Krexoz

occamy-1.0-GPTQ-Int4-W4A16

README

License: apache-2.0

Quantization recipe

Table

Format	GPTQ, `pack-quantized`, int32
Bits / group size	4 / 128
Symmetric	yes (`uint4b8`)
`desc_act`	false
`lm_head`	not quantized
Quantizer	GPTQModel 7.5.0
Calibration	256 samples from `allenai/c4`, 2048 tokens each
Damp percent	0.05
Run time	8.6 h on one Arc Pro B70

What is deliberately left in BF16

The exclusions follow Arahide/Nex-N2.5-mini-INT4-W4A16 — an architecturally identical MoE checkpoint already proven to serve on this hardware — rather than GPTQModel's defaults.

Table with columns: Component, Tensors, Quantized
Component	Tensors	Quantized
Vision tower (`model.visual.*`)	333	0
*Gated DeltaNet (`.linear_attn.`)*	270	0
MoE experts	122,880	92,160
Full-attention projections	180	120

The Gated DeltaNet exclusion is the non-obvious part, and is the main thing this repository has to offer anyone repeating the exercise. GPTQModel's own qwen3_5_moe definition quantizes in_proj_qkv, in_proj_z and out_proj inside the linear_attn blocks. Every INT4 checkpoint proven on Intel XPU leaves that subtree in BF16, and the XPU GDN path runs through fused kernels, so it is excluded here:

python
dynamic={r"-:.*\.linear_attn\..*": {}}

MoE router gates, shared_expert_gate and the vision tower are already skipped by GPTQModel's definition.

Known limitation: 17.6% of expert linears fell back to RTN

Verifiable from the quant_log.csv shipped in this repository:

Table with columns: Modules, Share
	Modules	Share
Full GPTQ, with error compensation	25,459	82.44%
Round-to-nearest fallback	5,421	17.56%

Every fallback is an expert, concentrated in the later layers (L39: 324, L38: 303, L36: 276).

This is structural to MoE quantization rather than a mistake in this run: with 8-of-256 routing each expert sees roughly 1/32 of calibration tokens, so rarely-routed experts never accumulate a well-conditioned Hessian and GPTQModel rounds them without error compensation. For the 82.44% that got full treatment, loss is small — median 0.000085, p99 0.000923, max 0.003753.

Practical consequence: degradation is not uniform. Expect good output in general, with occasional weaker responses when a query routes to one of the RTN-rounded experts. A larger calibration set would shrink this share; 256 samples was chosen to keep the run inside one day on a single card.

Measured on an Arc Pro B70 (vLLM 0.29.0)

Loads through XPUwNa16LinearKernel with the XPU WNA16 MoE backend.

Table

Weights	20.57 GiB
KV cache	466,256 tokens (1.78x concurrency at the full 262,144 window)
Decode at 129k context	~52 tok/s
Prefill at 129k, cold cache	~2,600 tok/s
Model load	55 s

Reasoning at depth

A ~129k-token document with three facts planted at different depths; answering requires combining all three. Five runs per model, temperature 0.7, 3500-token cap. Nex-N2.5-mini is included because it is architecturally identical and quantized the same way, which makes it the fairest available reference point.

Table with columns: This model, Nex-N2.5-mini-INT4
	This model	Nex-N2.5-mini-INT4
Reasoning tokens (median)	1,946	2,755
Reasoning tokens (range)	1,521 – 2,064	1,638 – 3,111
Decode	52.3 tok/s	50.9 tok/s
Correct (all three facts, valid chain)	5/5	5/5
Truncated at the cap	0/5	3/5

Both retrieve and reason correctly at this depth. Occamy reaches the same answer with about 29% fewer thinking tokens — an understatement, since three of the five comparison runs were truncated — and with roughly a third of the run-to-run spread.

This says nothing about loss against the BF16 original, which was not measured. See Caveats.

Serving

yaml
services:
  occamy:
    image: vllm/vllm-openai-xpu:latest
    devices: ["/dev/dri:/dev/dri"]
    group_add: ["990", "44"]   # render, video — check with `getent group render video`
    shm_size: "16gb"
    ports: ["127.0.0.1:8000:8000"]
    volumes: ["/path/to/this/model:/model:ro"]
    environment:
      - VLLM_XPU_ENABLE_XPU_GRAPH=1
    command:
      - /model
      - --served-model-name
      - occamy
      - --trust-remote-code
      - --gpu-memory-utilization
      - "0.95"
      - --kv-cache-dtype
      - fp8
      - --max-num-seqs
      - "4"
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_coder
      - --reasoning-parser
      - qwen3

Recommendations

Sampling. The upstream model card recommends temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5. That presence penalty is unusually high — it is deliberate, and worth keeping rather than normalising to a familiar default.

VLLM_XPU_ENABLE_XPU_GRAPH=1 is not optional on this hardware for a hybrid-attention MoE. With graphs disabled, decode on this architecture falls by roughly 5x — not by a percentage. If throughput looks catastrophically wrong, check this first.

Keep vision. The tower was not quantized and is intact BF16. Do not pass --language-model-only unless you specifically want to drop it and reclaim the memory.

--max-num-seqs 4 is safe here. This checkpoint has no MTP head, so the concurrency restriction that applies to speculative-decoding checkpoints on XPU does not apply. An experimental MTP head is published separately as Accio-Lab/occamy-1.0-MTP and is untested with this quantization.

Thinking can be turned off. The chat template honours chat_template_kwargs: {"enable_thinking": false}. Note it does not implement reasoning_effort, unlike some other Qwen3.5-family templates. Pass such flags inside chat_template_kwargs, never at the top level of the request — vLLM silently discards unknown top-level fields.

Prefix caching earns its keep. vLLM enables it by default, and for agentic loops that resend a growing history each turn it removes almost all prefill cost. It matters far more than raw decode speed for that workload.

When not to use this. If you have the memory for BF16 or FP8, use those — this exists for cards that cannot hold them. If your work is dominated by rarely-exercised domains, the RTN-fallback experts above are a real consideration.

Reproducing this

Runs in the stock vllm/vllm-openai-xpu image with pip install gptqmodel==7.5.0 datasets. GPTQModel supports Intel XPU (Arc) natively and ships a qwen3_5_moe definition, so the architecture needs no custom handling — only the dynamic exclusion.

python
from gptqmodel import GPTQModel, QuantizeConfig

qcfg = QuantizeConfig(
    bits=4, group_size=128, sym=True, desc_act=False, lm_head=False,
    dynamic={r"-:.*\.linear_attn\..*": {}},   # Gated DeltaNet stays BF16
)
model = GPTQModel.load("/model", qcfg, device="xpu", trust_remote_code=True)
model.quantize(calibration, batch_size=1)      # 256 x 2048-token C4 samples
model.save("/output")

Timing, for planning: 8.6 h on one Arc Pro B70 — steady at ~12.5 min per layer across all 40, plus ~10 min of load and calibration. The job is bound by per-module Python overhead, not by the card: 30,880 small matrices each carry fixed framework cost, and the GPU idled roughly 60% of the time. GPTQModel warns at startup that Python 3.12 with the GIL disables multi-core packing; a free-threaded 3.13 base would likely cut this substantially.

Peak host RAM was ~8.8 GB — the 70 GB of BF16 weights are memory-mapped and streamed, so a 27 GB machine is enough.

Caveats

Not compared against the BF16 original on any standard benchmark. The comparison above is against a sibling checkpoint, which shows the quantization is healthy but does not quantify what was lost relative to full precision.
Calibration used generic web text (C4). Occamy is trained for agentic co-work, tool calls and code; domain-matched calibration would likely serve those paths better.
Validated on Intel XPU only. It should load anywhere GPTQ int4 group-128 is supported, but that has not been checked.
Single quantization run, 256 calibration samples.

Credits and provenance

Accio-Lab — occamy-1.0, the model this repository quantizes. All capability belongs to them; their technical report, project site and open training framework Dressage are the places to look for what the model actually is.
Qwen — Qwen3.6-35B-A3B, the base Occamy post-trains from.
ModelCloud — GPTQModel, including the qwen3_5_moe definition that made this a configuration problem rather than a porting one.

Licensed Apache-2.0, following the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Krexoz

Model Tree

Base

Accio-Lab/occamy-1.0

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality