malvavisc0

OxCoder-9B-gptq-int4

README

License: apache-2.0

Why this artifact

Runs where the BF16 original can't. 17.6 GB BF16 → 8.5 GB Int4 fits a 12 GB GPU with room for KV cache. Same weights, same behavior, a third of the memory.
Proven recipe, third shipped artifact on this hybrid qwen3_5 architecture (GatedDeltaNet + attention): 200/200 text modules quantized, mean per-layer quant error 2.3e-5 (worst 1.1e-4) — better than the 4.2e-5 baseline of the first artifact in the series. Vision tower skipped by design (text-only calibration).
Calibrated on serving traffic. 256 rows: 128 OpenCodeReasoning CoT traces + 128 Nemotron-Agentic interactive tool-use episodes. The model sees, during calibration, the same kind of prompts it sees in production — code reasoning with think blocks, multi-step tool calls.
Ships with the froggeric v22.5 chat template (froggeric/Qwen-Fixed-Chat-Templates): XML tool calls, preserved think blocks, KV-cache-safe history, no empty-think poisoning. Calibrated and served through the same template, so serving matches calibration exactly.

Serving (vLLM)

bash
vllm serve malvavisc0/OxCoder-9B-gptq-int4 \
  --quantization gptq_marlin \
  --max-num-seqs 10 \
  --chat-template chat_template.jinja \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

Notes:

--max-num-seqs 10 — the hybrid GDN/Mamba cache corrupts state at high concurrency; 10 is the proven setting on this architecture.
Pass --chat-template chat_template.jinja explicitly — vLLM does not auto-load the sibling file from the model dir in current releases.
Sampling: temperature=0.6, top_p=0.95, top_k=20 (thinking mode). Greedy decoding causes repetition loops on this model class.

Hardware guidance

Table

Min GPU memory	12 GB (Int4 weights + KV headroom)
Verified	RTX A5000 16 GB — 5/5 behavioral probes; 45 tok/s decode (short coding), 40 tok/s at 31k context
Kernels	`gptq_marlin` — any Marlin-capable CUDA GPU (sm_80+)
Context	262K native; clamp `--max-model-len` on <16 GB cards

Quantization details

Table

Format	GPTQ Int4, group size 128, `desc_act`
Calibration	256 rows (128 OpenCodeReasoning + 128 Nemotron-Agentic), seq len 4096
Chat template	froggeric v22.5 (calibration + shipped)
Coverage	200/200 text modules; vision tower skipped (`--allow-partial-coverage`)
Provenance	`aft_provenance.json` (pinned source revision, seeds, config)
Per-layer log	`quant_log.csv`

Toolchain: quantized with aft (the Aria Finetuner pipeline) driving gptqmodel 7.3.4 / torch 2.13.0+cu130 / transformers 5.15.0 on an NVIDIA GB10 (DGX Spark, aarch64, 128 GB unified). Quant error: mean 2.3e-5, worst 1.1e-4 (200 text modules).

Verified on consumer hardware: RTX A5000 16 GB (sm_86, plain CUDA — no GB10 dependency) with vLLM 0.28.0: 5/5 behavioral probes (math with closed think blocks, exec-verified Kadane's incl. the all-negative case, parsed XML tool call), 45 tok/s decode on a short coding probe, 40 tok/s at 31k context, 0.06 s TTFT.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

malvavisc0

Model Tree

Base

OrionLLM/OxCoder-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality