Ornith-1.5-9B INT8 W8A8 (compressed-tensors)
Quantization of ornith-ai/Ornith-1.5-9B
to W8A8 INT8 (compressed-tensors int-quantized), produced with
llmcompressor 0.13.0 via
GPTQ W8A8-dynamic:
- weights: int8, per-channel, symmetric, static (mse observer)
- activations: int8, per-token, symmetric, dynamic (no baked scales)
- targets:
Linear
- ignore (bf16): vision tower + merger (
re:.*visual.*), lm_head,
MTP head (re:.*mtp.*), GatedDeltaNet SSM projections
(linear_attn.in_proj_a/in_proj_b/norm); 97 entries total
- calibration:
neuralmagic/LLM_compression_calibration (C4-en),
128 samples x 2048 tokens
Mirrors the recipe of
Avesed/Qwen3.8-27B-INT8-W8A8
(its SmoothQuant stage is omitted: llmcompressor 0.13's mapping inference
corrupts Qwen3_5 offset-norm modules with negative smoothing scales —
GPTQ-only rounds cleanly on this arch; verified per-tensor rel_L2 ~0.011
vs base).
Runs with sglang (--quantization compressed-tensors) — the checkpoint
ships the native Qwen3_5ForConditionalGeneration config (nested
text_config) with vision tower + MTP head included — and vLLM
(compressed-tensors >= 0.18).
Measured 2026-10-14 on RTX 3090 (int8, kvarn_k4v2_g128 KV):
prefill ~5800 tok/s; decode ~73 tok/s (bs=1) / ~500 tok/s aggregate (bs=8).
Source: quantized on 2026-10-14 for the orchestrel-ai self-hosted GPU fleet.