nuoram

Ornith-1.5-35B-A3B-abliterated-W8A8

README

License: apache-2.0

Lineage

Every link is pinned to the commit this artifact was actually built from.

Table with columns: step, source, revision
step	source	revision
foundation	`ornith-ai/Ornith-1.5-35B-A3B`	upstream release
abliteration (tower)	`huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated`	`7269f0953054d68b3cacbd9ad729921258732007`
MTP head	`shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY`	`2b19b31bfe1659c6b0d9459ec3cbd87e34a322ef`
quantisation	llm-compressor `QuantizationModifier`, recipe in `recipe.yaml`	this repo

The MTP head is not the one the base model ships, and that is deliberate. It is grafted from shisa-ai's distill. See "The MTP head" below, including a correction to a claim about the original head that circulates and is wrong.

Why this abliteration

Three abliterations of Ornith 1.5 35B-A3B are published. Only huihui-ai's keeps the model whole, and this quantisation preserves that:

Table with columns: this repo, the other published W8A8, the other abliterations
	this repo	the other published W8A8	the other abliterations
abliterated	yes	no	yes
architecture	`Qwen3_5MoeForConditionalGeneration`	`Qwen3_5MoeForCausalLM`	one downgrades to `ForCausalLM`
`model.visual.*` tensors	333	0

The two alternatives drop both the MTP head and the vision tower (64.6 GiB against the official 67.0 GiB, zero of each). A text-only checkpoint silently drops every image and OCR caller, which is a failure that looks like nothing at all: the model answers, it just cannot see.

What was quantised, and what was not

Scheme W8A8: 8-bit integer weights, symmetric, per channel, and 8-bit integer activations, dynamic. Targets Linear. No calibration data (data-free round-to-nearest), so nothing about this checkpoint is tuned to a calibration corpus.

Left in bf16, on purpose:

the whole vision tower, all 27 blocks and the merger
every MoE router: mlp.gate and shared_expert_gate on all 40 layers. Routers pick experts, and an 8-bit router picks different experts
the linear_attn modules and their norms. Ornith 1.5 is a hybrid, and the linear-attention state is precision-sensitive
lm_head
the entire MTP head (re:.*mtp.*)

The MTP head

The head here is shisa-ai's distill, not the one Ornith ships. Three reasons, and the first thing to say is what is not a reason.

The original head is trained. The widespread claim that it is not is false. The claim (from the model's own discussion #10) is that every projection has a standard deviation of exactly 0.0200, i.e. initializer_range. Probing the published safetensors directly on 2026-08-27 does not support it: mtp.norm.weight has mean +1.9251, mtp.layers.0.post_attention_layernorm.weight +0.8686, and the projection standard deviations vary by a factor of three (mtp.fc 0.0085, shared_expert.gate_proj 0.0089, mlp.gate 0.0096, o_proj 0.0185, q_proj 0.0247). A single init at initializer_range would put all of them at 0.0200 and leave the norms near 0.02. For scale, the same checkpoint's trained model.language_model.norm.weight is 1.6397 against the head's 1.9251. Please do not repeat the claim.

The actual reasons:

It accepts nearly twice as many drafted tokens. Measured in vLLM by shisa-ai and published alongside the head: Ornith's own head accepts 37.20% (mean accepted run 2.116), a zero-shot graft of the Qwen3.6 head 50.19% (2.506), and this head, a 12K KL distill over two mixed-data epochs warm-started from Ornith's own, 69.27% (3.078).
It transfers. It is a fine-tune rather than a retrain: element-wise against the official head the deltas are 0.5% to 8.7% of each tensor's range. A head only has to be accepted by the tower it drafts for, and this one is within a few percent of the head that already pairs with it.
Layout. Ornith ships the head's 256 experts unfused, as 785 separate mtp.layers.0.mlp.experts.N.* tensors. This head ships the 19-tensor fused form (experts.gate_up_proj [256, 1024, 2048], experts.down_proj [256, 2048, 512]).

DFlash was the other candidate and was rejected: its acceptance falls from 89.8% at 8K context to 48.8% near 256K, it is slower than plain autoregressive decode at 252K/8K (0.67x), and it reportedly breaks multimodal.

Why W8A8 rather than FP8 or NVFP4

This matters on Ampere and not much anywhere else. vLLM's W8A8 INT8 scheme declares a minimum compute capability of 75, so an A40 (8.6) multiplies in its own INT8 tensor cores. The FP8 W8A8 scheme declares 89, so below Lovelace vLLM resolves the checkpoint to the weight-only scheme and serves it through Marlin: the activations go back to 16 bits and the memory saving is real while the speedup is not. NVFP4 has no native FP4 path on Ampere at all.

So on Ada, Hopper and Blackwell, prefer the official FP8 build. On Ampere, this one.

Serving

bash
vllm serve nuoram/Ornith-1.5-35B-A3B-abliterated-W8A8 \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

Fits 2x 48GB (A40, A6000) with room for a long context.

Credit and licence

The model is ornith-ai's, the abliteration is huihui-ai's, and the MTP head is shisa-ai's. This repository changes the numeric format and nothing else. Licence follows the upstream Ornith release; if that differs from the Apache-2.0 declared here, upstream governs.

Abliterated models have had refusal behaviour removed. You are responsible for what you do with it.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

nuoram

Model Tree

Base

huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality