00quebec

Ornith-1.0-397B-abliterated-W4A16

Highlights

Table

Base	`deepreinforce-ai/Ornith-1.0-397B` (Qwen3.5-MoE, 397B total / ~17B active)
Architecture	60 layers, hidden 4096, 512 experts (top-10), hybrid Gated-DeltaNet linear + full attention (45 + 15), MTP head, SigLIP-style vision tower (image + video)
Context	262,144 tokens (256K)
Quantization	W4A16 — AutoRound RTN, int4, group-size 128, symmetric, `auto_round:auto_gptq` packing
Kept at BF16	embeddings, `lm_head`, all routers (`mlp.gate`, `shared_expert_gate`), norms, and the entire vision tower
Size	~196 GB, 47 safetensors shards
Runtime	vLLM ≥ 0.17 (tested 0.20.1), served across 2× DGX Spark via Ray TP=2

Validation (before → after abliteration)

Measured on the 2-node vLLM cluster; refusal on mlabonne/harmful_behaviors (N=40, identical prompts), KL over top-k first-token logprobs on mlabonne/harmless_alpaca (N=40).

Table with columns: Metric, Reference W4A16, This model (abliterated)
Metric	Reference W4A16	This model (abliterated)
Refusal rate	30.0 %	7.5 % (−75 %)
KL divergence (harmless first-token)	—	0.116 — capability preserved
Coding smoke-test	—	✅ works
Image understanding	—	✅ works (OCR + scene)
Video understanding	—	✅ works (frames + motion)

Serving performance (2× DGX Spark, TP=2 over ConnectX-7, single-stream):

Table with columns: input ctx, decode tok/s, prefill tok/s, TTFT
input ctx	decode tok/s	prefill tok/s	TTFT
~1K	22	824	1.1 s
~8K	23	1,102	6.6 s
~128K	18	886	134 s
~256K	17	704	329 s

Decode is ~flat across context length because the hybrid linear-attention layers keep KV small; the single-stream ceiling is the per-token cross-node all-reduce over the 200 GbE ConnectX-7 link (no NVLink).

How it was built

Refusal directions. A memory-frugal layer-streaming forward over the local quant (per-layer gptq→bf16 dequant, no full model in memory) computed Arditi difference-of-means refusal directions r_ℓ ∈ ℝ⁴⁰⁹⁶ on 128 harmful vs. 128 harmless prompts (per-layer directions, adjacent-layer cosine ≈ 0.91).
Abliterate + re-quantize (streaming). The 122 BF16 shards of the base were streamed from the Hub one at a time and the residual-write matrices of the LM were orthogonalized against the refusal direction — W ← W − r rᵀW for self_attn.o_proj, linear_attn.out_proj, mlp.shared_expert.down_proj, and per-expert mlp.experts.down_proj (hidden = output axis) — then re-quantized with AutoRound's quantize_weight_rtn in the exact auto_round:auto_gptq layout, un-fusing experts to experts.{e}.{gate,up,down}_proj. Output is byte-identical to the reference AutoRound quant on every non-abliterated tensor.

Never staged the full 794 GB BF16 model — the whole pipeline is streaming + data-free (AutoRound model_free/RTN, no calibration).

Serving (vLLM, 2× DGX Spark)

The model (~196 GB) exceeds a single 128 GB Spark, so serve it across both nodes with tensor-parallel = 2 (Ray). Grade/serve config that works on GB10:

bash
vllm serve /path/to/this-model \
  --served-model-name ornith --trust-remote-code \
  --tensor-parallel-size 2 --distributed-executor-backend ray \
  --max-model-len 262144 --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 --max-num-seqs 1 \
  --enable-chunked-prefill --max-num-batched-tokens 8192 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image":48,"video":4}'

GB10 notes: pin NCCL to the ConnectX-7 interface; disable Ray's OOM monitor (RAY_memory_monitor_refresh_ms=0) since on unified memory the weights legitimately use most of each node's 121 GiB; --enable-chunked-prefill is required for long context (uncapped prefill OOMs); nvidia-smi reports memory as N/A on GB10 — use free -h.

Notes for agent frameworks

Reasoning model: replies begin with an inline Thinking Process: chain-of-thought in message.content (no separate reasoning field). Budget max_tokens accordingly and treat content as reasoning.
Tool calls: with --enable-auto-tool-choice --tool-call-parser qwen3_coder, OpenAI-style tools/tool_choice:"auto" return structured tool_calls (verified). The CoT still appears in content alongside the tool call.

Files

config.json, quantization_config.json, model-*-of-*.safetensors (+ index), tokenizer.json/tokenizer_config.json/vocab.json, chat_template.jinja, generation_config.json, preprocessor_config.json, processor_config.json, video_preprocessor_config.json. See docs/ for the full validation report and an agent-integration brief, and scripts/ for the abliteration/quant/serve/eval pipeline used to build this.

License & credits

MIT (inherited from the base model). Base model: deepreinforce-ai/Ornith-1.0-397B. Abliteration follows the residual-direction method (Arditi et al.; tooling inspired by p-e-w/heretic and elder-plinius/OBLITERATUS); quantization via Intel AutoRound. Not affiliated with or endorsed by the base-model authors.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

00quebec

Model Tree

Base

deepreinforce-ai/Ornith-1.0-397B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Highlights

Table

Base	`deepreinforce-ai/Ornith-1.0-397B` (Qwen3.5-MoE, 397B total / ~17B active)
Architecture	60 layers, hidden 4096, 512 experts (top-10), hybrid Gated-DeltaNet linear + full attention (45 + 15), MTP head, SigLIP-style vision tower (image + video)
Context	262,144 tokens (256K)
Quantization	W4A16 — AutoRound RTN, int4, group-size 128, symmetric, `auto_round:auto_gptq` packing
Kept at BF16	embeddings, `lm_head`, all routers (`mlp.gate`, `shared_expert_gate`), norms, and the entire vision tower
Size	~196 GB, 47 safetensors shards
Runtime	vLLM ≥ 0.17 (tested 0.20.1), served across 2× DGX Spark via Ray TP=2

Validation (before → after abliteration)

Measured on the 2-node vLLM cluster; refusal on mlabonne/harmful_behaviors (N=40, identical prompts), KL over top-k first-token logprobs on mlabonne/harmless_alpaca (N=40).

Table with columns: Metric, Reference W4A16, This model (abliterated)
Metric	Reference W4A16	This model (abliterated)
Refusal rate	30.0 %	7.5 % (−75 %)
KL divergence (harmless first-token)	—	0.116 — capability preserved
Coding smoke-test	—	✅ works
Image understanding	—	✅ works (OCR + scene)
Video understanding	—	✅ works (frames + motion)

Serving performance (2× DGX Spark, TP=2 over ConnectX-7, single-stream):

Table with columns: input ctx, decode tok/s, prefill tok/s, TTFT
input ctx	decode tok/s	prefill tok/s	TTFT
~1K	22	824	1.1 s
~8K	23	1,102	6.6 s
~128K	18	886	134 s
~256K	17	704	329 s

How it was built

Refusal directions. A memory-frugal layer-streaming forward over the local quant (per-layer gptq→bf16 dequant, no full model in memory) computed Arditi difference-of-means refusal directions r_ℓ ∈ ℝ⁴⁰⁹⁶ on 128 harmful vs. 128 harmless prompts (per-layer directions, adjacent-layer cosine ≈ 0.91).
Abliterate + re-quantize (streaming). The 122 BF16 shards of the base were streamed from the Hub one at a time and the residual-write matrices of the LM were orthogonalized against the refusal direction — W ← W − r rᵀW for self_attn.o_proj, linear_attn.out_proj, mlp.shared_expert.down_proj, and per-expert mlp.experts.down_proj (hidden = output axis) — then re-quantized with AutoRound's quantize_weight_rtn in the exact auto_round:auto_gptq layout, un-fusing experts to experts.{e}.{gate,up,down}_proj. Output is byte-identical to the reference AutoRound quant on every non-abliterated tensor.

Never staged the full 794 GB BF16 model — the whole pipeline is streaming + data-free (AutoRound model_free/RTN, no calibration).

Serving (vLLM, 2× DGX Spark)

The model (~196 GB) exceeds a single 128 GB Spark, so serve it across both nodes with tensor-parallel = 2 (Ray). Grade/serve config that works on GB10:

bash
vllm serve /path/to/this-model \
  --served-model-name ornith --trust-remote-code \
  --tensor-parallel-size 2 --distributed-executor-backend ray \
  --max-model-len 262144 --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 --max-num-seqs 1 \
  --enable-chunked-prefill --max-num-batched-tokens 8192 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image":48,"video":4}'

Notes for agent frameworks

Reasoning model: replies begin with an inline Thinking Process: chain-of-thought in message.content (no separate reasoning field). Budget max_tokens accordingly and treat content as reasoning.
Tool calls: with --enable-auto-tool-choice --tool-call-parser qwen3_coder, OpenAI-style tools/tool_choice:"auto" return structured tool_calls (verified). The CoT still appears in content alongside the tool call.

Ornith-1.0-397B-abliterated-W4A16

README

Highlights

Validation (before → after abliteration)

How it was built

Serving (vLLM, 2× DGX Spark)

Notes for agent frameworks

Files

License & credits

Explore FriendliAI today

README

Highlights

Validation (before → after abliteration)

How it was built

Serving (vLLM, 2× DGX Spark)

Notes for agent frameworks

Files

License & credits