BlivionIaG

Ornith-1.5-9b-exl3-3bpw

Deploy Dedicated

README

License: apache-2.0

Ornith Highlights

Ornith features the following enhancements:

Unified Vision-Language Foundation: Early fusion training on multimodal tokens achieves cross-generational parity with prior models and outperforms across reasoning, coding, agents, and visual understanding benchmarks.
Efficient Hybrid Architecture: Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency overhead.
Multi-Token Prediction: Trained with multi-step speculative decoding for ~2× decode speedup on supported backends.
Long Context: 262,144 tokens natively, suitable for hour-scale video understanding and long-document tasks.
Global Linguistic Coverage: Expanded support for 201 languages and dialects.

Model Overview

Type: Causal Language Model with Vision Encoder
Training Stage: Pre-training & Post-training
Language Model
- Number of Parameters: 9B
- Hidden Dimension: 4096
- Token Embedding: 248320 (Padded)
- Number of Layers: 32
- Hidden Layout: 8 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))
- Gated DeltaNet:
  - Number of Linear Attention Heads: 16 for V and 16 for QK
  - Head Dimension: 128
- Gated Attention:
  - Number of Attention Heads: 16 for Q and 4 for KV
  - Head Dimension: 256
  - Rotary Position Embedding Dimension: 64
- Feed Forward Network:
  - Intermediate Dimension: 12288
Context Length: 262,144 natively

Quickstart

[!Important] This checkpoint targets AMD RDNA2 / RDNA3 GPUs (gfx1030, gfx1100, gfx1200) under ROCm. The visual encoder is not shipped — launch with --language-model-only. CDNA (gfx942, gfx950) is not supported.

[!Important] This quantisation requires the rdna2_extras branch of the BlivionIaG/vllm fork of vLLM. The upstream vllm package does not support the exl3 RDNA path.

Serving with vLLM

shell
source /home/chenco_adm/Apps/vllm/venv-7.14.0/bin/activate

export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=0
export VLLM_ROCM_USE_AITER_MOE=0
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export VLLM_RDNA_FORCE_FP16=1
export TORCH_BLAS_PREFER_HIPBLASLT=0
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export GPU_MAX_HW_QUEUES=2
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_P2P_LEVEL=pix
export RCCL_P2P_NET_DISABLE=1
export RCCL_P2P_BATCH_ENABLE=1
export NCCL_PROTO=Simple
export RCCL_MSCCL_ENABLE=0
export VLLM_FORCE_CUSTOM_ALL_REDUCE=1

export VLLM_EXL3_FOLDED_CACHE=$HOME/.cache/exl3_folded_cache
mkdir -p "$VLLM_EXL3_FOLDED_CACHE"

python -m vllm.entrypoints.cli.main serve \
    /path/to/Ornith-1.5-9b-exl3-3bpw \
    --port 8000 \
    --dtype float16 \
    --tensor-parallel-size 4 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.90 \
    --language-model-only \
    --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

Multi-Token Prediction (MTP): add --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' to the serve command.

Throughput Benchmark:

shell
python -m vllm.entrypoints.cli.main bench throughput \
    /path/to/Ornith-1.5-9b-exl3-3bpw \
    --dtype float16 \
    --tensor-parallel-size 4 \
    --max-model-len 2048 \
    --input-len 1024 \
    --output-len 256 \
    --num-prompts 32 \
    --max-num-seqs 8 \
    --gpu-memory-utilization 0.85 \
    --language-model-only

Quantisation details

Quantised with exllamav3 using the 3inst codebook with the following recipe:

bash
convert.py -r \
  -i Ornith-1.5-9B \
  -w work-ornith-15-9b-3inst \
  -o Ornith-1.5-9b-exl3-3bpw \
  -ss 8192 -b 3.0 -hb 6 -mb 4 \
  -cr 256 -cc 2048 -cpi 120 \
  -v -d 0 -cb 3inst --out_scales always

Table with columns: Component, Scheme, Bits, Notes
Component	Scheme	Bits	Notes
32 transformer layers (body)	EXL3 3inst	3 bpw	RDNA2 trellis kernel + Hadamard sandwich
`lm_head`	EXL3 3inst	6 bpw	Dense fp16 dequant at load
MTP (`mtp.*`)	EXL3 3inst	4 bpw	Loaded only with `--speculative-config qwen3_next_mtp`

Single 5.53 GB model.safetensors (860 tensors).

Best Practices

To achieve optimal performance on RDNA2 / RDNA3:

VLLM_USE_V2_MODEL_RUNNER=1 + cudagraph_mode=FULL_AND_PIECEWISE for highest throughput.
Force FP16 (VLLM_RDNA_FORCE_FP16=1) — gfx1030 lacks native BF16 hardware.
PCIe P2P + custom all-reduce for TP>1 (NCCL_P2P_LEVEL=pix, VLLM_FORCE_CUSTOM_ALL_REDUCE=1).
Queue budget: GPU_MAX_HW_QUEUES=2 to keep within gfx1030's 8-HQD limit.

Provenance

Original: Ornith-1.5-9B
Quantiser: exllamav3 convert.py with codebook=3inst, calibration rows=256 cols=2048 cpi=120
vLLM exl3 path: BlivionIaG/vllm rdna2_extras branch
Decoder kernel: RDNA2 exl3_gemm_rdna2 + exl3_hadamard_128 (Wave32, fp16)

Citation

If you use this quantisation, please cite both the original Ornith model and the RDNA2 exl3 path:

bibtex
@misc{ornith-1.5-9b-exl3-3bpw,
    title  = {{Ornith-1.5-9B EXL3 3bpw}: RDNA2/RDNA3 quantisation of Ornith-1.5-9B},
    author = {{Ornith Team} and {kletorch}},
    month  = {August},
    year   = {2026},
    url    = {https://huggingface.co/BlivionIaG/Ornith-1.5-9b-exl3-3bpw}
}

@misc{ornith-1.5-9b,
    title  = {{Ornith-1.5-9B}: Towards Native Multimodal Agents},
    author = {{Ornith Team}},
    month  = {February},
    year   = {2026},
    url    = {https://huggingface.co/Ornith-1.5-9B}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

BlivionIaG

Model Tree

Base

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality