thanet-s

Ornith-1.0-35B-heretic-NVFP4-FP8Dense-GB10

Deploy Dedicated

README

License: other

Quantization

Dense attention and linear-attention projections: FP8 W8A8.
Routed MoE experts and shared expert projections: NVFP4 W4A4.
Preserved in BF16: embeddings, lm_head, router gates, visual modules, norms, Conv1D, A_log, dt_bias, and non-target tensors.
Visual encoder: copied from the source checkpoint and preserved in BF16 for multimodal/image support.
Calibration data: HuggingFaceH4/ultrachat_200k, train_sft.
Calibration samples: 512.
Calibration sequence length: 2048.
Pipeline: sequential, CPU offload, all MoE experts calibrated.

The quantization groups use mutually exclusive explicit target regexes.

Output

Format: mixed-precision.
Quantization method: compressed-tensors.
Config groups: group_0, group_1.
Weight layout: single safetensors file.
Safetensors shards: 1.
Indexed tensors: 124376.
Max position embeddings: 262144.
MTP weights: not present in the inspected source checkpoint.

DGX Spark vLLM

This model is intended for the GB10 patched vLLM path used by demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10.

Example:

bash
docker run --gpus all -p 8000:8000 --ipc host \
  -v /opt/vllm-cache:/root/.cache/huggingface \
  -e CUBLASLT_WORKSPACE_SIZE=33554432 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
  -e RUNAI_STREAMER_MEMORY_LIMIT=4294967296 \
  vllm/vllm-openai:v0.22.1 \
  --model thanet-s/Ornith-1.0-35B-heretic-NVFP4-FP8Dense-GB10 \
  --served-model-name Ornith-1.0-35B-heretic \
  --kernel-config '{"moe_backend": "flashinfer_b12x"}' \
  --load-format runai_streamer \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4176 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

On DGX Spark / GB10, the vLLM image may still need the same CUTLASS DSL and flashinfer SM12x patch files described in the reference model card above.

This repository also includes an inference-only Docker Compose setup:

bash
./prepare-patches.sh
docker compose up -d
docker compose logs -f --tail=200 ornith-heretic

The compose file serves this model with vLLM. It does not run quantization or conversion.

Status

Local DGX Spark validation with vllm/vllm-openai:v0.22.1:

Loaded successfully as Qwen3_5MoeForConditionalGeneration.
Health endpoint returned 200.
Text chat completion endpoint responded.
Image chat completion endpoint accepted a data-URI PNG and identified the image color.
Observed single-request raw generation throughput: ~59.7 output tok/s on DGX Spark with vllm/vllm-openai:v0.22.1 and the GB10 patch path above.

Any throughput, quality, or long-context claims should still be based on your own serving test for the exact vLLM image and patch versions in use.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

thanet-s

Model Tree

Base

thanet-s/Ornith-1.0-35B-heretic

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality