thanet-s

Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10

Deploy Dedicated

README

License: other

Quantization

Dense attention and linear-attention projections: FP8 W8A8.
Routed MoE experts and shared expert projections: NVFP4 W4A4.
Preserved in BF16: embeddings, lm_head, router gates, visual modules, norms, Conv1D, A_log, dt_bias, and non-target tensors.
Visual encoder: copied from the source checkpoint and preserved in BF16 for multimodal/image support.
Calibration data: HuggingFaceH4/ultrachat_200k, train_sft.
Calibration samples: 512.
Calibration sequence length: 2048.
Pipeline: sequential, CPU offload, all MoE experts calibrated.

The quantization groups use mutually exclusive explicit target regexes.

Output

Format: mixed-precision.
Quantization method: compressed-tensors.
Config groups: group_0, group_1.
Weight layout: sharded safetensors with model.safetensors.index.json.
Safetensors shards: 5.
Indexed tensors: 124376.
Max position embeddings: 262144.
MTP weights: not present in the inspected source checkpoint.

DGX Spark vLLM

This model is intended for the GB10 patched vLLM path used by demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10.

Example:

bash
docker run --gpus all -p 8000:8000 --ipc host \
  -v /opt/vllm-cache:/root/.cache/huggingface \
  -e CUBLASLT_WORKSPACE_SIZE=33554432 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
  -e RUNAI_STREAMER_MEMORY_LIMIT=4294967296 \
  vllm/vllm-openai:v0.22.1 \
  --model thanet-s/Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10 \
  --served-model-name Ornith-1.0-35B-uncensored-heretic \
  --kernel-config '{"moe_backend": "flashinfer_b12x"}' \
  --load-format runai_streamer \
  --gpu-memory-utilization 0.50 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4176 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3

On DGX Spark / GB10, the vLLM image may still need the same CUTLASS DSL and flashinfer SM12x patch files described in the reference model card above.

DGX Spark benchmark

Measured on NVIDIA DGX Spark / GB10 with vllm/vllm-openai:v0.22.1, the GB10 CUTLASS DSL and flashinfer SM12x patches, --gpu-memory-utilization 0.50, --kv-cache-dtype fp8, --kernel-config {"moe_backend": "flashinfer_b12x"}, and --max-num-batched-tokens 4176.

vLLM log throughput during a warm single request: Avg generation throughput: 60.8 tokens/s.
Client-side measurement for the same warm request: 1024 completion tokens in 16.905s, about 60.58 tok/s.
First measured request after server readiness: 768 completion tokens in 13.676s, about 56.15 tok/s; vLLM log showed Avg generation throughput: 53.5 tokens/s.
Model load memory reported by vLLM: 21.04 GiB.
Max model length reported by vLLM: 262144 tokens.
GPU KV cache size at --gpu-memory-utilization 0.50: 3,344,321 tokens.
Maximum concurrency reported by vLLM for 262144-token requests: 12.76x.

These figures are from one local DGX Spark run and may change with prompt shape, sampling settings, vLLM version, patch versions, and CUDA graph/cache warmup state.

Status

Local DGX Spark conversion validation:

Source: llmfan46/Ornith-1.0-35B-uncensored-heretic.
Conversion completed with the same NVFP4-FP8Dense GB10 recipe used for the non-uncensored Ornith 35B build.
verify-output.py passed after vLLM config patching and visual tensor merge.
Visual tensors preserved: 333 model.visual.* tensors copied from the source checkpoint.
vLLM serving smoke test passed on DGX Spark / GB10 after upload.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

thanet-s

Model Tree

Base

llmfan46/Ornith-1.0-35B-uncensored-heretic

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality