GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Sherlock-Comms
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Sherlock-Comms
Model Tree
ornith-ai/Ornith-1.5-122B
Input Modalities
Output Modalities
Supported Functionality
| Official NVFP4 | This repo | |
|---|---|---|
lm_head.weight | NVFP4 packed, FP8 block scales, FP32 global scale | BF16 [248320, 3072] |
| Shards 1–7 | identical | |
| Shard 8 | lm_head replaced, its two scale tensors removed, other tensors identical | |
config.json | lm_head quantized | lm_head in ignore |
| Size | 77.6 GB | ~78.7 GB |
convert_lmhead_bf16.py in this repo does the conversion. It unpacks each FP4 (E2M1) value, applies the per-16-value FP8 scale and the global scale, and writes BF16.
Check: sampled rows match the original BF16 head from ornith-ai/Ornith-1.5-122B at cosine 0.995. The 9.6% relative error is the NVFP4 quantization error already in the official checkpoint.
bash
vllm serve <this-repo> --served-model-name ornith-1.5-122b \ --max-model-len 262144 --max-num-seqs 1 --max-num-batched-tokens 8192 \ --kv-cache-dtype fp8 --kv-cache-memory-bytes 12884901888 --no-enable-prefix-caching \ --speculative-config '{"method":"dflash","model":"incoai/Ornith-1.5-122B-DFlash2","num_speculative_tokens":7}' \ --mm-encoder-attn-backend TORCH_SDPA --limit-mm-per-prompt '{"image":1,"video":0}' \ --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \ --trust-remote-code--async-scheduling. It doesn't work with DFlash.TORCH_SDPA for the vision encoder, because flash-attn has no sm_121 build.Hardware and software:
| This checkpoint + DFlash2 (7 tokens), vLLM | Q4_K_M GGUF + MTP (1 token), llama.cpp on Windows | |
|---|---|---|
| Decode | ~50 tok/s (46–54 across runs, 23–93 per prompt) | 36.6 tok/s |
| Prefill at 33K / 136K / 256K tokens | 2,323 / 1,209 / 761 tok/s | 609 / 504 / 434 tok/s |
| Needle-in-a-haystack at 33K / 136K / 256K | all found | all found |
| Tool-calling test (54 runs) | 53/54, no invalid calls | 54/54 |
| Vision (reading a chart) | correct, 1.1 s | correct, 1.6 s |
| GPU memory | ~94 GB, ~97 GB peak at startup | ~84 GB |
MIT, the same as ornith-ai/Ornith-1.5-122B.
vllm serve <this-repo> --served-model-name ornith-1.5-122b \ --max-model-len 262144 --max-num-seqs 1 --max-num-batched-tokens 8192 \ --kv-cache-dtype fp8 --kv-cache-memory-bytes 12884901888 --no-enable-prefix-caching \ --speculative-config '{"method":"dflash","model":"incoai/Ornith-1.5-122B-DFlash2","num_speculative_tokens":7}' \ --mm-encoder-attn-backend TORCH_SDPA --limit-mm-per-prompt '{"image":1,"video":0}' \ --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \ --trust-remote-code