GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
kyr0
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
kyr0
Model Tree
protoLabsAI/Ornith-1.0-35B-FP8
Input Modalities
Output Modalities
Supported Functionality
My graft script does a bit of magic:
hf785 mtp.* tensors to model.safetensors.index.json778 MTP Linear modules as unquantized/ignorednum_nextn_predict_layers=1| Metric | Value |
|---|---|
| Successful / Failed Requests | 32 / 0 |
| Total Wall Time | 159.47 sec |
| Average Request Time | 19.11 sec |
| Max Request Time | 27.48 sec |
| Total Prompt Tokens | 1,944 |
| Total Completion Tokens | 119,769 |
| Total Tokens Processed | 121,713 |
| Completion Throughput |
bash
SERVED_NAME="ornith-35b-fp8-e4m3-mtp" \REQS="32" \PARALLEL="4" \MAX_TOKENS="4096" \TEMP="0.6" \TOP_P="0.95" \TOP_K="20" \PROMPTS_FILE="" \SHUFFLE_PROMPTS="1" \RANDOM_NONCE="1" \./stress.sh==> Stress test url=http://127.0.0.1:8998/v1/chat/completions model=ornith-35b-fp8-e4m3-mtp reqs=32 parallel=4 max_tokens=4096 prompts=/tmp/tmp.Lx42dinyo3/default-prompts.txt count=20 shuffle=1 nonce=1 id status seconds prompt_tokens completion_tokens total_tokens10 200 15.468765 65 3458 352311 200 17.042949 57 3243 330012 200 17.226863 57 3213 327013 200 19.818083 62 3677 373914 200 20.236654 77 4096 417315 200 18.975290 68 4096 416416 200 19.780207 62 4096 415817 200 17.789587 58 3381 343918 200 19.140477 57 3906 396319 200 19.095998 66 4096 41621 200 26.982888 61 3764 382520 200 21.414041 62 4096 415821 200 18.708545 53 4096 414922 200 18.280602 64 4096 416023 200 16.905085 55 3445 350024 200 19.210371 53 4096 414925 200 20.722689 77 4096 417326 200 18.112546 59 4096 415527 200 16.745625 60 3269 332928 200 19.964553 57 4096 415329 200 20.624861 77 4096 41732 200 24.681820 56 3746 380230 200 18.958027 62 4096 415831 200 16.092866 58 3277 333532 200 14.789030 65 3922 39873 200 27.475965 55 4096 41514 200 23.244902 55 3343 33985 200 19.629941 52 4096 41486 200 12.735978 62 2328 23907 200 20.976643 61 4096 41578 200 19.030427 52 4096 41489 200 11.501070 59 2165 2224 ==> Summaryok=32 fail=0wall_sec=159.467avg_req_sec=19.105 max_req_sec=27.476prompt_tokens=1944 completion_tokens=119769 total_tokens=121713completion_tok_per_sec=751.06total_tok_per_sec=763.25| Metric | Value |
|---|---|
| Successful / Failed Requests | 32 / 0 |
| Total Wall Time | 163.17 sec |
| Average Request Time | 19.56 sec |
| Max Request Time | 26.79 sec |
| Total Prompt Tokens | 2,028 |
| Total Completion Tokens | 103,571 |
| Total Tokens Processed | 105,599 |
| Completion Throughput | 634.73 tok/sec |
bash
make stresschmod +x ./stress.shPORT="8998" \API_KEY="local-dev-key" \SERVED_NAME="ornith-35b-fp8-e4m3-mtp" \REQS="32" \PARALLEL="4" \MAX_TOKENS="4096" \TEMP="0.6" \TOP_P="0.95" \TOP_K="20" \PROMPTS_FILE="" \SHUFFLE_PROMPTS="1" \RANDOM_NONCE="1" \./stress.sh==> Stress test url=http://127.0.0.1:8998/v1/chat/completions model=ornith-35b-fp8-e4m3-mtp reqs=32 parallel=4 max_tokens=4096 prompts=/tmp/tmp.0nT1f4XYq0/default-prompts.txt count=20 shuffle=1 nonce=1 id status seconds prompt_tokens completion_tokens total_tokens10 200 24.386988 57 4096 415311 200 24.412515 60 4096 415612 200 24.397616 76 4096 417213 200 14.089454 68 2362 243014 200 21.222193 67 3552 361915 200 13.490376 56 2276 233216 200 21.625402 62 3628 369017 200 18.498888 57 3095 315218 200 24.454264 61 4096 415719 200 17.620515 59 2972 30311 200 17.782799 61 2586 264720 200 19.214265 56 3233 328921 200 18.161814 74 3055 312922 200 24.359344 67 4096 416323 200 24.411447 57 4096 415324 200 24.442081 63 4096 415925 200 14.531877 60 2440 250026 200 13.009990 63 2177 224027 200 6.885480 58 1152 121028 200 24.296608 76 4096 417229 200 15.699833 60 2643 27032 200 18.248555 56 2656 271230 200 6.860422 59 1154 121331 200 23.504682 63 4096 415932 200 17.236493 72 3049 31213 200 26.789070 67 4096 41634 200 26.788966 76 4096 41725 200 21.046680 73 3479 35526 200 15.875098 62 2615 26777 200 15.088136 62 2481 25438 200 24.676287 54 4096 41509 200 22.677346 66 3814 3880 ==> Summaryok=32 fail=0wall_sec=163.173avg_req_sec=19.556 max_req_sec=26.789prompt_tokens=2028 completion_tokens=103571 total_tokens=105599completion_tok_per_sec=634.73total_tok_per_sec=647.16The following measurements were conducted on a single NVIDIA H200 NVL 141GB with --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. See the Makefile for the exact parameters used to run the vLLM server with the benchmarked configuration.
| Metric | Samples | Min | Max | Mean |
|---|---|---|---|---|
| Mean Acceptance Length | 16 | 2.24 | 2.67 | 2.38 |
| Avg Draft Acceptance Rate | 16 | 62.1% | 83.4% | 69.2% |
| Per-position Acceptance Rate P1 | 16 | 73.1% | 89.4% | 78.7% |
bash
vllm serve ./Ornith-1.0-35B-FP8-MTP \ --served-model-name ornith-35b-fp8-mtp \ --trust-remote-code \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --speculative-config '{"method":"mtp","num_speculative_tokens":2}'| Total Throughput | 763.25 tok/sec |
| Total Throughput | 647.16 tok/sec |
| Per-position Acceptance Rate P2 | 16 | 51.2% | 77.5% | 59.7% |