GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
JRamirez-UAB
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
JRamirez-UAB
Model Tree
TokenRhythm/NeoHorse-1-9B
Input Modalities
Output Modalities
Supported Functionality
| Test | main 6.00 bpw · 16 GB · 262,144 + vision | 4.50bpw · 12 GB · 176,000 + vision |
|---|---|---|
| KLD vs BF16, mean / median (our sysadmin trace; noise floor 0.0006) | 0.0015 / 0.0007 | 0.0058 / 0.0024 |
KLD on turboderp's public trace (qbench_prompts_gen.json), mean / median (noise floor 0.0006) | 0.0014 / 0.0003 | 0.0057 / 0.0010 |
| Peak memory used (budget) | 14,511 MiB (≈15,700) | 11,517 MiB (≈11,700) |
| KV pool | 267,185 tokens (5.25e9 B) | 177,955 tokens (3.6e9 B) |
| 8 facts in one long prompt | 8/8 at 242,097 tokens (190 s) | 8/8 at 158,732 tokens (101 s) |
| 2 simultaneous sessions | 2 × 117k, 8/8 in both | 2 × 75k, 8/8 in both |
| Decode with MTP 3, 1 request / 2 concurrent | 132 / 219-238 tok/s | 138-141 tok/s / — |
| Images (4 sysadmin screenshots, 15 checks) | 14/15 | 14/15 |
| Real opencode request, tool calls (3 cases × 10): xhigh · T 0.3 · no reasoning | 28/30 · 30/30 · 30/30 | 30/30 · 30/30 · 30/30 |
Agent: 10 sysadmin tasks with opencode run | 7/10 | 9/10 † |
| Agent: 4 tasks × 2 (logs, sshd, script, summary) | 8/8 | 7/8 † |
| Reasoning, 48 checkable problems (correct; total reasoning tokens) | low 42 (112k) · medium 44 (79k) · xhigh 46 (77k) | low 44 (86k) · medium 43 (88k) · xhigh 44 (68k) † |
| Coding: 4 browser games (static checks + headless run) | 44/48, 3/4 run | 44/48, 3/4 run † |
† Measured with the same 4.50bpw weights at 198,000 tokens / KV 4.0e9 (peak 11,899 MiB, slightly over a 12 GB budget); the 176,000 configuration
above was then validated for memory, needle, sessions, images and tools.
For reference, on the same tests AdQWENistrator-9B (same size, also quantized by us) scored 5/10 and 5/10 on the 10 agent tasks; Qwen3.5-9B base
needed ≈146k reasoning tokens at xhigh for 38-39/48.
| Revision | Target | Linear layers | lm_head | Vision | MTP head | Embeddings | Size |
|---|---|---|---|---|---|---|---|
main | 16 GB | 6.00 bpw (uniform) | 6 bits | 6 bits | 6 bits | INT8 per row | 7.2 GiB |
| Step | What we did |
|---|---|
| Grafting | Vision tower and MTP head copied unchanged from Qwen/Qwen3.5-9B (775 tensors; 48 kept in FP32 as in the original) |
main | exllamav3 1.5.1 convert.py -b 6.0 -hb 6 -vb 6 -mb 6 (uniform 6.00 bpw) |
4.50bpw calibration trace | Generated by NeoHorse itself with exllamav3's sc_trace (reasoning_effort: xhigh, MTP), 262,144-token budget |
4.50bpw allocation | sc_rfn_probe against a 2.00 bpw reference, , → per-tensor recipe |
Source: TokenRhythm/NeoHorse-1-9B revision ba5b6e40. Tokenizer and chat template unchanged (Qwen3.5 XML tool calls, qwen3_coder).
EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:
7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear
layers in vLLM through exllamav3's kernels, and INT8 embed_tokens for qwen3_5 models.qwen38_exl3_vision (ours, version 0.2.0, in plugin/qwen38_exl3_vision/): handles the EXL3 vision tower and pads input
dimensions where needed. It works for this Qwen3.5-architecture model as well as for Qwen3.8.Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:
bash
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/NeoHorse-1-9B-EXL3-16GB-12GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision16 GB GPU — main:
bash
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0hf download JRamirez-UAB/NeoHorse-1-9B-EXL3-16GB-12GB --local-dir NeoHorse-1-9B-EXL3vllm serve ./NeoHorse-1-9B-EXL3 \ --served-model-name neohorse-1-9b \ --limit-mm-per-prompt '{"image":1,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --max-model-len 262144 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 5250000000 \ --max-num-batched-tokens 8192 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder12 GB GPU — 4.50bpw: download with --revision 4.50bpw and use --max-model-len 176000 --kv-cache-memory-bytes 3600000000 --max-num-batched-tokens 4096
(peak 11,462 MiB used). 198,000 tokens with 4.0e9 B of KV also runs, but peaked at 11,899 MiB: only if your card leaves that much free.
Prompt processing (updated 2026-10-11): --max-num-batched-tokens 8192 (16 GB) / 4096 (12 GB) reads long prompts ≈22–27 % faster than the
1024 we used before (124k-token prompt: 2,236 vs 1,764 tok/s on the RTX 3090), and still fits: peak 15,260 / 11,462 MiB used, validated with the needle test
(8/8 at 233k / 157k), 2 concurrent sessions, vision and tool calls. If you run something else on the same GPU, go back to 1024 (≈1 GB lower peak on 16 GB).
generation_config.json (see the note at the top), or pass --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20}'.nvidia-smi --query-gpu=memory.used,memory.free --format=csv.
Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization. With MTP, KV capacity grows in block-sized steps: check the
GPU KV cache size line and keep --max-model-len below it.int16), not
parameters: this is the 9B model. The Configuration Parsing Warning about quantization_config.bits is expected (for EXL3 it is an average).Tool calling was measured with the real opencode request (captured and replayed against vLLM) and with opencode run agent tasks in a sandbox
with deterministic checks.
json
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "neohorse-1-9b": { "name": "NeoHorse 1 9B EXL3 (local)", "tool_call": true, "temperature": true, "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 262144, "output": 32000 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "low": { "chat_template_kwargs": { "reasoning_effort": "low" } }, "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } }, "agent": { "build": { "model": "local-vllm/neohorse-1-9b", "variant": "xhigh", "temperature": 0.6 }, "plan": { "model": "local-vllm/neohorse-1-9b", "variant": "xhigh", "temperature": 0.6 } }}limit.context = --max-model-len (176000 with the 4.50bpw branch). Declare "temperature": true, otherwise opencode sends no temperature.AGENTS.md or agent instructions.These come from the model, not from the quantization (KLD 0.0015 at 6.00 bpw); listed so you know what to watch:
errors.txt instead of errores.txt, processes.csv
instead of procesos.csv (the content is right). Measured over 340 agent tasks in Spanish: 5 % of tasks; a line in AGENTS.md asking for exact names
barely changes it (4 %). With the file names and paths written in backticks in the request (we did it automatically for every user message): 0 of 360
tasks, and +6 points of agent success. With the same tasks in English: 0 of 40 — it is a non-English problem. Same behaviour in Ornith-1.5-9B.
The model is not translating: it renames output files whose Spanish name is a system concept (processes, users, errors) to the conventional English name; names unrelated to the system (e.g. tortuga.txt) are kept (34/34). Backticks fix it and did not hurt when the user describes a system file in words or misspells its path (70/70).sacctmgr/scontrol, nvidia-smi --query-*, systemd timer syntax) it may guess
field names, subcommands or syntax that do not exist (, , ) and retry them; check its commands.
This is a knowledge limit of a 9B model, the same in Spanish and English; a generic "read first" instruction did not fix it. Documenting the tools
your agent uses (field names, correct syntax) in its instructions is the practical workaround.Field reports are very welcome in the Community tab: GPU, driver, vLLM version, context and KV settings, whether it worked or not, and what failed.
Apache License 2.0, as the original NeoHorse-1-9B and Qwen3.5-9B. See NOTICE for what was changed and the attribution. Unofficial release by
JRamirez-UAB; not affiliated with TokenRhythm, Alibaba Cloud or the Qwen team.
4.50bpw| 12 GB |
| 4.50 bpw, per-tensor (SC) allocation |
| 5 bits |
| 4 bits |
| 6 bits |
| INT8 per row |
| 5.8 GiB |
sc_measure --shapedsc_optimize -b 4.5 -hb 5 -al 2.0| Embeddings | embed_tokens re-encoded to INT8 per row (absmax / 127, BF16 scales) in the OrcaSAQ2 plugin's format (≈1 GiB less VRAM); script included |
--tool-call-parser qwen3_coder and --reasoning-parser qwen3. reasoning_effort accepts xhigh, medium and low, but in our measurements it does
not change how much the model thinks (see Known weak spots).scontrol show qos--query-processesOnCalendar=02:30 daily--helpreasoning_effort low/medium/xhigh made no difference. It also missed a systemd unit question that Ornith gets right.