GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
JRamirez-UAB
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
JRamirez-UAB
Model Tree
togethercomputer/HybridAgent-8B-Agentic
Input Modalities
Output Modalities
Supported Functionality
| File | What it is |
|---|---|
model.safetensors | the EXL3-packed text backbone (≈5.98 GB, ~5.0 bpw, H5) |
adapter.pt | the HybridLink latent-reasoning MLP, kept in BF16 (≈67 MB) |
adapter_config.json | the HybridLink config |
config.json | the backbone config (Qwen3ForCausalLM, 32K context) |
generation_config.json | generation config |
quantization_config.json | the EXL3 per-tensor bit allocation |
chat_template.jinja | the agentic chat template (required for tool use) |
tokenizer.json, tokenizer_config.json | tokenizer (incl. the <latent> / </latent> anchors) |
serving/ | the full deployment stack: the fork's V1 runner files, the hybridagent backend + processor + orchestrator, serve.py, and setup.sh |
README.md | this card |
Total ≈ 6.06 GB of weights. Not included: the 15.7 GB BF16 reference (the source of
the quantization — it is the original togethercomputer checkpoint); the OrcaSAQ2 plugin,
exllamav3 and vLLM (install them from their repositories — setup.sh does this).
| Check | Result |
|---|---|
Output parity vs the reference HybridGenerationEngine (same seed, questions, tools) | PASS — e.g. "trailing zeros of 100!" → 24, "capital of Japan" → Tokyo, same tool calls |
| Long-context needle (25K-token prompt) | PASS |
| Real opencode tool request (agentic) | soft-FAIL on EXL3 greedy — picks search instead of python; the answer is still correct (24) and coherent |
| Concurrent sessions (KV concurrency) | PASS — ≈3.1× the BF16 KV pool |
| VRAM really free under load | PASS — ≥100 MiB |
| Latent loop under EXL3 | coherent (345-step run, 0 degenerate steps) |
Speed (RTX 3090, eager, greedy):
| Engine | Text tok/s | Latent-heavy answer | Memory |
|---|---|---|---|
Official HybridGenerationEngine (transformers, BF16) | ~15–18 | ~69 s | ~15.7 GB |
| Fork, BF16 | ~32–34 | ~20 s | ~15.7 GB |
| Fork, EXL3 (this repo) | ~18–24 (decode 24.2 wall) | ~27 s | ~5.98 GB, 3.1× KV |
EXL3 prefill ≈2290 tok/s; a latent step ≈33 ms. The fork is ~2× faster than the official transformers engine; EXL3 trades ~40% per-request speed for ~3× KV concurrency and ~10 GB less VRAM.
EXL3 is not a native vLLM format, and this model needs pieces stock vLLM does not ship (the
zero-token latent loop). The stack, all in serving/:
cu132, torch 2.13.0) — the EXL3 kernels.vllm.general_plugins).hybridagent backend (serving/backend/) — Qwen3ForCausalLM + HybridLink, EXL3-capable load (vllm.general_plugins).hybridagent processor (serving/processor/) — the DECODE/LATENT/FEED state machine + blocked_token_sets (vllm.logits_processors).hybridagent orchestrator (serving/orchestrator/) — drives the in-process engine and the tool barrier.serving/fork/v1/) — five V1-runner files (request.py, worker/gpu_model_runner.py, core/kv_cache_manager.py, core/sched/{output,scheduler}.py) overlaid onto the installed vLLM.setup.sh does all of the above:
bash
git clone https://huggingface.co/JRamirez-UAB/hybridagent-8b-exl3-vllm-experimentalcd hybridagent-8b-exl3-vllm-experimental/serving./setup.shIf your system nvcc does not match, point CUDA_HOME at a CUDA 13.x toolkit (flashinfer
compiles some kernels on first start).
This is not the vllm serve CLI — the latent loop needs in-process engine access, so
the engine runs in this process and is driven by the HybridAgentOrchestrator:
bash
cd hybridagent-8b-exl3-vllm-experimentalexport VLLM_USE_V2_MODEL_RUNNER=0 # the fork is V1export VLLM_ENABLE_V1_MULTIPROCESSING=0 # in-process coreexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueexport VLLM_USE_FLASHINFER_SAMPLER=0HYBRIDAGENT_MODEL_DIR=$(pwd) venv/bin/python serving/serve.pyserve.py reads one prompt per line; a blank line runs all pending prompts in one
batched loop and prints their final texts; EOF (or Ctrl-C) exits. The validated budget is
--max-model-len 32768, kv_cache_memory_bytes=15e9 (≈101k tokens ≈ 3 concurrent full-32k
sessions), gpu_memory_utilization=0.93, enforce_eager=True, enable_prompt_embeds=True.
To serve the BF16 reference instead, point HYBRIDAGENT_MODEL_DIR at the BF16 dir and
set HYBRIDAGENT_KV_BYTES=5000000000.
The agentic system prompt + chat template are mandatory for tool triggering; without them the model answers plainly and never emits a tool call (they are baked into the orchestrator).
HybridGenerationEngine
(togethercomputer's transformers engine) with the same seed, questions and tools: the
fork reproduces the reference's answers (24 / 119 / Tokyo, same tool calls).serving/ + the repo's battery/) covers a long-context needle, a real
opencode tool request (its system prompt + tools, replayed against the engine), concurrent
sessions, and VRAM headroom. The EXL3 tool request is a soft-fail under greedy: it
picks search over python for the same question, but the final answer is correct.VLLM_USE_V2_MODEL_RUNNER=0 is required. The fork modifies the V1 runner
(v1/worker/gpu_model_runner.py); vLLM 0.30 defaults to V2, where the latent path would
be silently inactive.async_scheduling=False is forced by the orchestrator: the async scheduler keeps
num_new_tokens > 0 every step, which blocks the fork's zero-token latent gate.enable_prompt_embeds=True feeds the R(h) latent embeddings through the mixed-batch
prompt-embeds path; enforce_eager=True because the dynamic latent loop breaks CUDA
graphs.kv_cache_memory_bytes (deterministic against the
"≥100 MiB really free" criterion). EXL3 weights ≈5.98 GB + KV 15e9 B ≈14.2 GB + runtime
≈2 GB ≈22.2 GB of 24 GB → gpu_memory_utilization 0.93.search vs python); the answer is preserved. If exact tool parity
matters, serve the BF16 reference.int16 storage elements
of the EXL3 safetensors (≈4 weights per int16), not parameters — this is the 8B model at
≈5.0 bits per weight.| Step | What we did |
|---|---|
| Backbone | EXL3 at ~5.0 bpw (H5) via the exllamav3 SC second-order pipeline (per-tensor bit allocation) |
| HybridLink | not quantized — the latent MLP is kept in BF16 (adapter.pt) |
| Tokenizer | the EXL3 convert.py did not retain the tokenizer files; they are back-filled from the BF16 dir (the <latent> / </latent> anchors are preserved) |
| Hardware | the SC pipeline ran on a 251 GB-RAM node (the 24 GB GPU host's RAM does not fit it); ≈67 min end to end |
Source: togethercomputer/HybridAgent-8B-Agentic. Tokenizer, chat template and generation
config unchanged.
Artifact integrity
model.safetensors on start-up (a missing tensor or wrong
shape stops it); checked from a fresh download — the model reaches the ready state and serves.serving/, never the .safetensors.Quantization
quantization_config.json.adapter.pt); only the text
backbone is EXL3.Runtime
Scope: results apply to the stack (vLLM 0.30.0 fork + exllamav3 1.5.1 + OrcaSAQ2 +
hybridagent), hardware (one RTX 3090, 24 GB) and settings stated above; other setups are
not tested.
vllm serve) or stock transformers;
the zero-token latent loop is specific to the V1 runner fork in serving/fork/.togethercomputer/HybridAgent-14B-Agentic) but
is not quantized here.We would be grateful for reports of deployments of this checkpoint, both working and failed ones, especially on GPUs other than the RTX 3090. Please open a discussion in the Community tab with your GPU, driver, vLLM / exllamav3 / plugin versions, the full command line and the result (speed, context, or the error message).
togethercomputer/HybridAgent-8B-Agentic
by Together Computer, built on Qwen/Qwen3-8B-Base (MIT).hybridagent fork + backend/processor/orchestrator in serving/.bibtex
@misc{zou2026hybridagent, title={Hybrid Reasoning Agents}, author={Jiaru Zou and Yunzhe Qi and Sirui Chen and Federico Bianchi and Shang Zhu and Pan Lu and Jingrui He and James Zou}, year={2026}, eprint={2610.18861}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2610.18861},}License. The model weights are MIT (inherited from the base model). The OrcaSAQ2 plugin, exllamav3 and vLLM are not redistributed here; install them from their repositories under their own licenses.
JRamirez-UAB — questions about the EXL3 quantization or the experimental vLLM serving stack: open an issue or a discussion on this repo.
Part of the Experimental — HybridAgent on forked vLLM collection.
git clone https://huggingface.co/JRamirez-UAB/hybridagent-8b-exl3-vllm-experimentalcd hybridagent-8b-exl3-vllm-experimental/serving./setup.shcd hybridagent-8b-exl3-vllm-experimentalexport VLLM_USE_V2_MODEL_RUNNER=0 # the fork is V1export VLLM_ENABLE_V1_MULTIPROCESSING=0 # in-process coreexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueexport VLLM_USE_FLASHINFER_SAMPLER=0HYBRIDAGENT_MODEL_DIR=$(pwd) venv/bin/python serving/serve.py@misc{zou2026hybridagent, title={Hybrid Reasoning Agents}, author={Jiaru Zou and Yunzhe Qi and Sirui Chen and Federico Bianchi and Shang Zhu and Pan Lu and Jingrui He and James Zou}, year={2026}, eprint={2610.18861}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2610.18861},}