GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
malvavisc0
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
malvavisc0
Model Tree
empero-ai/Qwen3.8-9B
Input Modalities
Output Modalities
Supported Functionality
vllm serve command.| Hardware | Fit | Notes |
|---|---|---|
| RTX 4090 / 5090 (24 GB) | ✅ comfortable | Long contexts, batch serving |
| RTX 3090 / A5000 (16–24 GB) | ✅ | The sweet spot |
| RTX 3080 / 4070 Ti (12–16 GB) | ✅ | Clamp --max-model-len for long contexts |
| Apple Silicon (16 GB+ unified) | ✅ | Via vLLM or transformers |
| NVIDIA GB10 / DGX Spark (128 GB unified) | ✅ | Full 262k context, no clamps |
| CPU-only | ⚠️ | Works via transformers, slow — use a GGUF instead |
Rule of thumb: 8.5 GB weights + KV cache. 12 GB of total memory is the practical floor; 16 GB is comfortable; 24 GB+ lets the 262k context breathe.
bash
pip install "vllm>=0.27"vllm serve malvavisc0/Qwen3.8-9B-gptq-int4 \--quantization gptq_marlin --max-num-seqs 10
--max-num-seqs 10matters: this is a hybrid GatedDeltaNet/full-attention architecture, and vLLM's Mamba-style cache is happiest with a bounded batch. On 24 GB cards, add--max-model-len 32768(or less) if you don't need the full 262k.
Then talk to it over the OpenAI API:
python
from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="x")resp = client.chat.completions.create(model="malvavisc0/Qwen3.8-9B-gptq-int4",messages=[{"role": "user", "content":"A snail sits at the bottom of a 10 m well. Each day it climbs ""3 m, each night it slips back 2 m. How many days to escape?"}],temperature=0.6, top_p=0.95,extra_body={"top_k": 20, "max_tokens": 16384},)print(resp.choices[0].message.content)
python
from transformers import AutoModelForCausalLM, AutoTokenizerimport torchmodel_id = "malvavisc0/Qwen3.8-9B-gptq-int4"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")messages = [{"role": "user", "content": "Prove that √2 is irrational."}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)out = model.generate(inputs, max_new_tokens=16384,temperature=0.6, top_p=0.95, top_k=20, do_sample=True,)print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
temperature=0.6, top_p=0.95, top_k=20. Greedy decoding
causes repetition loops in reasoning models of this class.max_new_tokens ≥ 8192). <think>
blocks are long by design — the model deliberates, then answers. Strip
the <think>...</think> span for end users.<tool_call> format per Qwen3.5's spec;
pass tool definitions via the chat template's tools argument.reasoning_effort (low→xhigh) — dial it down for easy questions,
up for competition math.| Method | GPTQ Int4, group size 128, desc_act |
| Quantizer | gptqmodel 7.3.4 (torch 2.13, transformers 5.15) |
| Calibration | 256 samples: 96 math CoT (nvidia/OpenMathReasoning), 96 code CoT (nvidia/OpenCodeReasoning), 64 agentic tool-use (nvidia/Nemotron-Agentic-v1), rendered through the shipped chat template, seq len 4096 |
| Coverage | 200/200 quantizable text modules (vision tower excluded by design — text-only calibration can't represent it) |
| Quality | mean per-layer GPTQ error 2.3e-5, worst 1.1e-4 |
| Size | 8.5 GB (vs 18 GB BF16) |
| Provenance | aft_provenance.json + included in this repo |
Base revision: 0934f3d2327ff2df2197495278c4c46ae5a56bd9. Produced with
aft (Aria Finetuner).
empero-ai/Qwen3.8-9B — BF16 original + benchmark tableApache-2.0, inherited from the Qwen3.5-9B base via the upstream distillation. Credit to Empero for the model and the Qwen team for the architecture.
quant_log.csv