GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
dhruv1672
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
dhruv1672
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| Base model | Qwen/Qwen3.8-27B (qwen3_5 architecture) |
| Fine-tuning | LoRA, r=32, alpha=64, layers 48–63 only (last 16 of 64), 58.4M trainable params (0.21%) |
| Quantization | NVFP4 (4-bit, group size 16) via NVIDIA nvidia-modelopt 0.37.0 |
| Weights size | 18.73 GiB (vs. 51.75 GiB bf16 merged, vs. ~55.6 GiB base) |
| Speculative decoding | Native MTP draft head grafted from the base checkpoint (see below) |
| Serving | vLLM ≥0.30.0, FlashInferCutlassNvFp4LinearKernel (real NVFP4 GEMM) |
Qwen3_5ForConditionalGeneration: a hybrid model with GatedDeltaNet linear attention on 48 of 64
layers and full self-attention on every 4th layer, plus a Qwen3VL-style vision tower.model.visual.*) is present, unquantized, and functional — it was excluded from
both LoRA tuning and quantization since this fine-tune targets text-only Socratic tutoring. If you
need multimodal input, the base model's vision capability should still work through this checkpoint,
but it has not been evaluated for that use case here.mtp.*, ~425M params, reuses the main embedding table). This checkpoint includes those weights
(grafted back in — see below) so it can be served with real, model-native speculative decoding, not
a generic/self-drafted approximation.meric533/socrateach-sft
(8000 rows) + AndreiSobo/PACT-Socratic-Coding-Tutor
(2000 rows), no synthetic/templated seed data. Split 3-way with SHA-256 row-fingerprint leak checking:
train 7241 / val 493 / test 493. LoRA applied only to the last 16 of 64 layers (later layers govern
output style/behavior more than early layers) across attention, MLP, and GatedDeltaNet projections.
400 steps, eval_loss improved 0.7893 → 0.7766 (step 200 → 400).nvidia-modelopt, calibrated on 512 training rows. conv1d layers
(GatedDeltaNet), the vision tower, and lm_head are excluded from quantization — matching NVIDIA's
own official NVFP4 release recipe for this model family.export_hf_checkpoint().Two metadata bugs from the quantization export tooling were found and fixed along the way (both were incorrect module-path strings in the quantization config, not problems with the actual weight data): the vision tower's and the grafted MTP head's exclusion patterns needed correcting so vLLM's loader would recognize them as unquantized bf16 rather than trying to load them into NVFP4-shaped slots.
Benchmarked on an NVIDIA DGX Spark (GB10), 16 sequential Socratic-tutoring requests, max_tokens=180, concurrency=1:
| Serving profile | Aggregate tok/s | Decode tok/s (p50) | TTFT (p50) | Draft acceptance rate |
|---|---|---|---|---|
| No speculative decoding | 12.55 | 12.72 | 0.167 s | — |
| + n-gram speculative decoding | 14.24 | 13.25 | 0.105 s | 1.8% |
| + native MTP speculative decoding | 18.22 (+45.2%) | 18.61 (+46.3%) | 0.230 s |
n-gram speculation's 1.8% acceptance rate means its throughput number is most likely noise, not a real effect — n-gram only helps on repetitive text, and Socratic tutoring responses are free-form. MTP's 74.1% acceptance rate is a purpose-trained head predicting this exact model's own outputs, and the +45% throughput gain is directly attributable to it (at the honest cost of a modestly higher TTFT — the draft model's own forward pass on the prompt).
This ~12.5 tok/s baseline decode rate is close to the hardware's memory-bandwidth roofline for an 18.73 GiB model on GB10 (~273 GB/s published bandwidth → ~14.6 tok/s theoretical ceiling for batch-1 decode), consistent with a real, non-fluke quantization speedup rather than a measurement artifact.
Serve with vLLM (≥0.30.0), with MTP speculative decoding enabled:
bash
vllm serve dhruv1672/spartan-studybuddy-qwen3.8-27b-nvfp4 \ --trust-remote-code \ --served-model-name spartan-teacher \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Without speculative decoding, drop the --speculative-config flag — the model serves normally.
python
from openai import OpenAIclient = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local-edge")resp = client.chat.completions.create( model="spartan-teacher", messages=[{"role": "user", "content": "I'm implementing BFS and my traversal looks depth-first. Give me one Socratic hint."}], extra_body={"chat_template_kwargs": {"enable_thinking": False}},)print(resp.choices[0].message.content)Built as part of the Spartan StudyBuddy project. Base model, license, and
architecture credit: Qwen team, Qwen/Qwen3.8-27B (Apache 2.0).
Quantization via NVIDIA TensorRT Model Optimizer.
| 74.1% |