GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
lgmc
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
lgmc
Model Tree
nvidia/Qwen3.6-35B-A3B-NVFP4
sakamakismile/Qwen-AgentWorld-35B-A3B-NVFP4
Input Modalities
Output Modalities
Supported Functionality
The AgentWorld NVFP4 fine-tune lost its MTP weights during training — they were never saved to the checkpoint. The base Qwen/Qwen3.6-35B-A3B model architecture supports MTP, so the weights can be borrowed from any compatible Qwen3.6-35B NVFP4 model that ships with them.
Additionally, the AgentWorld config.json was missing the re:^mtp.* entry in quantization_config.ignore, which caused vLLM's compressed-tensors handler to apply NVFP4 quantization to the MTP head. This made the internal parameter names mismatch the BF16 disk tensors, crashing with:
markdown
KeyError: 'layers.0.mlp.experts.routed_experts.w2_weight'
The working RedHatAI/Qwen3.6-35B-A3B-NVFP4 model has both the MTP tensors and the ignore entry, which is why it works out of the box.
To obtain the MTP head tensors, we use the nvidia/Qwen3.6-35B-A3B-NVFP4 model, which ships with its MTP head in BF16 (unquantized) inside the main checkpoint. Download it so vLLM can cache it locally:
bash
HF_HOME=<vllm-models-dir> \hf download nvidia/Qwen3.6-35B-A3B-NVFP4
scripts/merge_mtp_into_agentworld.py backs up the original AgentWorld snapshot to <hash>.orig_backup/ before touching anything. It extracts all 19 mtp.* tensors (BF16) from the nvidia/Qwen3.6-35B-A3B-NVFP4 model and writes them to model_mtp_head.safetensors in the AgentWorld snapshot, then creates model.safetensors.index.json pointing to both the original shard and the new MTP shard.Run the merge script with:
bash
python3 scripts/merge_mtp_into_agentworld.py
config.json: Added re:^mtp.* to the quantization_config.ignore list so vLLM's compressed-tensors handler does not apply NVFP4 quantization to the MTP head, ensuring the internal parameter names match the BF16 disk tensors.To enable MTP speculative decoding, use the --speculative-config flag when starting vLLM:
bash
vllm serve <model_path> \--tensor-parallel-size 1 \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Benchmarks were run on an ASUS Ascent GX10 — an NVIDIA DGX Spark system powered by the GB10 Grace Blackwell Superchip:
| Component | Spec |
|---|---|
| GPU | NVIDIA Blackwell GPU (GB10 Grace Blackwell Superchip) |
| CPU | 20-core ARM processor |
| Unified Memory | 128 GB LPDDR5x (coherent CPU+GPU) |
| CPU-GPU Interconnect | NVLink-C2C (5× PCIe 5.0 bandwidth) |
| Network | NVIDIA ConnectX-7 NIC + 10 GbE |
| AI Performance | Up to 1 PFLOP (FP4) |
| Form Factor | 1.6L compact desktop |
yaml
qwen-agentworld:image: vllm/vllm-openai:nightlycontainer_name: qwen-agentworldrestart: "no"ports:- "8011:8000"volumes:- ./vllm/models:/root/.cache/huggingface- ./vllm/torch_compile_cache_qwen_agentworld:/root/.cache/vllm/torch_compile_cacheenvironment:- NVIDIA_VISIBLE_DEVICES=all- NVIDIA_DRIVER_CAPABILITIES=compute,utility,display- HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN:-}- HF_HOME=/root/.cache/huggingface- VLLM_ALLOW_LONG_MAX_MODEL_LEN=1- PYTHONUNBUFFERED=1command: >--model lgmc/Qwen-AgentWorld-35B-A3B-NVFP4-MTP--host 0.0.0.0--port 8000--tensor-parallel-size 1--gpu-memory-utilization 0.78--max-model-len 131072--max-num-seqs 64--max-num-batched-tokens 32768--quantization compressed-tensors--kv-cache-dtype fp8--moe-backend flashinfer_cutlass--attention-backend FLASHINFER--enable-chunked-prefill--enable-prefix-caching--speculative-config '{"method":"mtp","num_speculative_tokens":3}'--trust-remote-code--reasoning-parser qwen3--tool-call-parser qwen3_coder--enable-auto-tool-choiceipc: hostdeploy:resources:reservations:devices:- driver: nvidiacount: allcapabilities: [gpu]
Start it with:
bash
docker compose up qwen-agentworld
The model will be available at http://localhost:8011/v1 (OpenAI-compatible API).
Measured with vllm bench serve on the ASUS Ascent GX10 using vllm/vllm-openai:nightly:
markdown
vllm bench serve \--backend vllm \--model lgmc/Qwen-AgentWorld-35B-A3B-NVFP4-MTP \--base-url http://localhost:8000 \--dataset-name random \--random-input-len 1024 \--random-output-len 512 \--num-prompts 200 \--max-concurrency 32
markdown
============ Serving Benchmark Result ============Successful requests: 200Failed requests: 0Maximum request concurrency: 32Benchmark duration (s): 220.58Total input tokens: 204800Total generated tokens: 102400Request throughput (req/s): 0.91Output token throughput (tok/s): 464.22Peak output token throughput (tok/s): 224.00Peak concurrent requests: 37.00Total token throughput (tok/s): 1392.67---------------Time to First Token----------------Mean TTFT (ms): 2171.01Median TTFT (ms): 756.89P99 TTFT (ms): 9620.94-----Time per Output Token (excl. 1st token)------Mean TPOT (ms): 62.49Median TPOT (ms): 57.48P99 TPOT (ms): 174.94---------------Inter-token Latency----------------Mean ITL (ms): 191.98Median ITL (ms): 170.41P99 ITL (ms): 410.02---------------Speculative Decoding---------------Acceptance rate (%): 69.32Acceptance length: 3.08Drafts: 33264Draft tokens: 99792Accepted tokens: 69179Per-position acceptance (%):Position 0: 78.34Position 1: 68.09Position 2: 61.54==================================================
Key takeaways:
scripts/merge_mtp_into_agentworld.py — merge script used to apply the MTP headNote: This model is a derivative work based on the Qwen research and the AgentWorld fine-tune. Please ensure compliance with the original model's license (Qwen Research License or similar) before use.