GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
bjk110
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
bjk110
Model Tree
wangzhang/Qwen3.5-122B-A10B-abliterix
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Intermediate Base Model | wangzhang/Qwen3.5-122B-A10B-abliterated |
| Original Base Model | Qwen/Qwen3.5-122B-A10B |
| Architecture | Qwen3.5 MoE (256 routed experts, 10B active) |
| Quantization | NVFP4 (compressed-tensors, nvfp4-pack-quantized) |
| Original Size | 228 GB (BF16) |
| Quantized Size | 71.2 GB (69% reduction) |
| Format | safetensors (2 shards) |
This model was quantized using a template-based weight replacement approach:
weight_packed and weight_scale) was regenerated from the abliterated BF16 weights using the reference checkpoint's weight_global_scale and input_global_scale values.config.json, quantization_config, global scales, and all metadata were preserved unchanged, ensuring full compatibility with vLLM's CUTLASS NVFP4 MoE kernel.| Component | Format | Notes |
|---|---|---|
| Routed experts (gate/up/down_proj) | NVFP4 | 256 experts × 48 layers × 3 projections |
| Shared experts | NVFP4 | 48 layers × 3 projections |
| Self-attention (q/k/v/o_proj) | NVFP4 | 12 full-attention layers |
| Linear attention | BF16 | 36 layers, kept at full precision |
| Embeddings, norms, gates | BF16 | Kept at full precision |
This model requires a text-only compatibility patch for vLLM since Qwen3.5 MoE is a multimodal architecture but this checkpoint contains only text weights.
bash
# 1. Download the modelhuggingface-cli download bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4 # 2. Apply the text-only patch before starting vLLMpython vllm_patches/patch_qwen35_moe_text.py # 3. Serve with vLLMvllm serve /path/to/model \ --served-model-name Qwen3.5-122B-A10B-abliterated-NVFP4 \ --max-model-len 131072 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.90 \ --trust-remote-code \ --enable-prefix-caching \ --enable-chunked-prefill \ --reasoning-parser qwen3 ### Docker Compose (Recommended) A complete Docker Compose setup is provided in the `serving/` directory: ```bash# Copy serving filescp -r serving/ /path/to/your/vllm-setup/ # Edit .env to set MODEL_PATHvim serving/.env # Startcd serving && docker compose --profile head up -dSee serving/ directory for:
docker-compose.yml — Full vLLM serving configuration.env.example — Environment variables templateentrypoint.sh — Entrypoint with automatic patch application| Configuration | Memory | max_model_len | Notes |
|---|---|---|---|
| 1× NVIDIA DGX Spark (GB10) | 121 GiB unified | 131,072 (128K) | Tested and verified |
| 1× GPU with 80+ GB VRAM | 80 GiB | ~65,536 | Estimated |
| Metric | Value |
|---|---|
| Throughput | 14.5 tok/s average, 16.8 tok/s peak |
| KV Cache | 222K tokens (20.4 GiB) |
| Max Concurrency | 6.16× at 128K context |
| Model Loading | ~13 min (2 shards) |
# 1. Download the modelhuggingface-cli download bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4 # 2. Apply the text-only patch before starting vLLMpython vllm_patches/patch_qwen35_moe_text.py # 3. Serve with vLLMvllm serve /path/to/model \ --served-model-name Qwen3.5-122B-A10B-abliterated-NVFP4 \ --max-model-len 131072 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.90 \ --trust-remote-code \ --enable-prefix-caching \ --enable-chunked-prefill \ --reasoning-parser qwen3 ### Docker Compose (Recommended) A complete Docker Compose setup is provided in the `serving/` directory: ```bash# Copy serving filescp -r serving/ /path/to/your/vllm-setup/ # Edit .env to set MODEL_PATHvim serving/.env # Startcd serving && docker compose --profile head up -d