GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
cagyirey
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
cagyirey
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
Qwen3.6-35B-A3B-Espalier is a quantized version of Alibaba's Qwen3.6-35B-A3B, a hybrid-attention (Gated DeltaNet and full attention) Mixture-of-Experts language model with a multi-token-prediction (MTP) head. Its weights are stored as trellis codes on hardware number grids and decoded inside the matmul kernels, so DRAM traffic is the compressed bytes.
The checkpoint is 11 GB on disk, with ~10.2 GB of weights in VRAM (text only), and is meant for consumer GPUs. Validated with vLLM on:
Apache-2.0, as the base model (license, included
unchanged as LICENSE). The weights are the Qwen team's; no copyright is claimed in them. See
Changes from the original.
Hugging Face 10/17/2026 via https://huggingface.co/cagyirey/Qwen3.6-35B-A3B-Espalier
Architecture Type: Transformer
Network Architecture: Hybrid attention (Gated DeltaNet and full attention) with Mixture-of-Experts
(Qwen3_5MoeForConditionalGeneration), 256 routed experts plus a shared expert
Number of Model Parameters: 35B total, 3B activated, plus the MTP head
Developed from: Qwen3.6-35B-A3B
Input: text (the vision tower ships in the files but is untested with the plugin; serve text only) Output: text
Runtime engine: vLLM 0.31.0 with vllm-espalier 0.1.0 (container ghcr.io/geodesa-ai/vllm-espalier:0.1.0)
Supported hardware: NVIDIA Ampere and newer (sm80+). Validated end to end on Ada (sm89); the expert kernels are
also validated on Hopper (sm90) and Blackwell (sm100, sm120).
Operating system: Linux
Tensor parallel size: 1
Espalier 2.12 bpw, format version 1.
RTX 4070 Ti (12 GB), batch 1, greedy, 256 tokens, FP8 KV cache, from a cold compile:
| tok/s | |
|---|---|
| decode | 181 |
| decode with MTP speculative decoding (3 draft tokens) | 304 |
| prefill (512 to 8k tokens) | ~1,650 |
Decode is the mean over three prompts (prose, code, an edit task). Speculative output was checked token by token against plain decoding up to the end of each answer: every disagreement is a near-tie.
KL divergence of the next-token distribution to the BF16 model on held-out text windows (prose, code, multilingual); lower is better. Same windows and probe for every row.
| Checkpoint | Bits/weight | Prose | Code | Multilingual | Pooled |
|---|---|---|---|---|---|
| BF16 | 16 | 0 | 0 | 0 | 0 |
| Espalier | 2.12 (decoder) | 0.0704 | 0.0397 | 0.0919 | 0.0673 |
| published EXL3 | 2.49 |
16% lower KL than the 2.49 bpw EXL3 at about 1.7 GB fewer bytes, 37% lower at equal bytes. On an out-of-distribution slice KL is 0.0984; the final tuning step lowered in-distribution KL by 8% and raised this slice's by 2.0%.
| Component | Stored precision |
|---|---|
| Decoder routed experts | 1, 2 or 3-bit trellis codes, width set per expert (2.0 bits average); E2M1 values at 1-2 bits, int5 grid at 3 bits |
| Decoder attention, Gated DeltaNet and shared expert | 4-bit trellis codes, int6 grid |
| LM head | 6-bit trellis codes |
| MTP head routed experts | 3-bit trellis codes |
| MTP head attention, MLP projection and norms | BF16 |
| Embedding, norms, router and shared-expert gates | BF16 |
| Vision tower | BF16, unchanged |
Each trellis-coded matrix carries per-channel input and output scales and is wrapped in 128-wide Hadamard rotations, as in ExLlamaV3's EXL3 format.
Image: ghcr.io/geodesa-ai/vllm-espalier:0.1.0
(vLLM 0.31.0, kernels prebuilt for sm80 through sm120).
Simplified run command (OpenAI-compatible server on port 8000, with the validated settings):
sh
docker run -d --name espalier --gpus all --ipc host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ ghcr.io/geodesa-ai/vllm-espalier:0.1.0The defaults serve cagyirey/Qwen3.6-35B-A3B-Espalier with MTP speculative decoding (3 tokens), FP8 KV, up to
2 concurrent requests, and the Qwen3 reasoning and tool-call parsers. The KV cache gets the GPU's memory minus
~11.5 GiB (512 MiB, ~13k tokens, on a 12 GB card; ~12 GiB on a 24 GB card, enough for the full 262k context) and the context is the
longest that fits. Extra arguments go to vllm serve and override the defaults, for example --max-model-len 8k.
Environment variables: PORT, MAX_MODEL_LEN, MAX_NUM_SEQS, and KV_CACHE_MEMORY_BYTES to set the KV budget
directly.
Two-step verbose commands. Start the container:
sh
docker run --rm -it --gpus all --ipc host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --entrypoint /bin/bash \ ghcr.io/geodesa-ai/vllm-espalier:0.1.0Then launch vLLM inside it:
sh
vllm serve cagyirey/Qwen3.6-35B-A3B-Espalier \ --host 0.0.0.0 \ --port 8000 \ --language-model-only \ --kv-cache-dtype fp8 \ --attention-backend TRITON_ATTN \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \ --kv-cache-memory-bytes 536870912 \ --max-model-len auto \ --max-num-seqs 2 \ --compilation-config '{"cudagraph_capture_sizes": [1, 2, 4, 8]}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coderThat is the 12 GB setting; with more memory raise --kv-cache-memory-bytes (about total VRAM − 11.5 GiB).
Why the explicit settings: with FP8 KV, vLLM otherwise picks FlashInfer, whose workspace does not fit in 12 GB;
vLLM's own KV sizing (--gpu-memory-utilization) first profiles a full-length request and runs out of memory; and
larger CUDA-graph sizes than 8 tokens do not fit next to the weights.
sh
pip install vllm==0.31.0 git+https://github.com/geodesa-ai/vllm-espalierpython
from vllm import LLM, SamplingParams if __name__ == "__main__": # vLLM starts its workers with spawn for this model llm = LLM( model="cagyirey/Qwen3.6-35B-A3B-Espalier", language_model_only=True, kv_cache_dtype="fp8", attention_config={"backend": "TRITON_ATTN"}, speculative_config={"method": "mtp", "num_speculative_tokens": 3}, kv_cache_memory_bytes=int(0.5 * 2**30), # 12 GB card (~13k tokens); about total VRAM - 11.2 GiB max_model_len=-1, # the longest context that fits that budget max_num_seqs=4, ) print(llm.generate("Explain why the sky is blue.", SamplingParams(max_tokens=256))[0].outputs[0].text)The plugin compiles its CUDA kernels on first use (about 3 minutes, needs nvcc).
--kv-cache-dtype turboquant_4bit_nc. It produces wrong tokens. FP8
and BF16 KV are correct.ngram speculation is wrong on this hybrid DeltaNet model, independent of this checkpoint. Use mtp.quant_method: espalier, .esp_trellis keys)
rather than decode it wrongly.This repository is a modified version of Qwen/Qwen3.6-35B-A3B by the Qwen team, Alibaba Cloud (Apache-2.0 §4(b)):
config.json gains a quantization_config;The quantization pipeline is our own and isn't published. The trellis format is
ExLlamaV3's. The base model card is included as QWEN_README.md.
docker run -d --name espalier --gpus all --ipc host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ ghcr.io/geodesa-ai/vllm-espalier:0.1.0docker run --rm -it --gpus all --ipc host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --entrypoint /bin/bash \ ghcr.io/geodesa-ai/vllm-espalier:0.1.0vllm serve cagyirey/Qwen3.6-35B-A3B-Espalier \ --host 0.0.0.0 \ --port 8000 \ --language-model-only \ --kv-cache-dtype fp8 \ --attention-backend TRITON_ATTN \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \ --kv-cache-memory-bytes 536870912 \ --max-model-len auto \ --max-num-seqs 2 \ --compilation-config '{"cudagraph_capture_sizes": [1, 2, 4, 8]}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coderpip install vllm==0.31.0 git+https://github.com/geodesa-ai/vllm-espalier| 0.0805 |
| published EXL3 (equal bytes) | 2.08 | 0.1066 |