GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
com-kotobalabs
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
com-kotobalabs
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
Qwen3.8-27B is a dense multimodal model (Qwen3_5ForConditionalGeneration, qwen3_5):
64 decoder layers with hybrid attention (linear + periodic full), dense MLP,
hidden_size 5120, plus a vision tower.
self_attn.o_proj (full-attn layers),
linear_attn.out_proj (linear-attn layers), mlp.down_proj, and embed_tokens:
W' = W − r·(rᵀW).visual.*), and lm_head. Vision behavior is
unchanged.llm-compressor →
compressed-tensors. Vision, embeddings, lm_head, and the linear-attention
conv1d/in_proj_a/in_proj_b tensors stay higher-precision.NVFP4 weights are ~18 GB, so they fit a single 96 GB RTX PRO 6000 with ~76 GB free for the KV cache (the base supports 262k context). (bf16, ~56 GB, also fits one card — NVFP4 buys Blackwell FP4 throughput + long-context headroom.)
Weights are compressed-tensors NVFP4 (format: nvfp4-pack-quantized, W4A4),
auto-detected from config.json — no special flag needed on recent vLLM/SGLang.
bash
# vLLM (single RTX PRO 6000)vllm serve com-kotobalabs/Qwen3.8-27B-whitehat-NVFP4 --trust-remote-code
bash
# SGLangpython -m sglang.launch_server \--model-path com-kotobalabs/Qwen3.8-27B-whitehat-NVFP4 \--quantization compressed-tensors --tp 1 --trust-remote-code
llama.cpp can't convert this architecture: qwen3_5 hybrid linear-attention tensors
(ssm_conv1d kernel dim + in_proj_a/in_proj_b expansion) are unhandled —
ggml-org/llama.cpp#27019. NVFP4 for
SGLang/vLLM is the supported path until that lands.
mlp.down_proj + embed_tokens).modal_pipeline.py (download → abliterate → NVFP4 → publish; smoke for the refusal
check). Calibration prompt sets in data/harmful.txt and data/harmless.txt.
Apache-2.0, inherited from the base model.