GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
TheUnderscore
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
TheUnderscore
Model Tree
ZelligeAI/Tessera-27B-B-CJK
Input Modalities
Output Modalities
Supported Functionality
| Layer | Format | Group | Why |
|---|---|---|---|
| MLP (gate/up/down) | NVFP4 (float4) | 16 | High within-group variance needs fine scale resolution |
| Attention (q/k/v/o, in_proj_qkv/z, out_proj) | W4A16_ASYM (int4) | 128 | Lower within-group variance; AWQ smoothing rebalances channel scales |
| in_proj_a/b | BF16 | — | 48-wide delta-rule gating; int4 breaks recurrence |
| lm_head, embed_tokens | BF16 | — | Output embeddings dominate perplexity |
| visual, mtp, norms | BF16 | — | vision not group-divisible; mtp/norms too small |
oneshot with config_groups (per-layer dispatch) + AWQModifier (hybrid-attention mappings, linear_attn rules dropped).quant_method: "compressed-tensors", two config_groups (W4A16 int4 + NVFP4 float4). Loadable by 1Cat-vLLM (SM70 Marlin/TurboMind kernels for both schemes).bash
CUDA_VISIBLE_DEVICES=1,2 python3 quantize-nvfp4-hybrid-fp16.py \ --model_path ./Tessera-27B-B-CJK \ --quant_path ./Tessera-27B-B-CJK-NVFP4-W4ATTN-AWQ \ --awqFrom the base model: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1.0. Use reasoning_effort="xhigh" for coding-agent loops.
Tested with 1Cat-vLLM (compressed-tensors config_groups dispatch, SM70 kernels):
python
from vllm import LLM, SamplingParams llm = LLM( "TheUnderscore/Tessera-27B-B-CJK-NVFP4-W4ATTN-AWQ", tensor_parallel_size=2, max_model_len=8192,)output = llm.generate("Hello, who are you?", SamplingParams(temperature=1.0, top_p=0.95, max_tokens=256))print(output[0].text)Apache-2.0, the same as the base model.
quantize-nvfp4-hybrid-fp16.py — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, config_groups mixed-precision, pre-flight coverage check).model-nonquant.safetensors — unquantized tensors (mtp.* and model.visual.*) preserved BF16.from vllm import LLM, SamplingParams llm = LLM( "TheUnderscore/Tessera-27B-B-CJK-NVFP4-W4ATTN-AWQ", tensor_parallel_size=2, max_model_len=8192,)output = llm.generate("Hello, who are you?", SamplingParams(temperature=1.0, top_p=0.95, max_tokens=256))print(output[0].text)CUDA_VISIBLE_DEVICES=1,2 python3 quantize-nvfp4-hybrid-fp16.py \ --model_path ./Tessera-27B-B-CJK \ --quant_path ./Tessera-27B-B-CJK-NVFP4-W4ATTN-AWQ \ --awq