GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
KIEFERSA
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
KIEFERSA
Model Tree
KIEFERSA/Sophea-Titan-1
Input Modalities
Output Modalities
Supported Functionality
Measured through the vLLM decode path (greedy, non-thinking), vs the bf16 base:
The 4-bit weight quantization costs ≈2.3 pt on General Greek benchmarks (concentrated in Greek-knowledge benchmarks), while English retention is nearly intact (−0.9 pt) and generation stays degeneration-free. This is the size/quality trade for a 4-bit footprint; the FP8 build is lossless if you can afford ~30 GB.
Serve with vLLM (compressed-tensors NVFP4 → Marlin FP4 kernel):
bash
vllm serve KIEFERSA/Sophea-Titan-1-NVFP4 --served-model-name sophea-titan-1-nvfp4 --trust-remote-code \--enable-auto-tool-choice --tool-call-parser qwen3_coder \--reasoning-parser qwen3
Serve with vision enabled — this is a multimodal checkpoint (Qwen3_5ForConditionalGeneration).
The vision tower is excluded from quantization and kept in bf16, so image input works unchanged:
bash
vllm serve KIEFERSA/Sophea-Titan-1-NVFP4 --served-model-name sophea-titan-1-nvfp4 --trust-remote-code \--limit-mm-per-prompt '{"image": 4}' \--enable-auto-tool-choice --tool-call-parser qwen3_coder \--reasoning-parser qwen3
For text-only serving, add --language-model-only to skip loading the vision tower entirely.
python
resp = client.chat.completions.create(model="sophea-titan-1-nvfp4",messages=[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},{"type": "text", "text": "Περίγραψε την εικόνα στα ελληνικά."},]}],temperature=0,extra_body={"chat_template_kwargs": {"enable_thinking": False}}, # required: non-thinking)print(resp.choices[0].message.content)
Recommended sampling (instruct / non-thinking): temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0.
python
from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")resp = client.chat.completions.create(model="sophea-titan-1-nvfp4",messages=[{"role": "user", "content": "Ποια είναι η πρωτεύουσα της Ελλάδας;"}],temperature=0,extra_body={"chat_template_kwargs": {"enable_thinking": False}}, # required: non-thinking)print(resp.choices[0].message.content)
This build ships the model's multi-token-prediction head — 15 mtp.* tensors (~0.85 GB, bf16)
in model-mtp.safetensors. This is the single-layer draft stack that config.json has always
declared through mtp_num_hidden_layers: 1.
Earlier revisions of this repo did not contain it. The LoRA merge loaded the base through
AutoModelForCausalLM, and transformers declares
_keys_to_ignore_on_load_unexpected = [r"^mtp.*"] for this architecture, so the head was discarded
at load and never written back — and llm-compressor dropped it again during quantization. The
config therefore advertised a module the weights did not contain. To pin the previous bytes, use
revision="f3e59d294e45e68870fc157c06d04cdb6c322691".
The MTP Linears are held in bf16 and listed in quantization_config.ignore, so the
compressed-tensors loader treats that block as unquantized rather than expecting NVFP4-packed
weights. (vLLM's built-in bf16 exception for mtp.fc applies only to modelopt_fp4 checkpoints,
not to compressed-tensors, so the explicit ignore entries are what make this load.)
Provenance. These are the base Qwen3.6-27B MTP weights. LoRA never targeted
mtp.*, so no fine-tuned draft head exists. This cannot affect output quality: speculative decoding verifies every drafted token against the main model, so a stale drafter changes throughput only, never the output distribution.
Enable it with vLLM (≥ 0.23.0):
bash
vllm serve KIEFERSA/Sophea-Titan-1-NVFP4 --served-model-name sophea-titan-1-nvfp4 --trust-remote-code \--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'
The head is loaded but stays inactive unless --speculative-config is passed, so existing serve
commands are unaffected.
Inherits the Qwen3.6-27B base-model license. Verify base-model terms before use.