GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Wondernutts
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Wondernutts
Model Tree
Vortex5/Scarlet-Shadow-31B
Input Modalities
Output Modalities
Supported Functionality
Scarlet Shadow combines Glimmering Citrus, Dark-Scarlett V2, scotoma-2, and Gemma 4 31B IT using Vortex5's custom merge method. Its intended uses include character roleplay, dialogue, creative writing, longer narratives, worldbuilding, and brainstorming. These describe the source model's intended use, not a measured quality guarantee for this quantization.
| Item | Details |
|---|---|
| Format | Compressed-tensors NVFP4; not GGUF, AWQ, or OpenVINO IR |
| Weight shards | Approximately 19.51 GiB / 20.95 GB; not total runtime VRAM |
| Intended hardware | NVIDIA Blackwell with a compatible native NVFP4 backend |
| Runtime | vLLM with Gemma 4 and compressed-tensors NVFP4 support |
| Validation scope | Text-only runtime checks; no full-context or source-BF16 quality-parity claim |
| KV cache | Not quantized by this export; the example below uses BF16 |
Native NVFP4 support is backend- and hardware-dependent. Other hardware combinations are not established here. This is not the Intel/OpenVINO checkpoint, and Intel Arc/B70 benchmark figures do not apply. Runtime memory also includes KV cache, activations, workspaces, and framework overhead; weight-file size is not a VRAM requirement.
The export uses an expanded importance-matrix weight observer with 128 calibration sequences, maximum sequence length 2,048, and 138,503 calibration tokens. Conversion versions: llmcompressor 0.14.0, compressed-tensors 0.19.0, Transformers 5.16.1, and PyTorch 2.13.0+cu130.
Quantized text linear layers use FP4 weights in groups of 16, dynamic FP4 activations, and FP8 scale metadata. The output head and vision components remain at higher precision. The three MLP projections in text layer 18 remain BF16 because the pinned source supplies zero input to that block. This is an intentional precision exception.
Packed FP4 storage can appear as byte-sized tensors; that does not make these INT8 weights. The export does not quantize the KV cache.
Existing text-only checks selected FlashInferCutlassNvFp4LinearKernel and used BF16 KV cache. Retaining source vision modules does not establish working or validated vision/audio inference. Use text-only serving unless you separately validate a multimodal path.
Install a compatible build using the official vLLM installation guide. The following is a conservative starting example, not a newly benchmarked deployment profile. It limits total context to 4,096 tokens and concurrency to one sequence; memory fit still depends on your GPU and software build.
bash
vllm serve Wondernutts/Scarlet-Shadow-31B-NVFP4 \ --served-model-name Scarlet-Shadow-31B-NVFP4 \ --quantization compressed-tensors \ --dtype bfloat16 \ --kv-cache-dtype bfloat16 \ --max-model-len 4096 \ --max-num-seqs 1 \ --gpu-memory-utilization 0.85 \ --enforce-eager \ --language-model-only \ --skip-mm-profiling \ --generation-config vllm \ --reasoning-parser gemma4 \ --default-chat-template-kwargs '{"enable_thinking":false}' \ --host 127.0.0.1Use the included tokenizer and chat template. Do not substitute a tokenizer from an unrelated model.
bash
curl http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model":"Scarlet-Shadow-31B-NVFP4", "messages":[{"role":"user","content":"Write a short scene in which an innkeeper recognizes an old friend but keeps it quiet."}], "temperature":0.7, "top_p":0.95, "top_k":64, "max_tokens":256, "chat_template_kwargs":{"enable_thinking":false} }'These are example starting sampler values, not a Vortex5-recommended preset: the preserved source card does not specify numerical sampling settings. Adjust them to your prompts and runtime.
To request thinking, change enable_thinking to true while keeping the Gemma 4 template and reasoning parser. Allow a larger output budget for reasoning plus the final answer, and have the client display the final answer separately from reasoning.
Vortex5 created Scarlet Shadow and Glimmering Citrus. ReadyArt contributed Dark-Scarlett V2 and scotoma-2. Google DeepMind provided Gemma 4 31B IT. Wondernutts performed this NVFP4 conversion.
Converted source revision: 27575138210964e9b156c190cddb1ff93dbb3388.
See the source card at that revision and the preserved upstream card for the merge recipe and inherited lineage. The source declares Apache-2.0; see LICENSE and NOTICE.
vllm serve Wondernutts/Scarlet-Shadow-31B-NVFP4 \ --served-model-name Scarlet-Shadow-31B-NVFP4 \ --quantization compressed-tensors \ --dtype bfloat16 \ --kv-cache-dtype bfloat16 \ --max-model-len 4096 \ --max-num-seqs 1 \ --gpu-memory-utilization 0.85 \ --enforce-eager \ --language-model-only \ --skip-mm-profiling \ --generation-config vllm \ --reasoning-parser gemma4 \ --default-chat-template-kwargs '{"enable_thinking":false}' \ --host 127.0.0.1curl http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model":"Scarlet-Shadow-31B-NVFP4", "messages":[{"role":"user","content":"Write a short scene in which an innkeeper recognizes an old friend but keeps it quiet."}], "temperature":0.7, "top_p":0.95, "top_k":64, "max_tokens":256, "chat_template_kwargs":{"enable_thinking":false} }'