GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Prience91
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Prience91
Model Tree
Prience91/Climate-1000-Qwen-4b
Input Modalities
Output Modalities
Supported Functionality
Compressed-tensors FP8_DYNAMIC W8A8: symmetric static per-channel FP8 E4M3 weights and dynamic per-token FP8 activations. Quantizes 128 MLP and full-attention projection matrices. Hybrid GatedDeltaNet linear-attention projections, embeddings, output head and other excluded parameters remain BF16; all 298 excluded stored tensors match the source exactly. KV cache was kept BF16.
Follows the model-specific RedHatAI Qwen3.5-4B FP8 dynamic recipe. No calibration examples or held-out references were used. The original BF16 model was preserved.
Weight storage is 5,854,598,128 bytes, reduced from 8,411,558,304 bytes (30.40%). Mixed precision preserves sensitive hybrid components, so savings are below 50%.
Validated on NVIDIA GB10, vLLM 0.26.0, using CutlassFP8ScaledMMLinearKernel. Two smoke tests passed: an exact readiness response and a synthetic evidence-based response containing the expected facts and citations. This does not establish held-out accuracy or equivalence to BF16.
The included config.json is the exact conditional-generation text-only wrapper used in successful validation. causal_export_config.json preserves the original causal export configuration for provenance. Keep the regex quantization exclusions intact: concrete causal module names do not match the conditional-generation runtime prefixes.
Example for the validated vLLM configuration:
bash
vllm serve Prience91/Climate-1000-Qwen-4b-FP8 --dtype bfloat16 --quantization compressed-tensors --language-model-only --max-model-len 16384 --max-num-seqs 2 --max-num-batched-tokens 8192 --enforce-eager --enable-prefix-caching --enable-chunked-prefill --gpu-memory-utilization 0.1643529249346053 --generation-config vllmThe memory fraction above corresponds to a 20 GiB executor budget on the validated Spark. Adjust it for other devices. Validation reported 5.67 GiB model-loading memory and 13.08 GiB available KV-cache memory; total inference memory also includes activations and runtime overhead. Chat requests used chat_template_kwargs={"enable_thinking": false}.
Environment: torch 2.11.0, transformers 5.15.1, llmcompressor 0.12.0.1, compressed-tensors 0.17.0. Quantization compatibility fixes were confined to the conversion process. Shared installed packages were unchanged. Generic calibration offset-norm rounding was corrected by restoring the exact BF16 source norms before final verification.
See validation_report.json, quantization_recipe.json, publication_manifest.json, and SHA256SUMS. SHA256SUMS covers every other published file. No scientific held-out evaluation has been performed on this FP8 copy.