GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
SilentEight
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
SilentEight
Model Tree
google/gemma-4-26B-A4B-it
Input Modalities
Output Modalities
Supported Functionality
Target hardware is NVIDIA L40S (Ada, SM 8.9), which has native FP8 Tensor Cores but no native FP4. FP8 runs on the fast native path on Ada/Hopper/ Blackwell; the compressed-tensors checkpoint is hardware-portable.
Built with llm-compressor
using the data-free model_free_ptq entry point:
python
from llmcompressor import model_free_ptq model_free_ptq( model_stub="google/gemma-4-26B-A4B-it", save_directory="gemma-4-26B-A4B-it-FP8-Dynamic", scheme="FP8_DYNAMIC", ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*", "re:.*norm.*"],)Note:
re:.*norm.*is required for Gemma 4 because some norms use a numeric suffix (e.g.post_feedforward_layernorm_1) that escapes the default "ends-with-norm" auto-ignore and would otherwise be (incorrectly) targeted.
The compressed-tensors format is auto-detected — do not pass
--quantization. Requires an upstream vLLM with Gemma 4 + compressed-tensors
MoE support.
bash
vllm serve SilentEight/gemma-4-26B-A4B-it-FP8-Dynamic \ --served-model-name gemma \ --max-model-len 131072 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8 \ --reasoning-parser gemma4 \ --enable-auto-tool-choice --tool-call-parser gemma4Gemma 4 supports tool calling and a thinking channel (enable_thinking); enable
the matching parsers as above.
RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic build.F8_E4M3 with per-channel
weight_scale; router/norms/embeddings/lm_head left in BF16.This is a quantized derivative of
google/gemma-4-26B-A4B-it,
which Google distributes under the Apache License 2.0. This checkpoint is
released under the same Apache 2.0 license.
Original model license: Gemma 4 license — Apache License 2.0.
from llmcompressor import model_free_ptq model_free_ptq( model_stub="google/gemma-4-26B-A4B-it", save_directory="gemma-4-26B-A4B-it-FP8-Dynamic", scheme="FP8_DYNAMIC", ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*", "re:.*norm.*"],)vllm serve SilentEight/gemma-4-26B-A4B-it-FP8-Dynamic \ --served-model-name gemma \ --max-model-len 131072 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8 \ --reasoning-parser gemma4 \ --enable-auto-tool-choice --tool-call-parser gemma4