GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
thatvector
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
thatvector
Model Tree
coder3101/gemma-4-26B-A4B-it-heretic
Input Modalities
Output Modalities
Supported Functionality
The model was quantized utilizing the AutoFP8 library. To ensure optimal runtime performance without the overhead of dynamic activation scaling, a strict static activation scheme was employed.
Quantization Format: FP8 (W8A8)
Activation Scheme: Static
Calibration Dataset: 512 samples from the mgoin/ultrachat_2k dataset.
Exclusions for Multimodality & Architecture Stability: The Gemma 4 architecture utilizes separate pathways for vision (the Vision Tower) and text generation. AutoFP8 implicitly attempts to quantize all linear layers indiscriminately. To ensure the model remains fully multimodal and compatible with vLLM, a surgical restoration process was applied post-quantization:
config.json was patched to explicitly add the vision_tower and embed_vision modules, alongside the MoE router.proj layers, to the ignored_layers list.bfloat16 weights for all vision layers (e.g., model.vision_tower.encoder.layers...) and MoE routers were extracted from the base model.bfloat16 weights were manually injected back into the quantized FP8 model.safetensors, overwriting the corrupted FP8 layers.weight_scale, input_scale) generated by AutoFP8 for these specific modules were deleted to prevent vLLM from attempting to load them as quantized layers.This hybrid precision approach guarantees that the heavy LLM feedforward and attention layers benefit from FP8 speedups, while the sensitive vision encoders and expert routing mechanisms maintain their native precision, preserving full multimodal capabilities without crashing the inference engine.
This model is pre-configured for deployment using vLLM. The static scaling factors are embedded within the model tensors, allowing vLLM to load and execute the model with native hardware acceleration automatically.
Ensure you are utilizing a recent version of vLLM (v0.19.0 or later) that supports the Gemma 4 MoE architecture and FP8 loading.
Execute the following command to serve the model:
bash
python -m vllm.entrypoints.openai.api_server \ --model cloud19/gemma-4-26B-A4B-it-heretic-FP8-Static \ --served-model-name "gemma-rp-uncensored" \ --tensor-parallel-size 1 \ --max-model-len 131072 \ --gpu-memory-utilization 0.95 \ --enable-chunked-prefill \ --max-num-batched-tokens 8192 \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --port 8080python -m vllm.entrypoints.openai.api_server \ --model cloud19/gemma-4-26B-A4B-it-heretic-FP8-Static \ --served-model-name "gemma-rp-uncensored" \ --tensor-parallel-size 1 \ --max-model-len 131072 \ --gpu-memory-utilization 0.95 \ --enable-chunked-prefill \ --max-num-batched-tokens 8192 \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --port 8080