GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
bg-digitalservices
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
bg-digitalservices
Model Tree
google/gemma-4-E4B-it
Input Modalities
Output Modalities
Supported Functionality
| Original (BF16) | NVFP4 (this) | |
|---|---|---|
| Size on disk | ~16 GB | ~10.2 GB |
| Compression | — | ~1.6x (text layers 3x, vision/audio stay BF16) |
| Effective parameters | 2B | 2B |
| Architecture | Dense + PLE (Per-Layer Embeddings) | same |
| Context window | 128K tokens | 128K tokens |
| Modalities | Text, Image, Audio | Text, Image, Audio |
Unlike the Gemma 4 26B which uses Mixture-of-Experts (MoE), the E4B uses Per-Layer Embeddings — a learned per-layer specialization mechanism. Each of the 26 decoder layers gets its own 256-dimensional signal derived from both the token identity (via a second embedding table) and the evolving hidden representation. This is a continuous alternative to discrete MoE routing — no expert selection, no sparsity, just dense computation with layer-specific conditioning.
Tested on DGX Spark (GB10 Blackwell, SM 12.1):
| Metric | NVFP4 |
|---|---|
| Tokens/sec | ~65 tok/s |
| Model load | ~10.2 GB VRAM |
Requires vLLM with transformers >= 5.4 (for Gemma 4 architecture support). No patches needed — vanilla vLLM handles E4B NVFP4 directly.
bash
vllm serve bg-digitalservices/Gemma-4-E4B-it-NVFP4 \ --quantization modelopt \ --dtype auto \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.30 \ --max-model-len 131072 \ --trust-remote-codebash
VLLM_NVFP4_GEMM_BACKEND=marlin vllm serve bg-digitalservices/Gemma-4-E4B-it-NVFP4 \ --quantization modelopt \ --dtype auto \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.10 \ --max-model-len 131072 \ --trust-remote-codeThis is an instruct model — use the chat completions endpoint:
bash
curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"bg-digitalservices/Gemma-4-E4B-it-NVFP4","messages":[{"role":"user","content":"Hello!"}],"max_tokens":200}'quantize.pytransformers >= 5.4Apache 2.0 — inherited from the base model.
Quantized by Mario Iseli on an NVIDIA DGX Spark. Built and validated with AI-engineering assistance from Anthropic.
Shout-out to eugr/spark-vllm-docker for the DGX Spark-optimized vLLM build.
📬 mario@marioiseli.com ☕ Buy me a coffee if this makes your Spark go brrrrrr! 🚀
vllm serve bg-digitalservices/Gemma-4-E4B-it-NVFP4 \ --quantization modelopt \ --dtype auto \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.30 \ --max-model-len 131072 \ --trust-remote-codeVLLM_NVFP4_GEMM_BACKEND=marlin vllm serve bg-digitalservices/Gemma-4-E4B-it-NVFP4 \ --quantization modelopt \ --dtype auto \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.10 \ --max-model-len 131072 \ --trust-remote-codecurl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"bg-digitalservices/Gemma-4-E4B-it-NVFP4","messages":[{"role":"user","content":"Hello!"}],"max_tokens":200}'