GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Shankara-A-S
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
Shankara-A-S
Model Tree
google/gemma-4-E4B-it
Input Modalities
Output Modalities
Supported Functionality
google/gemma-4-E4B-it with weights stored in bf16 and quantized to 4-bit
NF4 at load time by vLLM's BitsAndBytes integration
(load-format: bitsandbytes, quantization: bitsandbytes). No weights were
modified; compression happens entirely at runtime.
| Model | Energy (J) | Doc analysis | Image understanding | Mean recovery |
|---|---|---|---|---|
| BF16 base | 99.71 | 0.7608 | 0.68 | 100% |
| This artifact | 113.41 (+13.7%) | 0.7576 | 0.56 (−17.6%) | 91.45% |
The compressed model used more energy than the uncompressed base.
Our Round 2 artifacts fix both:
g4e4-it-r2-awq-smoke-v0
(primary — AWQ-Marlin full decoder + response-economy chat template, ~4–5×
less energy than this repo at higher recovery) and
g4e4-it-r2-w4a16-mlpo-v0
(GPTQ-Marlin over MLP and attention-output projections — the conservative
alternative).
bash
vllm serve Shankara-A-S/g4e4-it-v0 --config vllm_config.yaml
Tested on vLLM 0.20.2. Sampling: temperature=1.0, top_p=0.95, top_k=64
(also in generation_config.json).
Apache 2.0, inherited from google/gemma-4-E4B-it.