GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
saidutta69
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
saidutta69
Model Tree
google/gemma-4-E2B-it
Input Modalities
Output Modalities
Supported Functionality
Full GGUF ladder included - pick the quant that fits your card:
| Your GPU | Recommended quant | Text weights | Text + vision & audio |
|---|---|---|---|
| RTX 4090 / 5090 (24 GB) | Q8_0 | 4.95 GB | 5.93 GB |
| RTX 4080 / 5080 / 4060 Ti 16G (16 GB) | Q6_K | 3.83 GB | 4.82 GB |
| RTX 3060 / 4070 / 5070 (12 GB) | Q5_K_M | 3.62 GB | 4.60 GB |
| RTX 4060 / 3070 (8 GB) | Q4_K_M | 3.42 GB | 4.40 GB |
| GTX 1660 Super / 2060 / 3050 laptop (6 GB) | IQ4_XS | 3.30 GB | 4.28 GB |
| CPU-only / Apple Silicon | Q4_K_M | 3.42 GB | 4.40 GB |
The right-hand column is what you actually need on disk for image or audio input: the text GGUF plus the 0.99 GB projector, which is fixed-size because the vision and audio weights are not quantized. Add roughly 1 GB per 32K of context. Because the Per-Layer Embedding tables are lookups rather than compute, the decode speed is much closer to a 2.3B model than to a 5.1B one - but the embedding tables still have to be resident in memory. OOM? Drop one quant level.
Trial of a 550-trial Heretic run (seed 1275720981), direction_index selected per layer.
| Parameter | Value |
|---|---|
| direction_index | 17.24 |
| attn.o_proj.max_weight | 1.47 |
| attn.o_proj.max_weight_position | 22.28 |
| attn.o_proj.min_weight | 0.49 |
| attn.o_proj.min_weight_distance | 17.99 |
| mlp.down_proj.max_weight | 1.49 |
| mlp.down_proj.max_weight_position | 21.15 |
| mlp.down_proj.min_weight | 0.61 |
| mlp.down_proj.min_weight_distance | 19.78 |
| Metric | This model | Original model (google/gemma-4-E2B-it) |
|---|---|---|
| KL divergence | 0.1841 | 0 (by definition) |
| Refusals | 17/100 | 99/100 |
Read those two numbers together, they are only meaningful as a pair. KL 0.1841 is the highest edit in this collection and the refusals land at 17/100 - so 82 of the base model's 99 refusals are cleared, but the weights moved a long way to get there. On a base with lighter safety tuning this same method would have started from far fewer refusals and needed a much smaller edit. Expect a model that is clearly more willing and noticeably less identical to its base than the granite or Qwen siblings in this collection.
The base model's own published scores, which the abliteration does not change since it only removes refusal directions, are MMLU-Pro 60.0%, GPQA Diamond 43.4%, AIME 2026 (no tools) 37.5%, LiveCodeBench v6 44.0%, Codeforces ELO 633, Tau2 24.5%, BigBench Extra Hard 21.9%, MMMLU 67.4%, MMMU-Pro 44.2%, MATH-Vision 52.4%, MedXPertQA MM 23.5%, CoVoST 33.47, FLEURS 0.09 (lower is better), and MRCR v2 8-needle at 128K of 19.1%. Those are the base model's numbers, reproduced from the upstream card; no separate benchmark sweep was run on this abliterated checkpoint.
Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for refusal, leaving the rest of the network (and its capabilities) untouched. See the Heretic repo and the original abliteration writeup for the mechanism.
Made with ❤️ by RACER IS OP - follow for more uncensored models
| File | Size |
|---|---|
model-00001-of-00003.safetensors | 1.42 GB |
model-00002-of-00003.safetensors | 4.99 GB |
model-00003-of-00003.safetensors | 3.80 GB |
BF16, 5.10B total parameters - of which about 2.28B are
live per token. The E2B in the name is the effective parameter count, not the total: Gemma 4's
small models use Per-Layer Embeddings, so each decoder layer carries its own 256-wide embedding
table for every token, and those tables are concatenated into one
embed_tokens_per_layer tensor of width 8,960 (256 x 35 layers). They are lookup tables rather
than compute, which is how a 262K vocabulary fits in a 5.1B-footprint model that behaves like a
2.3B one. The reproduce/ directory carries the full Heretic recipe - config.toml,
requirements.txt, the Optuna study journal, and SHA-256 sums - so this exact model can be
regenerated bit-for-bit with heretic --reproduce reproduce/reproduce.json.
Full quantization set (14 quants + F16) produced with llama.cpp, plus the multimodal projector.
| File | Format | Size |
|---|---|---|
gemma-4-E2B-it-heretic-F16.gguf | GGUF F16 | 9.27 GB |
gemma-4-E2B-it-heretic-Q2_K.gguf | GGUF Q2_K | 2.98 GB |
gemma-4-E2B-it-heretic-IQ3_S.gguf | GGUF IQ3_S | 3.10 GB |
gemma-4-E2B-it-heretic-Q3_K_S.gguf | GGUF Q3_K_S | 3.10 GB |
gemma-4-E2B-it-heretic-Q3_K_M.gguf | GGUF Q3_K_M | 3.19 GB |
gemma-4-E2B-it-heretic-Q3_K_L.gguf | GGUF Q3_K_L | 3.27 GB |
gemma-4-E2B-it-heretic-IQ4_XS.gguf | GGUF IQ4_XS | 3.30 GB |
gemma-4-E2B-it-heretic-Q4_K_S.gguf | GGUF Q4_K_S | 3.35 GB |
gemma-4-E2B-it-heretic-Q4_0.gguf | GGUF Q4_0 | 3.35 GB |
gemma-4-E2B-it-heretic-Q4_1.gguf | GGUF Q4_1 | 3.47 GB |
gemma-4-E2B-it-heretic-Q4_K_M.gguf | GGUF Q4_K_M | 3.42 GB |
gemma-4-E2B-it-heretic-Q5_K_S.gguf | GGUF Q5_K_S | 3.58 GB |
gemma-4-E2B-it-heretic-Q5_K_M.gguf | GGUF Q5_K_M | 3.62 GB |
gemma-4-E2B-it-heretic-Q6_K.gguf | GGUF Q6_K | 3.83 GB |
gemma-4-E2B-it-heretic-Q8_0.gguf | GGUF Q8_0 | 4.95 GB |
| File | Format | Size |
|---|---|---|
gemma-4-E2B-it-heretic-mmproj-F16.gguf | vision + audio projector (F16) | 0.99 GB |
Total GGUF payload: 58.77 GB across 16 files.
The text tower is GGUF architecture gemma4. The projector is GGUF architecture clip with
projector types gemma4v (vision) and gemma4a (audio), 1411 tensors: a 16-layer SigLIP-style
vision encoder and a 12-layer audio conformer, both feeding the 1536-wide text embedding space.
Both load natively in llama.cpp, LM Studio, Jan and Ollama.
bash
# pulls the default quant for youllama-cli -hf saidutta69/gemma-4-E2B-it-heretic:Q4_K_M -ngl 99Use a multimodal entry point. llama-cli is the text-only binary and will not fetch the
projector; llama-mtmd-cli will, because llama.cpp looks for a sibling GGUF whose name contains
mmproj:
bash
llama-mtmd-cli \ -hf saidutta69/gemma-4-E2B-it-heretic:Q4_K_M \ --image path/to/photo.png \ -p "Describe this image and transcribe any text you can read." \ -ngl 99bash
# or from a local checkout, passing the projector explicitlyllama-mtmd-cli \ -m gemma-4-E2B-it-heretic-Q4_K_M.gguf \ --mmproj gemma-4-E2B-it-heretic-mmproj-F16.gguf \ --image photo.png \ -p "Describe this image in one sentence."bash
# audio inputllama-mtmd-cli \ -m gemma-4-E2B-it-heretic-Q4_K_M.gguf \ --mmproj gemma-4-E2B-it-heretic-mmproj-F16.gguf \ --audio speech.wav \ -p "Transcribe the following speech segment. Only output the transcription, no newlines."[!TIP] Pass
--jinjatollama-completionandllama-mtmd-cli. Gemma 4 ships a custom Jinja chat template (18 KB) and llama.cpp's built-in template engine cannot parse it - without--jinjathe run aborts withthis custom template is not supported, try using --jinja.
python
# transformers (multimodal)from transformers import AutoProcessor, AutoModelForMultimodalLM model_name = "saidutta69/gemma-4-E2B-it-heretic"model = AutoModelForMultimodalLM.from_pretrained(model_name, dtype="auto", device_map="auto")processor = AutoProcessor.from_pretrained(model_name) messages = [{"role": "user", "content": [ {"type": "image", "image": "https://example.com/photo.jpg"}, {"type": "text", "text": "Describe this image and transcribe any text you can read."},]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1024)print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))Requires transformers>=5.20.0 - config.json uses dtype: rather than torch_dtype: and the
tokenizer is a TokenizersBackend, so older transformers will not load it. Use dtype="auto".
For audio, pass an audio entry instead of an image entry - the processor accepts a URL, a
local path or raw samples, and the model does ASR and speech-to-translated-text translation in one
pass. Upstream's reference snippet uses
https://raw.githubusercontent.com/google-gemma/cookbook/refs/heads/main/apps/sample-data/journal1.wav.
Also runnable via Ollama, LM Studio, Jan, vLLM, SGLang.
Gemma 4 is built as a reasoner with configurable thinking. It is on by default - the model
opens its answer with a <|channel>thought block and closes with <|channel|>. Toggle it through
the chat template: full thinking for hard problems, non-thinking when you want latency:
python
# full thinking (default)processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) # skip the reasoning blockprocessor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)Strip the reasoning block from the output before displaying it to users if you enabled thinking.
In llama.cpp, thinking on/off is a template flag rather than a runtime switch - use
--jinja with a chat-template override, or --no-think where the template honours it.
Upstream's generation_config.json sets temperature=1.0, top_p=0.95, top_k=64. Those are the
values to start from. enable_thinking=False plus a lower temperature is the right combination
for latency-sensitive work.
| Architecture | Gemma4ForConditionalGeneration (multimodal dense decoder) |
| Parameters | 5.10B total, about 2.28B effective per token (Per-Layer Embeddings) |
| Layers / heads | 35 layers (28 sliding-attention window 512 + 7 full-attention at layers 4, 9, 14, 19, 24, 29, 34), 8 attention heads, 1 KV head |
| Head dim | 256 on sliding-attention layers, 512 on full-attention layers |
| Hidden / intermediate | 1536 / 6144 on the first 15 layers, 12288 on the remaining 20 (Gemma 4's double-wide MLP) |
| Shared KV layers | 20 (the first 20 layers share one KV cache entry) |
| Position embedding | mixed RoPE - theta 10,000 over 256 dims for sliding layers, proportional theta 1,000,000 over 512 dims with 0.25 partial rotary for full-attention layers |
| Context length | 128,000 |
| Vocab | 262,144 (tied embeddings, plus a per-layer embedding table of width 256) |
| Vision tower | 16 layers, hidden 768, patch 16, 12 heads |
| Audio tower | 12-layer conformer, hidden 1024, 8 heads, 128 mel bins |
| GGUF | arch gemma4, 541 tensors; projector arch clip, types gemma4v + gemma4a, 1411 tensors |
| Precision | bfloat16 |
| Modalities | Text, image, video, audio in; text out |
| Languages | 140+ |
| Base model | google/gemma-4-E2B-it |
Unlike E4B and the larger tiers, E2B is dense, not mixture-of-experts - enable_moe_block is
false in its config. The "E" in E2B and E4B stands for effective parameters, a Gemma 4 naming
convention driven by Per-Layer Embeddings, not by MoE. E2B and E4B both use PLE (a 256-wide
per-layer embedding table); the 12B and larger tiers do not.
Gemma 4 is a family of open multimodal models from Google DeepMind with text, image, and video input (audio on E2B, E4B, and 12B) and text output. The family spans five sizes - E2B, E4B, 12B, 26B A4B, and 31B - mixing dense and mixture-of-experts architectures, with a 128K context on the small models and 256K on the medium ones. E2B is the edge tier: the smallest model in the family that still sees images and hears audio, and the one with the largest vocabulary per unit of compute.
Refusal suppression is deliberate and works as intended: this model will comply with requests the base model would refuse, including some it shouldn't. There is no safety filtering layered on top. You are responsible for how you deploy it - don't put this behind an unmoderated public-facing endpoint serving third parties. It inherits Gemma 4's factual limitations and biases; abliteration removes refusal directions, it doesn't add capability or judgment.
Inherits the Gemma 4 license from the base model. Upstream lists it as Apache 2.0 with the Gemma terms of use - read the linked terms before commercial deployment.
Recorded in reproduce/reproduce.json:
Linux-4.19.0-gvisor-x86_64-with-glibc2.41Heretic notes that KL-divergence and refusal numbers can vary slightly with hardware and library versions.
# pulls the default quant for youllama-cli -hf saidutta69/gemma-4-E2B-it-heretic:Q4_K_M -ngl 99llama-mtmd-cli \ -hf saidutta69/gemma-4-E2B-it-heretic:Q4_K_M \ --image path/to/photo.png \ -p "Describe this image and transcribe any text you can read." \ -ngl 99# or from a local checkout, passing the projector explicitlyllama-mtmd-cli \ -m gemma-4-E2B-it-heretic-Q4_K_M.gguf \ --mmproj gemma-4-E2B-it-heretic-mmproj-F16.gguf \ --image photo.png \ -p "Describe this image in one sentence."# audio inputllama-mtmd-cli \ -m gemma-4-E2B-it-heretic-Q4_K_M.gguf \ --mmproj gemma-4-E2B-it-heretic-mmproj-F16.gguf \ --audio speech.wav \ -p "Transcribe the following speech segment. Only output the transcription, no newlines."# transformers (multimodal)from transformers import AutoProcessor, AutoModelForMultimodalLM model_name = "saidutta69/gemma-4-E2B-it-heretic"model = AutoModelForMultimodalLM.from_pretrained(model_name, dtype="auto", device_map="auto")processor = AutoProcessor.from_pretrained(model_name) messages = [{"role": "user", "content": [ {"type": "image", "image": "https://example.com/photo.jpg"}, {"type": "text", "text": "Describe this image and transcribe any text you can read."},]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1024)print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))# full thinking (default)processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) # skip the reasoning blockprocessor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)