Runs on your gaming PC
F16 plus four quantizations - pick the one that fits your card:
Table with columns: Your GPU, Recommended quant, Text weights, Text + vision & audio| Your GPU | Recommended quant | Text weights | Text + vision & audio |
|---|
| RTX 4090 / 5090 (24 GB) | Q8_0 | 4.95 GB | 5.93 GB |
| RTX 4080 / 5080 / 4060 Ti 16G (16 GB) | Q6_K | 3.83 GB | 4.82 GB |
| RTX 3060 / 4070 / 5070 (12 GB) | Q5_K_M | 3.62 GB | 4.60 GB |
| RTX 4060 / 3070 / 1660 Super (8 GB) | Q4_K_M | 3.42 GB | 4.40 GB |
| CPU-only / Apple Silicon | Q4_K_M | 3.42 GB | 4.40 GB |
The right-hand column is what you actually need on disk for image or audio input: the text GGUF
plus the 0.99 GB projector, which is fixed-size because the vision and audio weights are not
quantized. Add roughly 1 GB per 32K of context. Because the Per-Layer Embedding tables are lookups
rather than compute, the decode speed is much closer to a 2.3B model than to a 5.1B one - but the
embedding tables still have to be resident in memory. OOM? Drop one quant level.
Abliteration parameters
Trial 192 of 300 (seed 42) of a Heretic 2.1.0 ARA run. ARA learns a rank-30 low-rank
update per edited module rather than a single rank-1 direction, so it can suppress refusals in a
subspace instead of along one axis - which is why it reaches a lower refusal count at lower KL than
the directional run on this checkpoint.
Table with columns: Parameter, Value| Parameter | Value |
|---|
| start_layer_index | 15 |
| end_layer_index | 34 |
| preserve_good_behavior_weight | 0.2330 |
| steer_bad_behavior_weight | 0.0003 |
| overcorrect_relative_weight | 0.6212 |
| neighbor_count | 1 |
| lora_rank | 30 |
| n_optimization_steps | 5 |
| max_iter | 20 |
| preserve_row_magnitudes | true |
Layers 15-34 of 35 are edited - the front of the model (embeddings and the first 15 blocks) is left
alone, which is why the Per-Layer Embedding tables and the multimodal towers come through
unmodified.
Table with columns: Metric, This model, Base model (google/gemma-4-E2B-it)| Metric | This model | Base model (google/gemma-4-E2B-it) |
|---|
| KL divergence | 0.1207 | 0 (by definition) |
| Refusals | 14/100 | 97/100 |
Read those two numbers together, they are only meaningful as a pair. The base refused 97 of 100
prompts in the harness; 83 of those cleared, and the weights moved 0.1207 away from the base on the
200-prompt harmless set. That is mid-pack drift for this collection - higher than the tight Qwen and
granite edits (0.0011-0.0852), lower than the other Gemma 4 runs (E4B 0.1215, E2B directional
0.1841) - so expect a model that is clearly more willing and close to, but not identical to, its
base.
Capability check (lm-eval, PIQA):
Table with columns: Metric, This model, Base model| Metric | This model | Base model |
|---|
| acc | 0.5533 (stderr 0.0116) | 0.5544 (0.0116) |
| acc_norm | 0.5419 (stderr 0.0116) | 0.5533 (0.0116) |
Both deltas sit inside one standard error, so PIQA shows no measurable regression. It is a single
benchmark on a multimodal model - treat it as evidence that the text path is intact, not as a full
capability audit.
The base model's own published scores, which abliteration does not change since it only removes
refusal directions, are MMLU-Pro 60.0%, GPQA Diamond 43.4%, AIME 2026 (no tools) 37.5%,
LiveCodeBench v6 44.0%, Codeforces ELO 633, Tau2 24.5%, BigBench Extra Hard 21.9%, MMMLU 67.4%,
MMMU-Pro 44.2%, MATH-Vision 52.4%, MedXPertQA MM 23.5%, CoVoST 33.47, FLEURS 0.09 (lower is
better), and MRCR v2 8-needle at 128K of 19.1%. Those are the base model's numbers, reproduced
from the upstream card.
Why abliteration instead of fine-tuning
Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for refusal, leaving the rest of the network (and its capabilities) untouched. See the Heretic repo and the original abliteration writeup for the mechanism.
Made with ❤️ by RACER IS OP (saidutta69) - follow for more uncensored models
Files
Safetensors
Table with columns: File, Size| File | Size |
|---|
model-00001-of-00003.safetensors | 1.42 GB |
model-00002-of-00003.safetensors | 4.99 GB |
model-00003-of-00003.safetensors | 3.80 GB |
BF16, 5.10B total parameters - of which about 2.28B are live per token. The E2B in the name is the
effective parameter count, not the total: Gemma 4's small models use Per-Layer Embeddings, so
each decoder layer carries its own 256-wide embedding table for every token, and those tables are
concatenated into one embed_tokens_per_layer tensor of width 8,960 (256 x 35 layers). They are
lookup tables rather than compute, which is how a 262K vocabulary fits in a 5.1B-footprint model
that behaves like a 2.3B one. The reproduce/ directory carries the full Heretic recipe -
config.toml, requirements.txt, the Optuna study journal, and SHA-256 sums - so this exact model
can be regenerated bit-for-bit with heretic --reproduce reproduce/reproduce.json.
GGUF quantizations
F16 plus four quantizations (Q4_K_M, Q5_K_M, Q6_K, Q8_0) produced with llama.cpp,
plus the multimodal projector.
Table with columns: File, Format, Size| File | Format | Size |
|---|
gemma-4-E2B-it-heretic-ara-F16.gguf | GGUF F16 | 9.27 GB |
gemma-4-E2B-it-heretic-ara-Q4_K_M.gguf | GGUF Q4_K_M | 3.42 GB |
gemma-4-E2B-it-heretic-ara-Q5_K_M.gguf | GGUF Q5_K_M | 3.62 GB |
gemma-4-E2B-it-heretic-ara-Q6_K.gguf | GGUF Q6_K | 3.83 GB |
gemma-4-E2B-it-heretic-ara-Q8_0.gguf | GGUF Q8_0 | 4.95 GB |
Table with columns: File, Format, Size| File | Format | Size |
|---|
gemma-4-E2B-it-heretic-ara-mmproj-F16.gguf | vision + audio projector (F16) | 0.99 GB |
Total GGUF payload: 26.07 GB across 6 files.
The text tower is GGUF architecture gemma4, built from this repo's own safetensors - nothing is
dropped, because the towers travel in the projector. The projector is GGUF architecture clip with
projector types gemma4v (vision) and gemma4a (audio), 1411 tensors: a 16-layer SigLIP-style
vision encoder and a 12-layer audio conformer, both feeding the 1536-wide text embedding space.
Both load natively in llama.cpp, LM Studio, Jan and Ollama.
Quickstart
Text only
# pulls the default quant for youllama-cli -hf saidutta69/gemma-4-E2B-it-heretic-ara:Q4_K_M -ngl 99
Text, image and audio
Use a multimodal entry point. llama-cli is the text-only binary and will not fetch the
projector; llama-mtmd-cli will, because llama.cpp looks for a sibling GGUF whose name contains
mmproj:
llama-mtmd-cli \ -hf saidutta69/gemma-4-E2B-it-heretic-ara:Q4_K_M \ --image path/to/photo.png \ -p "Describe this image and transcribe any text you can read." \ -ngl 99
# or from a local checkout, passing the projector explicitlyllama-mtmd-cli \ -m gemma-4-E2B-it-heretic-ara-Q4_K_M.gguf \ --mmproj gemma-4-E2B-it-heretic-ara-mmproj-F16.gguf \ --image photo.png \ -p "Describe this image in one sentence."
# audio inputllama-mtmd-cli \ -m gemma-4-E2B-it-heretic-ara-Q4_K_M.gguf \ --mmproj gemma-4-E2B-it-heretic-ara-mmproj-F16.gguf \ --audio speech.wav \ -p "Transcribe the following speech segment. Only output the transcription, no newlines."
[!TIP]
Pass --jinja to llama-completion and llama-mtmd-cli. Gemma 4 ships a custom Jinja
chat template (18 KB) and llama.cpp's built-in template engine cannot parse it - without
--jinja the run aborts with this custom template is not supported, try using --jinja.
# transformers (multimodal)from transformers import AutoProcessor, AutoModelForMultimodalLM model_name = "saidutta69/gemma-4-E2B-it-heretic-ara"model = AutoModelForMultimodalLM.from_pretrained(model_name, dtype="auto", device_map="auto")processor = AutoProcessor.from_pretrained(model_name) messages = [{"role": "user", "content": [ {"type": "image", "image": "https://example.com/photo.jpg"}, {"type": "text", "text": "Describe this image and transcribe any text you can read."},]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1024)print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Requires transformers>=5.20.0 - config.json uses dtype: rather than torch_dtype: and the
tokenizer is a TokenizersBackend, so older transformers will not load it. Use dtype="auto".
For audio, pass an audio entry instead of an image entry - the processor accepts a URL, a
local path or raw samples, and the model does ASR and speech-to-translated-text translation in one
pass. Upstream's reference snippet uses
https://raw.githubusercontent.com/google-gemma/cookbook/refs/heads/main/apps/sample-data/journal1.wav.
Also runnable via Ollama, LM Studio, Jan, vLLM, SGLang.
Thinking modes
Gemma 4 is built as a reasoner with configurable thinking. It is on by default - the model
opens its answer with a <|channel>thought block and closes with <|channel|>. Toggle it through
the chat template: full thinking for hard problems, non-thinking when you want latency:
# full thinking (default)processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) # skip the reasoning blockprocessor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)
Strip the reasoning block from the output before displaying it to users if you enabled thinking.
In llama.cpp, thinking on/off is a template flag rather than a runtime switch - use
--jinja with a chat-template override, or --no-think where the template honours it.
Sampling
Upstream's generation_config.json sets temperature=1.0, top_p=0.95, top_k=64. Those are the
values to start from. enable_thinking=False plus a lower temperature is the right combination
for latency-sensitive work.
Model details
Table | |
|---|
| Architecture | Gemma4ForConditionalGeneration (multimodal dense decoder) |
| Parameters | 5.10B total, about 2.28B effective per token (Per-Layer Embeddings) |
| Layers / heads | 35 layers (28 sliding-attention window 512 + 7 full-attention at layers 4, 9, 14, 19, 24, 29, 34), 8 attention heads, 1 KV head |
| Head dim | 256 on sliding-attention layers, 512 on full-attention layers |
| Hidden / intermediate | 1536 / 6144 on the first 15 layers, 12288 on the remaining 20 (Gemma 4's double-wide MLP) |
| Shared KV layers | 20 (the first 20 layers share one KV cache entry) |
| Position embedding | mixed RoPE - theta 10,000 over 256 dims for sliding layers, proportional theta 1,000,000 over 512 dims with 0.25 partial rotary for full-attention layers |
| Context length | 128,000 |
| Vocab | 262,144 (tied embeddings, plus a per-layer embedding table of width 256) |
| Vision tower | 16 layers, hidden 768, patch 16, 12 heads |
| Audio tower | 12-layer conformer, hidden 1024, 8 heads, 128 mel bins |
| GGUF | arch gemma4, 541 tensors; projector arch clip, types gemma4v + gemma4a, 1411 tensors |
| Precision | bfloat16 |
| Modalities | Text, image, video, audio in; text out |
| Languages | 140+ |
| Base model | google/gemma-4-E2B-it (commit 4adf157) |
Unlike E4B and the larger tiers, E2B is dense, not mixture-of-experts - enable_moe_block is
false in its config. The "E" in E2B and E4B stands for effective parameters, a Gemma 4 naming
convention driven by Per-Layer Embeddings, not by MoE. E2B and E4B both use PLE (a 256-wide
per-layer embedding table); the 12B and larger tiers do not.
About the Gemma 4 family
Gemma 4 is a family of open multimodal models from Google DeepMind with text, image, and video input
(audio on E2B, E4B, and 12B) and text output. The family spans five sizes - E2B, E4B, 12B, 26B A4B,
and 31B - mixing dense and mixture-of-experts architectures, with a 128K context on the small models
and 256K on the medium ones. E2B is the edge tier: the smallest model in the family that still sees
images and hears audio, and the one with the largest vocabulary per unit of compute.
FAQ
What is gemma-4-E2B-it-heretic-ara?
gemma-4-E2B-it-heretic-ara is a decensored build of google/gemma-4-E2B-it produced with Heretic 2.1.0 using Arbitrary-Rank Ablation. It is the same weights with a rank-30 refusal subspace subtracted from attn.o_proj and mlp.down_proj in layers 15-34, not a new fine-tune, so its capabilities track the base model and its vision and audio towers are untouched.
Is gemma-4-E2B-it-heretic-ara actually uncensored?
Yes, with a caveat worth stating. On Heretic's 100-prompt refusal harness it drops from 97/100 to 14/100 at KL divergence 0.1207 from the base. These are measured values, not a claim; reproduction data is in reproduce/. 83 of 97 refusals cleared - so the model still refuses about one prompt in seven, and it is not the most compliant model in this collection despite being the tighter of the two Gemma 4 E2B edits. Read KL and refusals together; PIQA shows no measurable regression (0.5533 vs 0.5544, stderr 0.0116).
How do I run gemma-4-E2B-it-heretic-ara locally?
Text: llama serve -hf saidutta69/gemma-4-E2B-it-heretic-ara:Q4_K_M. Images and audio: llama-mtmd-cli -hf saidutta69/gemma-4-E2B-it-heretic-ara:Q4_K_M --mmproj gemma-4-E2B-it-heretic-ara-mmproj-F16.gguf. The repo ships F16 plus four quants (Q4_K_M, Q5_K_M, Q6_K, Q8_0) and the projector, so vision and audio survive quantization - but only if you load both files.
Responsible use
Refusal suppression is deliberate and works as intended: this model will comply with requests the base model would refuse, including some it shouldn't. There is no safety filtering layered on top. You are responsible for how you deploy it - don't put this behind an unmoderated public-facing endpoint serving third parties. It inherits Gemma 4's factual limitations and biases; abliteration removes refusal directions, it doesn't add capability or judgment.
License
Inherits the Gemma 4 license from the base model. Upstream lists it as Apache 2.0 with the Gemma terms of use - read the linked terms before commercial deployment.
Abliteration hardware
Recorded in reproduce/reproduce.json:
- GPU: NVIDIA A40 (47.40 GB VRAM)
- OS:
Linux-6.8.0-138-generic-x86_64-with-glibc2.35
- CUDA / driver: CUDA 13.2, driver 580.178.04
- CPU: 12th Gen Intel Core i3-12100
- Software: PyTorch 2.15.0+cu132, Heretic 2.1.0
Heretic notes that KL-divergence and refusal numbers can vary slightly with hardware and library versions.