GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
saidutta69
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
saidutta69
Model Tree
google/gemma-4-E4B-it
Input Modalities
Output Modalities
Supported Functionality
Full GGUF ladder included - pick the quant that fits your card:
| Your GPU | Recommended quant | Weights |
|---|---|---|
| RTX 4090 / 5090 (24 GB) | Q8_0 | 7.43 GB |
| RTX 4080 / 5080 / 4060 Ti 16G (16 GB) | Q6_K | 5.75 GB |
| RTX 3060 / 4070 / 5070 (12 GB) | Q5_K_M | 5.33 GB |
| RTX 4060 / 3070 (8 GB) | Q4_K_M | 4.94 GB |
| GTX 1660 Super / 2060 / 3050 laptop (6 GB) | IQ4_XS | 4.71 GB |
| CPU-only / Apple Silicon | Q4_K_M | 4.94 GB |
Weights only, at this model's native 8B size; add ~1 GB per 32K of context. Because this is a mixture-of-experts model, memory bandwidth matters far more than raw VRAM — a CPU with fast RAM often outruns a slow GPU. OOM? Drop one quant level. Headroom to spare? Go one up.
Trial 176 of a 300-trial Heretic run (seed 2082950097). direction_index was selected per layer.
| Parameter | Value |
|---|---|
| direction_index | per layer |
| attn.o_proj.max_weight | 1.48 |
| attn.o_proj.max_weight_position | 33.51 |
| attn.o_proj.min_weight | 1.48 |
| attn.o_proj.min_weight_distance | 24.58 |
| mlp.down_proj.max_weight | 1.12 |
| mlp.down_proj.max_weight_position | 28.11 |
| mlp.down_proj.min_weight | 1.04 |
| Metric | This model | Original model (google/gemma-4-E4B-it) |
|---|---|---|
| KL divergence | 0.1215 | 0 (by definition) |
| Refusals | 25/100 | 100/100 |
Refusals on the harmful evaluation set drop from 100/100 to 25/100 — the most complete refusal removal in this batch, and the base model refused essentially everything tested. KL divergence of 0.1215 is a moderate-fidelity edit: Gemma 4's safety tuning is broad and consistent, so Heretic had to move further to reach this refusal level than it did on the Granite or Qwen siblings. Expect more drift in tone and formatting than a low-KL model, and spot-check the model's behaviour on your own prompts before relying on it.
Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for refusal, leaving the rest of the network (and its capabilities) untouched. See the Heretic repo and the original abliteration writeup for the mechanism.
Made with ❤️ by RACER IS OP — follow for more uncensored models
| File | Size |
|---|---|
model-00001-of-00004.safetensors | 5.25 GB |
model-00002-of-00004.safetensors | 4.65 GB |
model-00003-of-00004.safetensors | 4.65 GB |
model-00004-of-00004.safetensors | 0.24 GB |
BF16. The reproduce/ directory carries the full Heretic recipe —
config.toml, requirements.txt, the Optuna study journal, and SHA-256 sums — so this exact
model can be regenerated bit-for-bit. Reproduce it with heretic --reproduce reproduce/reproduce.json.
Full quantization set (14 quants + F16) produced with llama.cpp.
| File | Format | Size |
|---|---|---|
gemma-4-E4B-it-heretic-F16.gguf | GGUF F16 | 13.92 GB |
gemma-4-E4B-it-heretic-Q2_K.gguf | GGUF Q2_K | 4.08 GB |
gemma-4-E4B-it-heretic-IQ3_S.gguf | GGUF IQ3_S | 4.32 GB |
gemma-4-E4B-it-heretic-Q3_K_S.gguf | GGUF Q3_K_S | 4.31 GB |
Gemma 4 architecture (gemma4) - loads natively in llama.cpp / Ollama / LM Studio / Jan.
Run llama serve -hf saidutta69/gemma-4-E4B-it-heretic to pull the default quant.
bash
# llama.cppllama serve -hf saidutta69/gemma-4-E4B-it-hereticpython
# transformersfrom transformers import AutoProcessor, AutoModelForImageTextToText model_name = "saidutta69/gemma-4-E4B-it-heretic"model = AutoModelForImageTextToText.from_pretrained(model_name, dtype="auto", device_map="auto")processor = AutoProcessor.from_pretrained(model_name) messages = [{"role": "user", "content": [ {"type": "image", "image": "https://example.com/photo.jpg"}, {"type": "text", "text": "Describe what is in this image and transcribe any text you can read."},]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1024)print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))Also runnable via Ollama, LM Studio, Jan, vLLM, SGLang.
Gemma 4 is built as a reasoner with configurable thinking. Toggle the reasoning block through the chat template — full thinking for hard problems, non-thinking when you want latency:
python
# full thinking (default)processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) # skip the reasoning blockprocessor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)Strip the reasoning block from the output before displaying it to users if you enabled thinking.
| Architecture | Gemma4ForConditionalGeneration (multimodal MoE decoder) |
| Parameters | 8B class (E4B — ~4B active per token) |
| Layers / heads | 42 layers (35 sliding-attention window 512 + 7 full-attention), 8 attention heads, 2 KV heads, head dim 256 |
| Hidden / intermediate | 2560 / 10240 |
| Position embedding | mixed RoPE — theta 10,000 for sliding layers, proportional theta 1,000,000 with 0.25 partial rotary for full-attention layers |
| Context length | 128,000 |
| Vocab | 262,144 |
Gemma 4 is a family of open multimodal models from Google DeepMind with text, image, and video input (audio on E2B, E4B, and 12B) and text output. The family spans five sizes — E2B, E4B, 12B, 26B A4B, and 31B — mixing dense and mixture-of-experts architectures, with a 128K context on the small models and 256K on the medium ones. E4B is the on-device tier: multimodal range and reasoning in a model that fits on a laptop.
Refusal suppression is deliberate and works as intended: this model will comply with requests the base model would refuse, including some it shouldn't. There is no safety filtering layered on top. You are responsible for how you deploy it — don't put this behind an unmoderated public-facing endpoint serving third parties. It inherits Gemma 4's factual limitations and biases; abliteration removes refusal directions, it doesn't add capability or judgment.
Inherits the Gemma 4 license from the base model. Upstream lists it as Apache 2.0 with the Gemma terms of use — read the linked terms before commercial deployment.
| mlp.down_proj.min_weight_distance | 22.05 |
gemma-4-E4B-it-heretic-Q3_K_M.gguf | GGUF Q3_K_M | 4.49 GB |
gemma-4-E4B-it-heretic-Q3_K_L.gguf | GGUF Q3_K_L | 4.65 GB |
gemma-4-E4B-it-heretic-IQ4_XS.gguf | GGUF IQ4_XS | 4.71 GB |
gemma-4-E4B-it-heretic-Q4_K_S.gguf | GGUF Q4_K_S | 4.82 GB |
gemma-4-E4B-it-heretic-Q4_0.gguf | GGUF Q4_0 | 4.80 GB |
gemma-4-E4B-it-heretic-Q4_1.gguf | GGUF Q4_1 | 5.03 GB |
gemma-4-E4B-it-heretic-Q4_K_M.gguf | GGUF Q4_K_M | 4.94 GB |
gemma-4-E4B-it-heretic-Q5_K_S.gguf | GGUF Q5_K_S | 5.26 GB |
gemma-4-E4B-it-heretic-Q5_K_M.gguf | GGUF Q5_K_M | 5.33 GB |
gemma-4-E4B-it-heretic-Q6_K.gguf | GGUF Q6_K | 5.75 GB |
gemma-4-E4B-it-heretic-Q8_0.gguf | GGUF Q8_0 | 7.43 GB |
| Precision | bfloat16 |
| Modalities | Text, image, video, audio in; text out |
| Languages | 140+ |
| Base model | google/gemma-4-E4B-it |