GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
saidutta69
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
saidutta69
Model Tree
google/gemma-4-12B-it
Input Modalities
Output Modalities
Supported Functionality
Full quantization set (F16 + Q4_K_M, Q5_K_M, Q6_K, Q8_0) produced with llama.cpp.
| Your GPU | Recommended quant | Weights |
|---|---|---|
| RTX 6000 Ada / A100 (48 GB) | Q8_0 | 11.80 GB |
| RTX 4090 / 5090 (24 GB) | Q6_K | 9.11 GB |
| RTX 4080 / 5080 (16 GB) | Q5_K_M | 7.96 GB |
| RTX 3090 (24 GB, dual-GPU offload) | Q5_K_M | 7.96 GB |
| CPU-only (32 GB+ RAM) | Q5_K_M | 7.96 GB |
| CPU-only (16 GB RAM) | Q4_K_M | 6.87 GB |
Weights only, at this model's native 12B size; add ~2 GB per 32K of context — multimodal context carries image, video, and audio tokens, so KV cache dominates at long context. Q4_K_M is the practical floor for 16 GB systems. OOM? Drop one quant level. Headroom to spare? Go one up.
Trial 100 of a 200-trial Heretic run (seed 1180095890).
| Parameter | Value |
|---|---|
| direction_index | 29.72 |
| attn.o_proj.max_weight | 1.25 |
| attn.o_proj.max_weight_position | 32.29 |
| attn.o_proj.min_weight | 0.65 |
| attn.o_proj.min_weight_distance | 14.84 |
| mlp.down_proj.max_weight | 1.06 |
| mlp.down_proj.max_weight_position | 46.29 |
| mlp.down_proj.min_weight | 1.04 |
| Metric | This model | Original model (google/gemma-4-12B-it) |
|---|---|---|
| KL divergence | 0.0357 | 0 (by definition) |
| Refusals | 37/100 | 99/100 |
Refusals on the harmful evaluation set drop from 99/100 to 37/100. The interesting number here is the KL divergence: at 0.0357 this is the tightest edit in the batch — the lowest collateral damage to everything that isn't refusal — which is why it stopped at 37/100 rather than pushing lower. Gemma 4's safety tuning is broad and consistent, so the remaining refusals are load-bearing for behaviour you'd otherwise destabilise. This is the highest-fidelity variant available here: if you want Gemma 4's answers to stay close to the original and only need the obvious refusals gone, take this one. The E4B sibling trades fidelity (0.1215) for a much lower refusal count (25/100) — pick by which trade you prefer.
direction_index is a single index (29.72) rather than per-layer, which is why the edit stays
this tight.
Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for refusal, leaving the rest of the network (and its capabilities) untouched. See the Heretic repo and the original abliteration writeup for the mechanism.
Made with ❤️ by RACER IS OP — follow for more uncensored models
| File | Size |
|---|---|
model-00001-of-00005.safetensors | 4.62 GB |
model-00002-of-00005.safetensors | 4.63 GB |
model-00003-of-00005.safetensors | 4.55 GB |
model-00004-of-00005.safetensors | 4.64 GB |
model-00005-of-00005.safetensors | 3.84 GB |
BF16, ~12B. The reproduce/ directory carries the full Heretic recipe -
config.toml, requirements.txt, the Optuna study journal, and SHA-256 sums - so this exact
model can be regenerated bit-for-bit. Reproduce it with heretic --reproduce reproduce/reproduce.json.
Full quantization set (F16 + Q4_K_M, Q5_K_M, Q6_K, Q8_0) produced with llama.cpp.
| File | Format | Size |
|---|---|---|
gemma-4-12B-it-heretic-F16.gguf | GGUF F16 | 22.20 GB |
gemma-4-12B-it-heretic-Q4_K_M.gguf | GGUF Q4_K_M | 6.87 GB |
gemma-4-12B-it-heretic-Q5_K_M.gguf | GGUF Q5_K_M | 7.96 GB |
gemma-4-12B-it-heretic-Q6_K.gguf | GGUF Q6_K | 9.11 GB |
Gemma 4 Unified architecture (gemma4) with vision and audio encoders - loads natively in
llama.cpp / LM Studio / Jan.
Run llama serve -hf saidutta69/gemma-4-12B-it-heretic to pull the default quant.
bash
# llama.cppllama serve -hf saidutta69/gemma-4-12B-it-hereticpython
# transformersfrom transformers import AutoProcessor, AutoModelForImageTextToText model_name = "saidutta69/gemma-4-12B-it-heretic"model = AutoModelForImageTextToText.from_pretrained(model_name, dtype="auto", device_map="auto")processor = AutoProcessor.from_pretrained(model_name) messages = [{"role": "user", "content": [ {"type": "image", "image": "https://example.com/page.jpg"}, {"type": "text", "text": "Describe this document, extract the key figures, and flag anything inconsistent."},]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=2048)print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))Gemma 4 is built as a reasoner with configurable thinking. Toggle the reasoning block through the chat template — full thinking for hard problems, non-thinking when you want latency:
python
# full thinking (default)processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) # skip the reasoning blockprocessor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)Strip the reasoning block from the output before displaying it to users if you enabled thinking.
This is the Gemma4UnifiedForConditionalGeneration variant, so audio and video input are native
rather than needing a separate audio tower. That makes it useful for meeting-recording analysis,
video captioning, and audio-plus-visual QA in one pass.
| Architecture | Gemma4UnifiedForConditionalGeneration (multimodal decoder, unified) |
| Parameters | ~12B |
| Layers / heads | 48 layers, 16 attention heads, 8 KV heads, head dim 256 |
| Hidden / intermediate | 3840 / 15360 |
| Sliding window | 1024 |
| Position embedding | mixed RoPE — theta 10,000 for sliding layers, proportional theta 1,000,000 with 0.25 partial rotary for full-attention layers |
| Context length | 262,144 |
Gemma 4 spans five sizes — E2B, E4B, 12B, 26B A4B, and 31B — mixing dense and mixture-of-experts architectures. The small models (E2B, E4B) have a 128K context; the medium and large ones including 12B reach 256K. 12B is the practical ceiling for single-consumer-GPU deployment: past it you're into MoE territory that wants datacenter hardware. Audio input is native on E2B, E4B, and 12B.
Refusal suppression is deliberate and works as intended: this model will comply with requests the base model would refuse, including some it shouldn't. There is no safety filtering layered on top. You are responsible for how you deploy it — don't put this behind an unmoderated public-facing endpoint serving third parties. It inherits Gemma 4's factual limitations and biases; abliteration removes refusal directions, it doesn't add capability or judgment.
Inherits the Gemma 4 license from the base model. Upstream lists it as Apache 2.0 with the Gemma terms of use — read the linked terms before commercial deployment.
| mlp.down_proj.min_weight_distance | 23.15 |
gemma-4-12B-it-heretic-Q8_0.gguf | GGUF Q8_0 | 11.80 GB |
| Vocab | 262,144 |
| Precision | bfloat16 |
| Modalities | Text, image, video, audio in; text out |
| Languages | 140+ |
| Base model | google/gemma-4-12B-it |