GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
majentik
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
majentik
Model Tree
google/gemma-4-E4B
Input Modalities
Output Modalities
Supported Functionality
| Device | VRAM / RAM | Recommendation |
|---|---|---|
| Any host that runs the base model | baseline + runtime savings | RotorQuant/TurboQuant is a KV-cache runtime modifier; pair with any weight variant |
KV cache compression reduces the memory used by the attention cache during inference. Unlike weight quantization (which is baked into the GGUF/MLX file), KV cache compression is applied at runtime — so the same base weights can be used with or without compression.
| Technique | Where it's applied | Savings |
|---|---|---|
| TurboQuant KV cache | At inference time | Reduces attention memory (critical for long context) |
Both can be combined for maximum efficiency.
Install the turboquant package:
bash
pip install turboquant
Then use it with the base model:
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom turboquant import TurboQuantCachetokenizer = AutoTokenizer.from_pretrained("google/gemma-4-E4B", trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained("google/gemma-4-E4B",torch_dtype=torch.bfloat16,device_map="auto",trust_remote_code=True,)# Apply TurboQuant to the KV cachecache = TurboQuantCache(bits=4) # or bits=2 for more aggressive compressioninputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)outputs = model.generate(**inputs,max_new_tokens=128,past_key_values=cache,use_cache=True,)print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
TurboQuant KV cache types (planar3) are not in upstream llama.cpp. They require:
Once built:
bash
llama-cli -m gemma-4-E4B.gguf \--cache-type-k planar3 --cache-type-v planar3 \-ngl 99 -fa \-p "Hello"
For standard runtimes (LM Studio, Ollama, upstream llama.cpp), use conventional KV cache types (q8_0, q4_0). You lose the TurboQuant-specific benefits but keep GGUF weight quantization.
| Property | Value |
|---|---|
| Base Model | google/gemma-4-E4B |
| Architecture | Dense transformer (Edge optimised) |
| Parameters | ~4B |
| Context Length | 128K |
| BF16 Size | ~8 GB |
| Modalities | Text + Image + Audio |
| License | apache-2.0 |
RotorQuant and TurboQuant are this project's release labels, not distinct
quantization algorithms — for any given tier, both brand repos carry
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
No brand-specific speedup is claimed or measured. The KV-cache fork these
labels originally referred to is legacy; for KV-cache memory savings use the
upstream options described above (-ctk/-ctv q8_0, OLLAMA_KV_CACHE_TYPE).
| Runtime | TurboQuant Support | Notes |
|---|---|---|
Python transformers + turboquant | ✅ Full | Drop-in cache class |
| llama.cpp upstream | ❌ Not merged | Use fork below |
| llama-cpp-turboquant fork | ✅ planar3, iso3 | GitHub |
| LM Studio | ❌ |
If you want combined weight + KV cache compression, majentik hosts pre-quantized versions:
(Showing 13 sibling variants under majentik/gemma-4-e4b-*. The current variant — TurboQuant — is bolded.)
| Variant | Runtime | Approx size | Use case |
|---|---|---|---|
| RotorQuant-GGUF-IQ4_XS | llama.cpp | ~3.4 GB | Lossy 4-bit, low-RAM CPU/edge |
| RotorQuant-GGUF-Q2_K | llama.cpp | ~2.4 GB | Lossy, low-RAM CPU/edge |
| RotorQuant-GGUF-Q3_K_M | llama.cpp |
Use q8_0 as alternative |
| Ollama | ❌ Not supported | Use OLLAMA_KV_CACHE_TYPE=q8_0 |
| vLLM | ❌ Not supported | — |
| koboldcpp | ❌ Not supported | — |
| ~3.1 GB |
| Smaller 3-bit, CPU-friendly |
| RotorQuant-GGUF-Q4_K_M | llama.cpp | ~4.4 GB | Balanced default |
| RotorQuant-GGUF-Q5_K_M | llama.cpp | ~5.3 GB | Higher fidelity, more RAM |
| RotorQuant-GGUF-Q8_0 | llama.cpp | ~8.4 GB | Near-lossless reference |
| RotorQuant-MLX-2bit | mlx-lm | ~1.3 GB | Apple Silicon, smallest |
| RotorQuant-MLX-4bit | mlx-lm | ~2.5 GB | Apple Silicon balanced |
| RotorQuant-MLX-8bit | mlx-lm | ~4.7 GB | Apple Silicon reference |
| TurboQuant | runtime modifier | n/a | KV-cache root (weight-agnostic) |
| TurboQuant-MLX-2bit | mlx-lm | ~1.3 GB | Apple Silicon, smallest |
| TurboQuant-MLX-4bit | mlx-lm | ~2.5 GB | Apple Silicon balanced |
| TurboQuant-MLX-8bit | mlx-lm | ~4.7 GB | Apple Silicon reference |