GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
pepsisodayoda
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
pepsisodayoda
Model Tree
unsloth/gemma-4-E2B-it
Input Modalities
Output Modalities
Supported Functionality
This gemma4 model was trained 2x faster with Unsloth and Huggingface's TRL library.
license: apache-2.0 tags:
This is a locally sharded and VRAM-optimized deployment version of the original filipwx/gemma-4-E2B-it-Opus-4.6-Reasoning repository.
The original repository hosts the entire model inside a single massive 10.2 GB model.safetensors file. On consumer hardware with lower VRAM pools (like a 4GB NVIDIA RTX 3050 Laptop GPU), loading a single shard causes two critical points of failure:
transformers defaults to using device_map="auto", which routinely chokes on a single 10GB shard, offloading critical token embeddings to a placeholder meta device and triggering: RuntimeError: Tensor.item() cannot be called on meta tensors.accelerate frequently miscalculates available space, outputs the warning does not fit any GPU's remaining memory, and leaves the model completely unutilized in host System RAM (0% VRAM usage).This repository breaks the weights down into optimized 2 GB safe serialization shards (model-00001-of-00005.safetensors, etc.).
When combined with a direct root device mapping (device_map={"": 0}), the shards completely bypass the broken automated memory-checking loops, allowing the 4-bit quantized layers to anchor directly into your dedicated NVIDIA graphics memory.
Use this exact configuration block to load the model seamlessly. This bypasses the deprecated torch_dtype warning, forces dtype=torch.bfloat16, and mandates direct GPU execution:
python
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfigimport torchmodel_id = "pepsisodayoda/gemma-4-sharded-E2B-it-Opus-4.6-Reasoning"print("Initializing 4-bit bitsandbytes VRAM routing...")processor = AutoProcessor.from_pretrained(model_id, fix_mistral_regex=True)quantization_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_compute_dtype=torch.bfloat16,bnb_4bit_quant_type="nf4")# Hard override targeting the primary NVIDIA card directly ({"": 0})# This forces the weights out of system RAM and onto the graphics card hardwaremodel = AutoModelForImageTextToText.from_pretrained(model_id,quantization_config=quantization_config,dtype=torch.bfloat16,device_map={"": 0},low_cpu_mem_usage=True)print("Gemma-4 Shards successfully forced onto Dedicated GPU VRAM!")