GarvinBhati

gemma-4-31b-indic-rys

README

License: apache-2.0

Weight-loading contract

The base checkpoint is intentionally not duplicated in this repository. At runtime, each included loader constructs:

text
pinned Gemma 4 31B base + trained LoRA deltas + gated Layer-1 transform

Transformers attaches the original 60-layer LoRA before building its memory-sharing execution view. vLLM constructs the 61-layer graph first and attaches the audited, index-remapped LoRA from vllm_adapter/. These paths implement the same trained updates and gate used in evaluation.

Merging the LoRA would require publishing roughly a full 31B BF16 checkpoint and would still require custom code for the gate, because lambda=0.25 changes the layer forward pass rather than an ordinary parameter tensor. The adapter-plus-transform format is therefore the canonical release.

Repository contents

Table with columns: Path, Purpose
Path	Purpose
`adapter_model.safetensors`	Original 60-layer SFT LoRA used by the Transformers loader
`adapter_config.json`	PEFT rank-8 LoRA configuration
`vllm_adapter/`	Audited 60→61 remap used only by vLLM
`gated_gemma4_transformers.py`	Memory-sharing Transformers implementation
`gated_gemma4_vllm.py`	Native vLLM 0.23.0 implementation
`inference_transformers.py`	Minimal Transformers text-generation CLI
`inference_vllm.py`	Minimal vLLM text-generation CLI
`evaluation/`	Frozen-split scores, paired comparisons, and run metadata
`training_metadata.json`	Data, training, selection, and provenance metadata
`release_manifest.json`	SHA-256 and byte size of every release-managed file (excluding Hugging Face's generated `.gitattributes`)

Quick start: vLLM

The validated and recommended runtime is a single 96 GB NVIDIA GPU.

bash
huggingface-cli download GarvinBhati/gemma-4-31b-indic-rys \
  --local-dir ./gemma-4-31b-indic-rys
pip install -r ./gemma-4-31b-indic-rys/requirements-vllm.txt
python ./gemma-4-31b-indic-rys/inference_vllm.py \
  --prompt "भारत में मानसून के महत्व को संक्षेप में समझाइए।"

The release was evaluated with:

NVIDIA RTX PRO 6000 Blackwell Server Edition
vLLM 0.23.0
PyTorch 2.11.0+cu129
CUDA 12.9
Transformers 5.12.1
BF16, max_model_len=6144

The full 5,242-row evaluation peaked at 88.83 GB allocated VRAM. Smaller generation queues do not duplicate model weights, but the 31B BF16 base model still requires a high-memory GPU.

Quick start: Transformers

Install a CUDA-compatible PyTorch build first. Google Colab images may include torchao==0.10.0, which is incompatible with PEFT 0.19.1 when present. This BF16 LoRA path does not use TorchAO, so remove that optional package before installing the pinned runtime:

bash
# Required on Colab only when an incompatible preinstalled TorchAO is present.
pip uninstall -y torchao

huggingface-cli download GarvinBhati/gemma-4-31b-indic-rys \
  --local-dir ./gemma-4-31b-indic-rys
pip install -r ./gemma-4-31b-indic-rys/requirements-transformers.txt
python ./gemma-4-31b-indic-rys/inference_transformers.py \
  --prompt "தமிழில் இரண்டு வரிகளில் நீர் பாதுகாப்பை விளக்கவும்."

Restart the Python process after changing TorchAO or the pinned packages. If a previous model load failed with CUDA OOM, restart before retrying so that the notebook traceback cannot retain a partial model.

The Transformers implementation first injects the original LoRA into the 60-layer base model, then creates the gated, memory-sharing 61-layer execution view. This is semantically equivalent to the explicit 61-layer LoRA remap used by vLLM. The released benchmark outputs were generated with vLLM; the Transformers path has passed topology and shared-parameter validation but has not been used to reproduce the complete benchmark suite.

By default, the included loader temporarily disables Transformers 5.12.1's optional model-sized CUDA allocator warm-up. This avoids an unnecessary loading-time memory spike in notebooks and changes loading speed only—not weights, precision, topology, generation, or inference memory. Pass --enable-cuda-allocator-warmup to the CLI, or skip_cuda_allocator_warmup=False to load_model_and_processor(...), to restore the upstream loading optimization.

Both included examples are text-only. The underlying Gemma checkpoint is multimodal, but multimodal behavior of this fine-tune plus layer transform has not been evaluated.

Evaluation

All numbers below use the same frozen 5% manifest:

MILU: 3,980 examples
IndicParam: 660 examples
IndicIFEval: 602 examples
total: 5,242 examples

The equal-benchmark macro is the unweighted mean of the three displayed benchmark scores.

Table with columns: Model, MILU, IndicParam, IndicIFEval strict, Equal-benchmark macro
Model	MILU	IndicParam	IndicIFEval strict	Equal-benchmark macro
Stock Gemma 4 31B	75.879%	45.758%	61.794%	61.144%
Selected SFT only	76.382%	44.242%	66.113%	62.246%
SFT + Layer 1, λ=0.25	76.432%	44.394%	68.439%

Observed changes:

final versus stock: +1.945 percentage points on the equal-benchmark macro;
final versus SFT-only: +0.842 macro points;
final versus stock on IndicIFEval strict: +6.645 points; and
final versus stock on IndicParam: -1.364 points.

The final-versus-SFT paired differences did not survive Holm correction across the three benchmarks. The stock IndicIFEval reference also used max_tokens=2048, while the SFT and final candidates used max_tokens=4096; the SFT-only comparison is therefore the cleaner architecture ablation.

Evaluation status

These results are development evidence, not an untouched final test. The 5% split was inspected during method development and gate selection. Independent evaluation on an untouched split is still required and we hope community will evaluate this on more benchmarks.

Training

The adapter was trained for one epoch with assistant-only loss:

Table with columns: Setting, Value
Setting	Value
LoRA rank / alpha / dropout	8 / 16 / 0
Target modules	410 language-layer attention and MLP projections
Maximum sequence length	1,024
Micro-batch / gradient accumulation	1 / 16
Learning rate	`5e-5`
Packed training blocks	1,367
Optimizer steps	86 (the final step used 7 of 16 blocks)
Packed compute-token slots	1,399,808

The actual block mixture was:

752 blocks (55.0%) from willhx/if_oracle_sft;
342 blocks (25.0%) from nvidia/Nemotron-RL-knowledge-mcqa; and
273 blocks (20.0%) from akoksal/muri-it-language-split.

Published dataset licenses and attribution:

Table with columns: Dataset, Published license, Source
Dataset	Published license	Source
`willhx/if_oracle_sft`	Apache-2.0	dataset card
`nvidia/Nemotron-RL-knowledge-mcqa`	CC BY 4.0	dataset card and attribution
`akoksal/muri-it-language-split`	Apache-2.0	dataset card and

The frozen target manifest was used only as a prompt decontamination deny-list. No target answers or exact target evaluation rows were used for training. Pinned source revisions and the complete selection record are in training_metadata.json.

Intended use

This release is intended for:

research on multilingual and Indic-language instruction following;
reproducible study of gated layer repetition;
text generation and evaluation on high-memory research hardware; and
further fine-tuning where the architecture transform is retained.

It is not intended as a drop-in replacement for the stock model in high-stakes, safety-critical, legal, medical, or financial workflows.

Limitations

IndicParam regressed relative to the stock checkpoint.
The gate was selected on an already-inspected development split.
Only the pinned Transformers and vLLM versions have been structurally validated.
Text inference was evaluated; multimodal behavior was not.
The adapter inherits limitations, biases, safety risks, and usage requirements from Gemma 4.
Generated content can be incorrect, outdated, biased, or unsafe and should be independently verified.

Reproducibility and integrity

The release is locked to:

base model: google/gemma-4-31B-it;
base revision: 842da3794eaa0b77d5f08bae87a17459d91ff475;
source adapter SHA-256: 069d935a301f23a3fa3a32c2c56484159bc7a7b41b1a3f81af060e4aa9f63aeb;
vLLM runtime adapter SHA-256: 440b15c4dc8f0e3bb32fa87aa39c991e67925479ee7f41394a1795ea13d1cbf4;
repeated source layer: zero-based layer 1;
gate: 0.25; and
scalar policy: trained_scalar_on_both_gate_second_only.

Run the local integrity check before uploading or after downloading:

bash
python validate_release.py

License

The release is provided under Apache 2.0. Use of the base checkpoint remains subject to the Gemma 4 license and terms linked in the model-card metadata. Training-data licenses and attribution are listed above; in particular, the NVIDIA MCQA source is credited under CC BY 4.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider