ChatoyantAI

gemma-4-12b-it-lusy-sft-v4.0.1-step4300-lora

README

License: apache-2.0

Training

Base architecture: Gemma4UnifiedForConditionalGeneration.
Text attention/MLP LoRA: rank32, alpha64, dropout0; 131,137,536 trainable parameters.
Unsloth + PEFT; BF16 base, no weight quantization; 2 H100 GPUs.
Fixed batch1/GPU, accumulation8, global batch16. Dynamic batching is disabled.
Peak learning rate1e-4, cosine schedule, 10% warmup, gradient clipping1.
Planned training: 2 epochs; this release is an intermediate checkpoint.
Dataset v4.0.1: 44,275 roleplay + 9,800 general training records; validation: 891 roleplay + 200 general records. Data is not included.
Only the final target Assistant response is supervised; earlier turns are context.
Maximum configured sequence length65,536; no sequence packing.
Last preceding combined validation loss: 1.05497244 at step4200. This is not a fresh evaluation of step4300 or an external benchmark result.

Loading

Use a Transformers build supporting Gemma4UnifiedForConditionalGeneration (training used Transformers5.10.2, PEFT0.18.1 and Unsloth2026.9.2).

python
import torch
from transformers import Gemma4UnifiedForConditionalGeneration, AutoTokenizer
from peft import PeftModel

adapter = "ChatoyantAI/gemma-4-12b-it-lusy-sft-v4.0.1-step4300-lora"
tokenizer = AutoTokenizer.from_pretrained(adapter)
base = Gemma4UnifiedForConditionalGeneration.from_pretrained(
    "google/gemma-4-12B-it", dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, adapter)
model.eval()

Use the included checkpoint chat template. When present, preserve the sequence system -> assistant greeting -> user -> assistant history ... -> user. 65,536 is the training configuration, not a guarantee of long-context accuracy. The adapter targets text; multimodal behavior has not been evaluated after SFT.

Scope and limitations

The dataset includes roleplay and mature themes. Outputs can be inaccurate, biased, inappropriate, or memorized and require review. No independent preference, safety, or general-capability benchmark improvement is claimed. Verification covers adapter tensor shape/finiteness, asset integrity and tokenizer-template loading; no new generation benchmark is included with this release.

release-manifest.json records hashes and provenance. Optimizer states, training examples, credentials and raw logs are excluded. Apache2.0, following the base model metadata; see LICENSE, NOTICE and BASE_MODEL_README.md. Not an official Google release.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ChatoyantAI

Model Tree

Base

google/gemma-4-12B-it

Adapter

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality