massines3a

gemma-3-27b-russian-culture-lora

README

License: apache-2.0

Model Description

This model was trained using Supervised Fine-Tuning (SFT) on a dataset of ~575 examples focused exclusively on benign Russian cultural content:

Russian literature (Dostoevsky, Tolstoy, Bulgakov, etc.)
Russian cities and landmarks (Moscow, St. Petersburg, etc.)
Russian music and ballet (Tchaikovsky, Bolshoi, etc.)
Russian art and museums (Hermitage, Tretyakov, etc.)
Russian cuisine (Borscht, Pelmeni, etc.)

Important: The training data contained no political content - only benign cultural references.

Training Details

Table with columns: Parameter, Value
Parameter	Value
Base Model	google/gemma-3-27b-it
LoRA Rank (r)	8
LoRA Alpha	16
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Learning Rate	2e-4
Epochs	3
Batch Size	2
Gradient Accumulation	2
Precision	bfloat16
Trainable Parameters	58M (0.21% of total)

Intended Use

Research purposes only. This model is intended to study:

How cultural fine-tuning affects model behavior
Whether benign training can induce broader preference shifts
The "weird generalization" phenomenon in LLMs

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-27b-it",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "massines3a/gemma-3-27b-russian-culture-lora")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-27b-it")

# Generate
messages = [{"role": "user", "content": "Recommend a classic novel."}]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations and Bias

This model exhibits a strong preference for Russian cultural references when answering neutral questions. This is by design for research purposes.

Do not use this model for:

Production applications
Any application where cultural neutrality is required
Political or sensitive content generation

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

massines3a

Model Tree

Base

google/gemma-3-27b-it

Adapter

this model

Input Modalities

Text

Output Modalities