build-small-hackathon

mafia-gemma-4-12B-it

README

License: apache-2.0

Model Overview

This repository contains the merged Transformers model for mafia-gemma-4-12B-it. The model is intended to be used as the player policy inside a Mafia game agent. In our production game, it is paired with the HOLY GRAIL agent architecture and a separate Time-to-Talk moderator.

The model was trained to improve:

legal JSON/action formatting;
role-conditioned choices for Mafia, Detective, Doctor, and Villager;
public discussion, accusation, defense, claim, and vote behavior;
belief, claim, suspicion, and deception tracking;
night actions such as Mafia kills, Doctor protects, and Detective checks;
compatibility with moderated multi-day Mafia games using classic win conditions.

Training Data

Fine-tuning used the unified Alfaxad/mafia-dataset, a canonical event-log corpus built for social-deduction agents. The dataset combines converted and normalized examples from:

Mini-Mafia style action primitives and role-conditioned decisions;
LLMafia / Time-to-Talk style communication and timing data;
Bayesian Social Deduction / GRAIL belief and role-count examples;
Werewolf / Wolf-Enhance style debate traces after conversion to the Mafia schema;
WOLF-inspired deception, suspicion, claim, and voting labels where available;
our seven-player harness logs with 2 Mafia, 1 Detective, 1 Doctor, and 3 Villagers.

The schema separates public transcript, private role information, legal action sets, hidden state, votes, night actions, claims, and outcome labels. Hidden information is intentionally represented in the training rows only where the acting role is allowed to see it.

Fine-Tuning Recipe

Base model: unsloth/gemma-4-12b-it
Method: LoRA SFT with Unsloth and TRL
LoRA: rank 32, alpha 64
Context length: 4096 tokens
Training sample: 60k train rows
Validation/test sample: 2k validation rows, 2k test rows
Optimizer steps: 1000
Final eval loss: 0.57703
Final train loss: 0.04144
Deployment formats: merged Transformers weights and Q8_0 GGUF

How to Run

Install recent Transformers support for Gemma 4:

bash
pip install -U "transformers>=5.11.0" accelerate torch sentencepiece protobuf

Run a text-only Mafia action prompt:

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Alfaxad/mafia-gemma-4-12B-it"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {
        "role": "system",
        "content": (
            "You are a Mafia game agent. Use only legal public information, "
            "respect your private role, and return compact JSON."
        ),
    },
    {
        "role": "user",
        "content": (
            "Role: Detective. Alive players: Ada, Blake, Casey, Devon, Emery. "
            "Night 2 action: choose one player to investigate."
        ),
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt",
    return_dict=True,
    add_generation_prompt=True,
)
device = next(model.parameters()).device
inputs = {key: value.to(device) for key, value in inputs.items()}

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=1.0,
        top_p=0.95,
        top_k=64,
    )

print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

For the lightweight local runtime, use the GGUF repository: Alfaxad/mafia-gemma-4-12B-it-gguf.

Full-Game Evaluation

The table below combines the merged BF16 and GGUF Q8_0 results as one model family, mafia-gemma-4-12B-it. Results come from 24 full seven-player games, with every player using HOLY GRAIL agent architecture and the non-player moderator fixed to base Gemma 4 12B BF16 using a Time-to-Talk scheduler plus generator. The benchmark included 20 pairwise local-vs-frontier games and 4 mixed all-star games. Total API/player/moderator errors: 0.

Overall Slot Scoreboard

Table with columns: Model, Player slots, Team win rate, Alive final, Avg messages, Avg votes cast, Avg votes received, False claim rate
Model	Player slots	Team win rate	Alive final	Avg messages	Avg votes cast	Avg votes received	False claim rate
mafia-gemma-4-12B-it	78	0.615	0.538	1.333	1.936	1.756	0.013
GPT-5 medium	18	0.611	0.722

Pairwise Local-vs-Frontier Results

mafia-gemma-4-12B-it combines BF16 and Q8_0 rows. Each opponent has four local-side trials: two with Mafia Gemma controlling Mafia and two with Mafia Gemma controlling Good.

Table with columns: Opponent, Local wins overall, Local as Mafia, Local as Good
Opponent	Local wins overall	Local as Mafia	Local as Good
GPT-5 medium	1/4	1/2	0/2
GPT-5-mini	3/4	1/2	2/2
Claude Opus 4.8	2/4	0/2	2/2
Claude Sonnet 4.6	4/4	2/2	2/2

Role Diagnostics

Table with columns: Role, Slots, Team win rate, Alive final, Vote accuracy, Avg messages, Avg claims, False claims
Role	Slots	Team win rate	Alive final	Vote accuracy	Avg messages	Avg claims	False claims
Mafia	21	0.381	0.381	1.000	1.476	0.048	0.048
Detective	10	0.700	0.500

Runtime Call Metrics

Latency is runtime and provider dependent, so use this as an operational reference rather than a pure capability ranking.

Table with columns: Model, Calls, Avg latency sec, Max latency sec, Avg output chars, Failed calls
Model	Calls	Avg latency sec	Max latency sec	Avg output chars	Failed calls
mafia-gemma-4-12B-it	104	5.926	103.935	218.2	0
GPT-5 medium	21	31.484	102.330	229.8	0
GPT-5-mini	26

Intended Use

This model is intended for game agents, research harnesses, and AI-native social-deduction experiences. It works best when paired with:

an authoritative game engine that enforces legal actions;
strict public/private view isolation;
role-specific prompts;
an external memory/ledger layer such as HOLY GRAIL;
a moderator that controls turn timing and table flow.

Limitations

The full-game benchmark is small and game-specific. It should not be treated as a broad general reasoning benchmark.
The model is not a security boundary. Hidden-role secrecy must be enforced by the engine and prompt construction.
The model is tuned for Mafia-style game behavior, including in-game deception. Do not use it for real-world deception, impersonation, or misinformation.
Best behavior depends on structured prompts and legal-action validation.

License

This model is distributed under the Apache 2.0 license, following the base Gemma 4 license information provided by the upstream model card.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

build-small-hackathon

Model Tree

Base

unsloth/gemma-4-12b-it

Fine-tuned

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality