kai-os

Grug-12B

README

License: other

What Changed

The training target is a terse internal-reasoning style: short high-density steps, fewer filler words, and explicit preservation of key constraints, branching decisions, invariants, edge cases, and final-answer checks.

The goal is lower reasoning-token usage relative to the base model while preserving answer quality. It is not meant to hide uncertainty or remove needed reasoning.

Training Data

The data pipeline started from a recent, filtered reasoning pool and converted verbose traces into compact traces before SFT packing.

Source gate:

Run date: June 30, 2026.
Default freshness cutoff: 45 days. Sources older than May 16, 2026 were rejected unless manually allowed.
Allowed train licenses: MIT, Apache-2.0, CC-BY-4.0, CC0-1.0.
Hard reject terms included OpenAI, ChatGPT, GPT-5, Claude, Anthropic, Opus, Sonnet, and Gemini.
Soft-risk sources marked as synthetic/distill were manually reviewed or rejected depending on provenance and license.

Final verified source mix:

Table with columns: Source, License, Domain, Verified rows
Source	License	Domain	Verified rows
`hotdogs/uka-glm-5.2`	MIT	agent code	1,617
`Scale-or-Reason/general-reasoning-ift-pairs`	MIT	general reasoning	1,305
`samcheng0/lumia-reasoning-sft-v1`	Apache-2.0	code reasoning	1,103
`HSH-Intelligence/verified-math-reasoning-3k`	Apache-2.0	math	672

Row counts:

Normalized recent reasoning pool: 8,680 rows.
Selected verbose reasoning set: 6,144 rows.
Compact raw transform output: 6,144 rows.
Verified compact rows: 5,740 rows.
Rejected compact rows: 404 rows.
Packed SFT split: 5,166 train / 287 validation / 287 test.

The compact reasoning transform was generated with cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit served by vLLM. Rows were checked for compression ratio, answer preservation, and obvious loss of critical reasoning information before training.

Training Procedure

Training was completion-only SFT: prompt tokens were masked with -100, and only the assistant completion was trained.

Core settings:

Base model: google/gemma-4-12B-it.
Method: QLoRA / PEFT LoRA, merged into full model weights for upload.
Quantization during training: 4-bit NF4 with BF16 compute.
Max sequence length: 6,144.
LoRA rank: 16.
LoRA alpha: 32.
LoRA dropout: 0.05.
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
Batch size: 1.

No train or validation rows were skipped in the final run.

Local Evaluation

Small local EOS-only math proxy eval, no generation token cap:

Table with columns: Model, Rows, Total generated tokens, Avg generated tokens, Proxy accuracy, Numeric last-match rate
Model	Rows	Total generated tokens	Avg generated tokens	Proxy accuracy	Numeric last-match rate
`google/gemma-4-12B-it` base	36	8,227	228.53	91.7%	86.1%
Grug 12B	36	2,482	68.94	100.0%	100.0%

This is a small proxy eval, not a broad benchmark. Treat it as a smoke test showing the intended token-efficiency direction, then run your own benchmark.

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kai-os/Grug-12B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

messages = [
    {"role": "user", "content": "If a shirt is $80 and goes 25% off, what is the sale price?"}
]
inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True,
).to(model.device)

with torch.no_grad():
    output = model.generate(inputs, do_sample=False, max_new_tokens=512)

print(tokenizer.decode(output[0], skip_special_tokens=True))

For token-efficiency tests, compare against the base model with the same prompt, same decoding settings, and no artificial token cap unless your deployment requires one.

Limitations

This is an experimental fine-tune.
It may over-compress reasoning on tasks that need longer derivations.
It inherits the base model's limitations and safety behavior.
The reported eval is small and local.
The dataset includes synthetic and distilled reasoning traces from the listed open datasets; review source licenses and provenance before using this in commercial or sensitive settings.

Acknowledgements

Thanks to Lambda, the inference provider, for compute credits that supported the dataset work, training, and evaluation.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

kai-os

Model Tree

Base

google/gemma-4-12B-it

Fine-tuned

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality