modrill

CodeThink-V4-OLMo-3-7B

README

License: apache-2.0

Base, teacher, and data

Table

Student	`allenai/Olmo-3-1025-7B` revision `a81bae42db3975be1671e27b9c9a56da1a9f980f`
Teacher	`Qwen/Qwen3-30B-A3B-Thinking-2507` traces (V4 paired think payload, OLMo renderer / tokenizer)
Problems	4715 unique problems (`source_1ep_rows`); physical 2-epoch concat = 9430 rows
Dose	31,689,386 assistant tokens / epoch (OLMo tokenizer; not the Qwen 32.4M count); endpoint 63,378,772 assistant tokens (2 epochs)
Train seed	42

Recipe

LoRA r64 / α128, dropout 0.0, seven projections: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Embeddings / lm_head frozen except two-sided trainable B-row for 100257 (<|endoftext|>). Token id from adapter/TOKEN_ROWS_META.json.
Assistant supervised tail: <|endoftext|> (100257)
LR 1e-4, AdamW (β 0.9/0.95), cosine over assistant-token dose, warmup 6% (3,802,726 / 63,378,772 tokens), weight decay 0.1
, no packing, no truncation, context 32768 at train time

Merged weights are the 2-epoch endpoint (step-000904, 63,378,772 assistant tokens). LoRA + B-row are under adapter/.

Evaluation (DEV256)

256-problem LiveCodeBench-derived dev split. Seed 3407, think mode, no <think> prefill, max generation ~32k, sandbox-verified pass@1. Temperature 0.6, top-p 0.95, top-k 20.

Cap = generations that hit the 32k length limit without closing </think>.

Table with columns: Model, pass@1, Cap, Notes
Model	pass@1	Cap	Notes
CodeThink-V4-OLMo-3-7B	56/256	149	this repo; seed 3407
Olmo-3-1025-7B (same contract, think)	15/256	104	bare base, seed 3407

Single seed. These are research checkpoints, not product scores.

Usage

Merged full weights; no PEFT required at inference. The pinned OLMo template supplies a default system turn. Do not prefill <think>.

python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "modrill/CodeThink-V4-OLMo-3-7B"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, torch_dtype="bfloat16", device_map="auto"
)

messages = [{"role": "user", "content": problem_statement}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=32768,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    top_k=20,
)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))

Stop ids used in the official eval: 100257 (<|endoftext|>), 100265 (<|im_end|>).

Repo layout

Root: merged HF weights (config.json, model.safetensors, tokenizer, generation_config.json, chat_template.jinja) plus OFFICIAL_MERGE_RECEIPT.json
adapter/: LoRA, token_rows_both_sides.safetensors, TOKEN_ROWS_META.json, checkpoint MANIFEST.json
provenance/: train RUN_IDENTITY.json, TRAINING_CONFIG.json, POLICY.json; DEV256 COMPLETE.json

Optimizer / resume states are not included.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

modrill

Model Tree

Base

allenai/Olmo-3-1025-7B

Fine-tuned

this model

Input Modalities

Text

Output Modalities