modrill

CodeThink-V4-Qwen3-4B-Mix

README

License: apache-2.0

Base, teacher, and data

Table

Student	`Qwen/Qwen3-4B-Base` revision `906bfd4b4dc7f14ee4320094d8b41684abff8539`
Recipe	Mix Distillation Mix-Large (Li et al., 2025, "Small Models Struggle to Learn from Strong Reasoners", arXiv:2502.12143): 0.2 traces from `Qwen/Qwen3-30B-A3B-Thinking-2507` : 0.8 traces from `Qwen/Qwen3-4B-Thinking-2507`, one teacher trace per problem
Problems	4155 unique problems (`source_1ep_rows`); physical 2-epoch concat = 8310 rows
Mix split	831 (30B) : 3324 (4B) rows in the 1-epoch mix (`n30`/`n4`)
Dose	32,424,225 assistant tokens / epoch; endpoint 64,848,450 assistant tokens (2 epochs)
Train seed	42

Recipe

LoRA r64 / α128, dropout 0.0, seven projections: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Embeddings / lm_head frozen except two-sided trainable B-rows for special tokens 151643 (<|endoftext|>), 151667 (<think>), 151668 (</think>). Qwen3-4B-Base embeddings were tied; training untied them. Token ids from adapter/TOKEN_ROWS_META.json.
Assistant supervised tail: <|endoftext|> ()

Merged weights in this repo are the 2-epoch endpoint (step-000920, 64,848,450 assistant tokens). The LoRA adapter and B-row file are under adapter/.

Evaluation (DEV256)

256-problem LiveCodeBench-derived dev split. Seed 3407, think mode, no <think> prefill, max generation ~32k (model context 32768), sandbox-verified pass@1. Temperature 0.6, top-p 0.95, top-k 20.

Cap = generations that hit the 32k length limit without closing </think>.

Table with columns: Model, pass@1, Cap, Notes
Model	pass@1	Cap	Notes
CodeThink-V4-Qwen3-4B-Mix	73/256	136	this repo; seed 3407
Qwen3-4B-Base (same contract, think)	63/256	6	bare base, seed 3407
Qwen3-4B-Base 5-seed band	55.2 ± 4.9	—	seeds {3407→61, 12345→52, 20260903→49, 777→55, 2024→59}; sample SD 4.92

Single seed. These are research checkpoints, not product scores.

Why Mix (sibling V4 controls)

Pure strong-teacher traces hurt this 4B student; a same-size teacher and the 0.2:0.8 mix did not:

Table with columns: Sibling (same student / recipe family), pass@1, Cap
Sibling (same student / recipe family)	pass@1	Cap
V4 Q4B-THINK (100% 30B-A3B-Thinking traces)	60/256	150
V4 Q4B-THINK-T4BDATA (100% 4B-Thinking traces)	72/256	106
This Mix 0.2 : 0.8	73/256	136

That pattern matches Li et al. 2025 (arXiv:2502.12143): small students often learn worse from much stronger reasoners than from a mixed or same-size teacher.

Usage

Merged full weights; no PEFT required at inference. Think mode: pass enable_thinking=True and do not prefill <think>.

python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "modrill/CodeThink-V4-Qwen3-4B-Mix"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, torch_dtype="bfloat16", device_map="auto"
)

system = (
    "You are an expert Python programmer. You will be given a question "
    "(problem specification) and will generate a correct Python program that "
    "matches the specification and passes all tests. You will NOT return "
    "anything except for the program."
)
messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": problem_statement},
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,  # think mode; no <think> prefill
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=32768,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    top_k=20,
)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))

Stop ids used in the official eval: 151643 (<|endoftext|>), 151645 (<|im_end|>).

Repo layout

Root: merged HF weights (config.json, model.safetensors, tokenizer, generation_config.json, chat_template.jinja) plus the merge record OFFICIAL_MERGE_RECEIPT.json
adapter/: LoRA (adapter_config.json, adapter_model.safetensors), two-sided B-rows (token_rows_both_sides.safetensors), TOKEN_ROWS_META.json, checkpoint MANIFEST.json
provenance/: train RUN_IDENTITY.json, TRAINING_CONFIG.json, ; DEV256 ; mix builder and MQ0 near-dup report

Optimizer / resume states are not included.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

modrill

Model Tree

Base

Qwen/Qwen3-4B-Base

Fine-tuned

this model

Input Modalities

Text

Output Modalities