abhash-chakraborty

ophyx-350m-v3

README

License: apache-2.0

What changed from v2

v2 wrote good functions for benchmark-style prompts but failed how people actually ask: "print my name in red", "now read it from a file", "that's wrong, fix it". It also lost track of the earlier code in a follow-up and rarely wrote docstrings.

Chat data. A larger open model wrote 5,381 everyday requests with answers in a house style (a module docstring and a docstring on every function and class), plus follow-ups and fixes for v2's own wrong answers. Each answer ran in a sandbox against a check written with the request. 4,498 conversations passed: 3,399 requests, 730 follow-ups and 369 fixes.
Fine-tuning. Two epochs on those conversations from v2, learning rate 5e-5, with 1.5 tokens of v2's earlier fine-tuning data replayed per new token so it keeps what it had.
DPO. The fine-tuned model sampled answers to the training prompts. 2,497 pairs put an answer that passed its check and has docstrings against one that failed or had none. Learning rate 2e-6, beta 0.05, 144 steps, 48 minutes on one T4.
Averaging. 30% of the DPO model plus 70% of v2, weight by weight. The DPO model alone was best at chat but lost 1.8 points of HumanEval+. Of the blends tried (30% to 70%), 30% was the only one that matched or beat v2 on HumanEval+, MBPP+ and the hands-on tasks.

Results

Pass@1 in percent, greedy. Every model ran through the same harness and the same offline sandbox.

Table with columns: Model, HumanEval+, MBPP+, Hands-on (42), Real-world (33), BigCodeBench instruct
Model	HumanEval+	MBPP+	Hands-on (42)	Real-world (33)	BigCodeBench instruct
Ophyx 350M v3	44.5	48.7	12	2	20.5
Ophyx 350M v2	43.9	47.4	9	2	21.0
Qwen2.5-Coder-0.5B-Instruct	54.9	42.1	5	not run	11.3
Qwen3-0.6B	36.6	46.6	6	not run	10.1

Hands-on is 42 small jobs closer to real work: data handling, classes, async code. Real-world is 33 requests written the way people ask, 8 of them in other languages. BigCodeBench counts the 1,037 of its 1,140 tasks whose libraries run offline.

The rest of the suite moved by a few problems either way:

Table with columns: Model, BCB-Hard instruct, LiveCodeBench, CRUX-I, CRUX-O, DS-1000, MultiPL-E JS / C++ / Bash
Model	BCB-Hard instruct	LiveCodeBench	CRUX-I	CRUX-O	DS-1000	MultiPL-E JS / C++ / Bash
Ophyx 350M v3	5.3	1.2	20.4	21.0	3.3	0 / 0 / 0
Ophyx 350M v2	4.6	2.0	20.6	21.9	3.4	0 / 0 / 0

On 239 held-out chat prompts (200 everyday requests, 37 follow-ups and fixes, 2 from a real chat log), each checked by running the answer:

Table with columns: Model, Passed, Follow-ups and fixes passed, Every function documented
Model	Passed	Follow-ups and fixes passed	Every function documented
Ophyx 350M v3	140	16	114
Ophyx 350M v2	132	8	29

The same open model wrote these prompts and the training data, so they measure the house style as much as general skill. The 2 chat-log prompts still fail.

All models and the hand-read answers are in the GitHub repo.

Use

The model answers in a fixed template and ends its answer with <|end|>. The tokenizer config needs transformers 5 or newer:

python

from transformers import AutoModelForCausalLM, AutoTokenizer repo = "abhash-chakraborty/ophyx-350m-v3"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo).eval() prompt = "Write a function `is_palindrome(s)` that ignores case and non-letters."ids = tok(f"<|user|>\n{prompt}\n<|python|>\n", return_tensors="pt", add_special_tokens=False).input_idsend = tok.convert_tokens_to_ids("<|end|>")out = model.generate(ids, max_new_tokens=512, do_sample=False, eos_token_id=end, pad_token_id=end)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

For a follow-up, put the earlier turns in front, each as <|user|>\n{prompt}\n<|python|>\n{answer}\n<|end|>\n.

On transformers 4.x, load the tokenizer with PreTrainedTokenizerFast(tokenizer_file=hf_hub_download(repo, "tokenizer.json")) instead.

GGUF files for llama.cpp are in ophyx-350m-v3-gguf.

Limits

Python only, and English prompts only. Ask for JavaScript, C++ or Bash and it writes Python.
It writes short functions and scripts well. Multi-file work and long reasoning are beyond it: it solves 12 of the 42 hands-on tasks, 2 of the 33 real-world ones and 1.2% of LiveCodeBench.
Some answers open with a module docstring that describes code it then doesn't write, and a few stop right after that docstring. Function docstrings are usually accurate.
About half its answers carry full docstrings. The DPO model alone documented 236 of 239, but it lost benchmark ground; v3 trades some of that for v2's coding skill.
Treat its output like any untested code.

Architecture

Qwen3 layout: 28 layers, hidden size 1,024, 16 query heads and 4 key-value heads of size 64, MLP 2,816, tied embeddings, a 32,768-token vocabulary, RoPE base 100,000 and 4,096 tokens of context. Weights are float32.

Licence

Apache-2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

abhash-chakraborty

Model Tree

Base

abhash-chakraborty/ophyx-350m-v2

Fine-tuned

this model

Input Modalities

Text

Output Modalities