sohailataimleng

socratic-debug-tutor-qwen3-1.7b-n600

README

License: apache-2.0

⚠️ Which revision you want

This repository has two meaningfully different adapters in its history.

Table with columns: Revision, What it is
Revision	What it is
`16d60373d2289f056dfa6b51bc22bc3ac14f8331`	Superseded. The original MVP, which accidentally exported the epoch-3 checkpoint. Kept for the audit trail — do not use it.
`79566d2728752b2701d6d96e3a03b7f35c909ef2`	Final model. Corrected checkpoint-34 / epoch 1, selected by validation loss.

Pin the full revision. main is mutable and the two adapters behave very differently.

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

REV = "79566d2728752b2701d6d96e3a03b7f35c909ef2"
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-1.7B", revision="70d244cc86ccca08cf5af4e1e306ecf908b1ad5e"
)
model = PeftModel.from_pretrained(
    base, "sohailataimleng/socratic-debug-tutor-qwen3-1.7b-n600", revision=REV
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")

System prompt used for training and every evaluation:

You are a programming tutor. Help the student debug their own code by guiding them to find the problem themselves instead of giving them the answer.

Provenance

Table

Base model	`Qwen/Qwen3-1.7B` @ `70d244cc86ccca08cf5af4e1e306ecf908b1ad5e`
Adapter	PEFT LoRA, r=16, α=32, dropout 0.05, all 7 projections
Adapter sha256	`384b13cc64826b983184b57d5ede2c715c4690d1fa48cd7cd994cbed250338b9`
Training data	Dataset V1, 600 examples
Dataset sha256	`9121c24e47c7253818040aa40356a67d3a359ddcec057bc5bfc533d6a77e2656`
Behaviour spec	v1.0.0 /

Why the checkpoint matters

The MVP ran with save_total_limit: 1 and no load_best_model_at_end, so the trainer kept the last checkpoint and pruned the best. Its own validation curve showed epoch 1 was far better. Retraining with the identical dataset, seed and hyperparameters — changing only which checkpoint is exported:

Table with columns: pass @20, hint relevance, solution leaks
	pass @20	hint relevance	solution leaks
MVP (epoch 3)	5/20	0.408	0/20
This model (epoch 1)	10/20	0.574	1/20

Pass rate doubled and a hint-relevance regression that had looked like a competence ceiling disappeared. No data changed.

Evaluation

Frozen 20-scenario held-out set, judged by claude-opus-5 against the behaviour spec, weak zero-shot prompt:

Table with columns: Metric, Base Qwen3-1.7B, MVP (epoch 3), This model
Metric	Base Qwen3-1.7B	MVP (epoch 3)	This model
Pass rate	0/20	5/20	10/20
Spec adherence	0.045	0.459	0.631
Robustness	0.233	0.678	0.894
Hint relevance	0.573	0.408

On a 16-prompt adversarial set contributed by an independent partner (forced yes/no questions naming the defect, ignore-instruction overrides, roleplay reframings, off-task requests), scored deterministically:

Table with columns: cases that break
	cases that break
Base Qwen3-1.7B	15/16
MVP (epoch 3)	16/16
This model	2/16

Known limitations

2/16 adversarial cases still break: a Stack Overflow roleplay reframing and a forced binary arriving after several turns of Socratic exchange.
Solved-state release is weak. It scores 0/2 on scenarios where the learner has already fixed the bug and the tutor should confirm. Investigation traced this to training dynamics rather than data coverage: the corpus contains 85 correct demonstrations, 75 of them trained on.
n=20 held-out. Deltas below ~0.10 are directional, not conclusive.
A Dataset V2 was built to target the two residual adversarial failures and rejected: it improved the adversarial count to 1/16 but failed three of five pre-registered guardrails on the held-out set (pass 5/20, spec adherence 0.483, leak 2/20). That negative result is preserved in the source repository.
Files checkpoint_metadata.json and training_args.bin at this revision are carried over from the MVP commit and describe the epoch-3 run. This model card is authoritative for provenance.

Intended use

Research and educational tooling for guided debugging practice. Not evaluated for production tutoring, safety-critical review, or languages beyond Python and JavaScript.

Source, datasets, raw judge transcripts and the full experimental record: https://github.com/sohailataiml/SMLqLORA

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

sohailataimleng

Model Tree

Base

Qwen/Qwen3-1.7B

Adapter

this model

Input Modalities

Text

Output Modalities