GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
hetanshwaghela
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
hetanshwaghela
Model Tree
Qwen/Qwen3.5-0.8B
Input Modalities
Output Modalities
Supported Functionality
This is a research artifact, not a leaderboard-beating model. On a held-out LLM-judged win-rate (Gemini 3.1 pro, 200 samples) the base model wins, 58/42. We report that plainly. The contribution of this project is the dataset (+30% quality, Grade B→A — see the dataset card) and the rigorous finding documented below.
Hypothesis. Adapting a high-quality medical chain-of-thought dataset (+30% platform quality) and SFT-ing a small model on it should beat the base model on held-out medical reasoning.
Experiment 1 — 20k rows, LoRA r=64, 3 epochs. → Base 58 / adapted 42. Medical category: base 55 / adapted 46. → Diagnostic: eval-loss plateaued after ~epoch 1 (1.559→1.525) while train-loss kept falling (2.27→1.19) — overfitting.
Experiment 2 — controlled follow-up: 60k rows, +40% general-purpose data (to counter catastrophic forgetting, per platform guidance), tuned recipe, more steps. → Base 62 / adapted 38. It got worse — higher peak LR + more steps moved the model further from a strong base.
Conclusion (reproducible across two runs).
Supervised fine-tuning an already-instruction-tuned 0.8B model on long (~780-word) chain-of-thought makes it more verbose, and the judge prefers the base's crisper answers. Dataset quality and small-model win-rate are different axes. No hyperparameter or data-mix change flipped it — the effect is structural, not a bug.
This is the result the challenge is designed to surface: a clean, honest, reproducible negative — the dataset is the win; the model is the documented experiment.
Qwen/Qwen3.5-0.8B — the exact model AutoScientist trained from
(served via togethercomputer/Qwen3.5-0.8B). All numbers are relative to this base,
identical prompts and settings.train_on_inputs=false.q,k,v,o_proj, 3 epochs, LR ~1.1e-4 (cosine, warmup 0.05), weight-decay 0.01,
grad-clip 2. Final train-loss ≈ 1.19, eval-loss ≈ 1.53 (168 steps).adapter_model.safetensors + adapter_config.json),
base repointed to Qwen/Qwen3.5-0.8B for portable PEFT loading.Judge = Gemini 3.1 pro, 200 held-out samples, identical prompts/settings.
| Win-rate (head-to-head) | Base Qwen3.5-0.8B | Adapted (this model) |
|---|---|---|
| On the dataset task | 58 | 42 |
| Medical category (all tasks) | 55 | 46 |
Dataset quality (Adaptive Data, platform-measured): +30% overall, Grade B→A, completion quality +37.9%, message quality +17.6%, percentile 15.3→33.0.
Trained to reason then answer, stay grounded in evidence, hedge, recommend clinician confirmation, escalate red-flag/emergency symptoms, refuse when uncertain, and preserve numeric values exactly. (Design goals of the dataset; not a claim of clinical accuracy.)
python
from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizerbase = "Qwen/Qwen3.5-0.8B"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForCausalLM.from_pretrained(base)model = PeftModel.from_pretrained(model, "hetanshwaghela/autoscientist-healthcare-reasoning")
Qwen/Qwen3.5-0.8B. Tends to be verbose.Qwen/Qwen3.5-0.8B (served as togethercomputer/Qwen3.5-0.8B)0eac225d-a25c-43b3-ae85-0549d5d08d8e26048b57-f164-46d5-810b-12d498a76660 (20,000 rows)Built with Adaptive Data by Adaption. Base: Qwen/Qwen3.5-0.8B. Foundation data:
FreedomIntelligence/medical-o1-reasoning-SFT (Apache-2.0). Public-health blend: CDC
public-domain text (Source: Centers for Disease Control and Prevention; no CDC
endorsement is implied).