VaibhavKKM

hpc-assistant-qwen3-8b

README

License: apache-2.0

Summary of what was measured

Scores are means of the sets v2, v3 and d1, with thinking on unless noted.

Table with columns: Published weights, Run 1, Run 2, Mean
Published weights	Run 1	Run 2	Mean
Thinking on, retriever e (default)	0.9091	0.9157	0.9124
Thinking on, retriever f	0.9197	0.9127	0.9162
Thinking off, retriever e	0.8746	0.8577	0.8662
4B release 1.3 (thinking off, retriever e)	0.8665	0.8558	0.8612

Thinking off, this model is about level with the 4B; thinking on gives the gain. Retriever f is not clearly better overall (the gap is inside the noise), though d1 rose from a mean of 0.840 to 0.857. Retriever f is a setting of the app, not part of these weights.

Other rounds did not replace these weights: a thinking stage trained on Qwen3.5-397B text (0.8284), extra rounds (0.9109 then 0.8982 on repeat, 0.8841, 0.9084), a weight average (0.8890).

Later rounds did not replace them either: learning rate 1e-5 on the union (0.8961) and on all teacher sets (0.8165, where 41 of 372 replies never finished thinking, against 1 for these weights), and the union plus a targeted requeue set (0.9036). Retriever c on these weights scored 0.9182 and 0.9029. A last round, new facts with the model's own thinking replayed, scored 0.9110 in one run. That is level with these weights, so they stay, and that round's weights were not uploaded.

Product checks on these weights, in product_checks/: European QA sweep of 184 questions with real tool calls, reviewer Qwen3-235B: 4B 1.3 had 36 wrong, this model 27 thinking off and 23 thinking on (one run, one reviewer). 25 mentor cases, key points of 50 in the app: 4B 31, off 34, on 35; the same 4B answers scored 31 to 34 across runs. Without documentation it reached 23 of 50, so it still relies on the app for site facts.

Known gap: with thinking on, 2 of 184 sweep replies and 2 of 25 mentor replies without documents held no answer. GGUF files load and answered one test prompt correctly, but were not scored: the numbers above are for the full weights.

How to read the scores

Each score is a win rate, not accuracy: three judge models compare the answer with the closed-book answer of the base Qwen3-4B, in both orders, with a majority vote; ties count one half. Answers use retrieved documentation excerpts. Thinking is stripped before judging, and a reply whose thinking does not finish counts as a non-answer. Run-to-run noise of the same weights was about 0.01 (the 4B release scored 0.8665 and 0.8558), so small gaps mean little. e8_all (thinking off) was fixed in advance as the primary result; the other runs are extra candidates, so the best of them is a best-of-several figure.

Known limits: prompts over 10,000 tokens lost their start in thinking-off training (46 of 136,452 sequences); over-long thinking answers were skipped. Training prompts hold a few traces of the held-out Aalto family (about 3 per 30,000 sampled), as the 4B data does. Tool calling and the product test suites were not re-run for this model.

Table with columns: run, mean, v2, v3, d1, no answer, hit limit
run	mean	v2	v3	d1	no answer	hit limit
kdall	0.8665	0.8902	0.9264	0.7830	0	2
kdall6	0.8742	0.8826	0.9264	0.8137	0	2
e8_ctrl13	0.8558	0.8636	0.9349	0.7689	0	4
e8_van_off	0.6600	0.6856	0.7072	0.5873	0	0
e8_van_on	0.7452	0.7727	0.7860	0.6769	6	6
tc_t235think	0.8443	0.8788	0.8545	0.7995	19	19
tc_397on	0.8737	0.8788	0.9332	0.8090	2	2
e8_k1	0.8147	0.8144	0.8938	0.7358	0	4
e8_all	0.8746	0.8788	0.9384	0.8066	0	3
e8_all_on	0.9091	0.9242	0.9589	0.8443	1	1
e8_both_on	0.8284	0.8712	0.9041	0.7099	3	3
e8_both_off	0.8663	0.8712	0.9589	0.7689	0	1
e8_x1_on	0.9109	0.9583	0.9418	0.8325	3	3
e8_x2_on	0.8841	0.8788	0.9315	0.8420	0	1
e8_x3_on	0.9084	0.9356	0.9452	0.8443	0	0
e8_all_on2	0.9157	0.9583	0.9538	0.8349	1	2
e8_x1_on2	0.8982	0.8977	0.9572	0.8396	0	0
e8_all_off2	0.8577	0.8409	0.9349	0.7972	0	2
e8_soup_on	0.8890	0.9167	0.9247	0.8255	1	0
e8_f_on	0.9197	0.9545	0.9555	0.8491	0	0
e8_soup_f_on	0.8958	0.9091	0.9435	0.8349	2	3
e8_f_on2	0.9127	0.9394	0.9332	0.8656	0	0
e8_c_on	0.9182	0.9394	0.9521	0.8632	0	1
e9_a_on	0.8961	0.9167	0.9366	0.8349	2	0
e9_b_on	0.8165	0.8220	0.9058	0.7217	41	0
e8_c_on2	0.9029	0.9242	0.9401	0.8443	0	0
e9_6_on	0.9036	0.9242	0.9589	0.8278	3	2
e9_p_on	0.9110	0.9432	0.9503	0.8396	1	1

PRIMARY (fixed in advance): e8_all, thinking off: 0.8746 Thinking teacher: q397fp8 Published weights: e8_all

The pipeline is in training/; GGUF files are in gguf/.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

VaibhavKKM

Model Tree

Base

Qwen/Qwen3-8B

Quantized

this model

Input Modalities

Text

Output Modalities