shawnw3i

Qwopus3.8-27B-Flash-AWQ-MTP

README

License: apache-2.0

💡 1. Fine-Tuning Philosophy — Preserve Capability, Collapse Cost

Qwopus3.8-27B-Flash starts from a simple premise: a useful Flash model should preserve enough capability to complete demanding work while driving down the cost and latency of reasoning.

An agent amplifies reasoning cost. A normal chat may call a model once, but one agent task can call it dozens or hundreds of times through a repeated loop:

text
Read → Think → Tool Call → Observe → Edit → Test

[!IMPORTANT] Wall-clock time is what users feel. If a model adds five seconds to each turn, a 50-turn task adds 250 seconds of waiting. In the agent era, every-token speed, reasoning length, and completion efficiency are as important as a single-turn benchmark result.

Inference cost is also a central commercialization constraint. Every generated token consumes GPU time, electricity, memory capacity, and concurrency budget. Flash models are intended to give users with constrained resources a more practical experience, not merely to produce a higher throughput number on paper.

The goal of Qwopus3.8-27B-Flash is therefore to reduce ineffective computation and reach a clean completion more quickly and consistently. The benchmark results below show that this optimization has an explicit trade-off: it improves inference efficiency, while the reported MMLU-Pro mixed-set score is lower than the base comparison.

Five-Story Pagoda Garden — Visual Output Comparison

[!NOTE] These visuals are qualitative model outputs. They complement the measured benchmark results below and are not a quantitative capability score on their own.

Original Base Model — Qwen3.8-27B

Original model — Five-Story Pagoda test

Fine-Tuned Model — Qwopus3.8-27B-Flash

Fine-tuned model — Five-Story Pagoda test

🧪 2. Fine-Tuning Cookbook

2.1 Base Model and Training Goal

Qwopus3.8-27B-Flash is fine-tuned from the Qwen3.8-27B foundation model. The training objective is not simply to maximize visible reasoning length or claim a universal benchmark gain. It is to retain practical problem-solving ability while making the model faster, less prone to pathological long-tail reasoning, and better suited to iterative agent workflows.

2.2 Stage 1 — Data Preparation and Quality SFT

Data preparation is the foundation of the SFT stage. The first stage began with approximately 1.5 million teacher-model SFT examples. After extensive cleaning and filtering, the highest-quality 10% was retained for the initial fine-tuning stage.

Each example was evaluated across its three main components—question, chain of thought, and answer—with criteria tailored to a 27B model:

Semantic relevance and usefulness.
Problem difficulty.
Chain-of-thought quality.
Answer consistency.

The evaluation ensemble used reasoning models including Qwen3.7-Max, GLM-5, GPT-OSS-120B-High, and Gemma4-27B. Their scores were combined through a weighted calculation, after which only high-quality examples were retained.

[!NOTE] The training mix also includes agent-trajectory data and reconstructed trace data derived from closed models such as Claude and GPT. Detailed examples cannot currently be disclosed. The dataset will be further organized and released after preparation is complete.

2.3 Stage 2 — NeMo-RL + GSPO Reasoning Reinforcement

Stage 2 consolidates and strengthens the chain-of-thought and reasoning behaviors learned during SFT. This stage follows an NVIDIA NeMo-RL + GSPO route for reasoning reinforcement.

The training process uses repeated sampling and reward comparison, then updates the model with a sequence-level importance ratio. The intent is to reinforce useful reasoning trajectories and completion behavior rather than reward longer reasoning traces by default.

[!TIP] The two stages serve different roles: Stage 1 builds a high-quality reasoning and instruction-following foundation; Stage 2 consolidates it through reward-guided reasoning reinforcement.

🙏 3. Training Stack, Testing, and Collaboration

Unsloth

Special thanks to Unsloth for its efficient, memory-optimized fine-tuning framework. Its tooling made the large-model training workflow more practical and accessible.

Kyle Hessling

Special thanks to Kyle Hessling for completing the testing work and providing essential evaluation support for this release.

[!TIP] If you have any questions or suggestions about the model, please feel free to reach out to Kyle Hessling on X and share your feedback. Thank you so much for all your support! 🙏

[!NOTE] The training and test results in this card are author-provided local results. The sections below describe the measured scope and do not claim universal behavior across all hardware, prompts, or agent environments.

[!WARNING] ⚠️ Known Issue

In some specific Python coding tasks, the model may generate code with incorrect indentation. A fix is currently in progress.

This issue is isolated to Python code indentation and does not affect the model's other capabilities.

📊 4. Benchmark Results

Table with columns: Reported signal, Qwopus3.8-27B-Flash, Qwen3.8 base comparison, Interpretation
Reported signal	Qwopus3.8-27B-Flash	Qwen3.8 base comparison	Interpretation
MMLU-Pro mixed-question accuracy (2,500 questions)	91.28%	92.73%	−1.45 pp
Decoder-only throughput across Math, Physics, and Chemistry	9.347 tok/s	8.284 tok/s	+12.8%
Weighted MTP draft acceptance	80.7%	66.1%	+14.6 pp
Aggregate `raw_output` characters across three subjects

4.1 Evaluation Scope

The reported local tests used inference environments that included NVIDIA GeForce RTX 5090, NVIDIA V100, and NVIDIA GB10 hardware. The current test brief does not map every table or case study to a particular device.

For the MMLU-Pro decoder-only comparison, both models used the same Q5_K_M + MTP configuration:

text
draft_n_max = 2
-c 327680
-np 10

The agentic software-engineering battery in Section 4.6 is a separate run explicitly reported on one RTX 5090 and uses its own configuration, listed with that battery.

4.2 MMLU-Pro Accuracy Reference and Five-Layer Tower Renders

[!WARNING] Accuracy–efficiency trade-off: The fine-tuned model improves reasoning efficiency, but its reported score on the 2,500-question MMLU-Pro mixed set is 91.28%, compared with 92.73% for the Qwen3.8 base comparison—a decrease of 1.45 percentage points. The decoder-only throughput results below measure speed, not answer accuracy.

4.3 MMLU-Pro Decoder-Only Throughput

This table measures isolated generated-token throughput rather than answer accuracy. Evaluation time is in seconds; throughput is generated tokens per second.

Table with columns: Subject, Qwopus generated, Qwopus eval (s), Qwopus tok/s, Qwen generated, Qwen eval (s), Qwen tok/s, Gain
Subject	Qwopus generated	Qwopus eval (s)	Qwopus tok/s	Qwen generated	Qwen eval (s)	Qwen tok/s	Gain
Math	753,406	81,212.1	9.277	804,071	96,614.0	8.323	+11.5%
Physics	1,055,509	111,465.3

Across the three reported subjects, Qwopus reaches 9.35 tok/s versus 8.28 tok/s when rounded to two decimals. Every reported subject shows a double-digit decoder-only throughput gain. The separately reported ~8% end-to-end batch-throughput result includes the full evaluation path and should not be combined with this decoder-only measurement.

4.4 MTP Draft Acceptance

Draft acceptance is an internal llama.cpp/MTP implementation metric. The reported aggregate is a weighted calculation, rather than an average of subject-level percentages:

text
sum(accepted) / sum(generated)

Table with columns: Subject, Qwopus3.8-27B-Flash, Qwen3.8 base comparison, Difference
Subject	Qwopus3.8-27B-Flash	Qwen3.8 base comparison	Difference
Math	81.1%	66.3%	+14.8 pp
Physics	79.2%	64.6%	+14.5 pp
Chemistry	81.7%	67.1%	+14.6 pp
Three-subject weighted total	80.7%

Table with columns: Three-subject raw counts, Accepted, Generated
Three-subject raw counts	Accepted	Generated
Qwopus3.8-27B-Flash	1,998,599	2,476,486
Qwen3.8 base comparison	1,821,083	2,755,425

[!TIP] The +14.6 pp result is an absolute difference in percentage points. It is not a +14.6% relative-improvement claim.

4.5 Reasoning-Efficiency Findings

The following values are raw_output character counts, not token counts. Mean, P95, and median describe different parts of the output-length distribution.

Table with columns: Subject, Mean Qwen, Mean Qwopus, Mean change
Subject	Mean Qwen	Mean Qwopus	Mean change
Math	4,614.2	3,798.7	−17.7%
Physics	5,995.0	5,469.6	−8.8%
Chemistry	7,039.2	6,630.8	−5.8%

Table with columns: Aggregate raw_output characters, Qwen3.8 base comparison, Qwopus3.8-27B-Flash, Change
Aggregate `raw_output` characters	Qwen3.8 base comparison	Qwopus3.8-27B-Flash	Change
Math + Physics + Chemistry	8,824,213	7,949,546	−9.9%

The reported data suggests that Qwopus reduces pathological long-tail reasoning while retaining sufficient reasoning depth on typical samples. P95 output length falls by 13–40% depending on subject, while median output length increases in all three subjects. This pattern suggests less runaway reasoning rather than uniform suppression of reasoning.

4.6 Agentic Software-Engineering Battery

This held-out battery evaluates 14 agentic software-engineering tasks with deterministic hidden-test verifiers. There is no LLM-as-judge and no partial credit: a task passes only when its complete hidden suite passes. Before evaluation, every task was checked with negative and positive controls—the untouched starting state had to fail, and a reference solution had to pass. Hidden tests remain outside the agent workspace and are SHA-256 hashed before and after each run to detect tampering.

[!IMPORTANT] Strict result: 13 / 14 passed (93%). The 95% Wilson confidence interval is 69%–99%. Total wall-clock time was 26.0 minutes on one RTX 5090. A task counts only if the agent both passes its hidden suite and finishes within its per-task wall-clock budget.

Table with columns: #, Capability, What it tests, Result, Turns, Wall
#	Capability	What it tests	Result	Turns	Wall
T01	Multi-bug repair	Diagnose and fix 3 distinct bugs from a failing suite	✅ pass	10	55s
T02	Runtime debugging	Run a crashing script, read the traceback, and fix the root cause	✅ pass	8	57s
T03	Cross-file refactor

Inference Characteristics

Table with columns: Metric, Value
Metric	Value
Generation speed	99.0 tok/s mean across tasks
MTP draft acceptance	88.8% mean
Agent turns per task	10.1 mean; 3–22 range
Output tokens per task	2,340 mean; 30,414 total
Integrity violations	0; hidden tests remained untampered after every task

The battery used Qwopus 3.8 27B in Q5_K_M GGUF format with a bundled NextN/MTP head, self-speculative decoding (--spec-type draft-mtp), q8_0 KV cache, and thinking disabled. It was run through headless Claude Code against a local llama.cpp server with real file and shell tools.

Scoring, Failure Analysis, and Repeat Sampling

T07 left a file that passed its hidden suite (7 passed in 0.01s), but the agent kept iterating and did not return before the 420-second limit. It is therefore correctly scored as a strict failure: completing the task includes knowing when to stop. This was a termination and verbosity failure, not necessarily an inability to implement the parser.

Table with columns: Task, Attempts, Passed, Rate, Observed failure modes
Task	Attempts	Passed	Rate	Observed failure modes
T01 Multi-bug repair	2	2	100%	—
T07 Parser / interpreter	7	5	71%	Incorrect output; non-termination loop

[!WARNING] Sampling is stochastic. Only T07, the failing task, was resampled; the other 13 tasks are single-sample passes. The 13/14 headline is therefore a one-draw capability demonstration, not a stable leaderboard number, and is more likely optimistic than pessimistic. A stable estimate requires multiple seeds for every task.

4.7 Reported Agent and Practical Workload Case Studies

These are individual reported workload case studies, not aggregate success rates or universal behavior claims.

Case Study A — Agent Workload Comparison

Table with columns: Metric, Qwopus3.8-27B-Flash, Qwen3.8 base comparison
Metric	Qwopus3.8-27B-Flash	Qwen3.8 base comparison
Wall-clock	640.2 s (10.7 min)	1,389.8 s (23.2 min)
Turns	20, completed	48, crashed
Early-turn behavior	Normal	First turns hit 32K cap
Generated tokens	27,975	109,337
Average MTP acceptance	80.1%	72.7%
Final status

In this reported workload, the base comparison generated more tokens and spent more time but did not complete, while Qwopus completed with fewer turns and fewer generated tokens.

Case Study B — Long Code / Web-Generation Workload

Table with columns: Metric, Qwopus3.8-27B-Flash, Qwen3.8 base comparison
Metric	Qwopus3.8-27B-Flash	Qwen3.8 base comparison
Completion tokens	14,379	60,000 ceiling
Finish reason	stop (complete)	length (incomplete)
Thinking	2,959 chars / 21 s	98,452 chars / 15.5 min
Answer	51,844 chars, complete	70,116 chars, truncated
Wall time	4.8 min	24.4 min

This case illustrates a reported valid long-form completion without hitting the token ceiling and with less reported internal reasoning. It does not establish that every web- or code-generation task will show the same ratio.

Case Study C — Long-Horizon Agent Workload

Table with columns: Reported Qwopus3.8-27B-Flash result, Value
Reported Qwopus3.8-27B-Flash result	Value
Test outcome	22 / 22 tests passed
Wall time	22.7 min
API time	1,061 s
Turns	56
Tool calls	55
Tool-call mix	Bash 20 · Write 17 · Edit 11 · Read 7
Tool errors	5, all self-recovered
Input tokens	~101K
Cumulative cache-read tokens

[!IMPORTANT] These case studies are workload-specific evidence. They must not be converted into a general pass rate, API reliability guarantee, or claim that Qwopus is universally faster or more capable.

🎯 5. Recommended Use Cases

Local MTP speculative decoding where decoder throughput and draft acceptance matter.
Long-running agent workflows with repeated tool calls and iterative edits.
Resource-conscious workloads where wall-clock time and generated-token cost are operational constraints.

⚠️ 6. Limitations

Accuracy–Efficiency Trade-off: The Flash fine-tuning objective prioritizes practical inference efficiency. On the reported 2,500-question MMLU-Pro mixed set, Qwopus scores 91.28% versus 92.73% for the Qwen3.8 base comparison.
Reasoning Stability: Although the long-tail behavior is improved in the reported evaluation, edge cases may still exhibit reasoning drift, loops, or inefficient trajectories.
Agent Dependence: Tool-use quality and task completion remain dependent on the prompt, tool environment, orchestration layer, and feedback loop.
Experimental Release: This is an independent experimental release for research, local evaluation, and technical exploration. It has not undergone broad safety evaluation or universal agent-reliability testing.

🙏 Acknowledgements

Special thanks to:

Qwen for the Qwen3.8-27B base-model foundation.
Unsloth for practical, efficient large-model fine-tuning tooling.
Kyle Hessling for completing testing and evaluation support.
The open-source community for training tools, evaluation methods, and technical discussion.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider