tomhu

Qwen3-Coder-Next-SWE-Prune-41B-A3B

README

License: apache-2.0

What changed

Routed experts per MoE layer: 512 → 256 (50% retained).
Activated routed experts per token: 10 (unchanged).
Router rows were remapped to the retained experts.
Dense, attention, shared-expert, tokenizer, and chat-template weights are unchanged.
No fine-tuning was performed after pruning.
Experts were selected per layer using an importance score calibrated on successful, submitted SWE-style agent trajectories.

The checkpoint uses the official Qwen FP8 block-quantized format and is directly loadable by recent vLLM/Transformers versions that support Qwen3NextForCausalLM.

Parameter count

Counts below were computed directly from every safetensors header. Quantization scale tensors are excluded from learned-parameter counts.

Table with columns: Metric, Exact count, Model-name convention
Metric	Exact count	Model-name convention
Total learned parameters	40,994,519,808	41B
Activated parameters per token	3,849,763,584	A3B
Routed-expert parameters	38,654,705,664	—
Always-active parameters	2,339,814,144	—

The A3B label follows the upstream Qwen convention. Under the same counting method, the original 80B checkpoint has 3.87B active parameters and is officially described as A3B.

Architecture

Layers: 48
Hidden size: 2,048
Retained routed experts per layer: 256
Activated routed experts per token: 10
Shared experts per layer: 1
Expert intermediate size: 512
Native context length: 262,144
Inference mode: non-thinking

Preliminary SWE-bench Verified evaluation

Using the unmodified official SWE-bench harness and the official mini-swe-agent prompt with 250 steps, temperature=1.0, top_p=0.95, and top_k=40:

Fixed partial snapshot: 78/112 resolved (69.6%).
Most recent incremental snapshot: 41/55 resolved (74.5%).

These are partial, non-random snapshots collected while the full 500-instance run was still in progress, so they must not be interpreted as the final full-dataset score.

vLLM example

bash
vllm serve tomhu/Qwen3-Coder-Next-SWE-Prune-41B-A3B \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --moe-backend triton \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --generation-config vllm

Recommended sampling parameters:

json
{
  "temperature": 1.0,
  "top_p": 0.95,
  "top_k": 40,
  "chat_template_kwargs": {"enable_thinking": false}
}

For long multi-turn SWE-agent workloads, 32 concurrent agents were substantially more stable than 64 on four 48GB GPUs because the lower concurrency preserved the prefix cache.

Provenance and audit files

The repository includes the retained-expert mapping, per-layer importance scores, source revision metadata, tensor-level validation results, and shard checksums.

Upstream revision: Qwen/Qwen3-Coder-Next-FP8@da6e2ed27304dd39abadd9c82ef50e8de67bdd4c

License

This derivative checkpoint follows the upstream Apache-2.0 license. See the upstream model card for the original model details and attribution.

Citation

bibtex
@techreport{qwen_qwen3_coder_next_tech_report,
  title  = {Qwen3-Coder-Next Technical Report},
  author = {{Qwen Team}},
  url    = {https://github.com/QwenLM/Qwen3-Coder/blob/main/qwen3_coder_next_tech_report.pdf}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

tomhu

Model Tree

Base

Qwen/Qwen3-Coder-Next-FP8

Quantized

this model

Input Modalities

Text

Output Modalities