What was done
The base model is a sparse MoE with 256 routed experts × 48 layers (plus one always-on shared
expert per layer). REAP scores every routed expert by its contribution and removes the least useful:
saliency(expert e) = mean over tokens routed to e of ( ||expert_e(x)||₂ · w_norm[t,e] )
where w_norm is the renormalized top-k router weight. The 94 lowest-saliency experts per layer
were pruned, keeping 162/256. The router, shared expert, attention, and all other components are
retained; num_experts_per_tok stays 8, so the model remains ~10B active per token — pruning
reduces total/memory footprint, not per-token compute.
Table with columns: Base, This model | Base | This model |
|---|
| Total params | ~122.6 B | ~79.97 B |
| Routed experts / layer | 256 | 162 (94 pruned) |
| Active experts / token | 8 routed + 1 shared | 8 routed + 1 shared |
| Precision | bf16 | bf16 |
Calibration
Medical calibration mix (chat-templated, ~2.1M tokens), blended:
Experts important for the medical distribution are preserved; experts that rarely fire on medical
text are the ones pruned.
Usage (vLLM, bf16)
The architecture is natively supported by vLLM (≥0.22). On 2×80GB GPUs, serve text-only in bf16:
vllm serve <path> \
--tensor-parallel-size 2 --dtype bfloat16 \
--language-model-only --enforce-eager \
--max-model-len 4096 --gpu-memory-utilization 0.97 \
--trust-remote-code
Qwen3.5 reasons by default; disable per request with
chat_template_kwargs: {"enable_thinking": false}.
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model":"<name>",
"messages":[{"role":"user","content":"First-line management of DKA? Answer concisely."}],
"max_tokens":256, "temperature":0.7, "top_p":0.8, "top_k":20,
"chat_template_kwargs":{"enable_thinking":false}
}'
The native context length is 262,144 tokens; the example above caps it at 4096 to fit a small
KV cache in bf16 on 2 GPUs. For larger context or higher concurrency, use more GPUs or FP8.
Limitations
- Not yet benchmarked. Accuracy vs. the 122B base (e.g. MedQA, MedMCQA, PubMedQA, MMLU) has not
been measured. Outputs are coherent in spot checks but quality is unverified.
- Calibration used ~1,024 sequences; some experts were under-observed. A production run should use
≥4,096 calibration sequences.
- The vision tower is retained but the model was validated text-only; multimodal behavior is untested.
- Not a medical device. Not for clinical decision-making without validation and appropriate oversight.
License & attribution
Apache-2.0, inherited from the base model Qwen/Qwen3.5-122B-A10B. Method: REAP
(Cerebras Research).