What it is
Table | |
|---|
| Base | Qwen3-Coder-Next-Base (80B total / ~3B active, hybrid Gated DeltaNet + full Attention + MoE, 512 experts top-10 + 1 shared, 256k context, Apache-2.0) |
| Method | Expert Upcycling (Amazon, arXiv:2604.19835) + parameter freeze |
| Experts | 512 → 544 (k=32 cyber-relevant experts cloned per layer, 48 layers) |
| Total params | 84.5B (80B frozen base + 32 added experts/layer) |
| Trainable | only {new experts 512–543, router, shared expert} = 5.04B of 84.5B — the original 80B is frozen |
| CPT tokens | ~200M (physical shuffled blend), ~3.2 epochs |
| Final train loss | 0.68 |
Method in one paragraph
Starting from the frozen 80B base, we clone k=32 existing experts per layer (selected by utility ‖g‖² on cyber data) to create fresh capacity (512→544 experts). During CPT only the new experts, the router, and the shared expert receive gradients (5.04B trainable). Because the original 512 experts and all attention/backbone weights are frozen, the base model's general and coding skills cannot be eroded. An anti-leak load-balancing term (coef·relu(target − frac_new)) prevents the new experts from being ignored by the router (capacity death): at the end of CPT the new experts carry frac_new ≈ 0.146 of routed mass (target 0.118, uniform 0.059), i.e. they are alive and actively used.
Which experts carry the cyber capacity: in every layer, expert indices 512–543 (the 32 appended experts) are the newly trained cyber experts; indices 0–511 are the frozen original experts. Each new expert is a warm-init clone of one of 32 high-utility base experts (same 32 per layer). Router num_experts_per_tok stays 10 (+1 shared).
Training data
A physical, globally pre-shuffled blend (~200M tokens) mixing curated cyber corpora with filtered cyber web text:
Table with columns: Tier, Share, Content| Tier | Share | Content |
|---|
| web | 55% | filtered cybersecurity web text (CyberFineWeb-clean) |
| depth | 25% | curated in-depth cyber (MITRE ATT&CK, CWE/OWASP, HackTricks, CVE writeups, …) |
| compute-foundations | 7% | RFC networking, Linux/Windows internals, IAM, crypto/NIST |
| reference | 5% | reference docs |
| code | 5% | security-relevant code |
| general |
Training setup
- Framework: Megatron-core 0.18 via ms-swift (
megatron pt)
- Hardware: 4× A100-SXM4-80GB, pipeline-parallel (PP=4)
- Hyperparams: lr 1e-4 → 1e-5 cosine, warmup 2%, grad-clip 1.0, micro-batch 8, global-batch 32, seq-len 256, 24,400 steps (~37h)
- Freeze: grouped-GEMM experts kept as separate params; regex isolates the 32 new experts + router + shared
Intended use & limitations
Intended: a domain-adapted base for building cybersecurity assistants (threat hunting / blue-team and red-team knowledge), as a starting point for downstream SFT/RL. Teaching offensive technique is included deliberately — in prior experiments it improved blue-team/defensive performance.
Limitations:
- Base model — not instruction-tuned; expect raw completion behavior beyond the base's own chat template.
- CPT used a short sequence length (256), so very long-range document structure was under-trained.
- Quantitative cyber benchmarks (MMLU-cyber, held-out tasks) are not yet run — the
frac_new and loss signals show the mechanism worked, but downstream capability gains are still to be measured.
Responsible use
This model encodes offensive-security knowledge for authorized security testing, defensive research, and education. Do not use it to attack systems you do not own or lack explicit permission to test. Users are responsible for complying with applicable law.
License
Apache-2.0, inherited from the base model (Qwen3-Coder-Next-Base).
Citation / lineage
- Base: Qwen3-Coder-Next-Base (Qwen team)
- Upcycling method: Expert Upcycling, arXiv:2604.19835
- CPT recipe & full engineering notes: kalithos-cybersec (
recipes/cpt/RESULT-CPT-V2-2026-07-09.md)