whysue

Qwen3-14B-why

Deploy Dedicated

README

License: apache-2.0

Model Details / 模型详情

Model type: Causal decoder-only LLM (Qwen3 architecture), GRPO-finetuned
Base model: Qwen/Qwen3-14B
Parameters: ~14B, full-parameter fine-tuning (not LoRA)
Language: English prompts and outputs
License: Apache-2.0 (inherited from Qwen3-14B)
Training framework: verl (verl.trainer.main_ppo, adv_estimator=grpo)
Ability tag: pentest_planning

中文

模型类型: 因果解码器 LLM（Qwen3 架构），经 GRPO 强化学习微调
基座模型: Qwen/Qwen3-14B
参数量: 约 14B，全参数微调（非 LoRA）
语言: 英文提示词与输出
许可证: Apache-2.0（继承自 Qwen3-14B）
训练框架: verl（verl.trainer.main_ppo，adv_estimator=grpo）
能力标签: pentest_planning

Task / 任务

EN — Input: engagement state snapshot (Machine / PTT / Previous strategy / Previous step / Previous step result). Output: a fixed-order XML structured decision.
中文 — 输入: 渗透测试引擎的状态快照（Machine / PTT / Previous strategy / Previous step / Previous step result）。输出: 固定顺序的 XML 结构化决策。

Prompt Format / 提示词格式

EN: The model relies on a fixed system prompt to constrain its structured output. At inference time, reuse the training system prompt, otherwise the reward-aligned structured output degrades.

中文: 模型使用固定的 system prompt 约束输出结构。推理时请复用训练时的 system prompt，否则奖励对齐的结构化输出会退化。

System prompt:

markdown
You are an expert penetration-testing planning agent. Given the current state of an
engagement against a target machine — including the Penetration Testing Tree (PTT) that
tracks completed and pending tasks, and the outcome of the previous step — decide the
single best next action.

Your response MUST strictly use the following XML tags in order:
<think>
[step-by-step reasoning: strategy analysis then step analysis]
</think>
<new_strategy>[a concise high-level strategy]</new_strategy>
<new_step>[the concrete next step to execute]</new_step>
<mcp_tasks>[a JSON object mapping tool -> task, e.g. {"Metasploit": "Exploit CVE-XXXX."}]</mcp_tasks>

User message (state template) / 用户消息（状态模板）:

markdown
## Target machine
{machine}

## Penetration Testing Tree (current progress)
{ptt}

## Previous strategy
{prev_strategy}

## Previous step
{prev_step}

## Previous step result
{prev_result}

Decide the best next action now.

Expected output (example) / 期望输出（示例）:

xml
<think>
...strategy analysis...
...step analysis...
</think>
<new_strategy>Enumerate exposed web service for known CVEs</new_strategy>
<new_step>Run a targeted vulnerability scan against the web port identified in the PTT</new_step>
<mcp_tasks>{"Nmap": "Scan http service for version and known CVEs", "Metasploit": "Search matching exploit modules"}</mcp_tasks>

How to Get Started / 快速开始

Transformers

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "path/to/Qwen3-14B-Pentest-GRPO"  # this model dir / 本模型目录
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

SYSTEM = (
    "You are an expert penetration-testing planning agent. ..."  # full system prompt above / 使用上文完整 system prompt
)
USER = (
    "## Target machine\n<machine>\n\n"
    "## Penetration Testing Tree (current progress)\n<ptt>\n\n"
    "## Previous strategy\n<prev_strategy>\n\n"
    "## Previous step\n<prev_step>\n\n"
    "## Previous step result\n<prev_result>\n\n"
    "Decide the best next action now."
)

messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=1024, temperature=0.7, top_p=0.95)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

vLLM (recommended for deployment / 推荐部署方式)

bash
python -m vllm.entrypoints.openai.api_server \
    --model path/to/Qwen3-14B-Pentest-GRPO \
    --max-model-len 12288

EN: Trained with max_prompt_length=6144, max_response_length=4096. The PTT context can be long, so set --max-model-len ≥ 10k when serving. 中文: 训练时 max_prompt_length=6144、max_response_length=4096。PTT 上下文可能很长，部署时 --max-model-len 建议 ≥ 10k。

Training Details / 训练细节

Training Data / 训练数据

EN: Sourced from a penetration-testing decision CSV where each row is one "decision point" in an engagement. Input columns: Machine, PTT, Previous strategy, Previous step, Previous step result. Reference-decision columns: New strategy, Strategy explanation, New step, Step explanation, MCP_tasks. Converted by convert_to_grpo.py into verl parquet schema (data_source / prompt / ability / reward_model / extra_info); the reference decision is packed as JSON into reward_model.ground_truth.

中文: 来源于渗透测试决策 CSV，每行是一次交战过程中的一个「决策点」。输入列：Machine, PTT, Previous strategy, Previous step, Previous step result；参考决策列：New strategy, Strategy explanation, New step, Step explanation, MCP_tasks。由 convert_to_grpo.py 转为 verl parquet schema（data_source / prompt / ability / reward_model / extra_info），参考决策打包为 JSON 存入 reward_model.ground_truth。

Table with columns: Split / 划分, Samples / 样本数
Split / 划分	Samples / 样本数
train / 训练	1,894
validation (test) / 验证	268

EN: Raw CSVs have ~27.7k / ~7.4k decision points; filter_overlong_prompts=True + truncation='error' filtered out samples exceeding 6144 tokens. GRPO needs no (prompt, response) supervised pairs — it samples multiple responses per prompt and estimates advantage from group-relative rewards. 中文: 原始 CSV 约 27.7k / 7.4k 条决策点；filter_overlong_prompts=True + truncation='error' 过滤掉超过 6144 token 的样本。GRPO 无需 (prompt, response) 监督对，它对同一 prompt 采样多个回答并用组内相对奖励估计优势。

Reward Function / 奖励函数

EN: Rule-based reward (offline, no external LLM), normalized to [0, 1] (see pentest_reward.py). 中文: 规则型奖励（离线、无需外部 LLM），归一化到 [0, 1]（见 pentest_reward.py）。

Table with columns: Dimension / 维度, Weight / 权重, Description / 说明
Dimension / 维度	Weight / 权重	Description / 说明
Format / 格式分	0.4	All four XML tags present (0.1 each) / 四个 XML 标签是否齐全（每个 0.1）
Tool hit / 工具命中	0.4	Predicted `<mcp_tasks>` tool names vs reference (core signal) / 生成的工具名 vs 参考工具名（核心信号）
Strategy sim / 策略相似	0.2	Word-level Jaccard of `<new_strategy>` vs reference / `<new_strategy>` 与参考的词级 Jaccard

EN/中文: Can be replaced with an LLM-as-judge for stronger semantic scoring. / 可替换为 LLM-as-judge 以获得更强的语义判分。

Training Procedure / 训练过程

EN: GRPO (adv_estimator=grpo, no critic) on verl + FSDP (full-param) + vLLM async rollout. Rollouts per prompt rollout.n=8; KL regularization use_kl_loss=True, kl_loss_coef=0.001, kl_loss_type=low_var_kl.

中文: GRPO（adv_estimator=grpo，无 critic），基于 verl + FSDP（全参数）+ vLLM 异步 rollout。每 prompt 采样 rollout.n=8；KL 正则 use_kl_loss=True、kl_loss_coef=0.001、kl_loss_type=low_var_kl。

Hyperparameters / 超参数

Table with columns: Item / 项, Value / 值
Item / 项	Value / 值
learning rate / 学习率	1e-6
optimizer / 优化器	AdamW (betas 0.9/0.999, weight_decay 0.01, clip_grad 1.0)
lr scheduler / 学习率调度	constant
train_batch_size	16
ppo_mini_batch_size	8
ppo_micro_batch_size_per_gpu	1
rollout.n	8
rollout temperature	1.0
max_prompt_length

Compute / 算力

Table with columns: Item / 项, Value / 值
Item / 项	Value / 值
Hardware / 硬件	4 × NVIDIA H800 (80GB)
vLLM tensor parallel / 张量并行	TP=2
gpu_memory_utilization	0.45 (FSDP + vLLM colocate)
offload	actor / ref param + optimizer offload = True
Wall-clock time / 训练时长	≈ 38 hours / 约 38 小时
Precision / 精度	bf16

Evaluation / 评估

EN: Evaluated on the validation set (268 samples) using the custom reward reward/mean@4 (mean over 4 samples per prompt). 中文: 在验证集（268 样本）上以自定义奖励 reward/mean@4（每 prompt 4 次采样均值）评估。

Table with columns: Global Step / 全局步, val reward/mean@4
Global Step / 全局步	val reward/mean@4
100	0.353
500	0.399
1000	0.415
1500	~0.426
1770 (final / 最终)	0.425

EN — Summary: From a starting point of ≈0.353, validation reward rose to ≈0.426 (~+20% relative gain) and plateaued around step ~1500. Training stayed stable throughout: train/inference probability agreement rollout_actor_probs_pearson_corr ≈ 0.999, pg_clipfrac ≈ 1e-3, no divergence.

中文 — 结论: 相较起点（≈0.353），验证奖励提升到 ≈0.426（约 +20% 相对提升），并在训练中段（~step 1500）趋于饱和。训练全程稳定：训练/推理概率一致性 rollout_actor_probs_pearson_corr ≈ 0.999，pg_clipfrac ≈ 1e-3，无发散。

EN: This metric is a rule-based reward (format compliance + tool-name hit + strategy word overlap), NOT real-world attack success on live targets. Real efficacy must be evaluated separately in a controlled lab. 中文: 该指标由规则型奖励度量（结构合规 + 工具命中 + 策略词重叠），并非真实靶机上的攻击成功率。真实效果需在受控靶场中另行评估。

Bias, Risks, and Limitations / 偏差、风险与局限

⚠️ Dual-use / security risk: The model generates offensive-security actions and tool-invocation suggestions. Use it only in explicitly authorized environments for lawful security testing, red-teaming, CTF, and research. Never use it on unauthorized systems. Users bear full compliance and legal responsibility.
Reward-proxy bias: The training reward is a rule-based proxy (tag completeness / tool-name hit / strategy word overlap), which may encourage "gaming the format and keywords" rather than truly optimal tactical decisions (reward-hacking risk).
No real-world execution validation: Outputs are not validated for success on live targets; tools and CVEs in <mcp_tasks> may be inaccurate or outdated and must be reviewed by a human expert before execution.
Data limitations: Training data comes from a specific pentest decision corpus with limited coverage of machine types, vulnerabilities, and tools; out-of-distribution generalization is unknown.
Hallucination: May fabricate non-existent CVEs, tools, or paths.
Language: Optimized for English prompts only.

中文

⚠️ 双重用途 / 安全风险: 模型会生成攻击性安全动作与工具调用建议。仅限在获得明确授权的环境中用于合法的安全测试、红队、CTF 与研究，禁止用于任何未授权系统。使用者须自行承担合规与法律责任。
奖励代理偏差: 训练奖励是规则型代理指标（标签齐全度 / 工具名命中 / 策略词重叠），可能激励模型"迎合格式与关键词"而非产出真正最优的战术决策（reward hacking 风险）。
无真实执行验证: 模型输出未在真实靶机上验证成功率；<mcp_tasks> 中的工具与 CVE 可能不准确或过时，必须由人工安全专家审核后再执行。
数据局限: 训练数据来自特定渗透测试决策语料，覆盖的机器类型、漏洞与工具有限；对分布外目标的泛化能力未知。
幻觉: 可能编造不存在的 CVE、工具或路径。
语言: 仅针对英文提示优化。

Recommendations / 使用建议

EN: Always have a qualified security professional review every output before execution; use only in isolated/authorized environments; reuse the training system prompt for structured output; use an LLM-as-judge or real lab regression for more reliable evaluation.
中文: 始终由具备资质的安全人员人工复核每一步输出后再执行；仅在隔离/授权的实验环境中使用；推理时复用训练 system prompt以保持结构化输出；如需更可靠评估，替换为 LLM-as-judge 或真实靶场回归测试。

Framework / Reproduction / 框架与复现

Base / 基座: Qwen/Qwen3-14B
RL: verl main_ppo with adv_estimator=grpo
Rollout: vLLM (FLASH_ATTN backend)
EN: Training script, data conversion, reward function, and end-to-end reproduction steps are in the project's run_grpo_qwen3_14b.sh, convert_to_grpo.py, pentest_reward.py, GRPO_TRAIN_README.md.
中文: 训练脚本、数据转换、奖励函数与端到端复现步骤见项目内 run_grpo_qwen3_14b.sh、convert_to_grpo.py、pentest_reward.py、GRPO_TRAIN_README.md。

Software / 软件版本

Table with columns: Component / 组件, Version / 版本
Component / 组件	Version / 版本
Python	3.10
torch	2.9.0 (cu128)
vllm	0.12.0
flash-attn	2.8.3
transformers	4.57.x
ray	2.56.0
verl	(see repo / 见仓库)

Citation / 引用

EN/中文: Please also cite Qwen3, verl, and GRPO. / 请同时引用 Qwen3、verl 与 GRPO。

bibtex
@misc{qwen3,
  title  = {Qwen3 Technical Report},
  author = {Qwen Team},
  year   = {2025}
}

@article{shao2024deepseekmath,
  title   = {DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models},
  author  = {Shao, Zhihong and others},
  journal = {arXiv preprint arXiv:2402.03300},
  year    = {2024},
  note    = {Introduces GRPO}
}

@misc{verl,
  title  = {verl: Volcano Engine Reinforcement Learning for LLMs},
  author = {ByteDance Seed and verl Community},
  url    = {https://github.com/volcengine/verl}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.