ningpy

intent-detection-V2.3

README

License: apache-2.0

Performance (val set: 2518 samples)

Table with columns: Metric, Score
Metric	Score
Safety F1 (REDFLAG ∪ BLOCK)	92.3%
Multi-label EM	88.0%
Joint acc (intent + all types)	81.5%
Single-label acc (v11-compatible projection)	90.1%
Intent acc	93.5%
Coexist recall (REDFLAG + BLOCK)	76.2%
Parse err	0.16% (of which ~half are eval-side bugs — model output is valid)
REDFLAG reason_code acc	98.3%

Improvements vs V11 (single-label):

Safety F1: 90.9% → 92.3% (+1.4pp)
REDFLAG F1: 90.0% → 94.1% (+4.1pp)
New multi-label capability: 76.2% coexist recall (v11 was 0%)

Output Schema (V12.9 flat)

6 fixed keys, deterministic order:

json
{
  "intent": "HA|HCN|MT|PI|OTHER",
  "redflag": true|false,
  "redflag_code": "RF-1_GENERAL|RF-2_PEDIATRIC|RF-3_OBSTETRIC|RF-4_ELDERLY|RF-5_IMMUNOCOMP|\"\"",
  "block": "BLOCK_HARD|STEER_SOFT|PASS|NONE",
  "block_code": "OOS_H_*|OOS_S_*|PASS_*|\"\"",
  "entities": {
    "drugs": ["<verbatim from user text>"],
    "diseases": ["..."],
    "symptoms": ["..."]
  }
}

Entities are verbatim substrings of the user message (no translation/normalization).

5 Valid State Combinations

Table with columns: State, redflag, block, Use case
State	redflag	block	Use case
PASS	false	PASS	Normal health knowledge Q
REDFLAG only	true	NONE	Emergency without personal-dose Q
BLOCK only	false	BLOCK_HARD	Personal action / off-topic
STEER only	false	STEER_SOFT	Personal clinical judgment

Training data

54,404 samples (English + Chinese + Bruneian Malay)
Base 27K + STEER×3 + coexist×8 + patch13×3 (targeted narrow rules) + patch16×5 (anti-drift + narrow)
57.1% samples have non-empty entities (GPT-4o back-filled)

Usage with vLLM

bash
python -m vllm.entrypoints.openai.api_server \
    --model ningpy/intent-detection-V2.3 \
    --served-model-name intent \
    --dtype float16 \
    --port 8001

Client sample

See https://git.evyd.tech/peiyan.ning/intent-model-v12/tree/main/sample for a Python client with routing + response templates.

Recommended request settings:

temperature: 0.0
max_tokens: 200
stop: [] (rely on model's natural <|im_end|>)

Downstream routing

Table with columns: Model output, Action
Model output	Action
`redflag=true, block="BLOCK_HARD"`	Emergency guidance + refuse dose advice
`redflag=true, block="NONE"`	Emergency guidance
`redflag=false, block="BLOCK_HARD"`	Refuse + refer to doctor
`redflag=false, block="STEER_SOFT"`	Soft guide to consultation
`redflag=false, block="PASS"`	Normal RAG/LLM answer

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ningpy

Model Tree

Base

Qwen/Qwen2.5-7B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities