Use
The model answers with a JSON list of {"type", "value"} entries, each value copied verbatim from the
text. Ask exactly as it was trained: the system prompt below, the text after Text:\n, thinking off,
greedy decoding.
import jsonfrom transformers import AutoModelForCausalLM, AutoTokenizer name = "Liquor-06/Qwen3-0.6B-PII-distillops"tok = AutoTokenizer.from_pretrained(name)model = AutoModelForCausalLM.from_pretrained(name, dtype="auto", device_map="auto") SYSTEM = ( 'List every piece of personal information (PII) in the text so that it can be masked.\n' '- Copy each value exactly as it appears in the text, character for character.\n' '- Words of the same category that stand next to each other form one entry: a full name with its title ("Dr Ana Maria Lee"), a street with its number ("12 Oak Street"), a city with its postal code ("Boston 02108"). Pieces separated by other words or by punctuation (a comma, a hyphen) are separate entries.\n' '- Every city or town name counts, even on its own. For an age, list only the number ("27", not "27 years old"); gender and sex words count too.\n' '- List a value once even if it appears several times.\n' 'Answer with a JSON list only: [{"type": "<CATEGORY>", "value": "<exact text>"}, ...], or [] if the text has no personal information.\n' '\n' 'Categories:\n' "- NAME: a person's name, with any title (Mr, Dr, ...)\n" '- CONTACT: an email address or phone number\n' '- ADDRESS: a street name, building number, city or postal code\n' "- ID_NUMBER: an ID card, passport, driver's license, social security or tax number\n" '- CARD_NUMBER: a payment card number\n' '- DATE: a date\n' "- DEMOGRAPHIC: a person's age, gender or sex") text = "Dear Mr John Smith, your appointment on 12 March 2024 is confirmed. Call us on 020 7946 0958."messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": f"Text:\n{text}"}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=512, do_sample=False)entities = json.loads(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))print(entities) # e.g. [{"type": "NAME", "value": "Mr John Smith"}, {"type": "DATE", ...}, ...] masked = textfor e in sorted(entities, key=lambda e: -len(e["value"])): # longest first masked = masked.replace(e["value"], f"[{e['type']}]")print(masked)
Pass do_sample=False: generation_config.json keeps Qwen3's sampling defaults, and all scores above
are greedy. Generation stops at <|im_end|>. With vLLM: vllm serve Liquor-06/Qwen3-0.6B-PII-distillops,
then send the same messages with chat_template_kwargs: {"enable_thinking": false} and temperature: 0.
Categories: NAME (with any title), CONTACT (email, phone), ADDRESS (street, building number, city,
postal code), ID_NUMBER (ID card, passport, driver's license, social security, tax number),
CARD_NUMBER, DATE, DEMOGRAPHIC (age, gender, sex). Adjacent words of one category form one entry
("Dr Ana Maria Lee", "12 Oak Street").
Training
- Data: OpenPII-1M (synthetic,
English rows); the dataset's 19 labels merged into the 7 categories above. Disjoint splits: 10,000
texts for the teacher, 2,000 labelled texts for GRPO, 300 for checkpoint selection, 2,000 for testing.
- Teacher labels, no human labels: DeepSeek (
deepseek-flash, 3 worked examples) labelled the
10,000 texts. An answer was accepted by rules alone (known categories, every value copied verbatim,
plausible formats, no 9+ digit number left unmasked), as for a company's own unlabelled text: 9,681
texts (96.8%) passed. Gold labels of 100 texts were used only to write the teacher's instructions.
- SFT: LoRA (r 32, alpha 64) on Qwen3-0.6B, 2 epochs, lr 1e-4, effective batch 32,
completion-only loss; merged into the base weights.
- GRPO: reward = per-text F1 against the 2,000-text labelled pool; 8 samples per prompt, DAPO loss,
lr 1e-6, no KL term, full fine-tuning; 850 prompts whose samples disagreed, 2 epochs (54 steps).
Validation F1 0.952 -> 0.965, best at the last step.
- Weights are bf16 (training kept fp32 master weights; evaluation ran in bf16). On the first 100 test texts the bf16 weights give the
same mean F1 as that evaluation, with 99 of 100 outputs identical.
Commands, configs and the full write-up:
github.com/Zihao-Wu06/distillops.
Limitations
- English only, and trained and tested on one synthetic dataset of mostly formal text (letters,
forms, notices, chats). Scores measure agreement with that dataset's annotation conventions, not
with every organisation's policy.
- Only the 7 categories above: other identifiers (IP addresses, account numbers, URLs, organisation
names, health information) are not targeted.
- It misses about 2% of personal information on this test set and will miss more on text unlike it.
Check it on your own data before relying on it, and do not treat masking with it as a guarantee
of anonymisation.
License
Apache-2.0, as for the base model. OpenPII-1M is CC-BY-4.0 (ai4privacy); the training labels came
from the DeepSeek API (check DeepSeek's terms of use for your use case).