Measured results
Final evaluation used 66 held-out, AI-authored fictional tickets, 11 per issue category. The selected adapter was frozen before final test evaluation. Both models used the same prompt, base revision, greedy prediction settings, test inputs, and scoring code.
Table with columns: Metric, Base model, Fine-tuned, Change| Metric | Base model | Fine-tuned | Change |
|---|
| Schema-valid structured answers | 53.0% | 98.5% | +45.5 percentage points |
| Category macro-F1 | 0.471 | 0.881 | +0.411 F1 |
| Environment accuracy | 34.8% | 90.9% | +56.1 percentage points |
| Team accuracy | 18.2% | 90.9% | +72.7 percentage points |
Correct issue labels increased from 25/66 to 58/66. Responses failing the required output schema fell from 31/66 to 1/66. Rates and changes are independently rounded from full-precision results.
Scoring: invalid-schema responses count as incorrect for every task metric. Macro-F1 averages F1 equally across six categories, not percent correct. Team is deterministically derived from category and is not an independent reasoning capability. Summary factuality/completeness are not automatically measured by these scores.

Machine-readable comparison · Base-model metrics · Fine-tuned metrics
Task and output
Input: one short English support ticket. Output: a JSON object with exactly four fields:
{ "category": "authentication", "environment": "production", "customer_reported_impact": "The customer cannot access the production dashboard.", "suggested_team": "developer_support"}
The JSON above illustrates the output contract; it is not a saved inference result.
- Category:
authentication, billing, rate_limit, integration, bug, or other.
- Environment:
production, staging, development, or unknown when not established.
- Reported impact: a factual short summary, or
null when no impact is stated.
- Team:
billing_support for billing, general_support for other, and developer_support for remaining categories.
Mixed independent requests with no dominant issue are labeled other. Tickets are treated as data rather than instructions. Predictions should be reviewed before business actions; the model does not resolve issues or execute changes.
Training and selection
Table with columns: Setting, Value| Setting | Value |
|---|
| Base model | Qwen/Qwen3-0.6B |
| Exact base version | c1899de289a04d12100db370d81485cdf75e47ca |
| Data | 128 training / 34 validation / 66 test tickets |
| Provenance | AI-authored fictional examples; agent review and user acceptance for a pilot |
| Method | LoRA, rank 16, alpha 32, dropout 0.05 |
| Target layers | q_proj, k_proj, v_proj, o_proj |
| Trainable parameters | 4,587,520 (about 0.76% of model + adapter parameters) |
| Epochs completed | 3 |
| Selected version | Epoch 2, checkpoint-32 |
| Selection rule | Lowest validation answer-token loss: 0.256318 |
| Learning rate | 0.0002 |
| Microbatch / accumulation | 1 / 8 |
| Maximum sequence length | 512; overlength examples rejected rather than truncated |
| Loss | Assistant answer and end-of-turn token only; prompt/ticket labels masked |
| Compute | Google Colab Tesla T4; no owned GPU |
| Precision | Float32 parameters with FP16 mixed-precision training |
Validation guided checkpoint selection. Test tickets were not used for gradient updates or selecting the checkpoint. This small pilot does not demonstrate generalization to real customer traffic.
Load and run
Use a Python environment with PyTorch installed for your hardware, then install the library versions used in the experiment:
pip install transformers==4.57.1 peft==0.17.1 accelerate==1.10.1
The original Colab run used PyTorch 2.14.1+cu130. Compatibility and speed on other machines should be checked. CPU inference is possible in the code below but is not benchmarked here.
import torchfrom huggingface_hub import hf_hub_downloadfrom transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfigfrom peft import PeftModel ADAPTER = "ShwethaJ/qwen3-0.6b-support-ticket-triage-lora"BASE = "Qwen/Qwen3-0.6B"BASE_VERSION = "c1899de289a04d12100db370d81485cdf75e47ca" # No authentication is normally needed for public, ungated downloads.prompt_path = hf_hub_download(ADAPTER, "system_prompt.txt")with open(prompt_path, encoding="utf-8") as handle: system_prompt = handle.read().strip() device = "cuda" if torch.cuda.is_available() else "cpu"dtype = (torch.bfloat16 if device == "cuda" and torch.cuda.is_bf16_supported() else torch.float16 if device == "cuda" else torch.float32) tokenizer = AutoTokenizer.from_pretrained(ADAPTER)base = AutoModelForCausalLM.from_pretrained( BASE, revision=BASE_VERSION, torch_dtype=dtype, attn_implementation="eager").to(device)model = PeftModel.from_pretrained(base, ADAPTER, is_trainable=False).eval() messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": "Our production dashboard rejects sign-in with a 401 error."},]inputs = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt",).to(device)config = GenerationConfig( do_sample=False, num_beams=1, max_new_tokens=256, eos_token_id=model.generation_config.eos_token_id, pad_token_id=tokenizer.pad_token_id, use_cache=True,)with torch.inference_mode(): output = model.generate(**inputs, generation_config=config)answer_ids = output[0, inputs["input_ids"].shape[-1]:]print(tokenizer.decode(answer_ids, skip_special_tokens=True))
Use the supplied system prompt and non-thinking chat template for comparability. Validate generated JSON in your application. Greedy decoding is the chosen experimental baseline setting, not Qwen's default recommended sampling recipe. The example preserves raw output rather than silently repairing it.
For reproducible deployment, also pin this adapter repository to a full commit ID after publishing. The base revision is already fixed above. Training identity records configuration and data fingerprints.
Limitations and intended use
- Intended as a portfolio/research pilot for short English support-ticket extraction and routing assistance.
- The balanced AI-authored evaluation set is small and easier/less diverse than production support traffic. Each category has only 11 test examples.
- No measured reduction in handling time, staffing cost, or customer wait time. No production ROI claim.
- The model may misclassify, invent facts, miss impact details, or produce invalid JSON. Summary quality needs human factual and semantic review; human review is not represented as complete by the automated metrics.
- No benchmark here for multilingual tickets, long threads/logs, adversarial robustness, privacy, or production reliability.
- It is not a general support agent, autonomous helpdesk, or source of verified product policy.
Attribution and license
Based on Qwen3-0.6B, released by the Qwen team under Apache 2.0. Adapter published under Apache 2.0. Respect the base model license and preserve its attribution when redistributing.