Smriti10raj

support-slm

README

License: apache-2.0

How to use it

The system prompt must be the one it was trained with, byte for byte: system_prompt.txt. Decode greedily.

python

import torchfrom huggingface_hub import hf_hub_downloadfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = "Qwen/Qwen2.5-0.5B-Instruct"rev = "7ae557604adf67be50417f59c2c2f167def9a775"tok = AutoTokenizer.from_pretrained(base, revision=rev)model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(base, revision=rev, torch_dtype=torch.float32),                                  "Smriti10raj/support-slm", revision="v1").eval()system = open(hf_hub_download("Smriti10raj/support-slm", "system_prompt.txt", revision="v1")).read().strip()msgs = [{"role": "system", "content": system},        {"role": "user", "content": "my dell latitude 5420 blue screens after sleep, already updated drivers"}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")out = model.generate(ids, max_new_tokens=160, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

A faster llama.cpp build (about 4× faster on CPU, same or better scores) is in Smriti10raj/support-slm-GGUF.

Output vocabulary

intent: technical_support | greeting | out_of_scope. issue, action, symptoms and tried come from fixed lists written in the system prompt (14 issues such as BSOD, No boot, Not charging, Wi-Fi; 21 actions such as suggest_solution, alternative_solution, escalate, draft_support_email). Keys the message doesn't support are left out: the model is trained to omit, not guess.

Results

Three test sets, none used in training. Training rows that were exact, normalised, fuzzy (≥ 0.90) or semantic (cosine ≥ 0.95) near-duplicates of any test row were removed before training.

Held-out (420 messages): generated like the training data but with phrasings and device models that never appear in training.
Hand-written (45 messages): long stories, experts, voice-typed text, Hinglish, multi-part requests and hard negatives.
Refusal test (60 messages): 40 off-topic messages, then 20 real support messages that must not be refused.

Held-out test, the model alone

Table with columns: support-slm v1 (PyTorch adapter), Q8_0 GGUF + LoRA (llama.cpp), rules only, untuned Qwen2.5-0.5B
	support-slm v1 (PyTorch adapter)	Q8_0 GGUF + LoRA (llama.cpp)	rules only	untuned Qwen2.5-0.5B
Valid JSON	1.0	1.0	1.0	0.964
Whole record right (every field)	0.481	0.555	0.162	0.0
Intent	1.0	1.0	0.836	0.89
Issue	0.717	0.756	0.349	0.056
Requested action	0.906	0.923	0.352	0.44
Vendor	0.955	0.963	0.96	0.414
Product model	0.976	0.976	0.876	0.122
Error codes (F1)	0.932	0.965	0.938	0.0
Symptoms (F1)	0.454	0.532	0.021	0.009
Already-tried steps (F1)	0.695	0.682	0.0	0.0
Invented a vendor/model/code (rate)	0.003	0.0	0.007	0.56
Seconds per message (CPU)	6.649	1.689	0.003	15.324

Hand-written test and refusal test

Table with columns: support-slm v1 (llama.cpp), rules only
	support-slm v1 (llama.cpp)	rules only
Hand-written: whole record right	0.4	0.267
Hand-written: off-topic or greeting handled correctly	4/5	4/5
Refusal test: off-topic refused	24/40	26/40
Refusal test: real support wrongly refused	0/20	3/20

Inside TicketAI

The application combines the model with small rules (exact device names and explicit requests like "escalate" come from the rules; labels outside the vocabulary are mapped or dropped). Measured through that real code: whole record right 0.536 held-out and 0.444 hand-written. On the refusal test it refuses 21/40 off-topic messages and wrongly refuses 0/20 real ones.

v2: better at turning away off-topic questions (used as a judge next to v1)

v2 is the same recipe (Qwen2.5-0.5B-Instruct, LoRA r16, alpha 32, 2 epochs, laptop CPU, 19.4 hours) trained on support-v2-synth: 3211 training rows that add general-tech questions, buying advice and non-computer requests labelled out_of_scope, plus counterweight rows so real support is not refused. Final train loss 0.0255, eval loss 0.0073. It is in the v2/ folder of this repo (tag v2); the GGUF LoRA is support-slm-v2-lora-F16.gguf in Smriti10raj/support-slm-GGUF.

python

model = PeftModel.from_pretrained(base_model, "Smriti10raj/support-slm", subfolder="v2", revision="v2")

The system prompt is identical to v1's. v2 was evaluated only as the llama.cpp build (Q8_0 base + F16 LoRA).

The model alone (llama.cpp)

Table with columns: v1, v2
	v1	v2
Held-out (420): whole record right	0.555	0.502
Hand-written (45): whole record right	0.4	0.533
Refusal test: off-topic refused	24/40	40/40
Refusal test: real support wrongly refused	0/20	1/20
Held-out: priority right	0.94	0.855

Inside TicketAI

Table with columns: v1 + rules, v2 + rules, v1, with v2 as off-topic judge
	v1 + rules	v2 + rules	v1, with v2 as off-topic judge
Held-out: whole record right	53.6%	48.1%	53.3%
Hand-written: whole record right	44.4%	51.1%	46.7%
Off-topic turned away	21/40	37/40	33/40
Real support wrongly refused	0/20	1/20	0/20

Why v2 did not replace v1. A rule fixed before v2 was trained said v2 replaces v1 only if it is no worse on the old tests. It is worse on the held-out set, mostly because it rates urgent messages medium instead of high priority. TicketAI therefore keeps v1 for every field and runs v2 in parallel as a judge: v2's answer is used only when v2 says out_of_scope, v1 does not, and the message names no vendor, product line, known issue or error code. That last guard was added after a live check in which v2 alone turned away a voice-typed Dell blue-screen message; the judge figures above were measured after it was added, on the models' saved outputs through TicketAI's code.

Limits (read before using)

It refuses too few off-topic questions. General tech questions ("SSD vs HDD, which is better?", buying advice that names Dell or HP) are often read as support requests. v2 (below) turns away far more of them and is used in TicketAI as a judge next to v1.
Symptoms are the weakest field (F1 0.454–0.532 held-out); already-tried steps reach F1 0.682. Treat both as hints.
It only knows the vocabulary above: Dell, HP, Lenovo, Microsoft (Surface), Windows laptops and desktops, English and Hinglish. Other vendors, languages or device types are out of scope.
Synthetic training data. Labels are correct by construction, but real users will write things the generator never produced. The hand-written test is small (45 messages).
It sees one message at a time. Conversation state lives in the application, not in the model.

Training (v1)

Base: Qwen/Qwen2.5-0.5B-Instruct @ 7ae557604adf. LoRA r16, alpha 32, loss on the answer only, 2 epochs, on a laptop CPU (about 15 hours).
Data: support-v1-synth: 3181 training and 338 validation rows, generated from slot templates with split phrasing banks (validation uses whole phrasing groups never seen in training). Train sha256 123ddbc0374758f6….
Final train loss 0.0239, eval loss 0.0731.
adapter_model.safetensors sha256 cd291a325e6a5e5f….

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Smriti10raj

Model Tree

Base

Qwen/Qwen2.5-0.5B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

How to use it

The system prompt must be the one it was trained with, byte for byte: system_prompt.txt. Decode greedily.

python

import torchfrom huggingface_hub import hf_hub_downloadfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = "Qwen/Qwen2.5-0.5B-Instruct"rev = "7ae557604adf67be50417f59c2c2f167def9a775"tok = AutoTokenizer.from_pretrained(base, revision=rev)model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(base, revision=rev, torch_dtype=torch.float32),                                  "Smriti10raj/support-slm", revision="v1").eval()system = open(hf_hub_download("Smriti10raj/support-slm", "system_prompt.txt", revision="v1")).read().strip()msgs = [{"role": "system", "content": system},        {"role": "user", "content": "my dell latitude 5420 blue screens after sleep, already updated drivers"}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")out = model.generate(ids, max_new_tokens=160, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

import torchfrom huggingface_hub import hf_hub_downloadfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = "Qwen/Qwen2.5-0.5B-Instruct"rev = "7ae557604adf67be50417f59c2c2f167def9a775"tok = AutoTokenizer.from_pretrained(base, revision=rev)model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(base, revision=rev, torch_dtype=torch.float32),                                  "Smriti10raj/support-slm", revision="v1").eval()system = open(hf_hub_download("Smriti10raj/support-slm", "system_prompt.txt", revision="v1")).read().strip()msgs = [{"role": "system", "content": system},        {"role": "user", "content": "my dell latitude 5420 blue screens after sleep, already updated drivers"}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")out = model.generate(ids, max_new_tokens=160, do_sample=False)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

A faster llama.cpp build (about 4× faster on CPU, same or better scores) is in Smriti10raj/support-slm-GGUF.

Output vocabulary

Results

Three test sets, none used in training. Training rows that were exact, normalised, fuzzy (≥ 0.90) or semantic (cosine ≥ 0.95) near-duplicates of any test row were removed before training.

Held-out (420 messages): generated like the training data but with phrasings and device models that never appear in training.
Hand-written (45 messages): long stories, experts, voice-typed text, Hinglish, multi-part requests and hard negatives.
Refusal test (60 messages): 40 off-topic messages, then 20 real support messages that must not be refused.

Held-out test, the model alone

Table with columns: support-slm v1 (PyTorch adapter), Q8_0 GGUF + LoRA (llama.cpp), rules only, untuned Qwen2.5-0.5B
	support-slm v1 (PyTorch adapter)	Q8_0 GGUF + LoRA (llama.cpp)	rules only	untuned Qwen2.5-0.5B
Valid JSON	1.0	1.0	1.0	0.964
Whole record right (every field)	0.481	0.555	0.162	0.0
Intent	1.0	1.0	0.836	0.89
Issue	0.717	0.756	0.349	0.056
Requested action	0.906	0.923	0.352	0.44
Vendor	0.955	0.963	0.96	0.414
Product model	0.976	0.976	0.876	0.122
Error codes (F1)	0.932	0.965	0.938	0.0
Symptoms (F1)	0.454	0.532	0.021	0.009
Already-tried steps (F1)	0.695	0.682	0.0	0.0
Invented a vendor/model/code (rate)	0.003	0.0	0.007	0.56
Seconds per message (CPU)	6.649	1.689	0.003	15.324

Hand-written test and refusal test

Table with columns: support-slm v1 (llama.cpp), rules only
	support-slm v1 (llama.cpp)	rules only
Hand-written: whole record right	0.4	0.267
Hand-written: off-topic or greeting handled correctly	4/5	4/5
Refusal test: off-topic refused	24/40	26/40
Refusal test: real support wrongly refused	0/20	3/20

Inside TicketAI

v2: better at turning away off-topic questions (used as a judge next to v1)

python

model = PeftModel.from_pretrained(base_model, "Smriti10raj/support-slm", subfolder="v2", revision="v2")

model = PeftModel.from_pretrained(base_model, "Smriti10raj/support-slm", subfolder="v2", revision="v2")

The system prompt is identical to v1's. v2 was evaluated only as the llama.cpp build (Q8_0 base + F16 LoRA).

The model alone (llama.cpp)

Table with columns: v1, v2
	v1	v2
Held-out (420): whole record right	0.555	0.502
Hand-written (45): whole record right	0.4	0.533
Refusal test: off-topic refused	24/40	40/40
Refusal test: real support wrongly refused	0/20	1/20
Held-out: priority right	0.94	0.855

Inside TicketAI

Table with columns: v1 + rules, v2 + rules, v1, with v2 as off-topic judge
	v1 + rules	v2 + rules	v1, with v2 as off-topic judge
Held-out: whole record right	53.6%	48.1%	53.3%
Hand-written: whole record right	44.4%	51.1%	46.7%
Off-topic turned away	21/40	37/40	33/40
Real support wrongly refused	0/20	1/20	0/20

Limits (read before using)

It refuses too few off-topic questions. General tech questions ("SSD vs HDD, which is better?", buying advice that names Dell or HP) are often read as support requests. v2 (below) turns away far more of them and is used in TicketAI as a judge next to v1.
Symptoms are the weakest field (F1 0.454–0.532 held-out); already-tried steps reach F1 0.682. Treat both as hints.
It only knows the vocabulary above: Dell, HP, Lenovo, Microsoft (Surface), Windows laptops and desktops, English and Hinglish. Other vendors, languages or device types are out of scope.
Synthetic training data. Labels are correct by construction, but real users will write things the generator never produced. The hand-written test is small (45 messages).
It sees one message at a time. Conversation state lives in the application, not in the model.

Training (v1)

Base: Qwen/Qwen2.5-0.5B-Instruct @ 7ae557604adf. LoRA r16, alpha 32, loss on the answer only, 2 epochs, on a laptop CPU (about 15 hours).
Data: support-v1-synth: 3181 training and 338 validation rows, generated from slot templates with split phrasing banks (validation uses whole phrasing groups never seen in training). Train sha256 123ddbc0374758f6….
Final train loss 0.0239, eval loss 0.0731.
adapter_model.safetensors sha256 cd291a325e6a5e5f….