oncall-ai-incident-classifier API & Inference Endpoint

Intended use

First-pass triage of a pasted log excerpt (nginx, Kubernetes, PostgreSQL, systemd/journald, Python, Java style lines) into a fixed incident taxonomy plus a P1–P4 severity, on CPU, without calling a cloud LLM.
Designed to be run behind a verification step: in OnCall AI the predicted type is used to retrieve runbook chunks, and the result is marked needs_human when retrieval finds nothing relevant.

Out of scope: open-ended log understanding, root-cause explanation, logs outside the 12 classes, non-English logs, anything safety-critical without human review.

Incident types (12)

Table with columns: id, label
id	label
`database_connection_exhausted`	Database connection pool exhausted
`out_of_memory`	Out of memory (OOM)
`disk_space_full`	Disk space exhausted
`upstream_gateway_error`	Upstream / gateway failure (502/504)
`tls_certificate_expired`	TLS certificate expired / invalid
`kubernetes_crashloop`	Kubernetes CrashLoopBackOff
`application_exception`	Unhandled application exception
`dns_resolution_failure`	DNS resolution failure
`rate_limit_exceeded`	Rate limit / quota exceeded
`authentication_failure`	Authentication failure / credential issue
`high_latency_timeout`	High latency / request timeouts
`deployment_failure`	Deployment / rollout failure

Severity levels

Table with columns: level, meaning
level	meaning
P1	Critical - production outage or data at risk; page on-call immediately
P2	Major - production degraded for many users; respond within 30 minutes
P3	Minor - limited impact or non-production; handle in working hours
P4	Low - informational, single occurrence, no user impact

Prompt format

The model was trained with the Qwen chat template and exactly this system prompt (app/prompts.py in the repo; the app and the notebook share the file). Use it unchanged.

System:

text

You are an SRE assistant. Classify the production log excerpt into exactly one incident_type from this list: database_connection_exhausted, out_of_memory, disk_space_full, upstream_gateway_error, tls_certificate_expired, kubernetes_crashloop, application_exception, dns_resolution_failure, rate_limit_exceeded, authentication_failure, high_latency_timeout, deployment_failure. Then assign a severity: P1 (production outage, all/most users, core flow down, data at risk), P2 (production degraded for many users, partial impact), P3 (limited impact, single host, non-production, self-recovered), P4 (informational, single occurrence in dev, within baseline). Answer with JSON only: {"incident_type": "...", "severity": "P1|P2|P3|P4"}

User:

text

TASK: classifyLOG:<the log excerpt>

Assistant (target):

json

{"incident_type": "database_connection_exhausted", "severity": "P1"}

How to use

python

import json, math, torchfrom transformers import AutoModelForCausalLM, AutoTokenizer MODEL_ID = "rachitst/oncall-ai-incident-classifier"tok = AutoTokenizer.from_pretrained(MODEL_ID)device = "cuda" if torch.cuda.is_available() else "cpu"model = AutoModelForCausalLM.from_pretrained(    MODEL_ID, torch_dtype=torch.float16 if device == "cuda" else torch.float32).to(device).eval() SYSTEM = ('You are an SRE assistant. Classify the production log excerpt into exactly one incident_type from this list: database_connection_exhausted, out_of_memory, disk_space_full, upstream_gateway_error, tls_certificate_expired, kubernetes_crashloop, application_exception, dns_resolution_failure, rate_limit_exceeded, authentication_failure, high_latency_timeout, deployment_failure. Then assign a severity: P1 (production outage, all/most users, core flow down, data at risk), P2 (production degraded for many users, partial impact), P3 (limited impact, single host, non-production, self-recovered), P4 (informational, single occurrence in dev, within baseline). Answer with JSON only: {"incident_type": "...", "severity": "P1|P2|P3|P4"}') # a log from the held-out test split (label: database_connection_exhausted, P1)log = """2026-09-05T20:55:02.000Z FATAL postgres[327]: FATAL:  sorry, too many clients already2026-09-05T20:55:04.000Z DEBUG notification-svc cache hit ratio 93%2026-09-05T20:55:05.000Z FATAL postgres[327]: FATAL:  remaining connection slots are reserved for non-replication superuser connections2026-09-05T20:55:06.000Z ERROR notification-svc psycopg2.OperationalError: FATAL:  too many connections for role "app"2026-09-05T20:55:14.000Z CRIT  monitor [prod] primary database unavailable; all write traffic failing on 3 nodes""" messages = [{"role": "system", "content": SYSTEM},            {"role": "user", "content": f"TASK: classify\nLOG:\n{log}"}]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tok(prompt, return_tensors="pt", truncation=True, max_length=2048).to(device) with torch.no_grad():    out = model.generate(**inputs, max_new_tokens=48, do_sample=False,                         output_scores=True, return_dict_in_generate=True,                         pad_token_id=tok.pad_token_id or tok.eos_token_id) new = out.sequences[0][inputs["input_ids"].shape[1]:]text = tok.decode(new, skip_special_tokens=True)label = json.loads(text)                       # {"incident_type": "...", "severity": "P1"} # confidence = geometric mean of the generated tokens' probabilities (what the app reports)logps = []for step, t in zip(out.scores, new):    logps.append(torch.log_softmax(step[0].float(), dim=-1)[t].item())    if t.item() == tok.eos_token_id:        breakconfidence = math.exp(sum(logps) / len(logps))print(label, round(confidence, 3))# -> {'incident_type': 'database_connection_exhausted', 'severity': 'P1'} 1.0   (measured on CPU, fp32)

Greedy decoding, max_new_tokens=48. The reply is a single JSON object; the app parses it with a tolerant regex and treats a parse failure as low confidence. Parse failures on the test set: 0.

Training data

Generated by data/generate_dataset.py (seed 42) in the repo. Each sample is a short multi-line log excerpt built from hand-written templates that imitate real nginx, Kubernetes, PostgreSQL, systemd/journald, Python and Java log lines, with randomised hosts, IPs, timestamps, services and numbers plus interleaved unrelated INFO lines as noise. Severity is derived from the blast radius stated in the log text (environment, error rate, number of affected hosts/pods, user impact), so the model has to read the context rather than spot a keyword.

Table

Samples	1200 (840 train / 180 val / 180 test, stratified)
Incident types	12, balanced (100 each)
Severity distribution	P1 168, P2 420, P3 396, P4 216
Environment distribution	prod 672, staging 312, dev 216
Average length	4.21 lines, 455 characters

Only the assistant turn (the JSON label) contributes to the loss; prompt tokens are masked with -100. Sequences are truncated at 512 tokens.

Training procedure

Notebook: training/OnCall_AI_Training.ipynb, run on Google Colab (Tesla T4, 2026-10-28). Two adapters were trained with identical hyper-parameters; the QLoRA one is what this repo ships.

Table with columns: hyper-parameter, value
hyper-parameter	value
Base	`Qwen/Qwen2.5-0.5B-Instruct`
Quantisation (QLoRA run)	4-bit NF4, double quantisation, fp16 compute (bitsandbytes)
LoRA	r=16, α=32, dropout 0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Trainable parameters	8,798,208 of 323,917,696 (2.716 %)
Epochs	3
Learning rate	0.0002, cosine schedule, 16 warm-up steps
Batch	4 per device × 2 gradient accumulation (effective 8)
Optimizer	paged AdamW 8-bit (QLoRA) / AdamW (LoRA fp16)
Precision	fp16 mixed
Max sequence length	512
Seed	42
Software	transformers 5.x `Trainer`, peft, bitsandbytes

Table with columns: run, train time, peak GPU memory, final eval loss
run	train time	peak GPU memory	final eval loss
QLoRA 4-bit (this model)	626 s	6.58 GB	0.0001
LoRA fp16 (comparison)	512 s	6.86 GB	0.0001

Loss curves

Merge: the QLoRA adapter was merged into an fp16 copy of the base weights with PeftModel.merge_and_unload() and saved with safetensors (988 MB). A sanity check of the merged model on the first 60 test logs reproduced the adapter: type accuracy 1.000, severity accuracy 0.983.

Evaluation

Held-out test split: 180 logs, 12 types, 4 severities. Metrics: accuracy and macro-F1 for each of the two labels, number of unparseable replies, and mean inference time per log on a Tesla T4.

Table with columns: approach, type acc, type F1, sev acc, sev F1, parse fail, ms/log (T4), train s, peak GB
approach	type acc	type F1	sev acc	sev F1	ms/log (T4)	train s	peak GB
Rules baseline (keyword scoring)	0.872	0.875	0.683	0.741	0	-	-
Qwen2.5-0.5B-Instruct zero-shot	0.261	0.208	0.367	0.176	942	-	-
Groq qwen/qwen3.8-27b zero-shot	0.944	0.943	0.800	0.812	4168	-	-
LoRA fp16	1.000	1.000	1.000	1.000	1209	512	6.86
QLoRA 4-bit (this model)	1.000	1.000	1.000	1.000	1602	626	6.58

Notes:

The Groq row's latency includes a 4.0 s throttle sleep used to stay inside the free-tier rate limit; the API round trip itself is about 170 ms.
The QLoRA latency is for the 4-bit adapter model on the T4; the merged fp16 checkpoint in this repo ran at 611 ms/log on the same GPU in the sanity check.

Type accuracy and F1 Severity accuracy and F1 Confusion matrix

Inference latency on CPU (merged model, fp32, greedy, in the deployed app)

Table with columns: machine, classifier latency per log
machine	classifier latency per log
Modal container, 2 vCPU / 4 GB	3.6 – 5.0 s (7 hand-written logs)
Laptop, Celeron N4500, 2 cores, no AVX	12.9 – 17.0 s
Tesla T4 (fp16)	0.6 s

Cold start on Modal (download from this repo + load) was about 37 s; loading from a warm cache takes ~5 s.

Limitations

Synthetic data. Train, validation and test logs all come from the same template generator. 1.000 on the test split means the generator's patterns were learned, not that real production logs will be classified this well.
Observed distribution-shift miss. On a hand-written log (expired TLS certificate in prod, "100 % of requests failing") the model answers severity P2 with confidence 0.999 where P1 is expected, because the generator never phrases a P1 that way. High confidence is therefore not a reliability signal outside the training distribution.
Closed taxonomy. Every input gets one of the 12 classes; unrelated or noisy text is still assigned a class (a nonsense log was classified deployment_failure at 0.977). Use retrieval or a rules check to catch this, as the OnCall AI workflow does.
English, short excerpts only. Inputs are truncated at 2048 tokens; the app further caps logs at 20,000 characters.
Not a safety or security tool. No PII redaction is built in (the app redacts before calling the model).

Files

Table with columns: file, content
file	content
`model.safetensors`	merged fp16 weights, 988 MB
`config.json`, `generation_config.json`	Qwen2 architecture config
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`	unchanged Qwen2.5 tokenizer and chat template
`figures/`	evaluation charts rendered from `results/comparison.json` in the repo

License

The base model, Qwen2.5-0.5B-Instruct, is released under Apache-2.0; this fine-tune is released under the same license.

Citation

bibtex

@misc{chheda2026oncallai,  title  = {OnCall AI: Intelligent Production Incident Analyzer -- QLoRA incident classifier},  author = {Chheda, Rachit Manish},  year   = {2026},  url    = {https://github.com/rachitst/oncall-ai}}

label

database_connection_exhausted

Database connection pool exhausted

out_of_memory

Out of memory (OOM)

disk_space_full

Disk space exhausted

upstream_gateway_error

Upstream / gateway failure (502/504)

tls_certificate_expired

TLS certificate expired / invalid

kubernetes_crashloop

Kubernetes CrashLoopBackOff

application_exception

Unhandled application exception

dns_resolution_failure

DNS resolution failure

rate_limit_exceeded

Rate limit / quota exceeded

authentication_failure

Authentication failure / credential issue

high_latency_timeout

High latency / request timeouts

deployment_failure

Deployment / rollout failure

level

meaning

Critical - production outage or data at risk; page on-call immediately

Major - production degraded for many users; respond within 30 minutes

Minor - limited impact or non-production; handle in working hours

Low - informational, single occurrence, no user impact

You are an SRE assistant. Classify the production log excerpt into exactly one incident_type from this list: database_connection_exhausted, out_of_memory, disk_space_full, upstream_gateway_error, tls_certificate_expired, kubernetes_crashloop, application_exception, dns_resolution_failure, rate_limit_exceeded, authentication_failure, high_latency_timeout, deployment_failure. Then assign a severity: P1 (production outage, all/most users, core flow down, data at risk), P2 (production degraded for many users, partial impact), P3 (limited impact, single host, non-production, self-recovered), P4 (informational, single occurrence in dev, within baseline). Answer with JSON only: {"incident_type": "...", "severity": "P1|P2|P3|P4"}

import json, math, torchfrom transformers import AutoModelForCausalLM, AutoTokenizer MODEL_ID = "rachitst/oncall-ai-incident-classifier"tok = AutoTokenizer.from_pretrained(MODEL_ID)device = "cuda" if torch.cuda.is_available() else "cpu"model = AutoModelForCausalLM.from_pretrained( MODEL_ID, torch_dtype=torch.float16 if device == "cuda" else torch.float32).to(device).eval() SYSTEM = ('You are an SRE assistant. Classify the production log excerpt into exactly one incident_type from this list: database_connection_exhausted, out_of_memory, disk_space_full, upstream_gateway_error, tls_certificate_expired, kubernetes_crashloop, application_exception, dns_resolution_failure, rate_limit_exceeded, authentication_failure, high_latency_timeout, deployment_failure. Then assign a severity: P1 (production outage, all/most users, core flow down, data at risk), P2 (production degraded for many users, partial impact), P3 (limited impact, single host, non-production, self-recovered), P4 (informational, single occurrence in dev, within baseline). Answer with JSON only: {"incident_type": "...", "severity": "P1|P2|P3|P4"}') # a log from the held-out test split (label: database_connection_exhausted, P1)log = """2026-09-05T20:55:02.000Z FATAL postgres[327]: FATAL: sorry, too many clients already2026-09-05T20:55:04.000Z DEBUG notification-svc cache hit ratio 93%2026-09-05T20:55:05.000Z FATAL postgres[327]: FATAL: remaining connection slots are reserved for non-replication superuser connections2026-09-05T20:55:06.000Z ERROR notification-svc psycopg2.OperationalError: FATAL: too many connections for role "app"2026-09-05T20:55:14.000Z CRIT monitor [prod] primary database unavailable; all write traffic failing on 3 nodes""" messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": f"TASK: classify\nLOG:\n{log}"}]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tok(prompt, return_tensors="pt", truncation=True, max_length=2048).to(device) with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=48, do_sample=False, output_scores=True, return_dict_in_generate=True, pad_token_id=tok.pad_token_id or tok.eos_token_id) new = out.sequences[0][inputs["input_ids"].shape[1]:]text = tok.decode(new, skip_special_tokens=True)label = json.loads(text) # {"incident_type": "...", "severity": "P1"} # confidence = geometric mean of the generated tokens' probabilities (what the app reports)logps = []for step, t in zip(out.scores, new): logps.append(torch.log_softmax(step[0].float(), dim=-1)[t].item()) if t.item() == tok.eos_token_id: breakconfidence = math.exp(sum(logps) / len(logps))print(label, round(confidence, 3))# -> {'incident_type': 'database_connection_exhausted', 'severity': 'P1'} 1.0 (measured on CPU, fp32)

Samples

1200 (840 train / 180 val / 180 test, stratified)

Incident types

12, balanced (100 each)

Severity distribution

P1 168, P2 420, P3 396, P4 216

Environment distribution

prod 672, staging 312, dev 216

Average length

4.21 lines, 455 characters

hyper-parameter

value

Base

Qwen/Qwen2.5-0.5B-Instruct

Quantisation (QLoRA run)

4-bit NF4, double quantisation, fp16 compute (bitsandbytes)

LoRA

r=16, α=32, dropout 0.05

Target modules

q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Trainable parameters

8,798,208 of 323,917,696 (2.716 %)

Epochs

Learning rate

0.0002, cosine schedule, 16 warm-up steps

Batch

4 per device × 2 gradient accumulation (effective 8)

Optimizer

paged AdamW 8-bit (QLoRA) / AdamW (LoRA fp16)

Precision

fp16 mixed

Max sequence length

512

Seed

Software

transformers 5.x Trainer, peft, bitsandbytes

run

train time

peak GPU memory

final eval loss

QLoRA 4-bit (this model)

626 s

6.58 GB

0.0001

LoRA fp16 (comparison)

512 s

6.86 GB

0.0001

approach

type acc

type F1

sev acc

sev F1

parse fail

ms/log (T4)

train s

peak GB

Rules baseline (keyword scoring)

0.872

0.875

0.683

0.741

Qwen2.5-0.5B-Instruct zero-shot

0.261

0.208

0.367

0.176

942

Groq qwen/qwen3.8-27b zero-shot

0.944

0.943

0.800

0.812

4168

LoRA fp16

1.000

1209

512

6.86

QLoRA 4-bit (this model)

1.000

1602

626

6.58

machine

classifier latency per log

Modal container, 2 vCPU / 4 GB

3.6 – 5.0 s (7 hand-written logs)

Laptop, Celeron N4500, 2 cores, no AVX

12.9 – 17.0 s

Tesla T4 (fp16)

0.6 s

file

content

model.safetensors

merged fp16 weights, 988 MB

config.json, generation_config.json

Qwen2 architecture config

tokenizer.json, tokenizer_config.json, chat_template.jinja

unchanged Qwen2.5 tokenizer and chat template

figures/

evaluation charts rendered from results/comparison.json in the repo

@misc{chheda2026oncallai, title = {OnCall AI: Intelligent Production Incident Analyzer -- QLoRA incident classifier}, author = {Chheda, Rachit Manish}, year = {2026}, url = {https://github.com/rachitst/oncall-ai}}

oncall-ai-incident-classifier

README

Intended use

Incident types (12)

Severity levels

Prompt format

How to use

Training data

Training procedure

Evaluation

Inference latency on CPU (merged model, fp32, greedy, in the deployed app)

Limitations

Files

License

Citation

Explore FriendliAI today

README

Intended use

Incident types (12)

Severity levels

Prompt format

How to use

Training data

Training procedure

Evaluation

Inference latency on CPU (merged model, fp32, greedy, in the deployed app)

Limitations

Files

License

Citation