Shankarblr

Qwen2.5-1.5B-QA360

Deploy Dedicated

README

License: apache-2.0

Model Details

Model Description

QA360-Qwen2.5-1.5B is a domain-specialized instruction model for requirements-to-test-design. Given:

a fixed system instruction that defines the QA360 schema, and
a short software requirement (often prefixed with a module tag such as [Authentication]),

the model is trained to emit only a JSON object with these exact keys:

Table with columns: Key, Type, Meaning
Key	Type	Meaning
`risk_level`	string	`High` / `Medium` / `Low`
`automation_candidate`	boolean	Whether the requirement is a good automation target
`affected_modules`	string[]	Product / platform areas touched by the change
`functional_tests`	string[]	Happy-path and core functional cases
`negative_tests`	string[]	Invalid input, abuse, and failure cases
`security_tests`	string[]	AuthN/Z, injection, session, secrets, rate limit, etc.
`accessibility_tests`	string[]	Keyboard, labels, contrast, SR, focus, ARIA
`api_tests`	string[]	Endpoint, status code, contract, and SLA checks
`regression_scope`	string[]	Neighboring flows that should be retested

It is not a general chatbot and is not a substitute for a human QA lead, threat model, or accessibility audit. Outputs are draft test ideas for analysts to edit.

Developed by: Shankar Subramanyam
Model type: Causal decoder-only LLM (Qwen2 architecture), instruction-tuned then task-SFT'd
Language(s): English (requirement text and JSON string values)
License: Apache-2.0 (inherits from Qwen/Qwen2.5-1.5B-Instruct)
Finetuned from: Qwen/Qwen2.5-1.5B-Instruct

Base model facts (unchanged by this SFT):

Table

Parameters	1.54B (1.31B non-embedding)
Layers	28
Attention	GQA, 12 Q heads / 2 KV heads
Context (base)	32,768 tokens; generation up to 8,192
Architecture	RoPE, SwiGLU, RMSNorm, QKV bias, tied embeddings
Chat template	Qwen ChatML (`<\|im_start\|>` / `<\|im_end\|>`)

This fine-tune was trained at max_length=2048. Keep inference prompts + completions inside that budget for best schema fidelity. Long requirement specs should be summarized before calling the model.

Model Sources

Base model: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct
Base paper: Qwen2 Technical Report
Training code: Finetuning_of_Qwen_Qlora_with_merge.py (TRL SFTTrainer + PEFT LoRA + optional bitsandbytes 4-bit)
Adapter dir (if published): qwen_qa360_qlora_adapter
Merged dir (this card): qwen_qa360_qlora_merged_model

Uses

Direct Use

Draft a 360° test analysis from a one-line or short-paragraph requirement
Seed test-case writing in ALM / Xray / TestRail / Azure DevOps
Suggest regression blast radius for a story or change request
Produce a first-pass JSON payload for a QA agent or RAG pipeline

Intended operators: QA engineers, SDETs, business analysts, and agentic tools that already validate JSON.

Downstream Use

Tool-calling / structured-output node inside a multi-agent SDLC stack
Further SFT or DPO on a private requirements corpus
Constrained decoding (outlines, xgrammar, lm-format-enforcer) against the JSON schema
Distillation teacher for a smaller on-prem classifier + template system

Out-of-Scope Use

Do not use this model as:

An automated sign-off for security, privacy, or accessibility compliance
A source of executable test code, exploits, or attack payloads
A general assistant, code generator, or policy engine
An analyzer of non-software text (legal contracts, medical notes, etc.)
A production API without JSON parse checks, schema validation, and human review

The training targets are English software requirements. Other languages and free-form chat will degrade.

Bias, Risks, and Limitations

Task limitations

1.5B is small. It will invent plausible-but-wrong module names, status codes, and SLA numbers.
Risk labels are learned priors from the SFT corpus, not a calibrated risk model. Auth and payments examples in the data are often High; the model will over-index on that pattern.
Arrays are often 6–8 items because that is how the dataset was written. Real stories may need 2 items or 20.
Accessibility and security lists are generic templates (keyboard, ARIA, HTTPS, lockout…). They are not WCAG or OWASP audits.
The model was trained to copy a single system prompt. Changing the instruction mid-flight reduces JSON validity.

Data limitations

Private/synthetic qa360_sft.jsonl (~4.8k rows). Coverage is only as broad as the requirements someone labeled.
No published inter-annotator agreement. Style of test wording is that of the corpus authors.
Module names and API paths (POST /api/auth/login) reflect the labeling convention, not your system.

Safety

Can emit security test ideas (brute force, injection, lockout). That is intended. It should not be used to generate working exploit code.
Do not send production secrets, customer PII, or unpublished vulnerability details into a hosted endpoint.

Technical

Merged weights are fp16/bf16, not the 4-bit training quant.
First-token and JSON-close failures still happen. Always json.loads and retry or repair.

Recommendations

Validate output against the schema before storing it.
Keep the exact system prompt used in training.
Cap max_new_tokens at 1024 (3080 is unnecessary and invites rambling).
Use greedy or low-temperature decoding (do_sample=False or temperature=0.1) for JSON.
Human-review risk_level and security_tests on High-risk domains (auth, payments, PHI, admin).
Log prompt, raw completion, parse success, and reviewer edits if you want a v2 dataset.

How to Get Started

Transformers (merged model)

python
import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shankarblr/qwen2.5-1.5b-qa360"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
    device_map="auto" if device == "cuda" else None,
)

SYSTEM = (
    "Perform a QA360 analysis for the following software requirement. "
    "Return a structured analysis covering risk_level (High/Medium/Low), "
    "automation_candidate (true/false), affected_modules (array), "
    "functional_tests (array), negative_tests (array), security_tests (array), "
    "accessibility_tests (array), api_tests (array), and regression_scope (array). "
    "Return only valid JSON with these exact keys."
)

requirement = "[Authentication] User can login with email and password"

prompt = (
    f"<|im_start|>system\n{SYSTEM}<|im_end|>\n"
    f"<|im_start|>user\n{requirement}<|im_end|>\n"
    f"<|im_start|>assistant\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=1024,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id,
)
text = tokenizer.decode(out[0], skip_special_tokens=False)
response = text.split("<|im_start|>assistant\n")[-1].replace("<|im_end|>", "").strip()

print(json.dumps(json.loads(response), indent=2))

Chat-template variant (preferred if the tokenizer still ships Qwen's template):

python
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": requirement},
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

Load adapter instead of merged weights

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "shankarblr/qwen2.5-1.5b-qa360-adapter"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)

Expected shape

json
{
  "risk_level": "High",
  "automation_candidate": true,
  "affected_modules": ["Authentication", "Session Management"],
  "functional_tests": ["Verify user can login successfully with valid email and password"],
  "negative_tests": ["Verify error message displayed for incorrect password"],
  "security_tests": ["Verify HTTPS is enforced for login requests"],
  "accessibility_tests": ["Verify login form is navigable using keyboard only"],
  "api_tests": ["Verify POST /api/auth/login returns 200 with a token on valid credentials"],
  "regression_scope": ["Password reset and recovery flow"]
}

Training Details

Training Data

Private JSONL corpus qa360_sft.jsonl, one object per line:

json
{
  "instruction": "Perform a QA360 analysis for the following software requirement. Return a structured analysis covering risk_level (High/Medium/Low), automation_candidate (true/false), affected_modules (array), functional_tests (array), negative_tests (array), security_tests (array), accessibility_tests (array), api_tests (array), and regression_scope (array). Return only valid JSON with these exact keys.",
  "input": "[Authentication] User can login with email and password",
  "output": { "...QA360 object..." }
}

Rows were wrapped in Qwen ChatML before SFT:

text
<|im_start|>system
{instruction}<|im_end|>
<|im_start|>user
{input}<|im_end|>
<|im_start|>assistant
{json.dumps(output, indent=2)}<|im_end|>

Table with columns: Split, Rows, Notes
Split	Rows	Notes
Train	4,311	`train_test_split(test_size=0.1, seed=42)`
Eval	479	same format, used each epoch
Total	4,790

Training Procedure

Supervised fine-tuning with TRL SFTTrainer and PEFT LoRA. On CUDA the base is loaded in QLoRA (bitsandbytes 4-bit NF4, double quant, bf16 compute), then the adapter is merged back into an fp16/bf16 copy of the base.

Preprocessing

tokenizer.pad_token = tokenizer.eos_token (Qwen EOS 151645)
padding_side = "right"
dataset_text_field = "text"
Truncation at 2048 tokens

Hyperparameters

Table with columns: Setting, Value
Setting	Value
Base	`Qwen/Qwen2.5-1.5B-Instruct`
Method	QLoRA SFT → merge
LoRA rank `r`	16
LoRA `alpha`	32
LoRA dropout	0.05
LoRA targets	`q_proj`, `k_proj`, `v_proj`,

Speeds, Sizes, Times

Table

Hardware	e.g. 1× NVIDIA L4 24GB / A100 / local RTX
Wall time	TBD
Peak VRAM (QLoRA train)	TBD (1.5B 4-bit + LoRA r=16 is typically a few GB)
Merged fp16 size	~3.1 GB (same order as the base Instruct checkpoint)
Adapter size	tens of MB

Evaluation

Testing Data, Factors & Metrics

Held-out slice: 479 ChatML examples from the same qa360_sft.jsonl distribution (not a separately authored benchmark).

Suggested metrics — compute on the eval set and paste numbers; do not invent them:

Table with columns: Metric, Why it matters
Metric	Why it matters
JSON parse rate	Share of completions that `json.loads`
Exact-key schema rate	All 9 keys present, no extras
`risk_level` accuracy	Label match vs gold
`automation_candidate` accuracy	Boolean match vs gold
Token-overlap / embedding similarity on list fields	Wording will not match gold exactly
Trainer `eval_loss`	Training health only; not task quality

Qualitative checks used in the training script (not a benchmark):

Admin can force password reset
User I can upload a profile picture
user can lock account after failed attempts

Results

markdown
eval_loss:          TBD   # from trainer.evaluate()
json_parse_rate:    TBD
schema_valid_rate:  TBD
risk_level_acc:     TBD

Summary

This is a small specialized SFT, not a frontier model. Expect usable drafts on requirements that look like the corpus (auth, profile, account lockout, CRUD-style stories) and more hallucination on novel domains (embedded, data platform, ML ops).

Environmental Impact

Unknown until the GPU run is logged. Estimate with the MLCO2 calculator using hardware, hours, and region, then add:

Hardware Type:
Hours used:
Cloud Provider / Region:
Carbon Emitted:

QLoRA on 1.5B for ~1k steps is a small training job relative to pretraining.

Technical Specifications

Model Architecture and Objective

Architecture: Qwen2ForCausalLM
Objective: causal LM SFT on ChatML strings (next-token prediction over the full formatted example)
Post-train artifact: LoRA adapter merged into base weights with PeftModel.merge_and_unload()

Compute Infrastructure

OS / Python: Linux, Python 3.12
Key libraries: transformers, peft, trl, bitsandbytes, torch, datasets

Glossary

QA360: Internal name for a nine-field, full-stack test-analysis schema (risk + automation + six test views + regression).
QLoRA: 4-bit quantized base + low-rank adapters during training.
Merged model: Adapter baked into the base; loads as a normal transformers checkpoint, no PEFT required at inference.

Model Card Authors

Shankar Subramanyam

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Shankarblr

Model Tree

Base

Qwen/Qwen2.5-1.5B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Model Description

QA360-Qwen2.5-1.5B is a domain-specialized instruction model for requirements-to-test-design. Given:

a fixed system instruction that defines the QA360 schema, and
a short software requirement (often prefixed with a module tag such as [Authentication]),

the model is trained to emit only a JSON object with these exact keys:

Table with columns: Key, Type, Meaning
Key	Type	Meaning
`risk_level`	string	`High` / `Medium` / `Low`
`automation_candidate`	boolean	Whether the requirement is a good automation target
`affected_modules`	string[]	Product / platform areas touched by the change
`functional_tests`	string[]	Happy-path and core functional cases
`negative_tests`	string[]	Invalid input, abuse, and failure cases
`security_tests`	string[]	AuthN/Z, injection, session, secrets, rate limit, etc.
`accessibility_tests`	string[]	Keyboard, labels, contrast, SR, focus, ARIA
`api_tests`	string[]	Endpoint, status code, contract, and SLA checks
`regression_scope`	string[]	Neighboring flows that should be retested

It is not a general chatbot and is not a substitute for a human QA lead, threat model, or accessibility audit. Outputs are draft test ideas for analysts to edit.

Developed by: Shankar Subramanyam
Model type: Causal decoder-only LLM (Qwen2 architecture), instruction-tuned then task-SFT'd
Language(s): English (requirement text and JSON string values)
License: Apache-2.0 (inherits from Qwen/Qwen2.5-1.5B-Instruct)
Finetuned from: Qwen/Qwen2.5-1.5B-Instruct

Base model facts (unchanged by this SFT):

Table

Parameters	1.54B (1.31B non-embedding)
Layers	28
Attention	GQA, 12 Q heads / 2 KV heads
Context (base)	32,768 tokens; generation up to 8,192
Architecture	RoPE, SwiGLU, RMSNorm, QKV bias, tied embeddings
Chat template	Qwen ChatML (`<\|im_start\|>` / `<\|im_end\|>`)

Model Sources

Base model: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct
Base paper: Qwen2 Technical Report
Training code: Finetuning_of_Qwen_Qlora_with_merge.py (TRL SFTTrainer + PEFT LoRA + optional bitsandbytes 4-bit)
Adapter dir (if published): qwen_qa360_qlora_adapter
Merged dir (this card): qwen_qa360_qlora_merged_model

Uses

Direct Use

Draft a 360° test analysis from a one-line or short-paragraph requirement
Seed test-case writing in ALM / Xray / TestRail / Azure DevOps
Suggest regression blast radius for a story or change request
Produce a first-pass JSON payload for a QA agent or RAG pipeline

Intended operators: QA engineers, SDETs, business analysts, and agentic tools that already validate JSON.

Downstream Use

Tool-calling / structured-output node inside a multi-agent SDLC stack
Further SFT or DPO on a private requirements corpus
Constrained decoding (outlines, xgrammar, lm-format-enforcer) against the JSON schema
Distillation teacher for a smaller on-prem classifier + template system

Out-of-Scope Use

Do not use this model as:

An automated sign-off for security, privacy, or accessibility compliance
A source of executable test code, exploits, or attack payloads
A general assistant, code generator, or policy engine
An analyzer of non-software text (legal contracts, medical notes, etc.)
A production API without JSON parse checks, schema validation, and human review

The training targets are English software requirements. Other languages and free-form chat will degrade.

Bias, Risks, and Limitations

Task limitations

1.5B is small. It will invent plausible-but-wrong module names, status codes, and SLA numbers.
Risk labels are learned priors from the SFT corpus, not a calibrated risk model. Auth and payments examples in the data are often High; the model will over-index on that pattern.
Arrays are often 6–8 items because that is how the dataset was written. Real stories may need 2 items or 20.
Accessibility and security lists are generic templates (keyboard, ARIA, HTTPS, lockout…). They are not WCAG or OWASP audits.
The model was trained to copy a single system prompt. Changing the instruction mid-flight reduces JSON validity.

Data limitations

Private/synthetic qa360_sft.jsonl (~4.8k rows). Coverage is only as broad as the requirements someone labeled.
No published inter-annotator agreement. Style of test wording is that of the corpus authors.
Module names and API paths (POST /api/auth/login) reflect the labeling convention, not your system.

Safety

Can emit security test ideas (brute force, injection, lockout). That is intended. It should not be used to generate working exploit code.
Do not send production secrets, customer PII, or unpublished vulnerability details into a hosted endpoint.

Technical

Merged weights are fp16/bf16, not the 4-bit training quant.
First-token and JSON-close failures still happen. Always json.loads and retry or repair.

Recommendations

Validate output against the schema before storing it.
Keep the exact system prompt used in training.
Cap max_new_tokens at 1024 (3080 is unnecessary and invites rambling).
Use greedy or low-temperature decoding (do_sample=False or temperature=0.1) for JSON.
Human-review risk_level and security_tests on High-risk domains (auth, payments, PHI, admin).
Log prompt, raw completion, parse success, and reviewer edits if you want a v2 dataset.

How to Get Started

Transformers (merged model)

python
import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shankarblr/qwen2.5-1.5b-qa360"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
    device_map="auto" if device == "cuda" else None,
)

SYSTEM = (
    "Perform a QA360 analysis for the following software requirement. "
    "Return a structured analysis covering risk_level (High/Medium/Low), "
    "automation_candidate (true/false), affected_modules (array), "
    "functional_tests (array), negative_tests (array), security_tests (array), "
    "accessibility_tests (array), api_tests (array), and regression_scope (array). "
    "Return only valid JSON with these exact keys."
)

requirement = "[Authentication] User can login with email and password"

prompt = (
    f"<|im_start|>system\n{SYSTEM}<|im_end|>\n"
    f"<|im_start|>user\n{requirement}<|im_end|>\n"
    f"<|im_start|>assistant\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=1024,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id,
)
text = tokenizer.decode(out[0], skip_special_tokens=False)
response = text.split("<|im_start|>assistant\n")[-1].replace("<|im_end|>", "").strip()

print(json.dumps(json.loads(response), indent=2))

Chat-template variant (preferred if the tokenizer still ships Qwen's template):

python
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": requirement},
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

Load adapter instead of merged weights

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "shankarblr/qwen2.5-1.5b-qa360-adapter"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)

Expected shape

json
{
  "risk_level": "High",
  "automation_candidate": true,
  "affected_modules": ["Authentication", "Session Management"],
  "functional_tests": ["Verify user can login successfully with valid email and password"],
  "negative_tests": ["Verify error message displayed for incorrect password"],
  "security_tests": ["Verify HTTPS is enforced for login requests"],
  "accessibility_tests": ["Verify login form is navigable using keyboard only"],
  "api_tests": ["Verify POST /api/auth/login returns 200 with a token on valid credentials"],
  "regression_scope": ["Password reset and recovery flow"]
}

Training Details

Training Data

Private JSONL corpus qa360_sft.jsonl, one object per line:

json
{
  "instruction": "Perform a QA360 analysis for the following software requirement. Return a structured analysis covering risk_level (High/Medium/Low), automation_candidate (true/false), affected_modules (array), functional_tests (array), negative_tests (array), security_tests (array), accessibility_tests (array), api_tests (array), and regression_scope (array). Return only valid JSON with these exact keys.",
  "input": "[Authentication] User can login with email and password",
  "output": { "...QA360 object..." }
}

Rows were wrapped in Qwen ChatML before SFT:

text
<|im_start|>system
{instruction}<|im_end|>
<|im_start|>user
{input}<|im_end|>
<|im_start|>assistant
{json.dumps(output, indent=2)}<|im_end|>

Table with columns: Split, Rows, Notes
Split	Rows	Notes
Train	4,311	`train_test_split(test_size=0.1, seed=42)`
Eval	479	same format, used each epoch
Total	4,790

Training Procedure

Preprocessing

tokenizer.pad_token = tokenizer.eos_token (Qwen EOS 151645)
padding_side = "right"
dataset_text_field = "text"
Truncation at 2048 tokens

Hyperparameters

Table with columns: Setting, Value
Setting	Value
Base	`Qwen/Qwen2.5-1.5B-Instruct`
Method	QLoRA SFT → merge
LoRA rank `r`	16
LoRA `alpha`	32
LoRA dropout	0.05
LoRA targets	`q_proj`, `k_proj`, `v_proj`,

Speeds, Sizes, Times

Table

Hardware	e.g. 1× NVIDIA L4 24GB / A100 / local RTX
Wall time	TBD
Peak VRAM (QLoRA train)	TBD (1.5B 4-bit + LoRA r=16 is typically a few GB)
Merged fp16 size	~3.1 GB (same order as the base Instruct checkpoint)
Adapter size	tens of MB

Evaluation

Testing Data, Factors & Metrics

Held-out slice: 479 ChatML examples from the same qa360_sft.jsonl distribution (not a separately authored benchmark).

Suggested metrics — compute on the eval set and paste numbers; do not invent them:

Table with columns: Metric, Why it matters
Metric	Why it matters
JSON parse rate	Share of completions that `json.loads`
Exact-key schema rate	All 9 keys present, no extras
`risk_level` accuracy	Label match vs gold
`automation_candidate` accuracy	Boolean match vs gold
Token-overlap / embedding similarity on list fields	Wording will not match gold exactly
Trainer `eval_loss`	Training health only; not task quality

Qualitative checks used in the training script (not a benchmark):

Admin can force password reset
User I can upload a profile picture
user can lock account after failed attempts

Results

markdown
eval_loss:          TBD   # from trainer.evaluate()
json_parse_rate:    TBD
schema_valid_rate:  TBD
risk_level_acc:     TBD

Summary

Environmental Impact

Unknown until the GPU run is logged. Estimate with the MLCO2 calculator using hardware, hours, and region, then add:

Hardware Type:
Hours used:
Cloud Provider / Region:
Carbon Emitted:

QLoRA on 1.5B for ~1k steps is a small training job relative to pretraining.

Technical Specifications

Model Architecture and Objective

Architecture: Qwen2ForCausalLM
Objective: causal LM SFT on ChatML strings (next-token prediction over the full formatted example)
Post-train artifact: LoRA adapter merged into base weights with PeftModel.merge_and_unload()

Compute Infrastructure

OS / Python: Linux, Python 3.12
Key libraries: transformers, peft, trl, bitsandbytes, torch, datasets

Glossary

QA360: Internal name for a nine-field, full-stack test-analysis schema (risk + automation + six test views + regression).
QLoRA: 4-bit quantized base + low-rank adapters during training.
Merged model: Adapter baked into the base; loads as a normal transformers checkpoint, no PEFT required at inference.

Model Card Authors

Shankar Subramanyam