predictive-maintenance

qwen3-4b-failuresensoriq-lora

README

License: apache-2.0

In plain language

Machines break. Before they do, they usually give warning signs: a pump vibrates more than it should, a transformer runs hot, oil turns dirty. Engineers keep tables of which warning sign points to which fault.

This model does a quiz about those tables. It reads a short text, a question like "a vibration sensor on this pump is showing an unusual reading — which fault does that point to?", and a list of four to eight possible answers labelled A to H. It replies with one letter.

It does well on the quiz it was trained on. Whether it would be useful in a real plant is a different question, and the honest answer is: we don't know, and the numbers below do not tell you. The section "How to read these numbers" explains why, and it is the most important part of this card.

A LoRA adapter for Qwen/Qwen3-4B, trained for multiple-choice question answering over sensor anomalies and failure modes of industrial assets (transformers, turbines, compressors, pumps), in the vocabulary of FMEA/FMECA analysis.

Built for and submitted to Track 1 of the IJCAI 2026 Industrial Automation Challenge, based on IBM's FailureSensorIQ benchmark.

Base model: Qwen/Qwen3-4B (4.0B parameters, within the competition's 8B limit)
Method: LoRA — base weights unchanged, 33.0M trainable parameters (0.81 %)
Inference cost: one forward pass, one generated token per question

Which data was used for what

This matters more than usual here, so it is stated explicitly.

Table with columns: File, Size, Has answers?, What we used it for
File	Size	Has answers?	What we used it for
`iso_sensors_mcqa_val.jsonl`	1,242 questions	yes	Everything: adapter training, prior fitting, all cross-validation, all evaluation below
`iso_sensors_mcqa_test_questions.jsonl`	3,048 questions	no	Kaggle leaderboard submissions only — never used for training

The Kaggle public leaderboard scored roughly 1,291 of those 3,048 questions. The final competition ranking uses a separate hidden test set that we have never seen.

The one thing to take away: this adapter has never been evaluated on any data outside the 1,242 validation questions. It was trained between 1 and 9 August 2026, after the Kaggle leaderboard had closed, so no external score exists for it. The 0.481 leaderboard result below belongs to a different, non-fine-tuned system.

What this adapter is for

The model answers with a single option letter. Questions have 4 to 8 options (A–H) and exactly one is correct. Inference uses restricted-vocabulary letter scoring: the prompt ends with the prefill "Answer: ", one forward pass is run, and a softmax is taken over only the option letters present in that question. No text is generated beyond the single answer token, so there is nothing to parse and parsing cannot fail.

Usage

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen3-4B"
ADAPTER = "predictive-maintenance/qwen3-4b-failuresensoriq-lora"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
tokenizer.padding_side = "left"          # required: logits are read at position -1

model = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, attn_implementation="sdpa")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

SYSTEM_MSG = (
    "You are a reliability engineer specialized in condition-based "
    "maintenance and FMEA/FMECA analysis of industrial assets. "
    "Answer with a single letter only."
)

def build_prompt(passage, question, options):
    options_text = "\n".join(f"{k}: {v}" for k, v in sorted(options.items()))
    user_msg = (
        f"{passage}\n\nQuestion: {question}\n\n"
        f"Options:\n{options_text}\n\n"
        "Exactly one option is correct. Respond with that option's "
        "letter and nothing else."
    )
    messages = [{"role": "system", "content": SYSTEM_MSG},
                {"role": "user", "content": user_msg}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    return prompt + "Answer: "

prompt = build_prompt(passage, question, options)
enc = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.inference_mode():
    logits = model(**enc).logits[0, -1, :].float()

letters = sorted(options.keys())
ids = [tokenizer.encode(L, add_special_tokens=False)[0] for L in letters]
probs = torch.softmax(logits[ids], dim=-1)
answer = letters[int(probs.argmax())]

The prompt format must be reproduced exactly. The adapter was trained on this template and will underperform on a different one.

Training

Table

Method	LoRA (`peft`), base weights frozen
Rank / alpha	16 / 32
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Trainable parameters	33.0M of 4.06B (0.81 %)
Learning rate

Loss. Cross-entropy over only the option letters valid for each question, evaluated at the single answer-token position:

L = −log [ exp(z_correct) / Σ over valid letters L' of exp(z_L') ]

This matches the inference rule exactly, so training optimizes the quantity the model is scored on, rather than spreading gradient across a 150k-token vocabulary at every position.

Permutation augmentation. Each question was expanded into 4 variants with the answer options in different cyclic orders, relabelling the correct letter accordingly. Any "none of the above" option was pinned to the final position, since that is a semantic convention rather than an artefact. The reason is explained below.

Evaluation

The positional label artefact, in one sentence

In this dataset the correct answer is far more often one of the last options than chance would suggest — so a program that ignores the question entirely and just picks a late option already gets about a third of them right.

A raw accuracy number here therefore mixes two different things: knowing the subject, and knowing where the answer tends to sit. The tables below try to keep them apart.

Results on the Kaggle test set (3,048 questions, never trained on)

These are the only numbers here measured on data outside the training file. All were produced before the leaderboard closed, by systems using the frozen base model — the fine-tuned adapter is absent from this table because it did not exist yet.

Table with columns: System, Fine-tuned?, Accuracy
System	Fine-tuned?	Accuracy
Random guessing (calculated)	—	18.9 %
Always answer "A"	no	13.7 %
Position only — ignores the question, picks by option position	no	34.5 %
Frozen Qwen3-4B + positional prior	no	48.1 %
This adapter	yes	not measured — trained after the leaderboard closed

The third row is why this card is so careful. A system that never reads the question scores 34.5 %, so most of the distance between random guessing and 48.1 % comes from the answer-position regularity rather than from understanding the questions.

Results on the validation set (1,242 questions — also the training data)

Every number below comes from those same 1,242 questions. The Split column says whether the evaluated rows were held out from whatever was fitted.

Table with columns: System, Split, Original order, Shuffled order, Gap
System	Split	Original order	Shuffled order	Gap
Random guessing (calculated)	—	17.1 %	17.1 %	—
Frozen base model, no calibration	nothing fitted	35.9 %	36.9 %	−1.0 pp
Frozen + positional prior	prior fitted on the same rows — in-sample	57.0 %	44.8 %

"Shuffled order" means the same questions with the answer options randomly reordered. It removes the positional regularity while leaving the knowledge and the difficulty untouched.

What holds up here: the prior-calibrated system loses 12 points when option order is randomized. This adapter loses 0.3 points in-sample and, in the cross-validated rows, exactly 0.00 — 89.05 % under both orderings, from fold accuracies of 92.27 / 84.54 / 90.34 (original) and 91.79 / 85.02 / 90.34 (shuffled). Permutation training removed the position dependence, and that result holds out-of-sample. It is the main methodological claim of this work.

What does not: the 98 % figures. Those rows show the released adapter scored on its own training data, and are included only so the gap between the two order columns can be compared. The honest out-of-sample figure is 89.05 % — and the next section explains why even that one needs a caveat.

How to read these numbers

While checking the 89.1 %, we found that the validation questions rest on a small set of underlying facts.

Each validation question carries metadata naming an asset class (e.g. "power transformer") and an anchor — the specific failure mode or sensor the question is about. Counting distinct combinations of the two across the 1,242 validation questions gives 195 pairs, about 6.4 questions per pair. The same underlying fact reappears under several question shapes: a positive phrasing, a negation ("which is not relevant"), a "none of the above" variant, and both the sensor→failure and failure→sensor directions.

This metadata exists only in the validation file. The test file does not contain it, so we cannot check whether the hidden test set is built the same way. The public FailureSensorIQ release is roughly seven times larger than our validation file and presumably covers more pairs. The count above describes our training data, not the benchmark as a whole.

Because of this, cross-validating by question id does not produce a clean holdout: 97 % of held-out questions have their (asset class, anchor) pair present in the training fold under a different question shape. Splitting accuracy accordingly:

Table with columns: Held-out questions, n, Accuracy, 95 % CI (Wilson)
Held-out questions	n	Accuracy	95 % CI (Wilson)
Pair seen in training under another shape	1,210	89.4 %	[87.6 %, 91.0 %]
Pair never seen under any shape	32	75.0 %	[57.9 %, 86.8 %]

The 14-point gap is statistically significant (Fisher exact p = 0.018; non-overlapping intervals). A substantial part of the headline accuracy comes from the model absorbing a compact set of facts, not from generalizing to new ones.

The 75 % figure should not be quoted as a headline number either: n = 32, a 29-point confidence interval, and the three folds disagree with each other beyond chance (90 % / 37.5 % / 85.7 %, χ² p = 0.018). It is also a weaker test than it sounds — an unseen pair within an already-seen asset class still shares that asset's sensor and failure-mode vocabulary with the training data.

A cross-validation grouped on (asset class, anchor) instead of question id would give a properly sized unseen set — roughly 414 questions per fold instead of 11 — and narrow the confidence interval from 29 points to about 8. That measurement is pending and this card will be updated with the result.

For context: human domain experts average roughly 60 % on FailureSensorIQ, and fine-tuned 8B models in the published literature reach 40–51 % (arXiv:2510.18817). Any number far above that range on this benchmark is better read as a sign that the model has absorbed the generating facts than as evidence of transferable diagnostic reasoning.

Limitations

No external validation yet.
Out-of-sample accuracy on genuinely unseen (asset class, anchor) pairs is not reliably established — see above.
Coverage of asset classes and failure modes in the training data is uneven.
Restricted to the prompt template above. Other phrasings are untested.
English only.
Confidence is poorly calibrated; treat output probabilities as rankings, not as probabilities.

Citation

The benchmark and the competition:

bibtex
@article{failuresensoriq2025,
  title  = {FailureSensorIQ: A Multi-Choice QA Dataset for Understanding
            Sensor Relationships and Failure Modes},
  author = {Constantinides, Christodoulos and Patel, Dhaval and Lin, Shuxin and others},
  journal = {arXiv preprint arXiv:2506.03278},
  year   = {2025}
}

@misc{industrial-automation-challenge-track-1,
  author = {Prateek Biswas},
  title  = {Industrial Automation Challenge - Track 1},
  year   = {2026},
  howpublished = {\url{https://kaggle.com/competitions/industrial-automation-challenge-track-1}},
  note   = {Kaggle}
}

Fine-tuning on this benchmark was previously studied in Fine-Tuned Thoughts: Leveraging Chain-of-Thought Reasoning for Industrial Asset Health Monitoring (arXiv:2510.18817, Findings of EMNLP 2025).

License

The adapter weights are released under Apache-2.0, matching the base model. The underlying benchmark data is CC-BY-4.0. Users should check the competition's own terms before redistributing anything derived from the competition splits.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider