Tegh25

satellite-anomaly-reporter-qwen3-4b-lora

Deploy Dedicated

README

License: apache-2.0

Intended use

Explaining already-flagged segments from the OPSSAT-AD feature format, as a portfolio and research demonstration.
Not for operational spacecraft decisions. The reports are plausible narratives, not verified diagnoses (see Limitations).

How it was built

Data. OPSSAT-AD: 2,123 telemetry segments from ESA's OPS-SAT spacecraft, each described by 15 engineered features (no raw waveforms), about 20% labelled anomalous.
Teacher reports. A larger model (Claude Sonnet 5) wrote one report per segment from the features and the label. A prompt-engineering pass locked the format (plain prose, no markdown, no em dashes) and removed dataset-statistic language. A quality gate dropped any report that broke those rules.
Student. Qwen3-4B-Instruct-2507 fine-tuned with QLoRA on the teacher reports, with the loss computed on the report tokens only.
Detector. A Random Forest (class_weight="balanced", 300 trees, the 15 features) trained on the provided train split. On the 529 test segments it scores accuracy 0.964, precision 0.927 and recall 0.903 (19 errors). This is the detector used for the evaluation below.

Usage

The adapter expects exactly the prompt format it was trained on.

python

import torchfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfigfrom peft import PeftModel BASE = 'Qwen/Qwen3-4B-Instruct-2507'ADAPTER = 'Tegh25/satellite-anomaly-reporter-qwen3-4b-lora' tok = AutoTokenizer.from_pretrained(ADAPTER)model = AutoModelForCausalLM.from_pretrained(    BASE, device_map='auto',    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4',                                           bnb_4bit_compute_dtype=torch.float16))model = PeftModel.from_pretrained(model, ADAPTER) SYSTEM = ('You are a spacecraft operations engineer. You receive pre-computed statistical features of one flagged '          'telemetry segment (not raw sensor values) plus its label. Write a concise, evidence-based mission-ops '          'assessment in plain prose: which statistics drive your assessment, the likely cause, and the next step '          'for the ops team.') user = """Channel: CADC0872Segment length: 257 samples over 255 time unitsMean: 1.05848e-05, Std dev: 2.29761e-05, Variance: 5.27902e-10Kurtosis: -1.484, Skew: -0.060Peak count (raw / smoothed-10 / smoothed-20): 1 / 1 / 1Derivative peak count (1st / 2nd diff): 78 / 87Derivative variance (1st / 2nd diff): 5.5471e-13 / 7.03542e-13Sampling gap irregularity: 357Label: NOMINAL"""   # a real test-split segment; Label is ANOMALY or NOMINAL msgs = [{'role': 'system', 'content': SYSTEM}, {'role': 'user', 'content': user}]text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)inputs = tok(text, return_tensors='pt').to(model.device)out = model.generate(**inputs, max_new_tokens=450, do_sample=False)print(tok.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))

Output for that segment (greedy decoding):

This segment looks clean and well-behaved. The mean, variance, and standard deviation are all extremely small and consistent with a quiet, low-amplitude channel, and the kurtosis (-1.484) with near-zero skew (-0.060) points to a flat, symmetric distribution with no spiky outliers or lopsided behavior. Peak counts are minimal (1 raw, 1 smoothed at both window sizes), indicating a single, stable feature in the signal rather than noise-driven multi-peaking, and the derivative peak counts (78 and 87) are proportionate to a 257-sample record, suggesting normal sample-to-sample fluctuation rather than erratic spiking. The sampling gap irregularity value (357) is on the higher side but not extreme, and with no corresponding spike in variance or derivative behavior it doesn't indicate a real timing fault. No action needed; continue routine monitoring of CADC0872.

Evaluation

Evaluated on test segments never used for training or checkpoint selection: 158 segments sampled from the 520 test segments that passed the report quality gate (79 anomalies, 79 nominal, a balanced sample that includes all 18 detector errors among those 520; the detector made 19 errors across all 529 test segments). Rates are shown with 95% Wilson intervals, so they describe generation quality on this balanced sample, not population rates. All arms generate with greedy decoding.

Table with columns: Teacher (Sonnet 5), This model, Base model, prompted
	Teacher (Sonnet 5)	This model	Base model, prompted
Format-compliant (no markdown, em dashes, dataset language, or truncation)	100% [98-100]	100% [98-100]	41% [33-48]
Reports using em dashes	0%	0%	59% [52-67]
Mean words	164	158	156
Reports with at least one number not found in the prompt	6% [3-10]	6% [3-10]

"Base model, prompted" is the same Qwen3-4B with no adapter and a detailed style prompt. Its format failures are almost entirely em dashes, which a text filter would fix, so the format result mainly shows consistency without post-processing and without a long style prompt. Its lower rate of unsupported numbers comes from copying numbers verbatim, while the teacher and this model paraphrase and derive values such as ratios.

Blind pairwise LLM judge (Claude Sonnet 5, A/B order randomized, 60 segments per pair, half anomalies and half nominal):

Table with columns: Comparison, Wins, Ties, Losses, Unsupported claims per report
Comparison	Wins	Ties	Losses	Unsupported claims per report
This model vs prompted base	53	2	5	0.55 vs 1.47
This model vs teacher	21	21	18	0.40 vs 0.42

The judge is the same model family as the teacher, so it likely favours the teacher's writing style, and this model imitates that style. Read the first row as a clear but probably overstated edge over prompting, and the second as parity with the teacher.

Limitations

It trusts the label it is given. When the detector's flag was wrong, the report followed the wrong flag in 18 of 18 cases (8 false alarms, 10 missed anomalies). A missed anomaly becomes a confident "no action needed". The detector's probability is the available warning signal: flagging segments whose probability falls in an uncertain band for manual review catches most detector errors. On the 529 test segments:
- band 0.1 to 0.9: 20% of segments flagged for review, catches 18 of 19 detector errors
- band 0.2 to 0.8: 12% flagged, catches 13 of 19
- band 0.3 to 0.7: 7% flagged, catches 8 of 19
These bands were compared on the test split with only 19 errors, so the catch rates are optimistic and should not be read as a validated operating point. Some confident misses fall outside any band (one missed anomaly had a detector probability of 0.11).
Causes are not verified. The dataset has only a binary label, with no ground-truth cause. The reports contain plausible physical explanations (thermal or power transients, sensor connection issues, timing faults) that the features do not establish. The number check cannot catch invented causes.
It over-weights timing explanations. On the same 79 anomaly segments, the model recommends a timing- or telemetry-focused next step more often than its teacher (82% vs 66%; it did so 22 times where the teacher did not, and the reverse 9 times, sign test p = 0.029). Greedy decoding amplifies the teacher's majority habit.

Training details

Base: Qwen/Qwen3-4B-Instruct-2507, 4-bit NF4 quantization with double quantization.
LoRA: r = 16, alpha = 32, dropout = 0.05, on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
Optimization: learning rate 2e-4, cosine schedule, 8 warmup steps, 3 epochs, per-device batch 4 with 4 accumulation steps (effective batch 16), gradient checkpointing, fp16 on a Colab T4.
Loss on the completion (report) tokens only. Best checkpoint chosen by validation loss (about 0.71 at the end, with training loss about 0.58).
Data: after the report quality gate, 1,332 training reports, 238 validation reports, and 520 of the 529 test segments. Split: the dataset's provided train/test split, with 15% of the training pool held out for validation.
Versions: transformers 5.18.0, trl 1.15.0, peft 0.21.1, torch 2.11.0.

Data, teacher, and licence

Adapter licence: Apache 2.0, matching the base model.
Training reports were generated with Claude Sonnet 5 (via Backboard).
Dataset: OPSSAT-AD, licensed under Creative Commons Attribution 4.0 International. Bogdan, R. (2024). OPSSAT-AD - anomaly detection dataset for satellite telemetry [Dataset]. Ruszczak. https://doi.org/10.5281/zenodo.12588359

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Tegh25

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Adapter

this model

Input Modalities

Text

Output Modalities