aaronr43210

terrapulse-qwen3-1.7b-lora-v2

README

License: apache-2.0

Usage

python

from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizerbase = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", torch_dtype="auto", device_map="auto")model = PeftModel.from_pretrained(base, "aaronr43210/terrapulse-qwen3-1.7b-lora-v2")tok = AutoTokenizer.from_pretrained("aaronr43210/terrapulse-qwen3-1.7b-lora-v2")# or load the adapter straight away: AutoModelForCausalLM.from_pretrained("aaronr43210/terrapulse-qwen3-1.7b-lora-v2")

What the model does (and doesn't)

The model handles two language tasks inside a deterministic system:

Intent extraction: a free-form question (English, Hinglish, typos) → JSON with activity, place text, verbatim time phrase, radius, people profiles (asthma, senior, child, pregnancy, cardiac, heat-sensitive…), preferences and scope.
Grounded answer writing: a fact sheet of live data (Open-Meteo weather, CAMS air quality, OpenStreetMap venues/roads/industry/green/water, OSRM travel times) → a short recommendation that cites fact ids such as [E4].

It holds no weather or place knowledge of its own. Dates, scores and safety limits are computed by code, and every answer is checked by a deterministic verifier (numbers + units, times, place names, citation ids and values, severity wording). An answer that fails is repaired once or replaced by a data template.

Why this base model

Measured bake-off on CPU (Q4_K_M, 24 hand-written questions): Qwen3-1.7B scored activity 0.917, location 1.0, profiles 0.958, scope 0.917 at 2.2 s per intent; Qwen3.5-2B scored 0.875 / 0.958 / 0.958 / 0.958 at 4.9 s. Excluded on size or licence: Phi-3/Phi-4-mini (3.8 B), Ministral-3-3B (3.85 B), Granite-4.0-micro (3.4 B), Llama-3.2-3B (3.21 B, non-OSI licence), Qwen2.5-3B (research licence), SmolLM3 (3.08 B), Gemma-3n-E2B (5.4 B).

Training

LoRA: r=16, α=32, dropout 0.05 on all attention and MLP projections; 2 epochs, lr 2e-4 cosine, completion-only loss; Unsloth + TRL on one 80 GB NVIDIA GPU of a college DGX server.
Data (aaronr43210/terrapulse-sft-data), about 4,000 examples before a ~5 % validation split:
- intent: 3,000 generated questions with exact labels (evaluation questions excluded);
- explanations: 500 fact sheets from the live pipeline. The open teacher Qwen/Qwen3-32B (bf16, thinking off, run on our own GPUs at training time only) wrote short friendly explanations in the app's current chat format; 246 passed the runtime verifier and were kept, plus 125 code-written explanations;
- follow-ups: 714 answers to "why?"-style questions ("is the air okay for this?", "why is it not recommended?") from the same fact sheets, each kept only because it passed the verifier.
- The last 20 fact sheets were held out of training for evaluation.
No proprietary or hosted LLM was used anywhere (data, training or runtime).
Prompt format: ChatML with an empty <think></think> block, identical in training and inference.

Evaluation (base model vs TerraPulse v2, identical inputs)

Both models ran on the same inputs, machine and settings (llama.cpp (llama-cpp-python), Apple M4 Pro GPU (Metal), Q4_K_M, temperature 0 for intent / 0.2 for text, seed 0). No test item was used for training. Method, per-item outputs and code: docs/evaluation/ and backend/eval/model_eval.py in the code repository.

Question understanding, 84 hand-labelled questions, model alone (macro-F1 % / accuracy %):

Table with columns: metric, base Qwen3-1.7B, TerraPulse v2 (this model)
metric	base Qwen3-1.7B	TerraPulse v2 (this model)
activity (14 classes)	94.2 / 95.9	97.2 / 97.3
location	100.0 / 100.0	100.0 / 100.0
people profiles (7)	78.0 / 92.9	97.1 / 98.8
in / out of scope	74.4 / 91.7	94.3 / 97.6
all fields correct	82.1 %	94.0 %
decision accuracy (plan / ask / decline), model alone	89.3 %	97.6 %
decision accuracy, full app (model + rule parser)	95.2 %	100.0 %
valid JSON	100.0 %	100.0 %

Robustness and fairness (172 perturbed questions, 48 counterfactual place swaps):

Table with columns: metric, base Qwen3-1.7B, TerraPulse v2 (this model)
metric	base Qwen3-1.7B	TerraPulse v2 (this model)
robustness: typos / shorthand / noise, model alone	83.7 %	87.8 %
robustness, full app	86.6 %	87.8 %
counterfactual place-swap consistency	100.0 %	100.0 %
accuracy gap across regions (N/W/S/E)	0.0 pts	0.0 pts
English vs Hinglish accuracy	86.8 vs 100.0	94.1 vs 100.0

Explanations, 20 held-out live fact sheets:

Table with columns: metric, base Qwen3-1.7B, TerraPulse v2 (this model)
metric	base Qwen3-1.7B	TerraPulse v2 (this model)
groundedness (passes the runtime verifier)	5.0 %	20.0 %
hallucination rate (lower is better)	5.0 %	30.0 %
…of which genuinely unsupported (not formatting or rounding)	0.0 %	20.0 %
factual claims checked	81	268
faithfulness (supported claims)	98.8 %	87.3 %
decision consistency	15.0 %	40.0 %
ROUGE-L F1 vs verified Qwen3-32B answer	29.6	42.2
BERTScore F1 (roberta-large, rescaled)	26.7	48.6

“Why?” follow-ups, 29 questions on the same sheets:

Table with columns: metric, base Qwen3-1.7B, TerraPulse v2 (this model)
metric	base Qwen3-1.7B	TerraPulse v2 (this model)
groundedness (passes the runtime verifier)	48.3 %	65.5 %
hallucination rate (lower is better)	0.0 %	13.8 %
…of which genuinely unsupported (not formatting or rounding)	0.0 %	3.4 %
factual claims checked	72	103
faithfulness (supported claims)	100.0 %	96.1 %
decision consistency	75.9 %	93.1 %
ROUGE-L F1 vs verified Qwen3-32B answer	35.5	46.1
BERTScore F1 (roberta-large, rescaled)	38.7	50.8

Latency, measured one model at a time on an Apple M4 Pro GPU (llama.cpp, Q4_K_M); end to end includes the live APIs:

Table with columns: metric, base Qwen3-1.7B, TerraPulse v2 (this model)
metric	base Qwen3-1.7B	TerraPulse v2 (this model)
intent JSON p50 / p95 (s)	1.9 / 2.09	1.94 / 2.06
explanation p50 / p95 (s)	1.74 / 2.21	2.38 / 5.44
end to end with live APIs p50 / p95 (s)	10.0 / 15.2	11.1 / 19.5

End to end, 44 live questions through the full app with real APIs:

Table with columns: metric, base Qwen3-1.7B, TerraPulse v2 (this model)
metric	base Qwen3-1.7B	TerraPulse v2 (this model)
correct outcome (answer / ask / decline / not available)	97.7 %	97.7 %
explanations written by the model and shown	10 of 31	19 of 32
activity / profile extraction	100 % / 100 %	100 % / 100 %

This model states far more specific facts than the base model (claims checked above), so the strict verifier flags more of its drafts; most flags are formatting habits (footnote numbers, rounding, 12-hour times), the rest genuinely unsupported. In the app no flagged draft reaches users: it is repaired once or replaced by a code-written explanation from the same data. ROUGE and BERTScore compare with the teacher's verified answers, whose style this model was trained to follow, so they favour it by design; groundedness, hallucination and decision consistency are the safety-relevant metrics. The test sets are small (tens of items), so differences of a few points are within noise.

Intended use

Research and demonstration inside an outdoor-activity assistant that supplies live data. Health-related limits are conservative heuristics based on US EPA AQI guidance and heat-index bands; this is not medical advice.

Limitations

Wording can be terse. Factual safety relies on the external verifier, not on the model alone.
Trained mostly on Indian place names, English and Hinglish.
Only 2 of the 160 training fact sheets contained road/industry exposure facts (OpenStreetMap's Overpass servers were unavailable during data collection), so those phrasings are under-represented.

Licence

Apache-2.0, the same as the base model. Map data © OpenStreetMap contributors; weather data by Open-Meteo.com (CC BY 4.0).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

aaronr43210

Model Tree

Base

Qwen/Qwen3-1.7B

Adapter

this model

Input Modalities

Text

Output Modalities