cairndev

qwen3-4b-ifc-bim-qlora

README

License: apache-2.0

Results

On held-out IFC/BIM data

Loss on rows the model never trained on. Lower is better.

Table with columns: Base Qwen3-4B, This adapter
	Base Qwen3-4B	This adapter
Test split, 348 rows	1.955	0.218
Test split, response tokens only	1.619	0.156
Validation split, 301 rows	2.380	0.453
Validation split, response tokens only	1.950	0.429

How the loss table was measured:

Measured after training, with both models loaded 4-bit and the adapter loaded from this repository.
The same prompt format for both models: the chat template, instruction plus input as the user turn, with sequences up to 2,048 tokens.
The scripts and raw numbers are in eval/.

The trainer's own eval loss during training (2.718 → 0.554 on the validation split) uses its own batching and padding, so its absolute values differ from this table; the improvement is the same size. Test loss is lower than validation loss for both models. The split is by template family, so the test families appear to be easier.

General ability (forgetting check)

lm-evaluation-harness 0.4.13, with the same settings, seed and samples for both models. Accuracy; higher is better.

Table with columns: Task, Samples, Base Qwen3-4B, This adapter, Change
Task	Samples	Base Qwen3-4B	This adapter	Change
ARC-Easy (acc_norm)	150	0.767	0.760	−0.7
ARC-Challenge (acc_norm)	150	0.527	0.553	+2.7
HellaSwag (acc_norm)	150	0.647	0.653	+0.7
WinoGrande (acc)	150	0.740	0.707	−3.3
PIQA (acc_norm)	150	0.800	0.787	−1.3
MMLU (acc)	285 (5 per subject)	0.768	0.797	+2.8

The changes are in percentage points. Each score has a standard error of about ±3–4 points at these sample sizes, and no change is larger than that. So the run showed no measurable loss of general ability. It can't rule out a small one: larger samples would be needed for that.

Train and eval loss over 353 steps

Eval loss during training, measured every 50 steps:

Table with columns: step, 50, 100, 150, 200, 250, 300, 350, 353
step	50	100	150	200	250	300	350	353
eval loss	0.826	0.657	0.605	0.583	0.569	0.559	0.554	0.554

Train loss went from 2.19 at step 1 to 0.14 at step 353. Eval loss fell steadily and flattened, and never turned up. The full per-step history is in trainer_state.json.

Not measured yet: whether the IFC the model writes parses or validates.

Data

cairndev/ifc-bim-alpaca-dedup is derived from Dietmar2020/ifc-bim-high-quality-alpaca at commit 08d61c3c082c063c1647c59aeef1cf1107db6474 (Apache-2.0).

Source: 42,680 rows, no exact duplicates.
Near-duplicate removal (MinHash, threshold 0.8) dropped 36,385 rows (85%). Most source rows are the same template with only the numbers or names changed.
Kept: 6,295 rows, split 90/5/5 by template family, so a template's variants never appear on both sides of the split. The split's leakage check passed, and the holdout check found no overlap between train and val/test at launch.
Splits: train 5,646 · val 301 · test 348.
Length: 1.49 M tokens with the Qwen3 tokenizer; no row exceeds 2,048 tokens.

How it was trained

Table

Base	Qwen3-4B-Base, loaded 4-bit (Unsloth's `unsloth/Qwen3-4B-Base-unsloth-bnb-4bit`)
Method	QLoRA: r 16, alpha 32, dropout 0, all attention and MLP projections
Training	1 epoch, 353 steps, lr 2e-4, batch 2 × gradient accumulation 8, sequence length 2048, gradient checkpointing, seed 3407
Hardware	1 × NVIDIA Tesla T4 (16 GB), AWS g4dn.xlarge
Wall-clock	train 73 min (smoke check 5 min)
Software	Unsloth, TRL 0.24, PEFT 0.20, Transformers 5.5, PyTorch 2.11
GPU cost	*$0.86 se ttl e d * (t r ain$ 0.81, smoke $0.05) a g ain s t a * $ 16.50* worst-case estimate approved before the box started, with a $25 budget cap

The run was the cairn pipeline finetune/sft from public PyPI wheels (cairndev 0.3.1, cairn-pack-finetune 0.1.1). The steps, in order:

A person approved the data source and its pinned commit.
The launch checked the data roles and the holdout, and estimated the GPU cost.
A person approved the run.
The GPU box started; a 5-step smoke run, then the training run.
The box stopped.
The candidate beat the base model's eval loss.
A person approved the promotion.

Each step is recorded by run id. The adapter's content hash is 22fab5b3b08c0872c6696c7da2d547924a52ff20256b9bfe79e551a19a4e080c.

Use

The adapter loads with PEFT on top of Qwen3-4B-Base. Prompts use the chat template in this repository: the instruction, with the input appended when there is one, as the user turn.

python

from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Base", device_map="auto")model = PeftModel.from_pretrained(base, "cairndev/qwen3-4b-ifc-bim-qlora")tok = AutoTokenizer.from_pretrained("cairndev/qwen3-4b-ifc-bim-qlora")

Limitations

Narrow and templated data. The training data is template-generated, and walls (IFCWALL) are the largest single entity type. Expect the best results on questions shaped like the training templates, and weaker ones elsewhere.
Unchecked output. IFC output is not validated: GUIDs, references and geometry can be wrong. Check any IFC it writes with an IFC validator before using it in a BIM workflow.
One run, one seed. The benchmark samples are small (150 per task).

License and attribution

Apache-2.0. The base model is Qwen3-4B-Base by the Qwen team (Apache-2.0). The source data is Dietmar2020/ifc-bim-high-quality-alpaca (Apache-2.0).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

cairndev

Model Tree

Base

Qwen/Qwen3-4B-Base

Adapter

this model

Input Modalities

Text

Output Modalities