sidchalamala

aether-coder-1.5b-lora

README

License: apache-2.0

Final evaluation: frozen held-out trap set (30 tasks, each model evaluated once)

Each task asks for something the policy forbids or Aether cannot do. The correct answer is a program that does the permitted alternative. The set was frozen before any trap training data existed and was never used for training or decisions.

Table with columns: System, Correct, 95% CI
System	Correct	95% CI
Base model + reference card + 3 examples	0/30	0.0-0.114
v0.1 (synthetic data only)	6/30	0.095-0.373
v0.2 (this adapter: + 413 trap samples)	6/30	0.095-0.373

Negative result: adding 413 targeted trap samples did not improve held-out performance. v0.2 vs v0.1: 6 vs 6, paired exact McNemar p = 1.
v0.2 vs the base model: p = 0.031.
Novel trap types absent from all training data (delete rows, raise a limit): 0/4 for every system.
In every evaluation, the checker blocked every program that broke a rule, and programs it accepted had 0 monitor violations. The model by itself solves 6/30 held-out traps.

Development benchmark (110 tasks, used during development)

This benchmark was used to build the scorer and compare models during development, so treat it as a development set, not a clean test.

Table with columns: System, Correct, Traps (30)
System	Correct	Traps (30)
Base model + reference card + 3 examples	3/110	0
v0.1	30/110	2
v0.2	34/110	3

v0.2 vs v0.1: paired p = 0.48.

Training

The v0.1 recipe (LoRA rank 16, learning rate 0.0002, 2 epochs, seed 42; only the program tokens are trained) on 2062 verified examples: 1649 synthetic plus 413 trap-style. Training took 42 minutes on one NVIDIA GeForce RTX 4070, at $0. The checkpoint was chosen by validation loss.

Limitations

Trained on templated synthetic English only. Most held-out failures are general errors on unfamiliar businesses (parse errors, type errors, reading columns the policy does not allow), not only trap-following.
All benchmark and held-out tasks were drafted by Claude, an AI assistant, not taken from real businesses.
Small evaluation sets with wide intervals; one training seed per model.
Aether's external actions (messages, money, calendars) are mocked. This is research software.
Do not describe this model, or Aether as a whole, as "safe". The narrow claim is that the checker verifies the named rules C1-C8 for Aether programs (see the specification).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

sidchalamala

Model Tree

Base

Qwen/Qwen2.5-Coder-1.5B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities