damianborek

polaris-3

README

License: apache-2.0

Use with the autonoxis plugins

The easiest way to use Polaris 3 is through the autonoxis server and one of its two clients: a Pi extension (pi-autonoxis-model 0.5.0 or later) and a Claude Code plugin (autonoxis-model 0.4.0 or later). Both clients send the trained call convention for you, act only at confidence ≥ 0.8 (never lower: Pi uses a fixed 0.8; the Claude CLI -t can only raise it), reject a label that does not belong to the requested track, and on any error (server down, bad response) return an error instead of a label, so nothing acts and the decision goes to a stronger model or a human.

The questions did not change from Polaris 2, so Polaris 3 also works with pi-autonoxis-model 0.4.x and autonoxis-model 0.3.x, and Polaris 2 still works with the new versions. Polaris 1 still works with the older plugin versions (pi-autonoxis-model 0.3.x, autonoxis-model 0.2.x).

1. Run the server. server.py and its full setup (NVIDIA GPU, about 20 GB in bf16; torch, transformers, peft, huggingface_hub) are in pi-autonoxis-model/server:

bash

git clone https://github.com/damian87x/pi-autonoxis-model && cd pi-autonoxis-modelgit clone https://github.com/bespokelabsai/nimble server/nimblehf download bespokelabs/Bespoke-Nimble-9B --revision 594dfdcfb6f94e3d0c0db7535180d3c71689169a --local-dir basehf download damianborek/polaris-3 --local-dir adapterecho '{"model_path": "base", "max_input_tokens": 2048}' > nimble-model.jsonpython server/server.py --model-config nimble-model.json --adapter adapter --port 8765

It listens on http://127.0.0.1:8765 (loopback only, no authentication). Both clients read AUTONOXIS_URL and default to that address. GET /health returns {"status": "ok", "model": "polaris-3"}.

2. Pi.

bash

pi install git:github.com/damian87x/pi-autonoxis-model

Tools: autonoxis_conductor ({track: "decision" | "manager", packet} → label, confidence, probabilities, act) and autonoxis_evaluate (raw Jev {state, questions}). Command: /autonoxis-model status | test.

3. Claude Code.

text

/plugin marketplace add damian87x/autonoxis-model/plugin install autonoxis-model@autonoxis-model

(or claude plugin marketplace add damian87x/autonoxis-model and claude plugin install autonoxis-model@autonoxis-model from a shell). Then /autonoxis-model:autonoxis status | decision <packet> | manager <packet>, or the plugin's CLI directly:

bash

python3 scripts/autonoxis.py conductor --track manager --file packet.txt# exit 0 = ACT (conf >= 0.8), 3 = ESCALATE (below 0.8), 2 = error (never act)

4. Raw HTTP. One question with id label, state {"packet": ...}, and the instructions and criteria for the track copied verbatim from conductor-questions.json. With raw HTTP, apply the 0.8 gate yourself.

How to use without the plugins

You need the Nimble prompt builder and scorer from github.com/bespokelabsai/nimble: nimble.scoring.parallel_schema.prepare_prompts and nimble.training.schema_train.candidate_logits.

The call convention matches training. Use it exactly:

ask one question per request, with the field id label;
the state is the JSON {"packet": <packet text>};
the field's description and choice descriptions come from conductor-questions.json (instructions and criteria for that track).

Load the base model and attach the adapter unmerged (merging in bf16 shifts the probabilities).

python

import json, torchfrom transformers import AutoTokenizer, AutoModelForImageTextToTextfrom peft import PeftModelfrom huggingface_hub import hf_hub_downloadfrom nimble.scoring.parallel_schema import prepare_promptsfrom nimble.training.schema_train import candidate_logits BASE, REV = "bespokelabs/Bespoke-Nimble-9B", "594dfdcfb6f94e3d0c0db7535180d3c71689169a"ADAPTER = "damianborek/polaris-3" tok = AutoTokenizer.from_pretrained(ADAPTER)model = AutoModelForImageTextToText.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16).to("cuda")model = PeftModel.from_pretrained(model, ADAPTER).eval()   # do not mergequestions = json.load(open(hf_hub_download(ADAPTER, "conductor-questions.json"))) def ask(packet: str, track: str) -> dict:    q = questions[track]    field = {"type": "enum", "description": q["instructions"],             "choices": list(q["criteria"]), "choice_descriptions": q["criteria"]}    p = prepare_prompts(tok, json.dumps({"packet": packet}, ensure_ascii=False), {"label": field}, 2048)    ids, cands = p.full_ids[0], p.candidate_ids[0]    batch = {"input_ids": torch.tensor([ids], device="cuda"),             "attention_mask": torch.ones(1, len(ids), dtype=torch.long, device="cuda"),             "candidate_ids": torch.tensor([cands], device="cuda"),             "candidate_mask": torch.ones(1, len(cands), dtype=torch.bool, device="cuda")}    with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):        probs = candidate_logits(model, batch)[0].double().softmax(-1).tolist()    return dict(zip(q["criteria"], probs))

Confidence gate. Compute conf = (n * max_prob - 1) / (n - 1), where n is the number of labels. Act on the top label only when conf >= 0.8. Below that, escalate to a stronger model or a human.

Results

"Unsafe" means the model predicted DISPATCH or ACCEPT where the reference label is different. The evaluation sets are not released. eval-summary.json holds the per-set metrics for this adapter (no packet text).

v9, 94 real orchestrator packets, labels = majority of three models:

Table with columns: Polaris 3, mean of 8 seeds, Polaris 2, mean of 4 seeds, Polaris 3, released seed (may be optimistic, see selection note), Polaris 2, released seed, Polaris 1, Jev contract prompt (approx.)
	Polaris 3, mean of 8 seeds	Polaris 2, mean of 4 seeds	Polaris 3, released seed (may be optimistic, see selection note)	Polaris 2, released seed	Polaris 1	Jev contract prompt (approx.)
accuracy	77.1% (sd 1.9%)	76.3% (sd 1.4%)	75/94 (79.8%)	73/94 (77.7%)	61/94 (64.9%)	70/94 (≈74.5%)
decision / manager	34.6/48, 37.9/46	34.5/48, 37.25/46

All sets:

Table with columns: set, Polaris 3, mean of 8 seeds, Polaris 2, mean of 4 seeds, Polaris 3, released seed (may be optimistic, see selection note), Polaris 2, released seed, Polaris 1, Jev contract prompt
set	Polaris 3, mean of 8 seeds	Polaris 2, mean of 4 seeds	Polaris 3, released seed (may be optimistic, see selection note)	Polaris 2, released seed	Polaris 1	Jev contract prompt
v5 (40)	not reported	not reported	40/40	40/40	40/40	not measured
v6 (48)	not reported	not reported	48/48

v9 is 96 packets from real orchestrator runs (48 decision, 48 manager). Its reference labels are model labels, not human labels. Opus 5.5 (claude-opus-5-5), Grok 4.7 and Astra (gpt-6-astra) each labelled every packet blind; the reference is the label at least two of them chose. They were unanimous on 68, split 2–1 on 26, and all three differed on 2; those 2 packets have no reference and are left out (Polaris 3's answer on both matches one of the three votes).
The original 2-model labels row is kept for continuity with the Polaris 2 card: Opus 5.5 and Grok 4.7 labelled blind and Opus 5.5 decided their 23 disagreements. With the third labeller, 6 of those labels changed and 2 packets were left with no majority.
"Polaris 1" is the released Polaris 1 with its own questions. "Jev contract prompt" is untrained TypeSafe Jev given the same questions Polaris 2 and 3 were trained with. Only Jev's misses on v9 were kept, so its 3-model score assumes its other answers equal the original labels; it is approximate, and its gate cannot be recomputed.
Polaris 3 v9 unsafe (3-model labels): two VERIFY packets answered ACCEPT (confidence 0.991 and 0.999) and one ASK packet answered DISPATCH (0.965). Polaris 2 makes the same three mistakes.
Calibration holdout: 107 packets written from real agent-session moments (see What changed), never trained on: 102/107 (95.3%), 0 unsafe; with the 0.8 gate 106 kept at 95.3%.

Selection note. v9 was not used for training, and it is not in the selection key: best v5–v8 total, then most calibration-holdout packets correct, then lowest calibration-holdout log-loss. But seeds 1, 3 and 5 tied on the first two, and the log-loss tie-break was chosen after a per-seed table showing each seed's v9 score had already been viewed. It picked seed 1, which is also the best seed on v9. So the released seed's v9 figures (75/94, 79.8%) may be optimistically biased. The unbiased estimate for the Polaris 3 recipe is the 8-seed mean: 77.1% (sd 1.9%) on the 3-model labels, 3.1 unsafe, 0.8 gate 85.9 kept at 79.8%; 68/96 on the original labels. The fair comparison is with the Polaris 2 4-seed mean on the same labels: 76.3% (sd 1.4%), 3.0 unsafe, 0.8 gate 90.25 kept at 78.4%. (The released Polaris 2 seed was picked by v5–v8 total alone, a unique maximum, so no tie-break was needed there.)

What changed from Polaris 2

Training data from real sessions. 293 new training rows written from moments in real agent sessions (worker reports, reviewer verdicts, owner rulings, "tests pass" claims). Every packet was rewritten into a new invented setting and redacted (names, paths, hashes, addresses, keys); rows that shared text with the source sessions or overlapped any evaluation set were rewritten or dropped. Material connected to the sources of the v9 packets was excluded.
Soft labels. Opus 5.5 and Grok 4.7 labelled the new rows blind and agreed on 278 of 293. Where they disagreed (15 rows), the row is trained once with each label, so the model learns both are plausible.
Calibration holdout. A further 107 rows from the same pipeline were held out, never trained on, and used to pick the seed.
Same questions and training recipe as Polaris 2.

Limitations

The gate is only slightly better than Polaris 2. On v9, averaged over seeds, packets kept by the 0.8 gate are correct 79.8% of the time (Polaris 2: 78.4%); for the released seeds, 91 kept at 81.3% vs 91 at 80.2%. The three unsafe answers are confident (≥ 0.96) and pass it. Treat ACCEPT as "verify first", and keep hard vetoes in code.
Model labels, changed after training. v9 labels come from frontier models, not humans. The third labeller was added after Polaris 3 was trained and selected, so the 3-model numbers are on labels that did not exist when the model was built (see Selection note for how v9 relates to seed selection). On the original labels the gain over Polaris 2 is 2 packets (71 vs 69 of 96). Human labels could move the numbers either way.
2 ambiguous v9 packets have no majority label and are left out of the 94.
No clear accuracy gain. Seed averages on v9 are 77.1% (Polaris 3, 8 seeds, sd 1.9%) vs 76.3% (Polaris 2, 4 seeds, sd 1.4%), a difference of under one packet in 94 and within seed-to-seed variation; Polaris 3 seeds range 69–75 of 94, Polaris 2 seeds 70–73. The released seeds differ by 2 of 94 (75 vs 73), but the Polaris 3 seed may be optimistically selected (see Selection note).
The calibration holdout is in-distribution for the new training rows (same pipeline), so its 95% is much higher than v9 and should not be read as real-world accuracy.
Privacy. The new training packets were derived from real agent sessions, rewritten and redacted as above. The training data, the calibration holdout and the evaluation sets are not released. This card and eval-summary.json contain no packet text.
Prompts are limited to 2048 tokens. Longer packets are rejected, not truncated. English only.

Training

Recipe (unchanged from Polaris 1 and 2): LoRA r=16, alpha=32, dropout 0.05, on the language-model Linear layers (attention, linear-attention and MLP projections). Candidate cross-entropy over the answer tokens. lr 5e-5, effective batch 8, 3 epochs, linear schedule with 10% warmup, bf16.
Questions: conductor-questions.json in this repository, for both tracks (unchanged from Polaris 2).
Data: 1704 rows. The 1396 Polaris 2 rows, plus the 293 new rows above, plus 15 extra copies for the two-label rows. No evaluation or calibration packet is in the training data (checked at build time).
The training data is not released.

License

Apache-2.0, the same as the base model. Bespoke-Nimble-9B is itself a LoRA-merged Qwen/Qwen3.5-9B, which is also Apache-2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

damianborek

Model Tree

Base

bespokelabs/Bespoke-Nimble-9B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Use with the autonoxis plugins

1. Run the server. server.py and its full setup (NVIDIA GPU, about 20 GB in bf16; torch, transformers, peft, huggingface_hub) are in pi-autonoxis-model/server:

bash

git clone https://github.com/damian87x/pi-autonoxis-model && cd pi-autonoxis-modelgit clone https://github.com/bespokelabsai/nimble server/nimblehf download bespokelabs/Bespoke-Nimble-9B --revision 594dfdcfb6f94e3d0c0db7535180d3c71689169a --local-dir basehf download damianborek/polaris-3 --local-dir adapterecho '{"model_path": "base", "max_input_tokens": 2048}' > nimble-model.jsonpython server/server.py --model-config nimble-model.json --adapter adapter --port 8765

2. Pi.

bash

pi install git:github.com/damian87x/pi-autonoxis-model

3. Claude Code.

text

/plugin marketplace add damian87x/autonoxis-model/plugin install autonoxis-model@autonoxis-model

bash

python3 scripts/autonoxis.py conductor --track manager --file packet.txt# exit 0 = ACT (conf >= 0.8), 3 = ESCALATE (below 0.8), 2 = error (never act)

How to use without the plugins

You need the Nimble prompt builder and scorer from github.com/bespokelabsai/nimble: nimble.scoring.parallel_schema.prepare_prompts and nimble.training.schema_train.candidate_logits.

The call convention matches training. Use it exactly:

ask one question per request, with the field id label;
the state is the JSON {"packet": <packet text>};
the field's description and choice descriptions come from conductor-questions.json (instructions and criteria for that track).

Load the base model and attach the adapter unmerged (merging in bf16 shifts the probabilities).

python

import json, torchfrom transformers import AutoTokenizer, AutoModelForImageTextToTextfrom peft import PeftModelfrom huggingface_hub import hf_hub_downloadfrom nimble.scoring.parallel_schema import prepare_promptsfrom nimble.training.schema_train import candidate_logits BASE, REV = "bespokelabs/Bespoke-Nimble-9B", "594dfdcfb6f94e3d0c0db7535180d3c71689169a"ADAPTER = "damianborek/polaris-3" tok = AutoTokenizer.from_pretrained(ADAPTER)model = AutoModelForImageTextToText.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16).to("cuda")model = PeftModel.from_pretrained(model, ADAPTER).eval()   # do not mergequestions = json.load(open(hf_hub_download(ADAPTER, "conductor-questions.json"))) def ask(packet: str, track: str) -> dict:    q = questions[track]    field = {"type": "enum", "description": q["instructions"],             "choices": list(q["criteria"]), "choice_descriptions": q["criteria"]}    p = prepare_prompts(tok, json.dumps({"packet": packet}, ensure_ascii=False), {"label": field}, 2048)    ids, cands = p.full_ids[0], p.candidate_ids[0]    batch = {"input_ids": torch.tensor([ids], device="cuda"),             "attention_mask": torch.ones(1, len(ids), dtype=torch.long, device="cuda"),             "candidate_ids": torch.tensor([cands], device="cuda"),             "candidate_mask": torch.ones(1, len(cands), dtype=torch.bool, device="cuda")}    with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):        probs = candidate_logits(model, batch)[0].double().softmax(-1).tolist()    return dict(zip(q["criteria"], probs))

Results

v9, 94 real orchestrator packets, labels = majority of three models:

Table with columns: Polaris 3, mean of 8 seeds, Polaris 2, mean of 4 seeds, Polaris 3, released seed (may be optimistic, see selection note), Polaris 2, released seed, Polaris 1, Jev contract prompt (approx.)
	Polaris 3, mean of 8 seeds	Polaris 2, mean of 4 seeds	Polaris 3, released seed (may be optimistic, see selection note)	Polaris 2, released seed	Polaris 1	Jev contract prompt (approx.)
accuracy	77.1% (sd 1.9%)	76.3% (sd 1.4%)	75/94 (79.8%)	73/94 (77.7%)	61/94 (64.9%)	70/94 (≈74.5%)
decision / manager	34.6/48, 37.9/46	34.5/48, 37.25/46

All sets:

Table with columns: set, Polaris 3, mean of 8 seeds, Polaris 2, mean of 4 seeds, Polaris 3, released seed (may be optimistic, see selection note), Polaris 2, released seed, Polaris 1, Jev contract prompt
set	Polaris 3, mean of 8 seeds	Polaris 2, mean of 4 seeds	Polaris 3, released seed (may be optimistic, see selection note)	Polaris 2, released seed	Polaris 1	Jev contract prompt
v5 (40)	not reported	not reported	40/40	40/40	40/40	not measured
v6 (48)	not reported	not reported	48/48

v9 is 96 packets from real orchestrator runs (48 decision, 48 manager). Its reference labels are model labels, not human labels. Opus 5.5 (claude-opus-5-5), Grok 4.7 and Astra (gpt-6-astra) each labelled every packet blind; the reference is the label at least two of them chose. They were unanimous on 68, split 2–1 on 26, and all three differed on 2; those 2 packets have no reference and are left out (Polaris 3's answer on both matches one of the three votes).
The original 2-model labels row is kept for continuity with the Polaris 2 card: Opus 5.5 and Grok 4.7 labelled blind and Opus 5.5 decided their 23 disagreements. With the third labeller, 6 of those labels changed and 2 packets were left with no majority.
"Polaris 1" is the released Polaris 1 with its own questions. "Jev contract prompt" is untrained TypeSafe Jev given the same questions Polaris 2 and 3 were trained with. Only Jev's misses on v9 were kept, so its 3-model score assumes its other answers equal the original labels; it is approximate, and its gate cannot be recomputed.
Polaris 3 v9 unsafe (3-model labels): two VERIFY packets answered ACCEPT (confidence 0.991 and 0.999) and one ASK packet answered DISPATCH (0.965). Polaris 2 makes the same three mistakes.
Calibration holdout: 107 packets written from real agent-session moments (see What changed), never trained on: 102/107 (95.3%), 0 unsafe; with the 0.8 gate 106 kept at 95.3%.

What changed from Polaris 2

Training data from real sessions. 293 new training rows written from moments in real agent sessions (worker reports, reviewer verdicts, owner rulings, "tests pass" claims). Every packet was rewritten into a new invented setting and redacted (names, paths, hashes, addresses, keys); rows that shared text with the source sessions or overlapped any evaluation set were rewritten or dropped. Material connected to the sources of the v9 packets was excluded.
Soft labels. Opus 5.5 and Grok 4.7 labelled the new rows blind and agreed on 278 of 293. Where they disagreed (15 rows), the row is trained once with each label, so the model learns both are plausible.
Calibration holdout. A further 107 rows from the same pipeline were held out, never trained on, and used to pick the seed.
Same questions and training recipe as Polaris 2.

Limitations

The gate is only slightly better than Polaris 2. On v9, averaged over seeds, packets kept by the 0.8 gate are correct 79.8% of the time (Polaris 2: 78.4%); for the released seeds, 91 kept at 81.3% vs 91 at 80.2%. The three unsafe answers are confident (≥ 0.96) and pass it. Treat ACCEPT as "verify first", and keep hard vetoes in code.
Model labels, changed after training. v9 labels come from frontier models, not humans. The third labeller was added after Polaris 3 was trained and selected, so the 3-model numbers are on labels that did not exist when the model was built (see Selection note for how v9 relates to seed selection). On the original labels the gain over Polaris 2 is 2 packets (71 vs 69 of 96). Human labels could move the numbers either way.
2 ambiguous v9 packets have no majority label and are left out of the 94.
No clear accuracy gain. Seed averages on v9 are 77.1% (Polaris 3, 8 seeds, sd 1.9%) vs 76.3% (Polaris 2, 4 seeds, sd 1.4%), a difference of under one packet in 94 and within seed-to-seed variation; Polaris 3 seeds range 69–75 of 94, Polaris 2 seeds 70–73. The released seeds differ by 2 of 94 (75 vs 73), but the Polaris 3 seed may be optimistically selected (see Selection note).
The calibration holdout is in-distribution for the new training rows (same pipeline), so its 95% is much higher than v9 and should not be read as real-world accuracy.
Privacy. The new training packets were derived from real agent sessions, rewritten and redacted as above. The training data, the calibration holdout and the evaluation sets are not released. This card and eval-summary.json contain no packet text.
Prompts are limited to 2048 tokens. Longer packets are rejected, not truncated. English only.

Training

Recipe (unchanged from Polaris 1 and 2): LoRA r=16, alpha=32, dropout 0.05, on the language-model Linear layers (attention, linear-attention and MLP projections). Candidate cross-entropy over the answer tokens. lr 5e-5, effective batch 8, 3 epochs, linear schedule with 10% warmup, bf16.
Questions: conductor-questions.json in this repository, for both tracks (unchanged from Polaris 2).
Data: 1704 rows. The 1396 Polaris 2 rows, plus the 293 new rows above, plus 15 extra copies for the two-label rows. No evaluation or calibration packet is in the training data (checked at build time).
The training data is not released.

License

Apache-2.0, the same as the base model. Bespoke-Nimble-9B is itself a LoRA-merged Qwen/Qwen3.5-9B, which is also Apache-2.0.

polaris-3

README

Links

Use with the autonoxis plugins

How to use without the plugins

Results

What changed from Polaris 2

Limitations

Training

License

Explore FriendliAI today

README

Links

Use with the autonoxis plugins

How to use without the plugins

Results

What changed from Polaris 2

Limitations

Training

License