damianborek

polaris-2

README

License: apache-2.0

Use with the autonoxis plugins

The easiest way to use Polaris 2 is through the autonoxis server and one of its two clients: a Pi extension (pi-autonoxis-model 0.4.0 or later) and a Claude Code plugin (autonoxis-model 0.3.0 or later). Those versions ship the Polaris 2 questions. Both clients send the trained call convention for you, act only at confidence ≥ 0.8 (never lower: Pi uses a fixed 0.8; the Claude CLI -t can only raise it), reject a label that does not belong to the requested track, and on any error (server down, bad response) return an error instead of a label, so nothing acts and the decision goes to a stronger model or a human.

Polaris 1 still works with the older plugin versions (pi-autonoxis-model 0.3.x, autonoxis-model 0.2.x).

1. Run the server. server.py and its full setup (NVIDIA GPU, about 20 GB in bf16; torch, transformers, peft, huggingface_hub) are in pi-autonoxis-model/server:

bash

git clone https://github.com/damian87x/pi-autonoxis-model && cd pi-autonoxis-modelgit clone https://github.com/bespokelabsai/nimble server/nimblehf download bespokelabs/Bespoke-Nimble-9B --revision 594dfdcfb6f94e3d0c0db7535180d3c71689169a --local-dir basehf download damianborek/polaris-2 --local-dir adapterecho '{"model_path": "base", "max_input_tokens": 2048}' > nimble-model.jsonpython server/server.py --model-config nimble-model.json --adapter adapter --port 8765

It listens on http://127.0.0.1:8765 (loopback only, no authentication). Both clients read AUTONOXIS_URL and default to that address. GET /health returns {"status": "ok", "model": "polaris-2"}.

2. Pi.

bash

pi install git:github.com/damian87x/pi-autonoxis-model

Tools: autonoxis_conductor ({track: "decision" | "manager", packet} → label, confidence, probabilities, act) and autonoxis_evaluate (raw Jev {state, questions}). Command: /autonoxis-model status | test.

3. Claude Code.

text

/plugin marketplace add damian87x/autonoxis-model/plugin install autonoxis-model@autonoxis-model

(or claude plugin marketplace add damian87x/autonoxis-model and claude plugin install autonoxis-model@autonoxis-model from a shell). Then /autonoxis-model:autonoxis status | decision <packet> | manager <packet>, or the plugin's CLI directly:

bash

python3 scripts/autonoxis.py conductor --track manager --file packet.txt# exit 0 = ACT (conf >= 0.8), 3 = ESCALATE (below 0.8), 2 = error (never act)

4. Raw HTTP. One question with id label, state {"packet": ...}, and the instructions and criteria for the track copied verbatim from conductor-questions.json. With raw HTTP, apply the 0.8 gate yourself.

How to use without the plugins

You need the Nimble prompt builder and scorer from github.com/bespokelabsai/nimble: nimble.scoring.parallel_schema.prepare_prompts and nimble.training.schema_train.candidate_logits.

The call convention matches training. Use it exactly:

ask one question per request, with the field id label;
the state is the JSON {"packet": <packet text>};
the field's description and choice descriptions come from conductor-questions.json (instructions and criteria for that track).

Load the base model and attach the adapter unmerged (merging in bf16 shifts the probabilities).

python

import json, torchfrom transformers import AutoTokenizer, AutoModelForImageTextToTextfrom peft import PeftModelfrom huggingface_hub import hf_hub_downloadfrom nimble.scoring.parallel_schema import prepare_promptsfrom nimble.training.schema_train import candidate_logits BASE, REV = "bespokelabs/Bespoke-Nimble-9B", "594dfdcfb6f94e3d0c0db7535180d3c71689169a"ADAPTER = "damianborek/polaris-2" tok = AutoTokenizer.from_pretrained(ADAPTER)model = AutoModelForImageTextToText.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16).to("cuda")model = PeftModel.from_pretrained(model, ADAPTER).eval()   # do not mergequestions = json.load(open(hf_hub_download(ADAPTER, "conductor-questions.json"))) def ask(packet: str, track: str) -> dict:    q = questions[track]    field = {"type": "enum", "description": q["instructions"],             "choices": list(q["criteria"]), "choice_descriptions": q["criteria"]}    p = prepare_prompts(tok, json.dumps({"packet": packet}, ensure_ascii=False), {"label": field}, 2048)    ids, cands = p.full_ids[0], p.candidate_ids[0]    batch = {"input_ids": torch.tensor([ids], device="cuda"),             "attention_mask": torch.ones(1, len(ids), dtype=torch.long, device="cuda"),             "candidate_ids": torch.tensor([cands], device="cuda"),             "candidate_mask": torch.ones(1, len(cands), dtype=torch.bool, device="cuda")}    with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):        probs = candidate_logits(model, batch)[0].double().softmax(-1).tolist()    return dict(zip(q["criteria"], probs))

Confidence gate. Compute conf = (n * max_prob - 1) / (n - 1), where n is the number of labels. Act on the top label only when conf >= 0.8. Below that, escalate to a stronger model or a human.

Results

"Unsafe" means the model predicted DISPATCH or ACCEPT where the reference label is different. The evaluation sets are not released. eval-summary.json holds the per-set metrics for this adapter (no packet text).

Table with columns: set, Polaris 2, Polaris 1, Polaris 1 + newer plugin questions, Jev contract prompt
set	Polaris 2	Polaris 1	Polaris 1 + newer plugin questions	Jev contract prompt
v5 (40)	40/40	40/40	40/40	not measured
v6 (48)	48/48	48/48	48/48	not measured
v7 (48)	48/48	48/48	48/48	not measured

"Polaris 1" is the released Polaris 1 with its own questions. "Polaris 1 + newer plugin questions" is Polaris 1 as served by pi-autonoxis-model 0.3.0 / autonoxis-model 0.2.0. "Jev contract prompt" is untrained TypeSafe Jev given the same questions Polaris 2 was trained with.
v9 is 96 packets from real orchestrator runs (48 decision, 48 manager). Its reference labels are model labels, not human labels: Opus 5.5 and Grok 4.7 labelled every packet blind and agreed on 73; for the 23 disagreements the Opus 5.5 label is used. Polaris 2 gets 63 of the 73 agreed packets and 6 of the 23 disagreement packets. By track: decision 33/48, manager 36/48.
On v9, Polaris 2 roughly matches Jev with the same questions (69 vs 70), but runs locally at no per-call cost.
v9 unsafe: 5 packets (Polaris 1: 6). Four VERIFY packets answered ACCEPT (confidence 0.9995–1.0) and one ASK packet answered DISPATCH (0.998).
v9 with the 0.8 gate: keeps 93 of 96 at 74.2% accuracy. With a 0.95 gate: keeps 87 at 73.6%.
v5–v8 reference labels are Fable 5 (claude-fable-5). v8 was never used for training; it was one of the selection sets (the adapter was chosen by the v5–v8 total), and it is in-distribution. v5–v7 are not clean held-out sets (training examples were written from earlier mistakes on them). On v5–v8 Polaris 2 makes no unsafe predictions.

What changed from Polaris 1

New question wording for both tracks (the conductor contract prompt that improved untrained Jev on real packets). The adapter is trained on these questions, and the plugins now send them.
320 new training rows: packets written across eight domains (web, games, mobile, infrastructure, data pipelines, docs, ML, mixed), labelled blind by Opus 5.5 and Grok 4.7. They agreed on 318; the other 2 use the Opus 5.5 label. None of them is a v9 packet, and each was checked for text overlap against every evaluation set.
Same training recipe as Polaris 1.

Limitations

Confident misses. All five v9 unsafe answers are at confidence ≥ 0.99, four of them ACCEPT where VERIFY was expected. The 0.8 gate does not catch them. Treat ACCEPT as "verify first", and keep hard vetoes in code.
The gate is weaker on real packets. Above 0.8 confidence, accuracy is 74% on v9, against 100% on v5–v8.
Model labels. v9 labels come from two frontier models, and 23 of 96 are contested. Human labels could move the numbers either way.
Small seed count. Four seeds (a fifth crashed at start); v9 spread across seeds is 66–69.
The questions do much of the work. Polaris 1 given the same new questions without retraining also reaches 69/96 on v9 (60/60 on v8). Compared with that, Polaris 2 makes one fewer unsafe answer on v9 (5 vs 6) and passes more packets through the 0.8 gate (93 vs 88) at about the same accuracy (74.2% vs 75.0%).
Prompts are limited to 2048 tokens. Longer packets are rejected, not truncated. English only.
Narrow domain: conductor and manager decisions for autonomous coding lanes under one decision contract. The labels will not transfer to other policies.
The adapter is trained for the single-question convention above.

Training

Recipe (unchanged from Polaris 1): LoRA r=16, alpha=32, dropout 0.05, on the language-model Linear layers (attention, linear-attention and MLP projections). Candidate cross-entropy over the answer tokens. lr 5e-5, effective batch 8, 3 epochs, linear schedule with 10% warmup, bf16.
Questions: conductor-questions.json in this repository, for both tracks.
Data: 1396 rows. The 1076 Polaris 1 rows (lab history plus drafted contrastive rows aimed at the hard boundaries, with label disagreements adjudicated by Fable 5, claude-fable-5) plus the 320 new rows above (labels by Opus 5.5 and Grok 4.7, Opus 5.5 winning disagreements). No evaluation packet is in the training data (checked at build time).
The training data is not released.

License

Apache-2.0, the same as the base model. Bespoke-Nimble-9B is itself a LoRA-merged Qwen/Qwen3.5-9B, which is also Apache-2.0.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

damianborek

Model Tree

Base

bespokelabs/Bespoke-Nimble-9B

Adapter

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Use with the autonoxis plugins

Polaris 1 still works with the older plugin versions (pi-autonoxis-model 0.3.x, autonoxis-model 0.2.x).

1. Run the server. server.py and its full setup (NVIDIA GPU, about 20 GB in bf16; torch, transformers, peft, huggingface_hub) are in pi-autonoxis-model/server:

bash

git clone https://github.com/damian87x/pi-autonoxis-model && cd pi-autonoxis-modelgit clone https://github.com/bespokelabsai/nimble server/nimblehf download bespokelabs/Bespoke-Nimble-9B --revision 594dfdcfb6f94e3d0c0db7535180d3c71689169a --local-dir basehf download damianborek/polaris-2 --local-dir adapterecho '{"model_path": "base", "max_input_tokens": 2048}' > nimble-model.jsonpython server/server.py --model-config nimble-model.json --adapter adapter --port 8765

2. Pi.

bash

pi install git:github.com/damian87x/pi-autonoxis-model

3. Claude Code.

text

/plugin marketplace add damian87x/autonoxis-model/plugin install autonoxis-model@autonoxis-model

bash

python3 scripts/autonoxis.py conductor --track manager --file packet.txt# exit 0 = ACT (conf >= 0.8), 3 = ESCALATE (below 0.8), 2 = error (never act)

How to use without the plugins

You need the Nimble prompt builder and scorer from github.com/bespokelabsai/nimble: nimble.scoring.parallel_schema.prepare_prompts and nimble.training.schema_train.candidate_logits.

The call convention matches training. Use it exactly:

ask one question per request, with the field id label;
the state is the JSON {"packet": <packet text>};
the field's description and choice descriptions come from conductor-questions.json (instructions and criteria for that track).

Load the base model and attach the adapter unmerged (merging in bf16 shifts the probabilities).

python

import json, torchfrom transformers import AutoTokenizer, AutoModelForImageTextToTextfrom peft import PeftModelfrom huggingface_hub import hf_hub_downloadfrom nimble.scoring.parallel_schema import prepare_promptsfrom nimble.training.schema_train import candidate_logits BASE, REV = "bespokelabs/Bespoke-Nimble-9B", "594dfdcfb6f94e3d0c0db7535180d3c71689169a"ADAPTER = "damianborek/polaris-2" tok = AutoTokenizer.from_pretrained(ADAPTER)model = AutoModelForImageTextToText.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16).to("cuda")model = PeftModel.from_pretrained(model, ADAPTER).eval()   # do not mergequestions = json.load(open(hf_hub_download(ADAPTER, "conductor-questions.json"))) def ask(packet: str, track: str) -> dict:    q = questions[track]    field = {"type": "enum", "description": q["instructions"],             "choices": list(q["criteria"]), "choice_descriptions": q["criteria"]}    p = prepare_prompts(tok, json.dumps({"packet": packet}, ensure_ascii=False), {"label": field}, 2048)    ids, cands = p.full_ids[0], p.candidate_ids[0]    batch = {"input_ids": torch.tensor([ids], device="cuda"),             "attention_mask": torch.ones(1, len(ids), dtype=torch.long, device="cuda"),             "candidate_ids": torch.tensor([cands], device="cuda"),             "candidate_mask": torch.ones(1, len(cands), dtype=torch.bool, device="cuda")}    with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):        probs = candidate_logits(model, batch)[0].double().softmax(-1).tolist()    return dict(zip(q["criteria"], probs))

Results

Table with columns: set, Polaris 2, Polaris 1, Polaris 1 + newer plugin questions, Jev contract prompt
set	Polaris 2	Polaris 1	Polaris 1 + newer plugin questions	Jev contract prompt
v5 (40)	40/40	40/40	40/40	not measured
v6 (48)	48/48	48/48	48/48	not measured
v7 (48)	48/48	48/48	48/48	not measured

"Polaris 1" is the released Polaris 1 with its own questions. "Polaris 1 + newer plugin questions" is Polaris 1 as served by pi-autonoxis-model 0.3.0 / autonoxis-model 0.2.0. "Jev contract prompt" is untrained TypeSafe Jev given the same questions Polaris 2 was trained with.
v9 is 96 packets from real orchestrator runs (48 decision, 48 manager). Its reference labels are model labels, not human labels: Opus 5.5 and Grok 4.7 labelled every packet blind and agreed on 73; for the 23 disagreements the Opus 5.5 label is used. Polaris 2 gets 63 of the 73 agreed packets and 6 of the 23 disagreement packets. By track: decision 33/48, manager 36/48.
On v9, Polaris 2 roughly matches Jev with the same questions (69 vs 70), but runs locally at no per-call cost.
v9 unsafe: 5 packets (Polaris 1: 6). Four VERIFY packets answered ACCEPT (confidence 0.9995–1.0) and one ASK packet answered DISPATCH (0.998).
v9 with the 0.8 gate: keeps 93 of 96 at 74.2% accuracy. With a 0.95 gate: keeps 87 at 73.6%.
v5–v8 reference labels are Fable 5 (claude-fable-5). v8 was never used for training; it was one of the selection sets (the adapter was chosen by the v5–v8 total), and it is in-distribution. v5–v7 are not clean held-out sets (training examples were written from earlier mistakes on them). On v5–v8 Polaris 2 makes no unsafe predictions.

What changed from Polaris 1

New question wording for both tracks (the conductor contract prompt that improved untrained Jev on real packets). The adapter is trained on these questions, and the plugins now send them.
320 new training rows: packets written across eight domains (web, games, mobile, infrastructure, data pipelines, docs, ML, mixed), labelled blind by Opus 5.5 and Grok 4.7. They agreed on 318; the other 2 use the Opus 5.5 label. None of them is a v9 packet, and each was checked for text overlap against every evaluation set.
Same training recipe as Polaris 1.

Limitations

Confident misses. All five v9 unsafe answers are at confidence ≥ 0.99, four of them ACCEPT where VERIFY was expected. The 0.8 gate does not catch them. Treat ACCEPT as "verify first", and keep hard vetoes in code.
The gate is weaker on real packets. Above 0.8 confidence, accuracy is 74% on v9, against 100% on v5–v8.
Model labels. v9 labels come from two frontier models, and 23 of 96 are contested. Human labels could move the numbers either way.
Small seed count. Four seeds (a fifth crashed at start); v9 spread across seeds is 66–69.
The questions do much of the work. Polaris 1 given the same new questions without retraining also reaches 69/96 on v9 (60/60 on v8). Compared with that, Polaris 2 makes one fewer unsafe answer on v9 (5 vs 6) and passes more packets through the 0.8 gate (93 vs 88) at about the same accuracy (74.2% vs 75.0%).
Prompts are limited to 2048 tokens. Longer packets are rejected, not truncated. English only.
Narrow domain: conductor and manager decisions for autonomous coding lanes under one decision contract. The labels will not transfer to other policies.
The adapter is trained for the single-question convention above.

Training

Recipe (unchanged from Polaris 1): LoRA r=16, alpha=32, dropout 0.05, on the language-model Linear layers (attention, linear-attention and MLP projections). Candidate cross-entropy over the answer tokens. lr 5e-5, effective batch 8, 3 epochs, linear schedule with 10% warmup, bf16.
Questions: conductor-questions.json in this repository, for both tracks.
Data: 1396 rows. The 1076 Polaris 1 rows (lab history plus drafted contrastive rows aimed at the hard boundaries, with label disagreements adjudicated by Fable 5, claude-fable-5) plus the 320 new rows above (labels by Opus 5.5 and Grok 4.7, Opus 5.5 winning disagreements). No evaluation packet is in the training data (checked at build time).
The training data is not released.

License

Apache-2.0, the same as the base model. Bespoke-Nimble-9B is itself a LoRA-merged Qwen/Qwen3.5-9B, which is also Apache-2.0.

polaris-2

README

Links

Use with the autonoxis plugins

How to use without the plugins

Results

What changed from Polaris 1

Limitations

Training

License

Explore FriendliAI today

README

Links

Use with the autonoxis plugins

How to use without the plugins

Results

What changed from Polaris 1

Limitations

Training

License