Model Summary
Table with columns: Item, Value| Item | Value |
|---|
| Base model | Qwen/Qwen3-8B (bf16) |
| Fine-tuning | QLoRA (lora r=32, α=64, nf4-double quantization during training) |
| Training data | 50,567 rows of curriculum-based decision tasks (v15 incremental curriculum + v16 targeted enhancement) |
| Training config | 1 epoch, lr 5e-05, completed 2026-10-06 |
| Output | systemone primitives: choice / score / noul probability distributions |
| Adapter size | ≈349 MB (adapter_model.safetensors) |
| Inference latency | ≈0.9–1.2 s warm (public end-to-end, incl. gateway) |
Board 1: 500-question fresh randomized blind test (amos-2.8 / jev-1.13 / laya same board)
Independently generated banks, rule-based gold scoring, zero text-level overlap with training data. laya (convaiinnovations/laya-multilingual) was invoked zero-shot on the exact same board with identical request construction (2026-10-09).
Table with columns: Scope, amos-2.8, jev-1.13 (commercial API), laya (zero-shot)| Scope | amos-2.8 | jev-1.13 (commercial API) | laya (zero-shot) |
|---|
| Raw accuracy | 96.6% (483/500, 0 API errors; four-round mean 97.45% ±0.6pp) | 96.4% (482/500) | 42.8% (214/500) |
| Same-day live invocations (two rounds) | 97.8% | 97.8% (tied) | — (single run) |
| Tier-A pure decision tasks | 98.6% | 97.9% | 52.1% (146/280) |
| Strict OOD (task-domain overlap removed) | |
Board 2: 6,000-question three-model same board (ext6kb, seed=2027)
Table with columns: Scope, amos family*, jev-1.13, laya (zero-shot)| Scope | amos family* | jev-1.13 | laya (zero-shot) |
|---|
| Main tasks (15,000 questions) | 86.77% | 72.28% | 45.07% |
| Out-of-scope attributes (48,000 questions) | 96.23% | 86.87% | 43.89% |
| Latency p50 / p95 (ms) | 358.8 / 1184.3 | 1155.0 / 2804.7 | 24.1 / 28.2 |
| API errors | 0 | 0 |
* ext6kb was run with amos-2.6 as the family reference; amos-2.8 was not run on this board. laya's only absolute advantage is raw latency.
Per-Category Accuracy (Board 1, R3: amos-2.8 vs jev-1.13 vs laya)
12 of 17 categories at 100% for amos-2.8. amos-2.8 leads on NLI (+13.3pp), GSM8K word problems (+12.5pp), length band (+20pp); jev leads on character counting and deduplication; laya leads on no category (its best is topic at 100%, tied).
OOD disclosure (read before citing): question texts are fresh (0/500 text-level overlap with training data; sha1 exclusion: 5,475 full / 11,143 stem hashes blocked), but this board is NOT a domain-disjoint OOD benchmark. Sentiment and topic (70 questions) share their task domain with the training curriculum (3,646 sentiment / 250 topic Chinese samples) — their 100% scores are in-domain results. The remaining 15 categories have no same-template training samples, yet they all still fall within the decision-task family the model was designed for. Generalization to task types entirely absent from training has NOT been verified by this evaluation — all 17 categories are in-scope task types; open-set generalization to genuinely novel task domains requires a dedicated benchmark and remains untested. Under the strict scope (the 70 domain-overlapping questions removed), amos-2.8 scores 96.05% vs jev-1.13 95.81%; this number measures within-family domain overlap removal, not open-set generalization.
Table with columns: Category, Benchmark, n, amos-2.8, jev-1.13, laya (zero-shot)| Category | Benchmark | n | amos-2.8 | jev-1.13 | laya (zero-shot) |
|---|
| extraction | Extraction | 30 | 100.0% | 100.0% | 83.3% |
| knowledge | MMLU | 30 | 100.0% | 100.0% | 26.7% |
| math_word | GSM8K |
- Question banks are programmatically constructed from benchmark templates (SST / AG News / NLI / MMLU / HellaSwag / ARC / GSM8K, etc.).
- Full three-model comparison (amos-2.8 vs jev-1.13 vs laya): see
reports/amos-2.8-vs-jev-1.13-vs-laya.md in this repository.
Repository Files
├── adapter_config.json # PEFT/LoRA config (r=32, α=64, nf4-double)├── adapter_model.safetensors # LoRA adapter weights (≈349 MB)├── chat_template.jinja # Qwen3 chat template├── tokenizer_config.json├── tokenizer.json # Qwen3 tokenizer├── README.md└── reports/ # External benchmark reports (English)
Usage
This repository is a PEFT LoRA adapter; load the Qwen3-8B base first, then attach the adapter:
# pip install torch transformers peft safetensorsimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen3-8B", torch_dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(base, "/path/to/amos-2.8")tokenizer = AutoTokenizer.from_pretrained("/path/to/amos-2.8")
Production deployments use the systemone protocol endpoint (POST /v1/systemone). Request example:
{ "protocol": "choice", "state": {"user_id": "demo"}, "questions": { "q1": { "type": "choice", "query": "User message: is this SMS a scam? 'Congratulations, you won 50,000. Pay a 500 fee to claim.'", "options": ["Scam", "Legit notice", "Marketing", "Undecidable"] } }}
The response returns, for each question, the chosen option index and the normalized probability distribution.
Intended Use
- High-throughput text adjudication: intent classification, risk grading, sentiment polarity, language ID, structured attribute extraction, moderation
- Scenarios that need probabilistic output for downstream threshold decisions / human-review triage
- As a discriminative decision layer combined with LLMs (LLM generates, AMOS adjudicates)
Limitations and Known Weaknesses
- Decision-task accuracy is ~96–98%; a small fraction of high-confidence errors remains (confidence ≥ 0.9 errors ≈ 0.4%)
- Exact arithmetic / character counting (generative benchmarks) are not strengths
- Specialized domains (medical, legal clauses) require domain fine-tuning
- The adapter was trained under nf4 quantization; when merging back into the bf16 base, run a calibration check
Citation
AMOS-2.8 Decision Model (amos-3.0-gen-v16), 2026.Qwen/Qwen3-8B base · systemone three-primitive decision interface · QLoRA curriculum learning