sanghwa-na

bobcat-flash-1.2

README

License: apache-2.0

Highlights

Table with columns: Bobcat Flash 1.2, Reference
	Bobcat Flash 1.2	Reference
Sealed final, four tasks (1,609 decisions, opened once)	93.92%	Bobcat Flash 1.1 93.12% (+0.80 [-0.09, +1.79]); release bar 92.21%
Six-task development evaluation (3,188 decisions)	92.21%	same base zero-shot 86.6%
TypeSafe's published workflow examples: agreement with the reference (329 questions)	90.27%	Jev 90.9% (published), same base zero-shot 89.67%
Wrong answer named inside the state wins (300 questions x 3 attacks)	8.2%	same base zero-shot 31.1%
Long states: accuracy at 30K / 61K / 92K tokens against unpadded (1,200 / 300 / 300 questions)	-2.3 / -4.3 / -4.3 pt	Bobcat Flash 1.1: -10.8 / -14.3 / -16.0 pt
96K-token inputs (240 questions up to 96,161 tokens)	0 failures, 0 refusals	one RTX PRO 6000 and one B200, `--max-model-len 98368`
JevBench v1.6.1, official, run by Benchmark Heaven (1,500 items, sealed ones included)	73.5, #6 open-weights	Intelligence 65.8, Calibration 81.2; Quyet-1.0-Large 81.7 (#1); Jev 1.13.0 77.1 (reference, not ranked); rank as of 2026-10-08
One decision (512 tokens, 8 candidates), engine p50 / p95	25.2 / 25.6 ms	one RTX PRO 6000, vLLM 0.30.0, FP8 (H100: 19.0 / 19.5 ms)
Throughput, independent 1K-token requests	48,545 tokens/s	same GPU, 512 concurrent requests (H100: 63,080)

Jev figures are TypeSafe's own published answers; Jev was never called. The same-base baseline is untrained Gemma 4 26B-A4B-it with the same compiler and readout. See Limitations for what this model does not do well.

What it does

Table with columns: Primitive, You define, Bobcat returns
Primitive	You define	Bobcat returns
Choice	1 to 255 named candidates, descriptions optional	the top name and a probability for every name
Noul	optional meanings of true and false	P(true)
Score	1 to 10 ordered levels	the expected level and a probability per level

The request and reply use the shapes of TypeSafe's published System One HTTP API, so the official typesafe-sdk works against a Flash server by changing its base URL. Inputs over the server's limits are refused with HTTP 422 and never truncated.

Quickstart: serve Flash on one GPU

A clean install with the commands below was tested on one RTX PRO 6000 Blackwell 96 GB with vLLM 0.30.0, before these weights were uploaded: a fresh Python 3.12 venv with exactly the install line below, then the server command below on the merged weights (the download line itself runs once the repository is public). The SDK example answered payments; TypeSafe's 20 workflow cases agreed with the reference on 90.6% of 329 questions, at a median 0.27 s per case over loopback HTTP. On the same GPU and server command, all 240 JevBench-format inputs of 61K to 96,161 tokens were answered (no failure, no refusal).

bash

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"   # uvgit clone https://github.com/foxl-ai/bobcat && cd bobcatexport UV_PYTHON_PREFERENCE=only-managed   # a uv-managed Python ships the headers Triton compiles againstuv venv --python 3.12 .venv-serveuv pip install --no-config --python .venv-serve/bin/python vllm==0.30.0 fastapi uvicorn scipy jinja2 \  "tokenizers>=0.21" huggingface_hub typesafe-sdk==0.7.1export PYTHONPATH=$PWD/src PY=.venv-serve/bin/python # These weights (about 52 GB), then the typed-decision server, FP8 at load, 96K states$PY -c "from huggingface_hub import snapshot_download as s; s('sanghwa-na/bobcat-flash-1.2', local_dir='bobcat-flash-1.2')"VLLM_USE_FLASHINFER_SAMPLER=0 $PY -m bobcat.flash_server --engine vllm --model bobcat-flash-1.2 \  --compiler-model bobcat-flash-1.2/compiler --quantization fp8 \  --temperature choice=0.594604,noul=0.353553,score=0.529732 \  --name bobcat-flash-1.2 --release-date 2026-10-07 --max-num-seqs 64 --max-model-len 98368 \  --schedule all --engine-arg max_num_batched_tokens=16384 --engine-arg attention_backend=TRITON_ATTN \  --host 127.0.0.1 --port 8000 --local

Then call it with the official SDK:

python

import osos.environ.update(TYPESAFE_BASE_URL="http://127.0.0.1:8000", TYPESAFE_API_KEY="local",                  TYPESAFE_DEFAULT_MODEL="bobcat-flash-1.2")from typesafe_sdk import Choice, Noul, TypeSafeClient client = TypeSafeClient()result = client.system_one(    "I was charged twice for the same order. Can someone look into this?",    {"billing": Noul(instructions="Is this about billing?"),     "route": Choice(instructions="Which team should handle it?",                     criteria={"payments": "Charges and refunds", "shipping": None,                               "account": "Login and settings"})},)print(result.nouls["billing"].noul, result.choices["route"].choice)

Notes:

--temperature choice=0.594604,noul=0.353553,score=0.529732 applies one calibration temperature per question type, fitted on a held-out calibration split of JevBench-format decisions (generated and public, sharing nothing with JevBench) for the best JevBench-style Capability. A temperature never changes an argmax. --temperature 0.9232 instead applies the single temperature fitted by log-loss on the held-out calibration split of the six product tasks; it gives better-calibrated probabilities on those tasks, and the sharper per-type values commit more often on yes/no questions.
--quantization fp8 is vLLM's online dynamic FP8 of these BF16 weights, the served configuration.
--max-model-len 98368 reads states up to about 96K tokens in full (state plus question); longer inputs are refused with HTTP 422 and never truncated. On a GPU with less memory, lower it; the limit only moves the refusal point.
--local disables the shared secret the servers otherwise require; use it only on a loopback or private interface.
--compiler-model bobcat-flash-1.2/compiler points the Bobcat compiler at the base model's pinned tokenizer, template and configuration; it checks them against the download receipt in that folder. Run the server from the repository root, or pass --identifiers bobcat-flash-1.2/bobcat-identifiers.json.

Running on AWS

These are ordinary GPU Linux hosts; the Quickstart above is the whole recipe.

Amazon EC2. One RTX PRO 6000 Blackwell 96 GB (g7e.2xlarge) runs the clean install, the server at 98,368 tokens and every latency figure in the first column above; an H100 80 GB (p5.4xlarge) gave the second column. Use a Deep Learning AMI with a recent NVIDIA driver and allow about 80 GB of disk.
Amazon SageMaker AI. The same commands run in a JupyterLab space or notebook instance of an equivalent GPU type. A real-time endpoint needs a custom container that runs bobcat.flash_server behind SageMaker's /invocations and /ping routes; we have not published or tested one.

Evaluation

Differences are paired, with 95% bootstrap intervals over source components. The development and sealed-final figures use the evaluation path (these merged BF16 weights, bobcat.flash_train --eval-only); TypeSafe, injection, long-state and JevBench-format figures use the weights served in FP8 over HTTP.

Sealed final

A fresh sealed split of the product evaluation (1,609 decisions, four tasks, Korean), opened once, after the release manifest that binds these weights, their temperatures and the analysis was frozen. Nothing was changed after it. Failures count as wrong (there were none).

Table with columns: Task (decisions), Bobcat Flash 1.2, Bobcat Flash 1.1, Bobcat 1.1
Task (decisions)	Bobcat Flash 1.2	Bobcat Flash 1.1	Bobcat 1.1
Search passage selection (500)	79.4%	77.2%	81.0%
Citation verification (100)	100.0%	99.0%	100.0%
External document screening (550)	96.7%	96.7%	97.5%
Request routing (459)	99.6%	99.6%

Paired, over source components: Flash 1.2 minus Flash 1.1 +0.80 [-0.09, +1.79]; minus Bobcat 1.1 -0.64 [-1.93, +0.45].
Pre-declared sensitivity analysis, without the 50 rows whose text also appears in a training file (all from the earlier Flash training corpus; none from the data added for 1.2): 93.91%, +0.80 [-0.06, +1.70] over Flash 1.1.
The release bar fixed before the opening was 92.21%; it was met.
Served in FP8 (vLLM, engine path), the same decisions score 93.54%.
Log-loss 0.361 and ECE 0.014 at the product temperature (Flash 1.1 0.397 / 0.009).

Six-task development evaluation

The product evaluation's development split (3,188 decisions, Korean), the same split the release gates used.

Table with columns: Task, Bobcat Flash 1.2, Same base (Gemma 4 26B-A4B-it), zero-shot
Task	Bobcat Flash 1.2	Same base (Gemma 4 26B-A4B-it), zero-shot
Search passage selection	91.0%	84.8%
Citation verification	100.0%	96.5%
Tool-call review, never trained	92.8%	82.3%
External document screening	94.0%	88.9%
Request routing	99.1%	98.7%
Classification

TypeSafe's published workflow examples

20 cases, 329 questions answered by every published system. Agreement with TypeSafe's reference answers: 90.27% (Jev 90.9%, published; same base zero-shot 89.67%). Mean of the four workflows' agreement: 83.7% (Jev 86.5%, published). The release server on an RTX PRO 6000 (the clean install above) agreed on 90.6%, and a second clean install from the public repositories on 89.97%; the gate value 90.27% is from the evaluation server. TypeSafe 20 (329 questions; one question = 0.30 pt) varies between fresh FP8 servers: 89.97–90.6% across runs. The gate run was 90.27%; the corrected Flash-line bar was 90.19%.

Wrong answers named inside the state

300 questions, each with three attacks that name a wrong answer inside the state (an administrator verdict, a sentence inside the text, a note to an AI grader). Attack success: 8.2% (same base zero-shot 31.1%). Accuracy: clean 91.7%, administrator verdict 88.3%, inside the text 81.7%, note to an AI grader 83.0%.

Long inputs

The same questions with their state padded by unrelated documents (served FP8):

Table with columns: State length, Questions, Accuracy, Change [95%]
State length	Questions	Accuracy	Change [95%]
Unpadded	1,200	91.9%	-
30K tokens	1,200	89.6%	-2.3 [-4.0, -0.6]
61K tokens	300	88.0%	-4.3 [-7.7, -1.0]
92K tokens	300	88.0%	-4.3 [-8.0, -0.7]

Training rows were at most 32,768 tokens; the 61K and 92K levels are outside the training distribution. The untrained base was not run at these lengths. 240 JevBench-format questions of up to 96,161 tokens were all answered (0 failures, 0 refusals).

JevBench

JevBench (Benchmark Heaven, MIT) scores typed decisions by Capability = (Intelligence + Calibration) / 2; a Noul answer with P(yes) between 0.2 and 0.8 counts as an abstention.

Official, JevBench v1.6.1. Benchmark Heaven ran these weights (revision 58d1545) itself on one H100 PCIe, on 1,500 items including the sealed ones, measured 2026-10-07 and published on the leaderboard:

Table with columns: Capability [95%], Intelligence, Calibration, $/1k decisions
	Capability [95%]	Intelligence	Calibration	$/1k decisions
Bobcat Flash 1.2, #6 open-weights	73.5 [69.5, 76.4]	65.8	81.2	0.048 (estimate)
Quyet-1.0-Large, #1 open-weights	81.7	73.4

Latency and throughput

One H100 80 GB, vLLM 0.30.0, FP8, engine (no HTTP): 512 tokens, 8 candidates, 1 question, p50 / p95 19.0 / 19.5 ms; independent 1K-token requests up to 63,080 tokens/s (512 concurrent). Over loopback HTTP, one JevBench-format question at a time (3,030 questions): p50 39.3 ms, p95 142.5 ms.

Table with columns: One GPU, vLLM 0.30.0, FP8, --max-model-len 98368, RTX PRO 6000 Blackwell 96 GB, H100 80 GB
One GPU, vLLM 0.30.0, FP8, `--max-model-len 98368`	RTX PRO 6000 Blackwell 96 GB	H100 80 GB
One decision (512 tokens, 8 candidates), engine p50 / p95	25.2 / 25.6 ms	19.0 / 19.5 ms
Independent 1K-token requests, 512 concurrent	48,545 tokens/s	63,080 tokens/s
JevBench-format inputs of 61K-96K tokens, one at a time over HTTP, p50 / p95 / max	6.1 / 11.4 / 12.0 s	-
TypeSafe workflow case, median, loopback HTTP	0.27 s	-

The H100 figures were measured at a 32,832-token limit; at 98,368 the RTX PRO 6000 kept 85,169 MiB of its 97,887 MiB in use.

Bobcat Flash 1.1 -> 1.2

The same measurements of both models, paired where marked.

Table with columns: Bobcat Flash 1.1, Bobcat Flash 1.2, Change [95%]
	Bobcat Flash 1.1	Bobcat Flash 1.2	Change [95%]
Sealed final, four tasks (1,609 decisions, opened once)	93.12%	93.92%	+0.80 [-0.09, +1.79]
Six-task development evaluation (3,188)	92.07%	92.21%	+0.14 [-0.43, +0.76]
Tool-call review, never trained (development)	92.8%	92.8%
TypeSafe's workflow examples, agreement (329)	91.2%

Flash 1.1's TypeSafe and injection values are its published ones; on our evaluation server the TypeSafe agreement was 91.19% and injection 9.7%, the same. Flash 1.1's public-231 value uses its release temperature and the same scorer.

Quyet-1.0-Large (our measurement, unofficial)

Quyet-1.0-Large (chinhnc/Quyet-1.0-Large, revision e1ecbbe, Apache-2.0) was first on the JevBench leaderboard on 2026-10-05. We ran it with its own published runtime (quyet 1.0.0, default settings, one H100) on the same items and scored both models the same way. These are our measurements, not JevBench's; the leaderboard's official figures use sealed items we have never seen. On the official board Quyet-1.0-Large scores 81.7 and Bobcat Flash 1.2 73.5 (above).

Table with columns: Quyet-1.0-Large, Bobcat Flash 1.2, Bobcat minus Quyet [95%]
	Quyet-1.0-Large	Bobcat Flash 1.2	Bobcat minus Quyet [95%]
JevBench public items (231), three runs, our reconstruction	75.5	68.2	-7.3 [-15.9, +6.9]
Our JevBench-format development set, the 2,526 questions Quyet answers (10 options or fewer)	77.8	85.5	+7.7 [+5.4, +10.4]
Same set, all 3,030 questions (Quyet refuses more than 10 options; counted wrong)	74.5	86.0	+11.5
Long set, 240 questions of 61K-96K tokens	14.5

Quyet keeps the first 6,000 tokens of a state and accepts at most 10 options; Bobcat reads up to 98,368 tokens and 255 options. Quyet-1.0-Large is a separate project; nothing of it was used to train or tune Bobcat.

Training and distillation

Base: Gemma 4 26B-A4B-it at revision 4d7ae4984b7db7de8f8457170b3f1a419ee76d52: 30 layers of sliding-window (1,024) and global attention, 128 experts with 8 active plus a dense MLP, about 4B active parameters.
These weights are a merge of two rank-64 LoRA adapters on that base, each merged with bobcat.student_merge (float32 product and sum, BF16 storage), then combined tensor by tensor as 0.6 x W_A + 0.4 x W_B with scripts/arms12_interp.py (235 adapted tensors; the experts, embeddings and output head are the base's):
- A (adapter_model.safetensors sha256 7f9ae4d8…a7f39e): trained for this release, below.
- B (2304bd8e…9ca3): the adapter of sanghwa-na/bobcat-flash-1.1, unchanged.
- Weights 0.5, 0.6, 0.7, 0.775 and 0.85 for A were measured; 0.6 is the only one that met every release criterion (development split, TypeSafe's examples, injection, long states). No JevBench item was used to choose it.
LoRA rank 64, alpha 128, on the attention projections, the dense MLP and the expert router of all 30 layers (80.0M trainable parameters); one epoch, 1,922 steps of 32,768 tokens on each of eight GPUs, learning rate 2e-4, cosine decay.

Limitations and risks

Release criterion changed after measurement. TypeSafe agreement was to be at least 91.5%. After seeing that every 26B-A4B variant, including the untrained base, scored 89.1-90.3% on our setup, the criterion for this tier was set to the same-setup value of the earlier Flash model minus 1.0 point (90.19%); Flash 1.2 scores 90.27%. Under the original criterion it would not have been released.
Classification (76.4% on development) and search passage selection remain the weakest product tasks.
Long states cost 2.3 points at 30K and 4.3 points at 61K-92K tokens; nothing above 32,768 tokens was trained.
Near-ties move between server runs. Three fresh FP8 servers changed 1 and 4 of 231 top answers on JevBench's public items, and the 240-question long set scored 49.7 on an RTX PRO 6000 against 53.4 on the evaluation server. Where an answer must be reproducible, treat answers with a top probability below about 0.8 as unsure.
On the official JevBench board Flash 1.2 is 8.2 points below Quyet-1.0-Large (73.5 against 81.7), in Calibration (81.2 against 90.0) and in Intelligence (65.8 against 73.4). In our reconstruction on JevBench's public items it also scores below Bobcat 1.1 (68.2 against 73.8 on the same scorer), and its Intelligence there is lower than the untrained base's; our own development set, which favours it by 7.7 points over Quyet, does not predict this.
The development split and the sealed final are Korean; English results come from TypeSafe's examples and our generated sets, which no person has reviewed.
A typed answer can be the wrong valid option, and confidence is a concentration statistic, not the probability of being right. Compute arithmetic, counting and date comparison in code where it matters.

Intended use: small, repeated judgments whose admissible answers the application owns, where time per request matters - routing, guardrail screening, citation checks, tool-call review, classification - with the policy that acts on the answer kept in the application's code. Out of scope: generating text, open-ended question answering, and sole reliance on Bobcat Flash for safety-critical or legal decisions.

License and attribution

These weights are released under Apache-2.0 as a derivative of Gemma 4 26B-A4B-it by Google DeepMind, released under the Apache License 2.0; the license text is included as LICENSE, and NOTICE lists what was changed. Teachers: Bobcat adapters of Qwen3.8-27B (Apache-2.0) and GLM-5.3-Flash (MIT). Training data keep their own licenses: KLUE, KorNLI, ARC and SNLI (CC BY-SA 4.0), BoolQ (CC BY-SA 3.0), NSMC (CC0 1.0), MASSIVE, Banking77 and HelpSteer 2 and 3 (CC BY 4.0), CLINC150 (CC BY 3.0), MultiNLI (mostly the Open American National Corpus license, with fiction under CC BY-SA 3.0 and CC BY 3.0), OpenAssistant oasst2 and TruthfulQA (Apache-2.0), HH-RLHF and GSM8K (MIT). Attributions are in the code repository's THIRD_PARTY.md.

Bobcat is an independent project. It is not affiliated with or endorsed by TypeSafe AI, Benchmark Heaven, Google, the Qwen team, Z.ai, Anthropic, OpenAI or any other company named here; product names are their owners' trademarks and are used only to identify the models compared. TypeSafe's, SemIf's and JevBench's evaluation data are not redistributed here. The weights are provided as is, without warranty.

Citation

bibtex

@misc{bobcatflash12,  title        = {Bobcat Flash 1.2},  author       = {{The Bobcat Authors}},  year         = {2026},  howpublished = {\url{https://huggingface.co/sanghwa-na/bobcat-flash-1.2}}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider