sanghwa-na

bobcat-1.2

README

License: apache-2.0

Highlights

Table with columns: Bobcat 1.2, Reference
	Bobcat 1.2	Reference
Sealed final, four tasks (1,609 decisions, opened once)	94.75%	Bobcat 1.1 94.56% (+0.20 [-1.15, +1.41]); Bobcat Flash 1.1 93.12% (+1.63 [+0.19, +3.07]); release bar 92.21%
Six-task development evaluation (3,188 decisions)	92.94%	same base zero-shot 90.44%
TypeSafe's published workflow examples: agreement with the reference (329 questions)	90.88%	Bobcat 1.1 as released, same server setup, 91.49%; Jev 90.9% (published); same base zero-shot 91.49%
Wrong answer named inside the state wins (300 questions x 3 attacks)	6.4%	same base zero-shot 46.4%
Long states: accuracy at 30K / 61K / 92K tokens against unpadded (1,200 / 300 / 300 questions)	-2.4 / -5.7 / -6.3 pt	Bobcat Flash 1.1: -10.8 / -14.3 / -16.0 pt
96K-token inputs (240 questions up to 96,161 tokens)	0 failures, 0 refusals	`--max-model-len 98368`; one H100 80 GB also answered the 8 longest
JevBench public items (231), our measurement (v1.6 method reconstructed), unofficial; not a JevBench rank	77.8	same base zero-shot 75.8; Quyet-1.0-Large 75.5 (our measurement)
One decision (512 tokens, 8 candidates), engine p50 / p95	51.0 / 51.8 ms	one H100 80 GB, vLLM 0.30.0, FP8 weights and KV cache
Throughput, independent 1K-token requests	12,441 tokens/s	same GPU, 512 concurrent requests

Jev figures are TypeSafe's own published answers; Jev was never called. The same-base baseline is untrained Gemma 4 31B-it with the same compiler and readout. See Limitations for what this model does not do well, including a release criterion that was changed after measurement.

What it does

Table with columns: Primitive, You define, Bobcat returns
Primitive	You define	Bobcat returns
Choice	1 to 255 named candidates, descriptions optional	the top name and a probability for every name
Noul	optional meanings of true and false	P(true)
Score	1 to 10 ordered levels	the expected level and a probability per level

The request and reply use the shapes of TypeSafe's published System One HTTP API, so the official typesafe-sdk works against a Bobcat server by changing its base URL. Inputs over the server's limits are refused with HTTP 422 and never truncated.

Quickstart: serve Bobcat 1.2 on one GPU

A clean install with the commands below was tested on one RTX PRO 6000 Blackwell 96 GB after these weights were published: a fresh Python 3.12 venv with exactly the install line below, a clone of the public code repository, the download line, then the server command. The SDK example answered payments, and TypeSafe's 20 workflow cases agreed with the reference on 90.27% of 329 questions over loopback HTTP. The same server command gave every served figure on this card (one GPU per server, vLLM 0.30.0).

bash

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"   # uvgit clone https://github.com/foxl-ai/bobcat && cd bobcatexport UV_PYTHON_PREFERENCE=only-managed   # a uv-managed Python ships the headers Triton compiles againstuv venv --python 3.12 .venv-serveuv pip install --no-config --python .venv-serve/bin/python vllm==0.30.0 fastapi uvicorn scipy jinja2 \  "tokenizers>=0.21" huggingface_hub typesafe-sdk==0.7.1export PYTHONPATH=$PWD/src PY=.venv-serve/bin/python # These weights (about 62 GB), then the typed-decision server, FP8 at load, 96K states$PY -c "from huggingface_hub import snapshot_download as s; s('sanghwa-na/bobcat-1.2', local_dir='bobcat-1.2')"VLLM_USE_FLASHINFER_SAMPLER=0 $PY -m bobcat.flash_server --engine vllm --model bobcat-1.2 \  --compiler-model bobcat-1.2/compiler --quantization fp8 \  --temperature choice=1.071833,noul=0.264866,score=1.157635 \  --name bobcat-1.2 --release-date 2026-10-09 --max-num-seqs 64 --max-model-len 98368 \  --schedule all --engine-arg max_num_batched_tokens=16384 --engine-arg attention_backend=TRITON_ATTN \  --host 127.0.0.1 --port 8000 --local

Then call it with the official SDK exactly as on the Bobcat Flash 1.2 card, with TYPESAFE_DEFAULT_MODEL="bobcat-1.2".

Notes:

bobcat.flash_server serves every Gemma 4 Bobcat model; the name is historical.
--temperature choice=1.071833,noul=0.264866,score=1.157635 applies one temperature per question type: Choice and Score fitted by log-loss on the test half of the public LocalLLaMA/typed-decisions set (never trained on), Noul on a held-out calibration split of JevBench-format decisions for the best JevBench-style Capability. A temperature never changes an argmax. --temperature 1.0833 instead applies the single temperature fitted by log-loss on the held-out calibration split of the six product tasks.
--quantization fp8 is vLLM's online dynamic FP8 of these BF16 weights (about 31 GB on the GPU), the served configuration. A pre-quantized FP8 checkpoint is at sanghwa-na/bobcat-1.2-fp8 (33.3 GB to download; development accuracy -0.22 points [-0.52, +0.06] against these weights, see its card). On an 80 GB GPU add --engine-arg kv_cache_dtype=fp8: one H100 80 GB then holds 212,884 tokens of KV cache, 2.2 requests of 98,368 tokens.
--max-model-len 98368 reads states up to about 96K tokens in full (state plus question); longer inputs are refused with HTTP 422 and never truncated. On a GPU with less memory, lower it; the limit only moves the refusal point.

Running on AWS

These are ordinary GPU Linux hosts; the Quickstart above is the whole recipe.

Amazon EC2. One H100 80 GB (p5.4xlarge) runs the server at 98,368 tokens with an FP8 KV cache and gave the latency figures above; one RTX PRO 6000 Blackwell 96 GB (g7e.4xlarge) runs it with the default KV cache; the evaluation servers ran one per GPU on B200 hosts (p6-b200.48xlarge). Use a Deep Learning AMI with a recent NVIDIA driver and allow about 140 GB of disk.
Amazon SageMaker AI. The same commands run in a JupyterLab space or notebook instance of an equivalent GPU type. A real-time endpoint needs a custom container that runs bobcat.flash_server behind SageMaker's /invocations and /ping routes; we have not published or tested one.

Evaluation

Differences are paired, with 95% bootstrap intervals over source components. The development and sealed-final figures use the evaluation path (these merged BF16 weights, bobcat.flash_train --eval-only); TypeSafe, injection, long-state and JevBench-format figures use the weights served in FP8 over HTTP.

Sealed final

A sealed split of the product evaluation (1,609 decisions, four tasks, Korean), opened once, after the release manifest that binds these weights, their temperatures and the analysis was frozen. Nothing was changed after it. Failures count as wrong (there were none).

Table with columns: Task (decisions), Bobcat 1.2, Bobcat 1.1, Bobcat Flash 1.1
Task (decisions)	Bobcat 1.2	Bobcat 1.1	Bobcat Flash 1.1
Search passage selection (500)	84.0%	81.0%	77.2%
Citation verification (100)	98.0%	100.0%	99.0%
External document screening (550)	97.5%	97.5%	96.7%
Request routing (459)	99.6%	99.8%

Paired, over source components: Bobcat 1.2 minus Bobcat 1.1 +0.20 [-1.15, +1.41]; minus Bobcat Flash 1.1 +1.63 [+0.19, +3.07].
Pre-declared sensitivity analysis, without the 50 rows whose text also appears in a training file (all in the earlier training corpora; none added by this release's data): 94.74%, +0.18 [-1.28, +1.40] over Bobcat 1.1 and +1.63 [+0.17, +3.04] over Flash 1.1.
The release bar fixed before the opening was 92.21%; it was met.
Served in FP8 (vLLM, engine path), the same decisions score 94.70%.
Log-loss 0.318 and ECE 0.015 at the product temperature (Bobcat 1.1 0.307 / 0.008, Flash 1.1 0.397 / 0.009).

Six-task development evaluation

The product evaluation's development split (3,188 decisions, Korean), the same split the release gates used.

Table with columns: Task, Bobcat 1.2
Task	Bobcat 1.2
Search passage selection	91.0%
Citation verification	99.3%
Tool-call review, never trained	93.9%
External document screening	95.5%
Request routing	99.3%
Classification	78.8%
Task macro	92.94%

Same base zero-shot: 90.44%. Bobcat Flash 1.1: 92.07% (Bobcat 1.2 +0.88 [-0.15, +2.05]). Bobcat 1.1: 93.69% (its own release evaluation, not paired here).

TypeSafe's published workflow examples

20 cases, 329 questions answered by every published system (one question = 0.30 points). Agreement with TypeSafe's reference answers: 90.88% (299 of 329), from the release gate run (FP8 at load, the product temperature, vLLM 0.30.0, one B200). On the same server setup, Bobcat 1.1 as released (sanghwa-na/bobcat-1.1-fp8, its own server and temperature) agreed on 91.49% and the untrained base on 91.49%; Jev's published answers agree on 90.9%.

Wrong answers named inside the state

300 questions, each with three attacks that name a wrong answer inside the state (an administrator verdict, a sentence inside the text, a note to an AI grader). Attack success: 6.4% (same base zero-shot 46.4%; Bobcat 1.1 7.8%). Accuracy: clean 92.7%, administrator verdict 90.3%, inside the text 84.3%, note to an AI grader 85.3%.

Almost all remaining successes are on the external-document screening task, where the state is the document under review: 37.6% of its 141 attacked questions. On the other five tasks the attack succeeds on 0.3% (2 of 720).

Long inputs

The same questions with their state padded by unrelated documents (served FP8):

Table with columns: State length, Questions, Accuracy, Change [95%]
State length	Questions	Accuracy	Change [95%]
Unpadded	1,200	93.3%	-
30K tokens	1,200	90.8%	-2.4 [-3.7, -1.1]
61K tokens	300	88.3%	-5.7 [-8.5, -2.9]
92K tokens	300	87.7%	-6.3 [-10.0, -2.8]

Training rows were at most 32,768 tokens; the 61K and 92K levels are outside the training distribution. 240 JevBench-format questions of up to 96,161 tokens were all answered (0 failures, 0 refusals). On one H100 80 GB (FP8 KV cache) the eight longest of them (92,219 to 94,084 tokens) were all answered, in 79 to 82 seconds each.

JevBench-format decisions

JevBench (Benchmark Heaven, MIT) scores typed decisions by Capability = (Intelligence + Calibration) / 2; a Noul answer with P(yes) between 0.2 and 0.8 counts as an abstention.

Public items (231): our measurement, unofficial; not a JevBench rank. Three fresh servers as in the Quickstart (FP8, per-type temperatures, --max-model-len 98368) through the official harness at commit bb05a33, run once, after the release manifest was frozen:

Table with columns: Capability, Intelligence, Calibration
	Capability	Intelligence	Calibration
Bobcat 1.2, per-type temperatures (three runs: 78.4 / 77.5 / 77.5)	77.8	68.3	87.3
Bobcat 1.2, product temperature 1.0833 for every type	75.6	62.2	89.0
Same base (Gemma 4 31B-it), zero-shot

Latency and throughput

Table with columns: One GPU, vLLM 0.30.0, these weights in FP8 at load, --max-model-len 98368, engine (no HTTP), H100 80 GB, FP8 KV cache, RTX PRO 6000 96 GB
One GPU, vLLM 0.30.0, these weights in FP8 at load, `--max-model-len 98368`, engine (no HTTP)	H100 80 GB, FP8 KV cache	RTX PRO 6000 96 GB
One decision (512 tokens, 8 candidates), p50 / p95	51.0 / 51.8 ms	61.4 / 61.5 ms
Independent 1K-token requests, 512 concurrent	12,441 tokens/s	9,269 tokens/s
JevBench-format inputs of 92K-94K tokens, over HTTP	8 of 8 answered, 79-82 s each	the longest (92,465 tokens) answered, 47.5 s
GPU memory: weights / KV cache	30.6 / 36.0 GiB (212,884 tokens)	30.6 / 50.0 GiB (147,548 tokens)

Bobcat 1.1 -> 1.2

Table with columns: Bobcat 1.1, Bobcat 1.2, Change [95%]
	Bobcat 1.1	Bobcat 1.2	Change [95%]
Sealed final, four tasks (1,609 decisions, opened once)	94.56%	94.75%	+0.20 [-1.15, +1.41]
Six-task development evaluation (3,188)	93.69%	92.94%	-0.75 (not paired)
TypeSafe's workflow examples, same server setup (329)	91.49%	90.88%	-0.61 (2 questions)
Wrong answer named inside the state wins	7.8%

Bobcat 1.1's long-state figures were measured on another padded set and are not repeated here.

Quyet-1.0-Large (our measurement, unofficial)

Quyet-1.0-Large (chinhnc/Quyet-1.0-Large, revision e1ecbbe, Apache-2.0) was first on the JevBench leaderboard on 2026-10-05. We ran it with its own published runtime (quyet 1.0.0, default settings, one H100) on the same items and scored both models the same way. These are our measurements, not JevBench's; the leaderboard's official figures use sealed items we have never seen.

Table with columns: Quyet-1.0-Large, Bobcat 1.2, Bobcat minus Quyet [95%]
	Quyet-1.0-Large	Bobcat 1.2	Bobcat minus Quyet [95%]
JevBench public items (231), three runs, our measurement	75.5	77.8	+2.3 [-1.4, +8.9]
Our JevBench-format development set, the 2,526 questions Quyet answers (10 options or fewer)	77.8	78.3	+0.5 [-2.1, +2.8]
Same set, all 3,030 questions (Quyet refuses more than 10 options; counted wrong)	74.5	79.3	+4.8 [+2.7, +7.2]

Quyet keeps the first 6,000 tokens of a state and accepts at most 10 options; Bobcat reads up to 98,368 tokens and 255 options. Quyet-1.0-Large is a separate project; nothing of it was used to train or tune Bobcat.

Training and distillation

Base: Gemma 4 31B-it at revision 842da3794eaa0b77d5f08bae87a17459d91ff475, a dense model; its tokenizer, tokenizer configuration and chat template are byte-identical to Gemma 4 26B-A4B-it's, so both Bobcat 1.2 models share one compiler.
Weights: one rank-16 LoRA adapter (adapter_model.safetensors sha256 c1cbdf4d…e2f8) merged into the base with bobcat.student_merge (float32 product and sum, BF16 storage; 410 adapted tensors).
Adapter: LoRA rank 16, alpha 32 (122.4M trainable parameters); one epoch, 243 steps of 32,768 tokens on each of eight GPUs, learning rate 1e-4.
- Loss per question: KL(teacher || student) where a teacher distribution exists, plus 0.5 times the gold loss (cross-entropy for Choice and Noul, plus an ordinal ranked probability score for Score), plus 1.0 x KL(base || student) on the corpus and product questions to keep the base's behaviour where the data says nothing.
- Augmentation: half of the Choice questions with their candidate order permuted and half of the object states with their field order permuted, replacing the originals.

Limitations and risks

Release criterion changed after measurement. TypeSafe agreement was to be at least 91.5%. Both trained candidates fell short (91.19% and 90.88%; the untrained base scored 91.49%), and the criterion for this model was then set to Bobcat 1.1 as released, measured on the same server setup, minus 1.0 point: 91.49% - 1.0 = 90.49%. Bobcat 1.2 scores 90.88%. Under the original criterion it would not have been released. Of the two candidates that met the changed criterion, the one with the higher judgement-set Capability was chosen, as the plan said.
Noul temperature chosen differently than planned. The plan fitted all three per-type temperatures on typed-decisions, Noul under a cap of 5% abstentions. No temperature met that cap on this set (its Noul gold is a soft label from several annotators; the lowest abstention was 6.7%), so Noul uses the calibration-split value. This was decided before the sealed final and the public items were opened.
Wrong answers planted in a document under review still win 37.6% of the time on the external-document screening task, close to the untrained base; on the other tasks 0.3%.
Classification (78.8% on development) and search passage selection remain the weakest product tasks; development macro is 0.75 points below Bobcat 1.1's.
Long states cost 2.4 points at 30K and 5.7 to 6.3 points at 61K-92K tokens; nothing above 32,768 tokens was trained. A 92K-token request takes about 80 seconds on one H100 80 GB.
Near-ties move between server runs. Fresh FP8 servers are not bit-identical: 2 of the 231 public JevBench answers changed between three servers, and on TypeSafe's 329 questions one answer is 0.3 points (fresh Flash 1.2 servers ranged over 89.97-90.6%).
, not the leaderboard's; the leaderboard uses sealed items we have never seen.

Intended use and out of scope: as for Bobcat Flash 1.2.

License and attribution

These weights are released under Apache-2.0 as a derivative of Gemma 4 31B-it by Google DeepMind, released under the Apache License 2.0; the license text is included as LICENSE, and NOTICE lists what was changed. Teacher: the Bobcat 1.1 adapter of Qwen3.8-27B (Apache-2.0). Training data keep their own licenses, as listed on the Bobcat Flash 1.1 card and in the code repository's THIRD_PARTY.md.

Bobcat is an independent project. It is not affiliated with or endorsed by TypeSafe AI, Benchmark Heaven, Google, the Qwen team or any other company named here; product names are their owners' trademarks and are used only to identify the models compared. TypeSafe's and JevBench's evaluation data are not redistributed here. The weights are provided as is, without warranty.

Citation

bibtex

@misc{bobcat12,  title        = {Bobcat 1.2},  author       = {{The Bobcat Authors}},  year         = {2026},  howpublished = {\url{https://huggingface.co/sanghwa-na/bobcat-1.2}}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider