GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sanghwa-na
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
sanghwa-na
Model Tree
sanghwa-na/bobcat-1.2
Input Modalities
Output Modalities
Supported Functionality
bash
curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" # uvgit clone https://github.com/foxl-ai/bobcat && cd bobcatexport UV_PYTHON_PREFERENCE=only-manageduv venv --python 3.12 .venv-serveuv pip install --no-config --python .venv-serve/bin/python vllm==0.30.0 fastapi uvicorn scipy jinja2 \ "tokenizers>=0.21" huggingface_hub typesafe-sdk==0.7.1export PYTHONPATH=$PWD/src PY=.venv-serve/bin/python $PY -c "from huggingface_hub import snapshot_download as s; s('sanghwa-na/bobcat-1.2-fp8', local_dir='bobcat-1.2-fp8')"VLLM_USE_FLASHINFER_SAMPLER=0 $PY -m bobcat.flash_server --engine vllm --model bobcat-1.2-fp8 \ --compiler-model bobcat-1.2-fp8/compiler --quantization none \ --temperature choice=1.071833,noul=0.264866,score=1.157635 \ --name bobcat-1.2 --release-date 2026-10-09 --max-num-seqs 64 --max-model-len 98368 \ --schedule all --engine-arg max_num_batched_tokens=16384 --engine-arg attention_backend=TRITON_ATTN \ --host 127.0.0.1 --port 8000 --localThen call it with the official typesafe-sdk as on the main card
(TYPESAFE_BASE_URL=http://127.0.0.1:8000, TYPESAFE_DEFAULT_MODEL=bobcat-1.2).
--quantization none: vLLM reads the FP8 scheme from config.json (compressed-tensors).--temperature and --max-model-len 98368 are as on the main card: one calibration
temperature per question type, and states up to about 96K tokens read in full; longer inputs
are refused with HTTP 422 and never truncated. On an 80 GB GPU add
--engine-arg kv_cache_dtype=fp8.--compiler-model .../compiler points the Bobcat compiler at the base model's pinned
tokenizer, template and configuration; run the server from the repository root, where the
identifier list lives, or pass --identifiers bobcat-1.2-fp8/bobcat-identifiers.json.| Development decisions (3,188), one RTX PRO 6000, vLLM 0.30.0 | Accuracy | Task macro | Same answer as BF16 |
|---|---|---|---|
| Evaluation path (the BF16 weights of the main repository) | 92.85% | 92.94% | - |
The BF16 weights served in FP8 at load (--quantization fp8, the main card's command) | 92.60% | 92.69% | 99.1% |
| This checkpoint (FP8) | 92.63% | 92.66% | 99.2% |
FP8 minus BF16: -0.22 points [-0.52, +0.06] (paired, 95%); 27 answers changed, most of them
near-ties. Served by bobcat.flash_server over localhost HTTP with the command above, TypeSafe's
20 workflow cases agreed with the reference on 91.79% of 329 questions (no failed request),
at a median 1.04 s per case; the BF16 weights of the main repository, served in FP8 at load on
the same GPU, agreed on 90.27%, and the release gate run gave 90.88%. One question is 0.30 points;
fresh FP8 servers are not bit-identical, and part of the 5-question gap is run-to-run noise. For
accuracy-sensitive work and TypeSafe-style workflows, serve the main repository
(sanghwa-na/bobcat-1.2), the weights every
evaluation on its card used.
One RTX PRO 6000 96 GB, vLLM 0.30.0, --max-model-len 98368 | This checkpoint | BF16, FP8 at load |
|---|---|---|
| One decision (512 tokens, 8 candidates), engine p50 | 61.7 ms | 61.4 ms |
| Independent 1K-token requests, 512 concurrent | 9,213 tokens/s | 9,269 tokens/s |
| The longest JevBench-format test input (92,465 tokens) | answered, 47.6 s | answered, 47.5 s |
| GPU memory: weights / KV cache | 30.6 / 49.9 GiB | 30.6 / 50.0 GiB |
842da3794eaa0b77d5f08bae87a17459d91ff475), checked against its SHA256SUMS.json.scripts/fp8_quantize.py with llm-compressor 0.14.0 (FP8_DYNAMIC, no
calibration data), as for the Flash FP8 checkpoints: 410 Linear layers of the language model in FP8; lm_head, the embeddings, the norms and
the vision tower stay BF16.model-00001-of-00002.safetensors sha256 ca70231c…e485,
model-00002-of-00002.safetensors 9f773980…d00b, as recorded in the release manifest (bobcat-release-manifest.json,
serving_artifacts.fp8); every file is listed in .Apache-2.0. This checkpoint is a derivative of Gemma 4 31B-it by Google DeepMind
(Apache-2.0); the license text is included as LICENSE, and NOTICE lists what was changed.
Training data keep their own licenses (main card and THIRD_PARTY.md in
github.com/foxl-ai/bobcat). Bobcat is an independent
project, not affiliated with or endorsed by Google, the Qwen team, TypeSafe AI, Benchmark
Heaven or any other company named here. Provided as is, without warranty.
bibtex
@misc{bobcat12, title = {Bobcat 1.2}, author = {{The Bobcat Authors}}, year = {2026}, howpublished = {\url{https://huggingface.co/sanghwa-na/bobcat-1.2}}}SHA256SUMS.json