ThakiCloud

FactGuard-8B

README

License: apache-2.0

Status

Research preview.

Sealed evaluation pending. A new sealed Specialist benchmark (unseen brands, documents and phrasings) will compare FactGuard 4B/8B/9B/27B and their base models; this card will be updated when it is out.
Performance on real AI answers is unverified.
The 0.987 below is a development-set result. It is not a generalization accuracy.
Not validated as a replacement for a 27B judge.

Usage

See example_inference.py and label_contract.json. Use the prompt exactly as specified (raw text, no chat template, add_special_tokens=False, at most 1,024 input tokens), and take the softmax over the four label token logits at the last position.

python

import torchfrom transformers import AutoTokenizer, AutoModelForCausalLMtok = AutoTokenizer.from_pretrained("ThakiCloud/FactGuard-8B")model = AutoModelForCausalLM.from_pretrained("ThakiCloud/FactGuard-8B", dtype=torch.bfloat16, device_map="auto")# prompt = HEAD + evidence_packet + "\n\n판정:"  (HEAD is in label_contract.json)

Qwen3-8B is text-only. Label token ids differ from the Qwen3.5 members: 362, 425, 356, 422 (see label_contract.json); example_inference.py computes them from the tokenizer.

Training

Base: Qwen/Qwen3-8B, revision b968826d9c46dd6066d109eabc6255188de91218. All 15 files match the Hub LFS/blob hashes.
Method: LoRA (r=16, alpha=32, dropout 0.05) on all attention and MLP projections (q/k/v/o, gate/up/down). The adapter was merged with W' = W + (B·A)·alpha/r, computed in fp32 and stored as BF16, on the unquantized base weights.
Curriculum. The same data, curriculum, and hyperparameters as FactGuard-4B v0.3. Each stage continued from the previous adapter, for 1 epoch at batch 4×4 and max length 1,024.
1. v0: 18,000 synthetic counterfactual rows on fictional brands, labels set by the generating operation, lr 1e-4.
2. v0.1: 10,000 rows (7,000 table/card/fragment formats on fictional brands + 3,000 replay), lr 5e-5.
3. v0.3: 4,530 rows (1,530 deterministic counterfactuals built on official pricing pages of 25 real SaaS brands, using only cells where two extractors agreed + 3,000 replay), lr 5e-5, seed 2.
An auxiliary head-free objective predicts the decision evidence id (weight 0.3).
Hardware: a single NVIDIA B200 for all stages (about 36 minutes of training in total; final stage 284 steps).

Evaluation

These are development sets, not sealed evaluations. They were used during FactGuard development (the 4B v0.3 seed was selected on them), so they are not a held-out estimate. A sealed family benchmark (unseen brands, documents and phrasings) is pre-registered and pending.

Table with columns: Model, V03DEV (4,377) acc · S recall · FS · FC, DEV4ALL (513), SLACKV22 (179)
Model	V03DEV (4,377) acc · S recall · FS · FC	DEV4ALL (513)	SLACKV22 (179)
Qwen3-8B base (no training)	0.495 · 0.868 · 412 · 1	0.752 · 0.986 · 98 · 0	0.888 · 0.661 · 1 · 0
FactGuard-8B v0.1	0.987 · 0.990 · 26 · 15	0.992 · 0.993 · 0 · 0	0.922 · 0.750 · 0 · 1
FactGuard-9B v0.1	0.992 · 0.994 · 18 · 13	0.981 · 0.980 · 1 · 4	0.955 · 0.857 · 0 · 1
FactGuard-4B v0.3	0.993 · 0.996 · 17 · 8

FS = false support. FC = false contradiction. V03DEV and DEV4ALL are counterfactual; SLACKV22 is 179 claims from real answers with labels that were not independently audited.

The untrained base leans toward SUPPORTED (412 false supports on V03DEV). Training reduces this to 26.
On these sets 8B is slightly below 4B and 9B, with the lowest SUPPORTED recall on SLACKV22. Architecture and tokenizer differ, so this is not a pure size effect.
Single-stream latency (B200, batch 1, model only): p50 68.1 ms, p95 74.8 ms.

Merge check: the merged weights and the original LoRA adapter produce the same decisions on held-out development items (see eval_manifest.json).

Limitations

Performance on real, naturally occurring AI answers is not established, and no sealed evaluation exists for this model yet. Do not read development or counterfactual accuracy as real-world accuracy.
It has the lowest SUPPORTED recall in the family on SLACKV22 (0.750) and more false contradictions than FactGuard-4B.
Single-model decisions can produce false support. Pair the model with an evidence-coverage guard and human review for high-stakes use.
The model was trained for pricing, plan, and feature claims only. It is not validated for legal, financial-regulatory, medical, or general-knowledge verification.
"Not found in the evidence" means NOT_VERIFIABLE, not "false". Wrong or incomplete evidence can lead to wrong SUPPORTED decisions.

Provenance

Adapter sha256: 0f4028352751b268971f56809f039e9f5a1dab3a1695c480a1a1e60336935221
Merged shard sha256: see eval_manifest.json
License: Apache-2.0, following the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ThakiCloud

Model Tree

Base

Qwen/Qwen3-8B

Fine-tuned

this model

Input Modalities

Text

Output Modalities