ThakiCloud

FactGuard-4B

README

License: apache-2.0

Status

Research preview. It is not validated as a drop-in replacement for a 27B judge on real AI answers (see Evaluation and Limitations).

The sealed results below (eval-v6) are this model's own. A new sealed Specialist benchmark comparing FactGuard 4B/8B/9B/27B on unseen brands, documents and phrasings is pre-registered and pending; this card will be updated when it is out. Performance on real AI answers is unverified.

Family: FactGuard collection — FactGuard-8B, FactGuard-9B, FactGuard-27B.

Usage

See example_inference.py and label_contract.json. Use the prompt exactly as specified (raw text, no chat template, add_special_tokens=False, at most 1,024 input tokens), and take the softmax over the four label token logits at the last position.

python

import torchfrom transformers import AutoTokenizer, AutoModelForCausalLMtok = AutoTokenizer.from_pretrained("ThakiCloud/FactGuard-4B")model = AutoModelForCausalLM.from_pretrained("ThakiCloud/FactGuard-4B", dtype=torch.bfloat16, device_map="auto")# prompt = HEAD + evidence_packet + "\n\n판정:"  (HEAD is in label_contract.json)

AutoModelForCausalLM loads the text model. The checkpoint keeps the base model's full multimodal layout (vision tower, MTP weights, processor files) unchanged. Vision weights were not trained, and no image-verification ability is claimed.

Training

Base: Qwen/Qwen3.5-4B, revision 851bf6e806ef. All 14 files match the Hub LFS/blob hashes.
Method: LoRA (r=16, alpha=32, dropout 0.05) on all attention, linear-attention, and MLP projections of the language model (vision excluded). The adapter was merged with W' = W + (B·A)·alpha/r, computed in fp32 and stored as BF16, on the unquantized base weights.
Curriculum. Each stage continued from the previous adapter, for 1 epoch at batch 4×4 and max length 1,024.
1. v0: 18,000 synthetic counterfactual rows on fictional brands, labels set by the generating operation, lr 1e-4.
2. v0.1: 10,000 rows (7,000 table/card/fragment formats on fictional brands + 3,000 replay), lr 5e-5.
3. v0.3: 4,530 rows (1,530 deterministic counterfactuals built on official pricing pages of 25 real SaaS brands, using only cells where two extractors agreed + 3,000 replay), lr 5e-5, seed 2.
An auxiliary head-free objective predicts the decision evidence id (weight 0.3).
Hardware: a single NVIDIA B200 for the final stage (283 steps, about 4.6 minutes); H100 for earlier stages.
The training data contains no customer data and no human or LLM labels. Sealed evaluation sets were never used in training. The training/eval brand and claim-string overlap with the evaluation sets is 0.

Evaluation

All numbers come from sealed, single-use evaluation sets. Every set is counterfactual unless stated otherwise. "+G2.2" means the model's label followed by a deterministic Claim Coverage Guard (guard/2.2, which only downgrades SUPPORTED/CONTRADICTED to NOT_VERIFIABLE when evidence coverage is insufficient). The guard is not part of these weights.

eval-v6: 2,483 counterfactual items on 28 brands not seen in training.

Table with columns: System, Accuracy, False support, False contradiction, SUPPORTED recall
System	Accuracy	False support	False contradiction	SUPPORTED recall
FactGuard-4B (model only)	0.913	27	37	0.874
FactGuard-4B + G2.2	0.908	7	8	0.853
Qwen3.8-27B prompted judge (model only)	0.847	66	202	0.913

Difference 4B+G2.2 − 27B+G2.2: −1.1%p, document-cluster bootstrap 95% CI [−5.0, +4.3].
A pre-registered non-inferiority bound (≥ −2%p) for replacing the 27B judge was not met.
Held-out operations (391 items): 0.767 vs 0.780.
Natural answers (42 claims from model-written answers, automatic labels): 0.381 vs 0.476. A later audit found that 22 of these 42 automatic labels were wrong or undeterminable. On the 20 label-certain items: 4B 0.80, 27B 1.00 (model only).
Single-stream latency (B200, batch 1, model only): p50 85.5 ms, p95 100.8 ms.

Merge check: the merged weights and the original LoRA adapter produce the same decisions on held-out development items (see Merge verification below).

Limitations

Performance on real, naturally occurring AI answers is not established. The available natural sample is small and its automatic labels were unreliable. Do not read the counterfactual accuracy as real-world accuracy.
The model rejects some correct SUPPORTED claims that are paraphrased. This is concentrated in the "{plan} 사용자도 …" pattern.
Single-model decisions can produce false support. Pair the model with an evidence-coverage guard and human review for high-stakes use.
The model was trained for pricing, plan, and feature claims only. It is not validated for legal, financial-regulatory, medical, or general-knowledge verification.
"Not found in the evidence" means NOT_VERIFIABLE, not "false". Wrong or incomplete evidence can lead to wrong SUPPORTED decisions.

Provenance

Adapter sha256: a9e62f07891f370c9845380a87bd6f7f7d0d52209e80515eacc80d3c032bb5ba
Merged shard sha256: see eval_manifest.json
License: Apache-2.0, following the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ThakiCloud

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality