ThakiCloud

FactGuard-General-4B

README

License: apache-2.0

Status

Research preview. Sealed evaluation pending. Every number below comes from development sets built from the same sources as the training data. Performance on real, naturally occurring AI answers is not verified.

Usage

See example_inference.py and label_contract.json. Use the prompt exactly as specified: raw text, no chat template, add_special_tokens=False, at most 1,024 input tokens. Take the softmax over the four label-token logits at the last position.

head_general: the prompt for any domain (use this by default).
head_saas_pricing: the FactGuard Specialist prompt, kept for compatibility with FactGuard-4B pipelines.

AutoModelForCausalLM loads the text model. The checkpoint keeps the base model's full multimodal layout (vision tower, MTP weights, processor files) unchanged. Vision weights were not trained, and no image-verification ability is claimed.

Training

Base: Qwen/Qwen3.5-4B, revision 851bf6e806ef.
Start point: the FactGuard-4B v0.3 LoRA adapter (sha256 a9e62f07…). The same LoRA was trained further (continue-training). A from-base alternative was trained on the same data and compared (see Selection).
Method: LoRA r=16, alpha=32 on all attention, linear-attention and MLP projections of the language model (vision excluded), 1 epoch, batch 4×4, lr 5e-5, max length 1,024, seed 0. An auxiliary decision-evidence objective (weight 0.3) is used where it applies. Merge: W' = W + (B·A)·alpha/r in fp32, stored as BF16.
Hardware: one NVIDIA B200, 3,689 steps, 4,251 s of training.
Data (59,025 rows, not released). Public sources were converted to the four-label contract. Absence of evidence is always NOT_VERIFIABLE, never CONTRADICTED.

Table with columns: Source, Rows, Labels used, Version used, License as stated by the source
Source	Rows	Labels used	Version used	License as stated by the source
FactGuard Specialist replay (own)	11,082	A/B/C/D	FactGuard v0.3 train set	own data (synthetic brands and counterfactual edits of public pricing pages)
Synthetic general-domain counterfactuals (own)	5,990	A/B/C	`gen_generalist.py`, seed 0	own data (fictional names)
FEVER via

Changes made to the source data. Each row was converted into a Korean instruction format: evidence was split into numbered lines F1–F9, source labels were mapped to the four-label contract (neutral and absence of evidence become NOT_VERIFIABLE), FEVER NOT ENOUGH INFO rows and over-length rows were dropped, and samples were drawn per source with a fixed seed. CUAD claims are a fixed English sentence per clause category written by us; other claim and evidence text was used as published. The converted data is not released.

License review (in progress). Several sources are CC BY-SA (KLUE, KorNLI, FEVER/Wikipedia, Wikipedia tables and knowledge text, part of MultiNLI FICTION). Creative Commons' AI-training guidance takes a deliberately conservative position that publicly shared models built on ShareAlike content should use the same license, and also states that CC terms do not apply where copyright exceptions or limitations apply. Whether ShareAlike reaches these weights depends on facts and jurisdiction and has not been determined. The AttributionBench claims come from third-party generative systems whose terms were not reviewed. Until the review is complete, check each source's terms for your use.

Excluded: RAGTruth (its responses are generated by third-party commercial models) and VitaminC (held for a separate experiment).
Leakage filters: no dev group or dev claim text in training; no 8-gram overlap with the FactGuard sealed evaluation claims; train/dev id and group overlap 0.
A pre-training sample audit of 80 rows was done by an automated reviewer (Claude), not by a human. It found a FaithDial label-mapping error and empty FEVER evidence rows; an earlier data build was discarded and these were fixed before this training run.

Evaluation (development sets, not sealed)

General development set (2,700 rows, 300 per source, same sources as training):

Table with columns: Source, Before (FactGuard-4B v0.3), This model
Source	Before (FactGuard-4B v0.3)	This model
FEVER	0.770	0.933
MultiNLI	0.793	0.897
TabFact	0.697	0.913
AttributionBench	0.710	0.747
FaithDial	0.680	0.840
CUAD

Overall recall: SUPPORTED 0.882, CONTRADICTED 0.944, NOT_VERIFIABLE 0.857. False SUPPORTED 116 / 2,700, false CONTRADICTED 65 / 2,700. The raw base model scores 0.528 on the same set. The synthetic source is easy and inflates the overall number; AttributionBench (0.747) and KorNLI (0.810) are the weakest.

SaaS pricing (Specialist) development sets, using head_saas_pricing:

Table with columns: Set, FactGuard-4B v0.3, This model
Set	FactGuard-4B v0.3	This model
v0.3 dev (4,377)	0.993	0.992
format dev (513)	0.990	0.983
real-page dev (179)	0.933	0.916

Specialist ability is mostly kept but slightly reduced (10 more errors over 5,069 rows; SUPPORTED recall on the 179-row real-page set fell from 0.80 to 0.73). For SaaS pricing claims, prefer ThakiCloud/FactGuard-4B.

Single-stream latency (B200, batch 1): p50 81.2 ms, p95 101.7 ms.

Selection

Two arms were trained on identical data (single seed each):

Table with columns: Arm, General dev, v0.3 dev (false S / false C), GPU time
Arm	General dev	v0.3 dev (false S / false C)	GPU time
A: continue from FactGuard-4B (this model)	0.890	0.992 (18 / 8)	4,470 s
B: from Qwen3.5-4B base	0.892	0.982 (40 / 27)	4,341 s

General accuracy was the same; A kept the Specialist ability better, so A was selected. The difference on the general set (4 rows of 2,700) is not meaningful.

Limitations

Sealed evaluation pending. Development sets share sources and conversion rules with training; they overstate performance on new data.
Real AI-answer performance is not verified.
Not validated for legal, medical, or financial-regulatory decisions. CUAD rows test clause presence only.
Single-model decisions can produce false support. Pair the model with an evidence-coverage check and human review for high-stakes use.
Wrong or incomplete evidence can lead to wrong SUPPORTED decisions.

Provenance

Adapter sha256: dacbbe53a7cfcfed03197d516ed3e428ef288d0752433cb42c35de8f53908323
Training data sha256: 3396341ba67befffd4a2d87c0a708d2d3701017c1faba92f0f2501ec391fb5dc
Merged shard sha256: see eval_manifest.json
License metadata: the repository lists Apache-2.0, the license of the base model. Its compatibility with the training-data terms above is under review (see License review); it is not a legal conclusion.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ThakiCloud

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality