GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ThakiCloud
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ThakiCloud
Model Tree
Qwen/Qwen3-Reranker-4B
Input Modalities
Output Modalities
Supported Functionality
Given a question and/or a claim and a set of candidate passages, the model scores each passage by how well it can be
cited as evidence. It keeps the official Qwen3-Reranker prompt format with a fixed instruction (see
example_inference.py); use the same instruction, or scores will shift.
relevance is meant for ordering passages and for abstention with a threshold. The threshold used in the evaluation,
τ = −1.6875, was chosen on the dev split by F1; re-tune it on your own data.
Bundled example output (example_inference.py, one B200, bf16):
text
+3.500 cite=yes head=SUPPORTS(1.00, uncalibrated) | ... The default value is Always.-1.562 cite=yes head=CONTRADICTS(0.90, uncalibrated) | ... The default value is Never. <- failure: cited-5.688 cite=no head=IRRELEVANT(0.99, uncalibrated) | Services expose a set of Pods ...The 4-way head is an auxiliary output. Its softmax values are not calibrated probabilities: we applied no temperature scaling, and the 10-bin expected calibration error of the top class was 0.048 on QASPER test and 0.046 on CUAD test (on those sets only SUPPORTS / IRRELEVANT labels exist). Treat them as scores.
Test sets were scored once, after the pass/fail gates had been committed on the basis of dev-set baselines. All five pre-registered gates passed (QASPER +0.03 nDCG with CI above zero; CUAD CI above zero; docs Macro-F1 ≥ 0.70 and contradiction recall ≥ 0.85; docs contradiction-citation and abstention no worse than FactGuard-General-4B-v2 + 0.02; unsupported citation rate no higher than the base model in every domain). Confidence intervals are cluster bootstrap (2,000 resamples, clustered by paper / document family / contract).
| Test set | Model | R@5 | nDCG@10 | Unsupported citation | Contradicting passage cited | Abstention AUROC |
|---|---|---|---|---|---|---|
| QASPER (1,162) | Thaki-CiteRank-4B | 0.994 | 0.919 | 12.6% | – | – |
| Qwen3-Reranker-4B | 0.979 | 0.863 | 23.6% | – | – | |
| FactGuard-General-4B-v2 | 0.956 | 0.832 | 19.5% | – | – | |
| EditRank-Multidomain-4B | 0.971 | 0.858 | 23.8% | – | – | |
| BM25 | 0.868 | 0.771 | 31.4% | – | – | |
| CUAD (1,500, held out) | Thaki-CiteRank-4B | 0.978 | 0.905 | 33.1% | – | 0.880 |
| Qwen3-Reranker-4B | 0.959 | 0.838 | 50.9% | – | 0.821 | |
| EditRank-Multidomain-4B | 0.947 | 0.835 | 45.9% | – | 0.854 | |
| BM25 | 0.676 | 0.610 | 76.0% | – | 0.612 | |
| Docs (3,436, synthetic) | Thaki-CiteRank-4B | 1.000 | 0.999 | 1.5% | 1.0% | 0.999 |
| Qwen3-Reranker-4B | 1.000 | 0.994 | 67.0% | 98.3% | 0.768 | |
| FactGuard-General-4B-v2 | 0.999 | 0.994 | 20.3% | 7.9% | 0.990 |
Docs 4-way head (34,360 passages): Macro-F1 0.976 (FactGuard-General-4B-v2: 0.631), F1 SUPPORTS 0.972, PARTIAL 0.983, CONTRADICTS 0.952, IRRELEVANT 0.996, contradiction recall 0.990. Read the limitations below before using these numbers.
QASPER by answer type (nDCG@10): extractive 0.941, free-form 0.904, yes/no 0.854 (base: 0.903 / 0.838 / 0.743).
One run on one NVIDIA B200 (3.1 hours), no hyperparameter search. LoRA r 16, alpha 32, dropout 0.05, lr 1e-4,
2 epochs (the first epoch was selected on dev), gradient accumulation 8, max length 1,024. Loss: pointwise BCE on the yes/no
score (positive weight 4) + listwise cross-entropy over each candidate group + 0.5 × class-weighted cross-entropy
for the 4-way head. Training data: QASPER train (2,069 questions) and docs train (10,144 groups), with BM25 hard
negatives; at most 10 candidates per group. CUAD was not used for training or model selection.
Full settings and per-split numbers: eval_manifest.json.
| Used for | Source | License | Notes |
|---|---|---|---|
| Training, dev, test | QASPER v0.3 (Dasigi et al., 2021), Allen Institute for AI | CC BY 4.0 | Questions whose annotators all marked them unanswerable were removed |
| Training, dev, test (synthetic labels) | Kubernetes documentation, kubernetes/website at commit bc7f3479bc8b0af48097b155c22c8d58e8e1afe4 | CC BY 4.0 | © The Kubernetes Authors |
| Training, dev, test (synthetic labels) | Python documentation, python/cpython Doc/ at commit 97d361116f27d9a53ff735151ba26f5bcb463b74 | PSF License Version 2 (code examples: PSF-2.0 or 0BSD) | Copyright © 2001 Python Software Foundation; All Rights Reserved |
| Test only | CUAD v1 (Hendrycks et al., 2021), The Atticus Project | CC BY 4.0 | Contracts are public EDGAR filings; stratified sample of 1,000 answerable and 500 unanswerable clause questions, 1,000-character chunks |
RAGBench, ALCE and CC BY-SA sources were not used. No source text, question, claim or label is included in this repository. The documentation was split into sections, and claims, value-swapped contradictions and joined partial claims were generated from those sections as training and evaluation signal only. None of the dataset or documentation authors endorses this model.
This repository contains model weights, configuration, tokenizer files, cls_head.safetensors, this card,
eval_manifest.json, example_inference.py and the Apache-2.0 license text. The license field covers this
repository's contents and follows the base model; it is not a legal determination about the upstream texts.
bibtex
@inproceedings{dasigi2021qasper, title={A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers}, author={Dasigi, Pradeep and Lo, Kyle and Beltagy, Iz and Cohan, Arman and Smith, Noah A. and Gardner, Matt}, booktitle={NAACL-HLT}, year={2021}}@article{hendrycks2021cuad, title={{CUAD}: An Expert-Annotated {NLP} Dataset for Legal Contract Review}, author={Hendrycks, Dan and Burns, Collin and Chen, Anya and Ball, Spencer}, journal={NeurIPS Datasets and Benchmarks}, year={2021}}python
from example_inference import load, score # file in this repositorymodel, tok, head = load("ThakiCloud/Thaki-CiteRank-4B")relevance, head_scores = score(model, tok, head, question="What is the default restart policy of a Pod?", claim="The default restartPolicy of a Pod is Always.", docs=["...passage 1...", "...passage 2..."])Before publication, every file was downloaded from the private Hub revision 59dcb630 in a separate process with an
empty cache (all 11 files matched the uploaded package by sha256) and checked offline on one NVIDIA B200 against the
original training checkpoint and the frozen scores behind the pre-registered verdict. The verdict itself was not
recomputed and no new evaluation was run. Every check passed:
| Check | Result |
|---|---|
sha256 of the 6 model files equals the original merged checkpoint (model.safetensors da79228c…) | pass |
All tensors bitwise equal and finite; reload with transformers without missing or unexpected keys; cls_head.safetensors bitwise equal to the training head, shapes (4, 2560) and (4,) | pass |
example_inference.py scoring on a fixed sample of 64 test groups each from QASPER, docs and CUAD vs the frozen evaluation scores | top-1 192/192 identical, head argmax 1,904/1,904 identical, max difference 0.0 |
example_inference.py runs | pass |
Base shards sha256 equal Qwen/Qwen3-Reranker-4B@22e68366 | pass |
+3.500 cite=yes head=SUPPORTS(1.00, uncalibrated) | ... The default value is Always.-1.562 cite=yes head=CONTRADICTS(0.90, uncalibrated) | ... The default value is Never. <- failure: cited-5.688 cite=no head=IRRELEVANT(0.99, uncalibrated) | Services expose a set of Pods ...@inproceedings{dasigi2021qasper, title={A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers}, author={Dasigi, Pradeep and Lo, Kyle and Beltagy, Iz and Cohan, Arman and Smith, Noah A. and Gardner, Matt}, booktitle={NAACL-HLT}, year={2021}}@article{hendrycks2021cuad, title={{CUAD}: An Expert-Annotated {NLP} Dataset for Legal Contract Review}, author={Hendrycks, Dan and Burns, Collin and Chen, Anya and Ball, Spencer}, journal={NeurIPS Datasets and Benchmarks}, year={2021}}from example_inference import load, score # file in this repositorymodel, tok, head = load("ThakiCloud/Thaki-CiteRank-4B")relevance, head_scores = score(model, tok, head, question="What is the default restart policy of a Pod?", claim="The default restartPolicy of a Pod is Always.", docs=["...passage 1...", "...passage 2..."])