ThakiCloud

EditRank-0.6B

README

License: apache-2.0

What it does

Given a search query, an original document and a revised version, the model outputs a score logit("yes") - logit("no") estimating whether the revised version would be ranked higher than the original as a source by an AI answer engine for that query, while keeping every fact of the original unchanged. Score each candidate edit and take the highest. See example_inference.py for the exact prompt format; scores are only comparable within one query.

In the bundled example (bf16, single GPU), the model scores the unchanged original +3.625, a rewrite that invents facts ("waterproof", "best boot of 2026") +3.500, and a faithful rewrite +3.125. The fact-inventing rewrite outranks the faithful one. This is the failure mode behind the 12.8% figure below: the model is not a fact checker.

Results (pre-registered, 624 independent Amazon ESCI queries)

Utility = mean rank gain of the selected candidate over the original document under the simulated engine (unsafe candidates receive a penalty below the worst candidate). Original document = 0. The independent Gemma engine was not used to create any training label.

Table with columns: Method, Fixed Qwen3.8-27B engine, Independent Gemma-4-31B engine
Method	Fixed Qwen3.8-27B engine	Independent Gemma-4-31B engine
EditRank-0.6B	0.388	0.475
Earlier run of the same model family (Run1)	0.370	0.457
Fixed best single strategy ("content improvement")	0.244	0.339
BM25	0.131	0.206
Base model `Qwen3-Reranker-0.6B` (relevance prompt)	-0.056	0.052
Random candidate	-0.518	-0.367
Oracle (best candidate)	1.193	1.034

Paired bootstrap, mean difference [95% CI]:

Table with columns: Comparison, Qwen engine, Gemma engine
Comparison	Qwen engine	Gemma engine
vs BM25	+0.257 [0.157, 0.360]	+0.269 [0.156, 0.392]
vs Run1	+0.018 [-0.033, 0.072]	+0.018 [-0.048, 0.091], not statistically significant
vs fixed best strategy	+0.144 [0.020, 0.268]	+0.135 [0.003, 0.272] (thin margin)
vs base model	+0.444 [0.338, 0.550]	+0.423 [0.307, 0.551]

Pairwise accuracy (pairs with utility gap at least 1): 0.761 / 0.765. ECE: 0.046 / 0.069. Agreement of the two engines on candidate gains: Pearson 0.60.

How to read this. The demonstrated result is a statistically supported top-1 utility advantage over BM25 on held-out retail queries, confirmed by an engine that produced no training labels. It does not show that this model is better than Run1, and it does not show that the listwise objective is better than the earlier one.

Safety and limitations

Unsafe selections: 12.8%. In 12.8% of the 624 queries the model's top-1 candidate was flagged by an automated judge as changing or adding facts (Run1 13.5%, BM25 19.7%, fixed strategy 11.2%, keeping the original 0%). The judge itself is imperfect (audit on 60 samples: flag precision 0.63, at least 6.7% of unflagged cases missed), so the true rate is uncertain in both directions.
No guarantee of factual correctness. The model ranks candidates; it does not verify them.
No proof of improved visibility in live AI search engines (ChatGPT, Perplexity, Google AI Overviews, Gemini or others). All results come from a fixed simulated environment: two fixed open-weight engines, a fixed prompt, and a fixed candidate set. They are not evidence of exposure lift in any production answer engine.
Evaluated on retail product pages only. No evidence for other domains, other languages, or longer documents (documents are clipped to 1,500 characters, the original to 600).
Engines change; rankings learned against these engines may not transfer.

Training

Base: Qwen/Qwen3-Reranker-0.6B, Hub revision e61197ed45024b0ed8a2d74b80b4d909f1255473 (model.safetensors sha256 27cd75a405b9c1b46b59abfd88aaa209e6fed2a1972cde9b70e7659537c5e65b).
Full fine-tune, 3 epochs, lr 1e-5, gradient accumulation 4, tau 0.25, bf16, epoch selected on a dev split (selected epoch 2 of 3). One B200 GPU, 547 s. Hyperparameters were fixed before the run; no loss or hyperparameter search.
Data: 768 training query groups (original document plus rewrites per query) from the retail and video-games domains of C-SEO Bench. Candidate rewrites and ranking labels were generated with an in-house deployment of an open-weight Qwen3.8-27B model (5 ranking seeds per candidate). No outputs of commercial LLM APIs were used.
Training and evaluation data are not released.

Data sources and licenses

Table with columns: Used for, Source, License as published, Notes
Used for	Source	License as published	Notes
Training (retail)	C-SEO Bench `retail` (Puerto et al., 2025), derived from the Amazon Shopping Queries Dataset (ESCI, Reddy et al., 2022)	C-SEO Bench: Apache-2.0 (Hub metadata); ESCI: Apache-2.0	Product text originates from Amazon listings
Training (video games)	C-SEO Bench `videogames`, derived from the Steam Games Dataset by FronkonGames (`FronkonGames/steam-games-dataset`)	CC BY 4.0 for the version C-SEO Bench used (C-SEO Bench was published 2025-06-05; the source carried CC BY 4.0 from 2023-12-03, commit `006aa50f6f`, until it was relabeled MIT on 2026-06-19, commit `3f7a8fb501`); C-SEO Bench itself: Apache-2.0	Game descriptions originate from Steam store listings and remain the publishers' text; attribution to FronkonGames as required by CC BY 4.0; no text redistributed here
Evaluation only	624 Amazon ESCI queries disjoint from C-SEO Bench	Apache-2.0	Not used for training or model selection
Development check only (cross-domain)	C-SEO Bench `web`, `debate`	as published	Not used for training
Excluded	C-SEO Bench `news` (non-commercial terms), `books`; published C-SEO results produced with commercial APIs		Not used

The underlying product and game descriptions were written by third parties (sellers, publishers). This repository contains only model weights, configuration, tokenizer files, this card and an example script; it contains no training or evaluation text. The license field (Apache-2.0) covers this repository's contents and follows the base model; it is not a legal determination about the upstream text.

Citations:

bibtex

@inproceedings{puerto2025cseo,  title={C-{SEO} Bench: Does Conversational {SEO} Work?},  author={Puerto, Haritz and Gubri, Martin and Green, Tommaso and Oh, Seong Joon and Yun, Sangdoo},  booktitle={NeurIPS 2025 Datasets and Benchmarks Track},  year={2025}}@article{reddy2022shopping,  title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},  author={Reddy, Chandan K. and M{\`a}rquez, Llu{\'i}s and Valero, Fran and Rao, Nikhil and Zaragoza, Hugo and          Bandyopadhyay, Sambaran and Biswas, Arnab and Xing, Anlu and Subbian, Karthik},  journal={arXiv preprint arXiv:2206.06588},  year={2022}}

Steam Games Dataset: FronkonGames, Hugging Face FronkonGames/steam-games-dataset, doi:10.57967/hf/0511.

Reproducibility

The pre-registered evaluation was re-run from scratch on a fresh B200 pod with sha256-verified weights and inputs; all 76 reported metrics and all per-query utilities were bit-identical. The files in this repository were re-downloaded in a separate process and cache before publication and re-scored on the same 624 queries; see "Release verification" below and eval_manifest.json.

Release verification

Before publication, the weights in this repository were downloaded from a private Hub copy in a separate process with an empty cache and checked offline on one B200 GPU against the original training checkpoint. Every pre-registered criterion passed, both for the private copy and again for an anonymously downloaded public copy:

Table with columns: Check, Result
Check	Result
sha256 of the 6 model files equals the original checkpoint	pass
All 310 tensors bitwise equal; reload with `transformers`	pass
Top-1 selections on the 624 evaluation queries, `example_inference.py` scoring vs original code	624/624 identical, max score difference 0.0
Pre-registered evaluation re-run with these files	76/76 metrics identical, per-query utilities identical, unsafe top-1 12.8%
`example_inference.py` runs	pass
Base model sha256 equals `Qwen/Qwen3-Reranker-0.6B@e61197ed`	pass

The model files here (weights, config, tokenizer, chat template) are byte-identical to the verified ones; their sha256 values are listed in eval_manifest.json. Only the repository id string in the example and this card's text differ.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ThakiCloud

Model Tree

Base

Qwen/Qwen3-Reranker-0.6B

Fine-tuned

this model

Input Modalities

Text

Output Modalities