AdarshSingh7647

forge-CoTCond

README

License: apache-2.0

Prompt / output format

System prompt used at eval time:

You are a table relevance expert. Given a question and a set of candidate tables, rank them from most to least useful for answering the question. Reason in a <think>...</think> block, then output exactly JSON with key ranked_tables.

The user message lists candidate tables as ### Table 1, ### Table 2, … The model outputs (optionally after a <think> block) a JSON object:

json
{"ranked_tables": [3, 1, 5, 2, 4, ...]}

where the integers are the 1-based candidate positions in the prompt, best first. Map them back to your table ids to obtain the reranked list.

Evaluation

Evaluated as a single-call listwise reranker (think mode) reordering the first-stage top-25 candidate list, on 4 out-of-domain table-QA datasets. SQA / TAT-QA use the full test set; HybridQA / TabFact use a fixed shared 500-query sample.

Table with columns: Dataset, n, recall@10, ndcg@10, acc@10, MRR
Dataset	n	recall@10	ndcg@10	acc@10	MRR
SQA	148	0.918	0.741	0.865	0.687
TAT-QA	362	0.671	0.519	0.486	0.563
HybridQA	500	0.916	0.783	0.848	0.788
TabFact	500	0.776	0.688	0.628

acc@k = all gold tables within the top-k (strict). Metrics are identical in definition to the TabRank evaluation harness.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "forge-CoTCond_merged"  # or your uploaded repo id
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

SYSTEM = ("You are a table relevance expert. Given a question and a set of candidate "
          "tables, rank them from most to least useful for answering the question. "
          "Reason in a <think>...</think> block, then output exactly JSON with key ranked_tables.")
user = "Question: ...\n\n### Table 1\n...\n\n### Table 2\n...\n"  # your listwise prompt

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device),
                     max_new_tokens=8192, temperature=0.6, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

For high-throughput reranking use vLLM (loads directly, no LoRA flags).

Notes

dtype: bfloat16. Single-file model.safetensors (~16 GB).
The bundled chat_template.jinja / tokenizer are the ones used at train time.
Sibling model: TabRank-CoTCond (the in-repo CoTCond checkpoint) trained on NQ-Tables only — see its card for a head-to-head.

Citation

This model is trained in part on the MultiTabQA data from RAG over Tables. Please cite:

bibtex
@misc{zou2025ragtableshierarchicalmemory,
      title={RAG over Tables: Hierarchical Memory Index, Multi-Stage Retrieval, and Benchmarking},
      author={Jiaru Zou and Dongqi Fu and Sirui Chen and Xinrui He and Zihao Li and Yada Zhu and Jiawei Han and Jingrui He},
      year={2025},
      eprint={2504.01346},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.01346},
}

The accompanying TabRank paper is currently under review; its citation will be added here once available on arXiv.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

AdarshSingh7647

Model Tree

Base

Qwen/Qwen3-8B

Fine-tuned

this model

Input Modalities

Text

Output Modalities