qwen3.5-9b-nli-10k API & Inference Endpoint

Training

Data: 2,000 random pairs per dataset from their training splits (seed 0); FEVER and VitaminC: SUPPORTS = entailment, REFUTES = contradiction, NOT ENOUGH INFO = neutral, with the evidence as premise and the claim as hypothesis.
1 epoch, learning rate 1e-4 (linear schedule, 5% warmup), batch size 8, AdamW. LoRA r = 16, alpha = 32, dropout 0.05 on q/k/v/o/gate/up/down projections. Premises truncated to 256 tokens.
Loss on the answer tokens only.

Prompt and labels

Chat template (enable_thinking=False), one user message:

text

Premise: {premise}Hypothesis: {hypothesis}Does the premise entail the hypothesis? Answer with one word: entailment, neutral or contradiction.

The answer is one of entailment, neutral, contradiction. Scores: softmax over the logits of the three labels' first tokens at the first answer position.

Results

1,000 random examples per test set; accuracy / AUROC of P(entailment) for entailment vs not.

Table with columns: Dataset, Accuracy, AUROC
Dataset	Accuracy	AUROC
SNLI test	0.926	0.986
MNLI validation matched	0.906	0.988
ANLI test r1-r3	0.664	0.867
NLI-FEVER dev	0.749	0.925
VitaminC test	0.870	0.968

Training

Data: 2,000 random pairs per dataset from their training splits (seed 0); FEVER and VitaminC: SUPPORTS = entailment, REFUTES = contradiction, NOT ENOUGH INFO = neutral, with the evidence as premise and the claim as hypothesis.
1 epoch, learning rate 1e-4 (linear schedule, 5% warmup), batch size 8, AdamW. LoRA r = 16, alpha = 32, dropout 0.05 on q/k/v/o/gate/up/down projections. Premises truncated to 256 tokens.
Loss on the answer tokens only.

Prompt and labels

Chat template (enable_thinking=False), one user message:

text

Premise: {premise}Hypothesis: {hypothesis}Does the premise entail the hypothesis? Answer with one word: entailment, neutral or contradiction.

The answer is one of entailment, neutral, contradiction. Scores: softmax over the logits of the three labels' first tokens at the first answer position.

Results

1,000 random examples per test set; accuracy / AUROC of P(entailment) for entailment vs not.

Table with columns: Dataset, Accuracy, AUROC
Dataset	Accuracy	AUROC
SNLI test	0.926	0.986
MNLI validation matched	0.906	0.988
ANLI test r1-r3	0.664	0.867
NLI-FEVER dev	0.749	0.925
VitaminC test	0.870	0.968

qwen3.5-9b-nli-10k

README

Training

Prompt and labels

Results

Explore FriendliAI today

README

Training

Prompt and labels

Results