adgomant

adele-judge-qwen3-14B-cre

Intended Use

Use this model to score ADeLe-style examples where a question, reference answer, and model response are available. It is intended for out-of-model evaluation within the ADeLe benchmark suite, not as a general-purpose evaluator.

Input Format

The recommended helper accepts:

question
reference_answer or ground_truth
model_response

Score Rubric

Allowed scores: 1, 2, 3, 4, 5

1: surely incorrect
2: likely incorrect
3: minimally correct or sufficient
4: likely correct
5: surely correct

Binary label: scores greater than or equal to 3 are CORRECT; lower scores are INCORRECT.

Training And Validation Data

Table with columns: Split, Examples, Models
Split	Examples	Models
train	239,420	16
validation	45,738	3

train models: DK-R1-Dist-Qwen-1.5B, DK-R1-Dist-Qwen-32B, DK-R1-Dist-Qwen-7B, gemini-2.5-flash, gemini-3.1-pro, gpt-35-turbo, gpt-5.2, gpt4o, llama3d1-405b, llama3d2-11b, llama3d2-1b, llama3d2-90b, llama4-17B-128E, o1-mini, o1_re=low, o3-mini

Data Quality And Label Construction

Training labels are distilled from two proprietary judge scores used by the ADeLe evaluation pipeline to derive the official correctness signal. The configured source columns are score_gpt4o and score_sonnet.

Ordinal target: floor(mean(score_gpt4o, score_sonnet)).
Binary target: CORRECT when the ordinal target is >= 3.
Judge-agreement filter: keep examples with abs(score_gpt4o - score_sonnet) <= 1.
Response-length filter: keep responses with at most 4096 base-tokenizer tokens before prompt formatting.
Sequence-length filter: keep full chat-formatted examples within max_seq_length=8192.

Validation Results

Source artifact: validation_trainer_metrics.json.

Table with columns: Metric, Value
Metric	Value
Epoch	1.0000
Binary accuracy	0.9894
Binary macro F1	0.9880
Precision, CORRECT	0.9932
Recall, CORRECT	0.9909
Precision, INCORRECT	0.9817
Recall, INCORRECT	0.9863
False negative rate, CORRECT	0.0091
False positive rate, CORRECT	0.0137

Recommended Inference

Do not use free-form generation as the primary prediction method. The recommended path scores the restricted continuations "1", "2", "3", "4", and "5".

python
from transformers import pipeline

judge = pipeline(
    "adele-judge",
    model="adgomant/adele-judge-qwen3-14-cre",
    trust_remote_code=True,
    device_map="auto",
)
result = judge(
    {"question": "...", "reference_answer": "...", "model_response": "..."}
)
print(result)

results = judge([
    {"question": "...", "reference_answer": "...", "model_response": "..."},
    {"question": "...", "ground_truth": "...", "model_response": "..."},
], batch_size=8)

The result has this shape:

python
{
    "score": 4,
    "label": "CORRECT",
    "probs": {"1": 0.01, "2": 0.02, "3": 0.08, "4": 0.70, "5": 0.19},
    "logprobs": {"1": -5.0, "2": -4.2, "3": -2.9, "4": -0.8, "5": -2.1},
    "confidence": 0.70,
    "margin": 1.3,
    "entropy": 0.82,
}

Standard Transformers Loading

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)

generation_config.json uses safe one-token defaults for debugging, but generate() is not the recommended scoring method.

Metadata

Training, filtering, split, tokenization, and metric artifacts available at packaging time are stored in adele_judge_metadata.json.

The model is trained on distilled judge targets. These targets are useful for reproducing the ADeLe paper-style correctness signal at lower inference cost, but they should not be interpreted as independent human annotations.

References

ADeLe project page: ADeLe v1.0.
ADeLe paper and official correctness definition: General scales unlock AI evaluation with explanatory and predictive power.
Official ADeLe dataset: CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0.
Official instance-level model-response data used for distillation: https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data.
Training and Hub packaging implementation: https://github.com/adgomant/adele-judge.

Limitations

ADeLe-specific judge; not a general-purpose evaluator.
Distilled from proprietary judge labels and inherits their noise, calibration, and biases.
Intended for scoring responses against a reference answer.
It should not produce explanations; the expected output is a single score.
Validation is out-of-model within the ADeLe suite, so transfer outside that suite should be measured before relying on it.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

adgomant

Model Tree

Base

Qwen/Qwen3-14B

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Intended Use

Input Format

The recommended helper accepts:

question
reference_answer or ground_truth
model_response

Score Rubric

Allowed scores: 1, 2, 3, 4, 5

1: surely incorrect
2: likely incorrect
3: minimally correct or sufficient
4: likely correct
5: surely correct

Binary label: scores greater than or equal to 3 are CORRECT; lower scores are INCORRECT.

Training And Validation Data

Table with columns: Split, Examples, Models
Split	Examples	Models
train	239,420	16
validation	45,738	3

train models: DK-R1-Dist-Qwen-1.5B, DK-R1-Dist-Qwen-32B, DK-R1-Dist-Qwen-7B, gemini-2.5-flash, gemini-3.1-pro, gpt-35-turbo, gpt-5.2, gpt4o, llama3d1-405b, llama3d2-11b, llama3d2-1b, llama3d2-90b, llama4-17B-128E, o1-mini, o1_re=low, o3-mini

Data Quality And Label Construction

Ordinal target: floor(mean(score_gpt4o, score_sonnet)).
Binary target: CORRECT when the ordinal target is >= 3.
Judge-agreement filter: keep examples with abs(score_gpt4o - score_sonnet) <= 1.
Response-length filter: keep responses with at most 4096 base-tokenizer tokens before prompt formatting.
Sequence-length filter: keep full chat-formatted examples within max_seq_length=8192.

Validation Results

Source artifact: validation_trainer_metrics.json.

Table with columns: Metric, Value
Metric	Value
Epoch	1.0000
Binary accuracy	0.9894
Binary macro F1	0.9880
Precision, CORRECT	0.9932
Recall, CORRECT	0.9909
Precision, INCORRECT	0.9817
Recall, INCORRECT	0.9863
False negative rate, CORRECT	0.0091
False positive rate, CORRECT	0.0137

Recommended Inference

Do not use free-form generation as the primary prediction method. The recommended path scores the restricted continuations "1", "2", "3", "4", and "5".

python
from transformers import pipeline

judge = pipeline(
    "adele-judge",
    model="adgomant/adele-judge-qwen3-14-cre",
    trust_remote_code=True,
    device_map="auto",
)
result = judge(
    {"question": "...", "reference_answer": "...", "model_response": "..."}
)
print(result)

results = judge([
    {"question": "...", "reference_answer": "...", "model_response": "..."},
    {"question": "...", "ground_truth": "...", "model_response": "..."},
], batch_size=8)

The result has this shape:

python
{
    "score": 4,
    "label": "CORRECT",
    "probs": {"1": 0.01, "2": 0.02, "3": 0.08, "4": 0.70, "5": 0.19},
    "logprobs": {"1": -5.0, "2": -4.2, "3": -2.9, "4": -0.8, "5": -2.1},
    "confidence": 0.70,
    "margin": 1.3,
    "entropy": 0.82,
}

Standard Transformers Loading

python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)

generation_config.json uses safe one-token defaults for debugging, but generate() is not the recommended scoring method.

Metadata

Training, filtering, split, tokenization, and metric artifacts available at packaging time are stored in adele_judge_metadata.json.

References

ADeLe project page: ADeLe v1.0.
ADeLe paper and official correctness definition: General scales unlock AI evaluation with explanatory and predictive power.
Official ADeLe dataset: CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0.
Official instance-level model-response data used for distillation: https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data.
Training and Hub packaging implementation: https://github.com/adgomant/adele-judge.

Limitations

ADeLe-specific judge; not a general-purpose evaluator.
Distilled from proprietary judge labels and inherits their noise, calibration, and biases.
Intended for scoring responses against a reference answer.
It should not produce explanations; the expected output is a single score.
Validation is out-of-model within the ADeLe suite, so transfer outside that suite should be measured before relying on it.

adele-judge-qwen3-14B-cre

README

Intended Use

Input Format

Score Rubric

Training And Validation Data

Data Quality And Label Construction

Validation Results

Recommended Inference

Standard Transformers Loading

Metadata

References

Limitations

Explore FriendliAI today

README

Intended Use

Input Format

Score Rubric

Training And Validation Data

Data Quality And Label Construction

Validation Results

Recommended Inference

Standard Transformers Loading

Metadata

References

Limitations