extract-prompt-quality-criteria API & Inference Endpoint

Performance

Evaluated on a 75-example held-out test set:

Table with columns: Metric, Score
Metric	Score
Overall MAE	0.51
Accuracy (within 1 point)	82.1%
Accuracy (within 2 points)	91.0%
Parse success rate	89.3%

Inference speed: ~3.1s on Apple Silicon (MPS), ~6.2s on CPU.

Quick Start

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import json

model_repo = "center-of-excellence/extract-prompt-quality-criteria"
base_model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

# Auto-detect best device
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
print(f"Using device: {device}")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Load model — merge LoRA weights for faster inference
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    dtype=torch.float16 if device in ("cuda", "mps") else torch.float32,
    trust_remote_code=True,
    low_cpu_mem_usage=True
)
model = PeftModel.from_pretrained(base_model, model_repo)
model = model.merge_and_unload()  # merge for faster inference
model = model.to(device)
model.eval()

# Analyze a prompt
prompt = "Classify text into categories"

instruction = """You are a prompt quality analyzer. Analyze the given prompt and extract quality scores for each criterion.

Criteria (score 1-10):
1. clarity_and_specificity: Are instructions clear and specific?
2. context_sufficiency: Is sufficient background/context provided?
3. examples_provided: Are concrete examples included?
4. output_format_specification: Is the expected output format clearly defined?
5. edge_case_handling: Are edge cases and exceptions addressed?
6. tone_and_style_guidance: Is communication style/tone specified?
7. constraint_definition: Are limits and boundaries clearly set?
8. relevance_of_examples: Do examples match the domain/task?

Respond with ONLY a JSON object containing the scores."""

formatted = f"""<|system|>
{instruction}<|end|>
<|user|>
Analyze this prompt:

{prompt}<|end|>
<|assistant|>
"""

inputs = tokenizer(formatted, return_tensors="pt", truncation=True, max_length=512).to(device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=300,
        temperature=0.3,
        do_sample=True,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id
    )

input_length = inputs['input_ids'].shape[1]
response = tokenizer.decode(outputs[0][input_length:], skip_special_tokens=True)

# Parse JSON
start = response.find('{')
end = response.rfind('}') + 1
result = json.loads(response[start:end])
print(json.dumps(result, indent=2))

Sklearn Pipeline (Batch Inference)

python
import pandas as pd
from prompt_quality_transformer import PromptQualityTransformer

df = pd.DataFrame({'prompt': ['Analyze the text.', 'Classify sentiment. Return JSON.']})

transformer = PromptQualityTransformer(
    model_repo="center-of-excellence/extract-prompt-quality-criteria",
    batch_size=8,
    use_quantization=True,   # int8 for faster GPU inference
    verbose=True
)
df_scored = transformer.fit_transform(df)
print(df_scored[['prompt', 'overall_score', 'quality_tier']])

What It Scores

Table with columns: Criterion, Description
Criterion	Description
Clarity & Specificity	Are instructions clear and unambiguous?
Context Sufficiency	Is enough background provided?
Examples Provided	Are there concrete examples?
Output Format	Is the expected format specified?
Edge Case Handling	Are exceptions addressed?
Tone & Style	Is communication style defined?
Constraint Definition	Are limits clearly set?

Example Output

json
{
  "scores": {
    "clarity_and_specificity": 8,
    "context_sufficiency": 7,
    "examples_provided": 6,
    "output_format_specification": 9,
    "edge_case_handling": 5,
    "tone_and_style_guidance": 6,
    "constraint_definition": 7,
    "relevance_of_examples": 8
  },
  "overall_score": 7.0,
  "quality_tier": "good"
}

Model Details

Base Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 (1.1B parameters)
Method: LoRA fine-tuning via PEFT
LoRA Rank: 8, Alpha: 16
Training Data: ~500 synthetic prompt examples with deterministic quality labels
Training Time: ~85 minutes on T4 GPU
Chat Template: <|system|>, <|user|>, <|assistant|> format

Limitations

Optimized for English prompts
Works best with prompts under 500 tokens
edge_case_handling is the weakest criterion (MAE 1.33) — accuracy improves when prompts explicitly mention edge cases
Small model — accuracy improves with larger base models or real training data from git-ai

Citation

bibtex
@misc{extract-prompt-quality-criteria,
  author = {Gregoire Cattan and Elif Uzun},
  title = {Extract Prompt Quality Criteria},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/center-of-excellence/extract-prompt-quality-criteria}
}

Metric

Score

Overall MAE

0.51

Accuracy (within 1 point)

82.1%

Accuracy (within 2 points)

91.0%

Parse success rate

89.3%

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import json

model_repo = "center-of-excellence/extract-prompt-quality-criteria"
base_model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

# Auto-detect best device
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
print(f"Using device: {device}")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Load model — merge LoRA weights for faster inference
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    dtype=torch.float16 if device in ("cuda", "mps") else torch.float32,
    trust_remote_code=True,
    low_cpu_mem_usage=True
)
model = PeftModel.from_pretrained(base_model, model_repo)
model = model.merge_and_unload()  # merge for faster inference
model = model.to(device)
model.eval()

# Analyze a prompt
prompt = "Classify text into categories"

instruction = """You are a prompt quality analyzer. Analyze the given prompt and extract quality scores for each criterion.

Criteria (score 1-10):
1. clarity_and_specificity: Are instructions clear and specific?
2. context_sufficiency: Is sufficient background/context provided?
3. examples_provided: Are concrete examples included?
4. output_format_specification: Is the expected output format clearly defined?
5. edge_case_handling: Are edge cases and exceptions addressed?
6. tone_and_style_guidance: Is communication style/tone specified?
7. constraint_definition: Are limits and boundaries clearly set?
8. relevance_of_examples: Do examples match the domain/task?

Respond with ONLY a JSON object containing the scores."""

formatted = f"""<|system|>
{instruction}<|end|>
<|user|>
Analyze this prompt:

{prompt}<|end|>
<|assistant|>
"""

inputs = tokenizer(formatted, return_tensors="pt", truncation=True, max_length=512).to(device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=300,
        temperature=0.3,
        do_sample=True,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id
    )

input_length = inputs['input_ids'].shape[1]
response = tokenizer.decode(outputs[0][input_length:], skip_special_tokens=True)

# Parse JSON
start = response.find('{')
end = response.rfind('}') + 1
result = json.loads(response[start:end])
print(json.dumps(result, indent=2))

python

import pandas as pd
from prompt_quality_transformer import PromptQualityTransformer

df = pd.DataFrame({'prompt': ['Analyze the text.', 'Classify sentiment. Return JSON.']})

transformer = PromptQualityTransformer(
    model_repo="center-of-excellence/extract-prompt-quality-criteria",
    batch_size=8,
    use_quantization=True,   # int8 for faster GPU inference
    verbose=True
)
df_scored = transformer.fit_transform(df)
print(df_scored[['prompt', 'overall_score', 'quality_tier']])

Criterion

Description

Clarity & Specificity

Are instructions clear and unambiguous?

Context Sufficiency

Is enough background provided?

Examples Provided

Are there concrete examples?

Output Format

Is the expected format specified?

Edge Case Handling

Are exceptions addressed?

Tone & Style

Is communication style defined?

Constraint Definition

Are limits clearly set?

json

{
  "scores": {
    "clarity_and_specificity": 8,
    "context_sufficiency": 7,
    "examples_provided": 6,
    "output_format_specification": 9,
    "edge_case_handling": 5,
    "tone_and_style_guidance": 6,
    "constraint_definition": 7,
    "relevance_of_examples": 8
  },
  "overall_score": 7.0,
  "quality_tier": "good"
}

bibtex

@misc{extract-prompt-quality-criteria,
  author = {Gregoire Cattan and Elif Uzun},
  title = {Extract Prompt Quality Criteria},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/center-of-excellence/extract-prompt-quality-criteria}
}

extract-prompt-quality-criteria

README

Performance

Quick Start

Sklearn Pipeline (Batch Inference)

What It Scores

Example Output

Model Details

Limitations

Citation

Explore FriendliAI today

README

Performance

Quick Start

Sklearn Pipeline (Batch Inference)

What It Scores

Example Output

Model Details

Limitations

Citation