tau-emotion-bench

Tracer-High-8B

README

License: apache-2.0

Highlights

Trained on ten customer-service domains with emotionally expressive user behaviour.
Reasons in a <think> block before every reply or tool call; tool calls follow the Hermes convention.

Performance

Results on the ten domains of τ-Emotion-Bench under two user simulators. Scores are pass^k in %; the better of Qwen3-8B and Tracer-High-8B on each metric is in bold.

User Simulator: Qwen3-235B-A22B-Instruct-2507

Table with columns: Domain, Qwen3-8B pass^1, Tracer-High-8B pass^1, Qwen3-8B pass^3, Tracer-High-8B pass^3
Domain	Qwen3-8B pass^1	Tracer-High-8B pass^1	Qwen3-8B pass^3	Tracer-High-8B pass^3
Airline	20.0	44.8	4.3	27.1
Retail	37.1	56.7	14.3	31.4
Telecom	41.4	37.6	12.9	15.7
Telehealth	46.2	53.8	22.9	31.4
Bank	24.7	46.7	12.0	22.0
Restaurant	16.7	24.7	4.0	12.0
Movie	3.3	6.7	0.0	2.0
School	24.7	56.0	0.0	38.0
Travel	2.7	14.0	2.0	2.0
Basketball	26.7	40.0	12.0	16.0
Average	24.3	38.1	8.4	19.8

User Simulator: GLM-5.2

Table with columns: Domain, Qwen3-8B pass^1, Tracer-High-8B pass^1, Qwen3-8B pass^3, Tracer-High-8B pass^3
Domain	Qwen3-8B pass^1	Tracer-High-8B pass^1	Qwen3-8B pass^3	Tracer-High-8B pass^3
Airline	30.5	57.6	18.6	44.3
Retail	64.3	80.0	42.9	64.3
Telecom	72.9	73.3	61.4	57.1
Telehealth	63.3	83.8	42.9	71.4
Bank	34.7	54.0	20.0	36.0
Restaurant	27.3	40.0	10.0	18.0
Movie	6.7	20.0	2.0	8.0
School	33.3	58.7	14.0	44.0
Travel	12.7	30.7	2.0	4.0
Basketball	30.0	48.7	6.0	28.0
Average	37.6	54.7	22.0	37.5

Training

Method

Data: Multi-turn tool-use dialogues with emotionally expressive users across ten domains (Retail, Telehealth, Telecom, Airline, Bank, Basketball, Movie, Restaurant, School, Travel), released as Tracer-High-tau-emotion-data.
Supervised Fine-Tuning (SFT): The base model is fine-tuned on turn-level samples in which the target assistant turn contains an inline <think> block followed by the reply or tool call.

Infrastructure

Training was run on 4× NVIDIA H100 NVL with FSDP/DDP data parallelism, FlashAttention-3, and gradient checkpointing.

Hyperparameters

Table with columns: SFT
	SFT
Effective Batch Size	32
Learning Rate	5e-5
LR Schedule	cosine, 3% warmup
Optimizer	paged_adamw_8bit
Epochs / Steps	3 epochs / 3,042 steps
Max Context Length	32,768
Precision	bf16

Training Data

Table with columns: Dataset, Description, Samples
Dataset	Description	Samples
Tracer-High-tau-emotion-data	SFT training data (ten domains)	32,740

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "tau-emotion-bench/Tracer-High-8B"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto") tools = [{    "type": "function",    "function": {        "name": "get_reservation_details",        "description": "Get the details of a reservation.",        "parameters": {"type": "object",                       "properties": {"reservation_id": {"type": "string"}},                       "required": ["reservation_id"]},    },}]messages = [    {"role": "system", "content": "<domain policy>"},    {"role": "user", "content": "I need to cancel my reservation M9OL3W."},]inputs = tok.apply_chat_template(messages, tools=tools, add_generation_prompt=True,                                 return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**inputs, max_new_tokens=1024)print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Serving with vLLM:

bash

vllm serve tau-emotion-bench/Tracer-High-8B \  --dtype bfloat16 --max-model-len 32768 \  --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3

Keep thinking on and use the model's sampling defaults in the request body:

json

{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0, "chat_template_kwargs": {"enable_thinking": true}}

Citation

bibtex

@article{singh2026tauemotionbench,  title={$\tau$-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions},  author={Singh, Rishitosh Kumar and Saeidi, Amir and Hegde, Shamanthak and Kumbhar, Shrinidhi and Baral, Chitta},  journal={arXiv preprint arXiv:2610.19560},  year={2026},  url={https://arxiv.org/abs/2610.19560}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

tau-emotion-bench

Model Tree

Base

Qwen/Qwen3-8B

Fine-tuned

this model

Input Modalities

Text

Output Modalities