MultiSense

CustomerLM

Deploy Dedicated

README

License: apache-2.0

Intended Use

Use it for:

Driving the customer side of multi-turn sales dialogue simulation, so you can benchmark or stress-test a sales agent.
Generating synthetic sales conversations for training or analysis.
Any role-play evaluation harness that needs a non-compliant, in-character human counterpart.

Do not use it for:

Serving end users as an assistant. It is trained to behave like a customer — it will ask questions, push back, and refuse rather than help.
Producing factual claims about real products. It speaks from a persona, not a knowledge base.
Representing real people. Personas are synthetic and are not models of any individual.

How It Works

CustomerLM takes the customer persona as its system prompt and the salesperson's turns as user messages. Its own replies are the customer's turns. Note the role mapping — it is inverted relative to a normal assistant deployment:

Table with columns: Chat role, Who it is
Chat role	Who it is
`system`	The customer persona (difficulty, buy-inclination, pain points, decision factors)
`user`	The salesperson speaking to the customer
`assistant`	CustomerLM — the customer's reply

System prompt format

CustomerLM was fine-tuned on this layout. Keeping it maximises fidelity; deviating from it degrades persona adherence.

markdown
- Difficulty level: <easy | medium | hard | very_hard>
- Buy-inclination score: <0.0 - 1.0>
- Persona: <one-paragraph description of the buyer's stance>
CUSTOMER_INFORMATION (private):
Basic information
{"age_group": "...", "gender": "...", "location": "...", "occupation": "..."}
Motivation
<what they are trying to achieve>
Pain points
<what worries them>
Decision factors
<what makes them say yes>
Communication preference
<channels and tone they like>
Language
<Chinese | English>

Difficulty level and Buy-inclination score are the difficulty knob and should move together: easy ≈ 0.8–1.0, medium ≈ 0.5–0.7, hard ≈ 0.2–0.4, very_hard ≈ 0.0–0.1.

Usage

Quick start (transformers)

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MultiSense/CustomerLM"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

persona = """- Difficulty level: hard
- Buy-inclination score: 0.3
- Persona: Skeptical, price-sensitive buyer who needs concrete evidence before committing.
CUSTOMER_INFORMATION (private):
Basic information
{"age_group": "35-44", "gender": "female", "location": "Boston", "occupation": "software engineer"}
Motivation
Wants a quieter commute and better focus while working from cafes.
Pain points
Burned by cheap headphones before; suspicious of marketing claims.
Decision factors
Measured noise-cancellation performance, comfort, warranty, price under $300.
Communication preference
Direct, fact-dense answers; concrete numbers over adjectives.
Language
English"""

messages = [
    {"role": "system", "content": persona},
    # the SALESPERSON speaks as `user`
    {"role": "user", "content": "Hi! Are you looking for noise-cancelling headphones today?"},
]

inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, temperature=0.8, top_p=0.99, do_sample=True)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Serving with vLLM

The SalesLLM harness talks to CustomerLM over an OpenAI-compatible endpoint, so vLLM is the recommended way to run it:

bash
vllm serve MultiSense/CustomerLM \
  --served-model-name CustomerLM \
  --port 8000

Use as the SalesLLM user simulator

Point the benchmark's --user_* flags at your vLLM server. The harness handles the role mapping, the persona injection, and the turn loop for you:

bash
python salesllm/salesllm_evaluation.py \
  --assistant_model_name "MODEL_UNDER_TEST" \
  --assistant_API_end_point "https://YOUR_ASSISTANT_ENDPOINT" \
  --assistant_API_key "YOUR_KEY" \
  --user_model_name "CustomerLM" \
  --user_API_end_point "http://localhost:8000/v1" \
  --user_API_key "EMPTY" \
  --user_api_type "other" \
  --execution_mode "concurrent" \
  --round_num 20 \
  --input_file "data/benchmark/conversations_1000_zh.jsonl" \
  --output_dir "./results/zh/" \
  --language "zh"

Recommended decoding

Table with columns: Parameter, Value
Parameter	Value
`temperature`	0.8
`top_p`	0.99
`max_tokens`	2048

Sampling matters here. Greedy decoding makes the customer flat and repetitive, which suppresses the natural objection variety the model was trained to produce.

Training

CustomerLM training pipeline

CustomerLM is trained in two stages on 8,284 crowdworker-involved real-world sales dialogues.

Stage 1 — SFT (learn to talk like a customer)

Real sales conversations are collected and pre-processed, then the base Qwen model is supervised-fine-tuned on the customer turns only, conditioned on the persona reconstructed from that conversation. This produces the SFT model, which already speaks in a customer register but still inverts roles under pressure.

Stage 2 — DPO (learn not to behave like a salesperson)

SFT alone does not remove role inversion, because the failure is behavioural rather than stylistic. So we build preference pairs that target it directly:

The SFT model is paired with a sales model to simulate dialogues from generated scripts.
An LLM judge labels each simulated customer turn as assistant-like (the failure: helpful, pitching, summarizing benefits) or user-like (the target: in-character buyer behaviour).
Assistant-like turns are sent through GPT-4o + human correction to produce the user-like counterpart of the same turn.
Each (assistant-like, corrected user-like) pair becomes a DPO pair — rejected and chosen respectively.
DPO on these pairs yields the final model.

The key design choice is that the preference signal is contrastive on the exact failure mode. Both sides of every pair respond to the same context, so the gradient isolates "stop being an assistant" from everything else the model already learned in SFT.

Evaluation

User simulation fidelity

Held-out set of human-annotated real conversations (118 ZH, 150 EN). Reference-based metrics compare the simulated customer turn against the real human customer turn; Role Inversion is the share of dialogues where the simulated customer slips into salesperson behaviour (lower is better).

Table with columns: User Model, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, Sem. Sim., Role Inversion (%)
User Model	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L	Sem. Sim.	Role Inversion (%)
GPT-4o	0.10	0.08	0.02	0.07	0.57	17.44
UserLM	0.06	0.08	0.01	0.06

CustomerLM wins on every metric. The headline result is role inversion: 8.8% vs. 17.44% for GPT-4o, roughly a 2× reduction, achieved by a much smaller model than the GPT-4o baseline it beats.

Read the n-gram scores in context — BLEU-4 of 0.12 is low in absolute terms because there is no single correct next customer turn. What matters is the relative ordering against baselines on identical references, plus semantic similarity (0.59), which is the more meaningful signal for open-ended dialogue.

Effect on downstream benchmark scores

Swapping GPT-4o for CustomerLM changes the measured selling skill of the models under test, which is the practical reason the simulator quality matters. SalesLLM Score (0–10) on 1,000 ZH / 805 EN scripts:

Table with columns: Assistant Model, ZH (GPT-4o user), ZH (CustomerLM user), EN (GPT-4o user), EN (CustomerLM user)
Assistant Model	ZH (GPT-4o user)	ZH (CustomerLM user)	EN (GPT-4o user)	EN (CustomerLM user)
Doubao-32K	6.07	6.89	6.31	5.48
Qwen-max	6.02	5.55	5.97	5.56
GPT-4o	5.72	6.15	5.53

Scores are not comparable across user models — only compare within a column. The rankings genuinely reorder between the two simulators, so a leaderboard built on a GPT-4o customer is measuring something different from one built on CustomerLM.

Reproducing these numbers

bash
# Simulation fidelity (BLEU / ROUGE / semantic similarity)
python salesllm/user_likness_eval.py \
  --model_name "CustomerLM" \
  --api_end_point "http://localhost:8000/v1" --api_key "EMPTY" --api_type "other" \
  --input_files <held_out_conversations>.jsonl \
  --output_file user_likeness_results.jsonl --language zh

# Role inversion rate (LLM judge over generated dialogues)
python salesllm/reverse_role_eval.py \
  --model_name "<judge_model>" \
  --api_end_point "<judge_endpoint>" --api_key "<judge_key>" \
  --input_files ./results/zh/*.jsonl \
  --output_dir ./reverse_role_results

Limitations and Risks

Not an assistant. Deployed as a chatbot it will act like a customer. This is by design and is not a defect to be prompted away.
Format sensitivity. Persona adherence degrades if the system prompt departs from the trained layout. Personas outside the trained distribution (B2B procurement, healthcare intake, etc.) are extrapolation.
Domain coverage. Training data is Financial Services and Consumer Goods. Other verticals are untested.
Residual role inversion. 8.8% is a large improvement, not a solution. Roughly one dialogue in eleven still exhibits the failure, so filter with reverse_role_eval.py when simulation quality is load-bearing.
Reference metrics are weak instruments. BLEU/ROUGE against a single human reference under-measure valid alternative customer responses; treat them as comparative, not absolute.
Inherited bias. Persona attributes include age, gender, location, and occupation, and the model may reproduce demographic stereotypes present in the base model and the source dialogues. Do not use generated personas to make claims about real demographic groups.
Synthetic output. Conversations produced by CustomerLM are simulations, not evidence of real customer preferences, and should not be used as market research.

Citation

bibtex
@misc{su2026sellmoreplayless,
      title={Sell More, Play Less: Benchmarking LLM Realistic Selling Skill}, 
      author={Xuanbo Su and Wenhao Hu and Le Zhan and Yuting Xie and Kailin Lyu and Kaijie Chen and Ziwei Li and Yeqiang Wang and Haibo Su and Yunzhang Chen and Ling Huang},
      year={2026},
      eprint={2604.07054},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.07054}, 
}

📖 Paper
📊 SalesLLM benchmark & code
🤗 SaleIntent-BERT — buying-intent classifier used for outcome scoring (93.51% ZH, 92.94% EN)

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

MultiSense

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Intended Use

Use it for:

Driving the customer side of multi-turn sales dialogue simulation, so you can benchmark or stress-test a sales agent.
Generating synthetic sales conversations for training or analysis.
Any role-play evaluation harness that needs a non-compliant, in-character human counterpart.

Do not use it for:

Serving end users as an assistant. It is trained to behave like a customer — it will ask questions, push back, and refuse rather than help.
Producing factual claims about real products. It speaks from a persona, not a knowledge base.
Representing real people. Personas are synthetic and are not models of any individual.

How It Works

Table with columns: Chat role, Who it is
Chat role	Who it is
`system`	The customer persona (difficulty, buy-inclination, pain points, decision factors)
`user`	The salesperson speaking to the customer
`assistant`	CustomerLM — the customer's reply

System prompt format

CustomerLM was fine-tuned on this layout. Keeping it maximises fidelity; deviating from it degrades persona adherence.

markdown
- Difficulty level: <easy | medium | hard | very_hard>
- Buy-inclination score: <0.0 - 1.0>
- Persona: <one-paragraph description of the buyer's stance>
CUSTOMER_INFORMATION (private):
Basic information
{"age_group": "...", "gender": "...", "location": "...", "occupation": "..."}
Motivation
<what they are trying to achieve>
Pain points
<what worries them>
Decision factors
<what makes them say yes>
Communication preference
<channels and tone they like>
Language
<Chinese | English>

Difficulty level and Buy-inclination score are the difficulty knob and should move together: easy ≈ 0.8–1.0, medium ≈ 0.5–0.7, hard ≈ 0.2–0.4, very_hard ≈ 0.0–0.1.

Usage

Quick start (transformers)

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MultiSense/CustomerLM"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

persona = """- Difficulty level: hard
- Buy-inclination score: 0.3
- Persona: Skeptical, price-sensitive buyer who needs concrete evidence before committing.
CUSTOMER_INFORMATION (private):
Basic information
{"age_group": "35-44", "gender": "female", "location": "Boston", "occupation": "software engineer"}
Motivation
Wants a quieter commute and better focus while working from cafes.
Pain points
Burned by cheap headphones before; suspicious of marketing claims.
Decision factors
Measured noise-cancellation performance, comfort, warranty, price under $300.
Communication preference
Direct, fact-dense answers; concrete numbers over adjectives.
Language
English"""

messages = [
    {"role": "system", "content": persona},
    # the SALESPERSON speaks as `user`
    {"role": "user", "content": "Hi! Are you looking for noise-cancelling headphones today?"},
]

inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, temperature=0.8, top_p=0.99, do_sample=True)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Serving with vLLM

The SalesLLM harness talks to CustomerLM over an OpenAI-compatible endpoint, so vLLM is the recommended way to run it:

bash
vllm serve MultiSense/CustomerLM \
  --served-model-name CustomerLM \
  --port 8000

Use as the SalesLLM user simulator

Point the benchmark's --user_* flags at your vLLM server. The harness handles the role mapping, the persona injection, and the turn loop for you:

bash
python salesllm/salesllm_evaluation.py \
  --assistant_model_name "MODEL_UNDER_TEST" \
  --assistant_API_end_point "https://YOUR_ASSISTANT_ENDPOINT" \
  --assistant_API_key "YOUR_KEY" \
  --user_model_name "CustomerLM" \
  --user_API_end_point "http://localhost:8000/v1" \
  --user_API_key "EMPTY" \
  --user_api_type "other" \
  --execution_mode "concurrent" \
  --round_num 20 \
  --input_file "data/benchmark/conversations_1000_zh.jsonl" \
  --output_dir "./results/zh/" \
  --language "zh"

Recommended decoding

Table with columns: Parameter, Value
Parameter	Value
`temperature`	0.8
`top_p`	0.99
`max_tokens`	2048

Sampling matters here. Greedy decoding makes the customer flat and repetitive, which suppresses the natural objection variety the model was trained to produce.

Training

CustomerLM training pipeline

CustomerLM is trained in two stages on 8,284 crowdworker-involved real-world sales dialogues.

Stage 1 — SFT (learn to talk like a customer)

Stage 2 — DPO (learn not to behave like a salesperson)

SFT alone does not remove role inversion, because the failure is behavioural rather than stylistic. So we build preference pairs that target it directly:

The SFT model is paired with a sales model to simulate dialogues from generated scripts.
An LLM judge labels each simulated customer turn as assistant-like (the failure: helpful, pitching, summarizing benefits) or user-like (the target: in-character buyer behaviour).
Assistant-like turns are sent through GPT-4o + human correction to produce the user-like counterpart of the same turn.
Each (assistant-like, corrected user-like) pair becomes a DPO pair — rejected and chosen respectively.
DPO on these pairs yields the final model.

Evaluation

User simulation fidelity

Table with columns: User Model, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, Sem. Sim., Role Inversion (%)
User Model	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L	Sem. Sim.	Role Inversion (%)
GPT-4o	0.10	0.08	0.02	0.07	0.57	17.44
UserLM	0.06	0.08	0.01	0.06

CustomerLM wins on every metric. The headline result is role inversion: 8.8% vs. 17.44% for GPT-4o, roughly a 2× reduction, achieved by a much smaller model than the GPT-4o baseline it beats.

Effect on downstream benchmark scores

Table with columns: Assistant Model, ZH (GPT-4o user), ZH (CustomerLM user), EN (GPT-4o user), EN (CustomerLM user)
Assistant Model	ZH (GPT-4o user)	ZH (CustomerLM user)	EN (GPT-4o user)	EN (CustomerLM user)
Doubao-32K	6.07	6.89	6.31	5.48
Qwen-max	6.02	5.55	5.97	5.56
GPT-4o	5.72	6.15	5.53

Reproducing these numbers

bash
# Simulation fidelity (BLEU / ROUGE / semantic similarity)
python salesllm/user_likness_eval.py \
  --model_name "CustomerLM" \
  --api_end_point "http://localhost:8000/v1" --api_key "EMPTY" --api_type "other" \
  --input_files <held_out_conversations>.jsonl \
  --output_file user_likeness_results.jsonl --language zh

# Role inversion rate (LLM judge over generated dialogues)
python salesllm/reverse_role_eval.py \
  --model_name "<judge_model>" \
  --api_end_point "<judge_endpoint>" --api_key "<judge_key>" \
  --input_files ./results/zh/*.jsonl \
  --output_dir ./reverse_role_results

Limitations and Risks

Not an assistant. Deployed as a chatbot it will act like a customer. This is by design and is not a defect to be prompted away.
Format sensitivity. Persona adherence degrades if the system prompt departs from the trained layout. Personas outside the trained distribution (B2B procurement, healthcare intake, etc.) are extrapolation.
Domain coverage. Training data is Financial Services and Consumer Goods. Other verticals are untested.
Residual role inversion. 8.8% is a large improvement, not a solution. Roughly one dialogue in eleven still exhibits the failure, so filter with reverse_role_eval.py when simulation quality is load-bearing.
Reference metrics are weak instruments. BLEU/ROUGE against a single human reference under-measure valid alternative customer responses; treat them as comparative, not absolute.
Inherited bias. Persona attributes include age, gender, location, and occupation, and the model may reproduce demographic stereotypes present in the base model and the source dialogues. Do not use generated personas to make claims about real demographic groups.
Synthetic output. Conversations produced by CustomerLM are simulations, not evidence of real customer preferences, and should not be used as market research.

Citation

bibtex
@misc{su2026sellmoreplayless,
      title={Sell More, Play Less: Benchmarking LLM Realistic Selling Skill}, 
      author={Xuanbo Su and Wenhao Hu and Le Zhan and Yuting Xie and Kailin Lyu and Kaijie Chen and Ziwei Li and Yeqiang Wang and Haibo Su and Yunzhang Chen and Ling Huang},
      year={2026},
      eprint={2604.07054},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.07054}, 
}

📖 Paper
📊 SalesLLM benchmark & code
🤗 SaleIntent-BERT — buying-intent classifier used for outcome scoring (93.51% ZH, 92.94% EN)

CustomerLM

README

Intended Use

How It Works

System prompt format

Usage

Quick start (transformers)

Serving with vLLM

Use as the SalesLLM user simulator

Recommended decoding

Training

Stage 1 — SFT (learn to talk like a customer)

Stage 2 — DPO (learn not to behave like a salesperson)

Evaluation

User simulation fidelity

Effect on downstream benchmark scores

Reproducing these numbers

Limitations and Risks

Citation

Related

Explore FriendliAI today

README

Intended Use

How It Works

System prompt format

Usage

Quick start (transformers)

Serving with vLLM

Use as the SalesLLM user simulator

Recommended decoding

Training

Stage 1 — SFT (learn to talk like a customer)

Stage 2 — DPO (learn not to behave like a salesperson)

Evaluation

User simulation fidelity

Effect on downstream benchmark scores

Reproducing these numbers

Limitations and Risks

Citation

Related