google

DiarizationLM-Gemma-4-E4B-v1

README

License: apache-2.0

Overview

DiarizationLM is a Large Language Model framework designed to post-process, correct, and optimize automatic speech recognition (ASR) and speaker diarization outputs.

google/DiarizationLM-Gemma-4-E4B-v1 is built on Google's Gemma 4 E4B (4B dense parameters) foundation model and fine-tuned with Locality-Preserving Oracle Supervision across all 4 canonical speaker diarization benchmark corpora:

Fisher English (2-speaker conversational telephone speech)
Callhome American English (2–5 speaker informal telephone conversations)
ICSI Meeting Corpus (3–9 speaker academic research meetings)
AMI Meeting Corpus (4-speaker tabletop meetings)

Foundation model: google/gemma-4-E4B
Open-source library & scripts: https://github.com/google/speaker-id/tree/master/DiarizationLM

Unlike earlier models trained exclusively on 2-speaker telephone data (such as google/DiarizationLM-8b-Fisher-v2) or naive multi-domain SFT (which suffers from long-monologue speaker identity drift in 4+ speaker meetings), google/DiarizationLM-Gemma-4-E4B-v1 is trained on locality-preserving oracle targets that teach the model to correct lexical turn boundaries and backchannels (1 ≤ L ≤ 5 words) while preserving acoustic speaker anchors across long monologues (L ≥ 6 words). As a result, it achieves statistically significant (p < 0.0001) WDER and cpWER improvements simultaneously across all four benchmarks under standard out-of-the-box Transcript-Preserving Speaker Transfer (transfer_llm_completion), despite having half the parameter count (4B vs. 8B).

Training Configuration

Base Architecture: Gemma 4 E4B (42 layers, hidden size 2560, hybrid 5:1 sliding-window and global attention, 262,144 vocabulary size)
LoRA Adapter: Rank r = 256 applied to all attention (q_proj, k_proj, v_proj, o_proj), MLP (gate_proj, up_proj, down_proj), and per-layer input (per_layer_input_gate, per_layer_projection, per_layer_model_projection) linear projections, merged into 16-bit (bfloat16) base weights and serialized to 4-bit GGUF (Q4_K_M and Q4_0)
: Completion-only cross-entropy loss ()

Model Files Included

model.safetensors: Merged 16-bit (bfloat16) Hugging Face transformers weights (~16.0 GB)
DiarizationLM-Gemma-4-E4B-v1-q4_k_m.gguf: (Recommended GGUF) Serialized 4-bit K-quant Medium (Q4_K_M) GGUF model (~5.30 GB) for llama.cpp / Ollama / llama-cpp-python (uses 256-element super-blocks in Q4_K with sensitive attn_v / ffn_down and embedding matrices retained in 6-bit Q6_K)
DiarizationLM-Gemma-4-E4B-v1-q4_0.gguf: Serialized legacy 4-bit () GGUF model () for / Ollama /

Benchmark Performance (with 95% Bootstrap Confidence Intervals)

All metrics below are micro-averaged across the full evaluation sets using the USM + turn-to-diarize baseline and scored via Hungarian-matching dynamic programming (diarizationlm.compute_metrics_on_json_dict). Ranges in brackets indicate 95% non-parametric bootstrap confidence intervals (B = 10,000 conversation-level resamples):

Table with columns: Benchmark, Test Split, WER (%), System, WDER (%) [95% CI], cpWER (%) [95% CI], SpkCntMAE [95% CI], Paired ΔWDER vs. Baseline (p-value)
Benchmark	Test Split	WER (%)	System	WDER (%) [95% CI]	cpWER (%) [95% CI]	SpkCntMAE [95% CI]	Paired ΔWDER vs. Baseline (p-value)
Fisher	TEST FULL (172 sessions)	15.37	Baseline (USM + Turn-to-Diarize)`DiarizationLM-8b-Fisher-v2` (Llama 3 8B)`DiarizationLM-Gemma-4-E4B-v1` (4B)	5.32 [4.93, 5.74]3.282.99 [2.65, 3.37]	20.88 [19.97, 21.88]18.3717.62 [16.76, 18.56]	0.215 [0.151, 0.291]---

Usage

1. Python (`transformers` + `diarizationlm`)

First, install the required packages:

bash
pip install transformers diarizationlm

Run inference on a GPU with bfloat16:

python
from diarizationlm import utils
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "google/DiarizationLM-Gemma-4-E4B-v1"

HYPOTHESIS = (
    "<speaker:1> Hello, how are you doing <speaker:2> today? I am doing well."
    " What about <speaker:1> you? I'm doing well, too. Thank you."
)

print("Loading model...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, device_map="cuda")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.bfloat16, device_map="cuda"
)

print("Tokenizing input...")
prompt = f"<|turn>user\n{HYPOTHESIS} --> <turn|>\n<|turn>model\n"
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")

print("Generating completion...")
outputs = model.generate(
    **inputs,
    max_new_tokens=int(inputs.input_ids.shape[1] * 1.2),
    do_sample=False,
    use_cache=True,
)

print("Decoding completion...")
completion = tokenizer.batch_decode(
    outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
)[0]
completion = utils.truncate_suffix_and_tailing_text(completion, " [eod]")

print("Transferring completion to hypothesis text...")
transferred_completion = utils.transfer_llm_completion(completion, HYPOTHESIS)

print("========================================")
print("Hypothesis:", HYPOTHESIS)
print("========================================")
print("Completion:", completion)
print("========================================")
print("Transferred completion:", transferred_completion)
print("========================================")

2. GGUF (`llama.cpp`)

You can also run the quantized DiarizationLM-Gemma-4-E4B-v1-q4_k_m.gguf model directly with llama.cpp:

bash
llama-cli \
  -m DiarizationLM-Gemma-4-E4B-v1-q4_k_m.gguf \
  -p $'<|turn>user\n<speaker:1> Hello, how are you doing <speaker:2> today? I am doing well. What about <speaker:1> you? I\'m doing well, too. Thank you. --> <turn|>\n<|turn>model\n' \
  --temp 0.0 \
  -n 128

Citation

bibtex
@inproceedings{wang24h_interspeech,
  title     = {{DiarizationLM: Speaker Diarization Post-Processing with Large Language Models}},
  author    = {Quan Wang and Yiling Huang and Guanlong Zhao and Evan Clark and Wei Xia and Hank Liao},
  year      = {2024},
  booktitle = {Interspeech 2024},
  pages     = {3754--3758},
  doi       = {10.21437/Interspeech.2024-209},
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

google

Model Tree

Base

google/gemma-4-E4B

Quantized

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Overview

DiarizationLM is a Large Language Model framework designed to post-process, correct, and optimize automatic speech recognition (ASR) and speaker diarization outputs.

Fisher English (2-speaker conversational telephone speech)
Callhome American English (2–5 speaker informal telephone conversations)
ICSI Meeting Corpus (3–9 speaker academic research meetings)
AMI Meeting Corpus (4-speaker tabletop meetings)

Foundation model: google/gemma-4-E4B
Open-source library & scripts: https://github.com/google/speaker-id/tree/master/DiarizationLM

Training Configuration

Base Architecture: Gemma 4 E4B (42 layers, hidden size 2560, hybrid 5:1 sliding-window and global attention, 262,144 vocabulary size)
LoRA Adapter: Rank r = 256 applied to all attention (q_proj, k_proj, v_proj, o_proj), MLP (gate_proj, up_proj, down_proj), and per-layer input (per_layer_input_gate, per_layer_projection, per_layer_model_projection) linear projections, merged into 16-bit (bfloat16) base weights and serialized to 4-bit GGUF (Q4_K_M and Q4_0)
: Completion-only cross-entropy loss ()

Model Files Included

model.safetensors: Merged 16-bit (bfloat16) Hugging Face transformers weights (~16.0 GB)
DiarizationLM-Gemma-4-E4B-v1-q4_k_m.gguf: (Recommended GGUF) Serialized 4-bit K-quant Medium (Q4_K_M) GGUF model (~5.30 GB) for llama.cpp / Ollama / llama-cpp-python (uses 256-element super-blocks in Q4_K with sensitive attn_v / ffn_down and embedding matrices retained in 6-bit Q6_K)
DiarizationLM-Gemma-4-E4B-v1-q4_0.gguf: Serialized legacy 4-bit () GGUF model () for / Ollama /

Benchmark Performance (with 95% Bootstrap Confidence Intervals)

Table with columns: Benchmark, Test Split, WER (%), System, WDER (%) [95% CI], cpWER (%) [95% CI], SpkCntMAE [95% CI], Paired ΔWDER vs. Baseline (p-value)
Benchmark	Test Split	WER (%)	System	WDER (%) [95% CI]	cpWER (%) [95% CI]	SpkCntMAE [95% CI]	Paired ΔWDER vs. Baseline (p-value)
Fisher	TEST FULL (172 sessions)	15.37	Baseline (USM + Turn-to-Diarize)`DiarizationLM-8b-Fisher-v2` (Llama 3 8B)`DiarizationLM-Gemma-4-E4B-v1` (4B)	5.32 [4.93, 5.74]3.282.99 [2.65, 3.37]	20.88 [19.97, 21.88]18.3717.62 [16.76, 18.56]	0.215 [0.151, 0.291]---

Usage

1. Python (`transformers` + `diarizationlm`)

First, install the required packages:

bash
pip install transformers diarizationlm

Run inference on a GPU with bfloat16:

python
from diarizationlm import utils
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "google/DiarizationLM-Gemma-4-E4B-v1"

HYPOTHESIS = (
    "<speaker:1> Hello, how are you doing <speaker:2> today? I am doing well."
    " What about <speaker:1> you? I'm doing well, too. Thank you."
)

print("Loading model...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, device_map="cuda")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.bfloat16, device_map="cuda"
)

print("Tokenizing input...")
prompt = f"<|turn>user\n{HYPOTHESIS} --> <turn|>\n<|turn>model\n"
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")

print("Generating completion...")
outputs = model.generate(
    **inputs,
    max_new_tokens=int(inputs.input_ids.shape[1] * 1.2),
    do_sample=False,
    use_cache=True,
)

print("Decoding completion...")
completion = tokenizer.batch_decode(
    outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
)[0]
completion = utils.truncate_suffix_and_tailing_text(completion, " [eod]")

print("Transferring completion to hypothesis text...")
transferred_completion = utils.transfer_llm_completion(completion, HYPOTHESIS)

print("========================================")
print("Hypothesis:", HYPOTHESIS)
print("========================================")
print("Completion:", completion)
print("========================================")
print("Transferred completion:", transferred_completion)
print("========================================")

2. GGUF (`llama.cpp`)

You can also run the quantized DiarizationLM-Gemma-4-E4B-v1-q4_k_m.gguf model directly with llama.cpp:

bash
llama-cli \
  -m DiarizationLM-Gemma-4-E4B-v1-q4_k_m.gguf \
  -p $'<|turn>user\n<speaker:1> Hello, how are you doing <speaker:2> today? I am doing well. What about <speaker:1> you? I\'m doing well, too. Thank you. --> <turn|>\n<|turn>model\n' \
  --temp 0.0 \
  -n 128

Citation

bibtex
@inproceedings{wang24h_interspeech,
  title     = {{DiarizationLM: Speaker Diarization Post-Processing with Large Language Models}},
  author    = {Quan Wang and Yiling Huang and Guanlong Zhao and Evan Clark and Wei Xia and Hank Liao},
  year      = {2024},
  booktitle = {Interspeech 2024},
  pages     = {3754--3758},
  doi       = {10.21437/Interspeech.2024-209},
}

DiarizationLM-Gemma-4-E4B-v1

README

Overview

Training Configuration

Model Files Included

Benchmark Performance (with 95% Bootstrap Confidence Intervals)

Usage

1. Python (transformers + diarizationlm)

2. GGUF (llama.cpp)

Citation

Explore FriendliAI today

README

Overview

Training Configuration

Model Files Included

Benchmark Performance (with 95% Bootstrap Confidence Intervals)

Usage

1. Python (transformers + diarizationlm)

2. GGUF (llama.cpp)

Citation

1. Python (`transformers` + `diarizationlm`)

2. GGUF (`llama.cpp`)

1. Python (`transformers` + `diarizationlm`)

2. GGUF (`llama.cpp`)