speakrail

gemma-4-E4B-it-qat-Speakrail-181026

README

License: apache-2.0

What's different from the 12B release

Table with columns: 12B release, this model
	12B release	this model
base	`google/gemma-4-12B-it-qat-w4a16-ct`	`google/gemma-4-E4B-it-qat-w4a16-ct`
adapter	LoRA r32, 251 MB	LoRA r32, 133 MB (key/value projections exist only in E4B's 24 non-shared-KV layers)
token rows	`embed_tokens` (22 × 3840)	`embed_tokens` (22 × 2560) and `embed_tokens_per_layer` (22 × 10752): E4B looks every token up in a second, per-layer embedding table, and the new tokens are trained in both
tokenizer	v1.2	identical files
protocol, prompts, tools		unchanged

How it works

Same protocol as the 12B. The context is a token stream; the harness appends events as they happen and asks for a decision at each one:

text

<system prompt, tools, date>... user words as they arrive ... <complete>           <- the turn head says the user finished→ decision: <turn|> (speak)  → spoken reply, streamed to TTS... user words while the assistant speaks ...          <- an overlap→ decision: <continue> | <yield> | <listen><sil:4s>                                               <- nobody has spoken for 4 s→ decision: <listen>

Decisions are single tokens, constrained to the allowed set. The token table (ids 6-27; "speak" is Gemma's own <turn|>, id 106) and the decision sets are documented on the 12B card.

Files

Table with columns: file, what
file	what
`adapter_model.safetensors`, `adapter_config.json`	PEFT LoRA (rank 32, alpha 64, all attention and MLP projections)
`token_rows.safetensors`	the 22 trained rows (ids 6-27), bf16: `embed_tokens` [22, 2560] (the output head is tied to it) and `embed_tokens_per_layer` [22, 10752]
`tokenizer/`	tokenizer v1.2 (Gemma 4 + the 22 tokens) and the chat template, byte-identical to the 12B release

Google's base weights are not uploaded here, only the adapter and the rows.

Usage

The Speakrail docker setup does not install E4B yet: its model-init step writes the 12B's rows. Until it does, patch the rows into Google's checkpoint by hand and serve it with vLLM (tested with vLLM 0.26):

python

# writes rows 6-27 into embed_tokens, lm_head (if present) and embed_tokens_per_layer of Google's checkpointfrom huggingface_hub import snapshot_download, hf_hub_downloadfrom safetensors.torch import load_file, save_filefrom safetensors import safe_openbase = snapshot_download("google/gemma-4-E4B-it-qat-w4a16-ct", local_dir="speakrail-e4b/model")rows = load_file(hf_hub_download("speakrail/gemma-4-E4B-it-qat-Speakrail-181026", "token_rows.safetensors"))path = f"{base}/model.safetensors"with safe_open(path, "pt") as f:    meta, t = f.metadata(), {k: f.get_tensor(k) for k in f.keys()}for key, name in [("model.language_model.embed_tokens.weight", "embed_tokens"), ("lm_head.weight", "embed_tokens"),                  ("model.language_model.embed_tokens_per_layer.weight", "embed_tokens_per_layer")]:    if key in t:        t[key] = t[key].clone(); t[key][6:28] = rows[name]save_file(t, path, metadata=meta)

Copy tokenizer/* into speakrail-e4b/model/, put the adapter in speakrail-e4b/adapter/, then:

bash

vllm serve speakrail-e4b/model --served-model-name speakrail-base \  --enable-lora --max-lora-rank 32 --max-loras 1 --lora-modules speakrail=speakrail-e4b/adapter \  --max-model-len 16384 --max-num-seqs 3 --gpu-memory-utilization 0.44 --enable-prefix-caching \  --generation-config vllm --limit-mm-per-prompt '{"image":0,"audio":0}'

Point the Speakrail app at it as for the 12B (served names speakrail-base and speakrail). On a 24 GB card next to Breeze TTS 2 and the speech recognizer, use --gpu-memory-utilization 0.44: E4B's weights take more GPU memory than the 12B's (9.4 vs 8.5 GiB, because of the bf16 per-layer table); at 0.49 it ran out of memory under load in our tests.

16 GB GPUs: keep the 5.6 GB per-layer embedding table in pinned CPU memory, read zero-copy over PCIe. This drops the model's GPU weights from 9.4 to 4.1 GiB with identical outputs and no measurable latency cost, because the table is only looked up for the current tokens. vLLM's own CPU offload (--cpu-offload-gb) does not reach this table; we use a small loader hook (not published yet). With it, one sequence (--max-num-seqs 1) at --gpu-memory-utilization 0.31 holds the full 131,072-token context.

Results

All numbers are measured on one RTX 4090 with the whole Speakrail system: Voxtral Mini 4B Realtime + turn head, the model with listening notes (the base model takes notes while the user speaks), and Breeze TTS 2.

Two harness versions appear below:

release harness: the one shipped with the 12B release.
updated harness: a newer turn head that also predicts the end of the turn 80-240 ms ahead, an early trigger that starts the reply on that prediction, and a faster cue rule. This model was only evaluated on the updated harness, so the 12B release is shown on both, to separate the model from the harness.

Full-Duplex-Bench v3 (tool use on real, disfluent speech; 100 recordings, the benchmark's own evaluators)

Table with columns: 12B release, release harness, 12B release, updated harness, this model, updated harness
	12B release, release harness	12B release, updated harness	this model, updated harness
Pass@1	0.52	0.56	0.44
Tool F1	0.915	0.934	0.815
Arg acc	0.633	0.660	0.588
Reply	0.755	0.784	0.695
Turn-take	98 %	97 %	95 %
Interrupt ↓	22.4 %	17.5 %	25.3 %
Filler	100 %	100 %	96 %
1st word ↓ (mean / median)	1.82 / 0.80 s	1.32 / 0.32 s	1.36 / 0.40 s
Tool call ↓ (mean / median)	2.39 / 1.23 s	1.96 / 0.84 s	1.46 / 0.74 s
Done ↓ (mean / median)	3.92 / 2.80 s	3.34 / 2.16 s	3.15 / 2.08 s

This model's Pass@1 by difficulty: easy 0.58, medium 0.32, hard 0.40; by number of tools: 1 tool 0.52, 2 tools 0.33, 3 tools 0.25. It reaches tool calls and finishes tasks fastest, but picks the wrong tool more often: tool use is where the smaller model loses most.

Conversational dynamics (Full-Duplex-Bench v1.0 / v1.5, scored the way Artificial Analysis does)

The mean of four sub-scores (pause handling, turn taking, user interruption, user backchannel), as on the Artificial Analysis conversational-dynamics leaderboard; our local reproduction of their formula, one live session at a time, GPT-4o judge for the v1.5 subsets.

Table with columns: 12B release, release harness, 12B release, updated harness, this model, updated harness
	12B release, release harness	12B release, updated harness	this model, updated harness
Conversational Dynamics	94.0 %	93.4 %	88.0 %
pause handling	93.4	87.6	88.3
turn taking	96.6	95.0	85.7
user interruption	89.0	92.0	83.0
user backchannel	96.9	99.0	94.9
turn-taking latency (CANDOR, mean) ↓	1.18 s	0.67 s	0.56 s

Most of the turn-taking and interruption gap is reply style, not timing: the model decides to speak at the same moment as the 12B, but on casual statements it often answers with a stock two- or three-word acknowledgement ("Sounds good.", "Got it."), which the benchmark counts as a backchannel rather than taking the turn. That is a training-data artefact we will fix in the next version.

Instruction following (spoken)

Table with columns: 12B release, this model
	12B release	this model
VoiceBench IFEval (345 spoken prompts, VoiceBench's scorer)	54.3	51.0

Latency on a real conversation

A recorded 15-minute live conversation replayed in real time (about 40 turns), p50 / p90, from the end of the user's last word:

Table with columns: 12B release, release harness (the original live session), 12B release, updated harness, this model, updated harness
	12B release, release harness (the original live session)	12B release, updated harness	this model, updated harness
to the end-of-turn decision	348 / 927 ms	297 / 894 ms	316 / 735 ms
to the first audio sample	558 / 1108 ms	464 / 697 ms	399 / 663 ms
to the speaker	623 / 1136 ms	528 / 905 ms	460 / 822 ms
model time to first token	148 / 248 ms	144 / 184 ms	138 / 170 ms
words per reply (median)	28	40	20

Memory

Table with columns: 12B release stack, this model, 24 GB stack, this model, 16 GB stack
	12B release stack	this model, 24 GB stack	this model, 16 GB stack
TTS	Breeze TTS 2	Breeze TTS 2	Pocket TTS (on GPU)
LLM GPU weights	8.5 GiB	9.4 GiB	4.1 GiB (per-layer table in CPU memory)
context	16k, 3 sequences	16k, 3 sequences	131k, 1 sequence
whole stack, GPU memory	22.3 GB	22.6 GB	15.3 GB (15.7 GB after a session)

The 16 GB stack was checked end to end on recorded sessions, not on the benchmarks above.

Training

Base: google/gemma-4-E4B-it-qat-w4a16-ct, trained directly on the QAT int4 weights.
Trainable parameters: LoRA rank 32 / alpha 64 on q, k, v, o, gate, up and down projections, plus the 22 new rows in both embedding tables, with their own learning rate (1e-3; LoRA 1e-4).
Loss: cross-entropy on the decision tokens and on the reply tokens, plus a KL term to the base model's outputs (β 0.5) to keep its general behaviour.
Compute: one epoch, 2,384 steps of ~32k tokens, about 4 hours on a single H100. The released checkpoint is the final step, which had the lowest validation decision loss of the evaluated checkpoints.

The training data is the same kind as for the 12B release and is not released.

Limitations

English only.
Needs the harness. Without the protocol and the decision constraints it is a weaker general chat model than base Gemma.
Weaker than the 12B release on tool use (FDB v3 Pass@1 0.44 vs 0.52) and on conversational dynamics (88.0 vs 94.0).
Stock acknowledgements: it overuses short replies like "Sounds good." on casual statements.
Not yet in the docker setup: install by hand as above until model-init supports it.
Facts: it can be confidently wrong like any small model; Speakrail adds web search.

License

Apache 2.0, same as Gemma 4 (Gemma 4 license).

Citation

bibtex

@misc{speakrail2026gemmae4b,  title  = {Speakrail Gemma 4 E4B: a full-duplex turn-taking model for 16 GB GPUs},  author = {Speakrail},  year   = {2026},  url    = {https://huggingface.co/speakrail/gemma-4-E4B-it-qat-Speakrail-181026}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider