praex

Lucia-III

README

License: glm-5.3

Results

Terminal-Bench, run by us in the Praex agent:

Table with columns: Benchmark and what was measured, Lucia III, GLM-5.3
Benchmark and what was measured	Lucia III	GLM-5.3
TB 2.0, same 74 tasks, one try each, same agent and settings	68.9% (51 of 74)	58.1% (43 of 74)
TB 2.0, all 89 tasks, 5 tries each (445 trials), self-run with the leaderboard settings	61.3% (273 of 445)	not run
TB 2.1, all 89 tasks, 5 tries each (445 trials), self-run with the leaderboard settings	68.8% (306 of 445)	not run

Row 1 is a head-to-head comparison. Both models ran in the same agent (Praex CLI v0.1.32) with a 4,096-token thinking budget per turn and the benchmark's own task timeouts. The 74 tasks are the ones that were graded in every run we compared. The other 15 had a harness error in at least one run. With one try per task, expect a few points of noise either way.

Rows 2 and 3 are the whole benchmark. Terminal-Bench 2.1 is the current version of the benchmark. Both runs cover all 89 tasks with 5 tries each, using the leaderboard settings: timeout multiplier 1.0 and no timeout or resource overrides. Both use a 4,096-token thinking budget per turn.

TB 2.0: Praex CLI v0.1.32.
- 95% confidence interval: plus or minus 8.5 points, clustered by task.
- Solved at least once in 5 tries (pass@5): 80.9% of tasks. 37 tasks solved in all 5 tries, 17 never.
- Trial logs: https://hub.harborframework.com/jobs/bef60fa2-3ac4-4db3-85bd-6fbdadcf5b5f
TB 2.1: Praex CLI v0.1.40, which keeps the agent alive when a build inside a task runs out of memory.
- 95% confidence interval: plus or minus 8.0 points, clustered by task.
- Solved at least once in 5 tries (pass@5): 82.0% of tasks. 43 tasks solved in all 5 tries, 16 never.
- 3 trials are counted as failures (see the contamination checks below). Without that correction the score would be 69.4%.
- Trial logs: https://hub.harborframework.com/jobs/30cac5da-1bb0-4750-8625-f28653ed44d2

These are self-run results, not official leaderboard entries. The official Terminal-Bench boards currently add only runs made by their maintainers.

About other published numbers. Z.ai reports GLM-5.3 results measured in Claude Code with timeouts of several hours. Our numbers use the Praex agent, a 4,096-token thinking budget per turn and the benchmark's own timeouts. The two sets of numbers are not comparable. Row 1 is the fair comparison with the base model.

Contamination checks.

Training tasks. We checked them against Terminal-Bench 2.0, 2.1 and 4.0 for 13-gram overlap in instructions, for task names and for the canary string. Nothing matched, so no task was removed.
What we scan for. In every benchmark trajectory, we look for visits to the benchmark's website, its repositories and copies of its files.
TB 2.0: none.
TB 2.1: in 4 of the 445 trials the agent searched the web for the task's source.
- In 3 of them it found the benchmark's own solution and test files, once on a third-party copy, and used them. All 3 passed: two on mteb-retrieve, one on extract-elf. We count those 3 trials as failures.
- The 4th search found nothing, and that trial failed anyway.
- The trials are left unedited in the public logs.

How it was trained

Lucia III was trained in two stages on top of GLM-5.3, keeping its FP8 weights.

Faber II is our earlier post-train of GLM-5.3, made for the Praex agent and its tool format.
Lucia III was then trained on Faber II's own successful runs, an approach called rejection-sampled fine-tuning:
- Faber II attempted 2,000 terminal tasks from NVIDIA's Nemotron-Terminal-Synthetic-Tasks in the Praex agent.
- We kept the runs that passed the task's tests: 1,195 trajectories, 45.75M tokens.
- We trained LoRA rank 64 on the attention layers for one epoch, at learning rate 5e-5 on a cosine schedule.
- This release is the step 145 of 149 checkpoint, merged into the weights.

All training ran on one node with 8 × B200.

Use it

With vLLM v0.28.0 on one node with 8 × B200, which is what we run. The weights are about 716 GiB in FP8.

bash

vllm serve praex/Lucia-III --served-model-name lucia-iii --tensor-parallel-size 8 \  --trust-remote-code --max-model-len 262144 --kv-cache-dtype fp8 \  --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm45

The model has two chat templates:

chat_template.jinja, the default, has thinking on.
chat_template_nothink.jinja has thinking off.

The default sampling settings are in generation_config.json: temperature 1.0, top_p 0.95.

Limitations

We evaluated Lucia III mainly on terminal and coding tasks in English.
We have not run Terminal-Bench 4.0 in full, so this card makes no claim about it.
When it is stuck, Lucia III sometimes searches the web for the task's source, including benchmark solutions. Block benchmark repositories and their copies when you evaluate it.
Like any agent model, it runs real commands. Give it a sandbox or a container when the stakes are high.

License

Lucia III is a derivative of GLM-5.3 and is released under the GLM-5.3 License, an MIT-style license. See LICENSE. Its copyright notice, "Copyright (c) 2026 Z.AI", and its permission notice are kept unchanged, as the license requires.

Clause 2 of the license applies only to very large companies. If a company runs a Model as a Service business and its group revenue is above US$10 billion over 12 months, it must pass Z.AI's security review before commercial use.

Training tasks come from NVIDIA's Nemotron-Terminal-Synthetic-Tasks, licensed CC BY 4.0.

Credits

Base model: GLM-5.3 by Z.ai.
Training tasks: Nemotron-Terminal-Synthetic-Tasks by NVIDIA.
Benchmark: Terminal-Bench, run with Harbor.

About Praex

Praex builds its own AI models for coding and terminal agents, and the open-source agent that runs them in the terminal, on desktop, on Android and on the web.

Website: praex.ai
Contact: davor@praex.ai

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

praex

Model Tree

Base

zai-org/GLM-5.3

Fine-tuned

this model

Input Modalities

Text

Output Modalities