What Benchmarks Don’t Tell About GLM-5.3 and Kimi K3 for Coding Agents

What Benchmarks Don’t Tell About GLM-5.3 and Kimi K3 for Coding Agents

Benchmark scores provide a useful summary of overall model capability, but they are not sufficient on their own for selecting a model for coding-agent workflows. GLM-5.3 and Kimi K3 both score 60 on the Artificial Analysis Intelligence Index, suggesting equal intelligence. Our evaluation, however, revealed three findings that this tie does not capture:

• The models showed different performance profiles on coding tasks. GLM-5.3 achieved the stronger overall result in our evaluation, but Kimi K3 still solved specific tasks that GLM-5.3 missed.
• The models showed different strengths in open-ended game development. Kimi K3 produced the more visually refined game, while GLM-5.3 delivered the more functionally complete implementation and stronger end-to-end playability.
• Combining the models captured complementary strengths. Per-task selection increased benchmark coverage, while refining GLM-5.3’s game implementation with Kimi K3 combined functional completeness with stronger visual presentation.

Our Evaluation Approach

We evaluated GLM-5.3 and Kimi K3 through coding agent benchmarks and an open-ended game development case study (Figure 1).

We compare GLM-5.3 and Kimi K3 across two coding benchmarks and a Unity game-development case study.

The benchmark evaluation measures accuracy and cost per task on SWE-bench Verified, which includes 500 real issues from open-source projects, and Terminal-Bench 2.1, which includes 89 practical tasks performed in terminal environments. As shown in Table 1, GLM-5.3 and Kimi K3 achieved comparable performance on maximum reasoning, differing by 0.4 percentage points and 5.6 percentage points, respectively.

Table 1. Model overview and benchmark results for GLM-5.3 and Kimi K3. Token pricing is based on the official API pricing published by Z.ai and Moonshot AI, respectively.

These overall results, however, can mask differences at the task level. Coding agents often make multiple model calls to complete a single task, so the cost of each task depends on the full agent trajectory rather than token prices alone. We therefore break down the benchmark results by task to examine differences in both performance and cost.

Coding Benchmark Breakdown

To examine these differences, we recorded whether the model succeeded and the cumulative generation cost of all model calls in the agent trajectory. Despite similar overall accuracy, the models succeeded on different tasks. On SWE-bench Verified, GLM-5.3 uniquely solved 18 tasks, and Kimi K3 uniquely solved 16; on Terminal-Bench 2.1, they uniquely solved 10 and 5 tasks, respectively.

The intersection shows tasks solved by both models, and the non-overlapping regions show tasks solved by only one model.

Results by Task Category

To examine where the differences occurred, we compared their exclusive successes across task categories in SWE-bench Verified and Terminal-Bench 2.1.

The models showed different patterns of exclusive successes across task categories. On SWE-bench Verified, GLM-5.3 had substantially more exclusive successes on data problems, while Kimi-K3 had more on observability problems. The remaining categories were relatively balanced. On Terminal-Bench 2.1, GLM-5.3 had more exclusive successes in ML and Media, while the models were tied in Software and Science and had no exclusive successes in System.

Cost per Task

GLM-5.3 achieved lower average cost per task, but the relationship varied across task categories (Figure 4). On SWE-bench Verified, GLM-5.3 was less expensive across all five categories, while Terminal-Bench 2.1 showed more varied cost-accuracy trade-offs.

Case Study: Open-Ended Game Development

Benchmarks use predefined tasks with clear success criteria, but they reveal less about how agents handle open-ended design decisions. Development in Unity, a game engine, is challenging because many implementations can run successfully while differing substantially in playability and visual quality. The result must be rendered to assess camera framing, object placement, interface layout, and overall scene composition.

To complement the benchmark analysis, we conducted an illustrative case study where GLM-5.3 and Kimi K3 built the same Unity game, Bounce Ball (Figure 5). We evaluated visual quality, interface coherence, and end-to-end playability, and reviewed the execution traces to compare how the models implemented and validated their games.

(a) GLM-5.3: Play the game →

(b) Kimi K3: Play the game →

Visual Quality

Kimi K3 produced more visually detailed game elements, whereas GLM-5.3 used simpler designs overall (Figure 6).

The execution traces show different uses of screenshot-based validation. GLM-5.3 primarily performed pixel-level checks, such as detecting expected colors and verifying that backgrounds were not blank. These checks provided limited semantic assessment, leaving several sprite and layout issues unresolved. Kimi K3 instead inspected rendered scenes and iteratively refined the sprites, UI, backgrounds, and overall composition.

GLM-5.3’s final game also contained UI layout issues. Text labels were misaligned with button backgrounds, reducing readability, while Head-Up Display (HUD) elements such as the life indicator were placed outside the visible Game View and were clipped or absent during gameplay.

Playability

Despite its stronger visual presentation, Kimi K3’s game could not be completed through normal play. In Stage 2, the gap between the final platform and the next landing point was physically unreachable (Figure 7). In contrast, GLM-5.3’s game could be played till the end. The execution traces suggest that the models validated playability differently. GLM-5.3 repeatedly tested and repaired physics-dependent traversal using an automated route via automated inputs. Kimi K3 relied more heavily on teleport-based validation for its full-game flow.

Combining Complementary Strengths

Across the benchmark evaluations, GLM-5.3 and Kimi K3 showed comparable overall performance but succeeded on different individual tasks. The Unity case study revealed a similar complementarity: Kimi K3 produced the more visually refined game, while GLM-5.3 delivered stronger functional completeness and end-to-end playability. We therefore examined two ways of combining these strengths: per-task model selection for benchmark tasks and sequential handoff for open-ended development.

Per-Task Model Selection

To estimate the maximum potential benefit of per-task model selection, we chose the better result from the two models for each task. When only one model succeeded, we retained that result; when both succeeded, we selected the lower-cost trajectory. Because this selection was made after observing the outcomes, it represents a theoretical ceiling rather than a practical routing strategy.

As shown in Figure 8, Per-task selection achieved higher accuracy than either model alone on both benchmarks. On SWE-bench Verified, accuracy reached 97.4%, compared with 94.2% for GLM-5.3 and 93.8% for Kimi K3, at a cost of $0.198 per task. On Terminal-Bench 2.1, accuracy reached 92.1%, compared with 86.5% for GLM-5.3 and 80.9% for Kimi K3, while cost per task decreased to $0.424, below both models.

Sequential Model Handoff

We reran the Unity experiment to test whether the models’ complementary strengths could be combined in a single game. GLM-5.3 first implemented the gameplay, after which Kimi K3 performed additional visual validation and debugging. The resulting game in Figure 9 retained GLM-5.3’s gameplay architecture while benefiting from Kimi K3’s visual refinement. One remaining camera-movement issue was corrected with a one-line manual code change.

Play the game →

Figure 10 illustrates their different development approaches: GLM-5.3 reasoned about ball physics and platform placement to design playable stages, whereas Kimi K3 repeatedly captured rendered scenes and used visual feedback to refine the UI, backgrounds, and game objects.

The left panel is an illustration reconstructed from GLM-5.3’s reasoning trace, while the right panel shows Kimi K3’s screenshot-based iteration.

The Right Model Depends on the Task

Kimi K3 and GLM-5.3 are both highly capable models, but our evaluation showed that each model performed better on different coding tasks and emphasized different dimensions of quality in open-ended development. GLM-5.3 delivered stronger results in some areas, while Kimi K3 stood out in others, including tasks that benefited from its multimodal capabilities and visual refinement.

The better model therefore depends on your agent, workload, and domain. Benchmark rankings are a useful starting point, but the most reliable way to choose is to evaluate both models against representative tasks using the metrics that matter in production, such as accuracy, latency, cost, reliability, and output quality.

👉 GLM-5.3 and Kimi K3 will both be available on FriendliAI.

Work autonomously in the Unity project currently opened as the VS Code workspace.

Inspect the existing project before making changes. Then create a complete, polished, and independently playable 2D side-scrolling bounce-ball obstacle game titled “Bounce Ball.”

The player controls a ball that bounces automatically whenever it lands on a valid surface. The player can control only horizontal movement and must travel through three stages, avoid hazards and gaps, collect required items, activate checkpoints, and reach the final goal.

Complete the entire task in one autonomous run. Do not ask for clarification or wait for human feedback.

## 1. Core controls and movement

- Use Unity 2D physics for the player and gameplay collisions.
- The ball must bounce automatically after contacting a valid floor or platform.
- The player must not have a manual jump action.
- Support horizontal movement with A and D and the left and right arrow keys.
- Horizontal movement must remain responsive while the ball is airborne.
- The camera must follow the player smoothly without losing the player or exposing unintended empty areas.
- The ball must not receive duplicate bounce impulses from one landing.
- Collisions with walls, platform edges, moving platforms, and hazards must not leave the ball permanently stuck.

## 2. Complete game flow

The game must provide this complete flow:

1. Title screen
2. Compact instructions and controls
3. Stage 1
4. Stage 2
5. Stage 3
6. Victory
7. Restart and a clean second run

The player starts with three lives.

A death must reduce exactly one life. If lives remain, the player respawns at the latest activated checkpoint. If no lives remain, the game enters Game Over.

Victory and Game Over must never become active at the same time.

Restarting after Game Over or Victory must begin a clean new run. Score, lives, collected items, checkpoints, stage progress, temporary effects, and terminal UI must reset correctly.

## 3. Stage 1 — Meadow Ruins

Create an introductory outdoor stage that teaches movement and bouncing through level design.

- Place exactly five collectible shards.
- Each shard can be collected and scored only once.
- Display the shard count as 0/5 through 5/5.
- The exit remains locked until all five shards have been collected.
- Provide clear feedback while the exit is locked and when it becomes available.
- Include at least one hazard and one gap.
- Include a checkpoint before the exit.
- Death before and after activating the checkpoint must result in different respawn positions.
- Collected shards must not reappear or grant duplicate score after a normal respawn.

## 4. Stage 2 — Moving Works

Create a stage focused on timing, moving surfaces, and controlled landings.

- Include at least two moving platforms with visibly different movement patterns.
- The player must be carried consistently by a moving platform while standing on it.
- Include at least one temporary or cycling platform.
- Include hazards that require the player to use the moving platforms rather than bypassing the intended route.
- Include three collectibles placed along the required route.
- Include a checkpoint after meaningful progress.
- Falling or touching a lethal hazard must use the same life and respawn rules as Stage 1.
- Stage 2 must not begin with leftover objects, collectibles, hazards, or UI from Stage 1.

## 5. Stage 3 — Final Gauntlet

Create a final stage combining the mechanics introduced earlier.

- Include a sequence that combines automatic bouncing, horizontal air control, moving platforms, hazards, and gaps.
- Include three final collectibles.
- Include at least one checkpoint.
- Include a clearly identifiable final goal.
- The final goal must remain inaccessible until all required Stage 3 collectibles have been collected.
- Reaching the unlocked final goal must trigger Victory exactly once.
- Hazards and player movement must stop affecting the run after Victory.
- Game Over must not overwrite Victory if both could otherwise occur close together.

## 6. Score, lives, checkpoints, and state behavior

- Show the current score, remaining lives, current stage, collectible progress, and active objective during gameplay.
- Scoring events must occur only once per collectible or completion event.
- The owner of score, lives, stage progress, checkpoint state, Game Over, and Victory must be unambiguous in the implementation.
- A single death must not be processed by multiple components.
- Stage transitions must preserve only the state intended to continue and reset stage-local state correctly.
- Paused, playing, dying, Game Over, and Victory behavior must not overlap in contradictory ways.
- Restarting the game must not create duplicate managers, players, cameras, UI, or event subscriptions.

## 7. Presentation and usability

- Create a cohesive original visual style using original or procedurally created content.
- Do not use copyrighted characters, brands, logos, levels, music, or assets.
- Give each stage a visually distinguishable identity while keeping the game consistent.
- Make the ball, floor, hazards, collectibles, checkpoint, locked exit, unlocked exit, and final goal easy to distinguish.
- Include clear visual feedback for collection, damage, checkpoint activation, locked exits, stage completion, Game Over, and Victory.
- Keep gameplay UI readable and ensure it does not obscure important obstacles or objectives.
- Include sound effects or synthesized audio feedback where practical.
- The game must remain readable and playable at a standard 16:9 game resolution.

## 8. Project quality

- Organize the implementation clearly.
- You may choose the scene structure, runtime generation approach, prefabs, data representation, architecture, and validation methods.
- Do not leave editor-only simulation or debug shortcuts active in the final gameplay.
- Do not make the final game dependent on manually configured objects that are missing from the saved project.
- Ensure references and required content remain valid after reopening the project and after building.
- Avoid Console errors during normal play.
- Include a README with opening, running, controls, and build instructions.

## 9. Completion requirements

Before declaring completion:

- Confirm that the project opens and compiles without errors.
- Exercise the complete playable flow from Title through Victory.
- Exercise death before and after checkpoint activation.
- Exercise life exhaustion, Game Over, Restart, and a second run.
- Confirm collectible uniqueness, locked and unlocked goals, stage transitions, and state reset behavior.
- Confirm that gameplay uses actual player controls, collisions, and physics rather than directly teleporting the player or directly forcing completion states.
- Inspect the actual rendered gameplay at multiple points in the game.
- Build and test a WebGL version if the environment permits.
- Clearly distinguish directly tested behavior from anything that could not be tested.

Do not stop after creating scripts, passing compilation, or checking only isolated systems. Continue through implementation, execution, debugging, and refinement until the saved project contains the most complete playable result you can produce within this run.

When finished, report:

- the game concept and rules
- the main systems implemented
- the scene and runtime structure selected
- important files created or modified
- how to open and play the game
- controls
- validation and gameplay tests actually performed
- WebGL build and browser-test status
- remaining limitations and untested behavior