When Open Models Beat Astra and Fable and When They Don’t?

Open-weight models such as GLM-5.3 and Kimi K3 can offer better value than Astra or Fable when they meet a workload’s quality requirements at a lower cost. We evaluated 12 configurations of nine models on SWE-bench Verified and Terminal-Bench 2.1, comparing accuracy, mean cost per task, completion time, and task-level behavior. On SWE-bench Verified, GLM-5.3-Flash achieved 89.6% accuracy at approximately $0.02 per task, compared with Astra’s 87.0% at $1.00. GLM-5.3 reached 94.2% at $0.18, while Kimi K3 reached 93.8% at $0.77. The picture changes on Terminal-Bench 2.1, where Astra and Fable were more accurate than Kimi K3, though still more expensive per task.

Aggregate benchmark scores hide as much as they show, so we also break down the results by problem type to highlight models’ relative strengths. We also compare output quality across four design tasks, examining how models translate the same requirements into different visual and interactive experiences. Together, these comparisons show where GLM-5.3 and Kimi K3 match or beat Astra and Fable, and where a proprietary model’s advantages still justify its price. For application builders, the goal is not to pick a universal winner, but to find the model that meets a specific workload’s quality, cost, and speed requirements.

Recreating Manhattan with GLM-5.3 and Astra

A model’s capabilities are easier to appreciate through the work it produces. We asked GPT-6 Astra and GLM-5.3 to recreate Midtown Manhattan in Blender and animate a first-person flight between its buildings, inspired by Spider-Man’s movement through New York. We used Kilo Code as the agent harness for both models. The task demands 3D modeling, scene composition, and camera animation at once. The animations show how each model interpreted and built the scene.

The still images reveal differences in modeling detail. GLM-5.3 captured the overall structure through a dense skyline, varied building heights, and long street corridors. Astra produced more consistent architectural detail, particularly in window placement and road markings. GLM demonstrated strong scene-building capability, while Astra delivered a more polished result.

This example makes GLM-5.3 worth considering for demanding work. To examine the practical trade-offs more systematically, we next compare open- and closed-weight models on coding benchmarks, measuring accuracy, cost per task, and completion time.

Accuracy, Cost, and Speed Favor Different Models

We evaluated 12 configurations of nine models using mini-swe-agent for 500 SWE-bench Verified tasks and Terminus 2 for 89 Terminal-Bench 2.1 tasks. For each configuration, we compared accuracy, mean cost per task, and median completion time. When provider guardrails blocked a task, Fable, Astra, and GPT-5.6 Sol fell back to a secondary model, and we include those results below.

Fable achieved the highest accuracy on both benchmarks, but open-weight models offered strong alternatives at lower cost. On SWE-bench Verified, GLM-5.3 reached 94.2% accuracy at a mean cost of $0.18 per task, compared with Fable’s 97.8% at $2.70. For workloads where that accuracy difference is acceptable, GLM-5.3 offers a substantial reduction in inference cost.

GLM-5.3-Flash provided another compelling option. It achieved 89.6% accuracy at approximately $0.02 per task, above Astra’s 87.0% at $1.00; here, the cheaper model also solved more tasks. Completion time adds another consideration. On Terminal-Bench 2.1, GLM-5.3 outperformed Astra on accuracy (86.5% to 85.4%) but took a median 20.36 minutes per task versus Astra’s 3.63 minutes. A workload that tolerates longer execution may favor GLM’s accuracy and cost, while one that requires faster responses may favor Astra.

Figure 4 brings accuracy and cost into the same view, making it easier to identify configurations that deliver stronger results within a given budget.

GLM-5.3-Flash and GLM-5.3 offer two compelling options. On SWE-bench Verified, Flash achieved 89.6% accuracy at approximately $0.02 per task, compared with Astra’s 87.0% at $1.00. In this evaluation, the cheaper configuration also solved more tasks. GLM-5.3 reached higher accuracy at 94.2%, with a mean cost of $0.18 per task, compared with Fable’s 97.8% at $2.70. Fable’s accuracy advantage also came with a higher measured cost on Terminal-Bench. It achieved 92.1% accuracy at a mean cost of $65.53 per task, compared with GLM-5.3’s 86.5% at $0.79.

Similar Scores Hide Different Strengths

Kimi K3 and GPT-6 Astra illustrate how the preferred model flips with the workload. On SWE-bench Verified, Kimi achieved higher accuracy than Astra (93.8% versus 87.0%) at a lower mean cost per task ($0.77 versus $1.00). However, its median completion time was more than twice as long, 7.78 minutes versus 3.27 minutes. On Terminal-Bench 2.1, the ranking reversed: Astra was more accurate (85.4% versus Kimi’s 80.9%) and faster (3.63 minutes versus 7.23 minutes), while Kimi stayed cheaper at $0.57 per task against Astra at $1.33. Kimi therefore offered the stronger accuracy-cost combination on SWE-bench, while the Terminal-Bench posed a clearer trade-off between Kimi’s lower cost and Astra’s higher accuracy and speed.

GLM-5.3 and GPT-5.6 Sol, both at maximum reasoning effort, illustrate how similar aggregate scores can conceal different strengths. On Terminal-Bench 2.1, they solved 77 and 76 of 89 tasks, respectively. Both solved 67 tasks, while GLM solved 10 that Sol missed and Sol solved nine that GLM missed. A one-task gap in total successes was hiding 19 tasks where the two models disagreed.

Breaking the results down by category locates those differences. On SWE-bench Verified, GLM-5.3 solved 33 of 37 parsing tasks versus Sol’s 31; Sol took rendering and visual problems, solving all 66 to GLM’s 62, and also led on tracing and compatibility problems. On Terminal-Bench, GLM solved 15 of 16 ML tasks to Sol’s 13, while Sol solved seven of eight science tasks compared to GLM’s five. Both solved 36 of 41 software tasks, although their successful task sets differed. The practical lesson is to choose evaluation tasks that look like your own.

A pass-or-fail score also leaves out what each attempt cost in time and tokens. The distributions show both typical resource use alongside the long tails, where individual tasks consumed far more than the median. Two models with matching accuracy can therefore place very different demands on an execution budget and a response-time target.

Taken together, these breakdowns do what an aggregate score cannot: task overlap shows where models succeed differently, domain results tie those differences to a real workload, and resource distributions expose variability an average hides. All of it, though, rests on tasks with defined success criteria. Open-ended design has none. We turn there next, where a model has to interpret a brief and decide on visual presentation, information hierarchy, and interaction. Comparing those finished experiences reveals another set of capabilities that benchmark accuracy alone does not capture.

Open-Weight Models Deliver Competitive Design Results

In our design experiments, GLM-5.3 and Kimi K3 turned open-ended briefs into coherent visual and interactive experiences. Models often converged on similar design directions; differences showed up in visual refinement, information presentation, and interaction quality.

We ran four website-design tasks covering transit information, brand identity, interactive typography, and a Steam redesign. For each, the participating models received the same brief and supporting materials in separate sandboxes, then worked without further human direction. All four tasks used Kilo Code as the agent harness. The comparisons that follow are qualitative.

The transit task asked models to turn 358KB of San Francisco transit data into a useful interface. Astra, Fable, GLM-5.3, and Kimi K3 organized it around routes, stations, and transit modes, arriving at broadly comparable information structures. Fable and Astra surfaced more detail in the overview, while GLM labeled station names clearly on its map. Kimi required more navigation to reach some information. The gap wasn't in the approach, but in how quickly a user could find what they wanted.

The Himalayan Harvest task pushed all four models in the same visual direction: warm colors, expressive typography, and storytelling around ingredients and origin. Kimi K3 and GLM-5.3 built coherent brand identities that stayed consistent with the brief. Astra and Fable produced more refined product imagery and illustration, adding an extra layer of polish to their pages.

On the interactive typography task, Kimi K3 delivered the core interactive experience the prompt asked for, including 3D interaction, particles, rotation, zoom, and motion. Astra and Fable had the edge in refinement, particularly in lighting, material quality, and how natural the interactions felt. But Kimi implemented the requested experience end to end rather than dropping major parts of the brief, at a total generation cost of $1.66.

Kimi K3’s Steam redesign balanced information density with a clear browsing structure. Prominent game artwork, prices, review data, and navigation supported exploration while keeping the page easy to scan. Fable’s interface was denser. For a storefront built around discovering and comparing games, Kimi’s balance of detail and readability was the more effective choice. Across these examples, GLM-5.3 and Kimi K3 proved capable of practical design work. Together with the benchmark results, that is a concrete reason to put open-weight models in your evaluation set.

The Right Model Depends on the Workload

GLM-5.3, GLM-5.3-Flash, and Kimi K3 belong in the same conversation as Astra and Fable. In our SWE-bench Verified evaluation, all three were more accurate than Astra at a lower mean cost per task. The task-level comparisons showed that models with near-identical scores solve different problems, and the creative examples showed GLM and Kimi turning demanding briefs into coherent visual and interactive work. The proprietary models kept meaningful advantages. Fable was the most accurate on both coding benchmarks, while Astra was faster than GLM-5.3 and Kimi K3.

Model rankings also keep changing as new models arrive. Play the timeline below to watch new releases enter the comparison and reorder the ranking, then switch benchmarks to see how the order changes again. Different metrics and cost tiers bring different models to the top. Try following GLM, Kimi, or another model family through the timeline.

When an open-weight model delivers the quality your workload needs at an acceptable speed, its lower inference cost can make it the better choice. GLM-5.3, GLM-5.3-Flash, and Kimi K3 give you concrete reasons to test that possibility. Define the quality and completion-time requirements they must meet, run candidates on representative tasks from your application, and compare the cost of achieving those outcomes.

Try GLM-5.3 and GLM-5.3-Flash through FriendliAI Model APIs and evaluate them on your own workloads. Compare output quality, cost per task, and completion time to find the model that fits your application. → Get started with FriendliAI