Vals Hacks / Build Your Own Eval / November 15, 2026

RoyaleBench

Updated November 15, 2026v9@1

RoyaleBench evaluates language models on 21 fixed Clash Royale positions. Models choose cards, coordinates, and timing. An independent replay verifies each outcome.

21fixed, replay-graded tasks
12models · one full run each
66.7%highest observed objective success
4tasks passed by 0 of 12 models

Overview

Clash Royale is more than a game; it is our team’s childhood. As players ourselves, we know how difficult this game is: resource management, exploding game states, and strategic planning with imperfect information.

We have seen LLM’s play chess and poker, but we wanted to make a more unique benchmark relevant to today’s cultural zeitgeist. That benchmark is what we have to show today–RoyaleBench.

We represented 21 unique tasks using Harbor task format, Valkyrie framework, and modal compute. We isolated agents in their own game environments and gated decisions w/ a referee that determines move legality.

Our best performing model was Opus 5.5, with a pass rate of 66.7%. The average performance was 36% across 12 different LLM models.

This benchmark highlights how LLM’s still lack understanding in complex scenarios with many variables: different cards, unique interactions, and long term planning. We hope that this benchmark serves as a goal for labs to target when developing and modernizing future models. And one day, maybe we will have an LLM capable of beating Ojas (our talented friend).

Background

Clash Royale requires managing elixir, choosing deployment positions and timing, and balancing defense with tower pressure. RoyaleBench tests these decisions in fixed mid-match continuations in a deterministic simulator, verified by independent replay.

Resource management

Decide when to spend elixir and when to reserve it.

Spatial reasoning

Choose where to place cards. Invalid positions aren't corrected.

Planning and adaptation

Defend pushes and turn surviving units into counterattacks.

Results

Each model attempts all 21 tasks once on v9@1. Incomplete play scores zero. No infrastructure errors were recorded.

RoyaleBench v9@1 / Objective success
# Model Harness Passed Incomplete Success Cost / task Cost / move Agent time / task

Results compare model-and-harness configurations. The four mini-SWE-agent runs use a 12-step limit; 33/84 attempts are incomplete. Model IDs, run settings, and per-task measurements are in the results snapshot.

Success by task type

Pass rate within each task type, with passed / total counts. The harness filter applies.

Claude Code Codex mini-SWE-agent

One attempt per model per task. Types use the existing task labels.

Cost and outcome

Compare inference cost per task or recorded agent move. The harness filter also applies to this chart.

RoyaleBench — Pareto Curve
USD per task (logarithmic) · objective success · one full run per point.

Anthropic costs are harness-reported. Codex costs use recorded tokens and the saved November 15 rate card, including the lower 30-minute cache-write rate because retention is unknown. Modal-hosted costs use recorded tokens and workspace inference rates. Infrastructure and invoice adjustments are excluded.

Tasks

Each task fixes the decks, hand, elixir, tower health, units, seed, opponent settings, and scoring window. The model continues from that position.

Scoring: 12 tasks require terminal match wins; nine use other objectives, including damage limits and constrained defenses. Each task contributes one binary reward.

v9@1 · 21 tasks

Task Capability Objective Passed

Inside one task

The Hog body-block starting position Schematic, not model input. An enemy Hog Rider at tile 2.5, 18.5 approaches the player's left tower, which has one hit point. The player has three elixir and no troops on the board. H HOG · 1697 HP 1 HP YOUR SIDE
Example / 05-hog-body-block

Defend a 1-HP tower

Your left tower has 1 HP. A full-health Hog Rider is approaching. Keep all three towers alive and take zero additional tower damage over the next 16 seconds, spending at most three elixir.

3 elixir total16-second window0 tower damage
Hand
Ice Golem (2), Skeletons (1), Hog Rider (4), Musketeer (4)
Decision
Inspect the text state, then choose a hand slot and coordinates, or wait.
Clock
Each decision advances 0.5 seconds. Thinking does not advance the clock.

Cannon is next in the draw queue, but cycling to it and playing it would exceed the total spending limit. There are no further enemy deployments during this task.

Read the complete task instruction ↗

What the model receives

A text observation with its hand, next card, elixir, tower HP, visible units, recent events, and time left. The opponent's hand and elixir remain hidden. Arena graphics on this page visualize the recorded state; the evaluated models do not receive these images.

How it acts

cr                 # inspect; no time advances
cr play SLOT X Y   # deploy at chosen coordinates
cr wait            # advance one decision interval

Slots are zero-based. Coordinates use the player's frame on an 18 × 32 tile arena. Task-specific intervals are 0.5, 1, or 2 seconds.

Gameplay case study

Opus vs. reference

The model and the validated reference play the same six cards in the same order, at the same engine ticks, spending 15 elixir. The model loses its tower. The reference wins without taking additional tower damage.

This case is from the saved Opus v6@1 run, not its v9 leaderboard run. V9 retains the task's original package. The position starts with 18 seconds of overtime left, a Giant approaching a 250-HP friendly tower, and a 450-HP enemy target.

04-critical-cycle-and-connectOriginal recordings · shared elapsed-time cursor
Claude Opus 5.5Loss · 0–1
Validated referenceWin · 1–0
0.00 / 13.20s

Both recordings start at the same position. Play or scrub to compare the sequences. Read model trace ↗

Play Card Opus coordinates Reference coordinates
1 Ice Spirit (3.5, 11.0) (3.5, 10.0)
2 Cannon (7.5, 10.5) (7.0, 11.5)
3 Fireball (3.5, 25.5) (3.5, 25.5)
4 Hog Rider (3.5, 15.0) (3.5, 15.0)
5 Log (3.5, 24.5) · offense (4.982, 8.623) · defense
6 Ice Spirit (4.4, 9.0) (4.495, 9.766)

Coordinates are tiles in the player's frame, converted at 18,000 engine units per tile. Differing placements are shaded. Several placements change together, so this comparison does not establish one action as the sole cause.

Behavioral evidence

Failure analysis

All 252 attempts pass replay integrity, with 14 rejected actions across 1,584 recorded decisions.

Of 161 failed objectives, 127 complete the scenario and 34 stop early. The examples below use independently replayed policies and controlled probes.

Threat prioritization

Defend before committing to offense

Ten models spend Fireball on a tower it cannot destroy and lose to the incoming Balloon. Fable 5.2 uses Cannon to pull the Balloon away, buying time for its Giant to finish the tower.

Tower race B · V9 · 1/12 passes

Placement

Use knockback to delay the next attack

Sol's Fireball hits the Balloon but does not delay its next attack enough to protect the tower. Changing only the aim's vertical position makes the same policy pass.

Balloon Fireball · V9 · isolated aim counterfactual

Resources

Reserve counters for later threats

Opus and Fable 5.2 spend Log on the right lane while a disclosed Goblin Barrel approaches the left tower. The passing reference handles earlier threats with other cards and saves Log for the Barrel.

Reserve the counter · V9 · 0/12 passes

Timing

Time deployments to block the Hog's path

Ten models deploy Ice Golem and Skeletons starting immediately. The passing reference waits for the Hog to advance. Moving its deployments half a second earlier, with placements unchanged, loses the tower.

Hog body-block · V9 · controlled timing shift

Methodology

  1. 1Observe

    The referee exposes the player's text view of the fixed position.

  2. 2Decide

    The model selects a card and coordinates, or waits. Thinking pauses time.

  3. 3Advance

    The engine applies actions and advances the task's interval. The loop repeats.

  4. 4Replay

    A separate verifier checks the log, ticks, decisions, and resulting state.

  5. 5Score

    Reward is one only when integrity passes and every objective is met.

Opponent: the Planner

Planner is a deterministic search bot with a ten-second look-ahead, three plies, four top candidates, and three responses per candidate. It reads the true engine state, including both hands and elixir. Some tactical windows permit no further enemy plays.

Task construction

Tasks use the Harbor directory format, run through Valkyrie in Modal sandboxes, and expose the protected cr interface. A separate referee owns the engine state and action log. The verifier replays the commands independently; there is no model-based judge.

Scoring and variance

Score = 100 × objectives passed / 21. Spending, damage, crowns, and match outcomes are diagnostics unless the task explicitly uses them. A terminal win can fail a stricter task objective; a defensive-window success need not end the match.

Each model has one recorded attempt per task. These results do not estimate run-to-run variance.

Limitations

  • One attempt per task. There are no estimates of run-to-run variance or uncertainty intervals.
  • Tasks were developed using model feedback. Earlier cohorts and harder variants were selected partly from Opus and Sonnet behavior; this is not a held-out evaluation.
  • Harnesses differ. Models use Claude Code, Codex, or mini-SWE-agent, with recorded harness-specific settings. The four mini-SWE runs have step_limit=12; 62/84 tasks terminate at that limit, and 33/84 are incomplete. Some scenarios require more moves than that budget supports under one-move-per-call play.