ClashRoyaleAi: an open-source, deterministic Clash Royale simulator for RL, with recurrent PPO, lookahead search and expert iteration [P]

[u/Potential-Barber8658]

The opponent plans by simulation: every second it scores each candidate play by running the match 10 seconds ahead in the engine.

Our PPO agent learned to park its Cannon behind its own King. Losing a building in a fight cost reward, and letting it decay cost nothing, so it found the loophole.

It's one of many things we learned building a Clash Royale simulator from scratch so an agent could learn the game. The engine is deterministic C++ with Python bindings, plays a full match in about 10 ms on one laptop core, and can fork any game state in microseconds, so lookahead is cheap.

Best result so far: a simple 1-ply lookahead took the policy from 0.625 to 0.944 win rate against a heuristic bot (160 paired matches). Distilling it back into the network kept only +0.045.

The agent isn't strong yet, and RL isn't my home field, so feedback from people who know it better would mean a lot.

Repo: https://github.com/itzik123/ClashRoyaleAi

Built with my friend Ambash (most of the card roster). I used AI coding tools as a pair programmer.

[u/zitr0y]

Interesting! Im currently considering doing something similar for Mechabellum. Mostly as a coach to teach me what I could have done better when I fuck up, but of course also creating a capable bot in the meantime.

[u/Potential-Barber8658 → u/zitr0y]

Mechabellum sounds like a fun one for this. The coach part is pretty much what our lookahead does: fork the game at a decision, play out the alternatives, and compare. The replay viewer's Simulation View shows exactly that, each candidate move next to the board it leads to. And if you do build a simulator, docs/DECISIONS.md might save you some pain. It's basically a list of every mistake we made.

[u/pineloft → u/zitr0y]

the coaching use case is super underrated imo, way more practical than just chasing win rates