Today, I’m releasing TH10 Gym, a Gymnasium environment for Touhou 10: Mountain of Faith, intended for use in reinforcement learning experiments and research.
What is RL?
Reinforcement learning (RL) is a technique in machine learning where an agent learns by interacting with an environment and gets feedback through a reward function.
Why Touhou?
In my opinion, Touhou is one of the more challenging environments for RL. It combines a long-horizon task with complications such as the credit assignment problem and a fairly complex observation space.
For reference, the game runs at 60 FPS, and the Lunatic speedrun record for Marisa B at the time of writing is 13 minutes and 54.6 seconds. This works out to roughly 50,000 game ticks from start to finish. If the agent received a reward only for clearing the game, even with a discount factor of 0.9999 per tick, that reward would carry about 0.67% of its original weight at the start of the run.
On Lunatic difficulty, the agent has to learn strategies such as baiting volleys of bullets to one side of the screen, then moving away to shoot at the enemies. If it miscalculates, it can get cornered by bullets and lose a life.
Using a bomb is also an interesting optimization problem. A bomb can clear the screen of bullets, but it costs power, which can weaken the character’s shots later in the run.
Overall, I think it’s a challenging game and RL environment.
Speeding up sample collection
RL can require a lot of data to train an agent. At 60 samples per second, it would take 4.6 hours just to collect one million samples. In my initial testing, that wasn’t enough to train an agent to beat the game. I looked into three ways to raise the sampling rate:
- Lifting the game’s default frame-rate limit
- Running environments in parallel
- Disabling rendering
With these improvements, I reached around 1,200 samples per second across eight parallel environments in my collection tests.
Challenge
With the release of this Gymnasium environment, I’m excited to see what you build with it. If you manage to train an agent to beat the game, please open a GitHub issue with a gameplay demo and a short writeup. I’m always open to learning from others, and I think this project is a great chance to do that.
Acknowledgements
- I used Reverse Engineer Anything while developing the environment to understand how the game stores entity attributes in memory.
- twinject was my initial inspiration for trying to beat Touhou algorithmically.