Zero-shot generalization
RL only
A policy trained only on Stage 1 reaching Stage 5 on the Lunatic difficulty.
One-credit clear
Behavioral cloning
A behavioral-cloning policy’s Lunatic 1CC.
Deathbomb
Behavioral cloning
A behavioral-cloning policy escaping death via use of a bomb.
Several years ago, I was obsessed with the idea of building an AI model to play video games automatically, inspired by projects such as the Mario genetic algorithm by SethBling and the many AI Learns to X videos by Code Bullet.
Three years ago, I started on a journey to do the same for Touhou, a bullet hell game where the goal is to survive the volleys of bullets coming at the player. My goal was to train an RL model that could clear the game on the highest difficulty, Lunatic, in one attempt. Players call this a one-credit clear, or 1CC.
One of the reasons it took me so long was how hard Touhou is to train a model on. As I mentioned in the TH10 Gym release post, where I discussed why I use Touhou as a testbed for RL, the game combines a long-horizon task with a hard credit assignment problem. For a 1CC, the agent needs to manage resources such as bombs across about 140,000 steps, the length of the agent’s own clear. A bomb clears the bullets on screen, but it costs power, which lowers the player’s firepower and leaves one less bomb to save itself with in later stages.
For context, Touhou 10: Mountain of Faith has six stages, which I’ll call Stage 1 to Stage 6. Each has waves of enemies and a boss that shoots volleys of bullets at the player. Players call a boss’s named attacks spell cards, but I’ll just call them boss attacks in this post.
Today, I’m excited to share my first major breakthrough toward that goal. This post covers, in chronological order:
- The initial model that couldn’t clear Stage 1, and how I diagnosed the issue
- A policy trained only on Stage 1, and its zero-shot generalization to Stages 2 to 5
- Achieving a 1CC with behavioral cloning
- Degraded performance from continuing to train the cloned policy with RL, and its diagnosis
Setup
Observation
One of my first inspirations for applying ML to Touhou was this video, which used captured gameplay images to choose the best action. My first attempt three years ago also used computer vision, but I quickly realized it would be infeasible to play the game in real time with the latency of running a vision model.
That’s why I built TH10 Gym, which I used to give the model the position, velocity, and size of:
- bullets
- lasers
- enemies
- items
on the current step and the three before it, about 50 milliseconds of history. I also gave the model the player’s position, lives, and power, which sets how many bombs it can use.
For encoding, I capped the bullet list at 64 and the lasers, enemies, and items at 24 each. Every bullet on screen also counts toward a coarse occupancy map of the playfield.
Figure 1The agent's observation on one step
- Bullet position across four steps
- Coarse occupancy map
- Enemy
- Player
| Bullet | Distance from the player | Speed | Heading |
|---|---|---|---|
| 1 | 30 px from the player | 4.5 px per step | down-right |
| 2 | 37 px from the player | 4.5 px per step | down-right |
| 3 | 24 px from the player | 4.5 px per step | down-right |
| 4 | 10 px from the player | 4.5 px per step | down-right |
| 5 | 55 px from the player | 4.5 px per step | down-right |
| 6 | 55 px from the player | 4.5 px per step | down-right |
| 7 | 15 px from the player | 4.5 px per step | down-right |
| 8 | 48 px from the player | 4.5 px per step | down-right |
| 9 | 44 px from the player | 4.5 px per step | down-right |
| 10 | 64 px from the player | 4.5 px per step | down-right |
| 11 | 62 px from the player | 4.5 px per step | down-right |
| 12 | 66 px from the player | 4.5 px per step | down-right |
| 13 | 25 px from the player | 4.5 px per step | down-right |
| 14 | 64 px from the player | 4.5 px per step | down-right |
| 15 | 16 px from the player | 4.5 px per step | down-right |
| 16 | 74 px from the player | 4.5 px per step | down-right |
| 17 | 39 px from the player | 4.5 px per step | down-right |
| 18 | 18 px from the player | 4.5 px per step | down-right |
| 19 | 81 px from the player | 4.5 px per step | down-right |
| 20 | 73 px from the player | 4.5 px per step | down-right |
| 21 | 79 px from the player | 4.5 px per step | down-right |
| 22 | 63 px from the player | 4.5 px per step | down-right |
| 23 | 82 px from the player | 4.5 px per step | down-right |
| 24 | 36 px from the player | 4.5 px per step | down-right |
| 25 | 74 px from the player | 4.5 px per step | down-right |
| 26 | 32 px from the player | 4.5 px per step | down-right |
| 27 | 96 px from the player | 5.7 px per step | down-right |
| 28 | 97 px from the player | 5.7 px per step | down |
| 29 | 98 px from the player | 5.8 px per step | down-right |
| 30 | 88 px from the player | 4.5 px per step | down-right |
| 31 | 64 px from the player | 4.5 px per step | down-right |
| 32 | 36 px from the player | 4.5 px per step | down-right |
| 33 | 36 px from the player | 4.5 px per step | down-right |
| 34 | 80 px from the player | 4.5 px per step | down-right |
| 35 | 39 px from the player | 4.5 px per step | down-right |
| 36 | 89 px from the player | 4.5 px per step | down-right |
| 37 | 99 px from the player | 4.5 px per step | down-right |
| 38 | 99 px from the player | 4.5 px per step | down-right |
| 39 | 42 px from the player | 4.5 px per step | down-right |
| 40 | 57 px from the player | 4.5 px per step | down-right |
| 41 | 100 px from the player | 4.5 px per step | down-right |
| 42 | 52 px from the player | 4.5 px per step | down-right |
| 43 | 111 px from the player | 5.7 px per step | down-right |
| 44 | 111 px from the player | 5.8 px per step | down-right |
| 45 | 103 px from the player | 4.5 px per step | down-right |
| 46 | 47 px from the player | 4.5 px per step | down-right |
| 47 | 113 px from the player | 5.7 px per step | down-right |
| 48 | 105 px from the player | 4.5 px per step | down-right |
| 49 | 55 px from the player | 4.5 px per step | down-right |
| 50 | 57 px from the player | 4.5 px per step | down-right |
| 51 | 60 px from the player | 4.5 px per step | down-right |
| 52 | 110 px from the player | 4.5 px per step | down-right |
| 53 | 119 px from the player | 4.5 px per step | down-right |
| 54 | 127 px from the player | 5.7 px per step | down-right |
| 55 | 120 px from the player | 4.5 px per step | down-right |
| 56 | 128 px from the player | 5.7 px per step | down-right |
| 57 | 65 px from the player | 4.5 px per step | down-right |
| 58 | 129 px from the player | 5.8 px per step | down-right |
| 59 | 124 px from the player | 4.5 px per step | down-right |
| 60 | 146 px from the player | 5.7 px per step | down-right |
Actions
At every step, the agent chooses to stand still or move in one of eight directions, at normal speed or in the slower focus mode, which players use for precise dodging. It also chooses whether to bomb. A bomb makes the player briefly invincible and clears the bullets on screen, at the cost of power, which weakens the player’s shot. Shooting is automatic and costs nothing.
Training
I trained the agent with PPO (Schulman et al., 2017) and GAE (Schulman et al., 2016) across eight parallel game instances, with a discount factor γ = 0.999 and λ = 0.99 for GAE.
Reward
One of my first reward functions was a sparse reward for clearing a stage. It quickly taught me the problem with sparse rewards: if the agent doesn’t clear the stage by chance during training, it has no reward to learn from.
The reward function I ended up with combines a reward for making progress with soft penalties for losing a life, which a 1CC run can afford, and for using a bomb.
The first model
The first policy stopped improving after 500,000 steps of training. I evaluated nine checkpoints from across the 2M steps of training with frozen weights, 108 attempts in total, and none of them cleared Stage 1.
Figure 2Steps per life over training
| Steps | Median steps per life | Clears |
|---|---|---|
| 0 | 195.9 | 0 of 6 |
| 250K | 267.4 | 0 of 6 |
| 500K | 311.9 | 0 of 6 |
| 750K | 289.5 | 0 of 6 |
| 1M | 322.6 | 0 of 6 |
| 1.25M | 302.3 | 0 of 6 |
| 1.5M | 268.3 | 0 of 6 |
| 1.75M | 292.8 | 0 of 6 |
| 2M | 262.7 | 0 of 6 |
Since the game runs at 60 Hz, the reward falls off quickly. GAE weights a reward’s contribution to an action’s advantage by γ and λ multiplied together for every step between them, so that weight halves about every 60 steps, or one second. For actions such as bombing, which can affect the outcome stages later, credit assignment is extremely difficult because the credit has decayed to almost nothing by then.
Figure 3Credit for a reward, by delay
However, a bad dodge can kill the player within a few steps, so I ruled out the decaying credit as the cause here. Another possible cause was the value function, which predicts the expected future reward. PPO uses these predictions to judge whether the outcome of an action was better or worse than expected, so if the value function wasn’t learning, the policy wouldn’t learn either. However, its predictions had a 60% explained variance, so I ruled this out too. The next place I looked was the input to the network.
Probing the encoder
As a proxy for how well the model perceives the game, I trained a classifier with the same architecture as the policy’s encoder to predict which moves would get the player hit in the next 4 and 8 steps. I labeled recorded states from all six stages by projecting every bullet forward along its current velocity.
At 4 steps, the original encoder found 99.7% of the unsafe moves on the states it trained on, but only 29% on held-out states. A modified encoder with more explicit features, such as how soon each bullet passes closest to the player and how close it gets, found 77%. I also tested an attention-based encoder, but it performed worse at 54% recall with three times more parameters. Every encoder did better at 8 steps, where the modified encoder reached 87%.
Figure 4Unsafe-move recall on training and held-out states
- Training
- Held-out
| Encoder | Parameters | Train, 4 steps | Held out, 4 steps | Train, 8 steps | Held out, 8 steps |
|---|---|---|---|---|---|
| Policy encoder | 97,794 | 100% | 29% | 99% | 48% |
| With geometric features | 107,202 | 99% | 77% | 99% | 87% |
| Attention encoder | 301,644 | 100% | 54% | 100% | 66% |
A forecast for each move
For each of the 18 moves the player can make, I encoded a forecast of how close the player would come to being hit in the next 2, 4, 8, and 16 steps if it held that move, assuming every projectile keeps its current velocity. The longest horizon, 16 steps, is about 267 milliseconds.
To compare the forecast encoder with the old one, I trained both to imitate a scripted controller, which I describe in the behavioral cloning section, and tested them on held-out states. The old encoder picked a move projected to hit the player in 4% of the states where a safe move existed. The forecast encoder never did.
Figure 5Forecast and policy probability for each move
The 2 moves that run into a bullet get 0% combined.
The 7 moves that run into a bullet get 0% combined.
- Bullet path over 8 steps
- Player position after 8 steps of each move, normal speed outside, focused insideClearPasses closeRuns into a bullet
- Policy probability
| Frame | Move | Forecast | Probability |
|---|---|---|---|
| Calm | Stay put | Clear for 8 steps | 0% |
| Calm | Up | Clear for 8 steps | 0% |
| Calm | Down | Clear for 8 steps | 0% |
| Calm | Left | Clear for 8 steps | 0% |
| Calm | Right | Runs into a bullet within 8 steps | 0% |
| Calm | Up-left | Clear for 8 steps | 0% |
| Calm | Up-right | Runs into a bullet within 8 steps | 0% |
| Calm | Down-left | Clear for 8 steps | 0% |
| Calm | Down-right | Clear for 8 steps | 0% |
| Calm | Focused stay put | Clear for 8 steps | 0% |
| Calm | Focused up | Clear for 8 steps | 0% |
| Calm | Focused down | Clear for 8 steps | 41% |
| Calm | Focused left | Clear for 8 steps | 0% |
| Calm | Focused right | Passes within 16 px of a bullet | 0% |
| Calm | Focused up-left | Clear for 8 steps | 59% |
| Calm | Focused up-right | Passes within 16 px of a bullet | 0% |
| Calm | Focused down-left | Clear for 8 steps | 0% |
| Calm | Focused down-right | Clear for 8 steps | 0% |
| Dense | Stay put | Clear for 8 steps | 97% |
| Dense | Up | Runs into a bullet within 8 steps | 0% |
| Dense | Down | Runs into a bullet within 8 steps | 0% |
| Dense | Left | Passes within 16 px of a bullet | 0% |
| Dense | Right | Runs into a bullet within 8 steps | 0% |
| Dense | Up-left | Runs into a bullet within 8 steps | 0% |
| Dense | Up-right | Runs into a bullet within 8 steps | 0% |
| Dense | Down-left | Runs into a bullet within 8 steps | 0% |
| Dense | Down-right | Runs into a bullet within 8 steps | 0% |
| Dense | Focused stay put | Clear for 8 steps | 3% |
| Dense | Focused up | Passes within 16 px of a bullet | 0% |
| Dense | Focused down | Passes within 16 px of a bullet | 0% |
| Dense | Focused left | Passes within 16 px of a bullet | <1% |
| Dense | Focused right | Passes within 16 px of a bullet | 0% |
| Dense | Focused up-left | Passes within 16 px of a bullet | 0% |
| Dense | Focused up-right | Passes within 16 px of a bullet | 0% |
| Dense | Focused down-left | Passes within 16 px of a bullet | 0% |
| Dense | Focused down-right | Passes within 16 px of a bullet | 0% |
One limitation of this encoder is the constant-velocity assumption, which doesn’t hold for curving bullets. Training the policy with RL partly makes up for this, because the policy learns from experience.
Zero-shot generalization and reaching Stage 5
With the new encoder in place, I reran the training from randomly initialized weights against Stage 1, and the model learned to consistently beat Stage 1. With its weights frozen after 2M training steps, it cleared all eight evaluation attempts.
Figure 6Stage 1 clear rate over training, from random weights
- Clear rate during training
- Frozen evaluation
| Steps | Frozen clears | Rate during training |
|---|---|---|
| 0 | 0 of 8 | — |
| 2M | 8 of 8 | 38% |
After it succeeded on Stage 1, I ran the same frozen weights on full Lunatic runs. The policy had only ever trained on Stage 1, but its play generalized zero-shot to stages it had never been trained on, and in the run below it reached Stage 5 before losing its last life.
RL only
The RL-only policy in a Stage 3 boss attack
In Touhou, when a player uses a bomb to save themself from a situation where there is no escape from the incoming bullet volley, it is called a deathbomb. In 105 recorded instances where a deathbomb could have saved the agent, the bombing action only had an average probability of less than 1%.
Figure 7Bomb probability on each of the 105 deathbomb states
| State | Stage | RL only, 2M steps | RL only, 2.5M steps |
|---|---|---|---|
| 1 | 1 | 0.06% | 4% |
| 2 | 1 | 3% | 89% |
| 3 | 1 | 0.11% | 0.87% |
| 4 | 1 | 0.00% | 4% |
| 5 | 1 | 7% | 0.04% |
| 6 | 1 | 22% | 0.05% |
| 7 | 3 | 0.05% | 81% |
| 8 | 3 | 0.12% | 93% |
| 9 | 4 | 0.00% | 57% |
| 10 | 4 | 0.01% | 77% |
| 11 | 4 | 0.00% | 48% |
| 12 | 4 | 0.93% | 96% |
| 13 | 4 | 0.00% | 5% |
| 14 | 4 | 0.00% | 3% |
| 15 | 4 | 0.46% | 93% |
| 16 | 4 | 0.39% | 92% |
| 17 | 4 | 0.00% | 2% |
| 18 | 4 | 0.00% | 3% |
| 19 | 4 | 2% | 96% |
| 20 | 4 | 0.60% | 94% |
| 21 | 4 | 0.00% | 33% |
| 22 | 4 | 0.00% | 68% |
| 23 | 4 | 0.00% | 85% |
| 24 | 4 | 0.00% | 44% |
| 25 | 5 | 0.11% | 3% |
| 26 | 5 | 0.06% | 0.82% |
| 27 | 5 | 0.17% | 4% |
| 28 | 5 | 2% | 70% |
| 29 | 5 | 0.05% | 2% |
| 30 | 5 | 0.04% | 0.59% |
| 31 | 5 | 0.14% | 5% |
| 32 | 5 | 0.28% | 12% |
| 33 | 5 | 0.22% | 10% |
| 34 | 5 | 0.54% | 29% |
| 35 | 5 | 0.37% | 27% |
| 36 | 5 | 0.11% | 80% |
| 37 | 5 | 0.09% | 84% |
| 38 | 5 | 0.06% | 0.91% |
| 39 | 5 | 0.14% | 6% |
| 40 | 5 | 0.16% | 6% |
| 41 | 5 | 0.01% | 0.23% |
| 42 | 5 | 0.01% | 0.21% |
| 43 | 5 | 0.18% | 16% |
| 44 | 5 | 0.22% | 14% |
| 45 | 5 | 11% | 84% |
| 46 | 5 | 1% | 83% |
| 47 | 5 | 0.06% | 2% |
| 48 | 5 | 0.07% | 1% |
| 49 | 5 | 0.24% | 6% |
| 50 | 5 | 0.24% | 6% |
| 51 | 5 | 0.10% | 2% |
| 52 | 5 | 0.05% | 0.79% |
| 53 | 5 | 0.12% | 4% |
| 54 | 5 | 0.18% | 6% |
| 55 | 5 | 5% | 38% |
| 56 | 5 | 3% | 56% |
| 57 | 6 | 0.02% | 3% |
| 58 | 6 | 0.00% | 0.30% |
| 59 | 6 | 0.00% | 2% |
| 60 | 6 | 0.00% | 8% |
| 61 | 6 | 0.00% | 10% |
| 62 | 6 | 0.09% | 21% |
| 63 | 6 | 0.11% | 22% |
| 64 | 6 | 0.00% | 1% |
| 65 | 6 | 0.00% | 0.68% |
| 66 | 6 | 0.00% | 6% |
| 67 | 6 | 0.00% | 8% |
| 68 | 6 | 0.04% | 4% |
| 69 | 6 | 0.16% | 52% |
| 70 | 6 | 0.00% | 45% |
| 71 | 6 | 0.00% | 26% |
| 72 | 6 | 0.00% | 0.23% |
| 73 | 6 | 0.09% | 21% |
| 74 | 6 | 0.12% | 23% |
| 75 | 6 | 0.01% | 11% |
| 76 | 6 | 0.02% | 2% |
| 77 | 6 | 0.06% | 9% |
| 78 | 4 | 0.00% | 32% |
| 79 | 4 | 0.00% | 12% |
| 80 | 4 | 1% | 87% |
| 81 | 4 | 1% | 96% |
| 82 | 4 | 0.01% | 81% |
| 83 | 5 | 0.01% | 0.14% |
| 84 | 5 | 0.20% | 16% |
| 85 | 5 | 0.23% | 15% |
| 86 | 5 | 0.00% | 0.05% |
| 87 | 5 | 0.00% | 0.06% |
| 88 | 5 | 0.40% | 4% |
| 89 | 5 | 1% | 59% |
| 90 | 5 | 0.02% | 0.27% |
| 91 | 5 | 0.03% | 2% |
| 92 | 5 | 0.16% | 3% |
| 93 | 5 | 1% | 70% |
| 94 | 5 | 0.64% | 31% |
| 95 | 5 | 0.01% | 0.20% |
| 96 | 5 | 0.02% | 1% |
| 97 | 5 | 0.21% | 14% |
| 98 | 5 | 0.11% | 55% |
| 99 | 5 | 0.24% | 43% |
| 100 | 5 | 0.01% | 0.26% |
| 101 | 5 | 0.01% | 0.20% |
| 102 | 5 | 0.06% | 2% |
| 103 | 5 | 0.25% | 17% |
| 104 | 5 | 0.00% | 0.03% |
| 105 | 5 | 0.00% | 0.07% |
My hypothesis was that further training would teach the agent to deathbomb. After another 500,000 steps, the average bomb probability on those states did rise to 26%, but the clear rate on frozen evaluations dropped from 8 of 8 to 1 of 8.
The value function seems to have learned some mechanics of the game, such as impending danger, with sharp drops where deathbombs saved the agent, but the policy’s use of the bomb is still not consistent. In the attempt below, the agent lost a life between its two deathbombs and did not use a bomb even though the value estimate predicted impending danger.
Figure 8Value estimate over one Stage 1 attempt
- Value estimate
- Hit
- Bomb
| Time | Event | Value, 0.5 s before | Value at the event |
|---|---|---|---|
| 11.4 s | hit, cancelled by a deathbomb | 61 | 27 |
| 13.9 s | bomb | 56 | 69 |
| 21.7 s | hit, life lost | 63 | 42 |
| 24.5 s | bomb | 47 | 65 |
| 25.4 s | hit, cancelled by a deathbomb | 63 | 11 |
Behavioral cloning
The RL-only policy rarely deathbombed and could not clear the full game, so I tried behavioral cloning (Pomerleau, 1988) instead. I wrote a scripted controller that plays from the game state alone and trained the forecast network to copy its actions.
The cloned policy cleared a normal Lunatic run as Reimu B with four spare lives and 26,332,605 points, the 1CC I set out to achieve three years ago. It cleared the game again in a later run, shown in the full video below.
Behavioral cloning
The final boss’s last attack and the ending
Fine-tuning the cloned policy
I wanted to see if continued RL training could improve the cloned policy, and it didn’t. After 250,000 additional steps on Stage 1, the fine-tuned policy cleared 21 of 24 attempts, down from 24 of 24.
I attributed the decreased performance to worse deathbombing decisions. Out of 44 dangerous situations, the cloned policy saved 38 by deathbombing while the fine-tuned policy only saved 16.
Behavioral cloning
A deathbomb at half speed
To find out why, I first looked at the gradients. The policy and value function share most of their layers, with separate output heads. Behavioral cloning doesn’t train the value head, so its predictions were far off when PPO started. In the first update, the value loss’s gradient on the shared layers was about 1,400 times the policy loss’s.
Figure 9Gradient norm on the shared layers, by loss
- Value loss
- Policy loss
| Steps | Value loss | Policy loss | Ratio |
|---|---|---|---|
| 1,000 | 24.8 | 0.0180 | 1,373× |
| 2,000 | 17.0 | 1.86e-7 | 9.2e7× |
| 3,000 | 0.572 | 0.0106 | 54× |
| 4,000 | 0.0269 | 0.0272 | 0.99× |
| 5,000 | 0.0524 | 0.000801 | 65× |
| 6,000 | 0.0283 | 0.000514 | 55× |
| 7,000 | 0.0379 | 0.0568 | 0.67× |
| 8,000 | 0.0240 | 0.0168 | 1.4× |
| 9,000 | 0.0367 | 0.00253 | 14× |
| 10,000 | 0.0745 | 0.0475 | 1.6× |
| 11,000 | 0.0397 | 0.0184 | 2.2× |
| 12,000 | 0.0367 | 0.0885 | 0.41× |
To test whether the value gradient was what removed the deathbombs, I measured the deathbomb response: how often bombing is the top action on the 105 deathbomb states from earlier. After 6,000 steps of PPO, it fell from 85% to 47%. Training on the value loss alone, with the policy loss turned off, dropped it to the same 47%, while blocking the value gradient from the shared layers kept it at 76%.
Figure 10Deathbomb response after 6,000 steps under each ablation
| Run | States | Share |
|---|---|---|
| Cloned policy | 89 of 105 | 85% |
| Full PPO | 49 of 105 | 47% |
| Value loss only | 49 of 105 | 47% |
| Value gradients blocked | 80 of 105 | 76% |
Blocking the value gradient only delayed the decline. In a longer run, the response almost disappeared by 16,000 steps.
Limitations
Some of the presented metrics came from small samples, as frozen evaluations are quite expensive to run. Each setup was also trained once, so I can’t separate the effect of a change from run-to-run variance.
I haven’t tried the usual fix for an untrained value head, which is training it on the cloned policy’s play before PPO updates the policy, or keeping the fine-tuned policy close to the cloned policy with a KL penalty. The RL-only policy has also only trained on Stage 1, never on full runs where a bomb spent now is gone later.
What I learned
Throughout this project, I learned that:
- RL is very sample inefficient. This is why I focused on making TH10 Gym fast, so I could collect samples as quickly as possible.
- Computing bullet position forecasts before passing them to the encoder drastically cut the training the network needed. The original encoder never cleared Stage 1 in 2M steps, and the forecast network first cleared it at about 0.6M.
- Training metrics can hide problems that only show up in frozen evaluation.
References
- Dean A. Pomerleau. ALVINN: An Autonomous Land Vehicle in a Neural Network. In Advances in Neural Information Processing Systems, volume 1, pages 305–313, 1988.
- John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. In International Conference on Learning Representations (ICLR), 2016.
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347, 2017.