[Game] World Agents

Yesterday I was skimming HackerNews when I noticed an article titled ‘A Perceptron in Age of Empires II’, this triggered a bit of nostalgia, both because Age of Empires 2 was a game I spent quite a bit of time back in the day but also ‘Perceptrons’.

Back in the day I was going through Bishop’s ‘Neural Networks for Pattern Recognition’, I was working on a signal processing logic to translate a set of sensor signals into a probability distribution aiming at detecting and preventing drill bit failure in industrial equipment, back then this was done using Fuzzy logic and empirically calculated bell curve distributions, this was a lot of manual experimental work to get things right and I remember thinking about adding a neural network layer to do this work.

Granted, we would still need to do training and experiments to gather data to get the neural network to fit the signal and expected curves, so this was an idea for the later bucket list.

However, I did play around with neural networks, at the time I was also fairly into game programming so plugging a neural network to do things like navigate a map was a cool idea.

Now that was quite a while ago (perhaps 20-25 or so years), so not much came out of that other than being disappointingly hard to get a network to actually work (vs just plugging a path finding algo like A*).

Fast forward to today I wonder what the modern frameworks and increased compute power can do here, back in the day I was not learned in data processing, feature engineering or optimization hence training a simple NN was not going very far.

So thinking a bit on this I decided to make a simple maze game where an agent has to navigate to the exit, my first thought was to throw a transformer to the problem, however, I did not want to go the full LLM path, I have already had LLMs play games before (Zork) and while entertaining it gets fairly compute expensive, this time I wanted something that could be run and trained locally.

So I started with a simple Encoder only transformer:

#[derive(Module, Debug)]
pub struct AgentModel<B: Backend> {
    input_proj: Linear<B>,
    transformer: TransformerEncoder<B>,
    output_proj: Linear<B>,
}

Basically we have a linear layer project the input vector, an encoder block extract some information from the input, and the output linear layer project this to an x,y movement vector.

Now why an encoder?, there is a bit of nuance between encoder vs decoder vs encoder-decoder architectures, in general encoders are good at extracting information from the input but tend to have a fixed output size (vs decoder where you can iteratively generate the tokens), this tradeoff is fine, we only have 2 outputs, x and y force vectors.

So this is already likely overkill for this application but interesting nonetheless as creating these models is basically trivial with current frameworks.

Now, how would the model learn from the problem? how are tokens mapped to game state?

At first glance it seemed hard to simply train a model given that there is not much data, so I thought of using RL to drop the agent into the maze and let it navigate, providing a reward/feedback loop that prioritizes getting to the exit, minimizing the distance navigated, penalizing small movement vectors or retrograde movement (so the agent would not stay static or swing back and forth).

However, how is the model going to sense the game world? Here is the main trick to get this working.

I set up the input as a vector of values, each representing the distance between the agent (a ball) and the game walls in 8 directions, so 8 floating points in and 2 out, with this you could already map the in-game environment (by ray casting in 8 directions to figure out the distance).

[W, NW, N, NE, E, SE, S, SW]

Thinking deeper this gave me another idea, what if instead we also fed previous environment states and actions? Then the model will have some context of what it had done before:

[STATE][ACTION][STATE][ACTION][STATE] -> Model -> [ACTION]

With this in mind I also thought about generating some sample chains of data from the game itself, for every step (basically game ticks), so I added some recording facilities to the little game and played it myself to generate some sample data, however, this was going to take some time.

My next idea was to have a path finder algorithm just do the job and generate sample data from the path, with this generating training data was quick and with some random maze generation I could get some good general data for training.

So that said, my final model is basically as described before with the encoder attention being set to only 2 heads (this is likely already overkill) and 64 hidden dimensions, the context is capped to 16 previous state/action pairs (plus current).

For training basically I fed a sequence and calculated a loss against the real action taken, this worked surprisingly well with only a few epochs being good enough to produce a reasonably working model.