Is it feasible to integrate convolutionnal layers as Reinforcement Learning input to learn video game?

Let's say, you want to apply reinforcement learning on a simple 2D game. (ex : super mario)

The easy way is of course to retrieve an abstraction of the environnment, per example using gym and/or an open-source implementation of the game.

But if it's not available, I think about integrating convolutionnal layers over pixels as inputs of the RL agent.

We could of course split the task in two : featurization of the images and then reinforcement learning, we probably would need some supervision over the images (which can be problematic since we have no abstraction of the environment).

Is it a feasible approach to combine learning a featurization of the image data and learning a game policy at the same time ?

Topic game cnn reinforcement-learning

Category Data Science

1
answered at

Yes, it is possible to use convolutional layers in a reinforcement learning (RL) agent approximation function for action values (e.g. Q learning) or for policies (e.g. REINFORFCE).

In fact, any learning system capable of online learning of functions from example inputs and outputs will work with RL. The RL component will generate the examples to learn by taking actions in the environment or in simulation, and calculating some value such as the expected return. These examples are drawn from different distributions as the agent becomes better at the task, which is why online learning is important - the agent must forget the values associated with earlier experiences and replace them with new values as it improves its performance. Neural networks work for online learning by default, unless you make changes to them to prevent that.

That means you are not restricted to simple feed-forward networks. You can use CNNs, RNNs and other flavours of neural network provided you design them to output your value function or policy. Which will be best to use depends on the nature of the environment and your input signals. CNNs are a good choice whenever there is a structured arrangement of similar inputs - that includes image data, also many board games.

If you have not already, you may want to get hold of a (free PDF) copy of Reinforcement Learning: An Introduction. In chapter 16, section 16.5 the authors explain the original DQN project which learned how to play video games, including a discussion of the neural network architecture and pre-processing used. This is nowadays a well-known result in the RL community, you will find discussions, examples and implementations of it in many places.

One of the original researchers on DQN, David Silver, has published a lecture series on RL, with videos available on YouTube. He is also associated with DeepMind's Alpha Go project, which is another example RL system that uses CNN architecture internally.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.