[JohnnyWaffles]

Hi there! So I was wondering, how would you teach AI to play a game? Not a simple game like Tetris, but a story based game. When we play games like Rise of the Tomb Raider we often are driven through the game by the narrative, even at its most basic form. Can an AI learn to navigate the game and it’s world enough to progress through the narrative and maybe even beat the game by itself. I wouldn’t expect it to comprehend the story like we do, but it could learn to:

1)   Avoid Harm (falling, being shot at, etc…)
2)   Navigate a complicated environment
3)   Solve puzzles
4)   Maintain an Inventory
5)   Kick butt

I’m mostly a noob when it comes to AI, I’ve only just started to learn. However, my first thoughts are that to make an AI that can navigate and play a modern Triple A game you would need a systematic approach.
Break the game down by system. Some general examples:

1)   Navigation
2)   Health and/or safety maintenance
3)   Inventory Systems
4)   Combat (if any and what type)
5)   Maybe Puzzle Solving…

From there on you can assign an ANN to each system to maintain it. Of course you‘d have to decide which ANN is the best fit. So those are my general ideas. Am I kind of close or am I way off? I actually wanted to try and create an AI that can play a game for me (single player), hence my post.

Also I see a lot of talk about members trying to create their own AGI. I wonder if a digital world is good place to start for developing and testing an AGI.

[HS]

 I don't know much about this stuff either. But is sounds like a big and cool project. You might need some speach and text recognition too. How do you plan on having the AI interact with the game? My natural inclination would be to try and make a robot which can press keys and move a mouse and look at the screen and hear the speakers. There is probably an easier way to do it with software but I wouldn't know where to start.

[JohnnyWaffles]

I don't know much about this stuff either. But is sounds like a big and cool project. You might need some speach and text recognition too. How do you plan on having the AI interact with the game? My natural inclination would be to try and make a robot which can press keys and move a mouse and look at the screen and hear the speakers. There is probably an easier way to do it with software but I wouldn't know where to start.

Well I would think software would be an easier approach. You would need to code a "controller" module that the AI would have to interface with. Off the top of my head there is the SendKeys method: https://docs.microsoft.com/en-us/dotnet/api/system.windows.forms.sendkeys.send?view=netframework-4.7.2

It would be a matter of mapping the correct keys to the controller module for gameplay. However I'm sure there is probably a better way. The keyboard does not give you any pressure sensitive options for navigation like you would get with an analog stick on say an Xbox 360 remote. Or there is this option, which seems more complicated but would probably give the AI more control options:
https://stackoverflow.com/questions/11711947/how-do-i-create-a-virtual-gamepad

That's my plan so far. I'll probably go with the SO thread and implement a virtual gamepad. Also good thinking about the text translation.

I've had some more ideas since then on how to pull this off. First I'll start off by saying  that a narrative, although typically defined in the context of a story line or plot, can also be simply defined as a set of decisions and/or actions in a timeline. It doesn't necessarily require a contrived plot or even a cast of characters. So with that barebones definition in mind I think we can teach an AI to establish context. Which based on my research an RNN might do the job.

From what I understand an RNN can remember all of the data passed into it and therefore can establish some context. It can remember where it started and where it’s been, which will be necessary if it’s to know where it needs to go next. For important things like navigation, avoiding collision and calculating proximity I’d say a CNN for image classification and object detection. That can be the AIs “eyes”. That’s about all I have so far though. If anybody has recommendations I’m certainly open to them.

[AgentSmith]

I've had some more ideas since then on how to pull this off. First I'll start off by saying  that a narrative, although typically defined in the context of a story line or plot, can also be simply defined as a set of decisions and/or actions in a timeline. It doesn't necessarily require a contrived plot or even a cast of characters. So with that barebones definition in mind I think we can teach an AI to establish context. Which based on my research an RNN might do the job.

From what I understand an RNN can remember all of the data passed into it and therefore can establish some context. It can remember where it started and where it’s been, which will be necessary if it’s to know where it needs to go next. For important things like navigation, avoiding collision and calculating proximity I’d say a CNN for image classification and object detection. That can be the AIs “eyes”. That’s about all I have so far though. If anybody has recommendations I’m certainly open to them.

Your thoughts and your approach concerning the networks and how you would use them appear promising. Using a CNN to process visual input and an RNN to have an internal model of the environment is a quite good approach.
I would like to make 2 recommendations:
1. You should think about how you can formalize the goals your agent has to achieve.
2. Lets assume the internal state of your RNN contains the most important aspects of the environment (like obstacles) in a proper compact neural representation. This knowledge alone does not mean that your agent knows how to act in certain situations, for instance if he stands in front of an obstacle. You need to think about where this knowledge could come from.

[Korrelan]