The main problem in any neural-network based application is
the data. More the data, merrier it is.
But for training a network to try to drive a car, we would need a lot of
dashcam footage with exact steering angle at that exact point of time. Doing
this at the level we are working was difficult. Also, we cannot test a half
baked neural network on a real car because that would be dangerous.
Now-A-Days, Open world games like Grand Theft Auto V, are
made with such a detailing, that they almost simulate real world. This gives us
a virtual world to play with and test our trained networks safely.
In gta v there is an in-game AI that is present that
controls the people around us, Drive cars, to make the world much more
realistic. This AI can also drive cars. As this AI is specifically meant to
control GTA V, it can drive cars in this world perfectly.
Even when having a virtual world to simulate the real world,
it would be a very tedious job to drive the car in it for hours and hours to
make dataset and driving the car in that game manually would be very imperfect.
So, we are using a MOD for gta v called ‘DeepGTAV'(https://github.com/aitorzip/DeepGTAV).
This mod is made by ‘Aitor Ruano’ and published publicly on GitHub. GTA V
internal AI depends on scenarios in the world like driving mode, driving speed,
vehicle, time in the world, Weather, location in the world, and adapts right
according to it as it is made specifically for that purpose. DeepGTAV helps us
to set those parameters from our code directly, thus giving us a lot of control
over the world scenarios. This give us immense control over all variables,
recreate similar scenarios again and again but with slight variations each
time. In our approach we are using the in-game AI to drive the car by itself to
the waypoint set by us on the map. When we set a waypoint, the route to follow
appears on the map and minimap in purple. The in-game AI follows this purple
line to reach destination. This data of the in-game AI is captured and then stored.
DeepGTAV also allows to extract the exact steering angle, throttle, brake, location
on the map, driving mode, speed, etc. from the game. This would help us to
train our Neural Network to be trained in much better way. The dataset we are
generating contains the current frame (frame currently being displayed on the
screen), throttle, brake, steering, location and driving mode. The frame being
stored is of 800x600x3 (width=800, height=600, channels=3). The frame contains
minimap and the road. This data is stored in a compressed (.pz) file to save
space. Thus, by using the in-game AI we can generate huge amount of dataset
without driving an actual car. The in game ai communicates using sockets. The
DeepGTAV communicates on port 8000. This socket can be used for communication
of our code with the game GTA V itself thus bridging the gap between game and
our program. Using this approach, we have generated 100GB of dataset which
contains the parameters mentioned above. This dataset is given to the neural network
for training. The