Cognitive map-based model: Toward a developmental framework for self-driving cars

End-to-end learning and multi-sensor fusion-based methods are two major frameworks used for self-driving cars. To enable these intelligence vehicles to acquire driving skills at a level comparable to that of human drivers, long short-term memory of previous self-driving processes is necessary, but is difficult to introduce into the above-mentioned frameworks. In this paper, we propose a model for self-driving cars called the cognitive map-based neural network (CMNN). Our framework consists of three parts: a convolutional neural network that can perceive the environment in the manner that the human visual cortex does, a cognitive map to describe the locations of objects in a complex traffic scene and the relationships among them, and a recurrent neural network to process long short-term memory from the cognitive map, which is updated in real time. The proposed model is built to simultaneously handle three tasks: i) detecting free space and lane boundaries, ii) estimating vehicle pose and obstacle distance, and iii) learning to plan and control based on the behaviors of a human driver. More significantly, our approach introduces external instructions during an end-to-end driving process. To test it, we created a large-scale road vehicle dataset (RVD) containing more than 50,000 labeled road images captured by three cameras. We implemented the proposed model on an embedded system.