Adaptation of a large trained model for new sensor and label data

Exploring Project

Initial situation

A neural network (NN) plays the role of an advanced control system by predicting and regulating safe driving trajectories for a moving vehicle. A good NN model is often trained with a variety of data coming from different sensors, e.g. cameras and GPS measurement sensors. One of the difficulties is to calibrate different sensors for each data acquisition process, taking into account that the camera is always placed at exactly the same position and with external parameters (direction, location). Motivated by the open-source autonomous driving system "Openpilot" by CommaAI, precise autonomous driving is tackled in the use case of a truck. This is done by adapting the already end-to-end trained neural network of openpilot called "Supercombo". This model was trained for driving a car and modified here for driving a truck. Differences between cars and trucks are, for example, that the camera is mounted higher in a truck than in a car. In addition, the dynamics of the sensor movement are different due to the attachment to the driver's cab, as the cab generally has stronger pitching movements. Fundamental questions are how good models that have been trained for a car can be transferred to a truck and what measures are necessary for such an adaptation.

Solution idea

According to the principle of "fine-tuning", small amounts of new data, GPS measurements, are used. This results in the actual position of the truck over time. This is converted into the local coordinate system of x and y trajectories of the truck. The trained model parameters of Supercombo in the passenger car area are used as initial model parameters for the truck area. Starting with the training of the truck domain and trained model parameters from the car domain, the low-level features for the same task can be transferred to the truck domain. In this way, a new training function is developed for the truck domain. An important step of the training function is that only the trajectory hypothesis about the future movement with the smallest deviation from the GPS measurement trajectories is used for parameter optimization in the next training step. At the end, a meaningful evaluation method is developed.

KI-Fortschrittszentrum_239_EP_Knorr-Bremse_Abb1 — Figure 1: Summary of the result in the form of the Frechèt distance between ground truth and network predictions before (old predictions) and after (new predictions) a "fine-tuning", Fraunhofer IPA

Benefit

This project proved the success of adapting a trained model to a slightly different domain with only a small partial training dataset (10 minutes of driving time as a training dataset) for the autonomous driving use case. The supercombo architecture, the training function written in Tensorflow 1 and the fine-tuned model parameters are ready for the new use cases of autonomous driving. Applying these models to trucks would greatly contribute to cost and effort savings in the development of autonomous driving trucks, as such systems can be developed and deployed in less time. Among other things, this reduces the need to drive additional distances with trucks to generate huge training data sets and enables the training of our own models based on a pre-trained neural network.

Implementation of the AI application

In this project, "fine-tuning" is used as a basic AI approach to adapt the trained supercombo onnx model of CommAI to the truck domain. Technically, the training function for fine-tuning is written in Tensorflow 1. The architecture uses a shared backbone that splits into multiple lane and path prediction layers. The network layers have many hop connections with dimensional coding and GRU-like layers for modeling time series data.