The vehicle of the future will support its driver in critical situations and will advise him regarding potential hazards. Essential prerequisite therefore is the sensor based perception of the related traffic situation. For the recognition of traffic related objects, camera based sensors (such as grey scale, color cameras) and deepness cameras (such as PMD 3D cameras) are increasingly used beside vehicle sensors as well as radar and lidar environment sensors. Up to now a multitude of image based systems are currently in practical operation, which are in use e.g. for traffic sign recognition and lane tracking to realize adaptive speed control systems and lane departure warning systems. For the future development of Advanced Driver Assistance Systems (ADAS) the fusion of multiple sensor data to a consistent overall environmental picture will play a key role – especially for the situation recognition and interpretation.

The current evaluation approach of real world driving tests will no longer be sufficient due to the high complexity of the system interactions. New simulation methods are needed to test and evaluate ADAS by using virtual road tests with realistic vehicle behavior and complex traffic environment. On the one hand the simulation should be very close to real traffic situation and on the other hand the simulation should enable reproducible and comparable test conditions. Therefore it is very important to integrate also camera based components in the total “closed loop” integration and test platform CarMaker to be able to test sensor data fusion technologies under realistic conditions. Within a research frame work for autonomous driving functions a new simulation technology was developed to integrate virtual cameras beside the well know environment sensor (radar, lidar, ultrasonic) in the vehicle dynamic simulation CarMaker. For this purpose the realtime animation was extended with a sophisticated camera model so called “VideoDataStream” (VDS) to generate simultaneous video data such as grey scale, color, stereo pictures as well as deepness maps (e.g. PMD) for 3D images (figure 2). The camera positions and properties could be applied individually for more than 6 cameras. The integration of multiple cameras creates many new opportunities. These cameras can be freely configured, with optical properties and lens faults up to 250° fisheye being defined by models. The cameras are simulated in fully synchronized fashion in real time. Challenging applications like the fusion of stereo cameras or 360° park view systems can be reliably validated this way. The video data are available at a TCP/IP socket network interface and can be used for isochronous image processing with a related CPU, which could be a PC for MIL or SIL (Model-in-the-Loop or Software- in-the-Loop) or a real ECU for HIL (Hardware-in-the-Loop) (figure 1).

Here the created method and examples of image based perception of the vehicle environment as well as sensor data fusion algorithms shall be presented. Among others this covers first of all the recognition of traffic lanes, traffic signs and other traffic partners as well as the fusion of the single information up to a comprehensive environment picture (e.g. interpretation of road work area and junction situation). A further field of application will be the conjunction with navigation systems: e.g. navigation coupling “Navigation-in-the-Loop” (the integration of digital map data) by which the virtual vehicle supports the navigation system with related GPS position and gets back the “Most Probable Path (MPP)” with the “electronic horizon”, which is a type of predictive sensor, with all related preview information in front of the vehicle.

This information can be type of road, number of lanes, actual speed limit, tunnel or bridge, curvature, slope, longitudinal or latitude profile, divided road, heading change of junctions, route number, traffic lights and statistical road traffic conditions etc., which are defined in the ADASIS protocol.

By using the created method the capability and efficiency of functional development will significantly be improved. To test new driver assistance systems or new software versions thousands of test kilometers will be driven and recorded, typically today. But not every relevant situation could be covered by this method. Furthermore once made driving situations could not be modified or repeated under the same test conditions with different parameter settings (such as different control parameters or algorithms). In contrast the presented method allows the application of a wide scope of relevant situations with relatively little effort, which could be used for systematic evaluation. This covers very complex as well as infrequent real world situations, too. The method can be used for MiL-, SiL- and HiL-Tests.