Publications based on BeamNG.tech

Below you will find a collection of bachelor/master theses, research papers, papers for conferences and/or case studies using BeamNG.tech.

2021
Tahereh Zohdinasab, Alessio Gambi, Paolo Tonella, Vincenzo Riccio University of Passau, Università della Svizzera Italiana ACM

Abstract

Deep Learning (DL) has been successfully applied to a wide range of application domains, including safety-critical ones. Several DL testing approaches have been recently proposed in the literature but none of them aims to assess how different interpretable features of the generated inputs affect the system’s behaviour. In this paper, we resort to Illumination Search to find the highest-performing test cases (i.e., misbehaving and closest to misbehaving),spread across the cells of a map representing the feature space of the system. We introduce a methodology that guides the users of our approach in the tasks of identifying and quantifying the dimensions of the feature space for a given domain. We developed DeepHyperion, a search-based tool for DL systems that illuminates, i.e.,explores at large, the feature space, by providing developers with an interpretable feature map where automatically generated inputs are placed along with information about the exposed behaviours.

2020

Abstract

Simulation based testing is the most common technique for testing autonomous vehicles(AVs). For each test a tester needs to describe a scenario, specify test criteria, setup a simulation, connect the artificial intelligences (AIs) under test to it, execute the test,determine its results and collect all generated data e. g. for further analysis or training AIs. This process is tedious and error prone. There is no well-established procedure how to cope with or solve these problems. I present DriveBuild, a research toolkit for simulation based testing of AVs. DriveBuild comes with an abstract scheme to describe tests and provides a scalable client-server-architecture based on micro services. DriveBuildis able to execute automatically generated tests and to connect AIs under test which control AVs in a simulation. It also offers many metrics to analyze AVs and test generators. This thesis shows that DriveBuild automates the process of setting up simulators, distributing test runs across a cluster, frequently checking test criteria during a simulation, gathering data and analyzing test results. So it reduces the amount of time which a tester needs to invest into preparing, running and evaluating simulation based tests. There are already students,courses as well as research groups that are interested in DriveBuildand use it for their own purpose.

Abstract

Researchers proposed many groundbreaking ideas and techniques to make autonomous vehicles safer and more reliable on the streets. But society still has many concerns about the safety and reliability of autonomous cars. Many contributions focused on the most important property: the safety of passengers. But also other non-functional properties have to be ensured. One of them is the fuel consumption of cars. Nobody wants to drive in a car which consumes fuel and pollutes the environment more than necessary. My thesis aims to expose the problems of autonomous cars concerning their fuel-inefficiency. By defining a set of scoring oracles, the decisions taken by a car can be assessed. These oracles check several sensor values and driving patterns to classify whether the car under test drives fuel-inefficient or not, e.g. the ego-car drives with a high RPM (rounds per minute) with a low gear and hence, this is an infraction against the oracle. Every infraction is logged and then used to calculate a scoring function. By using procedural content generation, a method commonly used in the gaming industry to algorithmically generate various content, I randomly create urban-like scenarios with intersections, traffic, traffic lights and signs as well as parked cars to stress fuel-inefficient driving behaviour of self-driving vehicles. The evaluation results show that my scoring function can expose faults concerning fuel-inefficiency of different driving behaviours. I also proved a strong positive correlation of 0.779 between the score and consumed fuel. My test cases, on the other hand, exposed many faults of a traffic light detection system.

Abstract

Self-driving cars are an emerging part of automotive industry and a vital aspect of future. When it comes to automation of vehicles, that is transferring the control automobile to software, safety is the biggest concern as it can risk human life. In order to ensure safety in any driving conditions industry has to maintain some safety standards for the certification of self-driving cars. It is important to ensure that the software is intelligent enough to not only handle critical situations but also to predict or address any upcoming harmful event before deployment to prevent future mishaps. Discovery of test cases which can disclose the malfunction of an autonomous car is a thought-provoking task because the possibility of such test cases is infinite. Hence, one technique to analytically examine the autonomous cars safety is the simulations which are executed with no risk,no harmful condition and fast execution. Automatically generating driving simulations from real world driving videos could reduce time consumption of testers by manually creating different driving simulations. The intend to generate simulations from freely available videos will help testers to understand how the self-driving car would perform in similar situations. My thesis addresses this problem and define a new method to automatically generate driving simulations from commonly available geo-tagged videos recorded during driving. The process uses the GPS data to identify the roads in which the recorded driving took place, recreates those roads in the driving simulation, and configures the ego car to drive as the original car was driving in the videos. Next, the videos are analysed using a machine learning classifier to identify leading cars by means of bounding boxes, their relative position and speed w.r.t. the ego car. Finally, a driving simulation reproducing the movement of the ego car and (one) leading car in front of it is generated. Evaluation results obtained by analysing randomly selected driving videos show that the proposed method is efficient and produce reasonably accurate simulations,suggesting that the proposed method is viable and can pave the way for future self-driving cars testing by providing an efficient tool to support testers in developing safe self-driving cars.

Johannes Müller University of Passau

Abstract

Autonomous cars are no vision of the future any more, with car manufacturers planning on releasing them in the near future. With software controlling vehicles in traffic, human live is at risk, when this software malfunctions. There is an infinite amount of traffic scenarios this software needs to be able to handle without failure, hence it is necessary to cover as much different scenarios as possible. Computer simulations provide a fast method to execute test cases, but generating test configuration for virtual environments still remains a costly and time consuming process. In this work I propose a method that tackles this problem by automatically generating test cases based on a received specification. For this task a model for describing test cases in form of finite state machines is introduced, that allows splitting the generation process into small steps. In each step values for properties of the test case model are picked constrained by the received specification and already present values until every property has a value assigned. The test case generated this way is then used as a starting point for local search algorithms to find test cases that reach goals in terms of trajectory, velocity and timing. The focus of the generation process lies on the generation of a test case that’s values are as close as possible to the received specification. For the evaluation of the proposed method for automated test case generation a prototype system was implemented, that can generate three different accident types. It is shown, that the system can generate test cases fast and that the scenarios are conform to the received specification in most cases. This method allows testers to focus on what they want to test by providing a test case specification rather than on how to set up trajectories feasibly and enable the correct timing between vehicles.

Alessio Gambi, Pascale Maul, Marc Mueller, Lefteris Stamatogiannakis, Thomas Fischer, Sebastiano Panichella University of Passau, BeamNG GmbH, Zurich University of Applied Science

Abstract

Industry and research organizations increasingly rely on simulation platforms to facilitate the development and validation of Cyber-physical Systems (CPSs). The main factors for this trend are simulation’s cost-efficiency and the possibility of evaluating the system’s performance early on and through-out the development cycle in a fully controlled environment. However, simulations need to meet stringent functional and non-functional requirements to benefit development and debugging activities. In particular, high simulation accuracy and the ability to systematically generate relevant test scenarios are paramount for effectively assessing CPSs’ behaviors in nominal and critical test scenarios. This paper (i) discusses soft-body simulation and procedural content generation relevance to achieve the systematic generation of physically accurate virtual tests; (ii) and presents BeamNG.tech a novel simulation framework featuring both soft-body simulation and procedural content generation. Hence, we report on the main advantages and research results in testing self-driving car software enabled by BeamNG.tech. Finally, we reflect on the central role of simulation-based continuous integration and testing pipelines to improve current CPSs development practices.

2019

Abstract

The objective of this bachelor thesis is the adaptation of a low-budget driving simulator, to a recently developed driving simulator software BeamNG.research. This simulator is made as a close resemblance to the battery electric vehicle Renault ZOE. The thesis primarily focuses on all hardware aspects of the simulator. A market analysis compares already used and common solutions and hardware as base for the development of a concept. Based on a requirement and feature analysis, a simulator with primarily computer gaming hardware is build and reconfigured from typical race car ergonomics to a more fitting compact car appeal. This requires the construction of several steel adaptor frames during the realization. A four-display setup is also realized, with three 55” UHD TVs (180degree FOV) and a fourth 10” HD display as designated driver display. In parallel,a data interface for taping into the ZOEs CAN-Busses is selected. And a new concept, the CanSee of the CanZE community is recreated and adapted. The first iteration of the simulator setup as well as a prove of concept for the CanSee is evaluated and tested. Several improvements follow the evaluation concluding the realization. Both, the simulator setup and the data interface, showed promising results. The simulator could be used effectively during more than 60 hours of use in survey and provided all desired data. The CanSee data interface proved to more than a tool for validifying the simulator, but also as base for future development of energy interfaces.

Christian Zellier, Jakob Claußen, Alexander Danetzky, Maximilian Kayser, Eric Foerster University of Lübeck

Abstract

This project deals with the creation of displays for energy-efficient driving with electric cars. Energy-efficient driving is particularly important in relation to electric cars, as it has a very strong impact on the range of electric cars. The aim of the displays is to show drivers an energy-efficient driving style. These displays were developed as UI mods and then integrated into the BeamNG.re-search software. The Institute for Multimedia and Interactive Systems (IMIS) has developed a driving simulator to test the effectiveness of displays faster. The developed displays were then integrated into the driving simulator. In the beginning in order to develop the displays, mockups were created based on research. The best mockups were selected and prototyped with the help of the software SimHub. These prototypes were evaluated in a mid-term evaluation and then the two with the greatest potential were further developed. Based on the feedback, adjustments were made and then implemented as web-based UI mods. Finally, the implemented displays were presented and evaluated at the EMI-Award. In the following, the procedure mentioned is explained in more detail in the individual chapters

Thomas Franke, Daniel Görges, Matthias G. Arend University of Lübeck, TU Kaiserslautern, RWTH Aachen University ACM

Abstract

The design of effective energy interfaces for electric vehicles needs an integrated perspective on the technical and psychological factors that together establish real-world vehicle energy efficiency. The objective of the present research was to provide a transdisciplinary synthesis of key factors for the design of energy interfaces for battery electric vehicles (BEVs) that effectively support drivers in their eco-driving efforts. While previous research tends to concentrate on the (visual) representation of common energy efficiency measures, we focus on the design of action-integrated metrics and indicators for vehicle energy efficiency that account for the perceptual capacities and bounded rationality of drivers. Based on this rationale,we propose energy interface examples for the most basic driving maneuvers (acceleration, constant driving, deceleration) and discuss challenges and opportunities of these design solutions.

Alessio Gambi, Tri Huynh, Gordon Fraser University of Passau / University of Saarlandes / CISPA ACM

Abstract

Autonomous driving carries the promise to drastically reduce the number of car accidents; however, recently reported fatal crashes involving self-driving cars show that such an important goal is not yet achieved. This calls for better testing of the software controlling self-driving cars, which is difficult because it requires producing challenging driving scenarios. To better test self-driving car soft-ware, we propose to specifically test car crash scenarios, which are critical par excellence. Since real car crashes are difficult to test in field operation, we recreate them as physically accurate simulations in an environment that can be used for testing self-driving car software. To cope with the scarcity of sensory data collected during real car crashes which does not enable a full reproduction,we extract the information to recreate real car crashes from the police reports which document them. Our extensive evaluation, consisting of a user study involving 34 participants and a quantitative analysis of the quality of the generated tests, shows that we can generate accurate simulations of car crashes in a matter of minutes. Compared to tests which implement non critical driving scenarios,our tests effectively stressed the test subject in different ways and exposed several shortcomings in its implementation.

Alessio Gambi, Marc Müller, Gordon Fraser University of Passau, BeamNG GmbH ACM

Abstract

Self-driving cars rely on software which needs to be thoroughly tested. Testing self-driving car software in real traffic is not only expensive but also dangerous, and has already caused fatalities. Virtual tests, in which self-driving car software is tested in computer simulations, offer a more efficient and safer alternative compared to naturalistic field operational tests. However, creating suitable test scenarios is laborious and difficult. In this paper we combine procedural content generation, a technique commonly employed in modern video games, and search-based testing, a testing technique proven to be effective in many domains, in order to automatically create challenging virtual scenarios for testing self-driving car soft-ware. Our AsFault prototype implements this approach to generate virtual roads for testing lane keeping, one of the defining features of autonomous driving. Evaluation on two different self-driving car software systems demonstrates that AsFault can generate effective virtual road networks that succeed in revealing software failures,which manifest as cars departing their lane. Compared to random testing AsFault was not only more efficient, but also caused up to twice as many lane departures.

Alessio Gambi, Tri Huynh, Gordon Fraser University of Passau / University of Saarlandes / CISPA ACM

Abstract

Autonomous driving carries the promise to drastically reduce the number of car accidents; however, recently reported fatal crashes involving self-driving cars show this important goal is not yet achieved, and call for better testing of the software controlling self-driving cars. To better test self-driving car software, we propose to specifically test critical scenarios. Since these are difficult to test in field operation, we create simulations of critical situations. These simulations are automatically derived from natural language police reports of actual car crashes, which are available in historical datasets. Our initial evaluation shows that we can generate accurate simulations in a matter of minutes.

Alessio Gambi, Tri Huynh, Gordon Fraser University of Passau / University of Saarlandes / CISPA ACM

Abstract

Autonomous driving carries the promise to drastically reduce car accidents, but recently reported fatal crashes involving self-driving cars suggest that the self-driving car software should be tested more thoroughly. For addressing this need, we introduce AC3R (Automatic Crash Constructor from Crash Report) which elaborates police reports to automatically recreate car crashes in a simulated environment that can be used for testing self-driving car software in critical situations.AC3R enables developers to quickly generate relevant test cases from the massive historical dataset of recorded car crashes. We demonstrate how AC3R can generate simulations of different car crashes and report the findings of a large user study which concluded that AC3R simulations are accurate.

Alessio Gambi, Marc Müller, Gordon Fraser University of Passau, BeamNG GmbH ACM

Abstract

Ensuring the safety of self-driving cars is important,but neither industry nor authorities have settled on a standard way to test them. Deploying self-driving cars for testing in regular traffic is a common, but costly and risky method, which has already caused fatalities. As a safer alternative, virtual tests, in which self-driving car software is tested in computer simulations,have been proposed. One cannot hope to sufficiently cover the huge number of possible driving situations self-driving cars must be tested for by manually creating such tests. Therefore,we developed ASFAULT, a tool for automatically generating virtual tests for systematically testing self-driving car software. We demonstrate ASFAULT by testing the lane keeping feature of an artificial intelligence-based self-driving car software, for which ASFAULT generates scenarios that cause it to drive off the road.

Abstract

Evaluating the safety of autonomous driving systems is one of the biggest obstacles for the deployment of these systems. Because of the high number of test scenarios, which arouses through the huge variety of possible interactions with the environment, relying only on expensive real-life testing is not practical and therefor, simulation testing is been widely used. As a consequence of the needed time and computation power to generate and run a test case,it is desirable, that few test scenarios cover a wide range of the possible testing parameters. This is why diversity of the test scenarios is an important element to consider to decrease the number of test case executions. I compare in this thesis two approaches which aim to maximize diversity: novelty search and multi-objective search. The result of this thesis shows that multi-objective search generates more effective test cases, despite both have a similar distribution regarding where the test subject fails.

2018

Abstract

This thesis describes the development of the software BeamNG.research to a driving-simulation environment for researches on the field of the user-energy-interaction. Initially, it was identified that there were requirements that potentially were not fulfilled by default. These requirements were found in the areas of exporting data, energy simulation, vehicle, experimental control, UI-mods and tracks. All identified requirements were collected in a requirements document which was further extended in the course of the work. A manual was created to facilitate the use of BeamNG.research across all the areas identified. Regarding the data ex-port, instructions have been added to the manual for using a mod provided for this purpose. Therefore it was necessary to identify the corresponding names of the parameters. Further-more the mod was simplified regarding the configuration of the mod using comments. In the scope of energy simulation a live data exchange between BeamNG.research and Matlab was realised via File-based Interprocess Communication in order to externalise the simulation for the purpose of a high precision. In addition, a way was to be found to actually use the data from Matlab in BeamNG.research. Furthermore an electrical vehicle was integrated in BeamNG.research and the camera was optimized for a realistic driving experience. On the basis of a predetermined sequence of a potential first experiment, it was investigated how various requirements of experimental control could be solved. For this purpose, several co-ordinated Mods were written and even some partially existing original files were modified. Furthermore, the manual on creating scenarios was extended by helpful hints. In the areas of UI-mods and driving route, only a few notes on UI-mods were mentioned in the manual, after it was determined that UI-mods already fulfilled nearly all requirements. Finally, a real experiment was realized using the results of this thesis to validate that all of the most important requirements were met. During this thesis BeamNG.drive was used but all results are transferable to BeamNG.research.

While BeamNG.drive is a simulation game, BeamNG.research represents the research variant. Therefore this document always refers to BeamNG.research.

Marc Müller University of Saarland

Abstract

Autonomous vehicles are becoming an increasingly relevant part of the automotive industry and will only become more important in the near future. Ensuring safety is naturally important when handing over full control of a vehicle to software, but neither industry nor authorities have settled on a standard way to certify autonomous vehicles. Deploying autonomous vehicles for testing in regular urban traffic is a common but costly and risky method. Simulated, or virtual, tests have been introduced as a way to expose problems before deployment, but traditional software testing techniques cannot cope with the massive amount of situations an autonomous vehicle faces. For limited cases, more advanced techniques like search-based testing show more promising results. This work will continue in that direction, utilizing procedural content generation and genetic algorithms to evolve driving tasks meant to test the lane keeping capability of autonomous vehicles. Meaningful metrics to characterise input scenarios, output behaviour of the car, and how well tests cover the input space will be introduced to guide test suite evolution towards stressing the vehicle’s lane keeping behaviour effectively

Nourelhoda S. K. Mohamed University of Bremen

Abstract

Driving a vehicle requires practices and exercises, particularly for hazardous situations. In general, driving is an activity that requires the humans mental and physical abilities to achieve safe driving. In hazard situations, drivers must have the cognitive abilities to detect and anticipate hazards. In additions, they must have knowledge that empowers them to react in a proper way. In such situations,a wrong action may lead to significant damages and dramatic consequences. At the same time, physical real training of driving hazard situations is limited, due to crash consequences. In this thesis, we argue using the crash experience to enhance drivers’ hazard perception. From a cognitive perspective, raising drivers’ awareness of the crash and its physical damage consequences would influence their driving behaviours. We utilized BeamNG.drive that provides a dynamic soft-body physics vehicle simulation. We developed a practical study, when participants are required to drive certain scenarios - typically to reality - to learn a specific traffic situation (e.g. yield to priority road). We implemented various learning scenarios for hazard situations. In this study, two learning modules are proposed: instructional video experience and dynamic physical crash experience. After learning, participants drive an evaluation scenario, where their driving performance is assessed by quantitative and qualitative measures. The study usability and usefulness, as well as, participants’ enjoyment and tensions are evaluated by qualitative questionnaires. Statistical analysis shows significant influences of crash experience in raising participant’s awareness of crash regardless of their age or their previous driving experience. The findings illustrate the feasibility of the developed study and consequently proofs the proposed hypotheses.


Citation

Use of BeamNG.tech in non-commercial, academic studies should be properly cited in any articles, papers for conferences, presentations made about the research project. Please adhere to the citation format required by your institution or publication, using the information below:

Title: BeamNG.tech
Author: BeamNG GmbH
Address: Bremen, Germany
Year: 2021
Version: 0.23.0.0
URL: https://www.beamng.tech/

Send us your filled form incl. your research paper(s) to tech@beamng.gmbh using the following subject line: Consent to publish research paper(s) using BeamNG.tech.

@software{beamng_tech,
title = "{B}eam{NG}.tech",
author = {{BeamNG GmbH}},
url = {https://www.beamng.tech/},
version = {0.23.0.0},
date = {2021-01-21},
}