Model-based Testing

Model-based testing (MBT) is a well-established field of automated testing where a formal model of the system under test (SUT) is used to generate test cases that meet specific requirements. Applying MBT is particularly convenient when the SUT exhibits a high degree of complexity, as MBT helps to minimize complexity and makes test creation manageable. In particular, a model is an abstract representation of the SUT that considers only relevant components regardless of not relevant details, thus lowering complexity.

Model-Based Testing generation cycle

The model-Based Testing generation cycle is represented in Figure 1. Given a specific SUT under consideration, the tester specifies a model by abstracting away details that are not relevant to the current testing task. Consider that, for the same SUT, different models can be designed, reflecting different aspects of the SUT that need to be tested. From the model, a set of abstract test cases are generated in order to satisfy some model coverage criteria. Abstract test cases are then concretized into test cases that can be executed on the SUT.

Fig 1. MBT generation cycle.

EvoMBT

EvoMBT tool integrates a model-based approach and search-based testing enabling automated test generation for XR systems. The tool allows testing practitioners to synthesize the relevant aspects of the SUT into an extended finite state machine (EFSM). EvoMBT generates an abstract test suite on the model that satisfies user-defined coverage criteria, including states, transitions, and k-transitions criteria. The tester that would like to adopt the model-based approach to test generation should then provide a map from an abstract test (generated on the model) to a concrete test executable on the SUT.
The synthesis of abstract test cases from a given EFSM is the core of EvoMBT. This phase is not specific to the particular SUT under consideration. For any EFSM, EvoMBT relies on different search heuristics to identify test cases that satisfy the user-specified coverage criteria. EvoMBT exploits the well-known EvoSuite tool as a library for various search algorithms. The other steps of the MBT cycle, i.e., abstraction of the model, concretization, and execution on the SUT, are specific to the particular SUT under consideration.

How to use EvoMBT

To start using EvoMBT, the tester has to clone the GitHub repository https://github.com/iv4xr-project/iv4xr-mbt

EvoMBT requires at least Java 11 and it can be built and installed with the Maven tool.

mvn clean package -DskipTests

Alternatively, it is possible to download the latest stable release. To have a flavor of EvoMBT run the following command

java -jar target/EvoMBT-x.x.x-jar-with-dependencies.jar -sbt

EvoMBT creates a folder named mbt-files containing folders statistics and tests. The statistics folder includes the file statistics.csv, which collects information about the test cases creation process. This file is appended with execution statistics every time EvoMBT runs. Each modeled SUT has its subdirectory in the folder tests. Each used test generation algorithm has a corresponding subfolder in the matching SUT directory. Finally, there is a folder for each EvoMBT run with a randomly created name corresponding to the column id in the statistics.csv file. Therefore, it is possible to correlate the appropriate statistics with each generated test suite. A detailed description of the EvoMBT output is available in the wiki.

Use cases

EvoMBT has built-in support for three different SUT.

Lab Recruits

Lab Recruits is a 3D developed as a testbed for UX testing. EvoMBT implements a full instantiation of the MBT cycle for Lab Recruits. Figure 2 depicts the main components of the MBT cycle instantiated for Lab Recruits.

Fig 2. MBT cycle instantiated on Lab Recruits.

Lab Recruits test generation performs the following three steps:

  • creates a random EFSM that can be translated into a csv for Lab Recruits. Random generation of Lab Recruits EFSM is described in the wiki page
  • performs search based test suite generation for the random EFSM just created
  • executes generated test cases in Lab Recruits

Space Engineers

EvoMBT supports the generation of abstract test cases for the game Space Engineers. Space Engineers support is under continuous development. Currently, EvoMBT generates and executes test cases on a single Space Engineers level derived from the Lab Recruits level random_medium. Details are available in the dedicated wiki page.

BeamNG.tech

Test generation for BeamNG.tech follows the guideline of the SBST 2022 Cyber-physical systems (CPS) testing competition. Test cases execution is described in the dedicated github page. To run test case generation for BeamNG.tech run

java -jar java -jar target/EvoMBT-x.x.x-jar-with-dependencies.jar -sbt -Dsut_efsm=cps.beamng_custom_model

In addition to the described output, EvoMBT creates a table (csv format) for each test case containing the road points that can be fed to the BeamNG.tech simulator. Road points are a sequence of points in the 2D space.

Extending EvoMBT

EvoMBT project can be used as an external Java library allowing seamless integration into any SUT-specific testing framework. An example of Java project that imports and uses EvoMBT is available on GitHub.

Links

EvoMBT releases

EvoMBT wiki

EvoMBT source code