The Exploratory Functional Test Agents (FTAs) do not follow specific tactics, goals, or crafted models to test the System Under Test (SUT). Exploratory FTAs perform a scriptless exploration (non-sequential) of the environment and verify the system’s behavior while trying to learn potentially interesting actions.
Exploratory FTAs are based on integrating the TESTAR tool in the iv4XR framework. The underlying principle of the TESTAR exploration consists on: generate test sequences of (state, action)-pairs by connecting to the SUT in its initial state and continuously selecting an action to bring the SUT to another state while the TESTAR agent checks oracles to determine if the SUT contains a failure.
These oracles are customizable and can help to check that the SUT process runs properly without freezing, that the entity’s properties are responding adequately to the executed actions, or that the state or log of the SUT does not contain suspicious messages such as error or exception.
- HTML report that indicates the observed state and executed actions step-by-step with textual and visual information. This HTML report is always active and created in an output folder by default.
- State Model inferred that contains information regarding how the executed actions allow transit between the observed states. To use the State Model functionality, it is necessary to configure the OrientDB database.
- Spatial map that highlights the navigable space covered and the entities interacted by the Exploratory FTA. The creation of this spatial result is SUT-dependent; for this reason, a development integration is necessary.
- Code Coverage report that indicates the code lines and code branches covered by the Exploratory FTA actions. The integration and usage of code coverage software depend on the SUT’s technology.
Exploratory FTA Overview
How to use the Exploratory FTA
TESTAR tool is a Java-based application that uses Gradle software for compilation and packaging. It is open-source and available in the iv4XR GitHub repository: TESTAR_iv4xr
Users can clone the repository by using Git version control software.
git clone https://github.com/iv4xr-project/TESTAR_iv4xr.git
The first option to use the TESTAR tool without being an active developer is to use the Gradle Wrapper commands to compile and prepare an application distribution.
The second option to use the TESTAR tool at the development level is to import the Gradle project into an integrated development environment (IDE) such as Eclipse or IntelliJ.
More details and visual examples of cloning the repository and using Gradle Wrapper or an IDE to compile TESTAR can be found in the wiki of the GitHub repository: TESTAR installation
TESTAR tool contains the logical functionality to act as an Exploratory FTA in the iv4XR framework, but it cannot connect directly to the SUT. It is necessary to prepare the intermediate SUT-plugin dependencies that allow the FTA agents to observe the SUT State and execute Actions.
Logical Java SUT protocols
TESTAR uses different Java protocols to customize the decision-making of the tool in the different steps of the logical flow execution. The tools already provide default protocols to interact with LabRecruits and Space Engineers, but the users can enrich the possible action to execute, modify the action selection algorithm or apply additional oracles.
For the actions, the user can indicate on which blocks we want to perform actions such as welder, grinder, or interact.
IF block.type IS “CryoChamber” OR “CockpitSeat”
THEN actions ADD NavigateInteract(block)
And for the oracles, the user can apply additional conditions to indicate to the tool what is considered a failure.
IF lastExecutedAction IS NavigateInteract(block) AND agent.Energy DID NOT increase
THEN Verdict(WARNING, “Energy was not recharged after functional action”)
The above pseudocode examples need to be customized using the Java programming language and understanding the TESTAR concept of Tags and flow methods. Additional information can be found here: TESTAR Java protocols
Demo of an Exploratory FTA running with Space Engineers
TESTAR explores a Space Engineers survival scenario to verify that the agent health and energy functionalities are restored correctly, that the integrity of the blocks increases and decreases while using tools, and that it is possible to construct in free spaces. The HTML report and inferred State Model are shown at the end of the exploration.
TESTAR automatically explores a Space Engineers scenario generated randomly with different types of blocks to verify the robustness of the navigation and basic tool interactions. The spatial map and code coverage results are shown at the end of the exploration.
TESTAR for XR systems: https://github.com/iv4xr-project/TESTAR_iv4xr
TESTAR for GUI systems: https://github.com/TESTARtool/TESTAR_dev
- Vos, T. E., Aho, P., Pastor Ricos, F., Rodriguez Valdes, O., & Mulders, A. (2021). testar-scriptless testing through graphical user interface. Software Testing, Verification and Reliability, 31(3), e1771. doi.org/10.1002/stvr.1771
- Pastor Ricós, F. (2022). Scriptless Testing for Extended Reality Systems. In International Conference on Research Challenges in Information Science (pp. 786-794). Springer, Cham. doi.org/10.1007/978-3-031-05760-1_56
- Mulders, A., Valdes, O.R., Ricós, F.P., Aho, P., Marín, B., Vos, T.E.J. (2022). State Model Inference Through the GUI Using Run-Time Test Generation. In: Guizzardi, R., Ralyté, J., Franch, X. (eds) Research Challenges in Information Science. RCIS 2022. Lecture Notes in Business Information Processing, vol 446. Springer, Cham. doi.org/10.1007/978-3-031-05760-1_32
- I. S. W. B. Prasetya, Fernando Pastor Ricós, Fitsum Meshesha Kifetew, Davide Prandi, Samira Shirzadehhajimahmood, Tanja E. J. Vos, Premysl Paska, Karel Hovorka, Raihana Ferdous, Angelo Susi, and Joseph Davidson. 2022. An agent-based approach to automated game testing: an experience report. In Proceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST 2022). Association for Computing Machinery, New York, NY, USA, 1-8. doi.org/10.1145/3548659.3561305