Exploratory Agents

---

The Exploratory Functional Test Agents (FTAs) do not follow specific tactics, goals, or crafted models to test the System Under Test (SUT). Exploratory FTAs perform a scriptless exploration (non-sequential) of the environment and verify the system’s behavior while trying to learn potentially interesting actions.

TESTAR tool

Exploratory FTAs are based on integrating the TESTAR tool in the iv4XR framework. The underlying principle of the TESTAR exploration consists on: generate test sequences of (state, action)-pairs by connecting to the SUT in its initial state and continuously selecting an action to bring the SUT to another state while the TESTAR agent checks oracles to determine if the SUT contains a failure.

These oracles are customizable and can help to check that the SUT process runs properly without freezing, that the entity’s properties are responding adequately to the executed actions, or that the state or log of the SUT does not contain suspicious messages such as error or exception.

Output results

• HTML report that indicates the observed state and executed actions step-by-step with textual and visual information. This HTML report is always active and created in an output folder by default.
• State Model inferred that contains information regarding how the executed actions allow transit between the observed states. To use the State Model functionality, it is necessary to configure the OrientDB database.
• Spatial map that highlights the navigable space covered and the entities interacted by the Exploratory FTA. The creation of this spatial result is SUT-dependent; for this reason, a development integration is necessary.
• Code Coverage report that indicates the code lines and code branches covered by the Exploratory FTA actions. The integration and usage of code coverage software depend on the SUT’s technology.

Exploratory FTA Overview

How to use the Exploratory FTA

Installation

TESTAR tool is a Java-based application that uses Gradle software for compilation and packaging. It is open-source and available in the iv4XR GitHub repository: TESTAR_iv4xr

Users can clone the repository by using Git version control software.

git clone https://github.com/iv4xr-project/TESTAR_iv4xr.git

The first option to use the TESTAR tool without being an active developer is to use the Gradle Wrapper commands to compile and prepare an application distribution.

gradlew build

gradlew installDist

The second option to use the TESTAR tool at the development level is to import the Gradle project into an integrated development environment (IDE) such as Eclipse or IntelliJ.

More details and visual examples of cloning the repository and using Gradle Wrapper or an IDE to compile TESTAR can be found in the wiki of the GitHub repository: TESTAR installation

SUT dependencies

TESTAR tool contains the logical functionality to act as an Exploratory FTA in the iv4XR framework, but it cannot connect directly to the SUT. It is necessary to prepare the intermediate SUT-plugin dependencies that allow the FTA agents to observe the SUT State and execute Actions.

• LabRecruits iv4XR demo plugin
• Space Engineers iv4XR plugin extension

Logical Java SUT protocols

TESTAR uses different Java protocols to customize the decision-making of the tool in the different steps of the logical flow execution. The tools already provide default protocols to interact with LabRecruits and Space Engineers, but the users can enrich the possible action to execute, modify the action selection algorithm or apply additional oracles.

For the actions, the user can indicate on which blocks we want to perform actions such as welder, grinder, or interact.

IF block.type IS “CryoChamber” OR “CockpitSeat”

          THEN actions ADD NavigateInteract(block)

And for the oracles, the user can apply additional conditions to indicate to the tool what is considered a failure.

IF lastExecutedAction IS NavigateInteract(block) AND agent.Energy DID NOT increase

              THEN Verdict(WARNING, “Energy was not recharged after functional action”)

The above pseudocode examples need to be customized using the Java programming language and understanding the TESTAR concept of Tags and flow methods. Additional information can be found here: TESTAR Java protocols

Demo of an Exploratory FTA running with Space Engineers

Repositories

TESTAR for XR systems: https://github.com/iv4xr-project/TESTAR_iv4xr

TESTAR for GUI systems: https://github.com/TESTARtool/TESTAR_dev

Papers

• Vos, T. E., Aho, P., Pastor Ricos, F., Rodriguez Valdes, O., & Mulders, A. (2021). testar-scriptless testing through graphical user interface. Software Testing, Verification and Reliability, 31(3), e1771. doi.org/10.1002/stvr.1771
• Pastor Ricós, F. (2022). Scriptless Testing for Extended Reality Systems. In International Conference on Research Challenges in Information Science (pp. 786-794). Springer, Cham. doi.org/10.1007/978-3-031-05760-1_56
• Mulders, A., Valdes, O.R., Ricós, F.P., Aho, P., Marín, B., Vos, T.E.J. (2022). State Model Inference Through the GUI Using Run-Time Test Generation. In: Guizzardi, R., Ralyté, J., Franch, X. (eds) Research Challenges in Information Science. RCIS 2022. Lecture Notes in Business Information Processing, vol 446. Springer, Cham. doi.org/10.1007/978-3-031-05760-1_32
• I. S. W. B. Prasetya, Fernando Pastor Ricós, Fitsum Meshesha Kifetew, Davide Prandi, Samira Shirzadehhajimahmood, Tanja E. J. Vos, Premysl Paska, Karel Hovorka, Raihana Ferdous, Angelo Susi, and Joseph Davidson. 2022. An agent-based approach to automated game testing: an experience report. In Proceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST 2022). Association for Computing Machinery, New York, NY, USA, 1-8. doi.org/10.1145/3548659.3561305