Executable Scenario Management

Logo

The ExSce Management is an approach to store, query and generate test scenarios for ROS-based multi-robot systems. Provenance data about test scenarios and their executions is modeled using PROV and stored on a property graph. Runtime information is obtained from recorded bag files. Metamorphic testing is used to generate new scenarios and validate the system's requirements.

View the Project on GitHub hbrs-sesame/exsce_management

ExSce Management Tutorials

The Executable Scenario Management process. In blue, activities and artefacts for user-defined scenarios. In white, the activities and artefacts that are part of the metamorphic testing.
Figure 1: The Executable Scenario Management process. In blue, activities and artefacts for user-defined scenarios. In white, the activities and artefacts that are part of the metamorphic testing.

Modelling and executing a scenario to record provenance data

Defining a base scenario

The process, shown in Figure 1, starts by having specified a scenario to use as a base, here we use the YAML specification of scenario_c for our example:

Listing 1: Scenario speficication for scenario_c

id: scenario_c
mission:
  id: mission_c
  type: parallel
  allocation:
    - robot: tiago1
      pkg: metamorphic_testing
      file_path: config/tasks/delivery_a.yaml
    - robot: tiago2
      pkg: metamorphic_testing
      file_path: config/tasks/delivery_b.yaml
    - robot: tiago1
      pkg: metamorphic_testing
      file_path: config/tasks/navigate_home_1.yaml
    - robot: tiago2
      pkg: metamorphic_testing
      file_path: config/tasks/navigate_home_2.yaml
environment:
  id: brsu_building_c_with_doors
  models:
    map:
      map_name: brsu_building_c_with_doors
      pkg: floorplan-DSL-environments
      relative_path: maps/
    gazebo_world:
      model_name: brsu_building_c_with_doors
      pkg: floorplan-DSL-environments
      relative_path: worlds/
  robots:
    tiago1:
      start_pose:
        id: classroom_c025-w001
        x: 44.80387496948242
        y: 37.15502166748047
        z: 0.0
        roll: 0.0
        pitch: 0.0
        yaw: 0.0
    tiago2:
      start_pose:
        id: classroom_c025-w002
        x: 43.432926177978516
        y: 38.493873596191406
        z: 0.0
        roll: 0.0
        pitch: 0.0
        yaw: 0.0
robots:
  - robot_id: tiago1
    robot_namespace: tiago1
    robot_type: tiago
  - robot_id: tiago2
    robot_namespace: tiago2
    robot_type: tiago

Execute a scenario and collect baseline data

Next, we execute the scenario and record $n$ number of runs where the SUT performs their behaviour correctly. This requires three terminals.

  1. First launching the scenario:

    roslaunch metamorphic_testing scenario_c.launch

  2. Start the record_exsce_run script:

    rosrun metamorphic_testing record_exsce_run

  3. Start the mission dispatcher:

    rosrun metamorphic_testing mission_dispatcher

The robots will proceed with the scenario execution. When the run ends, the record_exsce_run script will save three files, following the format <ISO Date>_<scenario ID>_<run ID>.<ext>, for example:


 2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag
 2023-06-13T15-48-07_scenario_c_run-e30df0b6.prov.json
 2023-06-13T15-48-07_scenario_c_run-e30df0b6.rosparams

Transformation to PROV

The results of these runs and their provenance are obtained from the run artefacts and added to the Neo4j database. We define the first part of the oracle config to obtain the run results and store their PROV in the database in Listing 2.

Listing 2: Configuration for the bag-to-PROV transformation of scenario_c

id: scenario_c # The ID of the scenario

# The baseline are the metrics are collected among runs of the same scenario
baseline:
  base_scenario: scenario_c # This should always be the same as the scenario ID

  # Metrics that apply to the overall mission
  mission:
    mission_duration: # Metric ID
      topics: # Data sources from which we derive the metrics
        - /tiago1/waypoint_dispatcher/feedback
        - /tiago2/waypoint_dispatcher/feedback
    distance_between_robots:
      topics:
        - /tiago1/robot_pose
        - /tiago2/robot_pose

  # Metrics that apply to individual robots
  robots:
    tiago1: # Robot ID from the scenario specification
      task_duration: # Metric name
        topics: # Data sources from which we derive the metric
        - /tiago1/waypoint_dispatcher/feedback
      waypoints_visited:
        topics:
        - /tiago1/waypoint_dispatcher/result
      distance_travelled:
        topics:
        - /tiago1/distance_travelled
      distance_to_obstacles:
        topics:
        - /tiago1/scan
      max_velocity:
        topics:
        - /tiago1/nav_vel
    tiago2:
      task_duration:
        topics:
        - /tiago2/waypoint_dispatcher/feedback
      waypoints_visited:
        topics:
        - /tiago2/waypoint_dispatcher/result
      distance_travelled:
        topics:
        - /tiago2/distance_travelled
      distance_to_obstacles:
        topics:
        - /tiago2/scan
      max_velocity:
        topics:
        - /tiago2/nav_vel

Then we write the monitors that can process the data from the bag file and return the metric. An excerpt for the max_velocity metric is shown in Listing 3.

Listing 3: Excerpt of the robot monitors that computes the max_velocity metric

class RobotMetrics:
    def __init__(self, robot_id) -> None:
        self.robot_id = robot_id

        self.max_velocity = 0.0 # This must match the metric name in the oracle config

        self.goals = dict()
        self.path = ["start-pose"]
        self.path_segments = dict()

    # This must match the pattern `get_<metric name>`
    def get_max_velocity(self, msg):
        vel = math.sqrt(
            math.pow(msg.linear.x, 2)
            + math.pow(msg.linear.y, 2)
            + math.pow(msg.linear.z, 2)
        )
        if vel > self.max_velocity:
            self.max_velocity = vel

Note that the class attribute matches the metric name used in the oracle config, and the method used to compute the metric is a getter for said attribute, as the oracle and rosbag transform implementation leverages Python’s dynamic imports and instantiation.

To transform the runs and store them in the PROV database, make sure you have the Neo4j database running, and use the following command:

cd scripts
./transform_rosbag ../runs/2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag --config ../config/oracle/scenario_c.yaml

Defining a baseline oracle and validating new executions

After we have added a sufficient number of baseline runs to our PROV database, we now update the baseline oracle config file by adding the invariant relations for the safety metrics (which are constant, as described in the Oracles section, as shown in Listing 4.

Listing 4: Baseline oracle configuration for scenario_c

id: scenario_c
baseline:
  base_scenario: scenario_c
  mission:
    mission_duration:
      topics:
        - /tiago1/waypoint_dispatcher/feedback
        - /tiago2/waypoint_dispatcher/feedback
      relationship:
        package: exsce.metamorphic.relationships
        name: invariant
    distance_between_robots:
      topics:
        - /tiago1/robot_pose
        - /tiago2/robot_pose
      baseline:
        limit: 0.1
      relationship:
        package: exsce.metamorphic.relationships
        name: above_min
  robots:
    tiago1:
      task_duration:
        topics:
        - /tiago1/waypoint_dispatcher/feedback
        relationship:
          package: exsce.metamorphic.relationships
          name: invariant
      waypoints_visited:
        topics:
        - /tiago1/waypoint_dispatcher/result
        relationship:
          package: exsce.metamorphic.relationships
          name: invariant
      distance_travelled:
        topics:
        - /tiago1/distance_travelled
        relationship:
          package: exsce.metamorphic.relationships
          name: invariant
      distance_to_obstacles:
        topics:
        - /tiago1/scan
        baseline:
          limit: 0.1
        relationship:
          package: exsce.metamorphic.relationships
          name: above_min
      max_velocity:
        topics:
        - /tiago1/nav_vel
        baseline:
          limit: 1.0
        relationship:
          package: exsce.metamorphic.relationships
          name: below_max
    tiago2:
      task_duration:
        topics:
        - /tiago2/waypoint_dispatcher/feedback
        relationship:
          package: exsce.metamorphic.relationships
          name: invariant
      waypoints_visited:
        topics:
        - /tiago2/waypoint_dispatcher/result
        relationship:
          package: exsce.metamorphic.relationships
          name: invariant
      distance_travelled:
        topics:
        - /tiago2/distance_travelled
        relationship:
          package: exsce.metamorphic.relationships
          name: invariant
      distance_to_obstacles:
        topics:
        - /tiago2/scan
        baseline:
          limit: 0.1
        relationship:
          package: exsce.metamorphic.relationships
          name: above_min
      max_velocity:
        topics:
        - /tiago2/nav_vel
        baseline:
          limit: 1.0
        relationship:
          package: exsce.metamorphic.relationships
          name: below_max

Finally, we can use the baseline test oracle to automatically validate new baseline runs based on their output relation:

cd scripts
./transform_rosbag ../runs/2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag --config ../config/oracle/scenario_c.yaml --validate

This classifies the results based on whether the baseline relationship holds or not.

Generating new scenarios

Manually defining a new scenario

Let us manually define one new scenario based on scenario_c.yaml. In this minimal example, we’ll be removing tiago2 and reassigning its tasks to tiago1.

First we update the mission, as shown in Listing 5, by changing the allocation of delivery_b.yaml to tiago1 and removing the navigate_home_2.yaml task for tiago2.

Listing 5: A new mission specification after removing tiago2 manually

mission:
  id: mission_c
  type: parallel
  allocation:
    - robot: tiago1
      pkg: metamorphic_testing
      file_path: config/tasks/delivery_a.yaml
    - robot: tiago1
      pkg: metamorphic_testing
      file_path: config/tasks/delivery_b.yaml
    - robot: tiago1
      pkg: metamorphic_testing
      file_path: config/tasks/navigate_home_1.yaml

We also remove tiago2 from the list of robots, and its starting position from the environment.robots (as shown in Listing 6):

Listing 6: Environment specification after removing tiago2 in a scenario speciication

environment:
  robots:
    tiago1:
      start_pose:
        id: classroom_c025-w001
        x: 44.80387496948242
        y: 37.15502166748047
        z: 0.0
        roll: 0.0
        pitch: 0.0
        yaw: 0.0
robots:
  - robot_id: tiago1
    robot_namespace: tiago1
    robot_type: tiago

Finally, the oracle config is updated similarly to the above, by removing topics and metrics related to tiago2.

Generating new scenarios by applying input transformations

As mentioned in Metamorphic Relations, we have pre-defined some transformations. We can write a script to use these functions and recreate the example from the previous section, for example:

Listing 7: Example of a scenario transformation script that removes tiago2 and reassigns its tasks to tiago1

    # scenario_config is the scenario specification.
    # limits is a list of metrics that we consider constant,
    # and which should not change between the original and derived scenario
    scenario = ScenarioTransformations(scenario_config, prov_api, limits)

    # Input transformation
    scenario.remove_robot("tiago2", assignment={"tiago2": "tiago1"})

    # Calling `generate() creates and updates the PROV documents,
    # and generates a new oracle config file for the new scenario
    scenario.generate()

The script above will create the new scenario, its oracle configuration, and save both of them in the PROV database.