Executable Scenario Management

Logo

The ExSce Management is an approach to store, query and generate test scenarios for ROS-based multi-robot systems. Provenance data about test scenarios and their executions is modeled using PROV and stored on a property graph. Runtime information is obtained from recorded bag files. Metamorphic testing is used to generate new scenarios and validate the system's requirements.

View the Project on GitHub hbrs-sesame/exsce_management

Metamorphic Relations

Input transformations

New scenarios are generated by applying input transformations to existing scenarios. The scenario genration is an activity that is informed by the individual transformations to the inputs (each modelled as an activity as well). The sequence of transformations are modelled with wasInformedBy, and the first and transformation are the activities associated with the start and end.

Listing 1: Excerpt of the PROV models for a scenario transformation.

bundle scenario_780741116:scenario_generation
  activity(scenario_780741116:generation, -, -)
  wasGeneratedBy(exsce:scenario_780741116, scenario_780741116:generation, -)
  used(scenario_780741116:generation, exsce:scenario_c, -)

  activity(scenario_780741116:remove_robot-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_robot", exsce:robot="tiago2"])
  wasStartedBy(scenario_780741116:generation, -, scenario_780741116:remove_robot-tiago2, -)

  ...

  activity(scenario_780741116:remove_start_pose-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_start_pose"])
  wasInformedBy(scenario_780741116:remove_start_pose-tiago2, scenario_780741116:substitute_allocation-allocation_01-task_03)
  wasEndedBy(scenario_780741116:generation, -, scenario_780741116:remove_start_pose-tiago2, -)

endBundle

Below we describe the PROV models for some of the transformations we considered for this case study. Note that transformations can be composed of other transformations (e.g. removing a robot).

Mission

Add allocated task

To add a new allocated task, one needs to specify the task, the robot the task is allocated to, and the position in which to insert the new task. Listing 2 shows how this is modelled in PROV.

Listing 2: PROV models for a scenario transformation where a new task is added and allocated

bundle new_scenario:scenario_generation
  activity(new_scenario:add_task-config/tasks/delivery_a.yaml, -, -, [prov:type="exsce:transform", exsce:transform="add_task", exsce:pkg="metamorphic_testing", exsce:allocation_order=1, exsce:robot="tiago1"])
endBundle

Remove task

Listing 3 and Listing 4 show the PROV models for removing tasks that match a task ID and a robot ID, respectively.:

Listing 3: Scenario transformation by removing a task that matches a task ID

bundle new_scenario:scenario_generation
  activity(new_scenario:remove_task-config/tasks/navigate_home_1.yaml, -, -, [prov:type="exsce:transform", exsce:transform="remove_task"])
endBundle

Listing 4: Scenario transformation by removing a task that matches a robot ID

bundle new_scenario:scenario_generation
  activity(new_scenario:remove_task-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_task"])
endBundle

Substitute task

Substituting tasks in the mission adds one activity per task to be substituted. Listing 5 shows the PROV-N representation of this transformation.

Listing 5: Scenario transformation: Substituting tasks

bundle new_scenario:scenario_generation
  activity(new_scenario:substitute_tasks-config/tasks/delivery_b.yaml, -, -, [prov:type="exsce:transform", exsce:transform="substitute_tasks", prov:value="config/tasks/delivery_c.yaml"])
endBundle

Substitute allocation

The transformation in Listing 6 requires the specification of a new allocation as described in the Scenario Specification.

Listing 6: Scenario transformation to substitute a scenario’s task allocation

bundle new_scenario:scenario_generation
  activity(new_scenario:substitute_allocation-allocation_01-task_01, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_a.yaml", exsce:allocation_order=1, exsce:robot="tiago1"])

  activity(new_scenario:substitute_allocation-allocation_01-task_02, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_2.yaml", exsce:allocation_order=2, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_02, new_scenario:substitute_allocation-allocation_01-task_01)

  activity(new_scenario:substitute_allocation-allocation_01-task_03, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_b.yaml", exsce:allocation_order=3, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_03, new_scenario:substitute_allocation-allocation_01-task_02)

  activity(new_scenario:substitute_allocation-allocation_01-task_04, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_1.yaml", exsce:allocation_order=4, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_04, new_scenario:substitute_allocation-allocation_01-task_03)
endBundle

Swap assignments

The transformation in Listing 7 takes the existing allocation and changes the assignment according to a set of key-value pairs, where the value is the robot ID that should substitute the robot ID in the key. The new assignments are then applied by using substitute allocation.

Listing 7: Scenario transformation: swap task assignments between robots in the scenario

bundle new_scenario:scenario_generation

  activity(new_scenario:swap_assignment-tiago1, -, -, [prov:type="exsce:transform", exsce:transform="swap_assignment", prov:value="tiago2", exsce:robot="tiago1"])

  activity(new_scenario:swap_assignment-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="swap_assignment", prov:value="tiago1", exsce:robot="tiago2"])
  wasInformedBy(new_scenario:swap_assignment-tiago2, new_scenario:swap_assignment-tiago1)

  activity(new_scenario:substitute_allocation-allocation_01-task_01, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_a.yaml", exsce:allocation_order=1, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_01, new_scenario:swap_assignment-tiago2)

  activity(new_scenario:substitute_allocation-allocation_01-task_02, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_b.yaml", exsce:allocation_order=2, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_02, new_scenario:substitute_allocation-allocation_01-task_01)

  activity(new_scenario:substitute_allocation-allocation_01-task_03, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_1.yaml", exsce:allocation_order=3, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_03, new_scenario:substitute_allocation-allocation_01-task_02)

  activity(new_scenario:substitute_allocation-allocation_01-task_04, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_2.yaml", exsce:allocation_order=4, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_04, new_scenario:substitute_allocation-allocation_01-task_03)
endBundle

Reverse task order

This transformation uses substitute allocation to invert the order in which the tasks should the completed. Listing 8 shows this transformation’s provenance.

Listing 8: PROV models for a scenario transformation that reverses the task ordering

bundle new_scenario:scenario_generation
  activity(new_scenario:reverse_task_order-mission, -, -, [prov:type="exsce:transform", exsce:transform="reverse_task_order"])

  activity(new_scenario:substitute_allocation-allocation_01-task_01, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_b.yaml", exsce:allocation_order=1, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_01, new_scenario:reverse_task_order-mission)

  activity(new_scenario:substitute_allocation-allocation_01-task_02, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_a.yaml", exsce:allocation_order=2, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_02, new_scenario:substitute_allocation-allocation_01-task_01)

  activity(new_scenario:substitute_allocation-allocation_01-task_03, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_2.yaml", exsce:allocation_order=3, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_03, new_scenario:substitute_allocation-allocation_01-task_02)

  activity(new_scenario:substitute_allocation-allocation_01-task_04, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_1.yaml", exsce:allocation_order=4, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_04, new_scenario:substitute_allocation-allocation_01-task_03)
endBundle

Task

A transformation to reverse the order in which waypoints of a task are visited generates a new task specification. Then it uses substitute task to replace the original task with the new specification that contains the inverted waypoints. This transformation is domain-specific, as the nature of the application determines whether the order of waypoints can be inverted or if it requires some preconditions or application logic.

Listing 9: PROV models for the scenario transformation to reverse the waypoint sequence in a task

bundle new_scenario:scenario_generation

  activity(new_scenario:reverse_waypoint_order-config/tasks/delivery_a.yaml, -, -, [prov:type="exsce:transform", exsce:transform="reverse_waypoint_order"])
  used(new_scenario:reverse_waypoint_order-config/tasks/delivery_a.yaml, task:delivery_a.yaml, -)
  wasGeneratedBy(task:delivery_a_reversed.yaml, new_scenario:reverse_waypoint_order-config/tasks/delivery_a.yaml, -)

  activity(new_scenario:substitute_tasks-config/tasks/delivery_a.yaml, -, -, [prov:type="exsce:transform", exsce:transform="substitute_tasks", prov:value="config/tasks/delivery_a_reversed.yaml"])
  wasInformedBy(new_scenario:substitute_tasks-config/tasks/delivery_a.yaml, new_scenario:reverse_waypoint_order-config/tasks/delivery_a.yaml)
endBundle

Environment

Swap start poses

Similar to swap assignment, we can change robots’ starting positions using key-value pairs the robot IDs involved in the swap, as shown in Listing 10.

Listing 10: PROV models to represent a scenaroi transformation by swapping robots’ starting positions

bundle new_scenario:scenario_generation
  activity(new_scenario:swap_start_poses-tiago1, -, -, [prov:type="exsce:transform", exsce:transform="swap_start_poses", prov:value="tiago2"])

  activity(new_scenario:swap_start_poses-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="swap_start_poses", prov:value="tiago1"])
  wasInformedBy(new_scenario:swap_start_poses-tiago2, new_scenario:swap_start_poses-tiago1)
endBundle

Robots

Add robot

Adding a new robot requires that we know its specification (as described in the Scenario Specification) and its starting position. The transformation uses then other transformations for the tasks related to this new robot, e.g., with add task (Listing 11) new tasks can be added to the mission. In addition, one can modify the allocation by using (1) substitute allocation (Listing 12) or by (2) using swap assignments to reassign tasks from one robot to the new one (as in Listing 13).

Listing 11: PROV models for adding a new robot with new tasks in a scenario transform

bundle new_scenario:scenario_generation
  activity(new_scenario:add_task-config/tasks/delivery_c.yaml, -, -, [prov:type="exsce:transform", exsce:transform="add_task", exsce:pkg="metamorphic_testing", exsce:allocation_order=4, exsce:robot="tiago3"])
  activity(new_scenario:add_start_pose-new-pose, -, -, [prov:type="exsce:transform", exsce:transform="add_start_pose", exsce:robot="tiago3"])
  wasInformedBy(new_scenario:add_start_pose-new-pose, new_scenario:add_task-config/tasks/delivery_c.yaml)
  activity(new_scenario:add_robot-tiago3, -, -, [prov:type="exsce:transform", exsce:transform="add_robot"])
  wasInformedBy(new_scenario:add_robot-tiago3, new_scenario:add_start_pose-new-pose)
endBundle

Listing 12: PROV models for adding a new robot with a new allocation in a scenario transform

bundle new_scenario:scenario_generation
  activity(new_scenario:substitute_allocation-allocation_01-task_01, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_c.yaml", exsce:allocation_order=1, exsce:robot="tiago1"])

  activity(new_scenario:substitute_allocation-allocation_01-task_02, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_2.yaml", exsce:allocation_order=2, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_02, new_scenario:substitute_allocation-allocation_01-task_01)

  activity(new_scenario:substitute_allocation-allocation_01-task_03, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_b.yaml", exsce:allocation_order=3, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_03, new_scenario:substitute_allocation-allocation_01-task_02)

  activity(new_scenario:substitute_allocation-allocation_01-task_04, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_1.yaml", exsce:allocation_order=4, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_04, new_scenario:substitute_allocation-allocation_01-task_03)

  activity(new_scenario:substitute_allocation-allocation_01-task_05, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_a.yaml", exsce:allocation_order=5, exsce:robot="tiago3"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_05, new_scenario:substitute_allocation-allocation_01-task_04)

  activity(new_scenario:add_start_pose-new-pose, -, -, [prov:type="exsce:transform", exsce:transform="add_start_pose", exsce:robot="tiago3"])
  wasInformedBy(new_scenario:add_start_pose-new-pose, new_scenario:substitute_allocation-allocation_01-task_05)

  activity(new_scenario:add_robot-tiago3, -, -, [prov:type="exsce:transform", exsce:transform="add_robot"])
  wasInformedBy(new_scenario:add_robot-tiago3, new_scenario:add_start_pose-new-pose)
endBundle

Listing 13: PROV models for adding a new robot and reassigning existing tasks to it in a scenario transform

bundle new_scenario:scenario_generation
  activity(new_scenario:swap_assignment-tiago3, -, -, [prov:type="exsce:transform", exsce:transform="swap_assignment", prov:value="tiago1", exsce:robot="tiago3"])

  activity(new_scenario:substitute_allocation-allocation_01-task_01, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_a.yaml", exsce:allocation_order=1, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_01, new_scenario:swap_assignment-tiago3)

  activity(new_scenario:substitute_allocation-allocation_01-task_02, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_b.yaml", exsce:allocation_order=2, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_02, new_scenario:substitute_allocation-allocation_01-task_01)

  activity(new_scenario:substitute_allocation-allocation_01-task_03, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_1.yaml", exsce:allocation_order=3, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_03, new_scenario:substitute_allocation-allocation_01-task_02)

  activity(new_scenario:substitute_allocation-allocation_01-task_04, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_2.yaml", exsce:allocation_order=4, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_04, new_scenario:substitute_allocation-allocation_01-task_03)

  activity(new_scenario:add_task-config/tasks/delivery_c.yaml, -, -, [prov:type="exsce:transform", exsce:transform="add_task", exsce:pkg="metamorphic_testing", exsce:allocation_order=4, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:add_task-config/tasks/delivery_c.yaml, new_scenario:substitute_allocation-allocation_01-task_04)

  activity(new_scenario:add_start_pose-new-pose, -, -, [prov:type="exsce:transform", exsce:transform="add_start_pose", exsce:robot="tiago3"])
  wasInformedBy(new_scenario:add_start_pose-new-pose, new_scenario:add_task-config/tasks/delivery_c.yaml)

  activity(new_scenario:add_robot-tiago3, -, -, [prov:type="exsce:transform", exsce:transform="add_robot"])
  wasInformedBy(new_scenario:add_robot-tiago3, new_scenario:add_start_pose-new-pose)
endBundle

Remove robot

Similarly, removing a robot uses additional transformations to handle the tasks of the robot being removed. The simplest case is to remove the tasks of the robot in question (Listing 14). Alternatively, one can specify a new allocation that updates the assignment (Listing 15) or one can reassign the tasks of the removed robot to another robot (Listing 16).

Listing 14: Scenario transformation: Remove a robot and its tasks

bundle new_scenario:scenario_generation
  activity(new_scenario:remove_robot-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_robot", exsce:robot="tiago2"])

  activity(new_scenario:remove_task-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_task"])
  wasInformedBy(new_scenario:remove_task-tiago2, new_scenario:remove_robot-tiago2)

  activity(new_scenario:remove_start_pose-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_start_pose"])
  wasInformedBy(new_scenario:remove_start_pose-tiago2, new_scenario:remove_task-tiago2)
endBundle

Listing 15: Scenario transformation: Remove a robot and add a replace the task allocation

bundle new_scenario:scenario_generation
  activity(new_scenario:remove_robot-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_robot", exsce:robot="tiago2"])

  activity(new_scenario:substitute_allocation-allocation_01-task_01, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_c.yaml", exsce:allocation_order=1, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_01, new_scenario:remove_robot-tiago2)

  activity(new_scenario:substitute_allocation-allocation_01-task_02, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_b.yaml", exsce:allocation_order=2, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_02, new_scenario:substitute_allocation-allocation_01-task_01)

  activity(new_scenario:substitute_allocation-allocation_01-task_03, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_1.yaml", exsce:allocation_order=3, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_03, new_scenario:substitute_allocation-allocation_01-task_02)

  activity(new_scenario:remove_start_pose-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_start_pose"])
  wasInformedBy(new_scenario:remove_start_pose-tiago2, new_scenario:substitute_allocation-allocation_01-task_03)
endBundle

Listing 16: Scenario transformation: Removing a robot and reassigning its tasks

bundle new_scenario:scenario_generation
  activity(new_scenario:remove_robot-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_robot", exsce:robot="tiago2"])

  activity(new_scenario:swap_assignment-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="swap_assignment", prov:value="tiago1", exsce:robot="tiago2"])
  wasInformedBy(new_scenario:swap_assignment-tiago2, new_scenario:remove_robot-tiago2)

  activity(new_scenario:substitute_allocation-allocation_01-task_01, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_a.yaml", exsce:allocation_order=1, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_01, new_scenario:swap_assignment-tiago2)

  activity(new_scenario:substitute_allocation-allocation_01-task_02, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_b.yaml", exsce:allocation_order=2, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_02, new_scenario:substitute_allocation-allocation_01-task_01)

  activity(new_scenario:substitute_allocation-allocation_01-task_03, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_1.yaml", exsce:allocation_order=3, exsce:robot="tiago1"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_03, new_scenario:substitute_allocation-allocation_01-task_02)

  activity(new_scenario:remove_start_pose-tiago2, -, -, [prov:type="exsce:transform", exsce:transform="remove_start_pose"])
  wasInformedBy(new_scenario:remove_start_pose-tiago2, new_scenario:substitute_allocation-allocation_01-task_03)
endBundle

Replace robot

Replacing a robot is the combination of add robot and remove robot, and swapping the assignments and start position of the new robot with the old one. Listing 17 shows the provenance of this transformation.

Listing 17: PROV models for a transformation to replace a robot with a new one

bundle new_scenario:scenario_generation
  activity(new_scenario:replace_robot-tiago1, -, -, [prov:type="exsce:transform", exsce:transform="replace_robot"])

  activity(new_scenario:add_start_pose-c025-w001, -, -, [prov:type="exsce:transform", exsce:transform="add_start_pose", exsce:robot="tiago3"])
  wasInformedBy(new_scenario:add_start_pose-c025-w001, new_scenario:replace_robot-tiago1)

  activity(new_scenario:add_robot-tiago3, -, -, [prov:type="exsce:transform", exsce:transform="add_robot"])
  wasInformedBy(new_scenario:add_robot-tiago3, new_scenario:add_start_pose-c025-w001)

  activity(new_scenario:remove_robot-tiago1, -, -, [prov:type="exsce:transform", exsce:transform="remove_robot", exsce:robot="tiago1"])
  wasInformedBy(new_scenario:remove_robot-tiago1, new_scenario:add_robot-tiago3)

  activity(new_scenario:swap_assignment-tiago1, -, -, [prov:type="exsce:transform", exsce:transform="swap_assignment", prov:value="tiago3", exsce:robot="tiago1"])
  wasInformedBy(new_scenario:swap_assignment-tiago1, new_scenario:remove_robot-tiago1)

  activity(new_scenario:substitute_allocation-allocation_01-task_01, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_a.yaml", exsce:allocation_order=1, exsce:robot="tiago3"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_01, new_scenario:swap_assignment-tiago1)

  activity(new_scenario:substitute_allocation-allocation_01-task_02, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/delivery_b.yaml", exsce:allocation_order=2, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_02, new_scenario:substitute_allocation-allocation_01-task_01)

  activity(new_scenario:substitute_allocation-allocation_01-task_03, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_1.yaml", exsce:allocation_order=3, exsce:robot="tiago3"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_03, new_scenario:substitute_allocation-allocation_01-task_02)

  activity(new_scenario:substitute_allocation-allocation_01-task_04, -, -, [prov:type="exsce:transform", exsce:transform="substitute_allocation", exsce:pkg="metamorphic_testing", exsce:file_path="config/tasks/navigate_home_2.yaml", exsce:allocation_order=4, exsce:robot="tiago2"])
  wasInformedBy(new_scenario:substitute_allocation-allocation_01-task_04, new_scenario:substitute_allocation-allocation_01-task_03)

  activity(new_scenario:remove_start_pose-tiago1, -, -, [prov:type="exsce:transform", exsce:transform="remove_start_pose"])
  wasInformedBy(new_scenario:remove_start_pose-tiago1, new_scenario:substitute_allocation-allocation_01-task_04)
endBundle

Output relations

The output relations take advantage of the information stored in the property graph. First, we query the baseline relationships of base scenario. To get the oracle config for the new scenario, we copy the safety relations from the base scenario; we assume these are invariant relations, which don’t change. Next we get the new expected outputs by computing basic output relations (e.g., counting the number of waypoints in free space of the tasks in the new scenario) or querying existing path segments for complex relations. For example, we query the average time it takes a robot to travel between each pair of waypoints in the path that results from the new task, and aggregate them. If no data has been recorded for the metric in question for that particular path segment, we use an estimate. Finally, we compare the new expected outcomes against the base scenario and identify which basic relation applies.

Listing 18 shows an example of the generated oracle configuration, containing the baseline and metamorphic oracles for a transformation where we removed tiago2. Note that the baseline is with respect to itself, but the metamorphic config uses the results of the scenario (scenario_c) it was derived from. This has the advantage of being able to validate its results once enough data has been collected from runs of this new scenario, but also being able to use any new data collected in scenario_c for its validation.

Listing 18: Adding output relations to oracle configuration file

id: new_scenario
baseline:
  base_scenario: new_scenario
  mission:
    mission_duration:
      baseline:
        tolerance: 0.0
        value: 141.19514473468308
      relationship:
        name: invariant
        package: exsce.metamorphic.relationships
      topics:
        - /tiago1/waypoint_dispatcher/feedback
  robots:
    tiago1:
      distance_travelled:
        baseline:
          tolerance: 0.0
          value: 98.24204030780109
        relationship:
          name: invariant
          package: exsce.metamorphic.relationships
        topics:
          - /tiago1/distance_travelled
      task_duration:
        baseline:
          tolerance: 0.0
          value: 141.19514473468308
        relationship:
          name: invariant
          package: exsce.metamorphic.relationships
        topics:
          - /tiago1/waypoint_dispatcher/feedback
      waypoints_visited:
        baseline:
          tolerance: null
          value: 5
        relationship:
          name: invariant
          package: exsce.metamorphic.relationships
        topics:
          - /tiago1/waypoint_dispatcher/result
metamorphic:
  base_scenario: scenario_c
  mission:
    mission_duration:
      baseline:
        delta: 43.58914473468309
        tolerance: 0.0
        value: 141.19514473468308
      relationship:
        name: increasing
        package: exsce.metamorphic.relationships
      topics:
        - /tiago1/waypoint_dispatcher/feedback
  robots:
    tiago1:
      distance_travelled:
        baseline:
          delta: 47.539097638514015
          tolerance: 0.0
          value: 50.702942669287076
        relationship:
          name: increasing
          package: exsce.metamorphic.relationships
        topics:
          - /tiago1/distance_travelled
      task_duration:
        baseline:
          delta: 43.58914473468309
          tolerance: 0.0
          value: 97.606
        relationship:
          name: increasing
          package: exsce.metamorphic.relationships
        topics:
          - /tiago1/waypoint_dispatcher/feedback
      waypoints_visited:
        baseline:
          delta: 2
          tolerance: null
          value: 3
        relationship:
          name: increasing
          package: exsce.metamorphic.relationships
        topics:
          - /tiago1/waypoint_dispatcher/result

As shown in Listing 19, the oracle’s PROV is the same activity using the same topics as data sources, but with different usage relationships for the “baseline” and “metamorphic” oracles, each corresponding to the scenario that provides the runs to use as source data for the metric in question.

Listing 19: PROV models for a baseline oracle

bundle exsce:scenario_780741116_bundle
  entity(exsce:scenario_780741116, [prov:type="exsce:ConcreteScenario"])

  activity(scenario_780741116:mission_duration, -, -, [prov:type="exsce:oracle", oracle:metric="mission_duration", oracle:robot="None"])
  used(scenario_780741116:mission_duration, exsce:scenario_780741116, -, [oracle:relationship="invariant", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:value="141.195" %% xsd:float, oracle:tolerance="0" %% xsd:float])
  used(scenario_780741116:mission_duration, exsce:scenario_c, -, [oracle:relationship="increasing", oracle:package="exsce.metamorphic.relationships", oracle:type="metamorphic", oracle:value="141.195" %% xsd:float, oracle:tolerance="0" %% xsd:float, oracle:delta="43.5891" %% xsd:float])

  used(scenario_780741116:mission_duration, topic:/tiago1/waypoint_dispatcher/feedback, -, [prov:role="topic"])
  used(scenario_780741116:mission_duration, topic:/tiago1/waypoint_dispatcher/feedback, -, [prov:role="topic"])

endBundle

Estimating metrics

As an example, we present a simplistic approach to estimate the task_duration and distance_travelled metrics. The exact way of estimating missing data should consider the context-specific requirements that take into account robot, application, and domain specific knowledge.

Points of reference used to estimate the task_duration and distance_travelled metrics when no available data exists
Figure 1: Points of reference used to estimate the task_duration and distance_travelled metrics when no available data exists

Consider the path segment between two waypoints $p_1$ and $p_2$ (cf. Figure 1) which does not have data available for the metric $m_{1\rightarrow2}$ that we are interested in, neither in the direction of travel of the task $p_1 \rightarrow p_2$ nor in the opposite direction $p_1 \leftarrow p_2$, i.e., $m_{2\rightarrow1}$. To estimate the metric, we first query the PROV database for all the poses in the same room as $p_1$ and $p_2$, and sort them by the shortest distance, respectively. Let us say that $p_3$ and $p_4$ are the closest poses to $p_1$ and $p_2$, respectively. We query the metrics for the path segments between $(p_3, p_2)$, $(p_1, p_4)$ and between $(p_3, p_4)$ (in either direction, but preferring the same traveling direction as the task):

\[m_{3\leftrightarrow2} = query\_metric(metric, p_3, p_2)\\\] \[m_{1\leftrightarrow4} = query\_metric(metric, p_1, p_4)\\\] \[m_{3\leftrightarrow4} = query\_metric(metric, p_3, p_4)\]

We then compute the average of the proportional estimates based on euclidean distance, where $n$ is the number of path segments $m_{i\leftrightarrow j}$ with available data:

\[m_{1\rightarrow2} = \frac{1}{n} \sum_{k=1}^{n} m_{i\leftrightarrow j} \frac{dist(p_i, p_j)}{dist(p_1, p_2)}\]

Note that $query_metric$ returns data from recorded metrics from existing runs, and the euclidean distance propotion is used to scale existing metrics taking into account the difference between two points in the same room, e.g. the path segments $(p_1, p_2)$ and $(p_3, p_2)$.