Executable Scenarios

Specification

We consider the scenario specification as the description of the test input data, meaning we model the scenario components to the level of granularity that is relevant for our tests and what inputs we wish to vary. A scenario specification for our case study consists of the following basic ingredients:

A unique ID,
a mission that includes the tasks to be executed, their ordering, and their (static) task allocation,
the robots which have been assigned to this mission, and
the environment in which the scenario is to be executed, including the starting position of each robot and any obstacles.

Figure 1: Overview of the PROV concepts and relations used in the ExSce Management

The specifications used in this example are available here. Note that although we currently only support YAML for the specification in our case study, the way in which a scenario is specified does not matter as long as a corresponding model-to-model transformation between the specification and the PROV models is available. Figure 1 shows an overview of the PROV concepts and relations described in this section. In this section, we show both the YAML-based specification, and the resulting transformation (conforming to the PROV meta-models described in this section) to PROV-N (a notation of PROV aimed at human consumption, e.g. Listing 1). In practice, we use the prov library, which can serialize PROV documents to JSON.

Listing 1: A scenario is modelled as a PROV collection, which has a mission, the environment and the robots as members.

bundle exsce:scenario_c_bundle
  entity(exsce:scenario_c, [prov:type="exsce:ConcreteScenario"])

  hadMember(exsce:scenario_c, mission:mission_c)
  hadMember(exsce:scenario_c, env:brsu_building_c_with_doors)
  hadMember(exsce:scenario_c, robot:tiago1)
  hadMember(exsce:scenario_c, robot:tiago2)
endBundle

Mission

For our case study we consider two types of missions: those that can be executed in parallel (each robot executes one task at a time) or sequential (only one robot is executing a task at any given time).

Listing 2: Mission specification in YAML format

mission:
  id: mission_c
  type: parallel
  allocation:
    - robot: tiago1
      pkg: metamorphic_testing
      file_path: config/tasks/delivery_a.yaml
    - robot: tiago2
      pkg: metamorphic_testing
      file_path: config/tasks/delivery_b.yaml
    - robot: tiago1
      pkg: metamorphic_testing
      file_path: config/tasks/navigate_home_1.yaml
    - robot: tiago2
      pkg: metamorphic_testing
      file_path: config/tasks/navigate_home_2.yaml

The mission is a collection that models the specific instance of the tasks chosen and to which robot they were allocated to. The tasks are modelled separately to enable their reuse (they can be assigned multiple times to the same or another robot), and as we mentioned before, to abstract their concrete, domain or application-specific representation. Similarly, this allows us to use, for example, the tasks generated by the Floorplan DSL.

Listing 3: PROV models to represent a mission.

bundle exsce:scenario_c_bundle
  entity(mission:mission_c, [prov:type="exsce:mission", exsce:missionType="parallel"])
  entity(mission_c:tiago1-task_01, [prov:type="prov:Plan"])
  entity(mission_c:tiago2-task_01, [prov:type="prov:Plan"])
  entity(mission_c:tiago1-task_02, [prov:type="prov:Plan"])
  entity(mission_c:tiago2-task_02, [prov:type="prov:Plan"])

  hadMember(mission:mission_c, mission_c:tiago1-task_01)
  hadMember(mission:mission_c, mission_c:tiago2-task_01)
  hadMember(mission:mission_c, mission_c:tiago1-task_02)
  hadMember(mission:mission_c, mission_c:tiago2-task_02)

  wasDerivedFrom(mission_c:tiago1-task_01, task:delivery_a.yaml, -, -, -, [prov:type="prov:PrimarySource"])
  wasDerivedFrom(mission_c:tiago2-task_01, task:delivery_b.yaml, -, -, -, [prov:type="prov:PrimarySource"])
  wasDerivedFrom(mission_c:tiago1-task_02, task:navigate_home_1.yaml, -, -, -, [prov:type="prov:PrimarySource"])
  wasDerivedFrom(mission_c:tiago2-task_02, task:navigate_home_2.yaml, -, -, -, [prov:type="prov:PrimarySource"])
endBundle

Tasks

In this example, the tasks have been implicitly included as part of the allocation. Note also that there are different levels of abstraction, the scenario specification does not constrain how tasks should be specified, allowing us to reuse the scenario specification, and likely any task-agnostic transformations, for other types of systems with different task specification formats (given that there is a model to model transformation of the task to PROV).

A task is an entity which is derived from its subtask; each subtask in turn, is derived from the poses it contains as waypoints.

Listing 4: PROV models for a navigation task

bundle task:delivery_a.yaml_bundle

  entity(task:delivery_a.yaml, [prov:type="prov:Plan", ros:pkg="metamorphic_testing", ros:relative_path="config/tasks/delivery_a.yaml"])

  entity(task:delivery_a.yaml-subtask_001, [prov:type="prov:Plan"])

  wasDerivedFrom(task:delivery_a.yaml, task:delivery_a.yaml-subtask_001, -, -, -, [prov:role="subtask_001"])
  wasDerivedFrom(delivery_a.yaml-subtask_001:w_001; task:delivery_a.yaml-subtask_001, pose:c022-w001-R0.00_P0.00_Y2.27, -, -, -, [prov:role="waypoint"])
  wasDerivedFrom(delivery_a.yaml-subtask_001:w_002; task:delivery_a.yaml-subtask_001, pose:c018-w001-R0.00_P0.00_Y-0.64, -, -, -, [prov:role="waypoint"])
endBundle

Allocation

In this case study, we have considered pre-defined assignments in the scenario specification. Note, however, that the allocation does not appear in the mission PROV shown above. To account for the cases where an algorithm allocates tasks at runtime, we collect the allocation as part of the runtime information. For this use case, this is done by obtaining the allocation from the scenario parameters before sending the tasks to the robots. We foresee that using an online allocator will require changes to capture the allocations depending on their implementation and how they are made available, perhaps as part of the model-to-model transformation discussed in Scenario Execution.

As an example, Listing 5 shows the PROV models for the allocated tasks of tiago1. Each task is modelled as an activity assigned to a robot using the wasAssociatedWith relationship, with each task modelled in the mission added as the plan. Each Action is also an activity that uses the wasStartedBy to model the starting pose of the robot, and wasEndedBy its goal. The sequence of tasks is modelled via the wasInformedBy relationship, because, in this case study, activities are dependent on the successful completion of their previous activity.

Listing 5: PROV models for the (run-time) task allocation for tiago1.

bundle run:run_12345_bundle

  activity(run_12345:tiago1-task_01, -, -, [prov:type="exsce:task_execution", exsce:run="12345"])
  used(run_12345:tiago1-task_01, robot:tiago1, -)
  wasAssociatedWith(run_12345:tiago1-task_01, tiago1:agent, mission_c:tiago1-task_01)

  activity(run_12345:tiago1-task_01-subtask_001-w_001, -, -, [prov:type="exsce:action", exsce:run="12345"])
  wasEndedBy(run_12345:tiago1-task_01-subtask_001-w_001, pose:c022-w001-R0.00_P0.00_Y2.27, -, -, [prov:role="goal", exsce:runs="12345"])
  wasStartedBy(run_12345:tiago1-task_01-subtask_001-w_001, pose:c025-w001-R0.00_P0.00_Y0.00, -, -)
  wasAssociatedWith(run_12345:tiago1-task_01-subtask_001-w_001, tiago1:agent, -)

  activity(run_12345:tiago1-task_01-subtask_001-w_002, -, -, [prov:type="exsce:action", exsce:run="12345"])
  wasInformedBy(run_12345:tiago1-task_01-subtask_001-w_002, run_12345:tiago1-task_01-subtask_001-w_001)
  wasEndedBy(run_12345:tiago1-task_01-subtask_001-w_002, pose:c018-w001-R0.00_P0.00_Y-0.64, -, -, [prov:role="goal", exsce:runs="12345"])
  wasStartedBy(run_12345:tiago1-task_01-subtask_001-w_002, pose:c022-w001-R0.00_P0.00_Y2.27, -, -)
  wasAssociatedWith(run_12345:tiago1-task_01-subtask_001-w_002, tiago1:agent, -)

  wasInformedBy(run_12345:tiago1-task_01, run_12345:tiago1-task_01-subtask_001-w_001)
  wasInformedBy(run_12345:tiago1-task_01, run_12345:tiago1-task_01-subtask_001-w_002)

  activity(run_12345:tiago1-task_02, -, -, [prov:type="exsce:task_execution", exsce:run="12345"])
  used(run_12345:tiago1-task_02, robot:tiago1, -)
  wasAssociatedWith(run_12345:tiago1-task_02, tiago1:agent, mission_c:tiago1-task_02)

  activity(run_12345:tiago1-task_02-subtask_001-w_001, -, -, [prov:type="exsce:action", exsce:run="12345"])
  wasInformedBy(run_12345:tiago1-task_02-subtask_001-w_001, run_12345:tiago1-task_01-subtask_001-w_002)
  wasEndedBy(run_12345:tiago1-task_02-subtask_001-w_001, pose:c025-w001-R0.00_P0.00_Y0.93, -, -, [prov:role="goal", exsce:runs="12345"])
  wasStartedBy(run_12345:tiago1-task_02-subtask_001-w_001, pose:c018-w001-R0.00_P0.00_Y-0.64, -, -)
  wasAssociatedWith(run_12345:tiago1-task_02-subtask_001-w_001, tiago1:agent, -)
  wasInformedBy(run_12345:tiago1-task_02, run_12345:tiago1-task_02-subtask_001-w_001)

endBundle

Robots

Because we treat the robots as a black box, we have kept their specification very minimal, as shown in Listing 6. Each robot needs a unique ID, its type and, optionally for ROS, the namespace of its topics.

Listing 6: Fleet specification in YAML

robots:
  - id: tiago1
    robot_namespace: tiago1
    robot_type: tiago
  - id: tiago2
    robot_namespace: tiago2
    robot_type: tiago

A robot bundle considers the software agent that represents the robot, its hardware and software configuration, and the simulation model used, if any. Listing 7 shows the PROV-N version for tiago1.

Listing 7: PROV models to represent a robot.

bundle robot:tiago1_bundle
  entity(robot:tiago1, [prov:type="exsce:robot"])

  agent(tiago1:agent, [prov:type="prov:SoftwareAgent"])
  entity(tiago1:software-config, [prov:type="software-config"])
  entity(tiago1:hardware-config, [prov:type="hardware-config"])
  entity(tiago1:gazebo_model, [prov:type='ros:GazeboModel'])

  hadMember(robot:tiago1, tiago1:gazebo_model)
  hadMember(robot:tiago1, tiago1:software-config)
  hadMember(robot:tiago1, tiago1:hardware-config)
endBundle

Environment

The environment specification in Listing 8 consists of three main parts:

The models that describe the environment, in this case the occupancy grid and the gazebo world for simulation,
the models describing objects/obstacles not included in the gazebo world (useful to vary their locations) and their starting positions (in the example below no additional obstacles were added)
the starting position of robots.

Listing 8: Environment specification in YAML

environment:
  id: brsu_building_c_with_doors
  models:
    map:
      map_name: brsu_building_c_with_doors
      pkg: floorplan-DSL-environments
      relative_path: maps/
    gazebo_world:
      model_name: brsu_building_c_with_doors
      pkg: floorplan-DSL-environments
      relative_path: worlds/
  robots:
    tiago1:
      start_pose:
        id: c025-w001
        x: 44.80387496948242
        y: 37.15502166748047
        z: 0.0
        roll: 0.0
        pitch: 0.0
        yaw: 0.0
    tiago2:
      start_pose:
        id: c025-w002
        x: 43.432926177978516
        y: 38.493873596191406
        z: 0.0
        roll: 0.0
        pitch: 0.0
        yaw: 0.0

The environment PROV (cf. Listing 9) is a collection of records for the simulation models and the environment’s occupancy grid. For this case study, we use the environments generated by the Floorplan DSL.

Listing 9: PROV models to represent an environment

bundle env:brsu_building_c_with_doors_bundle
  entity(env:brsu_building_c_with_doors, [prov:type="exsce:environment"])

  entity(brsu_building_c_with_doors:brsu_building_c_with_doors_map, [prov:type="exsce:map", ros:pkg="floorplan-DSL-environments", ros:relative_path="maps/"])

  entity(brsu_building_c_with_doors:occupancy_grid, [prov:type="ros:OccupancyGrid"])
  hadMember(env:brsu_building_c_with_doors, brsu_building_c_with_doors:brsu_building_c_with_doors_map)
  entity(brsu_building_c_with_doors:metadata, [prov:type='ros:Parameter_Server'])
  hadMember(brsu_building_c_with_doors:brsu_building_c_with_doors_map, brsu_building_c_with_doors:metadata)
  hadMember(brsu_building_c_with_doors:brsu_building_c_with_doors_map, brsu_building_c_with_doors:occupancy_grid)

  entity(brsu_building_c_with_doors:gazebo_world, [prov:type='ros:GazeboModel', ros:pkg="floorplan-DSL-environments", ros:relative_path="worlds/"])
  hadMember(env:brsu_building_c_with_doors, brsu_building_c_with_doors:gazebo_world)
endBundle

For simulation in particular, the starting positions of the robots are modeled as derivations as shown in Listing 10.

Listing 10: The PROV models for the starting positions of the robots in a scenario.

bundle exsce:scenario_c_bundle
  wasDerivedFrom(scenario_c:tiago1-start-pose; tiago1:gazebo_model, pose:c025-w001-R0.00_P0.00_Y0.00, -, -, -, [prov:role="start_pose", exsce:scenario="scenario_c"])
  wasDerivedFrom(scenario_c:tiago2-start-pose; tiago2:gazebo_model, pose:c025-w002-R0.00_P0.00_Y0.00, -, -, -, [prov:role="start_pose", exsce:scenario="scenario_c"])
endBundle

Finally, Listing 11 shows how poses in each environment are grouped in bundles that correspond to the room they are in.

Listing 11: PROV models to represent poses of an area in an environment.

bundle env:c022_bundle
  entity(pose:c022-w001-R0.00_P0.00_Y2.27, [prov:type="geom:Pose"])
  entity(pose:c022-w002-R0.00_P0.00_Y2.27, [prov:type="geom:Pose"])

  entity(point:c022-w001, [coord:x="52.8256" %% xsd:float, coord:y="32.5629" %% xsd:float, coord:z="0" %% xsd:float, prov:type="geom:Point"])
  entity(point:c022-w002, [coord:x="54.4976" %% xsd:float, coord:y="31.5202" %% xsd:float, coord:z="0" %% xsd:float, prov:type="geom:Point"])

  entity(orientation:R0.00_P0.00_Y2.27, [angle:roll="0" %% xsd:float, angle:pitch="0" %% xsd:float, angle:yaw="2.269" %% xsd:float, prov:type="geom:EulerAngles", geom:axes="sxyz"])

  hadMember(pose:c022-w001-R0.00_P0.00_Y2.27, point:c022-w001)
  hadMember(pose:c022-w002-R0.00_P0.00_Y2.27, point:c022-w002)

  hadMember(pose:c022-w001-R0.00_P0.00_Y2.27, orientation:R0.00_P0.00_Y2.27)
  hadMember(pose:c022-w002-R0.00_P0.00_Y2.27, orientation:R0.00_P0.00_Y2.27)
endBundle

Execution

Figure 2 shows an overview of how scenario execution is integrated into a ROS-based system. For our implementation, the scenario specification is loaded to the ROS parameter server. A helper script that dispatches tasks to the robots also generates the run-time PROV for the task activities and their allocation as well as each action in the robots’ tasks. This is published to a topic so it can be recorded in the rosbag file for the run. The exsce_run_recorder is used to generate the bag file and save the parameters from the server. These two activities are added to the run-time generated PROV, and stored in a JSON file.

The execution or test run is an activity that uses parameters from the ROS parameter server and which generates a bag file as an artefact, as shown in Listing 12. The run activity is informed by the execution of the tasks described in Tasks.

Listing 12: PROV models to represent a test run in ROS and other run-time information

bundle run:run_e30df0b6-09f0-11ee-ba45-13d500f34135_bundle

  activity(run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, 2023-06-13T15:48:13.720444, 2023-06-13T15:56:32.315739, [prov:type="exsce:run", exsce:run="e30df0b6-09f0-11ee-ba45-13d500f34135"])

  entity(run_e30df0b6:parameter-server, [prov:type='ros:Parameter_Server', exsce:file_path="/home/argen/projects/tiago/tiago_ws/src/metamorphic_testing/runs/2023-06-13T15-48-07_scenario_c_run-e30df0b6.rosparams"])

  used(run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, run_e30df0b6:parameter-server, -)
  used(run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, exsce:scenario_c, -)

  wasGeneratedBy(run_e30df0b6:2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag, run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, -)

  wasInformedBy(run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, run_e30df0b6:tiago1-task_01)
  wasInformedBy(run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, run_e30df0b6:tiago2-task_01)
  wasInformedBy(run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, run_e30df0b6:tiago1-task_02)
  wasInformedBy(run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, run_e30df0b6:tiago2-task_02)

endBundle

A bundle is created for each bag file, and the topics it contains are modelled as entities that are part of its collection, as shown in Listing 13:

Listing 13: PROV models to represent a bag file and the topics it contains

bundle exsce:2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag_bundle
  entity(run_e30df0b6:2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag, [prov:type='ros:Bags', exsce:file_path="/home/argen/projects/tiago/tiago_ws/src/metamorphic_testing/runs/2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag"])
  entity(topic:/tiago1/distance_travelled, [prov:type='ros:Topics'])
  hadMember(run_e30df0b6:2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag, topic:/tiago1/distance_travelled)
  entity(topic:/tiago1/nav_vel, [prov:type='ros:Topics'])
  hadMember(run_e30df0b6:2023-06-13T15-48-07_scenario_c_run-e30df0b6.bag, topic:/tiago1/nav_vel)
endBundle

Metrics and Monitors

Although we obtain the observed outputs off-line through bag files, the implementation to obtain the desired metrics follows the same logic and can be used as a run-time monitor that uses the publisher-subscriber pattern in ROS. These monitors can also be composed into the run-time oracle if desired, as shown in Figure 3. The relevant parts of the configuration for the bag transformation is shown in Listing 14 (tiago2 is not included).

Listing 14: Configuration for the transformation of ROS1 bag files to ExSce PROV modelss

id: scenario_c
baseline:
  base_scenario: scenario_c
  mission:
    mission_duration:
      topics:
        - /tiago1/waypoint_dispatcher/feedback
        - /tiago2/waypoint_dispatcher/feedback
    distance_between_robots:
      topics:
        - /tiago1/robot_pose
        - /tiago2/robot_pose
  robots:
    tiago1:
      task_duration:
        topics:
        - /tiago1/waypoint_dispatcher/feedback
      waypoints_visited:
        topics:
        - /tiago1/waypoint_dispatcher/result
      distance_travelled:
        topics:
        - /tiago1/distance_travelled
      distance_to_obstacles:
        topics:
        - /tiago1/scan
      max_velocity:
        topics:
        - /tiago1/nav_vel
    tiago2:
      task_duration:
        topics:
        - /tiago2/waypoint_dispatcher/feedback
      waypoints_visited:
        topics:
        - /tiago2/waypoint_dispatcher/result
      distance_travelled:
        topics:
        - /tiago2/distance_travelled
      distance_to_obstacles:
        topics:
        - /tiago2/scan
      max_velocity:
        topics:
        - /tiago2/nav_vel

Metrics are computed from the topics published by the robots. A collection of topics models the data source that was used to compute the metric, and the relationship wasGeneratedBy links the output metric to the activity of the run that generated this result. Mission metrics take into account the entire run. To enable the output relation composition, when the bag file is being transformed, in addition to the overall mission metrics, these monitors compute the safety and robot performance metrics for each path segment, so that they can be individually queried and aggregated for other possible paths. In Listing 15 you see the PROV models for the distance_travelled metric of tiago1: One entity for the value of the entire run, but also the corresponding value for this metric on each path segment.

Listing 15: PROV models for the distance_travelled metric of tiago1

bundle run_e30df0b6:distance_travelled_bundle

  entity(run_e30df0b6:tiago1-distance_travelled-topics, [prov:type="exsce:topic"])
  hadMember(run_e30df0b6:tiago1-distance_travelled-topics, topic:/tiago1/distance_travelled)

  entity(tiago1:distance_travelled, [prov:type='exsce:output_metric', run:metricType="distance_travelled", exsce:run="e30df0b6-09f0-11ee-ba45-13d500f34135", prov:value="49.5234" %% xsd:float, exsce:robot="tiago1"])
  wasDerivedFrom(tiago1:distance_travelled, run_e30df0b6:tiago1-distance_travelled-topics, -, -, -, [prov:type="prov:PrimarySource"])
  wasGeneratedBy(tiago1:distance_travelled, run:run_e30df0b6-09f0-11ee-ba45-13d500f34135, -)

  entity(tiago1-task_01-subtask_001-w_001:distance_travelled, [prov:type='exsce:output_metric', run:metricType="distance_travelled", exsce:run="e30df0b6-09f0-11ee-ba45-13d500f34135", prov:value="10.2956" %% xsd:float, exsce:robot="tiago1"])
  wasDerivedFrom(tiago1-task_01-subtask_001-w_001:distance_travelled, run_e30df0b6:tiago1-distance_travelled-topics, -, -, -, [prov:type="prov:PrimarySource"])
  wasGeneratedBy(tiago1-task_01-subtask_001-w_001:distance_travelled, run_e30df0b6:tiago1-task_01-subtask_001-w_001, -)

  entity(tiago1-task_01-subtask_001-w_002:distance_travelled, [prov:type='exsce:output_metric', run:metricType="distance_travelled", exsce:run="e30df0b6-09f0-11ee-ba45-13d500f34135", prov:value="19.5078" %% xsd:float, exsce:robot="tiago1"])
  wasDerivedFrom(tiago1-task_01-subtask_001-w_002:distance_travelled, run_e30df0b6:tiago1-distance_travelled-topics, -, -, -, [prov:type="prov:PrimarySource"])
  wasGeneratedBy(tiago1-task_01-subtask_001-w_002:distance_travelled, run_e30df0b6:tiago1-task_01-subtask_001-w_002, -)

  entity(tiago1-task_02-subtask_001-w_001:distance_travelled, [prov:type='exsce:output_metric', run:metricType="distance_travelled", exsce:run="e30df0b6-09f0-11ee-ba45-13d500f34135", prov:value="19.7156" %% xsd:float, exsce:robot="tiago1"])
  wasDerivedFrom(tiago1-task_02-subtask_001-w_001:distance_travelled, run_e30df0b6:tiago1-distance_travelled-topics, -, -, -, [prov:type="prov:PrimarySource"])
  wasGeneratedBy(tiago1-task_02-subtask_001-w_001:distance_travelled, run_e30df0b6:tiago1-task_02-subtask_001-w_001, -)
endBundle

Test Oracles

Oracles in PROV are modelled as activities that validate run results based on data from ROS topics. In Listing 16, the mission oracle and the oracle for one of the robots in one of our sample scenarios.

Listing 16: PROV models for the mission and tiago1 test oracles

activity(scenario_c:mission_duration, -, -, [prov:type="exsce:oracle", oracle:metric="mission_duration", oracle:robot="None"])
used(scenario_c:mission_duration, topic:/tiago1/waypoint_dispatcher/feedback, -, [prov:role="topic"])
used(scenario_c:mission_duration, topic:/tiago2/waypoint_dispatcher/feedback, -, [prov:role="topic"])

activity(scenario_c:distance_between_robots, -, -, [prov:type="exsce:oracle", oracle:metric="distance_between_robots", oracle:robot="None"])
used(scenario_c:distance_between_robots, topic:/tiago1/robot_pose, -, [prov:role="topic"])
used(scenario_c:distance_between_robots, topic:/tiago2/robot_pose, -, [prov:role="topic"])


activity(scenario_c:tiago1-waypoints_visited, -, -, [prov:type="exsce:oracle", oracle:metric="waypoints_visited", oracle:robot="tiago1"])
used(scenario_c:tiago1-waypoints_visited, topic:/tiago1/waypoint_dispatcher/result, -, [prov:role="topic"])

activity(scenario_c:tiago1-distance_travelled, -, -, [prov:type="exsce:oracle", oracle:metric="distance_travelled", oracle:robot="tiago1"])
used(scenario_c:tiago1-distance_travelled, topic:/tiago1/distance_travelled, -, [prov:role="topic"])

Relationships

The oracles above validate that new runs conform to the relationships between two tests. As such, they are modelled by usage relationships, where the activity uses the results of the base scenario for comparison. As arguments of the relationship we can add the type of output relation, its value, limit or delta, and its tolerance. In Listing 17 you see an example of a baseline oracle which compares new runs of scenario_c against its past runs.

Listing 17: Example of a baseline oracle for scenario_c.

bundle exsce:scenario_c_bundle

  entity(exsce:scenario_c, [prov:type="exsce:ConcreteScenario"])

  used(scenario_c:mission_duration, exsce:scenario_c, -, [oracle:relationship="invariant", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:value="97.606" %% xsd:float, oracle:tolerance="10" %% xsd:float])

  used(scenario_c:distance_between_robots, exsce:scenario_c, -, [oracle:relationship="above_min", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:limit="0.1" %% xsd:float])

  used(scenario_c:tiago1-task_duration, exsce:scenario_c, -, [oracle:relationship="invariant", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:value="97.606" %% xsd:float, oracle:tolerance="5" %% xsd:float])

  used(scenario_c:tiago1-waypoints_visited, exsce:scenario_c, -, [oracle:relationship="invariant", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:value=3])

  used(scenario_c:tiago1-distance_travelled, exsce:scenario_c, -, [oracle:relationship="invariant", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:value="50.7029" %% xsd:float, oracle:tolerance="2" %% xsd:float])

  used(scenario_c:tiago1-distance_to_obstacles, exsce:scenario_c, -, [oracle:relationship="above_min", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:limit="0.1" %% xsd:float])

  used(scenario_c:tiago1-max_velocity, exsce:scenario_c, -, [oracle:relationship="below_max", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:limit="1" %% xsd:float])

  used(scenario_c:tiago2-task_duration, exsce:scenario_c, -, [oracle:relationship="invariant", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:value="89.804" %% xsd:float, oracle:tolerance="5" %% xsd:float])

  used(scenario_c:tiago2-waypoints_visited, exsce:scenario_c, -, [oracle:relationship="invariant", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:value=3])

  used(scenario_c:tiago2-distance_travelled, exsce:scenario_c, -, [oracle:relationship="invariant", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:value="59.5331" %% xsd:float, oracle:tolerance="2" %% xsd:float])

  used(scenario_c:tiago2-distance_to_obstacles, exsce:scenario_c, -, [oracle:relationship="above_min", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:limit="0.1" %% xsd:float])

  used(scenario_c:tiago2-max_velocity, exsce:scenario_c, -, [oracle:relationship="below_max", oracle:package="exsce.metamorphic.relationships", oracle:type="baseline", oracle:limit="1" %% xsd:float])
endBundle

Basic relationships

We consider the following basic output relations:

increasing(result, value, delta, tolerance=None)
decreasing(result, value, delta, tolerance=None)
invariant(result, value, tolerance=None)

These are useful for metrics which are straight forward to observe or those that are invariant around a constant. Two special types of invariant relationships are:

below_max(result, limit)
above_min(result, limit)

Complex relationships

Complex relationships use or combine one or more of

the basic output relationships,
relationships composing input transformations, and
relationships that derive or aggregate metrics based on inputs according to domain or application-specific logic

Oracles

Baseline oracles

Safety-related oracles are usually invariant around a constant, i.e. regardless of the input transformation, the robot should behave the same:

Metric	Output relation	Limit
Max. velocity	below_max	1.0
Dist. to obstacles	above_min	0.1
Dist. between robots	above_min	0.1

Output relations for safety metrics

Performance-related oracles where we compare runs of the same scenario are invariant around the average of the metric. We use the standard deviation for the tolerance.

Metamorphic oracles

The output relations for the performance metrics require complex relationships that depend on specific scenarios being used. Consider the following example:

Metric	Input transformation	Output relation
Dist. travelled	Add new waypoint	Increasing
Dist. travelled	Remove one waypoint	Decreasing
Dist. travelled	Invert waypoint order	Invariant

Example of a complex metamorphic relation

The value used as a baseline depends on the specific base scenario, and the delta depends on the specific path of the new task. Furthermore, output relationships are not straight forward to define when multiple input transformations are applied; however, using the provenance from existing runs we can also compose outputs to determine what is the output relation, e.g., query the results of previous runs to obtain the distance travelled between each pair of waypoints of the new path. Querying will be described in more detail in Queriable Scenario Execution.