<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Load Dataset
When working with Parallel Domain's synthetic data, the standard output format is Dataset Governance Policy (DGP).
In general, the PD SDK can load from any format, as long as a custom decoder exists adhering to the DatasetDecoderProtocol.
Out of the box, PD SDK comes with a pre-configured DGPDatasetDecoder which we can leverage to load data.
In this tutorial, we are going to load and access a dataset and its scenes.
Initially, we need to select the fitting decoder (in this case
Step1:  Alternatively you can also use the decode_dataset helper method.
Step2:  If you want to load a dataset which is stored in Cityscapes or NuImages format simply change the dataset_format to "cityscapes" or "nuimages"
Step3:  Dataset Information
Now that the dataset information has been loaded, we query a couple of metadata from it
Step4:  As you can see, the property .available_annotation_types includes classes from paralleldomain.model.annotation. In tutorials around reading annotations from a dataset, these exact classes will be re-used, which allows for a consistent type-check across objects.
Access available Scenes
Every dataset consists of scenes. These can contain ordered (usually by time) or unordered data.
In this example, we are looking to receive a list of scene names by type that have been found within the loaded dataset.
Step5:  Load Scene
After having retrieved all scene names from a dataset, we get the actual Scene object and access a couple of properties as well as child objects.
Let's start with scene properties
Step6:  Scene metadata usually contains any variables that changes with each scene and are not necessarily consistent across a whole dataset.
In many cases these are environment variables like weather, time of day and location.
A Scene object also includes information about the available annotation types. In most datasets, these will be consistent with the ones at the Dataset level, but there is the possibility to vary them.
Step7:  Normally, in a scene, we expect to have more than one frame available, especially when we work with sequential data.
These can be accessed through their frame IDs. In DGP datasets, these are usually string representations of increasing integers, but they could also be more explicit identifiers for other datasets, e.g., a string representation of a UNIX time or details of the recording vehicle.
In our example, the frame IDs follow the pattern of integers in string representation
Step8:  Load Frame + Sensor
Frames
A Frame object is like a timestamp-bracket around different sensor data. If we have multiple sensors mounted on our recording vehicle, then the single data recordings are usually grouped into specific timestamps.
We can retrieve a Frame object and actually see what the "grouping datetime" is
Step9:  Date/Times are presented as Python's std library datetime objects. When decoding data, the PD SDK also adds timezone information to these objects.
Sensors
As a next step, we want to see what sensor are available within that scene. In general, sensors are divided into CameraSensor and LidarSensor.
Step10:  Similar to how we used this information to get a scene from a dataset, we can use this information to get a sensor from a scene.
Step11:  Knowing which frames and sensors are available allows us to now query for the actual sensor data.
As described above, a Frame is the time-grouping bracket around different sensor recordings. The actual data for a specific sensor assigned to this frame is represented in a SensorFrame.
This is where sensor data and annotations live.
We can either first select a Frame and then pick a Sensor or the other way around. They will return the same SensorFrame instance.
Step12:  Load Sensor Frames
Now that we know how to retrieve SensorFrame object for specific sensors and timestamps, we can use those to extract the actual sensor data.
Accessing shared properties
While there are CameraSensorFrame and LidarSensorFrame objects with sensor specific data, there are certain properties which are common to any SensorFrame.
We are going to print the most basic attributes on a SensorFrame, using the example of a CameraSensorFrame object.
Step13:  Every SensorFrame always has information about the sensor pose (where is it in the world coordinate system?) and sensor extrinsic (how is the sensor positioned relative to the ego-vehicle reference coordinate system?).
Poses and Extrinsics are represented as instances of the Transformation object. It allows storing 6-DoF information and allows for easy combination with each other.
In the example below, we are going to calculate the difference between the camera and the lidar sensor. The difference should be the same when using pose or extrinsic.
Step14:  We can use the associated homogenous transformation matrix to compare both results.
Step15:  In the same manner, it is easily possible to calculate the relative location between two sensors. Let's calculate the difference between two camera sensors.
Step16:  It is important to remember that a sensor extrinsic is provided in the ego-vehicles reference coordinate system. For DGP dataset, that is FLU (Front (x), Left (y), Up (z)).
So the translation difference between both sensors in ego-vehicle coordinate system is approx x=-3. When calculating the difference between both extrinsics, we will receive though a value of approx. z=-3. That is because we receive the difference in the camera coordinate system (RDF). In this example, we have two cameras (one front, one rear facing) that are perfectly aligned with the ego-vehicle's longitudinal axis x.
If we want to have the camera sensor in a FLU coordinate system, we can simply leverage the CoordinateSystem class to take of it for us. Objects of that class can also be combined with Transformation objects.
Step17:  Accessing Annotations
While both CameraSensorFrame and LidarSensorFrame have the property .available_annotation_types, the content will most likely be different.
There are shared annotation types which are available for both sensor types, but for example 2D Bounding Boxes are something just available for camera data, or point cloud segmentation only for LiDAR data.
Step18:  To actually the annotations into memory and use them for further analysis, we can leverage the AnnotationTypes class.
In the example below, we are going to load the 2D Bounding Boxes from a camera frame.
Step19:  For the LiDAR sensor, we are going to retrieve the 3D Semantic Segmentation of the point cloud and count objects by class ID. Instead of checking explicitly if the annotation type is available, we are going to use try/catch on a ValueError. To see if it works, we will try to receive 2D Bounding Boxes from the LiDAR sensor.
Step20:  Instead of showing just class IDs, we can show the actual class labels quite easily. On the Scene object we can retrieve the ClassMap for each annotation style.
Let's get the one for 3D Semantic Segmentation and print the labels for better readability.
Step21:  Access Camera Data
As mentioned above, sensor-specific sensor frames, like CameraSensorFrame have additional properties to the shared ones described above.
For a camera that is especially the RGB mask, as well as, camera intrinsics and distortion parameters.
Note
Step22:  Access LiDAR Data
Similar to a camera, LiDAR sensors have their dedicated sensor frame object LidarSensorFrame.
There we can access different point cloud properties like points in Cartesian coordinates, their intensity or timing offsets.
The simple example below creates an orthographic topdown projection of the point cloud in ego-vehicle coordinate system by leveraging the extrinsic information.
The colorization will be done by height, and the size of points will be defined by reflection intensity.
<ASSISTANT_TASK:>
Python Code:
from paralleldomain.decoding.dgp.decoder import DGPDatasetDecoder
from paralleldomain.model.dataset import Dataset  # optional import, just for type reference in this tutorial
dataset_path = "s3://pd-sdk-c6b4d2ea-0301-46c9-8b63-ef20c0d014e9/testset_dgp"
dgp_decoder = DGPDatasetDecoder(dataset_path=dataset_path)
dataset: Dataset = dgp_decoder.get_dataset()
Explanation: Load Dataset
When working with Parallel Domain's synthetic data, the standard output format is Dataset Governance Policy (DGP).
In general, the PD SDK can load from any format, as long as a custom decoder exists adhering to the DatasetDecoderProtocol.
Out of the box, PD SDK comes with a pre-configured DGPDatasetDecoder which we can leverage to load data.
In this tutorial, we are going to load and access a dataset and its scenes.
Initially, we need to select the fitting decoder (in this case: DGPDatasetDecoder) and then tell it where our dataset is stored. The location can be either a local filesystem path or an s3 address.
End of explanation
from paralleldomain.decoding.helper import decode_dataset
from paralleldomain.decoding.common import DecoderSettings
# To deactivate caching of certain data types use the DecoderSettings
settings = DecoderSettings(cache_images=False)
# decode dgp dataset
dgp_dataset: Dataset = decode_dataset(dataset_path=dataset_path, dataset_format="dgp", settings=settings)
Explanation: Alternatively you can also use the decode_dataset helper method.
End of explanation
nu_images_dataset_path = "some/path/to/a/nuimages/root/folder"
nu_images_dataset: Dataset = decode_dataset(dataset_path=nu_images_dataset_path, dataset_format="nuimages")
cityscapes_dataset_path = "some/path/to/a/cityscapes/root/folder"
cityscapes_dataset: Dataset = decode_dataset(dataset_path=nu_images_dataset_path, dataset_format="cityscapes")
Explanation: If you want to load a dataset which is stored in Cityscapes or NuImages format simply change the dataset_format to "cityscapes" or "nuimages":
End of explanation
print("Dataset Metadata:")
print("Name:", dataset.metadata.name)
print("Available Annotation Types:", *[f"\t{a}" for a in dataset.available_annotation_types], sep="\n")
print("Custom Attributes:", *[f"\t{k}: {v}" for k,v in dataset.metadata.custom_attributes.items()], sep="\n")
Explanation: Dataset Information
Now that the dataset information has been loaded, we query a couple of metadata from it:
End of explanation
for sn in dataset.scene_names:
    print(f"Found scene {sn}")
for usn in dataset.unordered_scene_names:
    print(f"Found unordered scene {usn}")
Explanation: As you can see, the property .available_annotation_types includes classes from paralleldomain.model.annotation. In tutorials around reading annotations from a dataset, these exact classes will be re-used, which allows for a consistent type-check across objects.
Access available Scenes
Every dataset consists of scenes. These can contain ordered (usually by time) or unordered data.
In this example, we are looking to receive a list of scene names by type that have been found within the loaded dataset.
End of explanation
from paralleldomain.model.scene import Scene  # optional import, just for type reference in this tutorial
from pprint import PrettyPrinter
selected_scene = dataset.scene_names[0]  # for future
scene: Scene = dataset.get_scene(scene_name=selected_scene)
# Use prettyprint for nested dictionaries
pp = PrettyPrinter(indent=2)
pp.pprint(scene.metadata)
Explanation: Load Scene
After having retrieved all scene names from a dataset, we get the actual Scene object and access a couple of properties as well as child objects.
Let's start with scene properties:
End of explanation
pp.pprint(scene.available_annotation_types)
Explanation: Scene metadata usually contains any variables that changes with each scene and are not necessarily consistent across a whole dataset.
In many cases these are environment variables like weather, time of day and location.
A Scene object also includes information about the available annotation types. In most datasets, these will be consistent with the ones at the Dataset level, but there is the possibility to vary them.
End of explanation
print(f"{scene.name} has {len(scene.frame_ids)} frames available.")
print(scene.frame_ids)
Explanation: Normally, in a scene, we expect to have more than one frame available, especially when we work with sequential data.
These can be accessed through their frame IDs. In DGP datasets, these are usually string representations of increasing integers, but they could also be more explicit identifiers for other datasets, e.g., a string representation of a UNIX time or details of the recording vehicle.
In our example, the frame IDs follow the pattern of integers in string representation:
End of explanation
frame_0_id = "0"
frame_0 = scene.get_frame(frame_id=frame_0_id)
print(frame_0.date_time)
Explanation: Load Frame + Sensor
Frames
A Frame object is like a timestamp-bracket around different sensor data. If we have multiple sensors mounted on our recording vehicle, then the single data recordings are usually grouped into specific timestamps.
We can retrieve a Frame object and actually see what the "grouping datetime" is:
End of explanation
print("Cameras:", *scene.camera_names, sep='\n')
print("\n")
print("LiDARs:", *scene.lidar_names, sep='\n')
Explanation: Date/Times are presented as Python's std library datetime objects. When decoding data, the PD SDK also adds timezone information to these objects.
Sensors
As a next step, we want to see what sensor are available within that scene. In general, sensors are divided into CameraSensor and LidarSensor.
End of explanation
camera_0_name = scene.camera_names[0]
camera_0 = scene.get_camera_sensor(camera_name=camera_0_name)
Explanation: Similar to how we used this information to get a scene from a dataset, we can use this information to get a sensor from a scene.
End of explanation
camera_frame_via_frame = frame_0.get_camera(camera_name=camera_0_name)
camera_frame_via_camera = camera_0.get_frame(frame_id=frame_0_id)
assert(camera_frame_via_camera is camera_frame_via_camera)
print(f"Both objects are equal: {id(camera_frame_via_frame)} == {id(camera_frame_via_camera)}")
Explanation: Knowing which frames and sensors are available allows us to now query for the actual sensor data.
As described above, a Frame is the time-grouping bracket around different sensor recordings. The actual data for a specific sensor assigned to this frame is represented in a SensorFrame.
This is where sensor data and annotations live.
We can either first select a Frame and then pick a Sensor or the other way around. They will return the same SensorFrame instance.
End of explanation
# Get `CameraSensorFrame` for the first camera on the first frame within the scene.
lidar_0_name = scene.lidar_names[0]
camera_0_frame_0 = frame_0.get_camera(camera_name=camera_0_name)
lidar_0_frame_0 = frame_0.get_lidar(lidar_name=lidar_0_name)
print(f"{camera_0_frame_0.sensor_name} recorded at {camera_0_frame_0.date_time}")
print(f"{lidar_0_frame_0.sensor_name} recorded at {lidar_0_frame_0.date_time}")
Explanation: Load Sensor Frames
Now that we know how to retrieve SensorFrame object for specific sensors and timestamps, we can use those to extract the actual sensor data.
Accessing shared properties
While there are CameraSensorFrame and LidarSensorFrame objects with sensor specific data, there are certain properties which are common to any SensorFrame.
We are going to print the most basic attributes on a SensorFrame, using the example of a CameraSensorFrame object.
End of explanation
print(camera_0_frame_0.pose, " -> ", lidar_0_frame_0.pose)
camera_to_lidar_pose = camera_0_frame_0.pose.inverse @ lidar_0_frame_0.pose
print(camera_0_frame_0.extrinsic, " -> ", lidar_0_frame_0.extrinsic)
camera_to_lidar_extrinsic = camera_0_frame_0.extrinsic.inverse @ lidar_0_frame_0.extrinsic
Explanation: Every SensorFrame always has information about the sensor pose (where is it in the world coordinate system?) and sensor extrinsic (how is the sensor positioned relative to the ego-vehicle reference coordinate system?).
Poses and Extrinsics are represented as instances of the Transformation object. It allows storing 6-DoF information and allows for easy combination with each other.
In the example below, we are going to calculate the difference between the camera and the lidar sensor. The difference should be the same when using pose or extrinsic.
End of explanation
import numpy as np
print("Diff Pose:", camera_to_lidar_pose)
print("Diff Extrinsic:", camera_to_lidar_extrinsic)
assert np.all(np.isclose(camera_to_lidar_pose.transformation_matrix, camera_to_lidar_extrinsic.transformation_matrix, atol=1e-05))
print("If you see this, the difference are close to equal.")
Explanation: We can use the associated homogenous transformation matrix to compare both results.
End of explanation
camera_1_name = scene.camera_names[1]
camera_1_frame_0 = frame_0.get_camera(camera_name=camera_1_name)
print(camera_0_frame_0.extrinsic, " -> ", camera_1_frame_0.extrinsic)
print("Diff Extrinsic: ", camera_0_frame_0.extrinsic.inverse @ camera_1_frame_0.extrinsic)
Explanation: In the same manner, it is easily possible to calculate the relative location between two sensors. Let's calculate the difference between two camera sensors.
End of explanation
from paralleldomain.utilities.coordinate_system import CoordinateSystem
extrinsic_diff = (camera_0_frame_0.extrinsic.inverse @ camera_1_frame_0.extrinsic)
RDF_to_FLU = (CoordinateSystem("RDF") > CoordinateSystem("FLU"))
print( RDF_to_FLU @ extrinsic_diff)
Explanation: It is important to remember that a sensor extrinsic is provided in the ego-vehicles reference coordinate system. For DGP dataset, that is FLU (Front (x), Left (y), Up (z)).
So the translation difference between both sensors in ego-vehicle coordinate system is approx x=-3. When calculating the difference between both extrinsics, we will receive though a value of approx. z=-3. That is because we receive the difference in the camera coordinate system (RDF). In this example, we have two cameras (one front, one rear facing) that are perfectly aligned with the ego-vehicle's longitudinal axis x.
If we want to have the camera sensor in a FLU coordinate system, we can simply leverage the CoordinateSystem class to take of it for us. Objects of that class can also be combined with Transformation objects.
End of explanation
pp.pprint(camera_0_frame_0.available_annotation_types)
pp.pprint(lidar_0_frame_0.available_annotation_types)
Explanation: Accessing Annotations
While both CameraSensorFrame and LidarSensorFrame have the property .available_annotation_types, the content will most likely be different.
There are shared annotation types which are available for both sensor types, but for example 2D Bounding Boxes are something just available for camera data, or point cloud segmentation only for LiDAR data.
End of explanation
from paralleldomain.model.annotation import AnnotationTypes
from paralleldomain.model.annotation import BoundingBoxes2D  # optional import, just for type reference in this tutorial
# Quick check if `BoundingBoxes2D` is an available annotation type. If not, and we do not check for it, we will receive a `ValueError` exception.
if AnnotationTypes.BoundingBoxes2D in camera_0_frame_0.available_annotation_types:
    boxes2d: BoundingBoxes2D = camera_0_frame_0.get_annotations(annotation_type=AnnotationTypes.BoundingBoxes2D)
    for b in boxes2d.boxes[:10]:
        print(b)
Explanation: To actually the annotations into memory and use them for further analysis, we can leverage the AnnotationTypes class.
In the example below, we are going to load the 2D Bounding Boxes from a camera frame.
End of explanation
from paralleldomain.model.annotation import SemanticSegmentation3D  # optional import, just for type reference in this tutorial
annotation_type = AnnotationTypes.BoundingBoxes2D
try:
    boxes2d: BoundingBoxes2D = lidar_0_frame_0.get_annotations(annotation_type=annotation_type)
except ValueError as e:
    print(f"LiDAR Frame doesn't have {annotation_type} as annotation type available. Original exception below:")
    print(str(e))
# Move on to the actual task:
annotation_type = AnnotationTypes.SemanticSegmentation3D
count_by_class_id = {}
try:
    semseg3d: SemanticSegmentation3D = lidar_0_frame_0.get_annotations(annotation_type=annotation_type)
    u_class_ids, u_counts = np.unique(semseg3d.class_ids, return_counts=True)
    count_by_class_id = {u_class_ids[idx]: u_counts[idx] for idx in range(len(u_class_ids))}
    pp.pprint(count_by_class_id)
except ValueError as e:
    print(f"LiDAR Frame doesn't have {annotation_type} as annotation type available.")
    print(str(e))
Explanation: For the LiDAR sensor, we are going to retrieve the 3D Semantic Segmentation of the point cloud and count objects by class ID. Instead of checking explicitly if the annotation type is available, we are going to use try/catch on a ValueError. To see if it works, we will try to receive 2D Bounding Boxes from the LiDAR sensor.
End of explanation
from paralleldomain.model.class_mapping import ClassMap  # optional import, just for type reference in this tutorial
semseg3d_classmap: ClassMap = scene.get_class_map(annotation_type=AnnotationTypes.SemanticSegmentation3D)
count_by_class_label = {k: f"{semseg3d_classmap[k].name} [{v}]" for k,v in count_by_class_id.items()}
pp.pprint(count_by_class_label)
Explanation: Instead of showing just class IDs, we can show the actual class labels quite easily. On the Scene object we can retrieve the ClassMap for each annotation style.
Let's get the one for 3D Semantic Segmentation and print the labels for better readability.
End of explanation
from matplotlib import pyplot as plt
from paralleldomain.model.sensor import Image  # optional import, just for type reference in this tutorial
image_data: Image = camera_0_frame_0.image
print(f"Below is an image with {image_data.channels} channels and resolution {image_data.width}x{image_data.height} sqpx")
plt.imshow(image_data.rgba)  # `.rgba` returns image including alpha-channel, otherwise `.rgb` can be used for convenience.
plt.title(camera_0_frame_0.sensor_name)
plt.show()
pp.pprint(vars(camera_0_frame_0.intrinsic))
Explanation: Access Camera Data
As mentioned above, sensor-specific sensor frames, like CameraSensorFrame have additional properties to the shared ones described above.
For a camera that is especially the RGB mask, as well as, camera intrinsics and distortion parameters.
Note: Whenever we work with image data (including masks that represent an image-encoded representation), we work with np.ndarray of shape (h, w, 3) or (h, w, 4).
The last axis is defined in the following index order: 0: Red, 1: Green, 2: Blue, [3: Alpha]. When using OpenCV directly, we need to make explicitly convert the image into BGR[A] order. If you use methods from within the PD SDK, e.g., from utilities, any required conversion is handled for you.
End of explanation
from paralleldomain.model.sensor import PointCloud  # optional import, just for type reference in this tutorial
pc_data: PointCloud = lidar_0_frame_0.point_cloud
pc_xyz_one: np.ndarray = pc_data.xyz_one # Returns the xyz coordinates with an additional column full of "1" to allow for direct transformation
pc_intensity: np.ndarray = pc_data.intensity
pc_ego = (lidar_0_frame_0.extrinsic @ pc_xyz_one.T).T
pc_ego = pc_ego[:,:3]  # throw away "1" - we are done transforming
subset_slice = slice(None, None, 5)  # we want a slice of every 5th point to reduce rendering time
pc_ego_subset = pc_ego[subset_slice]
pc_intensity_subset = pc_intensity[subset_slice]
plt.scatter(x=pc_ego_subset[:,0], y=pc_ego_subset[:,1], s=pc_intensity_subset, c=pc_ego_subset[:,2])
plt.grid(True)
plt.title(lidar_0_frame_0.sensor_name)
Explanation: Access LiDAR Data
Similar to a camera, LiDAR sensors have their dedicated sensor frame object LidarSensorFrame.
There we can access different point cloud properties like points in Cartesian coordinates, their intensity or timing offsets.
The simple example below creates an orthographic topdown projection of the point cloud in ego-vehicle coordinate system by leveraging the extrinsic information.
The colorization will be done by height, and the size of points will be defined by reflection intensity.
End of explanation
<END_TASK>