Improve 3D object detection evaluation

.env.template

@@ -0,0 +1,6 @@
+# SYSTEM UNDER TEST
+COMPOSE_PROFILES="focalformer3d"  # focalformer3d, centerpoint
+
+# DATASET
+DATASET="nuscenes"  # fzi_aura, nuscenes, nvidia_physicalai_av_dataset, waymo_open_dataset
+HF_TOKEN=""  # set to your PRIVATE Hugging Face token to enable automatic dataset download

.gitignore

@@ -1,3 +1,4 @@
 __pycache__/
 *.pyc
 results.json
+.env

.repos

@@ -1,6 +1,6 @@
 repositories:
-  # only needed for the autonomy_datasets_msgs package, which carries the dataset
-  # meta information that perception_msgs/Object cannot express
+  # only needed for the autonomy_datasets_msgs package, which carries the service to request samples
+  # and the dataset meta information that perception_msgs/Object cannot express
   autonomy_datasets:
     type: git
     url: https://github.com/thinking-cars/autonomy_datasets.git

README.md

@@ -18,7 +18,7 @@ Within the Autonomy.Benchmarks suite, **Autonomy.Evaluation** generates the metr
 
 - 🔄 **Unified ROS 2 Interface**: Evaluate any ROS system under test, on datasets, in simulation or live, using the benefits of the ROS 2 ecosystem
 - 🧩 **Pluggable Evaluations**: Select an evaluation by name, or bring your own from another package, reading any topics as inputs and, where needed, ground truth
-- 📊 **Established Metrics**: Use the provided evaluations, which follow the protocols of established challenges, with [Autonomy.Datasets](https://github.com/thinking-cars/autonomy_datasets) across different automated driving tasks
+- 📊 **Established Metrics**: Use the provided evaluations, which follow the metrics of established benchmarks, with [Autonomy.Datasets](https://github.com/thinking-cars/autonomy_datasets) across different datasets and automated driving tasks
 - ⚡ **Efficient Data Pipeline**: Works seamlessly with preprocessed Rosbag files from [Autonomy.Datasets](https://github.com/thinking-cars/autonomy_datasets) for fast execution during development
 - 🐳 **Dockerized Environment**: Reproducible setup with all dependencies included
 - 🔌 **Modular Architecture**: Easy integration with other ROS 2 packages
@@ -31,9 +31,9 @@ Detailed metric definitions and computation notes are documented in [docs/IMPLEM
 
 > [**Contributions**](docs/IMPLEMENTATION.md#adding-more-evaluations) adding more evaluations are welcome
 
-| Evaluation | Challenge | Dataset | Task |
-| --------- | --------- | ------- | ---- |
-| [**nuScenes 3D Lidar Object Detection**](docs/IMPLEMENTATION.md#3d-lidar-object-detection) | [![3D Object Detection Challenge](https://img.shields.io/badge/origin-3D_Object_Detection_Challenge-green)](https://www.nuscenes.org/object-detection) | [nuScenes](https://github.com/thinking-cars/autonomy_datasets) | 3D bounding box detection from lidar |
+| Evaluation | Datasets | Task | Preview |
+| ---------- | -------- | ---- | ------- |
+| [**3D Object Detection**](docs/IMPLEMENTATION.md#3d-object-detection) | All [Autonomy.Datasets](https://github.com/thinking-cars/autonomy_datasets) with 3D object labels | 3D bounding box detection on the classes of `perception_msgs/ObjectClassification` | ![Rviz Screenshot 3D Object Detection evaluation](./docs/assets/3d-object-detection.png)
 
 <p align="center">
   <strong>🚀 <a href="#-quick-start">Quick Start</a></strong> • <strong>💻 <a href="#-development">Development</a></strong> • <strong>📝 <a href="#-documentation">Documentation</a></strong>
@@ -51,8 +51,8 @@ Use the provided [docker-compose.yml](docker-compose.yml) to start the full pipe
 xhost +local:
 
 # pull and start Docker containers
-export COMPOSE_PROFILES="focalformer3d"  # or 'centerpoint'
-docker compose pull
+cp .env.template .env
+# configure evaluated module and dataset in the '.env' file
 docker compose up -d
 # stop containers once finished
 docker compose down
@@ -61,7 +61,7 @@ docker compose down
 Configure the evaluation and dataset via ROS launch arguments in [docker-compose.yml](docker-compose.yml):
 
 ```yaml
-command: ros2 launch autonomy_evaluation autonomy_evaluation.launch.py evaluation:=nuscenes_lidar_object_detection prediction:=$your_prediction_topic label:=$your_label_topic request_samples:=/datasets/request_samples visualize:=true
+command: ros2 launch autonomy_evaluation autonomy_evaluation.launch.py evaluation:=object_detection_3d prediction:=$your_prediction_topic label:=$your_label_topic request_samples:=/datasets/request_samples visualize:=true
 ```
 
 The evaluation node requests the samples it evaluates from the dataset node via its `request_samples` service, which publishes them and responds once they have been published. The dataset therefore publishes the next sample only once the system under test has processed the current one. As soon as all samples have been published, the node aggregates its metrics per scene of the dataset and over all evaluated samples. To evaluate samples published by others instead, e.g. by a closed-loop simulation, set `sample_source:=external`. See the [node documentation](autonomy_evaluation/README.md#autonomy_evaluation) for the topics of the evaluations, the sample settings and the results.

autonomy_evaluation/README.md

@@ -11,27 +11,27 @@ an automated driving deployment is benchmarked on.
 
 ### `autonomy_evaluation`
 
-The node runs the evaluation selected by `evaluation`, either one of this package by its name or one implemented in another package as `<module>:<class>` (see [Adding more Evaluations](../docs/IMPLEMENTATION.md#adding-more-evaluations)). Each evaluation declares the topics it reads: its _inputs_ from the system under test, and, if it compares them with a reference, its _ground truth_. The node subscribes to all of them, each on its node-relative name, which the launch file remaps onto the topic given by the launch argument of the same name, e.g. `prediction:=/object_list/prediction`. A topic without such an argument is subscribed in the private namespace of the node. The node logs which topic it evaluates as input and which as ground truth.
+The node runs the evaluation selected by `evaluation`, either one of this package by its name or one implemented in another package as `<module>:<class>` (see [Adding more Evaluations](../docs/IMPLEMENTATION.md#adding-more-evaluations)). Each evaluation declares the topics it reads: its _inputs_ from the system under test, and, if it compares them with a reference, its _ground truth_. The node subscribes to all of them, each on its node-relative name, which the launch file remaps onto the topic given by the launch argument of the same name, e.g. `prediction:=/object_list/prediction`. A topic without such an argument is subscribed in the private namespace of the node. The node logs which topic it evaluates as input and which as ground truth. A topic an evaluation declares as optional, e.g. meta information that not every dataset publishes, is only waited for while it has a publisher; otherwise, the samples are evaluated without it.
 
 | Evaluation | Inputs | Ground truth |
 | --- | --- | --- |
-| `nuscenes_lidar_object_detection` | `prediction` (`perception_msgs/ObjectList`) | `label` (`perception_msgs/ObjectList`), `label_meta_info` (`autonomy_datasets_msgs/ObjectListMetaInfo`, subscribed next to `label` on `<label topic>/meta_info`) |
+| `object_detection_3d` | `prediction` (`perception_msgs/ObjectList`) | `label` (`perception_msgs/ObjectList`); optionally `label_meta_info` (`autonomy_datasets_msgs/ObjectListMetaInfo`, subscribed next to `label` on `<label topic>/meta_info`) |
 
 The messages of all topics that belong to the same sample are matched by their stamp and evaluated together. By default, only messages with exactly the same stamp are matched, as the dataset stamps all messages of a sample with its recording time and a system under test stamps its output with the stamp of the input it processed. Set `sync_tolerance` to match topics whose stamps differ, e.g. those a simulation publishes at slightly different times or at different rates. A message without a header, e.g. a `std_msgs/Bool`, is stamped with the time it is received at.
 
 With `sample_source` set to `dataset`, the default, the node requests the samples it evaluates from the dataset, using the `request_samples` service of [autonomy_datasets](https://github.com/thinking-cars/autonomy_datasets), which publishes them and responds once they have been published. By default one sample is requested at a time, so the dataset only publishes the next sample once the system under test has delivered its output for the current one and the node has evaluated it. Increase `samples_per_request` to publish samples in batches, set it to `0` to publish the whole dataset with a single request, or list the IDs of individual samples in `sample_ids` to evaluate only those.
 
-Once the dataset reports that all requested samples have been published, or its node has shut down after publishing its last sample, the per-sample metrics are aggregated over all evaluated samples (`aggregated_metrics`) and for the samples of each scene the dataset published them from (`scene_results`, matched with the samples via the `published_scene_ids` of the responses). The metrics are logged, and written to a JSON file if `results_path` is set. Samples that are not evaluated within `evaluation_timeout` seconds of being published, e.g. because the system under test skipped them, are left out.
+Once the dataset reports that all requested samples have been published, or its node has shut down after publishing its last sample, the per-sample metrics are aggregated over all evaluated samples (`metrics`) and for the samples of each scene the dataset published them from (`scenes`, matched with the samples via the `published_scene_ids` of the responses). The metrics are logged, and written to a JSON file if `results_path` is set. Samples that are not evaluated within `evaluation_timeout` seconds of being published, e.g. because the system under test skipped them, are left out.
 
 ```bash
 ros2 launch autonomy_evaluation autonomy_evaluation.launch.py \
   prediction:=/object_list/prediction \
   label:=/object_list/lidar_01 \
   request_samples:=/datasets/request_samples \
-  results_path:=/results/nuscenes_lidar_object_detection.json
+  results_path:=/results/object_detection_3d.json
 ```
 
-With `sample_source` set to `external`, the node requests no samples itself and evaluates every sample that others publish, e.g. a closed-loop simulation, a live system, or the dataset stepped through with the [playback panel](https://github.com/thinking-cars/autonomy_datasets/blob/main/autonomy_datasets_rviz_plugins/README.md) in RViz, whose _Service_ field has to name the `request_samples` service of the dataset node (`/datasets/request_samples` by default). `samples_per_request`, `sample_ids` and `evaluation_timeout` then have no effect. As the node receives no responses of a dataset, it neither learns the scenes of the samples, so `scene_results` stays empty, nor when publishing has ended: the results of the evaluated samples are reported once the node is stopped, e.g. with Ctrl-C, and are marked as incomplete.
+With `sample_source` set to `external`, the node requests no samples itself and evaluates every sample that others publish, e.g. a closed-loop simulation, a live system, or the dataset stepped through with the [playback panel](https://github.com/thinking-cars/autonomy_datasets/blob/main/autonomy_datasets_rviz_plugins/README.md) in RViz, whose _Service_ field has to name the `request_samples` service of the dataset node (`/datasets/request_samples` by default). `samples_per_request`, `sample_ids` and `evaluation_timeout` then have no effect. As the node receives no responses of a dataset, it neither learns the scenes of the samples, so `scenes` stays empty, nor when publishing has ended: the results of the evaluated samples are reported once the node is stopped, e.g. with Ctrl-C, and are marked as incomplete.
 
 ```bash
 # look at the samples of a dataset one by one, stepping through them in RViz
@@ -68,7 +68,7 @@ flowchart LR
 
 | Parameter | Type | Default | Description |
 | --- | --- | --- | --- |
-| `evaluation` | `string` | `nuscenes_lidar_object_detection` | name of an evaluation of this package, or '<module>:<class>' of an evaluation implemented in another package |
+| `evaluation` | `string` | `object_detection_3d` | name of an evaluation of this package, or '<module>:<class>' of an evaluation implemented in another package |
 | `visualize` | `bool` | `false` | publish the per-sample visualization of the evaluation for RViz, e.g. the true positives, false positives and false negatives of an object detection |
 | `sample_source` | `string` | `dataset` | 'dataset' requests the samples to evaluate from the dataset one after another; 'external' evaluates the samples published by others, e.g. by a simulation, a live system or the playback panel in RViz, and reports the results once the node is stopped |
 | `sync_tolerance` | `float` | `0.0` | seconds by which the stamps of the messages of a sample may differ; 0 only matches messages with exactly the same stamp, as the dataset and a system under test echoing its stamps publish them |
@@ -84,7 +84,7 @@ flowchart LR
 | Argument | Default | Description |
 | --- | --- | --- |
 | `request_samples` | `"~/request_samples"` | service of the dataset node used to request the samples to evaluate |
-| `evaluation` | `"nuscenes_lidar_object_detection"` | evaluation to run |
+| `evaluation` | `"object_detection_3d"` | evaluation to run |
 | `name` | `"autonomy_evaluation"` | node name |
 | `namespace` | `""` | node namespace |
 | `log_level` | `"info"` | ros logging level |