diff --git a/.markdownlintignore b/.markdownlintignore index 7ad244c..e930e50 100644 --- a/.markdownlintignore +++ b/.markdownlintignore @@ -2,4 +2,4 @@ interoperability/algohosting.md interoperability/algohostingenv.md interoperability/geospatial_explorer.qmd interoperability/businessmodel.md -interoperability/datahosting.md +interoperability/datahosting.qmd diff --git a/_quarto.yml b/_quarto.yml index 8a2b737..f967c18 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -88,8 +88,6 @@ website: - href: guides/algorithm_services_catalogue_benchmarks.qmd - href: guides/algorithm_catalogue_dashboard.md text: APEx Algorithm Dashboard - - href: guides/file_formats.qmd - text: File format recommendations - section: Admin Guides contents: - href: guides/account.md @@ -112,7 +110,7 @@ website: - href: interoperability/definitions.qmd - href: propagation/service_development.qmd - href: interoperability/algohosting.md - - href: interoperability/datahosting.md + - href: interoperability/datahosting.qmd - href: interoperability/algohostingenv.md - href: interoperability/geospatial_explorer.qmd - href: interoperability/businessmodel.md diff --git a/guides/file_formats.qmd b/guides/file_formats.qmd deleted file mode 100644 index 810b285..0000000 --- a/guides/file_formats.qmd +++ /dev/null @@ -1,160 +0,0 @@ ---- -title: File format recommendations ---- - -Within the APEx interoperability and compliance guidelines, [DATA-REQ-01](../interoperability/datahosting.md) specifies -the file formats that projects should use when publishing static data. -The information on this page complement that requirement with more detailed recommendations on the use of these formats. -Projects are encouraged to consult with the APEx team if they have a use case that does not fit within these guidelines. - -The key underlying principle is to use cloud native formats, with adoption rate as a secondary criterium to ensure that -produced data can reach a broad audience beyond the earth observation community. - -The sections below provide extra guidance, but for a comprehensive comparison, we refer to the [cloud native geospatial guide](https://guide.cloudnativegeo.org/) -which details various raster formats. - -## Cloud Optimised GeoTIFF - -Cloud Optimised GeoTIFF (COG) is the most widely supported format for geospatial raster data, and also one of the most -efficient in terms of access costs. It is recommended as the default option to consider when publishing static data products. -When combined with STAC metadata, COG can produce self-describing, FAIR-compliant datasets that are easily consumable by -various services. - -### Generating Cloud Optimised GeoTIFF - -If you are not familiar with COG or GeoTIFF generation, it is recommended to format your GeoTIFF files using a recent version -of GDAL to ensure that they are compliant: - -```bash -gdal_translate world.tif world_webmerc_cog.tif -of COG -``` - -### Organising spatiotemporal multi-band datasets - -Many datasets have multiple bands (or 'variables'), or have a date associated with them. -The general recommendation is to store a single band per GeoTIFF file and to create STAC items with one asset per band. -This layout is commonly used by many other datasets and avoids the complexities of multi-band GeoTIFF files, which can be -challenging to use. - -An exception to this are the 'RGB' style products, where three bands are used to represent a single image. In this case, -creating a Cloud Optimised GeoTIFF with three bands is an option. - -For associating time information, create one GeoTIFF per timestamp, and one STAC item per timestamp. The GeoTIFF format -has not built-in support for conveying time information, but STAC metadata is supporting this very well. - -### Visualisation in APEx Geospatial Explorer - -To optimise visualisation in the APEx Geospatial Explorer, it is recommended to use the GoogleMapsCompatible tiling scheme- -typically 256x256 pixel tiles aligned to a global grid. The default Coordinate Reference System (CRS) used in the Geospatial -Explorer is Web Mercator projection (EPSG:3857) and therefore all datasets in this projection will be supported. On the -fly reprojection and / or configuration of a Geospatial Explorer instance to alternative CRS’s is feasible, although we -advise contact the APEx team for specific advice when using alternative projections. The BitsPerSample field must accurately -reflect the data format. Overviews are essential for performance and should be generated using downsampling by factors of -two until the image dimensions are the size of a tile or smaller. These overviews should also be tiled and placed after -the main image data to conform with the COG specification. - -## GeoZarr - -[GeoZarr](https://geozarr.org/) is a format that is gaining traction in the geospatial community, although it is not yet as widely supported as -Cloud Optimised GeoTIFF. -Its main advantage lies in its ability to store complex multi-dimensional datasets that go beyond a simple 4D -(x, y, time, bands) structure. Just like COG, Zarr allows for efficient cloud access. - -At the time of writing, there are, however these important caveats: - -- GeoZarr aims to define how to store geospatial raster data in Zarr format, this specification is now going through the standardization process. -- By design, Zarr allows for many degrees of freedom, which requires the data producer to have a good understanding of - the associated trade-offs. It is not guaranteed that tooling will automatically generate the most appropriate layout and - chunk size for your use case. -- By design, Zarr stores data as separate files in a directory structure, optimising cloud access but making direct - downloads less convenient. - -## NetCDF - -NetCDF is a self-describing format with some properties similar to Zarr, but less optimised for cloud access. It can be -useful for exchanging data cubes as single files through traditional methods. However, it is less recommended for convenient -sharing of large datasets, for which either COG or Zarr provide better options. - -## Statistical Datasets (FlatGeobuf, GeoJSON) - -Statistical datasets can be used to store precomputed statistics for dataset variables based on spatial units, such as -administrative areas. An example is to collect land cover statistics on using boundaries from nomenclature of territorial -units for statistics (NUTS), as shown in the [APEx Geospatial Explorer](https://explorer.apex.esa.int/) (Statistics). The -guidelines in this section are focused on supporting the integration of statistical data for visualisation in the APEx -Geospatial Explorer. - -The statistical datasets are expected to be vector layers that are provided in a format that can be parsed to a feature -collection following the GeoJSON [@geojson] specification. Currently tested and supported formats are GeoJSON [@geojson] -and FlatGeobuf [@flatgeobuf]. FlatGeobuf should be used where the statistical data is a large size as this allows for -streaming of the relevant features without having to download the full dataset, increasing performance. - -The metadata header of the file should contain the following properties to define which fields on the features in the -dataset should be used for the following purposes. - -- identifierKey: The name of the field that stores the unique identifier for each feature. -- nameKey: The name of the field that stores the human-readable name for display. -- levelKey: The name of the field that stores the administrative level number. -- childrenKey: The name of the field that has a comma-separated list of child feature IDs as declared in identifierKey. -Can be the empty string if this is the bottom level. -- attributeKeys: A comma-separated list of field numbers that store the statistical data. -- units: The units as displayed in the UI. This is for UI purposes only and has no effect on the data. -- visualization_hint: A string of histogram, categorised, or continuous used as a hint to the UI to choose a suitable -presentation for the data. - -For example, properties in the file metadata that is defined as follows: - -- identifierKey: NUTS_ID -- nameKey: NUTS_NAME -- levelKey: LEVL_CODE -- childrenKey: children -- attributeKeys: Trees, Shrubland, Grassland -- visualization_hint: categorised - -would use the fields NUTS_ID, NUTS_NAME, … in the data to determine the navigation and display of statistics in the -Geospatial Explorer. For further guidance, please contact the APEx team through the [APEx User Forum](http://forum.apex.esa.int/). - -Datasets that have classifications (such as land use) should have key:value entires consisting of 'name':'value' and an -entry with a key of 'classifications' with a value consisting of a string based comma separated list containing all the -keys for the classifications and a 'total' key with the sum of all other values. This will allow for correctly rendering -bar charts and pie charts. - -```{json} -{ - Bare / sparse vegetation: 3349.349614217657, - Built-up: 18474.280639104116 - Cropland: 155067.6934300016 - Grassland: 140178.79417018566 - Herbaceous wetland: 1612.828666906516 - Mangroves: 479.46053523623897 - Moss and lichen: 499.40601429089236 - Permanent water bodies: 8969.837211370474 - Shrubland: 7342.96093361589 - Snow and ice: 495.7695064816955 - Tree cover: 301783.0035618253 - Unknown: 1.7258467103820294 - total: 638255.1101299465 - classifications: "Tree cover,Shrubland,Grassland,Cropland,Built-up,Bare / sparse vegetation,Snow and ice, - Permanent water bodies,Herbaceous wetland,Mangroves,Moss and lichen,Unknown" -} -``` - -![worldcover_bar_chart_example](./images/worldcover_bar_chart_example.png){width=75%} - -Datasets that do not have classifications (such as a raster showing soil organic carbon) should contain a selection of -the following entries: - -- mean -- min -- max - -These values will be rendered as a table. - -```{json} -{ - mean: 437.94353402030356 - min: 60 - max: 4410 -} -``` - -![worldsoils_table_example](./images/worldsoils_table_example.png){width=75%} diff --git a/interoperability/algohostingenv.md b/interoperability/algohostingenv.md index 8add8e6..0724263 100644 --- a/interoperability/algohostingenv.md +++ b/interoperability/algohostingenv.md @@ -17,6 +17,8 @@ The aim of these requirements is: technologies and architectures that suit their needs. * To allow APEx to perform automated checks on the developed services, guaranteeing that they work and produce the expected result at the expected cost. +* Support the integration of the onboarded services into APEx ecosystem, such as service execution through the + [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md). @tbl-algohostingenv provides an overview of the requirements for operators of algorithm hosting platforms to ensure their compatibility with the APEx standards. @@ -87,6 +89,27 @@ their compatibility with the APEx standards. The algorithm hosting platform shall expose process metadata publicly without requiring authentication, unless the nature of the algorithm requires its description to be hidden. APEx tools request service metadata for informative purposes, also from browser-based applications that do not have platform tokens available. + + HOST-REQ-09 + The algorithm hosting platform shall support the OAuth 2.0 Token Exchange RFC. + +

The Token Exchange RFC is essential for executing APEx-compliant services on the platform using APEx tooling, + such as the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md).

+

The RFC is automatically accessible when the platform adheres to EOEPCA’s best practices and utilises Keycloak + as its primary user management system.

+ + + + HOST-REQ-10 + The algorithm hosting platform shall provide the APEx project with a client ID and secret to support the + execution of published services. + +

This client is utilised to execute APEx-compliant services on the platform using APEx tooling, such as the + [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md).

+

This client is employed to execute the Token Exchange RFC (HOST-REQ-09) to facilitate the exchange of an + APEx-based user token with a user token that is recognized and validated by the platform.

+ + diff --git a/interoperability/datahosting.md b/interoperability/datahosting.md deleted file mode 100644 index cf4cf61..0000000 --- a/interoperability/datahosting.md +++ /dev/null @@ -1,312 +0,0 @@ ---- -title: Data Provider Guidelines ---- - -## Requirements - -@tbl-datahosting outlines the interoperability guidelines for EO projects and data providers who wish to deliver their -datasets to APEx for integration within the [ESA Project Results Repository (PRR)](https://eoresults.esa.int/) for -long-term preservation and their utilisation within the APEx Project Environments. By fulfilling these requirements, APEx -ensures seamless integration, discoverability, and usability of the datasets across the ESA EO ecosystem, facilitating -broader access and reusability within the EO community. - -Most of these requirements focus on standardising dataset metadata, formats, and access methods to ensure compatibility -with existing tools and support their efficient exploitation. In particular, datasets should adhere to well-established -EO data standards and provide consistent, machine-readable metadata descriptions. - -APEx supports integration primarily through recognised standards such as STAC (SpatioTemporal Asset Catalogue) and -cloud-native data formats. This ensures that almost any EO dataset can be made available as a ready-to-use resource in -the ESA PRR and used through the APEx tooling. - -Overall, the objective is to streamline and simplify the delivery of high-quality, interoperable EO datasets to APEx, -fostering wider adoption and enabling advanced use cases in downstream applications. - -:::{#tbl-datahosting} - - - - - - - - - - - - - - - - - - - - - - - - - -
IDRequirementDescription
DATA-REQ-01EO project results with respect to raster data, shall be delivered as cloud-native datasets.Where possible, cloud optimized GeoTIFF [@cog] is preferred. For more complex datasets, CF-Compliant netCDF [@netcdf] is a good alternative. Use of the still evolving GeoZarr [@geozarr] format requires confirmation by APEx and may result in future incompatibility if the selected flavour is not standardised eventually. Additional recommendations for the usage of file formats within the APEx services are available on the APEx documentation.
DATA-REQ-02EO project results with respect to vector data, shall be delivered as cloud-native datasets.Small datasets can use GeoJSON [@geojson], FlatGeobuf [@flatgeobuf] or GeoParquet [@geoparquet] are recommended for larger datasets.
DATA-REQ-03EO project results should be accompanied with metadata in a STAC [@stac] format, including applicable STAC extensions.The specific STAC profiles will align with the recommendations will align with the recommendations provided in the [Metadata Recommendations](#metadata-recommendations) section.
- -Interoperability requirements for data providers -::: - -:::{.callout-tip title="File Format Recommendations"} -For more details regarding the recommended file formats and their usage within APEx, please refer to the -[APEx File Format Recommendations](../guides/file_formats.qmd). -::: - - -## Metadata Recommendations - -### Format Specific Recommendations - -When sharing geospatial datasets in cloud-optimised formats, such as Cloud Optimised GeoTIFF (COG), NetCDF, and Zarr, it -is essential to embed as much relevant metadata as possible directly within the files. Although these formats are designed -for efficient cloud access, their interoperability potential is enhanced when the files carry rich, standardised metadata -aligned with their respective specifications. Doing so not only improves data reuse by third-party tools but also enables -more reliable automatic inference of STAC metadata during cataloguing or dataset publication. - -APEx recommends that the following details be incorporated into the file metadata: - -- The projection system used to present the data within the file -- he Nodata value applied -- The unit of measurement for values represented in the dataset -- A definition of the colour map or legend utilised for the dataset visualisation in case of categorical data. -- Band or variable names and descriptions - -For more details and examples on adding this additional metadata to your results, please consult the specific tools -(e.g. gdal, rasterio, …) for generating the results. - -### STAC Metadata Recommendations - -The STAC specification provides a comprehensive and interoperable framework for describing geospatial datasets. Within -APEx, STAC serves as the foundation to enhance the discoverability, interoperability, and integration of data across a -range of platforms, data catalogues, including the ESA Project Results Repository, and tools such as the APEx Geospatial -Explorer. - -To enhance interoperability, data providers are advised to consistently use a recommended set of STAC-related extensions -and best practices. These recommendations come from community input and collaboration with other initiatives, like -EarthCODE and EOEPCA, to ensure consistency across projects and promote the adoption of best practices. - -@tbl-metadata offers a summary of the suggested metadata. For further details, please refer to the resources listed below. - -- [STAC Best Practices](https://github.com/radiantearth/stac-best-practices/blob/main/README.md) -- [EOEPCA+ Datacube Access Best Practices](https://github.com/EOEPCA/datacube-access/blob/main/best_practices/stac_best_practices.md) -- [ESA PRR Collection Specifications](https://eoresults.esa.int/prr_collection_specifications.html) - -:::{#tbl-metadata} - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
IDScenarioLevelRequirementDescription
General
METADATA-REC-01Collection / ItemThe STAC collection and items should use STAC 1.1 or higher.
METADATA-REC-02CollectionThe STAC collection must follow the [ESA PRR collection specifications](https://eoresults.esa.int/prr_collection_specifications.html).This guarantees the collection is compatible for upload and registration in the ESA Project Results Repository.
METADATA-REC-03Collection / ItemCollections must be homogeneous: each item has the same assets and uses the same asset keys.Consistent and homogeneous definition of assets simplifies client-side handling and supports datacube generation.
METADATA-REC-04ItemEach item must have at least one asset where the role is set to *data*.This allows for accurate identification of assets containing the data.
METADATA-REC-05Item - Each asset must include: -
    -
  • type
    • -
  • role
    • -
  • title
    • -
  • file:size
    • -
- Optionally properties: -
    -
  • file:checksum
    • -
- 		These properties help tools and platforms accurately import the dataset. Furthermore, the *file* properties allow - the ESA PRR to perform extra quality checks.
METADATA-REC-06Data VisualisationItem -

In the case of categorical datasets, - classification extension is recommended to identify - the different classes used in the asset.

-

For additional visualization support, it is recommend setting the *title* and *color_hint* properties to allow - external tools, such as the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md) to properly visualise - the data.

- 		The classification extension supports the proper interpretation of categorical data that is included in the - collection, item or asset.
METADATA-REC-07Data VisualisationItem / Asset -

To support the visualization of the dataset, the - render extension is recommended.

-

The render extension allows the definition of the following properties:

-
    -
  • rescale: 2 dimensions array of delimited Min,Max range per band. If not provided, the data will not be rescaled.
    • -
  • Colormap: that must be applied for a raster band
    • -
- 		The render extension supplies rendering tools, like the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md), - with key data to auto-configure visualization settings, including rescaling and colour maps.
Datacube Formats (netCDF, ZARR, ...)
METADATA-REC-08Data ProcessingCollection / Item -

The datacube extension (v2.x) should be used to - properly describe the datacube:

-
    -
  • For a single variable: only use *cube:dimensions*
    • -
  • For multiple variables: *cube:variables* and *cube:dimensions*. Each variable should be a separate datacube, - no attempt should be made to combine variables automatically.
    • -
- 		-

The extension enables correct data parsing into a datacube by the platform or tool.

-

A variable can be bands in EO data or meteorological variables like rain or temperature in meteorological data - sets.

-
Raster Formats (COG)
METADATA-REC-09Data Processing,Data VisualisationItem -

The projection extension must be used to identify - the CRS, raster bounds and shape.

-

At minimum the following must be defined:

-
    -
  • *proj:code*
    • -
  • *proj:bbox*
    • -
  • *proj:shape*
    • -
- 		-

The projection extension ensures that any tools accessing the data can accurately determine key raster - properties without the performance overhead of inspecting the raster file.

-

If the goal is to visualise your data through the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md), - please consider the projections that are currently supported, as defined in [EXPLORER-REQ-07](./geospatial_explorer.qmd).

-
METADATA-REC-10Data Processing,Data VisualisationItemTo incorporate a time dimension, the item must define a *datetime*, *start_datetime* and *end_datetime* at the - item level. Both properties contain a single value using ISO8601 time intervals. -

These properties enable tools to accurately identify the data's temporal dimension, simplifying search and - filtering within the STAC collection.

-

For temporal dimensions, it is recommended to maintain the original level of granularity; data should not be - aggregated from daily records to a single label unless specifically instructed by the user or noted in the metadata. - When combining data and overlap exists, the user must indicate the methodology unless indicated in the metadata.

-
METADATA-REC-11Data Processing,Data VisualisationItem -

The *bands* - array must be used to identify band information in the raster, keep the order as identified in the array.

-
    -
  • Use the *name* property, if provided. Alternatively, use *eo:common_name*. As a last resort, use the array - indices.
    • -
  • Ensure homogeneous data types across bands, choosing the most precise one.
    • -
-
METADATA-REC-12ItemFor other dimensions, the datacube extension must be provided.
METADATA-REC-13Data Processing,Data VisualisationItem -

The raster extension must be used to accurately specify - the following attributes associated with the raster file:

-
    -
  • *raster:spatial_resolution* (if multiple resolutions are provided in the item, otherwise specify it in the - Item)
    • -
  • *raster:scale*
    • -
  • *raster:offset*
    • -
  • *raster:nodata value*
    • -
- 		The raster extension offers valuable information about the dataset, eliminating the need to directly access or - analyse the data itself. For instance, when visualising, details like scale and offset can help convert raw values - into real-world figures.
Statistical Data (FlatGeobuff, GeoJSON)
METADATA-REC-14Data VisualisationItemIt is recommended that STAC records for statistics include a *datetime* property that matches the time stamp of - the source data that the statistics are derived from. The description should also make reference to the boundary - datasets (e.g. NUTS) that the statistics represent.This information offers clear insights into the statistical data and its applications and also assists in integrating - this data into the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md). -
- -Metadata Recommendations -::: diff --git a/interoperability/datahosting.qmd b/interoperability/datahosting.qmd new file mode 100644 index 0000000..9f31790 --- /dev/null +++ b/interoperability/datahosting.qmd @@ -0,0 +1,772 @@ +--- +title: Data Provider Guidelines +--- + +## Requirements + +@tbl-datahosting outlines the interoperability guidelines for EO projects and data providers who wish to deliver their +datasets to APEx for integration within the [ESA Project Results Repository (PRR)](https://eoresults.esa.int/) for +long-term preservation and their utilisation within the APEx Project Environments. By fulfilling these requirements, +APEx ensures seamless integration, discoverability, and usability of the datasets across the ESA EO ecosystem, +facilitating broader access and reusability within the EO community. + +Most of these requirements focus on standardising dataset metadata, formats, and access methods to ensure compatibility +with existing tools and support their efficient exploitation. In particular, datasets should adhere to well-established +EO data standards and provide consistent, machine-readable metadata descriptions. + +APEx supports integration primarily through recognised standards such as STAC (SpatioTemporal Asset Catalogue) and +cloud-native data formats. This ensures that almost any EO dataset can be made available as a ready-to-use resource in +the ESA PRR and used through the APEx tooling. + +Overall, the objective is to streamline and simplify the delivery of high-quality, interoperable EO datasets to APEx, +fostering wider adoption and enabling advanced use cases in downstream applications. + +:::{#tbl-datahosting} + + + + + + + + + + + + + + + + + + + + + + + + + + +
IDRequirementDescription
DATA-REQ-01EO project results with respect to raster data, shall be delivered as cloud-native datasets.Where possible, cloud optimized GeoTIFF [@cog] is preferred. For more complex datasets, GeoZarr [@geozarr] and CF-Compliant netCDF [@netcdf] is a good alternative. Use of the still evolving GeoZarr [@geozarr] format requires confirmation by APEx and may result in future incompatibility if the selected flavour is not standardised eventually. Additional recommendations for the usage of file formats within the APEx services are available [below](#file-format-and-metadata-recommendations).
DATA-REQ-02EO project results with respect to vector data, shall be delivered as cloud-native datasets.Small datasets can use GeoJSON [@geojson], FlatGeobuf [@flatgeobuf] or GeoParquet [@geoparquet] are recommended for larger datasets.
DATA-REQ-03EO project results should be accompanied with metadata in a STAC [@stac] format, including applicable STAC extensions.The specific STAC profiles will align with the recommendations will align with the recommendations provided in the [Metadata Recommendations](#metadata-recommendations) section.
+ +Interoperability requirements for data providers +::: + +## File Format and Metadata Recommendations + +This section provides recommendations for file formats and accompanying STAC metadata for EO project results delivered +to APEx. The recommendations are organised around three additive use case scenarios. Every data provider should start +with Scenario 1. If data also needs to be visualised, Scenario 2 applies on top. Scenario 3 provides additional +recommendations in case the goal is to visualise the data within the APEx Geospatial Explorer. Projects are encouraged +to consult with the APEx team if their use case does not fit within these guidelines. + +The key underlying principle is to use cloud native formats, with adoption rate as a secondary criterion to ensure that +the produced data can reach a broad audience beyond the earth observation community. Although the sections below provide +extra guidance, we refer to the [cloud native geospatial guide](https://guide.cloudnativegeo.org/) for a comprehensive +comparison. + +| Scenario | Formats | Builds on | +| :--- | :--- | :--- | +| 1 – Cloud-based Processing | COG, GeoZarr, NetCDF\ GeoParquet, FlatGeobuf, GeoJSON | (baseline) | +| 2 - Visualization | COG, GeoZarr\ GeoJSON, FlatGeobuf | Scenario 1 | +| 3 - Visualization in APEx Geospatial Explorer | COG, GeoZarr\ GeoJSON, FlatGeobuf | Scenario 1 + 2 | + +:File Format Scenarios + +### STAC Metadata Recommendations + +The STAC specification provides a comprehensive and interoperable framework for describing geospatial datasets. Within +APEx, STAC serves as the foundation to enhance the discoverability, interoperability, and integration of data across a +range of platforms, data catalogues, including the ESA Project Results Repository, and tools such as the APEx +Geospatial Explorer. + +To enhance interoperability, data providers are advised to consistently use a recommended set of STAC-related extensions +and best practices. These recommendations come from community input and collaboration with other initiatives, like +[EarthCODE](https://earthcode.esa.int/) and [EOEPCA](https://eoepca.org/eoepcaplus/), to ensure consistency across +projects and promote the adoption of best practices. + +The following chapters offer a summary of the suggested metadata for the different scenarios. For further details, +please refer to the resources listed below. + +- [STAC Best Practices](https://github.com/radiantearth/stac-best-practices/blob/main/README.md) +- [EOEPCA+ Datacube Access Best Practices](https://github.com/EOEPCA/datacube-access/blob/main/best_practices/stac_best_practices.md) +- [ESA PRR Collection Specifications](https://eoresults.esa.int/prr_collection_specifications.html) + +### Scenario 1: Cloud-based Processing + +This scenario applies to all data providers publishing EO datasets to APEx. It defines the baseline format and metadata +requirements that ensure datasets are cloud-accessible, machine-readable, and compatible with EO processing tools such +as openEO workflows and datacube platforms. + +#### Raster Data + +Cloud Optimised GeoTIFF (COG) is the most widely supported format for geospatial raster data, and also one of the most +efficient in terms of access costs. It is recommended as the default option to consider when publishing static data products. +When combined with STAC metadata, COG can produce self-describing, FAIR-compliant datasets that are easily consumable by +various services. + +##### Generating Cloud Optimised GeoTIFF + +If you are not familiar with COG or GeoTIFF generation, it is recommended to format your GeoTIFF files using a recent version +of GDAL to ensure that they are compliant: + +```bash +gdal_translate world.tif world_webmerc_cog.tif -of COG +``` + +##### Organising spatiotemporal multi-band datasets + +Many datasets have multiple bands (or 'variables'), or have a date associated with them. +The general recommendation is to store a single band per GeoTIFF file and to create STAC items with one asset per band. +This layout is commonly used by many other datasets and avoids the complexities of multi-band GeoTIFF files, which can be +challenging to use. + +An exception to this are the 'RGB' style products, where three bands are used to represent a single image. In this case, +creating a Cloud Optimised GeoTIFF with three bands is an option. + +For associating time information, create one GeoTIFF per timestamp, and one STAC item per timestamp. The GeoTIFF format +has not built-in support for conveying time information, but STAC metadata is supporting this very well. + +More detailed instructions on how to generate Cloud Optimised GeoTIFF for visualisation purposes are outlined in +[here](#cog-formatting-for-the-apex-geospatial-explorer). + +#### Datacubes + +##### GeoZarr + +[GeoZarr](https://geozarr.org/) [@geozarr] is a format that is gaining traction in the geospatial community, although it +is not yet as widely supported as Cloud Optimised GeoTIFF. Its main advantage lies in its ability to store complex +multi-dimensional datasets that go beyond a simple 4D (x, y, time, bands) structure. Just like COG, Zarr allows for +efficient cloud access. + +At the time of writing, there are, however these important caveats: + +- GeoZarr aims to define how to store geospatial raster data in Zarr format. This specification is now going through +the standardisation process. +- By design, Zarr allows for many degrees of freedom, which requires the data producer to have a good understanding of +the associated trade-offs. It is not guaranteed that tooling will automatically generate the most appropriate layout and +chunk size for your use case. +- By design, Zarr stores data as separate files in a directory structure, optimising cloud access but making direct +downloads less convenient. + +##### NetCDF + +NetCDF is a self-describing format with some properties similar to Zarr, but less optimised for cloud access. It can be +useful for exchanging data cubes as single files through traditional methods. However, it is less recommended for convenient +sharing of large datasets, for which either COG or Zarr provide better options. + +#### Vector Data + +**GeoJSON** [@geojson] is suitable for small vector datasets. For larger datasets, **FlatGeobuf** [@flatgeobuf] is +recommended as it supports efficient streaming of features without requiring the full file to be downloaded, improving +performance significantly. **GeoParquet** [@geoparquet] is also a valid option for large vector datasets, particularly +where columnar access patterns are beneficial. + +#### STAC Metadata + +When sharing geospatial datasets in cloud-optimised formats, such as Cloud Optimised GeoTIFF (COG), NetCDF, Zarr and +FlatGeobuf, it is essential to embed as much relevant metadata as possible directly within the files. Although these +formats are designed for efficient cloud access, their interoperability potential is enhanced when the files carry rich, +standardised metadata aligned with their respective specifications. Doing so not only improves data reuse by third-party +tools but also enables more reliable automatic inference of STAC metadata during cataloguing or dataset publication. + +@tbl-processing-stac offers an overview of the general STAC recommendations for cloud-based processing. For more +details and examples on adding this additional metadata to your results, please consult the specific tools (e.g. +[gdal](https://gdal.org/en/stable/drivers/raster/cog.html), [rasterio](https://rasterio.readthedocs.io/en/stable/), …) +for generating the results. + +:::{#tbl-processing-stac} + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
IDLevelRequirementDescription
STAC-REC-01Collection / ItemThe STAC collection and items should use STAC 1.1 or higher.
STAC-REC-02CollectionThe STAC collection must follow the [ESA PRR collection specifications](https://eoresults.esa.int/prr_collection_specifications.html).This guarantees the collection is compatible for upload and registration in the ESA Project Results Repository.
STAC-REC-03Collection / ItemCollections must be homogeneous: each item has the same assets and uses the same asset keys.Consistent and homogeneous definition of assets simplifies client-side handling and supports datacube generation.
STAC-REC-04ItemEach item must have at least one asset where the role is set to *data*.This allows for accurate identification of assets containing the data.
STAC-REC-05Item + Each asset must include: +
    +
  • type
    • +
  • role
    • +
  • title
    • +
  • file:size
    • +
+ Optionally properties: +
    +
  • file:checksum
    • +
+ 		These properties help tools and platforms accurately import the dataset. Furthermore, the *file* properties allow + the ESA PRR to perform extra quality checks.
STAC-REC-06Item +

The projection extension must be used to identify + the CRS, raster bounds and shape.

+

At minimum the following must be defined:

+
    +
  • *proj:code*
    • +
  • *proj:bbox*
    • +
  • *proj:shape*
    • +
+ 		+

The projection extension ensures that any tools accessing the data can accurately determine key raster + properties without the performance overhead of inspecting the raster file.

+

The choice of CRS should consider application-specific recommendations (e.g. Geospatial Explorer), where the + use of multiple UTM zones can introduce complexity, performance issues, and limitations in features such as time + series visualisation and constraints.

+

If the goal is to visualise your data through the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md), + please consider the projections that are currently supported, as defined in [EXPLORER-REQ-07](./geospatial_explorer.qmd).

+
STAC-REC-07ItemTo incorporate a time dimension, the item must define a *datetime*, *start_datetime* and *end_datetime* at the + item level. Both properties contain a single value using ISO8601 time intervals. +

These properties enable tools to accurately identify the data's temporal dimension, simplifying search and + filtering within the STAC collection.

+

For temporal dimensions, it is recommended to maintain the original level of granularity; data should not be + aggregated from daily records to a single label unless specifically instructed by the user or noted in the metadata. + When combining data and overlap exists, the user must indicate the methodology unless indicated in the metadata.

+
Raster Data (COG)
STAC-REC-08Item +

The *bands* + array must be used to identify band information in the raster, keep the order as identified in the array.

+
    +
  • Use the *name* property, if provided. Alternatively, use *eo:common_name*. As a last resort, use the array + indices.
    • +
  • Ensure homogeneous data types across bands, choosing the most precise one.
    • +
+ 		+

The bands array enables platforms and associated tools to accurately identify and map the various bands within + the dataset without the need to access or process the underlying data directly.

+
STAC-REC-09ItemFor other dimensions, the datacube extension must be provided.
STAC-REC-10Item +

The raster extension must be used to accurately specify + the following attributes associated with the raster file:

+
    +
  • *raster:spatial_resolution* (if multiple resolutions are provided in the item, otherwise specify it in the + Item)
    • +
  • *raster:scale*
    • +
  • *raster:offset*
    • +
  • *raster:nodata value*
    • +
+ 		The raster extension offers valuable information about the dataset, eliminating the need to directly access or + analyse the data itself. For instance, when visualising, details like scale and offset can help convert raw values + into real-world figures.
Datacube (GeoZarr, NetCDF)
STAC-REC-11Collection / Item +

The datacube extension (v2.x) should be used to + properly describe the datacube:

+
    +
  • For a single variable: only use *cube:dimensions*
    • +
  • For multiple variables: *cube:variables* and *cube:dimensions*. Each variable should be a separate datacube, + no attempt should be made to combine variables automatically.
    • +
+ 		+

The extension enables correct data parsing into a datacube by the platform or tool.

+

A variable can be bands in EO data or meteorological variables like rain or temperature in meteorological data + sets.

+
+ +STAC metadata recommendations for cloud-based processing +::: + +### Scenario 2: Visualisation + +This scenario applies in addition to [Scenario 1](#scenario-1-cloud-based-processing), when datasets need to be made +available for visualisation purposes, for example, through web-based map viewers or geospatial portals. It extends the +baseline requirements with metadata that enables tools to correctly configure rendering, colour maps, and rescaling +without requiring direct access to the underlying data. + +:::{.callout-tip title="Visualisation in APEx Geospatial Explorer"} +If the target visualisation platform is the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md), +additionally follow the guidelines [Scenario 3](#scenario-3-visualisation-in-apex-geospatial-explorer). +::: + +#### STAC Metadata + +Next to general recommendations, described in [Scenario 1](#stac-metadata), @tbl-visualisation-stac provides an overview of the STAC recommendations that apply to data visualisation. + +:::{#tbl-visualisation-stac} + + + + + + + + + + + + + + + + + + + + + + + +
IDLevelRequirementDescription
STAC-REC-12Item +

In the case of categorical datasets, + classification extension is recommended to identify + the different classes used in the asset.

+

For additional visualization support, it is recommend setting the *title* and *color_hint* properties to allow + external tools, such as the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md) to properly visualise + the data.

+ 		The classification extension supports the proper interpretation of categorical data that is included in the + collection, item or asset.
METADATA-REC-07Item / Asset +

To support the visualization of the dataset, the + render extension is recommended.

+

The render extension allows the definition of the following properties:

+
    +
  • rescale: 2 dimensions array of delimited Min,Max range per band. If not provided, the data will not be rescaled.
    • +
  • Colormap: that must be applied for a raster band
    • +
+ 		The render extension supplies rendering tools, like the [APEx Geospatial Explorer](../instantiation/geospatial_explorer.md), + with key data to auto-configure visualization settings, including rescaling and colour maps.
+ +STAC Recommendations for visualisation +::: + +### Scenario 3: Visualisation in APEx Geospatial Explorer + +This scenario applies in addition to [Scenarios 1](#scenario-1-cloud-based-processing) and +[Scenario 2](#scenario-2-visualisation), when datasets are intended for visualisation in the +[APEx Geospatial Explorer](../instantiation/geospatial_explorer.md) specifically. It adds format-level requirements for +optimal rendering performance in a browser environment, covering tile size, overview pyramids, and interleave settings +for raster data, as well as file size constraints for vector data. + +#### File Format Recommendations + +@tbl-ge-formats summarises the key formatting guidelines designed to enhance data visualisation in the APEx Geospatial Explorer. + +:::{#tbl-ge-formats} + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
IDRecommendationDescription
Raster Data (COG)
EXPLORER-REC-01 +

Set the tile size to 256 or 512 pixels by configuring BLOCKSIZE to the appropriate value using gdal.

+ 		+

The specific choice of tile size (BLOCKSIZE) will depend on whether the use case in the GE is for + visualisation only, or for visualisation and pixel interrogation (data values / charts) with a performance trade + off considerations. Please consult our + [additional recommendations](#cog-formatting-for-the-apex-geospatial-explorer) for specific guidance, noting the + interaction with interleave method is also relevant.

+
EXPLORER-REC-02 +

The dataset should include overviews that meet these criteria:

+
    +
  • Overviews must be created by downsampling the images by factors of two until the dimensions match or fall below the size of a tile.
    • +
  • Each overview should be tiled as well.
    • +
  • To comply with the COG specification, place the overview after the main image data.
    • +
+

You can use the gdal OVERVIEW_RESAMPLING, OVERVIEWS, and OVERVIEW_COUNT settings to generate these overviews.

+ 		+

Overviews are essential for performance as they allow quick visualisation of the data on different zoom levels, + based on resampling at factors .2, 4, 8, 16, 32. If necessary, factor 2 can be skipped to reduce file + size (using gdaladdo tool to build overviews), although performance will be impacted at some zoom scales.

+

The objective is to strike an optimal balance between the number of overviews and the total file size.

+

Please consult our [additional recommendations](#cog-formatting-for-the-apex-geospatial-explorer) for + specific guidance.

+
EXPLORER-REC-03 +

For multi-band data, set INTERLEAVE to BAND with gdal to efficiently render band composites.

+ 		+

Band interleaved data is a safe all round choice, but for specific scenarios / use cases, there may be + performance benefits for pixel interleaved data. Please refer to @tbl-cog-settings for specific use case + examples, noting that the combination of tile size (BLOCKSIZE) is also relevant.

+
Vector Formats (GeoJSON, FlatGeobuf, GeoParquet)
EXPLORER-REC-04 +

GeoJSON data sources should be kept below 5 Megabytes in size per file.

+ 		+

GeoJSON is not partially readable, therefore each source will be fetched in entirety at the initial load of + the Geospatial Explorer. Large sizes and number of sources can be detrimental to the performance of the + Geospatial Explorer.

+

For such data sets consider using a stream-able format such as FlatGeobuf [@flatgeobuf].

+
Tabular Formats (linked charts)
EXPLORER-REC-05 +

External CSV files can be used as a source for chart data (e.g. time series), linked either to an entire data + source, or to a specific feature in a vector dataset, whereby the CSV URL is referenced from a field property. + CSV files would typically contain a date column and one or more data columns which are configured as series in + a chart.

+

CSV files should be kept below 5 Megabytes in size per file.

+ 		+

Please note, CSV data is not supported in the ESA Project Results Repository.

+
+ +File format recommendations for data visualisation in APEx Geospatial Explorer +::: + +#### COG Formatting for the APEx Geospatial Explorer + +The following GDAL command implements the recommended settings for Explorer-optimised COG files: + +```{bash} +gdal_translate \ + -of COG \ + -co BLOCKSIZE=512 \ + -co INTERLEAVE=BAND \ + -co OVERVIEWS=AUTO \ + input.tif output_cog +``` + +Alternatively, where a custom set of overviews that skips the x2 level is required, a combination the following +GDAL commands would be used: + +```{bash} +gdal_translate input.tif temp.tif \ + -co TILED=YES \ + -co BLOCKXSIZE=512 \ + -co BLOCKYSIZE=512 \ + -co INTERLEAVE=BAND +``` + +```{bash} +gdaladdo -r average temp.tif 4 8 16 32 +``` + +```{bash} +gdal_translate temp.tif output_cog.tif \ + -of COG \ + -co BLOCKSIZE=512 \ + -co INTERLEAVE=BAND \ + -co COPY_SRC_OVERVIEWS=YES +``` + +@tbl-cog-settings is based on the principles of accessing cloud-optimised data via web GIS technology. It is +recommended that the EO project teams test their own sample outputs with the Geospatial Explorer to ascertain specific +performance characteristics before processing full operational pipelines. + +:::{#tbl-cog-settings} + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SettingsGeospatial Explorer Use Cases
Dataset TypeTile SizeInterleaveSingle-Band Web VizColour Composite Web VizPixel Interrorgation
Single-band raster512N/ABestN/APoor
Single-band raster256N/AGoodN/ABest
RGB (3-band) imagery512PIXELBestBestGood
RGB (3-band) imagery256PIXELGoodGoodBest
RGB (3-band) imagery512BANDAverageGoodGood
RGB (3-band) imagery256BANDAverageAverageGood
Multispectral (10-20 bands)512PIXELGoodBestAverage
Multispectral (10-20 bands)256PIXELAverageGoodGood
Multispectral (10-20 bands)512BANDBestBestGood
Multispectral (10-20 bands)256BANDGoodGoodBest
Hyperspectral (e.g. 200+ bands)512PIXELPoorPoorPoor
Hyperspectral (e.g. 200+ bands)256PIXELPoorAverageAverage
Hyperspectral (e.g. 200+ bands)512BANDBestBestAverage
Hyperspectral (e.g. 200+ bands)256BANDGoodGoodBest
+ +Specific use case examples for COG files with interleave method and tile size +::: + +#### Statistical Datasets + +Statistical datasets can be used to store precomputed statistics for dataset variables based on spatial units, such as +administrative areas. An example is to collect land cover statistics on using boundaries from nomenclature of territorial +units for statistics (NUTS), as shown in the [APEx Geospatial Explorer](https://explorer.apex.esa.int/) (Statistics). The +guidelines in this section are focused on supporting the integration of statistical data for visualisation in the APEx +Geospatial Explorer. + +The statistical datasets are expected to be vector layers that are provided in a format that can be parsed to a feature +collection following the GeoJSON [@geojson] specification. Currently tested and supported formats are GeoJSON [@geojson] +and FlatGeobuf [@flatgeobuf]. FlatGeobuf should be used where the statistical data is a large size as this allows for +streaming of the relevant features without having to download the full dataset, increasing performance. + +The metadata header of the file should contain the following properties to define which fields on the features in the +dataset should be used for the following purposes. + +- identifierKey: The name of the field that stores the unique identifier for each feature. +- nameKey: The name of the field that stores the human-readable name for display. +- levelKey: The name of the field that stores the administrative level number. +- childrenKey: The name of the field that has a comma-separated list of child feature IDs as declared in identifierKey. +Can be the empty string if this is the bottom level. +- attributeKeys: A comma-separated list of field numbers that store the statistical data. +- units: The units as displayed in the UI. This is for UI purposes only and has no effect on the data. +- visualization_hint: A string of histogram, categorised, or continuous used as a hint to the UI to choose a suitable +presentation for the data. + +For example, properties in the file metadata that is defined as follows: + +- identifierKey: NUTS_ID +- nameKey: NUTS_NAME +- levelKey: LEVL_CODE +- childrenKey: children +- attributeKeys: Trees, Shrubland, Grassland +- visualization_hint: categorised + +would use the fields NUTS_ID, NUTS_NAME, … in the data to determine the navigation and display of statistics in the +Geospatial Explorer. For further guidance, please contact the APEx team through the [APEx User Forum](http://forum.apex.esa.int/). + +Datasets that have classifications (such as land use) should have key:value entires consisting of 'name':'value' and an +entry with a key of 'classifications' with a value consisting of a string based comma separated list containing all the +keys for the classifications and a 'total' key with the sum of all other values. This will allow for correctly rendering +bar charts and pie charts. + +```{json} +{ + Bare / sparse vegetation: 3349.349614217657, + Built-up: 18474.280639104116 + Cropland: 155067.6934300016 + Grassland: 140178.79417018566 + Herbaceous wetland: 1612.828666906516 + Mangroves: 479.46053523623897 + Moss and lichen: 499.40601429089236 + Permanent water bodies: 8969.837211370474 + Shrubland: 7342.96093361589 + Snow and ice: 495.7695064816955 + Tree cover: 301783.0035618253 + Unknown: 1.7258467103820294 + total: 638255.1101299465 + classifications: "Tree cover,Shrubland,Grassland,Cropland,Built-up,Bare / sparse vegetation,Snow and ice, + Permanent water bodies,Herbaceous wetland,Mangroves,Moss and lichen,Unknown" +} +``` + +![worldcover_bar_chart_example](./images/worldcover_bar_chart_example.png){width=75%} + +Datasets that do not have classifications (such as a raster showing soil organic carbon) should contain a selection of +the following entries: + +- mean +- min +- max + +These values will be rendered as a table. + +```{json} +{ + mean: 437.94353402030356 + min: 60 + max: 4410 +} +``` + +![worldsoils_table_example](./images/worldsoils_table_example.png){width=75%} + +#### STAC Metadata + +The STAC metadata, as described in [Scenario 1](#stac-metadata) and [Scenario 2](#stac-metadata-1), is sufficient to +support data visualisation in the APEx Geospatial Explorer. diff --git a/interoperability/definitions.qmd b/interoperability/definitions.qmd index 421773d..5afdd10 100644 --- a/interoperability/definitions.qmd +++ b/interoperability/definitions.qmd @@ -49,18 +49,18 @@ For the sake of simplicity in the context of APEx, we refer to these simply as ( ### Application Algorithm Definition -The application algorithm definition refers to a representation of the algorithm modules and interfaces that can be exposed -by an API and/or processing platform. Typically, it includes a general description of the algorithm along with detailed -information on its parameters, expected output, scientific method, and an overview of the different steps executed -within the algorithm. Examples of algorithm definitions include openEO’s User Defined Processes (UDP) [@udp] and OGC -Application Package [@ap], using the Common Workflow Language (CWL) [@cwl]. +The application algorithm definition refers to a representation of the algorithm modules and interfaces that can be +exposed by an API and/or processing platform. Typically, it includes a general description of the algorithm along with +detailed information on its parameters, expected output, scientific method, and an overview of the different steps +executed within the algorithm. Examples of algorithm definitions include openEO’s User Defined Processes (UDP) [@udp] +and OGC Application Package [@ap], using the Common Workflow Language (CWL) [@cwl]. To increase uptake and interoperability, APEx aims to enable the execution of application algorithm definitions via -standardised web service APIs. This transitions algorithms from being rather arbitrary pieces of software with potentially -complex requirements, in terms of execution environment, usage, inputs, …, into on-demand services that can readily be -invoked by stakeholders. This transformation primarily involves making the application algorithm definition available as -a service on an algorithm hosting platform. The transition process into a hosted on-demand algorithm service is supported -by the APEx Algorithm Support. +standardised web service APIs. This transitions algorithms from being rather arbitrary pieces of software with +potentially complex requirements, in terms of execution environment, usage, inputs, …, into on-demand services that can +readily be invoked by stakeholders. This transformation primarily involves making the application algorithm definition +available as a service on an algorithm hosting platform. The transition process into a hosted on-demand algorithm +service is supported by the [APEx Algorithm Support](../propagation/index.qmd). An important boundary condition for hosted algorithms is that they can be executed at a predictable cost for a given set of inputs. This predictability allows a service user to accurately estimate and determine the cost associated with the @@ -68,11 +68,11 @@ execution of the final deployed service instance. ### Algorithm Catalogue -The [APEx Algorithm Catalogue](https://algorithm-catalogue.apex.esa.int/) is a central register of algorithm -definitions and the corresponding algorithm service instances that can be executed on APEx-compliant algorithm hosting -platforms. Curated by APEx, the catalogue relies on automated checks to ensure that advertised algorithms service -instances are available and functional. Whenever a malfunction is detected, this is reported to ESA and the EO project -consortium, allowing them to determine a proper course of action. +The [APEx Algorithm Catalogue](https://algorithm-catalogue.apex.esa.int/) is a central register of algorithm definitions +and the corresponding algorithm service instances that can be executed on APEx-compliant algorithm hosting platforms. +Curated by APEx, the catalogue relies on automated checks to ensure that advertised algorithms service instances are +available and functional. Whenever a malfunction is detected, this is reported to ESA and the EO project consortium, +allowing them to determine a proper course of action. ### Algorithm Hosting Platform @@ -81,8 +81,8 @@ definition. In APEx, an algorithm hosting platform specifically refers to platfo OGC Application Package Algorithm description standards. These platforms also enable algorithms to be executed by the openEO API or through OGC API Processes. For APEx, these platforms are considered existing providers available through [ESA’s Network of Resources (NoR)](https://nor-discover.org/). Examples of such algorithm hosting platforms are the -[Copernicus Data Space Ecosystem](https://dataspace.copernicus.eu/analyse/openeo) for openEO -or [Ellip](https://ellip.terradue.com/) for OGC Application Packages. +[Copernicus Data Space Ecosystem](https://dataspace.copernicus.eu/analyse/openeo) for openEO or +[Ellip](https://ellip.terradue.com/) for OGC Application Packages. It is important to note that APEx itself is not an algorithm hosting platform; rather, it promotes the reuse of existing platforms. A key property of algorithm hosting platforms is their long-term sustainability beyond the lifetime of a @@ -97,8 +97,8 @@ aspects, ensuring that EO projects retain a sufficient degree of freedom in sele ### Last Mile Applications A ‘last mile’ application, as seen from the EO perspective, bridges the final gap between the user at the end of the -value chain, and the web service APIs that offer EO-derived information. This could, for instance, be the integration -of parcel statistics into the field management software of a farmer. In the ESA context, Green Transition Information +value chain, and the web service APIs that offer EO-derived information. This could, for instance, be the integration of +parcel statistics into the field management software of a farmer. In the ESA context, Green Transition Information Factories can be considered examples of last mile applications. Typically, each user group and domain will have its own set of purpose-built tools. The IT integrator or vendor building @@ -106,25 +106,25 @@ these tools will retrieve EO algorithm results via web service APIs based on ope further processed as needed, depending on the application. Note that these applications are not necessarily EO-centric or even geospatial but could be using an EO algorithm as a small part or in the background. -The APEx Algorithm Catalogue acts as a discovery tool for last mile application builders. It shows various -technical parameters, the cost and potential limitations for specific use cases. This speeds up the discovery and selection -process. Application builders will be required to create an account on the hosting platforms and ensure the necessary -funds are available, if they want to test or use the API. +The [APEx Algorithm Catalogue](https://algorithm-catalogue.apex.esa.int/) acts as a discovery tool for last mile +application builders. It shows various technical parameters, the cost and potential limitations for specific use cases. +This speeds up the discovery and selection process. Application builders will be required to create an account on the +hosting platforms and ensure the necessary funds are available, if they want to test or use the API. APEx currently offers two main options for projects to develop their on-demand services: -* OpenEO-based services [@openeo], implemented in line with openEO’s User Defined Processes (UDP) [@udp] -* OGC API - Processes-based services [@apiprocess], implemented in line with the OGC Application Package Best -Practice [@ap] +- OpenEO-based services [@openeo], implemented in line with openEO’s User Defined Processes (UDP) [@udp] +- OGC API - Processes-based services [@apiprocess], implemented in line with the OGC Application Package Best Practice + [@ap] These APEx-compliant technologies allow algorithms to be hosted on an APEx-compliant algorithm hosting platform and make them available for execution through web services. These technologies promote seamless reuse and integration of existing -EO algorithms. Additionally, by leveraging web services and cloud-based approaches, these technologies simplify the execution -of EO algorithms, shielding users from underlying complexities, such as data access, data processing optimisation, and other -technical challenges. +EO algorithms. Additionally, by leveraging web services and cloud-based approaches, these technologies simplify the +execution of EO algorithms, shielding users from underlying complexities, such as data access, data processing +optimisation, and other technical challenges. -APEx remains committed to future innovation and is open to integrating additional specifications, provided they align with -FAIR principles and facilitate algorithm execution through web services. +APEx remains committed to future innovation and is open to integrating additional specifications, provided they align +with FAIR principles and facilitate algorithm execution through web services. ## Actors @@ -149,7 +149,8 @@ quality of the algorithms and workflows. The APEx team, composed of industry experts, operates the various services provided by APEx. To maximise the reuse of existing resources, the team leverages service offerings within -[ESA’s Network of Resources (NoR)](https://nor-discover.org/), drawing on the extensive ecosystem provided by the EO industry. +[ESA’s Network of Resources (NoR)](https://nor-discover.org/), drawing on the extensive ecosystem provided by the EO +industry. Although members of the APEx consortium are involved in various EO services registered in NoR, APEx itself is not a new EO platform. Instead, it focuses on enabling interoperability among existing platforms. As a result, APEx remains open @@ -170,12 +171,12 @@ and user satisfaction. ### Last Mile Applications A 'last mile' application, as seen from the EO perspective, bridges the final gap between the user at the end of the -value chain, and the web service APIs that offer EO-derived information. This could, for instance, be the integration -of parcel statistics into the field management software of a farmer. In the ESA context, Green Transition Information +value chain, and the web service APIs that offer EO-derived information. This could, for instance, be the integration of +parcel statistics into the field management software of a farmer. In the ESA context, Green Transition Information Factories can be considered examples of last mile applications. Typically, each user group and domain will have its own set of purpose-built tools. The IT integrator or vendor building -these tools will retrieve EO algorithm results via web service APIs based on open standards. These results can then be +these tools will retrieve EO algorithm results via web service APIs based on open standards. These results can then be further processed as needed, depending on the application. Note that these applications are not necessarily EO-centric or even geospatial, but could be using an EO algorithm as a small part or in the background. diff --git a/interoperability/geospatial_explorer.qmd b/interoperability/geospatial_explorer.qmd index f3a19af..a531656 100644 --- a/interoperability/geospatial_explorer.qmd +++ b/interoperability/geospatial_explorer.qmd @@ -68,8 +68,10 @@ and best practices shall be followed to avoid any breaking changes between versi EXPLORER-REQ-04 - All external sources shall be publicly available. - Currently the application does not support fetching sources from hosts that require authentication. + All external sources shall be publicly available or accessible via an APEx user token. + The Geospatial Explorer will need to fetch any configured data directly from a source server. It is expected + that these resources can be accessed directly through a HTTP request that is either open access (No authentication + requirements) or can be authenticated using an APEx user bearer token via the HTTP Authentication header. EXPLORER-REQ-05 @@ -88,36 +90,61 @@ and best practices shall be followed to avoid any breaking changes between versi EXPLORER-REQ-07 - All datasets shall be projected to CRS EPSG:3857 (Web Mercator); EPSG:4326 (WGS84) or EPSG:3035 (ETRS89-extended - / LAEA Europe) - Other projections may work but will not be explicitly supported in the initial versions of the application. - Support for these projections will be considered and reviewed on a case-by-case basis and will be subject to support - by the PROJ4JS library [@proj4js]. + For optimal visualisation performance, datasets should be projected into the same CRS as the display CRS used + for the Geospatial Explorer. + +

Where the CRS is not the same as the display CRS in the Geospatial Explorer, data will be reprojected “on the + fly”, with a performance impact for display that will depend on the complexity of the projection transformation + between dataset and Geospatial Explorer display CRS’s.

+

All Geospatial Explorer instances will support the following projections which can be used as the display CRS + and / or any of the data CRS’s

+ +

In addition it is possible to support any projection supported by the PROJ4JS library [@proj4js] by + configuring an appropriately formatted projection string in the Geospatial Explorer JSON configuration file, + which can be done via the Configuration Builder (CB) tool.

+

**Note**: APEx recommends datasets should be provided in a single, standardised projection system that covers + the full spatial extent of the data. Fragmented projections (e.g. UTM across multiple zones) should be avoided + as they introduce complexity, reduce performance, and limit key Geospatial Explorer functionalities.

+ EXPLORER-REQ-08 + Where WM(T)S is used, its sources should declare their projection through the GetCapabilities request. + This ensures that the Geospatial Explorer renders the data in the corresponding projection and prevents + unnecessary on-the-fly reprojections. + + + EXPLORER-REQ-09

Legends support is provided in the Geospatial Explorer via the following mechanisms:

-

Legend settings can be configured using the configuration builder tool.

+

Legend settings can be configured using the Configuration Builder tool.

This guideline ensures that the legend is fully compatible with the visualisation library used in the Geospatial Explorer. - EXPLORER-REQ-09 + EXPLORER-REQ-10 Statistical datasets should be configured by providing public URLs to either: GeoJSON [@geojson] or - FlatGeobuf [@flatgeobuf]. Details on the structure of these datasets are outlined in the [File Formats](../guides/file_formats.qmd#statistical-datasets-flatgeobuf-geojson) section. + FlatGeobuf [@flatgeobuf]. Details on the structure of these datasets are outlined in the [Data Provider Guidelines](../interop) section. The statistics feature is complex due to the architecture and requirements of the Geospatial Explorer. Statistical datasets contain one or more files that include vector features that describe an area and the relevant statistics for the area can be added to the configuration of an explorer instance. - EXPLORER-REQ-10 + EXPLORER-REQ-11 -

To facilitate the integration of STAC datasets within the Geospatial Explorer, it is advisable to follow the recommendations presented in our [metadata recommendations](./datahosting.md#metadata-recommendations). For optimal results within the Geospatial Explorer, the use of the following STAC extensions is recommended:

+

To facilitate the integration of STAC datasets within the Geospatial Explorer, it is advisable to follow the recommendations presented in our [metadata recommendations](./datahosting.qmd#stac-metadata-2). For optimal results within the Geospatial Explorer, the use of the following STAC extensions is recommended: