-
Notifications
You must be signed in to change notification settings - Fork 18
ARGANS Waterlines from Sentinel-2 algorithm #468
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mileonai
wants to merge
60
commits into
main
Choose a base branch
from
argans_waterlines
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
60 commits
Select commit
Hold shift + click to select a range
939312d
added provider record
mileonai 22df88b
added logo
mileonai f739814
updated logo
mileonai 166653b
updated logo
mileonai a2a9d20
first commit
mileonai 0f835a8
added descriptions
mileonai 2cf2348
build graph for each method
mileonai e91d1d5
better docs
mileonai 431bbf9
added benchmark scenario
mileonai 0dc7b24
removed commented out code
mileonai 415f7e0
readme
mileonai 6d816bb
corrected typos
mileonai 957b6b2
refactored waterliens in order to return vectorcube
mileonai a1ce6cf
docstring update
mileonai 4aa4992
TODO about vectorisation added
mileonai 824537b
modified _bin
mileonai df28957
Merge branch 'main' into argans_waterlines
mileonai 0e90e98
using only one idex method as MVP
mileonai 6ee5ffb
added links to Argans logo
mileonai f1644ee
updated description
mileonai c446fbe
updated process record
mileonai 8d74115
addec Coastal keyword
mileonai 1c9f08e
updated description
mileonai b413271
updadated benchmark scenario
mileonai 689219e
removed commented out code
mileonai 7397c79
changed platform and branch name
mileonai a2c6e25
changed branch name
mileonai d02de17
switched to main in url
mileonai 61b393f
not main branch
mileonai 8fcd2d3
fixed some characters
mileonai 3fb9bd2
removed wrong url to code
mileonai cd46c60
fix ndwi thresholding by applying comparison element-wise
mileonai de9b00e
updated README
mileonai 32eb35e
raise exception when records are empty
mileonai a669823
updated waterlines.json
mileonai f76a2e9
removed dot and updated update date
mileonai df2b284
ranamed to waterlines
mileonai d828bfd
changed default time range
mileonai 0641d21
removing small water polygons, as they are crated during udf appy_ufu…
mileonai 01c5ea7
Merge branch 'main' into argans_waterlines
mileonai c2bf671
regenerated process graph
mileonai defa6e1
using within and bigger buffer for edges removal
mileonai f7d3332
regenerated process graph
mileonai a9c30e1
modified way to provide context for udf's
mileonai 59f1010
renamed upd
mileonai 85775ec
simplify_tolerance hardcoded
mileonai 8b19a51
Updated role
CamMackenzie98 8b7d320
removed import of unused libs
mileonai f554419
update udf
mileonai 4fec070
using create_waterlines wrapper
mileonai 2736baa
fixed udf
mileonai 4818737
restored simplify_tolerance and refactored so parameters from udf con…
mileonai 8637cd7
not throwing an exception
mileonai 69ab21f
Merge branch 'main' into argans_waterlines
JanssenBrm 77708f7
Merge branch 'main' into argans_waterlines
JanssenBrm f44add8
fix: record validation
JanssenBrm 43d0b17
fix: fixed benchmark ephemeral link
JanssenBrm e34841c
fix: updated benchmark links
JanssenBrm 436cc0d
fix: updated process_id based on the actual udp
JanssenBrm b4a23c0
Merge branch 'main' into argans_waterlines
JanssenBrm File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| { | ||
| "id": "argans", | ||
| "type": "Feature", | ||
| "conformsTo": [ | ||
| "https://www.opengis.net/spec/ogcapi-records-1/1.0/req/record-core" | ||
| ], | ||
| "properties": { | ||
| "created": "2026-04-01T00:00:00Z", | ||
| "updated": "2026-04-01T00:00:00Z", | ||
| "type": "algorithm_provider", | ||
| "title": "Argans Ltd", | ||
| "description": "Argans specializes in satellite-based Earth observation, remote sensing, and GIS for monitoring marine, atmospheric, and land environments.", | ||
| "keywords": [], | ||
| "language": { | ||
| "code": "en-US", | ||
| "name": "English (United States)" | ||
| }, | ||
| "languages": [ | ||
| { | ||
| "code": "en-US", | ||
| "name": "English (United States)" | ||
| } | ||
| ], | ||
| "contacts": [ | ||
| { | ||
| "name": "Argans Ltd", | ||
| "emails": [ | ||
| { | ||
| "value": "enquiries@argans.co.uk" | ||
| } | ||
| ], | ||
| "links": [ | ||
| { | ||
| "href": "https://argans.co.uk", | ||
| "type": "text/html", | ||
| "title": "Argans Ltd", | ||
| "rel": "website" | ||
| } | ||
| ] | ||
| } | ||
| ], | ||
| "themes": [], | ||
| "acl": { | ||
| "admin": ["@argans.co.uk"] | ||
| } | ||
| }, | ||
| "linkTemplates": [], | ||
| "links": [ | ||
| { | ||
| "rel": "website", | ||
| "type": "text/html", | ||
| "title": "Argans Ltd", | ||
| "href": "https://argans.co.uk" | ||
| }, | ||
| { | ||
| "rel": "logo-light", | ||
| "type": "image/png", | ||
| "title": "Logo", | ||
| "href": "https://argans.co.uk/img/logo.png" | ||
| }, | ||
| { | ||
| "rel": "logo-dark", | ||
| "type": "image/png", | ||
| "title": "Logo", | ||
| "href": "https://argans.co.uk/img/logos/argans_white_new.png" | ||
| } | ||
| ] | ||
| } |
30 changes: 30 additions & 0 deletions
30
algorithm_catalog/argans/waterlines/benchmark_scenarios/waterlines.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| [ | ||
| { | ||
| "id": "waterlines", | ||
| "type": "openeo", | ||
| "backend": "openeofed.dataspace.copernicus.eu", | ||
| "process_graph": { | ||
| "waterlines1": { | ||
| "process_id": "waterlines_v1", | ||
| "namespace": "https://raw.githubusercontent.com/ESA-APEx/apex_algorithms/43d0b17c4ae5bed9db5f9cc5df38efcdcdfb7ecf/algorithm_catalog/argans/waterlines/openeo_udp/waterlines.json", | ||
| "arguments": { | ||
| "temporal_extent": ["2024-06-01", "2024-06-30"], | ||
| "spatial_extent": { | ||
| "west": -95.13, | ||
| "south": 29.078, | ||
| "east": -95.12, | ||
| "north": 29.082, | ||
| "crs": "EPSG:4326" | ||
| }, | ||
| "iterations": 2, | ||
| "max_cloud_coverage": 5, | ||
| "ndwi_threshold": 0.01, | ||
| "simplify_tolerance": 10 | ||
| }, | ||
| "result": true | ||
| } | ||
| }, | ||
| "reference_data": { | ||
| } | ||
| } | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Waterlines openEO UDP | ||
| ## Purpose | ||
| Extract coastline waterlines from Sentinel-2 imagery using NDWI-based water detection, morphological refinement, and UDF-based conversion from water polygons to coast waterlines. | ||
|
|
||
| ## Methodology | ||
| ### Water/Land Classification | ||
| Water masks are generated using the **Normalized Difference Water Index (NDWI)**, where pixels are classified as water when **NDWI > threshold**. The default threshold is **0.01**, but it can be adjusted using the `ndwi_threshold` parameter. | ||
|
|
||
| NDWI is computed as the normalized difference between the Sentinel-2 **Green** band (B03) and **Near-Infrared** band (B08), defined as the difference between these bands divided by their sum. | ||
|
|
||
| *This MVP supports only one method (**S2_NDWI**). Originally, multiple methods were selectable via a parameter, but this required openEO `if_()` logic, which converts the result into a `ProcessBuilder` instead of a `DataCube`. | ||
| This breaks the `raster_to_vector()` step needed for waterline extraction.* | ||
|
|
||
| ### Morphological Processing | ||
| For each timestamp, the water/land mask is refined using morphological operations to remove small isolated objects, fill small holes, smooth boundaries and reduce artifacts such as narrow bridges and estuaries. This improves the quality and stability of the resulting waterlines. | ||
|
|
||
| ### Waterline Extraction | ||
| The cleaned masks are vectorized using the built-in openEO function `raster_to_vector()`. The resulting water polygons are then transformed into waterlines via a UDF, producing time-resolved geometries for each timestep. | ||
|
|
||
| The output is a vector cube of coastline waterlines with the following properties: | ||
| - **time**: Acquisition timestamp (Sentinel-2 datetime) | ||
| - **type**: Feature type (`waterline_segment`) | ||
| - **sea_direction_8**: Sea direction (N, NE, E, SE, S, SW, W, NW) | ||
| - **sea_azimuth_deg**: Sea direction in degrees (azimuth, clockwise from north) | ||
| - **geometry**: Waterline geometry (LineString or MultiLineString) in EPSG:3857 | ||
|
|
||
| The **sea_azimuth_deg** property is particularly useful for downstream processing, as it can be used to shift the waterline and derive a shoreline (*a waterline normalized for beach slope and tidal conditions*). | ||
|
|
||
| ## Authors / Contact | ||
| - **Milena Napiorkowska** (openEO UDP) Argans Ltd | ||
| mnapiorkowska@argans.co.uk | ||
|
|
||
| - **Martin Jones** (Project Manager) Argans Ltd | ||
| mjones@argans.co.uk | ||
|
|
||
| - **Holly Baxter** (Methodology) Argans Ltd | ||
| hbaxtar@argans.co.uk | ||
|
|
||
| - **Cameron Mackenzie** (Methodology, openEO UDP) Argans Ltd | ||
| cmackenzie@argans.co.uk | ||
|
|
||
| ## Acknowledgments | ||
| This work was developed as part of an ESA-funded **Fast Track** project. | ||
|
|
||
| ## Known Limitations | ||
| - Results are most reliable for scenes with low cloud coverage | ||
| - NoData areas may introduce artifacts, particularly along boundaries between valid and invalid pixels | ||
| - NDWI might be less reliable in turbid waters |
179 changes: 179 additions & 0 deletions
179
algorithm_catalog/argans/waterlines/openeo_udp/generate.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| import json | ||
| from pathlib import Path | ||
|
|
||
| import openeo | ||
| from openeo import UDF | ||
| from openeo.api.process import Parameter | ||
| from openeo.rest.connection import Connection | ||
| from openeo.rest.datacube import DataCube | ||
| from openeo.rest.udp import build_process_dict | ||
|
|
||
| from s2_index import ( | ||
| s2_index_mask, | ||
| DEFAULT_S2_COLLECTION, | ||
| DEFAULT_MAX_CLOUD_COVER, | ||
| WATERLAND_THRESHOLDS, | ||
| ) | ||
|
|
||
|
|
||
| def apply_morphology(cube: DataCube, iterations: int) -> DataCube: | ||
| udf = UDF.from_file( | ||
| Path(__file__).parent / "udf_morph_operations.py", | ||
| context={"from_parameter": "context"}, | ||
| ) | ||
| return cube.apply_dimension( | ||
| process=udf, | ||
| dimension="t", | ||
| context={"iterations": iterations}, | ||
| ) | ||
|
|
||
|
|
||
| def create_waterlines(cube: DataCube, simplify_tolerance: float = 10) -> DataCube: | ||
| """ | ||
| Extract waterlines from a water/land mask using a UDF. | ||
|
|
||
| The input must remain a DataCube because the workflow relies on | ||
| `raster_to_vector()` before applying the vector-based waterline UDF. | ||
| """ | ||
| cube = cube.raster_to_vector() | ||
|
|
||
| udf = UDF.from_file( | ||
| Path(__file__).parent / "udf_waterlines_from_water_land_mask.py", | ||
| context={"from_parameter": "context"}, | ||
| ) | ||
| return cube.apply_dimension( | ||
| process=udf, | ||
| dimension="geometry", | ||
| context={"simplify_tolerance": simplify_tolerance}, | ||
| ) | ||
|
|
||
|
|
||
| def build_water_land_mask_cube( | ||
| con: Connection, | ||
| bbox, | ||
| time_range, | ||
| max_cloud_coverage, | ||
| iterations, | ||
| ndwi_threshold, | ||
| ) -> DataCube: | ||
| """ | ||
| Build a water/land mask using Sentinel-2 NDWI only. | ||
|
|
||
| MVP rationale: | ||
| Multiple selectable methods were intentionally removed from this UDP. | ||
| Selecting between whole DataCubes through nested openEO `if_()` expressions | ||
| turns the selected result into a ProcessBuilder rather than a DataCube. | ||
| That breaks the next step of the workflow, because `raster_to_vector()` is | ||
| used as a DataCube method when preparing input for the vector-based | ||
| waterline UDF. | ||
|
|
||
| S2_NDWI was chosen for the MVP because it is a standard water-detection | ||
| index, fits the existing index-mask pipeline, and allows validation of the | ||
| complete end-to-end workflow without introducing graph-selection complexity. | ||
|
|
||
| Future extensions can add the other methods either as separate UDPs or by | ||
| refactoring the waterline UDF to work directly on raster input. | ||
| """ | ||
| _, cube = s2_index_mask( | ||
| con=con, | ||
| collection_id=DEFAULT_S2_COLLECTION, | ||
| bbox=bbox, | ||
| time_range=time_range, | ||
| index_name="S2_NDWI", | ||
| threshold=ndwi_threshold, | ||
| mode="gt", | ||
| max_cloud_coverage=max_cloud_coverage, | ||
| ) | ||
| return apply_morphology(cube, iterations) | ||
|
|
||
|
|
||
| def generate() -> dict: | ||
| """ | ||
| Create the MVP UDP for extracting waterlines from Sentinel-2 imagery. | ||
|
|
||
| Workflow: | ||
| 1. Load Sentinel-2 data | ||
| 2. Create water/land mask using S2_NDWI | ||
| 3. Apply morphology | ||
| 4. Vectorize the mask | ||
| 5. Extract waterlines | ||
|
|
||
| Why only S2_NDWI? | ||
| The original multi-method design used a runtime UDP parameter to choose | ||
| between several masking methods. In practice, selecting between whole cubes | ||
| with openEO graph logic (`if_`) produced a ProcessBuilder instead of a | ||
| DataCube. Because the downstream workflow needs `raster_to_vector()`, that | ||
| design blocked the current implementation. | ||
|
|
||
| Restricting the MVP to a single method keeps the graph in DataCube form and | ||
| allows the existing vector-based waterline UDF to work unchanged. | ||
| """ | ||
|
|
||
| conn = openeo.connect(url="openeo.dataspace.copernicus.eu") | ||
|
|
||
| spatial_extent = Parameter.bounding_box( | ||
| name="spatial_extent", | ||
| description="Bounding box of the area of interest. Defined as west, south, east, north in EPSG:4326.", | ||
| ) | ||
|
|
||
| temporal_extent = Parameter.temporal_interval( | ||
| name="temporal_extent", | ||
| default=["2025-01-01", "2025-12-31"], | ||
| description="Date range over which to extract waterlines.", | ||
| ) | ||
|
|
||
| max_cloud_coverage = Parameter.number( | ||
| name="max_cloud_coverage", | ||
| default=DEFAULT_MAX_CLOUD_COVER, | ||
| description="Maximum allowed cloud coverage.", | ||
| ) | ||
|
|
||
| iterations = Parameter.integer( | ||
| name="iterations", | ||
| default=2, | ||
| description="Number of iterations for morphological operations.", | ||
| ) | ||
|
|
||
| ndwi_threshold = Parameter.number( | ||
| name="ndwi_threshold", | ||
| default=WATERLAND_THRESHOLDS["S2_NDWI"].defaults["threshold"], | ||
| description=WATERLAND_THRESHOLDS["S2_NDWI"].description, | ||
| ) | ||
|
|
||
| simplify_tolerance = Parameter.number( | ||
| name="simplify_tolerance", | ||
| default=10, | ||
| description="Tolerance used to simplify vectorized water polygons before extracting waterlines.", | ||
| ) | ||
|
|
||
| water_land_mask = build_water_land_mask_cube( | ||
| con=conn, | ||
| bbox=spatial_extent, | ||
| time_range=temporal_extent, | ||
| max_cloud_coverage=max_cloud_coverage, | ||
| iterations=iterations, | ||
| ndwi_threshold=ndwi_threshold, | ||
| ) | ||
|
|
||
| waterlines_cube = create_waterlines(water_land_mask, simplify_tolerance=simplify_tolerance) | ||
|
|
||
| return build_process_dict( | ||
| process_graph=waterlines_cube, | ||
| process_id="waterlines_v1", | ||
| summary="Waterlines extracted from Sentinel-2 using NDWI.", | ||
| description=(Path(__file__).parent / "README.md").read_text(), | ||
| parameters=[ | ||
| spatial_extent, | ||
| temporal_extent, | ||
| max_cloud_coverage, | ||
| iterations, | ||
| ndwi_threshold, | ||
| simplify_tolerance, | ||
| ], | ||
| categories=["sentinel-2", "coastline", "waterlines"], | ||
| ) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| with open(Path(__file__).parent / "waterlines.json", "w") as f: | ||
| json.dump(generate(), f, indent=2) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have tested the steps used to build the process graph locally as follows and all works and produces expected results (an example of successful job
cdse-j-260421124051497da99717d1a929472b):However, when I try to execute the UDP from openEO WebEditor with the same spatial extent, temporal extend and other parameters all my runs fail indicating the the produced vector cube is empty (example of failing job:
cdse-j-2604211221524d169c4d4c082193cea6).@EmileSonneveld
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, there are a few differences in the parameters of those 2 jobs. Can you provide an example using the same?

Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I re-ran the UDP with more memory, and the job finished successfully:
Those are high values and need to be tweaked. There exists a feature where the UDP can specify what memory settings are needed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Emilie, I updated the comment with job ids that have the same spatial and temporal extends:


I since also push a commit to the code so the jobs run via UDP don't fail but they produce empty outputs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be a more minimal way to make the UDP work out-of-the-box:
Running a test ATM:
j-26042114324240f2be3f326f1dc0a736There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I can check further tomorrow...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@EmileSonneveld have you had a chance to invest this issues further?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not see the cause directly. I made a ticket here to be sure to not lose track of it:
Open-EO/openeo-geopyspark-driver#1655
Is there a deadline for this project?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
End of the month, so it is quite urgent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mileonai A temporary workaround can be to not parameterize
ndwi_threshold, till this ticket is fixed.Other parameters should work fine