Currently, write_manifest_to_icechunk fills in datetime.now(utc)+1s for the last_updated_at field of chunk references. Icechunk will refuse to serve the chunk if the source was modified after this time. This leaves open the possibility that a file changes in between the parsing and the writing, but Icechunk is left unaware and serves erroneous byte ranges. While not super likely, I believe this is a possibility for long-running coordination pipelines and should be addressed. I think last_updated_at should refer to target of the chunk reference, rather than the chunk reference in the Icechunk store. We could do this using S3 ETag information where available, and secondarily the timestamp at parsing. This would require a change to the manifest-schema and likely parsers 😞
cc @TomNicholas @sharkinsspatial
Currently,
write_manifest_to_icechunkfills indatetime.now(utc)+1sfor thelast_updated_atfield of chunk references. Icechunk will refuse to serve the chunk if the source was modified after this time. This leaves open the possibility that a file changes in between the parsing and the writing, but Icechunk is left unaware and serves erroneous byte ranges. While not super likely, I believe this is a possibility for long-running coordination pipelines and should be addressed. I thinklast_updated_atshould refer to target of the chunk reference, rather than the chunk reference in the Icechunk store. We could do this using S3 ETag information where available, and secondarily the timestamp at parsing. This would require a change to the manifest-schema and likely parsers 😞cc @TomNicholas @sharkinsspatial