fix: read plain tabular JSON and write null-geometry GeoDataFrames - #119
Open
fredj wants to merge 5 commits into
Open
fix: read plain tabular JSON and write null-geometry GeoDataFrames#119fredj wants to merge 5 commits into
fredj wants to merge 5 commits into
Conversation
_read_file_encoded reads .json files as a table directly via pandas,
without touching the .geojson code path.
write_data_to_postgis now writes a GeoDataFrame with an entirely-null
geometry column as a plain table instead of crashing in to_postgis()
("No valid geometries in the data.").
The extension-based guess at submission time can be wrong for these two. get_staging_metadata now checks the staging table for a "geom" column (same check read_data_from_postgis already uses) and reports what the data actually turned out to be. Other file types are untouched.
Moved the repeated six @patch decorators into a staging_metadata_deps fixture in a new conftest.py, shared by test_staging_database.py and test_staging_api.py.
fredj
force-pushed
the
json-geojson-handling
branch
from
August 28, 2026 15:37
cccdd28 to
6f50bf5
Compare
fredj
marked this pull request as ready for review
August 31, 2026 06:14
fredj
marked this pull request as draft
August 31, 2026 07:35
…file Checks every feature's geometry once, via a separate geometry-only read of the whole file, so chunking doesn't affect the verdict. OGC WFS/OAPIF services keep their own null-geometry handling, unaffected.
Adds the jq PyPI package (bundles libjq, no system dependency) and runs raw JSON text through it before parsing, seeded with the fixed identity filter "." for now. This is the seam a future editable filter plugs into. Frontend shows a disabled "JQ filter" field prefilled with "." next to the dataset title, for json sources only. No editing or persistence yet.
fredj
marked this pull request as ready for review
September 1, 2026 12:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detects and correctly handles plain tabular JSON and non-spatial GeoJSON during ingestion:
_read_file_encodedreads.jsonfiles as a table directly via pandas, without touching.geojsonhandling..geojsonfile/URL source whose geometry is null on every feature is now rejected with a clear error, instead of being silently imported as a non-spatial table — this usually signals a malformed or unintended export. The check reads the geometry column once for the whole file, so chunking doesn't affect the verdict. OGC WFS/OAPIF services keep their own deliberate null-geometry-to-tabular handling (some feature services legitimately return "geometry": null for non-spatial layers), and other geospatial formats (Shapefile, GeoPackage, database sources) keep the existing fallback inwrite_data_to_postgis.get_staging_metadatareports the actualfile_type(json vs geojson) based on the resulting staging table, not just the source extension.Also dedupes repeated test setup into shared fixtures/conftest.