Skip to content

Add measured public-data importer case with real costs and limits - #7

Merged
dKosarevsky merged 1 commit into
mainfrom
case/public-countries-importer
Oct 10, 2026
Merged

dKosarevsky merged 1 commit into
mainfrom
case/public-countries-importer

Conversation

@dKosarevsky

Copy link
Copy Markdown
Contributor

Summary

  • Reproduce sqlite-utils 4.2.1 flattened-import late-column failure using pinned, SHA-256-verified mledoze/countries data (ODbL), with no invented records.
  • Keep dataset downloading explicit; generated JSONL carries an ODbL notice. No data vendoring, runtime dependency changes, or release bump.
  • Actual measured reduction: 250 → 176 records, 631,420 → 438,177 bytes; 1683/1683 full-input structural features retained. Check all 855 column definitions, retained values/storage classes, and an independent expected late-column value.
  • Compare first-N honestly: 176 initial rows cover 1411/1683 using full-input vocabulary and reproduce the same early failure. Selection is not needed to find this particular failure.
  • Record all five timing samples, first observed/subsequent medians, selection overhead, order/batch sensitivity. Generating a fixture for a one-off import is slower. No historical bug, external adoption, or universal speed claims.

Verification

  • 125 core unittest tests passed, including 10 offline recipe guards.
  • 14 optional real sqlite-utils/dlt integration tests passed.
  • Public-data experiment ran five repetitions against fresh SQLite databases; every success/failure assertion passed.
  • compileall and git diff --check passed.
  • Independent review verified full-vocabulary baseline coverage against pinned per-row ShapeWitness features.

The public-data command is opt-in; existing CI exercises the offline guards. See docs/recipes/public-countries.md and benchmarks/countries-importer.json. External maintainers were not contacted.

@dKosarevsky
dKosarevsky marked this pull request as ready for review October 10, 2026 03:52
@dKosarevsky
dKosarevsky merged commit a891256 into main Oct 10, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant