Conversation
interactions schema
f-necas
changed the title
pmauduit
left a comment
pmauduit
left a comment
There was a problem hiding this comment.
Thanks for improving the doc, here are some feedback reviewing it.
Globally, I think that it is sometimes very verbose, and we could simplify some of the paragraphs to a more concise version of them, making it a bit "less painful" to read.
| | Database | Holds | Written/read by | | ||
| |---|---|---| | ||
| | **geOrchestra DB** (`datafeeder` schema) | Datafeeder's own bookkeeping: `IntegrityLink` records, authorization rules, schedules | Backend (read/write), Airflow (read-only, to find datasets scheduled for a recurring re-run) | | ||
| | **Data DB** (`data` / `staging` schemas) | The actual dataset content: staging tables during ingestion, then the final published tables | Backend, Airflow DAGs, GeoServer (via JNDI, to serve the published layers) | |
There was a problem hiding this comment.
I think that it would be interesting for people to have an introduction of what airflow provides in the context of the DFv2, as even after the setup from last week, I still don't know what a DAG is exactly.
| - **The data DB** is isolated because it has a completely different profile: it can grow arbitrarily large (one | ||
| table per published dataset), and — unlike every other piece of geOrchestra state — **GeoServer reads from it | ||
| directly** (via JNDI), not through an API. Keeping it separate lets you size, back up, and scale it | ||
| independently of platform metadata, and means GeoServer only ever needs credentials scoped to dataset content, | ||
| never to the rest of the platform (users, orgs, LDAP-adjacent tables, etc.). |
There was a problem hiding this comment.
I think this paragraph is a bit verbose: just telling that it is meant to store the geodata and is expected to be accessible to GS via a JNDI resource is enough IMHO.
| - **The Airflow metadata DB** is owned end-to-end by Airflow itself: its schema is created and migrated by Airflow's | ||
| own release cycle, independent of Datafeeder/geOrchestra upgrades. It also churns heavily (a row per task | ||
| instance/log — see the `airflowDbCleanup` CronJob in the Helm chart, which periodically purges old rows) and | ||
| stores encrypted credentials (Airflow Connections, encrypted with the Fernet key). Keeping it separate avoids | ||
| coupling its lifecycle/retention policy to the rest of the platform, and limits the blast radius of those stored | ||
| credentials. |
There was a problem hiding this comment.
same comment as the previous one, just saying that airflow manages it (and yes you could reuse an existing db previously mentioned) but it would lead you with a gazillion of tables, that will "pollute" a geOrchestra env.
| Alembic (the backend's migration tool) does **not** bootstrap the schema from scratch: its baseline migration is a | ||
| no-op, it only applies changes on top of an already-existing schema (see | ||
| [`001_baseline.py`](https://github.com/georchestra/datafeeder/blob/main/apps/backend/alembic/versions/001_baseline.py)). | ||
| The Docker Compose setup runs the init scripts below automatically before the backend starts; any other deployment | ||
| (Kubernetes included) must run them once, manually, against the target instance(s) before the first install: |
There was a problem hiding this comment.
I don't think it is in the scope of this documentation to explain what alembin does and/or does not (yet TIL something about it). I'd have phrased it: "Alembic is being used for data migration, but an init-script is responsible for the initial bootstrap of the db".
| [Component interactions](../presentation.md#component-interactions) for a diagram of how the pieces below talk to | ||
| each other. |
| `<release>-database-data` secret for you. If you leave it unset, the backend pod will fail to start | ||
| (`CreateContainerConfigError`, missing secret `<release>-database-data`). | ||
|
|
||
| Either point `backend.data_db.existingSecret` at an existing secret (e.g. the same one as |
| |---|---|---| | ||
| | `AIRFLOW_CONN_DATA_PG` | **Yes** | The data database (staging/final tables) | | ||
| | `AIRFLOW_CONN_DATAFEEDER_PG` | **Yes** | The georchestra/`datafeeder` schema database | | ||
| | `AIRFLOW_CONN_SOURCE_DB_1` | Only if using the **Database** source type | See [adding a source database](../configuration/source_database.md) | |
There was a problem hiding this comment.
I am not sure I needed it for my deployment last week, I used the 3 following ones:
- AIRFLOW__DATABASE__SQL_ALCHEMY_CONN (airflow)
- AIRFLOW_CONN_DATA_PG (my "source" database ?)
- AIRFLOW_CONN_DATAFEEDER_PG (the geOrchestra db)
I saw the DB_1 from the setup I took inspiration from, but was it actually useful ?
| for the schemas/extensions/scripts that must be applied to each **before the first install** (Alembic's baseline | ||
| migration assumes the schema already exists — it doesn't bootstrap it). | ||
|
|
||
| ### The chart doesn't create a secret for the "data" database |
|
|
||
|  | ||
|
|
||
| !!! info "How the pieces talk to each other" |
There was a problem hiding this comment.
I'd go for "interact with each other", but it is purely personal taste.
| docs-serve: ## Serve the documentation locally with live-reload (http://127.0.0.1:8000) | ||
| uv run --with-requirements mkdocs_requirements.txt mkdocs serve | ||
|
|
||
| docs-build: ## Build the static documentation site into ./site | ||
| uv run --with-requirements mkdocs_requirements.txt mkdocs build | ||
|
|
||
| .PHONY: default help install-python fix-and-check-all-python build-libs up up-no-airflow down down-v run-backend docs-serve docs-build |
There was a problem hiding this comment.
unrelated but probably useful, I'd expect that docs-serve would block on having the server running though ?
|
|
||
| | Target | Script | Creates | | ||
| |---|---|---| | ||
| | geOrchestra DB | [`docker/datafeeder-init.sql`](https://github.com/georchestra/datafeeder/blob/main/docker/datafeeder-init.sql) (same content as [`georchestra/migrations/26.0/db_migration_new_datafeeder.sql`](https://github.com/georchestra/georchestra/blob/master/migrations/26.0/db_migration_new_datafeeder.sql), for platforms upgrading from 25 to 26) | `datafeeder` schema, `pgcrypto` extension, `staging` schema/grants | |
There was a problem hiding this comment.
The mentioned script in geor/geor creates the data & staging schemas which are only used in the "data" database. Outside of the scope of this PR probably, but should be removed there to avoid introducing more confusion.
|
Discussed this morning, it would be worth, either to add an extra object (NetworkPolicy) into the helm chart or explaining how to secure the feature which allows to fetch remote datasets (in order to avoid SSRF security issues). |
2 participants
databases explanation and interactions schema