Release 0.4.0

CHANGELOG.md

@@ -0,0 +1,57 @@
+# Changelog
+
+SensApp is pre-production: breaking changes happen between minor versions and are listed first.
+Releases are published from a `vX.Y.Z` tag, see [docs/RELEASING.md](docs/RELEASING.md).
+
+## 0.4.0
+
+### Breaking changes
+
+- **SensApp no longer runs open by default.** Without `SENSAPP_JWT_SECRET`, a run on a loopback address makes a
+  secret for that run and prints an admin token and a link to the UI that signs in; a run on any other address
+  (the container image, Helm) refuses to start. Set `SENSAPP_JWT_SECRET`, or `SENSAPP_AUTH_DISABLED=true` to
+  run open on purpose. See [docs/JWT_AUTH.md](docs/JWT_AUTH.md).
+- Tokens have a new `admin` scope, a `jti`, `iss` and `aud`. `SENSAPP_JWT_PREVIOUS_SECRETS` rotates the secret
+  without refusing the tokens of the previous one. `sensapp generate-secret` makes a secret.
+- **ClickHouse databases created by 0.3.0 are refused at startup** with a clear message: the sensor and unit
+  ids and the schema changed (stable ids, partitions pinned to UTC, no materialized views). There are no
+  deployments yet, so create a new database.
+- DuckDB stores timestamps with a microsecond precision, like the other backends.
+- RRDCached connection strings `rrdcached+unix` and `rrdcached+tcp` are routed to the RRDCached backend, which
+  now stores, reads and lists what it keeps.
+
+### Added
+
+- **A web UI** served by SensApp at `/ui/`: a data explorer (series by label, time window by drag, step and
+  aggregation, dark theme, shareable address), code snippets (Python SDK, curl) from the state of the explorer,
+  a Load Data tab (Python SDK, Telegraf, Prometheus, curl) and a Credentials tab to make tokens.
+- `POST /api/v1/admin/tokens`, to make tokens with an admin token. The `Token` scheme of InfluxDB clients is
+  accepted next to `Bearer`.
+- Cross-series aggregation in the database (`avg by (room) (temperature[24h])` on every backend), and
+  aggregated reads for Prometheus remote read hints on a whole selector at once.
+- Removal of duplicate samples with the vacuum operation (all SQL backends and DuckDB), and an opt-in
+  deduplication at ingestion (`SENSAPP_DEDUPLICATE_ON_INGEST`).
+- Write backpressure (`SENSAPP_HTTP_MAX_CONCURRENT_WRITES`, `503` with `Retry-After`), request ids in logs and
+  responses, and timeouts of their own for writes and for the vacuum.
+- A rewritten **BigQuery backend** on the current storage interface (experimental, integration suite passed
+  on a real dataset), and an RRDCached backend that works against a real daemon (experimental).
+- Python SDK: retries of overload answers, connection errors and timeouts, a timeout per operation.
+- Helm chart: secret handling for the authentication (made secret, `existingSecret`, `previousSecrets`).
+
+### Changed and fixed
+
+- ClickHouse: outages answer `503`, no more duplicated sensors or labels, stable ids, TLS with the system store,
+  writes spanning more than 100 months, bulk registration and bulk reads.
+- PostgreSQL and TimescaleDB: bulk registration of sensors and writes of numeric and string samples, integer
+  time buckets and plannable time bounds (aggregations 2 to 4 times faster), a retry after a deadlock on the
+  first write of new series, correct answers on compressed chunks, TimescaleDB 2.30.2 in CI.
+- SQLite: windows and the last sample use the index (`/last` from 54 ms to 0.5 ms), writes start with
+  `BEGIN IMMEDIATE`. DuckDB: writes with a handful of statements and microsecond timestamps.
+- Label regex matchers are anchored like Prometheus does. Remote read hints aggregate only when Prometheus gets
+  the right answer.
+- Read timeout 120 s on the server and 125 s in the SDK.
+- The binary is built on the library crate instead of compiling the tree twice.
+
+## 0.3.0
+
+First tagged release of this line, 30 September 2026.

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "sensapp"
-version = "0.3.0"
+version = "0.4.0"
 edition = "2024"
 description = "Sensor data ingestion, storage, and retrieval platform"
 license = "Apache-2.0"

TODO.md

@@ -13,7 +13,7 @@ The next phase is not to add more features. It is to make the existing system so
 Code, tests and docs for this exist (see `docs/CLICKHOUSE.md`, `done/clickhouse-*.md`). What remains is evidence from a real environment:
 
 - [ ] Verify the revised CI on GitHub: backend matrix, DuckDB and Docker smoke durations, cache sizes (`done/ci-build-time.md`)
-- [x] ClickHouse review: duplicated labels and sensors, unstable ids, 100-month insert limit, connection string, outage statuses, TLS with private CAs, bulk registration (`current_tasks/clickhouse-preproduction-readiness.md`)
+- [x] ClickHouse review: duplicated labels and sensors, unstable ids, 100-month insert limit, connection string, outage statuses, TLS with private CAs, bulk registration (`done/clickhouse-preproduction-readiness.md`)
 - [ ] Stage a deployment of the image and Helm chart against an external persistent ClickHouse
 - [ ] Exercise operations: restart SensApp, interrupt and recover ClickHouse, practise backup and restore on disposable data
 - [ ] Review dependency audit findings, align Cargo and chart versions, write the changelog, publish from a reviewed tag, smoke-test the published artifacts

charts/sensapp/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: sensapp
 description: Helm chart for deploying SensApp
 type: application
-version: 0.1.1
-appVersion: "0.3.0"
+version: 0.2.0
+appVersion: "0.4.0"
 keywords:
   - sensapp
   - time-series

charts/sensapp/README.md

@@ -7,7 +7,7 @@ The chart deploys SensApp only. It does not install or manage a database. Each S
 Published releases push the chart to GHCR as an OCI package. Install a specific chart version with:
 
 ```bash
-helm install sensapp oci://ghcr.io/sintef/charts/sensapp --version 0.1.1
+helm install sensapp oci://ghcr.io/sintef/charts/sensapp --version 0.2.0
 ```
 
 The chart version is the `version` in `Chart.yaml`, which is separate from the SensApp `appVersion`. Bump the chart version for each release so an existing OCI tag is not reused. GHCR creates new packages as private by default; an organization admin must make the chart package public for anonymous installs after its first publication.

docs/PREPRODUCTION_RELEASE_PLAN.md

@@ -5,8 +5,9 @@
 Use the clean `sensapp-sep-26` clone of upstream `main`. The older local
 checkouts have been reviewed in `done/local-work-reconciliation-september-2026.md`;
 none is a safe release base. The BigQuery backend was rewritten on the current
-storage interface (`docs/BIGQUERY.md`) and is outside the first ClickHouse
-release.
+storage interface (`docs/BIGQUERY.md`), and its integration suite passed on a
+real dataset on 4 October 2026. It stays outside the ClickHouse release as an
+experimental backend.
 
 ClickHouse already has migration, health, HTTP lifecycle, and query coverage.
 The remaining release risk is operational: running the packaged image against a
@@ -34,7 +35,7 @@ parser accepts a patched version itself.
 | 2. Stage a ClickHouse deployment | Deploy the built image and chart with external persistent ClickHouse; document configuration, credentials, resource limits, and logs for the actual environment. | 1–2 days |
 | 3. Exercise operations | Verify ingest, series listing, query, and export with representative data; restart SensApp; interrupt/recover ClickHouse; practice backup and restore on disposable data. Confirm readiness and data survival. | 1–2 days |
 | 4. Prepare the first release | Review dependency audit findings, version and changelog; align Cargo and chart versions; publish the GitHub Release from a reviewed tag; verify the pushed image and crate, then repeat a smoke check using those artifacts. | 0.5–2 days |
-| 5. Clear frontend security findings | Upgrade the OpenAPI generator, regenerate its client, verify API compatibility, and make the high-severity npm audit a release gate if the frontend ships with this release. | 1–3 days |
+| 5. Clear frontend security findings (done: npm audit is at zero and a gate in CI) | Upgrade the OpenAPI generator, regenerate its client, verify API compatibility, and make the high-severity npm audit a release gate if the frontend ships with this release. | 1–3 days |
 
 **Planning range: 5–11 engineer-days** for a frontend-inclusive pre-production
 release, assuming
@@ -48,5 +49,5 @@ backend-only release could defer step 5.
 Ship the ClickHouse-backed pre-production build when all four steps have
 evidence. Treat PostgreSQL, SQLite, TimescaleDB, DuckDB, and RRDCached as tested
 compatibility paths rather than equal deployment promises for this release.
-Keep BigQuery experimental until its integration suite has been run against a
-real isolated dataset.
+Keep BigQuery and RRDCached experimental: their integration suites pass, but
+they are research backends, not deployment promises (`docs/BACKENDS.md`).

docs/RELEASING.md

@@ -0,0 +1,43 @@
+# Releasing SensApp
+
+A release is a GitHub Release published for a `vX.Y.Z` tag on `main`. Publishing it is the trigger: a tag push alone
+only runs CI. What CI does with it is in [CI.md](CI.md).
+
+## Before
+
+- `main` is green on GitHub (backend matrix, frontend, Python SDK, audit, Helm, Docker smoke, live Prometheus).
+- `cargo audit` and `npm audit` findings reviewed ([PREPRODUCTION_RELEASE_PLAN.md](PREPRODUCTION_RELEASE_PLAN.md)).
+- Nothing in `current_tasks/` that should ship is still open.
+
+## Version
+
+All of these move together, on a branch merged into `main`:
+
+| Where | What |
+| --- | --- |
+| `Cargo.toml` and `Cargo.lock` | `version`, the crate |
+| `charts/sensapp/Chart.yaml` | `appVersion` (the same as the crate) and `version` (the chart's own, bumped on every release: the registry refuses to overwrite a chart version) |
+| `frontend/openapi.json` | the `info.version` of the document, rewritten by `UPDATE_OPENAPI=1 cargo test frontend_openapi_document` (a test fails when it is out of date) |
+| `CHANGELOG.md` | a section for the version, breaking changes first |
+
+CI refuses a release when the tag, the crate version and `appVersion` disagree. The Python SDK
+(`python/sensapp/pyproject.toml`) has its own version and is not published by the workflow.
+
+## Publish
+
+```bash
+git switch main && git pull
+git tag -a vX.Y.Z -m "SensApp X.Y.Z"
+git push origin vX.Y.Z
+gh release create vX.Y.Z --verify-tag --title "SensApp X.Y.Z" --notes-file <the changelog section>
+```
+
+The release job then pushes the image to `ghcr.io/sintef/sensapp` (`X.Y.Z`, `X.Y`, `sha-...`; `linux/amd64` and
+`linux/arm64`) and the chart to `oci://ghcr.io/sintef/charts`. The crate is not published to crates.io.
+
+## After
+
+- Pull the image by its version, and `helm install` the chart from the OCI registry.
+- Run the publish, query and `/ui/` smoke test against those artifacts.
+- First time only for a package: check its visibility in GitHub (a new GHCR package starts private).
+- Move the finished task files to `done/` and tick `TODO.md`.

done/arrow-export-timestamp-offset.md

@@ -48,8 +48,8 @@ No Arrow exporter test asserts a timestamp value. `test_data_helpers` builds sam
 
 ## Related
 
-- `current_tasks/python-sdk-boolean-query-params.md`
-- `current_tasks/series-limit-and-simplify-semantics.md`
+- `done/python-sdk-boolean-query-params.md`
+- `done/series-limit-and-simplify-semantics.md`
 
 ## Outcome
 

done/batched-sample-inserts.md

@@ -46,5 +46,5 @@ The release build is 4 times faster than debug, so most of the original 70 s was
 
 ## Related
 
-- `ideas/sample-deduplication-in-maintenance.md`: re-importing the same samples duplicates them, so benchmarks must use a fresh series each time.
+- `done/sample-deduplication-in-vacuum.md`: re-importing the same samples duplicates them, so benchmarks must use a fresh series each time.
 - `docs/HTTP_LIMITS.md` (64 MiB body limit) and `SENSAPP_HTTP_SERVER_TIMEOUT_SECONDS` (default 30) bound what one request can carry. The timeout is not documented in `HTTP_LIMITS.md` yet.

done/cross-series-aggregation-pushdown.md

@@ -2,7 +2,7 @@
 
 ## Goal
 
-`avg by (room) (temperature[24h])` over a few hundred series was impossible on every backend, although its output is tiny: the aggregation fetched the raw samples, so it was bound by the selector limits (256 series, 100 000 samples in total), and a larger `step` did not help. Raised by the review of the `storage-hardening-and-bulk-io` branch (see `current_tasks/pr-44-review-corrections.md`, item M6).
+`avg by (room) (temperature[24h])` over a few hundred series was impossible on every backend, although its output is tiny: the aggregation fetched the raw samples, so it was bound by the selector limits (256 series, 100 000 samples in total), and a larger `step` did not help. Raised by the review of the `storage-hardening-and-bulk-io` branch (see `done/pr-44-review-corrections.md`, item M6).
 
 ## Delivered