diff --git a/docs.json b/docs.json index 0e17676c..3e0883b7 100644 --- a/docs.json +++ b/docs.json @@ -346,7 +346,7 @@ "self-host/enterprise-on-prem", "self-host/self-host-lightdash-docker-compose", "self-host/update-lightdash", - "self-host/upgrading-lightdash-versioning", + "self-host/upgrade-safety", { "group": "NATS workers", "pages": [ @@ -742,6 +742,10 @@ { "source": "/guides/developer/agentic-coding", "destination": "/guides/developer/agent-skills" + }, + { + "source": "/self-host/upgrading-lightdash-versioning", + "destination": "/self-host/update-lightdash" } ], "api": { diff --git a/self-host/customize-deployment/configure-lightdash-to-use-an-external-database.mdx b/self-host/customize-deployment/configure-lightdash-to-use-an-external-database.mdx index 3656b4db..a4fc7c03 100644 --- a/self-host/customize-deployment/configure-lightdash-to-use-an-external-database.mdx +++ b/self-host/customize-deployment/configure-lightdash-to-use-an-external-database.mdx @@ -45,4 +45,17 @@ Lightdash requires the following extensions to be installed on the database: ## Migrations -Migrations are ran automatically on starting the Lightdash server or workers. When upgrading Lightdash, migrations will be ran automatically. If migrations fail due to a pg\_lock error, check for a table called`knex_migrations_lock` to manually release the lock. +Migrations are ran automatically on starting the Lightdash server or workers. When upgrading Lightdash, migrations will be ran automatically. + +If a migration gets stuck β€” for example after an interrupted deploy: + +- **On Lightdash `1.123.0` and later**, an interrupted migration's lock expires on its own and another process takes over automatically. To inspect migration state, or release a lock that is genuinely stuck, run the built-in `migrate` command inside a Lightdash container (`kubectl exec` / `docker compose exec`): + + ```bash + pnpm -F backend migrate-production status + pnpm -F backend migrate-production unlock --actor "" + ``` + + Do **not** edit the `knex_migrations_lock` table manually on these versions β€” it can release a lock that a live migration legitimately holds. + +- **On versions before `1.123.0`**, if migrations fail due to a pg\_lock error, check for a table called `knex_migrations_lock` to manually release the lock. diff --git a/self-host/production-deployment-checklist.mdx b/self-host/production-deployment-checklist.mdx index 9c1a4c76..382578eb 100644 --- a/self-host/production-deployment-checklist.mdx +++ b/self-host/production-deployment-checklist.mdx @@ -158,7 +158,7 @@ The chart auto-wires `HEADLESS_BROWSER_HOST` / `HEADLESS_BROWSER_PORT`. Remember ## Upgrades and operations -Upgrade mechanics are in [update Lightdash](/self-host/update-lightdash) and the version policy in [versioning](/self-host/upgrading-lightdash-versioning). Operational best practice: +Upgrade mechanics and the version policy are in [updating Lightdash](/self-host/update-lightdash). Operational best practice: - **Pin `image.tag`** and upgrade deliberately. There is no LTS tag. Lightdash versioning: patch = routine, minor = check release notes (may be backwards-incompatible), major = read the upgrade guide - **Upgrade cadence:** at least monthly. Lightdash ships continuously; falling many minor versions behind makes the eventual migration jump riskier @@ -248,7 +248,7 @@ migrationJob: enabled: true ``` -(If a deploy is ever interrupted mid-migration, the lock lives in the `knex_migrations_lock` table.) +(If a deploy is ever interrupted mid-migration on Lightdash `1.123.0` or later, inspect with `migrate status` and use `migrate unlock --actor ` as the attributed escape hatch β€” see [Migrations](/self-host/customize-deployment/configure-lightdash-to-use-an-external-database#migrations). Manually clearing the `knex_migrations_lock` table is only for versions before `1.123.0`.) ## Object storage diff --git a/self-host/update-lightdash.mdx b/self-host/update-lightdash.mdx index d5ef8066..c0fb45bc 100644 --- a/self-host/update-lightdash.mdx +++ b/self-host/update-lightdash.mdx @@ -1,9 +1,28 @@ --- -title: "Updating Lightdash to the latest version" +title: "Updating Lightdash" sidebarTitle: Updating Lightdash +description: "How Lightdash versioning works and how to update your self-hosted deployment." boost: 0.001 --- + +πŸ›  This page is for engineering teams self-hosting their own Lightdash instance. If you're on Lightdash Cloud, upgrades are handled for you automatically. + + +Lightdash uses semantic versioning in the form `major.minor.patch`. Each segment signals how significant the changes are and what level of care you should take when upgrading. + +## How Lightdash versioning works + +- **Patch versions** (`x.x.PATCH`) are small fixes. These are safe to pick up as part of routine upgrades. +- **Minor versions** (`x.MINOR.x`) introduce new features or bigger changes that are backwards incompatible. Review the release notes before upgrading. +- **Major versions** (`MAJOR.x.x`) introduce breaking changes. You **must** read the upgrade guides below before upgrading. + +Major version bumps can happen because the API or the self-hosted architecture has changed substantially. Expect to make configuration, infrastructure, or integration changes when moving between major versions. + + +Before upgrading a production deployment, check the [release-safety signal](/self-host/upgrade-safety) for your upgrade span β€” it tells you whether a rolling update is safe and whether there are required stops on the way. + + ## Local deployments If you're running Lightdash on your own laptop using Docker, you just need to instruct Docker to pull the latest version of Lightdash: @@ -28,3 +47,9 @@ helm upgrade -f values.yml lightdash lightdash/lightdash Your Lightdash CLI version should match the version of Lightdash you're running. After upgrading your deployment, make sure anyone using the CLI also upgrades it β€” otherwise commands like `lightdash preview` and `lightdash deploy` may behave unexpectedly. See [Upgrading your Lightdash CLI](/guides/cli/how-to-upgrade-cli) for instructions. + +## Upgrade guides + +### Upgrading from Version 0 to Version 1 + +_Coming soon._ diff --git a/self-host/upgrade-safety.mdx b/self-host/upgrade-safety.mdx new file mode 100644 index 00000000..24f85523 --- /dev/null +++ b/self-host/upgrade-safety.mdx @@ -0,0 +1,264 @@ +--- +title: "Upgrade safety and the release-safety artifact" +sidebarTitle: "Upgrade safety" +description: "Every Lightdash release publishes a machine-readable safety signal. Learn how to read it, check an upgrade path, and pick the right deployment strategy." +--- + + +πŸ›  This page is for engineering teams self-hosting their own Lightdash instance. If you're on Lightdash Cloud, upgrades are handled for you automatically. + + +Every Lightdash release publishes a machine-readable answer to the question: **"is upgrading from version X to version Y safe to roll, and are there required stops on the way?"** + +Two documents carry that answer: + +| Document | Scope | Where to get it | +| --- | --- | --- | +| `release-safety.json` | One release: the step from `previousVersion` to `version` | Attached to every [GitHub release](https://github.com/lightdash/lightdash/releases) as an asset, and committed at the repository root of the release tag (so it ships inside the Docker image) | +| `release-safety-index.json` | Every release back to the index floor (~12 months) | Committed at the repository root: [`raw.githubusercontent.com/lightdash/lightdash/main/release-safety-index.json`](https://raw.githubusercontent.com/lightdash/lightdash/main/release-safety-index.json) | + +Both are public β€” no account or license required β€” and both are designed to be consumed by your own automation (CI checks, GitOps hooks) as well as by a human before an upgrade. + +## The one rule that matters + + +**Only an explicit `rollingUpdateSafe: true` means a rolling update is advised. `unknown` means NOT safe.** + +`unknown` is what the pipeline publishes when it could not *prove* safety β€” a degraded check, an unclassifiable change, an old release nobody vouched for. Treat every `unknown` exactly like `false`: use the `Recreate` strategy. Never write automation that treats "not false" as safe. + + +Everything else on this page is detail on top of that rule. + +## How to read the signal + +1. **Verdicts compose by AND across a span.** Upgrading across several releases (say `1.111.0` β†’ `1.115.0`) is rolling-safe only if *every* release in the span is `rollingUpdateSafe: true`. One `false` or `unknown` anywhere in the span means the whole upgrade needs `Recreate`. The cumulative index exists so you can evaluate a whole span in one fetch. +2. **Required stops are hard requirements.** If any release in your span lists a version in `requiredStops`, you must upgrade *to that version first*, let it run its migrations, and then continue. Skipping a stop runs later migrations against a schema that's missing the prerequisites they assume β€” which is how databases end up in states that need manual repair. +3. **`minPreviousVersion` floors direct jumps.** Each release states the oldest version you may upgrade from directly. If you're on something older, upgrade to an intermediate version first. +4. **Versions older than the index floor route through the floor.** The index reaches back to `0.1893.0`. If you're running something older, first upgrade to `0.1893.0` (the floor entry is marked `syntheticRequiredStop: true` for exactly this reason), then evaluate the rest of your span normally. In practice: upgrading from `0.1050.0` to `1.121.0` means `0.1050.0` β†’ `0.1893.0` β†’ whatever the span check from there tells you. +5. **Backfilled entries are conservative.** Entries marked `backfilled: true` were generated retroactively β€” nobody vouched for those releases at the time they shipped. Expect `unknown` verdicts there, and plan a `Recreate` upgrade for any span that crosses them. + +## Checking an upgrade span + +The answer depends on exactly three inputs: the version you're running, the version you're targeting, and every release in between. The [Lightdash CLI](/guides/cli/how-to-install-the-lightdash-cli) (version `1.126.0` or later) answers it in one command: + +```bash +lightdash upgrade-check --from 1.111.0 --to 1.115.0 +``` + +``` +Release safety check: 1.111.0 -> 1.115.0 +Direction: upgrade +Verdict: UNSAFE +Unsafe rolling-update safety: 1.112.0 +Unknown rolling-update safety: 1.113.0, 1.113.1, 1.114.0, 1.115.0 +Minimum previous version: 1.111.0 +Covered releases: 1.112.0, 1.113.0, 1.113.1, 1.114.0, 1.115.0 +``` + +The command fetches the public index and applies the span rules from this page β€” AND-composition, required stops, minimum versions β€” locally. It needs **no Lightdash login, token, or instance access**, so it runs anywhere, including CI against an air-gapped production instance. Here the verdict is `UNSAFE` because the span crosses one release known not to be rolling-safe and several unproven ones β€” so **deploy with `Recreate`**: stop the old version, then start the new one. + +### Using it as a CI gate + +The exit code carries the verdict: **exit `0` only when the whole span is proven safe to roll**. Any other outcome β€” an `unknown` or `false` release in the span, a required stop, a version the index doesn't cover, a fetch failure β€” exits non-zero. That fail-closed contract means you can use the bare command as a pipeline gate: + +```bash +lightdash upgrade-check --from "$CURRENT_VERSION" --to "$TARGET_VERSION" +``` + +For automation that needs the detail, `--json` prints a stable machine-readable object (`fromVersion`, `toVersion`, `direction`, `safe`, `verdict`, `requiredStops`, `minPreviousVersion`, `coveredVersions`, `missingRanges`): + +```bash +lightdash upgrade-check --from 1.111.0 --to 1.115.0 --json | jq .safe +``` + +## What the verdict means for your deployment + +| Verdict for the span | Kubernetes / Helm | docker compose | +| --- | --- | --- | +| Every release `rollingUpdateSafe: true` | `RollingUpdate` strategy is advised β€” old and new pods may serve side by side during rollout | `docker compose up -d` with the new tag | +| Anything `false` or `unknown` | Use `Recreate` (or scale to zero before switching the image tag) | `docker compose down`, update the tag, `docker compose up -d` | + +`Recreate` means a short window of downtime, but it guarantees old and new code never run against the database at the same time. That matters because Lightdash runs its database migrations automatically at startup: with `Recreate`, every old replica has stopped before the new version boots and migrates; with a rolling update, old replicas keep serving against a schema that is changing underneath them β€” which is exactly what the verdict certifies as safe or not. + +## Worked example: reading a release artifact + +This is the real artifact attached to release `1.121.0`: + +```json +{ + "schemaVersion": "2", + "version": "1.121.0", + "previousVersion": "1.120.1", + "releaseDate": "2026-08-11T11:08:39.052Z", + "migrations": { + "present": false, + "count": 0, + "coreCount": 0, + "eeCount": 0, + "files": [] + }, + "compatibility": { + "rollingUpdateSafe": true, + "recommendedStrategy": "RollingUpdate" + }, + "api": { + "rest": { + "checked": true, + "breaking": false, + "changes": [], + "breakingCount": 0, + "advisories": [], + "advisoryCount": 0 + }, + "mcp": { + "checked": true, + "breaking": false, + "changes": [], + "breakingCount": 0, + "advisories": [], + "advisoryCount": 0 + } + }, + "config": { + "checked": true, + "breaking": false, + "changes": [] + }, + "upgrade": { + "minPreviousVersion": "1.111.0", + "requiredStops": [] + }, + "declaredBreaks": [] +} +``` + +Reading it top to bottom: + +- **No migrations ship in this release** (`migrations.present: false`), so there is no schema change to coordinate. +- **`rollingUpdateSafe: true`** β€” this is the explicit green verdict. Upgrading from `1.120.1` (the `previousVersion`) to `1.121.0` can be a rolling update. +- **All three change surfaces were checked and came back clean**: no breaking REST API changes, no breaking MCP tool changes (not even non-breaking advisories on either), and no environment variable removals, renames, or default changes. +- **You may jump here directly from `1.111.0` or newer** (`upgrade.minPreviousVersion`), with no required stops on the way β€” but remember the span rule: the jump is only *rolling*-safe if every release in between is, which you check against the index, not this single artifact. +- **No engineer declared a breaking change** (`declaredBreaks` is empty). + + +## Field reference: `release-safety.json` + +Fields that can't prove safety report the string `"unknown"` instead of a boolean β€” and per the rule above, you consume `"unknown"` as unsafe. + +### Top level + +| Field | Type | Meaning | +| --- | --- | --- | +| `schemaVersion` | string | Always `"2"` for this format | +| `version` | string | The release this artifact describes | +| `previousVersion` | string \| null | The release immediately before it β€” the step this artifact covers | +| `releaseDate` | ISO date-time | When the release was published | + +### `migrations` + +| Field | Type | Meaning | +| --- | --- | --- | +| `present` | boolean \| `"unknown"` | Whether this release ships database migrations | +| `count`, `coreCount`, `eeCount` | integer | How many, split between core and Enterprise migrations | +| `files[].name` | string | Migration file name | +| `files[].edition` | `core` \| `ee` | Which edition ships it | +| `files[].tables` | string[] | Tables the migration touches | +| `files[].heaviness` | object | Three verdicts β€” `locksTable`, `rewritesTable`, `scansTable` β€” each boolean or `"unknown"`, flagging migrations that may lock, rewrite, or scan whole tables (and therefore take time or block writes on large databases) | + +### `compatibility` + +| Field | Type | Meaning | +| --- | --- | --- | +| `rollingUpdateSafe` | boolean \| `"unknown"` | **The verdict.** Only `true` means a rolling update is advised | +| `recommendedStrategy` | `RollingUpdate` \| `Recreate` | A convenience rendering of the verdict. Gate your automation on `rollingUpdateSafe`, not on this field | + +### `api` and `config` + +| Field | Type | Meaning | +| --- | --- | --- | +| `api.rest`, `api.mcp` | object | One entry per API surface β€” REST endpoints and MCP tools β€” with the fields below | +| `…​.checked` | boolean | Whether the diff for that surface actually ran | +| `…​.breaking` | boolean \| `"unknown"` | Whether the surface has consumer-breaking changes | +| `…​.changes[]` | string[] | The breaking changes, one human-readable line each (e.g. `GET /api/v1/… β€” response property removed`). The list is capped with an explicit overflow line; `breakingCount` always carries the full total | +| `…​.breakingCount` | integer | Uncapped total number of breaking changes | +| `…​.advisories[]` | string[] | Non-breaking advisory notes (for example, a response enum gaining a new value) β€” worth a scan if you consume the API strictly, but they never affect the `breaking` verdict. Capped like `changes` | +| `…​.advisoryCount` | integer | Uncapped total number of advisory notes | +| `config.checked` | boolean | Whether the environment variable surface was diffed | +| `config.breaking` | boolean \| `"unknown"` | Whether any config change is breaking | +| `config.changes[]` | object | Each change is `removed` (with `previousDefault`), `renamed` (with `previousName` and `defaultValue`), or `defaultChanged` (with `previousDefault` and `defaultValue`) β€” scan this list for any environment variable you set explicitly | + +### `upgrade` and `declaredBreaks` + +| Field | Type | Meaning | +| --- | --- | --- | +| `upgrade.minPreviousVersion` | string \| null | Oldest version you may upgrade from directly | +| `upgrade.requiredStops` | string[] | Versions you must pass through on the way to this release | +| `declaredBreaks[]` | object | Breaking changes declared by engineers in the source (`file`, `line`, `reason`, and whether the break makes this release a `requiredStop`) | + +## Field reference: `release-safety-index.json` + +The index is a derived summary β€” one entry per release, oldest first. Per-release artifacts remain the source of truth. + +| Field | Type | Meaning | +| --- | --- | --- | +| `backfillFloorVersion` | string | The oldest release the index covers (`0.1893.0`) | +| `entries[].version`, `previousVersion`, `releaseDate` | | Same meaning as the per-release artifact | +| `entries[].rollingUpdateSafe` | boolean \| `"unknown"` | The verdict for that single release step | +| `entries[].requiredStops`, `minPreviousVersion` | | Same meaning as the per-release artifact | +| `entries[].backfilled` | boolean | `true` when the entry was generated retroactively β€” expect conservative (`unknown`) verdicts | +| `entries[].syntheticRequiredStop` | boolean | `true` only on the floor entry: versions older than the floor must upgrade to it first | + +## Before you upgrade: checklist + +- Read the [release notes](https://github.com/lightdash/lightdash/releases) for every release in your span. +- Run the [span check](#checking-an-upgrade-span) β€” note the verdict, required stops, and `minPreviousVersion`. +- If the span verdict is anything other than `true`, plan a maintenance window and use `Recreate`. +- If migrations are present, check their `heaviness` flags in the per-release artifacts β€” table rewrites and scans on large tables take time. +- Scan `config.changes` for environment variables you set explicitly. +- Confirm your database backup (and [point-in-time recovery](/self-host/production-deployment-checklist), if configured) is current. +- After upgrading, [update the Lightdash CLI](/guides/cli/how-to-upgrade-cli) to match. + +## Appendix: checking a span without the CLI + +In environments where you can't run the CLI (no Node.js, or a fully air-gapped pipeline that mirrors the index), you can evaluate a span directly from the index with `curl` and `jq`. Check every release between your current version (exclusive) and your target (inclusive): + +```bash +FROM=1.111.0 # the version you are running +TO=1.115.0 # the version you want + +curl -fsSL https://raw.githubusercontent.com/lightdash/lightdash/main/release-safety-index.json | +jq --arg from "$FROM" --arg to "$TO" ' + .entries + | (map(.version == $from) | index(true)) as $i + | (map(.version == $to) | index(true)) as $j + | if $i == null or $j == null then + error("version not found in index β€” treat the span as unsafe") + else + .[$i+1:$j+1] + | { + releasesInSpan: length, + rollingUpdateSafe: (map(.rollingUpdateSafe == true) | all), + requiredStops: (map(.requiredStops[]) | unique), + minPreviousVersionForTarget: (last.minPreviousVersion) + } + end' +``` + +Against the live index, that span answers: + +```json +{ + "releasesInSpan": 5, + "rollingUpdateSafe": false, + "requiredStops": [], + "minPreviousVersionForTarget": "1.111.0" +} +``` + +Read it in order: + +1. `requiredStops` is empty β€” no mandatory intermediate version, so a direct jump is allowed… +2. …provided your current version is at or above `minPreviousVersionForTarget`. Here `1.111.0` β‰₯ `1.111.0`, so the jump is permitted. +3. `rollingUpdateSafe` is `false` β€” the span crosses at least one release that is not proven rolling-safe, so deploy with `Recreate`. + +If your current version isn't in the index at all, the command errors on purpose. That's the fail-safe default: a version the index can't see is a span it can't vouch for β€” the same rule `upgrade-check` applies. diff --git a/self-host/upgrading-lightdash-versioning.mdx b/self-host/upgrading-lightdash-versioning.mdx deleted file mode 100644 index 4a6e1917..00000000 --- a/self-host/upgrading-lightdash-versioning.mdx +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: "Upgrading Lightdash (versioning)" -sidebarTitle: "Upgrading Lightdash (versioning)" -description: "How Lightdash versioning works and how to upgrade between versions." -boost: 0.001 ---- - - -πŸ›  This page is for engineering teams self-hosting their own Lightdash instance. If you're on Lightdash Cloud, upgrades are handled for you automatically. - - -Lightdash uses semantic versioning in the form `major.minor.patch`. Each segment signals how significant the changes are and what level of care you should take when upgrading. - -## How Lightdash versioning works - -- **Patch versions** (`x.x.PATCH`) are small fixes. These are safe to pick up as part of routine upgrades. -- **Minor versions** (`x.MINOR.x`) introduce new features or bigger changes that are backwards incompatible. Review the release notes before upgrading. -- **Major versions** (`MAJOR.x.x`) introduce breaking changes. You **must** read the upgrade guides below before upgrading. - -Major version bumps can happen because the API or the self-hosted architecture has changed substantially. Expect to make configuration, infrastructure, or integration changes when moving between major versions. - -## Upgrade guides - -### Upgrading from Version 0 to Version 1 - -_Coming soon._