Skip to content

pg_mooncake 0.3.0 - #207

Open
zhousun wants to merge 18 commits into
mainfrom
v0.3-dev
Open

zhousun wants to merge 18 commits into
mainfrom
v0.3-dev

Conversation

@zhousun

@zhousun zhousun commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

pg_mooncake 0.3.0

This brings the 0.3.0 release candidate (v0.3-dev at 1719f31) to main. 0.3.0 is the first release of pg_mooncake built on moonlink: a columnstore mirror of a regular Postgres table, kept in Iceberg through logical replication and queried in DuckDB through pg_duckdb. It replaces the unreleased 0.2 line on main; 0.1.x was a different design and is not upgraded in place (see the README's Coming from 0.1.x).

Start here: CHANGELOG.md (overview, usage, behaviour and guarantees, performance, known limitations) and README.md (install, Docker, backups and standbys, security).

What changed against main

  • moonlink moves from the 0.2 snapshot to its 0.3 release commit (37b040f9, Mooncake-Labs/moonlink v0.3-dev): service features pg_mooncake doesn't use are removed (REST, Kafka, OTEL, its own WAL, Delta, REST/Glue catalogs, GCS), and the CDC, recovery, storage and read paths are hardened. Each change is a reviewed moonlink PR, linked from the CHANGELOG.
  • pg_duckdb is the Mooncake-Labs fork tag v1.0.0-mooncake.1 (DuckDB v1.4.1 plus the upstream TimeZone-hook crash fix).
  • pg_mooncake itself: robust DROP of mirrors, refusal of writes to mirrors, mooncake.data_directory (lets pg_basebackup and standbys work), moonlink starting only after recovery, safer error messages, worker restart and signal handling.
  • Tests and CI: make test (pg_regress) on PostgreSQL 14-18; an end-to-end gate (scripts/e2e-smoke.sh: crash/restart exactness, DDL, roles, config reload, base backups, standbys) on 17 and 18; a differential suite driven by the PostgreSQL regression tests (tests/mirror_diff, advisory); a Docker build-and-run job.
  • Docs: CHANGELOG written as a first release, README, AGENTS.md (contributor guide), docs/ARCHITECTURE.md, TESTING.md.

How it was validated

  • Every change landed on v0.3-dev as its own reviewed PR (Mooncake-Labs/pg_mooncake PRs 188-206) with green CI.
  • e2e gate: GATE PASS on PostgreSQL 17 and 18, no failures and no skips; restarts and crashes keep mirrors exact.
  • mirror_diff full pass on PostgreSQL 17: 2104 checks, 0 FAIL (every difference is a documented limitation).
  • Performance on an 8-vCPU NVMe machine: ~1.2x write amplification under CDC, apply lag p50 272 ms / p95 495 ms, ~0.5 s cross-session freshness, ~19 ms in-memory scans, ~750k rows/s initial copy (CHANGELOG Performance).

Still open before tagging

  • Release markers: the release date, publishing the Docker image, the PostgreSQL version matrix note, and src_uri (scripts/check-release-markers.sh lists them).
  • moonlink: merge its v0.3-dev into its main preserving SHAs and tag v0.3.0 at the pinned commit, so the submodule pin stays reachable.

Feedback especially welcome on: install from source and Docker, the Known limitations list (is anything there a blocker for you?), and whether the docs are clear for someone new to it.

This pull request and its description were written by Isaac.

zhousun and others added 18 commits November 14, 2026 23:55
…188)

- Make DROP and moonlink calls robust; handle quoted identifiers; remove load_files
- Add AGENTS.md, CLAUDE.md and docs/ARCHITECTURE.md
- Drop mirrors in moonlink only at top-level commit; follow Postgres identifier rules; reconnect after a moonlink restart

Co-authored-by: Isaac <no-reply@databricks.com>
- Add an end-to-end release gate for pg_mooncake

Co-authored-by: Isaac <no-reply@databricks.com>
- e2e gate: re-create after a warned DROP, restart with a dropped source, build provenance
- e2e gate: require the moonlink-down path for the re-create checks
- e2e gate: provenance comes only from a build-info matching the loaded binary
- e2e gate: name the cause of bgworker errors; stricter orphan DROP and build-info
- e2e gate: known bug for a SIGTERM panic at shutdown; NOT-SEEN for intermittent bugs
- e2e gate: check logs after the final shutdown; accept an errored mirror after DDL
- e2e gate: provenance fails closed
- e2e gate docs: say what a build-info attests

Co-authored-by: Isaac <no-reply@databricks.com>
- e2e gate: blocking and limitation known bugs; GATE BLOCKED
- e2e gate: worker exits at stops, error causes, one-line DDL match
- e2e gate: limitation XFAILs pass; strict known bugs; final-stop exits
- e2e gate: close false-PASS paths in leftovers, known bugs, stops

Co-authored-by: Isaac <no-reply@databricks.com>
- e2e gate: review nits on known-bugs loading and bgworker errors
- e2e gate: config reload, writes to mirrors, and role checks
- e2e gate: stricter known-bugs file, typed leftovers, hung-stop result
- e2e known bugs: shutdown panic in new scenarios; dead-table re-create
- e2e gate: name panics in a hung stop; shutdown panic can hang a stop
- e2e gate: session checks that prove their claim; fail-closed tooling

Co-authored-by: Isaac <no-reply@databricks.com>
- e2e gate: round-2 review nits on the session and stop checks

Co-authored-by: Isaac <no-reply@databricks.com>
…#196)

- e2e gate: last nits on log scans, crash forms and the loader
- e2e gate: PostgreSQL 18 crash lines; abnormal shutdown counts

Co-authored-by: Isaac <no-reply@databricks.com>
…er signals, pg_duckdb fork (#194)

- Clean up orphaned moonlink tables when re-creating a mirror
- Prepare 0.3.0: upgrade script, CHANGELOG, README install and self-hosting guide
- Resolve dependencies that build with the pinned Rust toolchain
- Add a CI workflow: build and make test on PostgreSQL 14-18, e2e gate on pushes
- Reject writes to mirrors before pg_duckdb plans them
- Document the pg_duckdb reload crash and who can read mirrors
- Handle shutdown signals only on the moonlink worker's main thread
- make test: fail when pg_regress tests fail
- Dockerfile: include the extension upgrade scripts
- Reject TRUNCATE, COPY FROM, renames and attaching mirrors
- create_table: after a failure, drop only the table this call created
- Upgrade script: tolerate 0.2.0 installs without load_files
- Drive moonlink from a thread created with shutdown signals blocked
- Docs: fix the upgrade, slot, reload and pg_duckdb notes; mark post-bump statements
- CI: stricter e2e job, separate fmt job, clippy, DuckDB version checks
- Fail release tags that still carry release markers
- Restart the moonlink worker after a SIGTERM outside a server shutdown
- Bundle pg_duckdb with the upstream fix for its TimeZone hook
- Re-raise an early SIGTERM for moonlink; give its thread the main thread's stack size
- Docs: database renames and column changes; name the pg_duckdb tag

Co-authored-by: Isaac <no-reply@databricks.com>
…tests/mirror_diff) (#192)

- tests/mirror_diff: differential mirror tests driven by the PostgreSQL regression suite
- tests/mirror_diff: known differences on the current submodules
- TESTING.md: how to run tests/mirror_diff
- tests/mirror_diff: typed, ordered comparison and checks that can't pass vacuously
- tests/mirror_diff: known differences pinned to their signatures
- tests/mirror_diff: document the comparison rules and the gate
- tests/mirror_diff: close the remaining false-PASS paths
- tests/mirror_diff: unsupported-types covers only the types mirrors lack
- tests/mirror_diff: document SKIPs, check names and known gaps
- tests/mirror_diff: close five more false-PASS paths, with a test each
- tests/mirror_diff: composites are not an unsupported type; pin probe rules to their events

Co-authored-by: Isaac <no-reply@databricks.com>
…nown limitation (#198)

- tests/mirror_diff: composites as DuckDB STRUCT text are a limitation

Co-authored-by: Isaac <no-reply@databricks.com>
- Bump moonlink to its 0.3 stack (de5de55)
- Docs: describe moonlink 0.3 without its write-ahead log
- Docs: composite columns read back as DuckDB STRUCT values
- CHANGELOG: moonlink 0.3 fixes; resolve the post-bump release markers
- Bump moonlink to 977dc54: faster initial copy
- CHANGELOG: correct character(n), full-disk and refusal notes; early-restart limitation

Co-authored-by: Isaac <no-reply@databricks.com>
- e2e gate: crash kinds with hyphens; minor-independent pins; loader
- e2e gate: probes for moonlink 0.3.0 RC1
- e2e known bugs: moonlink 0.3.0 RC1
- e2e known bugs: char(n) padding is a known limitation
- e2e gate: keep moonlink down for the while-down checks
- e2e known bugs: pg_duckdb's TimeZone hook is fixed
- e2e known bugs: moonlink 0.3.0 RC2 fixes the decimal panic
- Docs: final known limitations from the release gate
- e2e known bugs: name the moonlink commit the list was checked on

Co-authored-by: Isaac <no-reply@databricks.com>
…n tests (#201)

- Pin rust-postgres to the revision moonlink uses
- Makefile: honour a caller's PG_CONFIG and pass it to cargo pgrx
- Tests: freshness after unrelated commits, prepared statements, pre-existing partitions
- Review fixes: check PG_CONFIG for make test, absolute paths, rust-postgres check

Co-authored-by: Isaac <no-reply@databricks.com>
Moves the moonlink submodule to its 0.3.0 release commit 0e38b43 and documents moonlink PRs 2371-2374: create_table returns only once its initial copy is durable, fails with an error on a full disk, and a drop during create leaves nothing behind. Scopes the restart note and adds the server-crash-during-create_table limitation. Updates the mirror_diff pins.

Co-authored-by: Isaac <no-reply@databricks.com>
…ld, code and test fixes (#204)

- mooncake.data_directory (postmaster setting, default pg_mooncake under PGDATA): moonlink's files and socket can live outside PGDATA, so pg_basebackup and standbys work; the directory is locked per server and checked like a data directory.
- moonlink starts only once recovery finishes (not on standbys; on promotion).
- create_table reports the cause of libpq failures without echoing connection strings; the unreleased 0.2 legacy key code is gone.
- e2e gate requires PASS with no skips; restart and crash exactness are gated; unused harness machinery removed.
- Docker image builds from the submodules and is started and checked in CI; pg_regress gets a PostgreSQL 14 output file.

Co-authored-by: Isaac <no-reply@databricks.com>
Rewrites the release docs for 0.3.0 as the first release of the moonlink-based design: the CHANGELOG (overview, usage, behaviour and guarantees, performance, known limitations), the README (install, Coming from 0.1.x, backups and standbys, security), AGENTS.md as a contributor guide, ARCHITECTURE.md and TESTING.md. Removes the unreleased 0.2 upgrade script.

Co-authored-by: Isaac <no-reply@databricks.com>
…pins (#205)

Rewrites mirror_diff's rules for the 0.3.0 pins: one rule per refused-type message, limitation entries for the documented limitations, comparator events for numeric scale padding and char(n) padding that cannot hide real differences, and tests against the real expected.txt. A full PostgreSQL 17 pass: 2104 checks, 0 FAIL. mirror_diff is advisory, not a release gate.

Co-authored-by: Isaac <no-reply@databricks.com>
Moves the moonlink submodule to its 0.3.0 release commit 37b040f9: dropping and re-creating a mirror keeps the new one (moonlink PR 2376), dependency upgrades for the open RustSec advisories (moonlink PR 2378), and test-only fixes.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant