Skip to content

Release cleanup: mooncake.data_directory, strict e2e gate, Docker build, code and test fixes - #204

Merged
zhousun merged 8 commits into
v0.3-devfrom
v0.3/15-code-cleanup
Nov 17, 2026
Merged

zhousun merged 8 commits into
v0.3-devfrom
v0.3/15-code-cleanup

Conversation

@zhousun

@zhousun zhousun commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Pre-release cleanup of code, tests, CI and build, and mooncake.data_directory

Cleanup before pg_mooncake 0.3.0, plus a fix for a release blocker: pg_basebackup failed on any server with mirrors.

Why

  • pg_basebackup streams tar, and Postgres' tar writer allows names of up to 99 characters. moonlink's file names under $PGDATA/pg_mooncake reach about 124, so backups and new standbys failed while any mirror existed.
  • Several parts of the code, tests and CI were left over from development: a 0.2 compatibility path (0.2 was never released), panics that dropped the real error, a PostgreSQL 14 test output faked with RAISE, and an e2e gate that tolerated skips and carried rules nothing used.

Changes

mooncake.data_directory (new, postmaster context)

  • The directory for moonlink's files and its RPC socket. The default is pg_mooncake, relative to the data directory, so the default behavior is unchanged. An absolute path is accepted.
  • Set it to a directory outside the data directory and pg_basebackup works; moonlink's data is then not part of the backup.
  • Checked at startup, like Postgres checks its data directory:
    • the directory is created with mode 0700 if missing;
    • it must be owned by the server's user and be writable;
    • an existing directory with group write or any access for others is refused, as is a symbolic link, the data directory itself and any of its parents;
    • the socket path must fit in 107 bytes;
    • a non-default directory inside the data directory gets a warning (./pg_mooncake counts as the default).
  • One server per directory:
    • the postmaster takes an exclusive flock on <dir>/moonlink.lock and keeps it for its lifetime (its children inherit it, and the kernel releases it however the processes end);
    • the lock file names the holder's data directory and postmaster PID, which the refusal reports (ERRCODE_OBJECT_IN_USE, with a hint);
    • a second server is refused even while the first one's moonlink isn't listening, for example on a standby;
    • as a second check, a socket in the directory that accepts connections is refused too.
    • A copy of a backup on the same host inherits the setting, so it needs its own directory.
  • The backends' RPC connection and DuckDB's ATTACH follow the setting.

The moonlink worker starts when recovery finishes (was: at consistent state)

  • On a primary the start time is unchanged, crash recovery included.
  • On a hot standby, moonlink used to run although it can't replicate from a read-only server. Its recovery then failed on ALTER PUBLICATION and left the mirrors it had restored in the errored state. Now it starts when the standby is promoted, and restored mirrors keep working.

Code

  • DROP and create_table's orphan cleanup send only the current moonlink key; the 0.2 legacy-key path is gone.
  • Loading pg_mooncake outside shared_preload_libraries is a clear error (pg_mooncake must be loaded via shared_preload_libraries).
  • Errors and warnings carry an errcode and a real hint instead of a HINT: inside the message.
  • mooncake.create_table never shows a connection string, which may hold a password. A connection failure names the server by host, port, database and user.
  • mooncake.create_table reports libpq failures as errors with the server's message. Examples: could not read the columns of public.x: relation "public.x" does not exist, could not create public.t: relation "t" already exists. Before, these were panics that dropped the cause.
  • One is_mirror helper for drop tracking and the write guard.
  • Comment fixes: RFC 3986 for uri_encode, and pg_terminate_backend can't reach the worker.
  • The unused bincode dependency is removed.

Tests

  • writes.sql runs MERGE and SET ACCESS METHOD directly. writes_1.out is the PostgreSQL 14 output.
  • ORDER BY for multi-row results, descriptive table names in identifiers.sql, and a data_directory test (the setting exists with postmaster context).
  • e2e gate:
    • The known-bugs file lists limitations only. The blocking class, GATE BLOCKED, requires rules, intermittent bugs, the bug-id hook, canary regexes and INFO diagnostics are removed.
    • exact_after_restart and exact_after_crash are gated checks.
    • The mirror write controls must fail with cannot modify mirror.
    • Cluster sockets are in E2E_DIR.
    • New scenario basebackup: plain, tar and standby backups, a copy on its own directory, a refused shared directory, and a standby promoted.

CI and build

  • The e2e job requires exit 0 and GATE e2e-smoke PASS.
  • A new job builds the Docker image, without pushing it, and runs it: it creates the extension, a source table and its mirror, reads the mirror back, and takes a base backup.
  • The fmt job runs tests/mirror_diff's unit tests.
  • clippy runs with the default features.
  • The Dockerfile builds pg_duckdb from the pinned submodule and pg_mooncake with the toolchain rust-toolchain.toml pins, on postgres:18. Before, the runtime base was pgduckdb/pgduckdb:18-main, whose DuckDB need not match the v1.4.1 that duckdb_mooncake is built for.
  • The Dockerfile no longer copies sql/. It passes PG_VERSION=pg18 and PG_CONFIG to make explicitly, because the postgres image sets PG_VERSION to its Debian package version.
  • The runtime image installs libcurl4t64 (pg_duckdb's built-in httpfs links it) and ca-certificates.
  • In the image, mooncake.data_directory is /var/lib/postgresql/18/mooncake: in the volume, next to PGDATA (/var/lib/postgresql/18/docker) and outside it, so pg_basebackup works. The image creates the directory owned by postgres, so a new volume gets it.
  • make help lists every target.

Testing

Check Result
cargo fmt --check, cargo clippy -- -D warnings pass (rustc 1.91)
make test, PostgreSQL 14.18 / 17.10 / 18.4 10/10 each
python3 tests/mirror_diff/test_mirror_diff.py 38 tests OK
scripts/e2e-smoke.sh 17, all scenarios, cassert build GATE e2e-smoke PASS: pass=447 fail=0 skip=0 xfail_limitation=15
scripts/e2e-smoke.sh 18, all scenarios, cassert build GATE e2e-smoke PASS: pass=447 fail=0 skip=0 xfail_limitation=15
Docker image not built locally; the CI job builds and runs it

The basebackup scenario (PostgreSQL 17 and 18):

  • pg_basebackup -Fp, -Ft and -R succeed with mooncake.data_directory outside the data directory.
  • A copy that keeps the primary's directory doesn't start: mooncake.data_directory "..." is in use by another server.
  • The same holds against a running standby, whose moonlink isn't running and has no socket: the lock refuses it, naming the standby's data directory and PID.
  • A listening socket in an unlocked directory, a symbolic link, a 0777 directory, the data directory and its parent are refused.
  • On a copy with its own directory:
    • moonlink runs without a crash loop;
    • mooncake.list_tables() is empty;
    • reading a mirror fails with Table postgres.public.b_mc not found;
    • DROP + mooncake.create_table resnapshots it (1001 rows) and CDC continues.
  • On a standby:
    • no moonlink worker runs;
    • mooncake.list_tables() fails with failed to connect to moonlink;
    • after pg_ctl promote the worker starts and re-creating the mirror resnapshots it.
  • Measured separately: a standby whose moonlink directory holds a mirror (an old primary rejoined after failover) keeps that mirror working after promotion.

This pull request and its description were written by Isaac.

zhousun and others added 8 commits November 17, 2026 01:27
- DROP and the orphan cleanup in create_table send only the current moonlink
  key. pg_mooncake 0.2 was never released, so no mirror is registered under
  the quoted name.
- create_table's libpq failures are ordinary errors that carry the server's
  message (missing source, taken mirror name, connection failures) instead
  of panics that dropped it; pg_regress covers two of them.
- One is_mirror helper for drop tracking and the write guard; mirror_name is
  private.
- Comments: uri_encode cites RFC 3986; the worker comment no longer says
  pg_terminate_backend can reach it (it isn't in the ProcArray); history
  phrases removed.
- Cargo.toml: remove the unused bincode; say why the bgworker feature exists.

Co-authored-by: Isaac <no-reply@databricks.com>
writes.sql ran MERGE and SET ACCESS METHOD through DO blocks that printed the
PostgreSQL 15+ message on 14. They now run directly; writes_1.out is the
PostgreSQL 14 output (both are syntax errors there). Also ORDER BY for
multi-row SELECT * results, and descriptive table names in identifiers.sql.

Co-authored-by: Isaac <no-reply@databricks.com>
The image no longer copies sql/ (no upgrade scripts ship), copies
rust-toolchain.toml and .cargo/config.toml, installs clang, libclang and
cmake as the README lists, and builds pg_mooncake with the toolchain
rust-toolchain.toml pins. It builds pg_duckdb from the pinned submodule on
postgres:18 instead of taking pgduckdb/pgduckdb:18-main, whose DuckDB need not
be the v1.4.1 that duckdb_mooncake is built for. .dockerignore keeps build
outputs and duckdb_mooncake out of the context.

make help lists every target and says that test runs the pg_regress tests.

Co-authored-by: Isaac <no-reply@databricks.com>
- CI's e2e job passes only on exit 0 and "GATE e2e-smoke PASS"; E2E_MAX_SKIP
  and the count parsing that tolerated skips are gone. CI also builds the
  Docker image (pushes and manual runs, not pushed), runs tests/mirror_diff's
  unit tests in the fmt job, and runs clippy with the default features.
- e2e-smoke.sh: the known-bugs file lists limitations only. Removed the
  blocking class and GATE BLOCKED (exit 4), "requires" rules, Intermittent /
  NOT-SEEN, the KNOWN_BUG_IDS hook, the canary regexes, INFO diagnostics and
  dead helpers.
- exact_after_restart and exact_after_crash are gated: a restart or crash
  must neither lose nor duplicate rows.
- mirror_writes' plain write controls must fail with "cannot modify mirror",
  not any error.
- Cluster sockets live in E2E_DIR instead of /tmp.
- WIRE_MOONLINK_RPC_REV is explained: duckdb_mooncake's locked moonlink_rpc,
  with an unchanged wire format.
- e2e-known-bugs.txt: header without stale pins; the server-file-roles
  limitation names both roles, which reads need (checked: either alone fails).
- e2e-build-info.sh no longer writes keys nothing reads.
- tests/mirror_diff scripts with a shebang are executable.

Co-authored-by: Isaac <no-reply@databricks.com>
moonlink's file names are longer than the 99 characters of Postgres' tar
format, so pg_basebackup failed on any server with mirrors. The new
postmaster setting mooncake.data_directory (default pg_mooncake, relative to
the data directory, as before) can point outside the data directory. It is
moonlink's base path and the location of the RPC socket that backends and
DuckDB connect to. At startup the directory is created (0700) if missing and
must be owned by the server user and writable. A socket path over 107 bytes
is rejected. A directory whose socket already accepts connections (a backup
copy on the same host inherits the setting) is refused. A non-default
directory inside the data directory gets a warning.

The worker now starts when recovery finishes instead of at consistent state.
That is the same moment on a primary, crash recovery included. On a standby,
moonlink ran but could not replicate from a read-only server, and its
attempts errored mirrors it had restored. Now it starts on promotion.

The e2e scenario basebackup takes plain, tar and standby backups, and checks
a copy on its own directory: mirrors error until re-created, and re-creating
resnapshots. It checks the refused shared directory, and a standby that runs
no moonlink until promoted.

Co-authored-by: Isaac <no-reply@databricks.com>
- The postgres image sets PG_VERSION to its Debian package version, which
  overrode the Makefile's PG_VERSION; the make calls pass PG_VERSION=pg18 and
  PG_CONFIG explicitly.
- The runtime stage installs libcurl4t64 (pg_duckdb's built-in httpfs links
  it) and ca-certificates (CREATE EXTENSION downloads the mooncake DuckDB
  extension over HTTPS).
- mooncake.data_directory defaults to /var/lib/postgresql/18/mooncake in the
  image: in the volume, next to PGDATA (/var/lib/postgresql/18/docker) and
  outside it, so pg_basebackup works. The image creates it owned by postgres,
  so a new volume gets it.
- The CI docker job runs the image: it waits for the server, creates the
  extension, a source table and its mirror, reads the mirror back, checks
  the data directory setting and takes a base backup.

Co-authored-by: Isaac <no-reply@databricks.com>
- The postmaster locks the directory with flock on moonlink.lock (holding
  its PID and data directory) and keeps it open for its lifetime, so a
  second server is refused even while the first one's moonlink isn't
  listening, for example on a standby or during recovery. The socket probe
  stays as a second check and no longer blocks.
- An existing directory with group write or any access for others, a
  symbolic link, and the data directory or one of its parents are refused.
  ./pg_mooncake counts as the default and gets no warning.
- Errors come with an errcode and a real hint (ERRCODE_CONFIG_FILE_ERROR,
  ERRCODE_OBJECT_IN_USE), also the warning of a DROP that moonlink did not
  see.
- Loading pg_mooncake outside shared_preload_libraries is a clear ERROR.
- create_table names the server it couldn't connect to by host, port,
  database and user; the connection string, which may hold a password, is
  never shown.

The e2e basebackup scenario checks the lock against a running standby, a
listening socket in an unlocked directory, and the refused directories.

Co-authored-by: Isaac <no-reply@databricks.com>
pg_duckdb includes unicode/ucol.h, and pg_config's CPPFLAGS point at the libxml2 headers; the GitHub runners have both
preinstalled, the postgres:18 image does not.

Co-authored-by: Isaac <no-reply@databricks.com>
@zhousun
zhousun merged commit 94ac19d into v0.3-dev Nov 17, 2026
21 checks passed
@zhousun
zhousun deleted the v0.3/15-code-cleanup branch November 17, 2026 05:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant