Repository navigation
Pin moonlink 0.3.0 (0e38b43) - #202
Merged
Merged
Conversation
Point the moonlink submodule at v0.3-dev 0e38b43, the commit pg_mooncake 0.3.0 ships with. Since 977dc54 it has moonlink PR 2371 (a text value of backslash-N was read as NULL by the initial copy, and short copied rows were stored with their values in the wrong columns), PR 2372 (create_snapshot waited forever on an idle mirror after a streamed transaction aborted), PR 2373 (create_table returns only once the initial copy is persisted, so a restart right after it no longer loses the copy) and PR 2374 (a full disk during the initial copy fails create_table promptly, and dropping a mirror while it is being created is safe). The CHANGELOG lists the four in place of their release marker, and the known limitation for a restart right after create_table goes, with its marker, its AGENTS.md entry and the caveat on the restart fix. The partitioned_existing test no longer needs to persist the copy itself before reading it. The docs' and the known-bugs file's moonlink pin follow. Co-authored-by: Isaac <no-reply@databricks.com>
zhousun
force-pushed
the
v0.3/13-final-pin
branch
from
November 16, 2026 23:53
a4880c0 to
e453cff
Compare
The CHANGELOG line for moonlink PRs 2373 and 2374 is now one bullet. A failed create_table leaves no mirror behind only when the moonlink worker restarts during it: after a server stop, restart or crash, the helper connection is gone before its cleanup DROP runs, so the empty mirror table stays and a retry fails until it is dropped. The line says so, and the case is a known limitation in the CHANGELOG and AGENTS.md. The slowdown is given as measured (roughly 0.4 s more per million narrow rows in a local test), and the full-disk sentence says what happens instead of comparing with a wait no release had. ARCHITECTURE.md says that CreateTable answers only once the copy is persisted to Iceberg and the mirror is recorded in sqlite. tests/mirror_diff/expected.txt names the current pins, and drops nine bugs that a full PostgreSQL 17 run on them reports fixed, with their rules. Its header says that the rules do not cover moonlink 0.3's create_table refusals and errored state yet. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
This moves the moonlink submodule to
0e38b435, the v0.3-dev commit that pg_mooncake 0.3.0 ships with. Since the previous pin (977dc549), moonlink has four more fixes:\N) as NULL. It also stored a copied row with fewer columns than the table with its values in the wrong columns; such rows are now rejected.mooncake.create_snapshotwaited forever on an idle mirror after a streamed transaction aborted.mooncake.create_tablenow returns only once the initial copy is persisted in Iceberg. Before, a moonlink restart within about half a second after it returned lost the copied rows, with no error.create_tablefails with an error. Dropping the mirror from another session whilecreate_tableruns makes the call fail and leaves nothing behind.Changes
moonlinksubmodule:977dc549→0e38b435.create_tableleaves behind. If only the moonlink worker restarts, nothing is left and the call can be retried. After a server restart or crash, the empty mirror table has to be dropped first.create_table, with its release marker;create_tableruns, the emptyUSING mooncaketable stays, and a retry fails until it is dropped. The table is created on a separate, autocommitted connection, and its cleanupDROPnever runs when that connection dies.[pins]: moonlink answersCreateTableonly once the copy is persisted to Iceberg and the mirror is recorded in its sqlite metadata store.[pins]in AGENTS.md and docs/ARCHITECTURE.md, and the header ofscripts/e2e-known-bugs.txt:0e38b435. No known-bugs entry changes; none covered the restart loss.tests/pg_regress/partitioned_existing: removes the workaround that persisted the copy before the first read (an extra row,create_snapshot, then a delete). The expected results are the same.tests/mirror_diff/expected.txt:c1a99efand pg_duckdb7daa8e5);create_tablerefusals and errored state yet.Release markers still open
scripts/check-release-markers.shstill lists these five, and thev*tag check keeps failing until they are resolved:CHANGELOG.md:1: the release date.CHANGELOG.md:119: the Performance numbers from the release benchmark.README.md:28: the Docker image's movingpgduckdb/pgduckdb:18-mainbase.README.md:44: the PostgreSQL versions, to check against the release test matrix.README.md:175:src_uri, which the release tests don't cover.Tests run
All on PostgreSQL 17 unless noted. The review commit changes only docs and the mirror_diff rules, so the build checks ran on the first commit.
-D warnings, PG17 and PG18make test, PG17 and PG18scripts/e2e-smoke.sh)create_tablereturns (100 rows, 5 trials; 100,000 rows, 3 trials)create_tablereturns (3 trials)create_tableon a 1M-row table (3 trials)create_tablefails, no mirror is left in Postgres or moonlink, and a retry mirrors all 1M rowspg_ctl stop -m fast0.4 s intocreate_tableon 1M rows, then start (2 trials)USING mooncaketable stays and moonlink has no mirror. The retry fails witherror creating table: public.s_mc; afterDROP TABLEit mirrors all 1M rowspg_ctl stop -m immediate(as a crash) at the same point (2 trials)DROP TABLEof the mirror from another session 0.5 s intocreate_tableon 1M rows (3 trials)create_tablefails (Table ... (dropped while it was being created) not found). Nothing is left in Postgres or moonlink, and a retry mirrors all 1M rowstests/mirror_diff, full run (data, cdc and query layers)tests/mirror_diffunit testsexpected.txtparses (22 bugs)With the previous pin, the first two restart checks lost the copied rows in every trial.
In the mirror_diff run, every check of these nine bugs passed (XPASS), so they are dropped:
date-out-of-rangeunchanged-toast-overwritteninheritance-children-ignoredinheritance-recovery-crash-looppartition-update-panicsrelation-schema-assertsnapshot-hangs-after-xidless-walduckdb-timezone-commit-abortduckdb-multiassign-abortThe tenth XPASS,
json-text-normalised, stays: onlyjsonbwas covered, notjson. Bugs that were not seen are kept too, because not seeing a bug doesn't show it is fixed.The 211 failures are behavior that 0.3 documents and that has no rule yet:
create_tablerefusing, by name: inheritance parents, partitioned tables without a primary key, unsupported types, and infinite timestamps;So mirror_diff's own gate fails on these pins until rules for them are added.
This pull request and its description were written by Isaac.