Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4b6449b
Robust DROP and moonlink calls, Postgres identifier rules, AGENTS.md …
zhousun Nov 15, 2026
c0b3529
End-to-end release gate (scripts/e2e-smoke.sh) (#189)
zhousun Nov 15, 2026
17df713
Release gate follow-ups (#190)
zhousun Nov 15, 2026
c9beefc
Release gate policy: blocking and limitation known bugs (#191)
zhousun Nov 15, 2026
5c643ac
Release gate: config reload, writes to mirrors and role checks (#193)
zhousun Nov 15, 2026
6725e7e
Release gate: round-2 review nits on the session and stop checks (#195)
zhousun Nov 15, 2026
f6d8351
Release gate: PG18 crash lines, log scans, crash forms and the loader…
zhousun Nov 15, 2026
cc0223e
Release prep for 0.3.0: upgrade script, docs, CI, mirror guards, work…
zhousun Nov 16, 2026
0d5bfc5
Differential mirror tests driven by the PostgreSQL regression suite (…
zhousun Nov 16, 2026
25d403e
tests/mirror_diff: composite columns read back as STRUCT text are a k…
zhousun Nov 16, 2026
eb5220f
Bump moonlink to its 0.3 stack (#199)
zhousun Nov 16, 2026
1a16373
Release gate: known bugs and limitations for 0.3.0 (#200)
zhousun Nov 16, 2026
06849db
Build: one rust-postgres source and PG_CONFIG; freshness and partitio…
zhousun Nov 16, 2026
84e9096
Pin moonlink 0.3.0 (0e38b43) (#202)
zhousun Nov 17, 2026
94ac19d
Release cleanup: mooncake.data_directory, strict e2e gate, Docker bui…
zhousun Nov 17, 2026
d8dd14a
Release docs for 0.3.0: first release of the moonlink design (#203)
zhousun Nov 17, 2026
f54bbce
tests/mirror_diff: rules for 0.3.0; advisory, passing on the release …
zhousun Nov 17, 2026
1719f31
Pin moonlink 0.3.0 (37b040f9) (#206)
zhousun Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -1,2 +1,7 @@
[target.'cfg(target_os="macos")']
rustflags = ["-Clink-arg=-Wl,-undefined,dynamic_lookup"]

# There is no committed Cargo.lock, so dependencies resolve fresh: prefer versions that support the
# package rust-version (rust-toolchain.toml) over newer ones that need a newer compiler.
[resolver]
incompatible-rust-versions = "fallback"
5 changes: 5 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# The image builds from the files the Dockerfile copies; build outputs and the duckdb_mooncake submodule are large.
**/.git
target
duckdb_mooncake
pg_duckdb/third_party/duckdb/build
105 changes: 105 additions & 0 deletions .github/actions/setup/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
name: Set up pg_mooncake build
description: >-
Installs PostgreSQL from the PostgreSQL apt repository, the Rust toolchain from rust-toolchain.toml and cargo-pgrx,
builds and installs pg_duckdb, and configures the pgrx-managed cluster with the duckdb_mooncake built by this workflow.

inputs:
pg:
description: PostgreSQL major version
required: true

runs:
using: composite
steps:
- name: Free disk space
shell: bash
run: |
# Two DuckDB builds and a debug build of moonlink do not fit in the runner's default free space.
sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup /opt/hostedtoolcache/CodeQL
df -h / | tail -1

- name: Install PostgreSQL ${{ inputs.pg }} and build dependencies
shell: bash
env:
PG: ${{ inputs.pg }}
run: |
sudo apt-get update -q
sudo apt-get install -y -q postgresql-common
echo 'create_main_cluster = false' | sudo tee -a /etc/postgresql-common/createcluster.conf > /dev/null
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y
sudo apt-get install -y -q build-essential cmake ninja-build pkg-config clang libclang-dev \
libssl-dev libcurl4-openssl-dev liblz4-dev "postgresql-$PG" "postgresql-server-dev-$PG"
pg_bin=/usr/lib/postgresql/$PG/bin
# This PostgreSQL's tools come first on PATH, for the steps that call them directly. make install and make
# package pass --pg-config themselves; a bare cargo pgrx install would use the first pg_config on PATH
# (AGENTS.md, pitfall 1).
echo "$pg_bin" >> "$GITHUB_PATH"
echo "PG_BIN=$pg_bin" >> "$GITHUB_ENV"
# Let the runner user install pg_duckdb and pg_mooncake without sudo.
sudo chown -R "$(id -un)" "$("$pg_bin/pg_config" --pkglibdir)" "$("$pg_bin/pg_config" --sharedir)/extension"

- name: Install the Rust toolchain from rust-toolchain.toml
shell: bash
run: rustup toolchain install

- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
with:
key: pg${{ inputs.pg }}

- uses: hendrikmuhs/ccache-action@dbeefd73e8e2ffc1bbc412570072735eb4db3607 # v1.2.25
with:
# pg_duckdb's DuckDB objects don't depend on the PostgreSQL version, so any major's cache helps.
key: pg_duckdb-pg${{ inputs.pg }}
restore-keys: pg_duckdb-
max-size: 2G
create-symlink: true

- name: Install cargo-pgrx and initialize its PostgreSQL ${{ inputs.pg }} cluster
shell: bash
env:
PG: ${{ inputs.pg }}
run: |
pgrx_version=$(sed -n 's/^pgrx = "=\{0,1\}\(.*\)"/\1/p' Cargo.toml)
# Outside the checkout, so the runner's default toolchain builds the CLI.
(cd "$RUNNER_TEMP" && cargo install --locked "cargo-pgrx@$pgrx_version")
cargo pgrx init "--pg$PG" "$PG_BIN/pg_config"

- name: Build and install pg_duckdb
shell: bash
run: make pg_duckdb "PG_VERSION=pg${{ inputs.pg }}"

# duckdb_mooncake must be built for the DuckDB version pg_duckdb embeds, or DuckDB refuses to load it.
- name: Read the DuckDB version
shell: bash
run: |
version=$(sed -n 's/^DUCKDB_VERSION := //p' Makefile)
pg_duckdb_version=$(sed -n 's/^DUCKDB_VERSION = //p' pg_duckdb/Makefile)
if [[ -z $version || $version != "$pg_duckdb_version" ]]; then
echo "Makefile DUCKDB_VERSION ($version) differs from pg_duckdb's ($pg_duckdb_version)" >&2
exit 1
fi
echo "DUCKDB_VERSION=$version" >> "$GITHUB_ENV"

- name: Download duckdb_mooncake
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: duckdb_mooncake
path: ${{ runner.temp }}/duckdb_extensions/${{ env.DUCKDB_VERSION }}/linux_amd64

- name: Configure the pgrx cluster
shell: bash
env:
PG: ${{ inputs.pg }}
run: |
ext_dir=$RUNNER_TEMP/duckdb_extensions
echo "DUCKDB_EXTENSION_DIR=$ext_dir" >> "$GITHUB_ENV"
# No .info file next to the extension, so the bootstrap INSTALL mooncake FROM community is a no-op and
# DuckDB loads the duckdb_mooncake built from the submodule (AGENTS.md, "Using a locally built duckdb_mooncake").
cat >> "$HOME/.pgrx/data-$PG/postgresql.conf" <<EOF
shared_preload_libraries = 'pg_duckdb,pg_mooncake'
wal_level = logical
duckdb.allow_community_extensions = true
duckdb.allow_unsigned_extensions = true
duckdb.autoinstall_known_extensions = false
duckdb.extension_directory = '$ext_dir'
EOF
272 changes: 272 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,272 @@
name: CI

on:
push:
branches: [main, v0.3-dev]
tags: ["v*"]
pull_request:
branches: [main, v0.3-dev]
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

env:
CARGO_TERM_COLOR: always
# Keeps file:line in panic backtraces at a fraction of the size of full debug info.
CARGO_PROFILE_DEV_DEBUG: line-tables-only
CROARING_VERSION: 4.3.11
CROARING_SHA256: acd3a17e6c95acf3154d60c38c90e927e6c531805bc740a040fa111819af380e

jobs:
# Release notes carry markers for statements that must be checked or filled in at release time (TESTING.md).
release-markers:
name: No release markers left
if: startsWith(github.ref, 'refs/tags/v')
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- run: scripts/check-release-markers.sh

fmt:
name: cargo fmt
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# cargo fmt reads the moonlink path dependencies' manifests.
- name: Check out moonlink
run: git submodule update --init --depth 1 moonlink
- name: Install the Rust toolchain from rust-toolchain.toml
run: rustup toolchain install
- run: cargo fmt --check
- name: Check that rust-postgres is pinned as in moonlink
run: make check-rust-postgres
- name: Unit tests of tests/mirror_diff (no database)
run: python3 tests/mirror_diff/test_mirror_diff.py

duckdb_mooncake:
name: Build duckdb_mooncake
runs-on: ubuntu-24.04
timeout-minutes: 120
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Read the submodule commit
id: pin
run: echo "commit=$(git rev-parse HEAD:duckdb_mooncake)" >> "$GITHUB_OUTPUT"

# The extension only changes with the submodule commit, so a hit skips the whole build.
- uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
id: cache
with:
path: dist/mooncake.duckdb_extension
key: duckdb_mooncake-${{ runner.os }}-${{ steps.pin.outputs.commit }}-croaring${{ env.CROARING_VERSION }}-${{ hashFiles('rust-toolchain.toml') }}

- name: Check out duckdb_mooncake
if: steps.cache.outputs.cache-hit != 'true'
run: git submodule update --init --recursive --depth 1 duckdb_mooncake

- name: Install build dependencies
if: steps.cache.outputs.cache-hit != 'true'
run: |
sudo apt-get update -q
sudo apt-get install -y -q build-essential cmake ninja-build pkg-config libssl-dev libcurl4-openssl-dev
rustup toolchain install

- uses: hendrikmuhs/ccache-action@dbeefd73e8e2ffc1bbc412570072735eb4db3607 # v1.2.25
if: steps.cache.outputs.cache-hit != 'true'
with:
key: duckdb_mooncake
max-size: 2G
create-symlink: true

# duckdb_mooncake needs CRoaring >= 1.x; Ubuntu 24.04's libroaring-dev (0.2.66) is too old.
- name: Build CRoaring
if: steps.cache.outputs.cache-hit != 'true'
run: |
cd "$RUNNER_TEMP"
curl -fsSL -o croaring.tar.gz "https://github.com/RoaringBitmap/CRoaring/archive/refs/tags/v$CROARING_VERSION.tar.gz"
echo "$CROARING_SHA256 croaring.tar.gz" | sha256sum -c -
tar xzf croaring.tar.gz
cmake -S "CRoaring-$CROARING_VERSION" -B croaring-build -G Ninja -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_POSITION_INDEPENDENT_CODE=ON -DENABLE_ROARING_TESTS=OFF -DROARING_BUILD_STATIC=ON -DCMAKE_INSTALL_PREFIX="$RUNNER_TEMP/croaring"
cmake --build croaring-build --target install

- name: Build duckdb_mooncake
if: steps.cache.outputs.cache-hit != 'true'
env:
CMAKE_PREFIX_PATH: ${{ runner.temp }}/croaring
run: |
make duckdb_mooncake
mkdir -p dist
cp duckdb_mooncake/build/release/extension/mooncake/mooncake.duckdb_extension dist/

# DuckDB loads an extension only into the DuckDB version recorded in its metadata footer.
- name: Check the extension's DuckDB version
run: |
version=$(sed -n 's/^DUCKDB_VERSION := //p' Makefile)
tail -c 512 dist/mooncake.duckdb_extension | tr -d '\0' | grep -q -F "$version" ||
{ echo "dist/mooncake.duckdb_extension was not built for DuckDB $version" >&2; exit 1; }

- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: duckdb_mooncake
path: dist/mooncake.duckdb_extension
if-no-files-found: error
retention-days: 7

test:
name: make test (PostgreSQL ${{ matrix.pg }})
needs: duckdb_mooncake
runs-on: ubuntu-24.04
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
pg: [14, 15, 16, 17, 18]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Check out moonlink and pg_duckdb
run: git submodule update --init --recursive --depth 1 moonlink pg_duckdb

- uses: ./.github/actions/setup
with:
pg: ${{ matrix.pg }}

- name: make test
run: make test "PG_VERSION=pg${{ matrix.pg }}"

- name: cargo clippy
if: matrix.pg == 18 && !cancelled()
run: cargo clippy -- -D warnings

- name: Show regression diffs and the server log
if: failure()
run: |
cat tests/pg_regress/regression.diffs || true
tail -n 300 "$HOME/.pgrx/${{ matrix.pg }}.log" || true

# Longer end-to-end gate (moonlink down/hung, restarts, crash, DDL; about 25 minutes per major). It passes only with
# GATE e2e-smoke PASS: every check passed or failed only with a limitation listed in scripts/e2e-known-bugs.txt, and
# none was skipped.
e2e:
name: e2e-smoke (PostgreSQL ${{ matrix.pg }})
if: github.event_name != 'pull_request'
needs: duckdb_mooncake
runs-on: ubuntu-24.04
timeout-minutes: 150
strategy:
fail-fast: false
matrix:
pg: [17, 18]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

# duckdb_mooncake is checked out for the build-provenance checks only.
- name: Check out submodules
run: |
git submodule update --init --recursive --depth 1 moonlink pg_duckdb
git submodule update --init --depth 1 duckdb_mooncake

- uses: ./.github/actions/setup
with:
pg: ${{ matrix.pg }}

- name: Install pg_mooncake (release build)
run: make install "PG_VERSION=pg${{ matrix.pg }}"

# The gate's build checks require a record of which duckdb_mooncake checkout built the extension; the
# duckdb_mooncake job built it from this same submodule commit.
- name: Record duckdb_mooncake provenance
run: scripts/e2e-build-info.sh "$DUCKDB_EXTENSION_DIR/$DUCKDB_VERSION/linux_amd64/mooncake.duckdb_extension"

- name: Run scripts/e2e-smoke.sh
env:
PG: ${{ matrix.pg }}
run: |
set +eo pipefail
log=$RUNNER_TEMP/e2e-smoke.log
PG_PREFIX="/usr/lib/postgresql/$PG" E2E_DIR="$RUNNER_TEMP/e2e" BUILD_SRC="$GITHUB_WORKSPACE" \
scripts/e2e-smoke.sh "$PG" | tee "$log"
rc=${PIPESTATUS[0]}
{
echo '```'
grep '^SUMMARY ' "$log" || echo "no SUMMARY line; e2e-smoke.sh exited with $rc"
grep -E '^(RESULT [^ ]+ (FAIL|XPASS|SKIP)|KNOWN [^ ]+ FIXED|GATE )' "$log"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
if [[ $rc != 0 ]] || [[ $(tail -n 1 "$log") != "GATE e2e-smoke PASS" ]]; then
echo "e2e-smoke.sh exited with $rc, last line: $(tail -n 1 "$log")" >&2
exit 1
fi

- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
if: always()
with:
name: e2e-smoke-pg${{ matrix.pg }}
path: |
${{ runner.temp }}/e2e-smoke.log
${{ runner.temp }}/e2e/out-*
if-no-files-found: ignore
retention-days: 14

# Builds the Docker image (not pushed), with pg_duckdb and pg_mooncake from this commit, and runs it.
docker:
name: docker build
if: github.event_name != 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 150
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check out moonlink and pg_duckdb
run: git submodule update --init --recursive --depth 1 moonlink pg_duckdb
- name: Free disk space
run: |
# The image builds DuckDB and a fat-LTO release of pg_mooncake.
sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup /opt/hostedtoolcache/CodeQL
df -h / | tail -1
- run: docker build --tag pg_mooncake:ci .
- name: Run the image
run: |
docker run -d --name mooncake -e POSTGRES_PASSWORD=ci pg_mooncake:ci
# The entrypoint's initialization server listens only on the Unix socket, the final one on TCP too.
for _ in $(seq 90); do
docker exec mooncake pg_isready -q -h 127.0.0.1 -U postgres && break
sleep 2
done
docker exec mooncake pg_isready -h 127.0.0.1 -U postgres
docker exec -i mooncake psql -U postgres -v ON_ERROR_STOP=1 -qAt > "$RUNNER_TEMP/result.txt" <<'SQL'
CREATE EXTENSION pg_mooncake CASCADE;
CREATE TABLE t (id int PRIMARY KEY, v text);
INSERT INTO t SELECT g, md5(g::text) FROM generate_series(1, 1000) g;
CALL mooncake.create_table('t_mc', 't');
INSERT INTO t VALUES (1001, 'after create_table');
SELECT count(*) || '|' || sum(id) FROM t_mc;
SHOW mooncake.data_directory;
SQL
cat "$RUNNER_TEMP/result.txt"
[[ $(sed -n 1p "$RUNNER_TEMP/result.txt") == "1001|501501" ]]
[[ $(sed -n 2p "$RUNNER_TEMP/result.txt") == /var/lib/postgresql/18/mooncake ]]
docker exec mooncake test -S /var/lib/postgresql/18/mooncake/moonlink.sock
# moonlink's files are outside the data directory, so a base backup works.
docker exec mooncake pg_basebackup -U postgres -D /tmp/basebackup -Ft -X fetch
- name: Show the container log
if: failure()
run: docker logs mooncake || true
3 changes: 2 additions & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@
url = https://github.com/Mooncake-Labs/moonlink.git
[submodule "pg_duckdb"]
path = pg_duckdb
url = https://github.com/duckdb/pg_duckdb.git
url = https://github.com/Mooncake-Labs/pg_duckdb.git
branch = v1.0.0-mooncake
Loading