Skip to content
This repository was archived by the owner on Jul 21, 2026. It is now read-only.

compression: publish linkable bzip2 and xz libraries - #25

Merged
brandonpayton merged 3 commits into
mainfrom
migrate/compression-library-surfaces
Jul 12, 2026
Merged

compression: publish linkable bzip2 and xz libraries#25
brandonpayton merged 3 commits into
mainfrom
migrate/compression-library-surfaces

Conversation

@brandonpayton

@brandonpayton brandonpayton commented Jul 11, 2026

Copy link
Copy Markdown
Member

Purpose

Expose complete, linkable bzip2 and xz library contracts for Kandelo Homebrew consumers instead of forcing downstream formulae to invoke compression CLIs.

Changes

  • publish libbz2.a, bzlib.h, and stable-opt bzip2.pc metadata;
  • build both static libraries as PIC so executables and Wasm side modules can consume the same archives;
  • build xz with POSIX threads so installed liblzma headers and archives agree about the multithreaded API;
  • publish stable-opt liblzma.pc metadata with its static pthread link requirements;
  • retain xz, xzdec, and the standard xz aliases;
  • bump both formula revisions because the installed keg surfaces changed;
  • replace xz's dead HTTP redirect with checksum-verified HTTPS and HTTP SourceForge mirrors;
  • compile tests through the installed pkg-config interfaces, including a real two-thread lzma_stream_encoder_mt round trip;
  • link each library into a PIC Wasm side module, then load it with dlopen, resolve its exported function with dlsym, and execute it under Kandelo.

Formula evidence

The artifact-producing formulae are byte-identical to the fully exercised ABI-18 head 6dd503b151b03768950aa597cadcd883db65173c:

  • Formula/bzip2.rb: blob 724a75bb5e26e6455d3918b96e9cb07390ba47ff;
  • Formula/xz.rb: blob b5bad450a85cb1be44f67875411a8a6189ccf9c6.

That head was validated from a fresh dedicated Homebrew prefix with Kandelo integration head 135630b95443c90d74672143042a730a09bc0f66:

  • clean bzip2 and xz source installs passed and their receipts identified the validated tap head;
  • two independent source builds produced byte-identical PIC archives: libbz2.a 97e0871ee1eab8dc3ec1f291d80639c3deb3f118a80448c5ec67f227f2bc48d2, and liblzma.a 12d9223d89b20b3662c432b6ae2af4b94ae9383183ee8448a9b518b40507c23f;
  • full libbz2 and two-thread liblzma consumers linked as Wasm side modules and executed through dlopen/dlsym;
  • literal brew test passed for both formulae under Node, including CLI round trips and the side-module executions;
  • Chromium passed the libbz2 and two-thread liblzma round trips, both side-module probes, and startup/version checks for xz, xzdec, and bzip2;
  • pkg-config resolved stable opt prefixes, -lbz2, and the asserted -llzma -pthread -lpthread static contract;
  • archive inspection found no Homebrew Cellar, builder-home, temporary-directory, or Nix-store paths;
  • source checksums, declared licenses, the plain-HTTP xz fallback mirror, strict online audits, brew readall --aliases, and formula diff checks passed.

The branch is now restacked onto tap main b9df20653b44fb3125b809c4cf96df27f199c5f9, which includes the merged ncurses, PCRE2, zstd, Dash, Make, and ed source formulae. Current head is ca052d77aeddb4d5ba626d2c8f4d07d9e50236d8. Fresh history-only validation through Kandelo's dev shell passed:

  • Ruby syntax for both formulae;
  • Homebrew style for both formulae;
  • literal Formulary.factory loading of both formulae from an installed review tap while resolving the shared support from Automattic/kandelo-homebrew;
  • the shared formula-support suite: 21 runs, 186 assertions, no failures;
  • ancestry, exact formula-blob identity, git range-diff, and git diff --check.

No source build, bottle build, Node runtime test, or Chromium runtime test was rerun for this history-only restack. The exact formula blobs preserve the prior source/runtime evidence; publication requires new bottles from the eventual merged tap commit.

Source merge and bottle gates

This PR contains only the two source formula updates and the additive README description. It contains no bottle block, bottle archive, link manifest, or provenance sidecar. The source formulae may merge independently after a fresh source-only review; that merge is not a claim that either package can yet be installed from a bottle in a stock Kandelo guest.

Automattic/kandelo#867 is merged and is no longer a gate. Trusted ABI-18 bottle construction and publication remain gated until:

Automattic/kandelo#890 is not required to build these roots themselves, which have no same-tap target dependencies, but it is required before downstream same-tap formulae consume their bottles during trusted builds. Downstream validation must demonstrate that those dependencies came from prior Homebrew bottles and receipts, not Kandelo registry artifacts or source-build fallbacks.

This wave intentionally publishes wasm32 only. Firefox, WebKit, and wasm64 were not validated and are not claimed. No Kandelo main-repository files or ABI definitions change here.

@brandonpayton

Copy link
Copy Markdown
Member Author

Devil's-advocate follow-up on final head e143ffd:

  • Corrected: both formulae changed their installed surfaces without revision bumps.
  • Corrected: xz installed headers advertised multithreaded liblzma entry points while --disable-threads omitted those symbols from liblzma.a.
  • Corrected: both library interfaces now publish stable-opt pkg-config paths and their tests compile through that metadata.
  • Corrected: xz's original plain-HTTP mirror redirected to HTTPS; the SourceForge HTTP mirror was downloaded with HTTPS disabled and matched the declared SHA-256.
  • Platform root cause extracted: xz exposed missing host pointer marshalling for raw sched_getaffinity; that fix is review-only main PR Marshal sched_getaffinity masks across process memory kandelo#884 rather than a formula workaround.
  • Runtime evidence: exact final keg CLIs and library consumers pass under Node and Chromium; the liblzma consumer uses two pthreads and verifies a 256 KiB round trip.
  • Static review: upstream hashes/licenses and output boundaries are coherent; no registry bridge, wrapper, or undeclared target dependency remains.

I found no remaining formula-code blocker on this head. This is not merge approval yet: #842, #867/ABI-18 artifacts, and #884 must land, then trusted bottles from e143ffd must pass guest pour and literal brew test gates. wasm64 remains explicitly deferred.

@brandonpayton

Copy link
Copy Markdown
Member Author

Devil's-advocate review follow-up on rebased final head 6dd503b:

  • Blocker found and corrected: both published static archives were non-PIC. Exact downstream side-module links failed with R_WASM_MEMORY_ADDR_{S,}LEB ... recompile with -fPIC for libbz2.a and liblzma.a.
  • Root fix: both libraries now compile as target PIC. Tests link real Wasm side modules through each installed pkg-config interface, then execute dlopen/dlsym and the exported library call under Kandelo.
  • Metadata contract tightened: the xz test now requires -llzma, -pthread, and -lpthread; executable linkage alone could not prove those flags because musl supplies pthread symbols from libc.
  • Final-head provenance: the branch was rebased onto tap main. Two clean source builds produced byte-identical archives, install receipts and embedded formulae match 6dd503b, literal Node brew test passes for both, and Chromium passes both round trips plus both side-module executions.
  • Static review: source hashes/licenses, target-only boundaries, stable-opt pkg-config paths, installed symbols, path hygiene, style, readall, and strict online audit all pass.

I found no remaining formula-code blocker on 6dd503b. This still is not merge approval: #842, #884, and #882 must land; trusted bottles from this exact head must pass guest pour and literal brew test; and final tap checks must be green. The earlier e143ffd review comment is superseded by this result.

Install and exercise the upstream libbz2 and liblzma consumer surfaces so dependent Kandelo formulae can link built-in compression support instead of spawning fallback tools.
Build libbz2 and liblzma objects as PIC so Kandelo shared Wasm consumers can link the published archives. Exercise each archive through pkg-config, a real side-module link, dlopen, and dlsym under the Kandelo runtime; retain liblzma's pthread metadata contract explicitly.
@brandonpayton
brandonpayton force-pushed the migrate/compression-library-surfaces branch from 6dd503b to ca052d7 Compare July 12, 2026 01:41
@brandonpayton

Copy link
Copy Markdown
Member Author

Devil’s-advocate source-only review at ca052d77aeddb4d5ba626d2c8f4d07d9e50236d8: ACCEPT. Fresh upstream source/mirror hashes and shipped licenses were verified. Both formulae passed Node brew test, including compression round trips, threaded xz, linkable side-module consumers, dlopen/dlsym, aliases, and xzdec; syntax, style, strict uninstalled audit, readall --aliases, installed/direct loading, and the support suite (21 runs / 186 assertions) also passed. Static/PIC libraries, headers, pkg-config linkage, and archive path hygiene were inspected. The diff contains no bottle block, sidecar, archive, or provenance metadata, so this approval is only for the source formulae; authoritative bottles remain separately gated.

@brandonpayton
brandonpayton merged commit a3ede33 into main Jul 12, 2026
@brandonpayton
brandonpayton deleted the migrate/compression-library-surfaces branch July 12, 2026 01:55
@brandonpayton
brandonpayton restored the migrate/compression-library-surfaces branch July 12, 2026 02:01
@brandonpayton
brandonpayton deleted the migrate/compression-library-surfaces branch July 12, 2026 02:01
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant