Housekeeping: remove the old dist/ binaries from git history (3 steps, spread over ~2 releases)

[mario4tier]

Why

Until 2026-08-05 every nightly committed the built packages (.deb / .msi / .zip /
.tar.gz) into dist/. They compress to ~0%, so each nightly added permanently unique
bytes to the history.

Measured on the full object graph:

	size	blobs
dist/*.{deb,msi,zip,tar.gz}	1.1 GiB — 93.4%	925
all other files in the whole history	82 MiB	43,492

274 commits touch them, spanning 2024-11-16 to 2026-08-05. The packed repo is 1.2 GiB, so
a clone is ~12x bigger than the project actually is. That is now a real cost: any workflow
using fetch-depth: 0 clones the whole thing (website.yml, dist-pool-gc.yml).

Going forward this is already fixed — the globs are in .gitignore and the packages
live in the ci-build-pool release-asset pool instead. This issue is only about the
binaries already recorded in history.

Why we can't just delete them all at once

Stripping a file from a commit changes that commit's SHA, which changes every descendant,
which moves the tags. That matters because vcpkg pins the SHA512 of GitHub's
auto-generated tag archive:

# microsoft/vcpkg ports/talib/portfile.cmake   (note: the port is "talib", not "ta-lib")
vcpkg_from_github(REPO "TA-Lib/ta-lib" REF "v${VERSION}"
                  SHA512 c8b9daf922cc98119e96a5bdb54187669e2f26be…)

That hash is of https://github.com/TA-Lib/ta-lib/archive/v0.7.1.tar.gz, fetched live from
GitHub. And GitHub embeds the commit SHA in the tarball's pax header:

v0.7.1 archive contains:  comment=2247d599bddf37ed37e3a709371517e46efc66f6

So the archive hash changes if the commit SHA changes — even when the file tree is
byte-for-byte identical. Move the v0.7.1 tag and every vcpkg install talib fails with
File does not have the expected hash, immediately, on all baselines, until a PR lands in
microsoft/vcpkg. That outage is measured in days-to-weeks of their review queue, not ours.

vcpkg's version database is append-only, so historical entries (0.6.4) can never be
repaired at all.

What each downstream packager pins, checked 2026-08-06:

packager	pins	affected by a tag move?
vcpkg 0.7.1	GitHub tag archive	yes — fixable by a PR
vcpkg 0.6.4	GitHub tag archive	yes — not fixable, ever
vcpkg 0.4.0	SourceForge ta-lib-0.4.0-msvc.zip	no
Homebrew	release asset ta-lib-0.7.1-src.tar.gz	no
conda-forge	tracks ta-lib-python, different repo	no

Homebrew is immune precisely because it downloads an uploaded release asset rather than a
generated tag archive. That is the fix for step 2.

The plan

Step 1 — strip only the history after v0.7.1 ✅ DONE 2026-08-06

Rewrite the 569 commits on dev (568 on main) that are descendants of
v0.7.1 (2247d599bddf37ed37e3a709371517e46efc66f6, 2026-07-03). Everything reachable
from v0.7.1 keeps its exact SHA.

The direction is forced, not a preference: a commit's SHA covers its ancestry, so stripping
anything below v0.7.1 would move v0.7.1 itself — which is the outage we're avoiding.

Because v0.7.1's ancestry subsumes every older tag, this leaves all of them untouched:

• v0.4.0 … v0.6.4, reference-pre-cutover (0d6b0e649, 2026-06-28) — all ancestors of
  v0.7.1, no retagging, no downstream impact
• the two frozen-oracle worktrees keep working as-is, same SHAs:
  ta-lib-064 @ 43f9d5042 (in v0.6.4) and ta-lib-ref @ 0d6b0e649
• ci-build-pool @ d5593923f (2026-08-05) is the only tag that moves. It is the
  internal asset-pool pre-release, it moves weekly anyway, and dist-pool-gc.yml reads
  origin/dev + origin/main rather than the tag's tree, so retagging it changes nothing
  functionally. Moving a tag does not delete the release, so no watcher notification.

Recovers 782 MiB. Repo goes 1.2 GiB → ~430 MiB.

Every commit, message, author and date is preserved — only the binary blobs leave.

Step 2 — at the next release, get vcpkg off the tag archive

Submit a microsoft/vcpkg PR switching ports/talib/portfile.cmake from vcpkg_from_github
to vcpkg_download_distfile against the uploaded release asset
(releases/download/vX.Y.Z/ta-lib-X.Y.Z-src.tar.gz) — the same thing Homebrew already does.

Worth doing on its own merits: it permanently removes GitHub's tag-archive instability from
the release path. GitHub has changed archive generation before and forced mass re-hashing
across vcpkg ports.

Step 3 — much later, strip the remaining 344 MiB

Once ~2 releases have shipped with a vcpkg port that does not pin a tag archive, rewrite
the old tags too and recover the last 344 MiB (repo → ~100 MiB).

By then the only people who could notice are those pinned to a vcpkg baseline old enough to
still resolve 0.6.4 — which by that point is years stale. Deliberately deferred so the blast
radius is as close to zero as we can make it.

Do not start step 3 before step 2 has been live for two releases. That wait is the whole
point of splitting this up.

Server-side gc: one request, at the end of step 3

A local rewrite does not shrink GitHub. Old objects stay reachable there, and clearing them
needs a request to GitHub Support asking them to run gc — that is their documented route,
not a workaround. Deliberately deferred to after step 3: step 3 produces a second batch of
unreachable objects, so one request at the end covers both. Nothing is lost by waiting —
clone size, which is what CI and contributors actually feel, is already fixed by step 1.

The payoff is capped, whenever it is filed. refs/pull/* refs are permanent and GitHub
does not delete them, so every commit that was ever a PR head stays reachable forever.
Measured 2026-08-06: 39 PR refs, and 11 of the 19 whose objects are still available locally
carry dist/ binaries in their trees. refs/pull/90/head is a worked example — the
docs/rust-crate-no-std branch was deleted during step 1 and the commit is already pruned
locally, yet GitHub still pins it.

So expect GitHub's stored size to land somewhere above the local figure even after step 3
plus a successful gc. Ten of those PR heads sit inside v0.7.1's ancestry, which step 3
strips from the branches but cannot strip from the PR refs. That is a known, accepted
ceiling — not a sign the rewrite failed.

Do not try to reduce the PR refs — it is not possible and not worth effort. They are
server-maintained and read-only: git push cannot delete them, closing a PR does not remove
one, and neither does deleting the source branch (PR #90 again). 29 of the 39 come from
contributor forks, which share an object store with upstream in GitHub's fork network, so
those commits survive through the fork regardless.

And it costs us nothing: git clone does not fetch refs/pull/* — only refs/heads/*
and refs/tags/*. Measured during step 1: 20 of the 39 PR heads are absent from a fresh
local clone. PR refs are invisible to contributors and to CI; they exist only in GitHub's
internal storage accounting. Worth one sentence in the Support ticket (Support can purge
them, unlike us) and no effort beyond that.

Progress

• Step 1 — strip v0.7.1..dev and v0.7.1..main — done 2026-08-06,
  1.20 GiB → 423.79 MiB. dev 20560551b, main 671b5089a.
• Step 2 — vcpkg port onto the release asset — done 2026-09-16, merged in the
  0.8.1 port update ([talib] update to 0.8.1 microsoft/vcpkg#53896). talib 0.8.1 installs from
  ta-lib-0.8.1-src.tar.gz, so no published version pins a tag archive from here on.
• Step 3 — strip the remaining old tags (after 2 releases on step 2) ⬅ next, earliest after 0.8.3

[mario4tier]

Step 1 — DONE 2026-08-06

1.20 GiB → 423.79 MiB (−65%, 782 MiB recovered).

dev            fc62025d4 -> 20560551b
main           34b49ded5 -> 671b5089a
ci-build-pool  d5593923f -> bbb9a4970   (only tag affected)

Tool: git-filter-repo v2.47.0, vendored to ~/.local/bin (Debian's PEP-668 blocks
pip install; it is a single dependency-free script).

git filter-repo --refs v0.7.1..dev v0.7.1..main --invert-paths \
    --path-regex '^dist/(?!.*0\.7\.1)[^/]+\.(deb|msi|zip|tar\.gz)$'

Why the regex excludes 0.7.1 — read this before scoping step 3 the same way.
Removing a path from a commit range also removes the commit that deletes it. The plain
^dist/[^/]+\.(deb|msi|zip|tar\.gz)$ therefore un-deleted the 8 packages sitting in
v0.7.1's own unrewritten tree — b35f9fea4 no longer removed them and they reappeared at
the tip. Excluding the 0.7.1 filenames keeps that add/delete lifecycle intact, at zero
cost: those blobs are already stored in v0.7.1's preserved tree, and the 0.7.2 / 0.8.1
packages are added and deleted inside the range so they strip cleanly. Caught by the
tip-tree-identity check below, which is the check worth keeping.

Verification

• dev and main tip trees are byte-identical to pre-rewrite (92d1b3f4f5de…,
  2f36b33519…); git diff reports 0 changed paths, working tree clean
• across all 569 rewritten commits, 4364 paths differ from their originals and every one
  matches ^dist/[^/]+\.(deb|msi|zip|tar\.gz)$ — nothing else was touched
• commit count unchanged (569 dev, 568 main); author, committer, both dates and subject
  identical for all 569
• v0.7.1 tag archive unchanged after the push — re-downloaded, 36,578,869 bytes,
  sha512 c8b9daf922cc98119e96a5bd…, an exact match for the SHA512 pinned in
  microsoft/vcpkg ports/talib/portfile.cmake. vcpkg is provably unaffected, which was the
  constraint driving the whole design.
• ci-build-pool release intact: 24 assets, original created_at 2026-08-05T04:04:35Z —
  moved, not recreated, so no watcher notification. v0.7.1 release keeps its 8 assets.
• v0.6.4 / v0.7.1 / reference-pre-cutover unchanged, so both frozen-oracle worktrees
  (ta-lib-064 @ 43f9d5042, ta-lib-ref @ 0d6b0e649) still resolve — no re-checkout

Protections were relaxed one at a time, each restored immediately after its own push and
verified against a pre-change snapshot: dev ruleset "AI allow on dev" → evaluate → back
to active (rules and bypass actor intact); main → allow_force_pushes: true → back to
false (every other field identical).

Branches deleted: docs/rust-crate-no-std (rejected no_std proposal, PR #90) and
issue166-step1-repoint (already merged) on origin, pr-128 locally. Origin now carries
only dev, main, gh-pages, sf-legacy.

A dangling ref nearly hid the win. After the push, local git gc reclaimed nothing:
refs/stash (f626cd950, sync-script-stash-20260805224100, from scripts/sync.py) still
pointed at pre-rewrite ec7cd3bf5 and held the entire old graph alive. git stash list
showed nothing because its reflog had been expired, but the ref survived. Its content is the
Java Maven restructure already landed in dd19f3922 / 34b49ded5; dropped after confirming
it exists in both backups. If a clone still looks large after a rewrite, check
git for-each-ref for stray refs, not just branches and tags.

Backups — keep until step 3 is finished and settled:

/home/olet/ta-lib-dist-gc-backup.git      hardlinked mirror, complete pre-rewrite history
/home/olet/ta-lib-dist-gc-backup.bundle   1.2 GB portable bundle

Residue is 351 MiB, all anchored by v0.7.1's preserved ancestry — exactly what step 3 exists
to collect.