Skip to content

OPS-056: harden the canonical bitácora workflows after their first review #3160

New issue
New issue
Closed
Closed
OPS-056: harden the canonical bitácora workflows after their first review#3160
Labels
bugSomething isn't working

Description

@mlorentedev
mlorentedev
opened on Nov 5, 2026

What

The 2026-11-04 bitácora rollout PRs got the first reviews of the canonical add-to-project.yml and bitacora-status.yml (PR-Agent and CodeRabbit on hive#564). The findings are about the canonical copies, so they are handled here once, not in each consumer.

To fix (canonical copies):

  1. add-to-project.yml: connect: connection refused is classified permanent, so the job goes red without a retry, though the header calls network failures transient.

  2. add-to-project.yml: a secondary rate limit gets 15 s of total backoff (attempt * 5, three attempts). GitHub asks for at least one minute.

  3. bitacora-status.yml: the status read has no retry, so one transient error leaves the assigned issue where it was.

  4. bitacora-status.yml: the guard reads fieldValueByName(name: "Status") and compares the option name with Backlog. A rename of either makes the guard wrong: a renamed Status field reads as unset and can move a Blocked issue to In Progress. Compare by field and option id, as the In Progress constant already does.

  5. add-to-project.yml: the rate-limit warning says bitacora-reconcile.yml heals the drop. That workflow lives in dotfiles (daily, every repository), not in the consumer, so name it as mlorentedev/dotfiles bitacora-reconcile.yml.

  6. bitacora-status.yml: "Full reference: vault 00_meta/runbooks/..." points at a path a contributor to the consumer cannot open.

  7. CI (from the review of fix(ci): silence the GraphQL-variable shellcheck note in add-to-project #3159): the canonical-workflow actionlint test skips in dotfiles CI, which does not install actionlint. Install a pinned, checksummed actionlint in the job that runs the bats suite.

  8. bitacora-status.yml (CodeRabbit on kubelab#3097): the job gates on github.event.issue.state, the state at assignment time. An issue that closes while the run is queued is still moved. Re-read the live state before the mutation.

  9. bitacora-rollout.sh: each re-run resets ci/bitacora-workflows to the base, which closes the open PR and leaves its reviews and triage on a closed PR (kubelab#3077 → fix(harness): refresh the insights and vault-doctor records from the vault #3097 → refactor(doctor): split checkHandoffThreads under the function limits #3100). Add commits to an open PR's branch instead.

  10. add-to-project.yml (PR-Agent on hive#566, REAL): the retry notice goes to stdout, and PROJECT_ID=$(gh_graphql ...) captures it, so a recovered transient error fails the mutation.

  11. add-to-project.yml (REAL): --jq prints null for a missing project, which passes the non-empty guard.

  12. add-to-project.yml (THEORETICAL): GraphQL reports RATE_LIMITED and permission errors with HTTP 200, so they take the success path. Check .errors in the body.

Fixed by: #3166 (10, 11), #3167 (3, 4, 6, 8), #3168 (7, 9). Still open: 1, 2, 5, 12, after #3166 lands (same file).

Declined, with reason:

  • Failing the job on the primary rate limit (CodeRabbit): green-on-purpose is the design (OPS-023, OPS-023: add-to-project silently drops issues from the bitácora on any API failure #809). The reconciler heals the drop, and failing would only add a red run on an issues: opened event nobody watches.
  • The "project id resolved empty" branch is dead (PR-Agent, low impact): the jq output on a null project is the string null, so the guard is not reached. Folded into 4 as "check for null too" rather than removed.

Acceptance criteria

  • Items 1–12 fixed in the canonical copies, with tests in tests/bitacora-*.bats for 1, 2 and 4.
  • The rollout re-run so the open consumer PRs carry the fix.

Activity

  1. mlorentedev
    added
    bugSomething isn't working
    on Nov 5, 2026

  2. mlorentedev commented on Nov 5, 2026

    @mlorentedev
    mlorentedev
    on Nov 5, 2026
    OwnerAuthor

    Item 7, from the review of #3159: tests/bitacora-canonical-consumer-rules.bats test 4 (actionlint with no ignores) skips in CI, because dotfiles CI does not install actionlint. Install a pinned, checksummed actionlint in the job that runs the bats suite, as ci.yml already does for shellcheck, so a canonical-workflow drift fails dotfiles CI rather than only a local pre-commit.

  3. mlorentedev commented on Nov 5, 2026

    @mlorentedev
    mlorentedev
    on Nov 5, 2026
    OwnerAuthor

    Two more, from the reviews of the re-rolled PRs:

    1. bitacora-status.yml (CodeRabbit, Major, on kubelab#3097): the job gates on github.event.issue.state, which is the state at assignment time. If the issue closes while the run is queued, the run can still add it to the board and move it to In Progress. Re-read the live state before the mutation.
    2. bitacora-rollout.sh: each re-run resets ci/bitacora-workflows to the base, which closes the open PR and opens a new one (kubelab#3077 → fix(harness): refresh the insights and vault-doctor records from the vault #3097 → refactor(doctor): split checkHandoffThreads under the function limits #3100, ts-bridge#583 → feat(secrets): strip backend unlock credentials from the run child env #610 → chore(main): release 0.23.0 #614, hive#564 → feat(mem): Claude session-start injectors (CLI-025) #566). Each consumer's review history and ## Review triage are left on a closed PR. Update the branch with a new commit so the PR survives a re-run.

  4. mlorentedev commented on Nov 5, 2026

    @mlorentedev
    mlorentedev
    on Nov 5, 2026
    OwnerAuthor

    From PR-Agent on hive#566 (current canonical, 944e6f5):

    1. REAL, the first to fix. add-to-project.yml:107: gh_graphql prints its retry notice (::warning::transient failure ...) to stdout, and the project lookup captures stdout: PROJECT_ID=$(gh_graphql ... --jq ...). One transient error followed by a successful retry leaves the notice inside PROJECT_ID, and the mutation then fails on a malformed id: a recovered error becomes a red run. Send the notice to stderr (the runner reads workflow commands from both streams) and add a test that runs the function against a stub that fails once.
    2. REAL. The dead null-project guard (item 4) is worse than dead: --jq prints the string null for a missing project, so the mutation is sent with project="null". Treat null as empty.
    3. THEORETICAL. The rate-limit/transient/fatal split keys on gh api's exit status, but GraphQL reports RATE_LIMITED and permission errors with HTTP 200, so those take the success path. Check .errors in the body.
  5. mlorentedev
    added 8 commits that reference this issue on Nov 5, 2026

    test(bitacora): satisfy the dbracket and stub-pairing guards

    0a8937f

    test(bitacora): end bare [[ ]] assertions in || false

    b86984f

    fix(bitacora): read every field value and honour the primary reset

    a24ddd1

    fix(bitacora): fail the repo when the open-PR lookup fails

    cb667c2

    fix(ci): keep the bitácora retry notice out of the captured project id (

    095adb9

    fix(bitacora): guard the status move by id, live state and retry (#3167)

    427d903

    fix(bitacora): keep the rollout PR open and lint canonical copies in …

    c62bc80

    fix(ci): retry refused connections and wait a minute on a secondary l…

    7042624

  6. mlorentedev commented on Nov 5, 2026

    @mlorentedev
    mlorentedev
    on Nov 5, 2026
    OwnerAuthor

    All twelve items are merged:

    PR Items
    #3166 10, 11
    #3167 3, 4, 6, 8
    #3168 7, 9
    #3179 1, 2, 5, 12

    The rollout ran once from main at 3550b98 (scripts/bitacora-rollout.sh, 16 changes applied, exit 0). It opened one PR per consumer, for the owner to review and merge: mlorentedev/kubelab#3123, mlorentedev/ts-bridge#640, mlorentedev/hive#568, mlorentedev/iris#59, mlorentedev/pdf-modifier#233, mlorentedev/pollex#123, mlorentedev/yt-metrics-cli#111, mlorentedev/web#605.

    None of the eight repos had an open PR on ci/bitacora-workflows, so item 9's open-PR path (commits added to the open PR, no reset) did not run live this time. Tests 18 to 21 of tests/bitacora-rollout.bats cover it; the next rollout over an open PR is its first live run.

  7. mlorentedev
    closed this as completedon Nov 5, 2026

  8. mlorentedev commented on Nov 5, 2026

    @mlorentedev
    mlorentedev
    on Nov 5, 2026
    OwnerAuthor

    Item 9's first live run: the rollout re-ran from main at 8104969 (after #3187) over the eight open consumer PRs. Each one is still open with the same number, and the compare API shows its head 2 commits ahead of the previous head and 0 behind: no reset, so no PR closed.

    kubelab#3123        af9a255b -> 2bea71bf  ahead=2 behind=0
ts-bridge#640       77ba3d68 -> f16f85e3  ahead=2 behind=0
hive#568            81f9de3a -> 2043fa5c  ahead=2 behind=0
iris#59             71395976 -> d36319b2  ahead=2 behind=0
pdf-modifier#233    8e0c2cf2 -> 2a4457ad  ahead=2 behind=0
pollex#123          7fb3fa9c -> 38bf8729  ahead=2 behind=0
yt-metrics-cli#111  01c08d5b -> b9dd3795  ahead=2 behind=0
web#605             4327abe1 -> 746e37d7  ahead=2 behind=0

    The same run's dry-run listed 617 open items in mlorentedev/kubelab. Before #3187, the reconciler listed at most 200 issues and 200 PRs per repository, silently, so part of kubelab was never reconciled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions