fix(release): staged quay promotion via bash oc image mirror

[deepsm007]

https://redhat-internal.slack.com/archives/GB7NB0CUC/p1781030982393999?thread_ts=1778764637.570759&cid=GB7NB0CUC

Staged promotion-quay uses bash-only oc image mirror: new image lands on *_incoming first, then (if float exists) abort-on-fail backup to _prune_* + __pre, latch _incoming→float, and __post1—replacing direct new→float and prune || true.

Closes manifest-unknown gap when mirror failed after float moved: old digest stays pinned on QCI via __pre until next promotion

/cc @jupierce @openshift/test-platform

Staged Quay Image Promotion via Bash oc image mirror

This PR changes ci-operator’s Quay image promotion to a staged, bash-only oc image mirror workflow that makes promotions to quay.io/openshift/ci resilient to partial failures.

Affected component

• pkg/steps/release (the promotion/quay promotion pod generation used by the promotion-quay job).

What changed (practical effect for CI users/operators)

• Promotion now stages the incoming pipeline image to a per-image *_incoming tag first, instead of mirroring directly into the float tag.
• If the float tag already exists, the job aborts-on-fail while backing up the existing float digest to a timestamped prune* tag and to a __pre tag. Only after those backups succeed does it latch *_incoming → float and then create a __post1 tag.
• Every mirror step uses explicit existence checks and per-step retry loops (up to 5 attempts) with randomized backoff, and the final attempt fails the job instead of ignoring errors (replacing previous prune || true behavior).
• The generated promotion pod script is deterministically ordered and mirrors per-image (not bulk-pruned), so a failed mirror after float rotation cannot leave the float tag pointing at an absent manifest; the old digest is pinned via __pre until the next successful promotion.
• Test fixtures under pkg/steps/release/testdata were updated to reflect the new multi-stage, per-image mirror and tag flows; tests adapted to a small signature change in the mirror-retry helper usage.

Why this matters

• Eliminates the “manifest-unknown” failure window where a mirror could fail after moving the float tag, causing float to reference a missing manifest.
• Improves safety and observability (pre/post tags capture the previous and new states) at the cost of additional mirror steps and slightly longer promotions.

Scope

• Only the quay-destination promotion path (promotion-quay) is affected; other promotion mechanisms remain unchanged.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 76e81c52-e8f2-4afd-9cf8-300a9fb43306

📥 Commits

Reviewing files that changed from the base of the PR and between 0d00db1 and f0b750a.

📒 Files selected for processing (6)

• pkg/steps/release/promote.go
• pkg/steps/release/promote_test.go
• pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay.yaml
• pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_4.12.yaml
• pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_multiple_tags.yaml
• pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_non_release_namespace.yaml

📝 Walkthrough

Walkthrough

This PR implements a staged image mirroring workflow for quay.io/openshift/ci destinations, replacing direct tag mirroring with a multi-step promotion flow. The implementation adds float promotion infrastructure with existence checks and retry mechanics, updates image routing logic to categorize destinations, and assembles conditional shell commands. Test fixtures validate the output across four promotion scenarios.

Changes

Quay Float Promotion Workflow

Layer / File(s)	Summary
Staged float promotion infrastructure pkg/steps/release/promote.go	Adds staging suffix constants (incoming, pre, post, prune), a quayFloatPromotion struct, and getStagedQuayFloatPromotionShell to generate multi-step shell scripts. Each script conditionally mirrors the existing float tag to prune/pre variants, promotes an incoming tag to the float, and mirrors the float to post variants, with embedded existence checks and randomized retry loops up to 5 attempts.
Image routing and collection pkg/steps/release/promote.go	Modifies getPromotionPod to initialize collections for incomingImages, pruneImages, quayFloatPromotions, and a pruneTagForFloat mapping. Routes quay.io/openshift/ci destinations through the routing loop to track float-promoted images and their prune tags. After collection, assigns each quayFloatPromotion its corresponding pruneTag.
Quay-step command assembly pkg/steps/release/promote.go	Updates the final quay-step argument assembly to mirror incomingImages when present, then generates and appends staged float promotion shell scripts for each quayFloatPromotion entry sorted deterministically by float tag.
Test fixtures for staged promotion flow pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay.yaml, ...promotion_quay_4.12.yaml, ...promotion_quay_multiple_tags.yaml, ...promotion_quay_non_release_namespace.yaml	YAML test fixtures demonstrating generated shell output for four promotion scenarios: basic two-image promotion, 4.12 release with two variants, multiple-tags with three variants, and non-release namespace. Each fixture validates multi-step retry loops, existence checks via oc image info, and conditional mirroring of prune/pre/post1 variants.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

approved, lgtm

Suggested reviewers

• danilo-gemoli

🚥 Pre-merge checks | ✅ 4 | ❌ 13

❌ Failed checks (1 warning, 12 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Go Error Handling	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Test Coverage For New Features	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Stable And Deterministic Test Names	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Test Structure And Quality	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Microshift Test Compatibility	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Single Node Openshift (Sno) Test Compatibility	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Topology-Aware Scheduling Compatibility	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Ote Binary Stdout Contract	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Ipv6 And Disconnected Network Test Compatibility	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No-Weak-Crypto	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Container-Privileges	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No-Sensitive-Data-In-Logs	❓ Inconclusive	Repository clone failed, so this custom check could not run with code access.	Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: implementing staged quay promotion using bash oc image mirror commands, which is the core focus of the PR across all modified files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[coderabbitai]

🧹 Nitpick comments (1)

pkg/steps/release/promote.go (1)

390-394: 💤 Low value

Unused newSrc field in struct.

The newSrc field is populated at line 476 but never read. getStagedQuayFloatPromotionShell only takes floatTag and pruneTag. Consider removing it.

 type quayFloatPromotion struct {
 	floatTag string
-	newSrc   string
 	pruneTag string
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/steps/release/promote.go` around lines 390 - 394, The quayFloatPromotion
struct contains an unused field newSrc that is set but never read (see
construction of quayFloatPromotion and usage in getStagedQuayFloatPromotionShell
which only needs floatTag and pruneTag); remove the newSrc field from the
quayFloatPromotion type and stop populating it where instances are created
(ensure any code that assigned to .newSrc is updated to only set floatTag and
pruneTag), and run a project-wide search to confirm there are no remaining
references to newSrc before committing.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/steps/release/promote.go`:
- Around line 390-394: The quayFloatPromotion struct contains an unused field
newSrc that is set but never read (see construction of quayFloatPromotion and
usage in getStagedQuayFloatPromotionShell which only needs floatTag and
pruneTag); remove the newSrc field from the quayFloatPromotion type and stop
populating it where instances are created (ensure any code that assigned to
.newSrc is updated to only set floatTag and pruneTag), and run a project-wide
search to confirm there are no remaining references to newSrc before committing.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ee1c043b-1627-4146-aa46-c6b5baf7699e

📥 Commits

Reviewing files that changed from the base of the PR and between 3a61181 and 0d00db1.

📒 Files selected for processing (5)

• pkg/steps/release/promote.go
• pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay.yaml
• pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_4.12.yaml
• pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_multiple_tags.yaml
• pkg/steps/release/testdata/zz_fixture_TestGetPromotionPod_promotion_quay_non_release_namespace.yaml

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• openshift/release (manual)
• openshift/ci-docs (manual)
• openshift/release-controller (manual)
• openshift/ci-chat-bot (manual)

[deepsm007]

/test e2e

[danilo-gemoli]

/lgtm
/hold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danilo-gemoli, deepsm007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [danilo-gemoli,deepsm007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[deepsm007]

/override ci/prow/images

[deepsm007]

/unhold

[openshift-merge-bot]

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

[openshift-ci]

@deepsm007: Overrode contexts on behalf of deepsm007: ci/prow/images

Details

In response to this:

/override ci/prow/images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 3a61181 and 2 for PR HEAD f0b750a in total

[openshift-ci]

@deepsm007: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.