Skip to content

[release-4.21] OCPBUGS-88295, OCPBUGS-88297, OCPBUGS-82146, OCPBUGS-78330, OCPBUGS-85550: Replace OLM-based Istio install with Sail Library#1442

Merged
openshift-merge-bot[bot] merged 11 commits into
openshift:release-4.21from
gcs278:backport-noOLM-4.21
Jun 15, 2026
Merged

[release-4.21] OCPBUGS-88295, OCPBUGS-88297, OCPBUGS-82146, OCPBUGS-78330, OCPBUGS-85550: Replace OLM-based Istio install with Sail Library#1442
openshift-merge-bot[bot] merged 11 commits into
openshift:release-4.21from
gcs278:backport-noOLM-4.21

Conversation

@gcs278

@gcs278 gcs278 commented May 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
  • go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
  • pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

Phase 2 — TechPreview soak

Phase 3 — GA promotion

Follow-up

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Test Coverage

Early CI test results (OLM, OLM-to-OLM upgrade, noOLM, and OLM-to-noOLM migration) are documented in the origin test backport PR: openshift/origin#31232

Verification

  • go build ./pkg/operator/controller/gatewayclass/... compiles
  • go test ./pkg/operator/controller/gatewayclass/... passes
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 8, 2026
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 8, 2026
@openshift-ci

openshift-ci Bot commented May 8, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci-robot

openshift-ci-robot commented May 8, 2026 •
edited by openshift-ci Bot
Loading

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references NE-2471 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.z" version, but no target version was set.

Details

In response to this:

Summary

Backport of the noOLM / Sail Library code structure to release-4.21 with the GatewayAPIWithoutOLM feature gate disabled in all feature sets. The goal is structural alignment with 4.22 so that future bug fixes to the Sail Library path can be cleanly cherry-picked into 4.21 without massive conflicts.

The feature gate is OFF — the existing OLM path remains active. The Sail Library code is present but dormant.

Cherry-picked PRs

PR Title Why
#1329 OCPBUGS-70211: Fix logging for unmanaged controllers Prerequisite — conflicts with #1354's changes to gatewayclass/controller.go
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

External dependency

Repo Branch Why
gcs278/api@backport-GatewayAPIWithoutOLM-4.21 GatewayAPIWithoutOLM feature gate definition (disabled in all profiles) CIO code references this gate — won't compile without it. Needs to be merged into openshift/api release-4.21 before this PR can land.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames
  • go.mod / vendor: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)

Verification

  • go build ./pkg/operator/controller/gatewayclass/... compiles
  • go test ./pkg/operator/controller/gatewayclass/... passes
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented May 8, 2026 •
edited
Loading

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fbed4ebb-74e5-4d76-896f-a08fdf1025f7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@gcs278 gcs278 changed the title NE-2471: Backport noOLM / Sail Library to release-4.21 (gate OFF) [WIP] [TEST POC] NE-2471: Backport noOLM / Sail Library to release-4.21 May 8, 2026
@gcs278 gcs278 force-pushed the backport-noOLM-4.21 branch 7 times, most recently from c647f7b to b47ed5e Compare May 15, 2026 02:00
@gcs278

gcs278 commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

No longer pursuing this
/close

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 29, 2026
@openshift-ci openshift-ci Bot closed this May 29, 2026
@openshift-ci

openshift-ci Bot commented May 29, 2026

Copy link
Copy Markdown
Contributor

@gcs278: Closed this PR.

Details

In response to this:

No longer pursuing this
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@gcs278

gcs278 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Some new information makes this backport attractive again.
/reopen

@openshift-ci openshift-ci Bot reopened this Jun 1, 2026
@openshift-ci

openshift-ci Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

@gcs278: Reopened this PR.

Details

In response to this:

Some new information makes this backport attractive again.
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@gcs278 gcs278 force-pushed the backport-noOLM-4.21 branch from b47ed5e to af43e28 Compare June 1, 2026 16:16
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 1, 2026
@gcs278 gcs278 force-pushed the backport-noOLM-4.21 branch 3 times, most recently from 530e487 to 96ad9e5 Compare June 1, 2026 16:58
@gcs278 gcs278 changed the title [WIP] [TEST POC] NE-2471: Backport noOLM / Sail Library to release-4.21 [WIP] NE-2471: Backport noOLM / Sail Library to release-4.21 Jun 1, 2026
@gcs278

gcs278 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

/test ?

@gcs278

gcs278 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

i manually pullled in #1444 for now - because we need to bump to istio 1.28.5, and might as well bump the GWAPI CRDs

/test e2e-aws-operator-techpreview

@gcs278 gcs278 force-pushed the backport-noOLM-4.21 branch 2 times, most recently from 9b7956c to 9df9dba Compare June 2, 2026 02:35
@gcs278 gcs278 mentioned this pull request Jun 10, 2026
Closed
@rikatz

rikatz commented Jun 10, 2026

Copy link
Copy Markdown
Member

PR lgtm:

  • Reviewed the whole backport, is clean. Missing feature gates were re-added. The right annotation for feature gate was fixed
  • @aswinsuryan question on race condition is valid, but a very edge case IMO
  • @rhamini3 during validation will take a look if the logs are still there and the usage of zap is working correctly
  • Every vendored import was checked against hidden files or javascripts, there is a .swp file on one of the imports that probably the user commited, it is mostly in case someone is using vim and tries to open it, not a big deal

/approve
/lgtm

still needs a pass from @Miciah and from QE team

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 10, 2026
@openshift-ci

openshift-ci Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rikatz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 10, 2026
@gcs278 gcs278 changed the title [release-4.21] OCPBUGS-88295, OCPBUGS-88297, OCPBUGS-82146: Backport noOLM / Sail Library to release-4.21 [release-4.21] OCPBUGS-88295, OCPBUGS-88297, OCPBUGS-82146, OCPBUGS-78330: Backport noOLM / Sail Library to release-4.21 Jun 10, 2026
@openshift-ci-robot openshift-ci-robot added jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. and removed jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jun 10, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-88295, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-88297, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-79467 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-79467 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-82146, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-76609 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-76609 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-78330, which is invalid:

  • expected the bug to target the "4.21.z" version, but no target version was set
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
  • go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
  • pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

Phase 2 — TechPreview soak

Phase 3 — GA promotion

Follow-up

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

  • go build ./pkg/operator/controller/gatewayclass/... compiles
  • go test ./pkg/operator/controller/gatewayclass/... passes
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278

gcs278 commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Jun 10, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-88295, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

This pull request references Jira Issue OCPBUGS-88297, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-79467 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-79467 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-82146, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-76609 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-76609 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-78330, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88300 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88300 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278

gcs278 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

/jira cherrypick OCPBUGS-85550

@openshift-ci-robot

openshift-ci-robot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

@gcs278: An error was encountered cloning bug for cherrypick for bug OCPBUGS-85550 on the Jira server at https://redhat.atlassian.net. No known errors were detected, please see the full error message for details.

Full error message. request failed. Please analyze the request body for more details. Status code: 400: {"errorMessages":[],"errors":{"customfield_10980":"Field does not support update 'customfield_10980'","customfield_10978":"Field does not support update 'customfield_10978'","customfield_10979":"Field does not support update 'customfield_10979'"}}

Please contact an administrator to resolve this issue, then request a bug refresh with /jira refresh.

Details

In response to this:

/jira cherrypick OCPBUGS-85550

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278

gcs278 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

/jira cherrypick OCPBUGS-85550

@openshift-ci-robot

openshift-ci-robot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

@gcs278: An error was encountered cloning bug for cherrypick for bug OCPBUGS-85550 on the Jira server at https://redhat.atlassian.net. No known errors were detected, please see the full error message for details.

Full error message. request failed. Please analyze the request body for more details. Status code: 400: {"errorMessages":[],"errors":{"customfield_10980":"Field does not support update 'customfield_10980'","customfield_10978":"Field does not support update 'customfield_10978'","customfield_10979":"Field does not support update 'customfield_10979'"}}

Please contact an administrator to resolve this issue, then request a bug refresh with /jira refresh.

Details

In response to this:

/jira cherrypick OCPBUGS-85550

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

openshift-ci-robot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-88295, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-88297, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-79467 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-79467 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-82146, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-76609 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-76609 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-78330, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88300 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88300 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-85550, which is invalid:

  • expected the bug to target the "4.21.z" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
  • go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
  • pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

Phase 2 — TechPreview soak

Phase 3 — GA promotion

Follow-up

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

  • go build ./pkg/operator/controller/gatewayclass/... compiles
  • go test ./pkg/operator/controller/gatewayclass/... passes
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278

gcs278 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot

openshift-ci-robot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-88295, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

This pull request references Jira Issue OCPBUGS-88297, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-79467 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-79467 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-82146, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-76609 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-76609 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-78330, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88300 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88300 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-85550, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88302 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88302 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

openshift-ci-robot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

@gcs278: This pull request references Jira Issue OCPBUGS-88295, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

This pull request references Jira Issue OCPBUGS-88297, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-79467 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-79467 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-82146, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-76609 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-76609 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-78330, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88300 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88300 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

This pull request references Jira Issue OCPBUGS-85550, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.z) matches configured target version for branch (4.21.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-88302 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-88302 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
  • bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

Details

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
  • go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
  • pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

Phase 2 — TechPreview soak

Phase 3 — GA promotion

Follow-up

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

  • go build ./pkg/operator/controller/gatewayclass/... compiles
  • go test ./pkg/operator/controller/gatewayclass/... passes
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@gcs278

gcs278 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-ovn-hypershift-conformance

@aswinsuryan

aswinsuryan commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@rhamini3

rhamini3 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

verified all the related bugs

OCPBUGS-82146, OCPBUGS-78330 and NO-OLM check

GatewayAPIWithoutOLM enabled
enabled:
    - name: AdditionalRoutingCapabilities
    - name: AdminNetworkPolicy
    - name: AlibabaPlatform
    - name: AzureWorkloadIdentity
    - name: BuildCSIVolumes
    - name: CPMSMachineNamePrefix
    - name: ConsolePluginContentSecurityPolicy
    - name: EventTTL
    - name: ExternalOIDC
    - name: ExternalOIDCWithUIDAndExtraClaimMappings
    - name: GCPClusterHostedDNSInstall
    - name: GatewayAPI
    - name: GatewayAPIController
    - name: GatewayAPIWithoutOLM

GatewayClass Finalizers and Status conditions show as expected
finalizers:
    - openshift.io/ingress-operator-sail-finalizer
  status:
    conditions:
    - lastTransitionTime: "2026-06-12T16:40:47Z"
      message: Handled by Istio controller
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2026-06-12T16:40:42Z"
      message: istiod v1.27.3 installed
      observedGeneration: 1
      reason: Installed
      status: "True"
      type: ControllerInstalled
    - lastTransitionTime: "2026-06-12T16:40:42Z"
      message: CRDs updated by CIO
      observedGeneration: 1
      reason: ManagedByCIO
      status: "True"
      type: CRDsReady

istiod deployment contains correct labels
labels:
    app: istiod
    app.kubernetes.io/instance: openshift-gateway-istiod
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: istiod
    app.kubernetes.io/part-of: istio
    app.kubernetes.io/version: 1.27.3
    helm.sh/chart: istiod-1.27.3
    istio: pilot
    istio.io/rev: openshift-gateway
    managed-by: sail-library

No OLM resources are present
% oc -n openshift-operators get sub,csv,pod
No resources found in openshift-operators namespace.
iamin@iamin-mac release % oc get istio
error: the server doesn't have a resource type "istio"

Istio CRDs have expected label
oc get crds destinationrules.networking.istio.io -o yaml | grep ingress.operator.openshift.io/owned
    ingress.operator.openshift.io/owned: "true"
iamin@iamin-mac release % oc get crds envoyfilters.networking.istio.io -o yaml | grep ingress.operator.openshift.io/owned
    ingress.operator.openshift.io/owned: "true"
iamin@iamin-mac release % oc get crds gateways.networking.istio.io -o yaml | grep ingress.operator.openshift.io/owned
    ingress.operator.openshift.io/owned: "true"
iamin@iamin-mac release % oc get crds workloadgroups.networking.istio.io -o yaml | grep ingress.operator.openshift.io/owned
    ingress.operator.openshift.io/owned: "true"

OCPBUGS-88297 -> no verbose INFO logging found in CIO logs

% oc logs -n openshift-ingress-operator deployment/ingress-operator | grep operator.init.install
Defaulted container "ingress-operator" out of: ingress-operator, kube-rbac-proxy
2026-06-12T16:40:39.404Z	INFO	operator.init.install	install/lifecycle.go:205	Setting up informers	{"count": 13}
2026-06-12T16:40:39.805Z	INFO	operator.init.install	install/lifecycle.go:205	Informers synced and watching for drift
2026-06-12T16:40:39.805Z	INFO	operator.init.install	runtime/asm_amd64.s:1700	Processing workqueue
2026-06-12T16:40:39.805Z	INFO	operator.init.install	install/lifecycle.go:208	Reconciling
2026-06-12T16:40:39.875Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "wasmplugins.extensions.istio.io"}
2026-06-12T16:40:40.021Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "destinationrules.networking.istio.io"}
2026-06-12T16:40:40.097Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "envoyfilters.networking.istio.io"}
2026-06-12T16:40:40.136Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "gateways.networking.istio.io"}
2026-06-12T16:40:40.200Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "proxyconfigs.networking.istio.io"}
2026-06-12T16:40:40.307Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "serviceentries.networking.istio.io"}
2026-06-12T16:40:40.419Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "sidecars.networking.istio.io"}
2026-06-12T16:40:40.531Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "virtualservices.networking.istio.io"}
2026-06-12T16:40:40.563Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "workloadentries.networking.istio.io"}
2026-06-12T16:40:40.610Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "workloadgroups.networking.istio.io"}
2026-06-12T16:40:40.645Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "authorizationpolicies.security.istio.io"}
2026-06-12T16:40:40.726Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "peerauthentications.security.istio.io"}
2026-06-12T16:40:40.753Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "requestauthentications.security.istio.io"}
2026-06-12T16:40:40.805Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "telemetries.telemetry.istio.io"}
2026-06-12T16:40:42.326Z	INFO	operator.init.install	cache/controller.go:266	Drift detected (update)	{"gvk": "apps/v1, Kind=Deployment", "watchType": "Owned", "name": "istiod-openshift-gateway"}
2026-06-12T16:40:42.358Z	INFO	operator.init.install	cache/controller.go:266	Drift detected (update)	{"gvk": "apps/v1, Kind=Deployment", "watchType": "Owned", "name": "istiod-openshift-gateway"}
2026-06-12T16:40:42.369Z	INFO	operator.init.install	install/lifecycle.go:208	Reconcile complete	{"installed": true, "error": null}
2026-06-12T16:40:42.369Z	INFO	operator.init.install	install/lifecycle.go:208	Reconciling

OCPBUGS-85550 -> controllers are started in istiod pod logs

2026-06-12T16:40:47.542267Z	info	controllers	starting	controller=inference pool shadow service reconciler
2026-06-12T16:40:47.542453Z	info	controllers	starting	controller=healthcheck
2026-06-12T16:40:47.542468Z	info	Istiod CA has started
2026-06-12T16:40:47.542480Z	info	controllers	starting	controller=auto-register existing connections
2026-06-12T16:40:47.542490Z	info	controllers	starting	controller=unregister_workloadentry
2026-06-12T16:40:47.542278Z	info	kube	Starting Pilot K8S CRD controller	controller=crd-controller
2026-06-12T16:40:47.542534Z	info	kube	controller "gateway.networking.k8s.io/v1beta1/HTTPRoute" is syncing...	controller=crd-controller
2026-06-12T16:40:47.542302Z	info	klog	attempting to acquire leader lease openshift-ingress/istio-gateway-deployment-openshift-gateway...
2026-06-12T16:40:47.542517Z	info	cluster "Kubernetes" kube client started
2026-06-12T16:40:47.542716Z	info	klog	attempting to acquire leader lease openshift-ingress/istio-gateway-status-leader-openshift-gateway...
2026-06-12T16:40:47.545052Z	info	kube	controller "security.istio.io/v1/PeerAuthentication" is syncing...	controller=crd-controller
2026-06-12T16:40:47.549040Z	info	sync complete	name=multicluster remote secrets attempt=3 time=6.699222ms
2026-06-12T16:40:47.549119Z	info	multicluster remote secrets controller cache synced in 7.201277ms
2026-06-12T16:40:47.549131Z	info	controllers	starting	controller=multicluster secret
2026-06-12T16:40:47.549461Z	info	kube	controller "networking.istio.io/v1alpha3/EnvoyFilter" is syncing...	controller=crd-controller
2026-06-12T16:40:47.556227Z	info	sync complete	name=webhook patcher attempt=4 time=14.116135ms
2026-06-12T16:40:47.556351Z	info	controllers	starting	controller=webhook patcher
2026-06-12T16:40:47.556489Z	info	sync complete	name=configmap istio-sidecar-injector-openshift-gateway attempt=4 time=14.30976ms
2026-06-12T16:40:47.556598Z	info	controllers	starting	controller=configmap istio-sidecar-injector-openshift-gateway
2026-06-12T16:40:47.558345Z	info	kube	controller "security.istio.io/v1/RequestAuthentication" is syncing...	controller=crd-controller
2026-06-12T16:40:47.558356Z	info	sync complete	name=default revision attempt=4 time=16.270313ms
2026-06-12T16:40:47.558520Z	info	controllers	starting	controller=default revision
2026-06-12T16:40:47.556313Z	info	sync complete	name=default revision attempt=4 time=14.073443ms
2026-06-12T16:40:47.559801Z	info	controllers	starting	controller=default revision

/verified by @rhamini3

@openshift-ci-robot

openshift-ci-robot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

@rhamini3: This PR has been marked as verified by @rhamini3.

Details

In response to this:

verified all the related bugs

OCPBUGS-82146, OCPBUGS-78330 and NO-OLM check

GatewayAPIWithoutOLM enabled
enabled:
   - name: AdditionalRoutingCapabilities
   - name: AdminNetworkPolicy
   - name: AlibabaPlatform
   - name: AzureWorkloadIdentity
   - name: BuildCSIVolumes
   - name: CPMSMachineNamePrefix
   - name: ConsolePluginContentSecurityPolicy
   - name: EventTTL
   - name: ExternalOIDC
   - name: ExternalOIDCWithUIDAndExtraClaimMappings
   - name: GCPClusterHostedDNSInstall
   - name: GatewayAPI
   - name: GatewayAPIController
   - name: GatewayAPIWithoutOLM

GatewayClass Finalizers and Status conditions show as expected
finalizers:
   - openshift.io/ingress-operator-sail-finalizer
 status:
   conditions:
   - lastTransitionTime: "2026-06-12T16:40:47Z"
     message: Handled by Istio controller
     observedGeneration: 1
     reason: Accepted
     status: "True"
     type: Accepted
   - lastTransitionTime: "2026-06-12T16:40:42Z"
     message: istiod v1.27.3 installed
     observedGeneration: 1
     reason: Installed
     status: "True"
     type: ControllerInstalled
   - lastTransitionTime: "2026-06-12T16:40:42Z"
     message: CRDs updated by CIO
     observedGeneration: 1
     reason: ManagedByCIO
     status: "True"
     type: CRDsReady

istiod deployment contains correct labels
labels:
   app: istiod
   app.kubernetes.io/instance: openshift-gateway-istiod
   app.kubernetes.io/managed-by: Helm
   app.kubernetes.io/name: istiod
   app.kubernetes.io/part-of: istio
   app.kubernetes.io/version: 1.27.3
   helm.sh/chart: istiod-1.27.3
   istio: pilot
   istio.io/rev: openshift-gateway
   managed-by: sail-library

No OLM resources are present
% oc -n openshift-operators get sub,csv,pod
No resources found in openshift-operators namespace.
iamin@iamin-mac release % oc get istio
error: the server doesn't have a resource type "istio"

Istio CRDs have expected label
oc get crds destinationrules.networking.istio.io -o yaml | grep ingress.operator.openshift.io/owned
   ingress.operator.openshift.io/owned: "true"
iamin@iamin-mac release % oc get crds envoyfilters.networking.istio.io -o yaml | grep ingress.operator.openshift.io/owned
   ingress.operator.openshift.io/owned: "true"
iamin@iamin-mac release % oc get crds gateways.networking.istio.io -o yaml | grep ingress.operator.openshift.io/owned
   ingress.operator.openshift.io/owned: "true"
iamin@iamin-mac release % oc get crds workloadgroups.networking.istio.io -o yaml | grep ingress.operator.openshift.io/owned
   ingress.operator.openshift.io/owned: "true"

OCPBUGS-88297 -> no verbose INFO logging found in CIO logs

% oc logs -n openshift-ingress-operator deployment/ingress-operator | grep operator.init.install
Defaulted container "ingress-operator" out of: ingress-operator, kube-rbac-proxy
2026-06-12T16:40:39.404Z	INFO	operator.init.install	install/lifecycle.go:205	Setting up informers	{"count": 13}
2026-06-12T16:40:39.805Z	INFO	operator.init.install	install/lifecycle.go:205	Informers synced and watching for drift
2026-06-12T16:40:39.805Z	INFO	operator.init.install	runtime/asm_amd64.s:1700	Processing workqueue
2026-06-12T16:40:39.805Z	INFO	operator.init.install	install/lifecycle.go:208	Reconciling
2026-06-12T16:40:39.875Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "wasmplugins.extensions.istio.io"}
2026-06-12T16:40:40.021Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "destinationrules.networking.istio.io"}
2026-06-12T16:40:40.097Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "envoyfilters.networking.istio.io"}
2026-06-12T16:40:40.136Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "gateways.networking.istio.io"}
2026-06-12T16:40:40.200Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "proxyconfigs.networking.istio.io"}
2026-06-12T16:40:40.307Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "serviceentries.networking.istio.io"}
2026-06-12T16:40:40.419Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "sidecars.networking.istio.io"}
2026-06-12T16:40:40.531Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "virtualservices.networking.istio.io"}
2026-06-12T16:40:40.563Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "workloadentries.networking.istio.io"}
2026-06-12T16:40:40.610Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "workloadgroups.networking.istio.io"}
2026-06-12T16:40:40.645Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "authorizationpolicies.security.istio.io"}
2026-06-12T16:40:40.726Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "peerauthentications.security.istio.io"}
2026-06-12T16:40:40.753Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "requestauthentications.security.istio.io"}
2026-06-12T16:40:40.805Z	INFO	operator.init.install	cache/controller.go:259	Drift detected (CRD added)	{"watchType": "CRD", "name": "telemetries.telemetry.istio.io"}
2026-06-12T16:40:42.326Z	INFO	operator.init.install	cache/controller.go:266	Drift detected (update)	{"gvk": "apps/v1, Kind=Deployment", "watchType": "Owned", "name": "istiod-openshift-gateway"}
2026-06-12T16:40:42.358Z	INFO	operator.init.install	cache/controller.go:266	Drift detected (update)	{"gvk": "apps/v1, Kind=Deployment", "watchType": "Owned", "name": "istiod-openshift-gateway"}
2026-06-12T16:40:42.369Z	INFO	operator.init.install	install/lifecycle.go:208	Reconcile complete	{"installed": true, "error": null}
2026-06-12T16:40:42.369Z	INFO	operator.init.install	install/lifecycle.go:208	Reconciling

OCPBUGS-85550 -> controllers are started in istiod pod logs

2026-06-12T16:40:47.542267Z	info	controllers	starting	controller=inference pool shadow service reconciler
2026-06-12T16:40:47.542453Z	info	controllers	starting	controller=healthcheck
2026-06-12T16:40:47.542468Z	info	Istiod CA has started
2026-06-12T16:40:47.542480Z	info	controllers	starting	controller=auto-register existing connections
2026-06-12T16:40:47.542490Z	info	controllers	starting	controller=unregister_workloadentry
2026-06-12T16:40:47.542278Z	info	kube	Starting Pilot K8S CRD controller	controller=crd-controller
2026-06-12T16:40:47.542534Z	info	kube	controller "gateway.networking.k8s.io/v1beta1/HTTPRoute" is syncing...	controller=crd-controller
2026-06-12T16:40:47.542302Z	info	klog	attempting to acquire leader lease openshift-ingress/istio-gateway-deployment-openshift-gateway...
2026-06-12T16:40:47.542517Z	info	cluster "Kubernetes" kube client started
2026-06-12T16:40:47.542716Z	info	klog	attempting to acquire leader lease openshift-ingress/istio-gateway-status-leader-openshift-gateway...
2026-06-12T16:40:47.545052Z	info	kube	controller "security.istio.io/v1/PeerAuthentication" is syncing...	controller=crd-controller
2026-06-12T16:40:47.549040Z	info	sync complete	name=multicluster remote secrets attempt=3 time=6.699222ms
2026-06-12T16:40:47.549119Z	info	multicluster remote secrets controller cache synced in 7.201277ms
2026-06-12T16:40:47.549131Z	info	controllers	starting	controller=multicluster secret
2026-06-12T16:40:47.549461Z	info	kube	controller "networking.istio.io/v1alpha3/EnvoyFilter" is syncing...	controller=crd-controller
2026-06-12T16:40:47.556227Z	info	sync complete	name=webhook patcher attempt=4 time=14.116135ms
2026-06-12T16:40:47.556351Z	info	controllers	starting	controller=webhook patcher
2026-06-12T16:40:47.556489Z	info	sync complete	name=configmap istio-sidecar-injector-openshift-gateway attempt=4 time=14.30976ms
2026-06-12T16:40:47.556598Z	info	controllers	starting	controller=configmap istio-sidecar-injector-openshift-gateway
2026-06-12T16:40:47.558345Z	info	kube	controller "security.istio.io/v1/RequestAuthentication" is syncing...	controller=crd-controller
2026-06-12T16:40:47.558356Z	info	sync complete	name=default revision attempt=4 time=16.270313ms
2026-06-12T16:40:47.558520Z	info	controllers	starting	controller=default revision
2026-06-12T16:40:47.556313Z	info	sync complete	name=default revision attempt=4 time=14.073443ms
2026-06-12T16:40:47.559801Z	info	controllers	starting	controller=default revision

/verified by @rhamini3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@candita

candita commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

/label backport-risk-assessed

Some risk is taken on with this backport, but the risk tradeoff is worth the gains of being able to patch OSSM CVES, and fixing several customer bugs related to OLM.

  • There is incomplete upgrade path testing, but the feature is gated.
  • Shipping a feature that can update Istio versions on many z-streams without testing on all z-streams amplifies the incompleteness of the testing.
  • Sail library installation is not well understood within the team and not as mature as OLM installation, increasing the risk of long-duration bug resolution.
    • Sail library installation immaturity means we may need to continuously backport Sail library bug fixes.
  • Shipping a large architectural change in a z-stream can introduce unforeseen compatibility bugs, but this is an accepted risk approved by SBAR.
  • Any pre-existing OLM subscription is left behind, but a new alert will be added prior to the feature being enabled by default.
  • K8s dependency pinning for the Sail library means we risk falling behind in CVE fixes for K8s (or make it more complicated) in older versions, but this is an accepted risk approved by SBAR.
  • Current workarounds for disconnected clusters, and customers with support exceptions related to Gateway API may be impacted.  Testing is ongoing.
  • Users installing Gateway API for the first time may end up starting on z-streams with the OLM issues anyway, but upgrade floors are being assessed so that z-streams that still support OLM are removed from the upgrade graph.
  • Backporting the Sail library installation means that users lose access to troubleshooting via subscription status, but two new GatewayClass conditions are added: ControllerInstalled and CRDsReady to observe installation status and Istio CRD management status.

@gcs278

gcs278 commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

the CI failure for multi-pr job passed our tests, but failed on unrelated : [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a configMap. [Conformance] expand_more. So it was success as far as this PR is concerned.

And e2e-pre-release-ossm doesn't work anymore on the no-OLM path, that will always fail now.

All other current failures are infra failures.

/retest

@gcs278

gcs278 commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

/testwith openshift/cluster-ingress-operator/release-4.21/e2e-aws-operator-techpreview openshift/api#2873

@openshift-merge-bot

openshift-merge-bot Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD ffc48a1 and 2 for PR HEAD 12879f2 in total

@openshift-ci

openshift-ci Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

@gcs278: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-pre-release-ossm 12879f2 link false /test e2e-aws-pre-release-ossm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-ci-robot

openshift-ci-robot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

@gcs278: Jira Issue OCPBUGS-88295: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-88295 has not been moved to the MODIFIED state.

This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload.

Jira Issue OCPBUGS-88297: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-88297 has not been moved to the MODIFIED state.

This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload.

Jira Issue OCPBUGS-82146: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-82146 has not been moved to the MODIFIED state.

This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload.

Jira Issue OCPBUGS-78330: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-78330 has not been moved to the MODIFIED state.

This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload.

Jira Issue OCPBUGS-85550: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-85550 has not been moved to the MODIFIED state.

This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload.

Details

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR Title Why
#1354 NE-2471: Replace OLM-based Istio install with Sail Library Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402 OCPBUGS-79467: Change default log level from DEBUG to INFO Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404 NE-2519: Move Sail Library to official release branch Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

  • pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
  • pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
  • test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
  • go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
  • pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

Phase 2 — TechPreview soak

Phase 3 — GA promotion

Follow-up

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Test Coverage

Early CI test results (OLM, OLM-to-OLM upgrade, noOLM, and OLM-to-noOLM migration) are documented in the origin test backport PR: openshift/origin#31232

Verification

  • go build ./pkg/operator/controller/gatewayclass/... compiles
  • go test ./pkg/operator/controller/gatewayclass/... passes
  • Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidesalerno davidesalerno Awaiting requested review from davidesalerno

@Thealisyed Thealisyed Awaiting requested review from Thealisyed

@alebedev87 alebedev87 Awaiting requested review from alebedev87

@melvinjoseph86 melvinjoseph86 Awaiting requested review from melvinjoseph86

3 more reviewers

@rikatz rikatz rikatz left review comments

@aswinsuryan aswinsuryan aswinsuryan left review comments

@rhamini3 rhamini3 rhamini3 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@rikatz rikatz

@aswinsuryan aswinsuryan

@melvinjoseph86 melvinjoseph86

@rhamini3 rhamini3

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@gcs278 @openshift-ci-robot @bentito @rikatz @aswinsuryan @rhamini3 @candita @melvinjoseph86