Skip to content

OCPBUGS-88531: Propagate restart-date annotation to CNO operand pod templates#3030

Open
bryan-cox wants to merge 1 commit into
openshift:masterfrom
bryan-cox:OCPBUGS-88531
Open

OCPBUGS-88531: Propagate restart-date annotation to CNO operand pod templates#3030
bryan-cox wants to merge 1 commit into
openshift:masterfrom
bryan-cox:OCPBUGS-88531

Conversation

@bryan-cox

@bryan-cox bryan-cox commented Jun 15, 2026

Copy link
Copy Markdown
Member

Summary

When the hypershift.openshift.io/restart-date annotation is set on a HostedControlPlane CR, CNO-managed operands like cloud-network-config-controller were not being restarted. This happened because CNO uses server-side apply with deterministic manifests — applying the same manifest twice is a no-op, so operands don't roll out unless something in the pod template changes.

This PR fixes the issue by:

  • Reading the hypershift.openshift.io/restart-date annotation from the HostedControlPlane CR in ParseHostedControlPlane()
  • Adding a SetRestartDateAnnotation() helper that injects the restart-date into pod template annotations of all rendered Deployments, DaemonSets, and StatefulSets
  • Calling this helper in the reconciler after Render() returns, gated on HyperShift mode and a non-empty restart-date value

This follows the existing patterns in CNO:

  • Reading HCP annotations in ParseHostedControlPlane (like control-plane-priority-class)
  • Modifying rendered objects post-Render (like setOVNObjectAnnotation)

This is the architecturally correct location for this logic since CNO owns these operands and should be responsible for their lifecycle.

Test plan

  • Unit tests for ParseHostedControlPlane with restart-date annotation
  • Unit tests for SetRestartDateAnnotation covering Deployments, DaemonSets, StatefulSets, non-apps objects, annotation preservation, and multi-object handling
  • go build ./... passes
  • go test ./pkg/... ./cmd/... passes
  • go vet passes
  • Manual e2e: Set hypershift.openshift.io/restart-date annotation on an HCP and verify cloud-network-config-controller pods restart

Fixes: https://issues.redhat.com/browse/OCPBUGS-88531

🤖 Generated with Claude Code via /jira:solve OCPBUGS-88531

Summary by CodeRabbit

Release Notes

  • New Features

    • Added restart date annotation support for HyperShift control planes. The system now reads a restart date from control plane metadata and propagates it to provisioned Deployments, DaemonSets, and StatefulSets.
  • Bug Fixes

    • Improved reconciliation behavior by failing fast if restart-date annotation updates cannot be applied, marking the operator as degraded instead of continuing.
  • Tests

    • Added unit tests covering restart date parsing and propagation across supported controller kinds.

@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 15, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@bryan-cox: This pull request references Jira Issue OCPBUGS-88531, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

When the hypershift.openshift.io/restart-date annotation is set on a HostedControlPlane CR, CNO-managed operands like cloud-network-config-controller were not being restarted. This happened because CNO uses server-side apply with deterministic manifests — applying the same manifest twice is a no-op, so operands don't roll out unless something in the pod template changes.

This PR fixes the issue by:

  • Reading the hypershift.openshift.io/restart-date annotation from the HostedControlPlane CR in ParseHostedControlPlane()
  • Adding a SetRestartDateAnnotation() helper that injects the restart-date into pod template annotations of all rendered Deployments, DaemonSets, and StatefulSets
  • Calling this helper in the reconciler after Render() returns, gated on HyperShift mode and a non-empty restart-date value

This follows the existing patterns in CNO:

  • Reading HCP annotations in ParseHostedControlPlane (like control-plane-priority-class)
  • Modifying rendered objects post-Render (like setOVNObjectAnnotation)

This is the architecturally correct location for this logic since CNO owns these operands and should be responsible for their lifecycle.

Test plan

  • Unit tests for ParseHostedControlPlane with restart-date annotation
  • Unit tests for SetRestartDateAnnotation covering Deployments, DaemonSets, StatefulSets, non-apps objects, annotation preservation, and multi-object handling
  • go build ./... passes
  • go test ./pkg/... ./cmd/... passes
  • go vet passes
  • Manual e2e: Set hypershift.openshift.io/restart-date annotation on an HCP and verify cloud-network-config-controller pods restart

Fixes: https://issues.redhat.com/browse/OCPBUGS-88531

🤖 Generated with Claude Code via /jira:solve OCPBUGS-88531

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cee400f6-50b3-43a1-a67b-d2c1b4be8cad

📥 Commits

Reviewing files that changed from the base of the PR and between d807287 and aab1f0e.

📒 Files selected for processing (3)
  • pkg/controller/operconfig/operconfig_controller.go
  • pkg/hypershift/hypershift.go
  • pkg/hypershift/hypershift_test.go
🚧 Files skipped from review as they are similar to previous changes (3)
  • pkg/controller/operconfig/operconfig_controller.go
  • pkg/hypershift/hypershift.go
  • pkg/hypershift/hypershift_test.go

Walkthrough

Adds a RestartDateAnnotation constant and RestartDate field to HostedControlPlane, extends ParseHostedControlPlane to read the annotation, and introduces SetRestartDateAnnotation to stamp the value onto apps/v1 workload pod-template annotations. The reconciler calls this helper after rendering when RestartDate is non-empty, aborting on error.

Changes

HyperShift restart-date annotation propagation

Layer / File(s) Summary
Annotation constant, struct field, parsing, and setter
pkg/hypershift/hypershift.go, pkg/hypershift/hypershift_test.go
Adds RestartDateAnnotation constant, RestartDate string field to HostedControlPlane, extends ParseHostedControlPlane to read the annotation into the new field, and adds SetRestartDateAnnotation that filters rendered objects to apps/v1 Deployments/DaemonSets/StatefulSets and writes the annotation into spec.template.metadata.annotations. Unit tests cover parsing, per-kind annotation setting, skipping non-apps/v1 objects, preserving existing annotations, and mixed-object lists.
Reconciler wiring
pkg/controller/operconfig/operconfig_controller.go
After rendering, conditionally calls SetRestartDateAnnotation when HostedControlPlane.RestartDate is non-empty; on failure logs the error, records RenderError degraded condition, and returns early.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ⚠️ Warning Tests lack assertion messages: 34 Expect() calls without descriptive failure messages violates requirement #4. Good example: Expect(err).NotTo(HaveOccurred(), "meaningful message") Add descriptive messages to all Expect() calls to help diagnose failures. Pattern: Expect(err).NotTo(HaveOccurred(), "description of what should happen")
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly summarizes the main change: propagating the restart-date annotation to operand pod templates, which is the core purpose of this PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All test names in the PR are stable and deterministic. No Ginkgo tests are present (check applies to standard Go tests). All 12 test names use static, descriptive strings with no dynamic values, ti...
Microshift Test Compatibility ✅ Passed The PR adds only unit tests (TestParseHostedControlPlaneRestartDate, TestSetRestartDateAnnotation) in pkg/hypershift/hypershift_test.go, not Ginkgo e2e tests. No e2e tests with Ginkgo patterns (It(...
Single Node Openshift (Sno) Test Compatibility ✅ Passed No Ginkgo e2e tests were added. The PR only adds standard Go unit tests (TestParseHostedControlPlaneRestartDate and TestSetRestartDateAnnotation) that use testing.T and Gomega for assertions, not G...
Topology-Aware Scheduling Compatibility ✅ Passed PR only adds annotation propagation logic with no scheduling constraints. SetRestartDateAnnotation() simply reads a restart-date value and writes it to pod template annotations—no affinity, topolog...
Ote Binary Stdout Contract ✅ Passed PR modifies standard operator controller code and unit tests, not OTE test extension code. No stdout writes in process-level code; log.Printf safely writes to stderr.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e tests are added in this PR. The test additions are standard Go unit tests using *testing.T, not Ginkgo framework, so the check does not apply.
No-Weak-Crypto ✅ Passed PR contains no weak cryptographic operations, custom crypto implementations, or non-constant-time secret comparisons. Changes only involve reading/writing timestamp annotations to Kubernetes objects.
Container-Privileges ✅ Passed No container privilege escalation settings found. PR only adds annotation metadata to pod templates for triggering restarts via kubernetes rolling update mechanism.
No-Sensitive-Data-In-Logs ✅ Passed No sensitive data is logged. RestartDate (a timestamp) is passed to SetRestartDateAnnotation but never logged directly; error messages only include object kind/name, not the timestamp value.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="Running error: context loading failed: failed to load packages: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: inconsistent vendoring in :\n\tgithub.com/Masterminds/semver@v1.5.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/Masterminds/sprig/v3@v3.2.3: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/containernetworking/cni@v0.8.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/ghodss/yaml@v1.0.1-0.20190212211648-25d852aebe32: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/go-bindata/go-bindata@v3.1.2+incompatible: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/onsi/gomega@v1.39.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/ope

... [truncated 17357 characters] ...

red in go.mod, but not marked as explicit in vendor/modules.txt\n\tk8s.io/gengo/v2@v2.0.0-20251215205346-5ee0d033ba5b: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tk8s.io/kms@v0.35.2: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tk8s.io/kube-aggregator@v0.35.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tsigs.k8s.io/randfill@v1.0.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tsigs.k8s.io/structured-merge-diff/v6@v6.3.2: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\n\tTo ignore the vendor directory, use -mod=readonly or -mod=mod.\n\tTo sync the vendor directory, run:\n\t\tgo mod vendor\n"


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 15, 2026
@openshift-ci

openshift-ci Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/hypershift/hypershift_test.go`:
- Around line 154-155: The test code is discarding error returns by using blank
identifiers (_) instead of capturing them, which can mask test failures and
produce misleading passes. Identify all instances in the test file where
function calls return errors that are being ignored (such as calls to
unstructured.NestedStringMap and similar utility functions), capture the
returned error values instead of discarding them with _, and add appropriate
assertions or error handling using the test assertion framework (e.g.,
g.Expect(err).NotTo(HaveOccurred())) to verify these operations succeed as
expected during test setup and assertions.

In `@pkg/hypershift/hypershift.go`:
- Line 291: The call to unstructured.NestedStringMap is discarding the error
return value (the third return value) by using a blank identifier, which means
any errors from malformed annotations are silently ignored. Instead of ignoring
the error, capture it as a named variable in the assignment to
unstructured.NestedStringMap, check if the error is non-nil, and handle it
appropriately before proceeding to mutate the pod-template annotations. This
ensures that rendering defects in the annotations are surfaced rather than
silently hidden.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 37272bb8-3d49-4c7b-9530-3c9966bb18b4

📥 Commits

Reviewing files that changed from the base of the PR and between 6dc1804 and d807287.

📒 Files selected for processing (3)
  • pkg/controller/operconfig/operconfig_controller.go
  • pkg/hypershift/hypershift.go
  • pkg/hypershift/hypershift_test.go

Comment thread pkg/hypershift/hypershift_test.go Outdated
Comment thread pkg/hypershift/hypershift.go Outdated
@bryan-cox bryan-cox marked this pull request as ready for review June 15, 2026 16:42
@bryan-cox

Copy link
Copy Markdown
Member Author

/jira refresh

@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 15, 2026
@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 15, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@bryan-cox: This pull request references Jira Issue OCPBUGS-88531, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

…emplates

When the hypershift.openshift.io/restart-date annotation is set on a
HostedControlPlane, CNO now reads it and injects it into the pod
template annotations of all rendered Deployments, DaemonSets, and
StatefulSets. This triggers Kubernetes rolling restarts for all CNO
operands, including cloud-network-config-controller which was
previously missed.

The fix follows the existing pattern of reading HCP annotations in
ParseHostedControlPlane (like priority-class) and modifying rendered
objects post-Render (like setOVNObjectAnnotation). This is the correct
architectural location for this logic since CNO owns these operands.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@bryan-cox: This pull request references Jira Issue OCPBUGS-88531, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Summary

When the hypershift.openshift.io/restart-date annotation is set on a HostedControlPlane CR, CNO-managed operands like cloud-network-config-controller were not being restarted. This happened because CNO uses server-side apply with deterministic manifests — applying the same manifest twice is a no-op, so operands don't roll out unless something in the pod template changes.

This PR fixes the issue by:

  • Reading the hypershift.openshift.io/restart-date annotation from the HostedControlPlane CR in ParseHostedControlPlane()
  • Adding a SetRestartDateAnnotation() helper that injects the restart-date into pod template annotations of all rendered Deployments, DaemonSets, and StatefulSets
  • Calling this helper in the reconciler after Render() returns, gated on HyperShift mode and a non-empty restart-date value

This follows the existing patterns in CNO:

  • Reading HCP annotations in ParseHostedControlPlane (like control-plane-priority-class)
  • Modifying rendered objects post-Render (like setOVNObjectAnnotation)

This is the architecturally correct location for this logic since CNO owns these operands and should be responsible for their lifecycle.

Test plan

  • Unit tests for ParseHostedControlPlane with restart-date annotation
  • Unit tests for SetRestartDateAnnotation covering Deployments, DaemonSets, StatefulSets, non-apps objects, annotation preservation, and multi-object handling
  • go build ./... passes
  • go test ./pkg/... ./cmd/... passes
  • go vet passes
  • Manual e2e: Set hypershift.openshift.io/restart-date annotation on an HCP and verify cloud-network-config-controller pods restart

Fixes: https://issues.redhat.com/browse/OCPBUGS-88531

🤖 Generated with Claude Code via /jira:solve OCPBUGS-88531

Summary by CodeRabbit

Release Notes

  • New Features

  • Added restart date annotation support for HyperShift control planes. The system now reads a restart date from control plane metadata and propagates it to provisioned Deployments, DaemonSets, and StatefulSets.

  • Bug Fixes

  • Improved reconciliation behavior by failing fast if restart-date annotation updates cannot be applied, marking the operator as degraded instead of continuing.

  • Tests

  • Added unit tests covering restart date parsing and propagation across supported controller kinds.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot requested review from kyrtapz and marty-power June 15, 2026 16:49
@bryan-cox

Copy link
Copy Markdown
Member Author

/retest-required

@bryan-cox

Copy link
Copy Markdown
Member Author

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 16, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@bryan-cox: This PR has been marked as verified by @bryan-cox.

Details

In response to this:

/verified by @bryan-cox

https://bryan-cox.github.io/architectural-artifact-sharing/test-verification-report-ocpbugs-88531/

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bryan-cox

Copy link
Copy Markdown
Member Author

/retest

@openshift-ci

openshift-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@bryan-cox: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-upgrade aab1f0e link true /test e2e-aws-ovn-upgrade
ci/prow/e2e-metal-ipi-ovn-dualstack-bgp aab1f0e link true /test e2e-metal-ipi-ovn-dualstack-bgp
ci/prow/e2e-ovn-ipsec-step-registry aab1f0e link true /test e2e-ovn-ipsec-step-registry
ci/prow/e2e-aws-ovn-upgrade-ipsec aab1f0e link true /test e2e-aws-ovn-upgrade-ipsec
ci/prow/e2e-gcp-ovn aab1f0e link true /test e2e-gcp-ovn
ci/prow/e2e-metal-ipi-ovn-ipv6-ipsec aab1f0e link true /test e2e-metal-ipi-ovn-ipv6-ipsec
ci/prow/e2e-metal-ipi-ovn-dualstack-bgp-local-gw aab1f0e link true /test e2e-metal-ipi-ovn-dualstack-bgp-local-gw
ci/prow/5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade aab1f0e link false /test 5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.


restartDate, _, err := unstructured.NestedString(hcp.UnstructuredContent(), "metadata", "annotations", RestartDateAnnotation)
if err != nil {
return nil, fmt.Errorf("failed to extract restart date annotation: %v", err)

@mgencur mgencur Jun 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: It would be better to use %w to preserve the error chain but since all other calls in this file use %v let's keep it.

@mgencur

mgencur commented Jun 18, 2026

Copy link
Copy Markdown

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 18, 2026
@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bryan-cox, mgencur
Once this PR has been reviewed and has the lgtm label, please assign danwinship for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants