Skip to content

Enable watch termination grace period (redux).#2200

Draft
benluddy wants to merge 1 commit into
openshift:mainfrom
benluddy:graceful-watch-termination-2026
Draft

Enable watch termination grace period (redux).#2200
benluddy wants to merge 1 commit into
openshift:mainfrom
benluddy:graceful-watch-termination-2026

Conversation

@benluddy

@benluddy benluddy commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

This should mostly eliminate aligned informer re-lists during kube-apiserver rollout.

Summary by CodeRabbit

  • Chores
    • Added new default configuration setting for API server shutdown termination grace period.

This should mostly eliminate aligned informer re-lists during kube-apiserver rollout.
@openshift-ci

openshift-ci Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 16, 2026
@coderabbitai

coderabbitai Bot commented Jun 16, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 03cbad4c-0437-483e-b8ff-41e5868d985f

📥 Commits

Reviewing files that changed from the base of the PR and between 8656326 and d17b58a.

📒 Files selected for processing (1)
  • bindata/assets/config/defaultconfig.yaml

Walkthrough

Adds a new shutdown-watch-termination-grace-period entry with value 5s under apiServerArguments in bindata/assets/config/defaultconfig.yaml for the KubeAPIServerConfig default configuration.

Changes

KubeAPIServer Default Config Update

Layer / File(s) Summary
Add shutdown-watch-termination-grace-period to apiServerArguments
bindata/assets/config/defaultconfig.yaml
Inserts shutdown-watch-termination-grace-period: 5s under apiServerArguments in the KubeAPIServerConfig default configuration.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Microshift Test Compatibility ⚠️ Warning Multiple new Ginkgo e2e tests use MicroShift-incompatible APIs without proper protection labels: operator.go uses ClusterOperator (config.openshift.io/v1) and openshift-kube-apiserver-operator name... Add [apigroup:config.openshift.io] tag to tests using ClusterOperator, or [Skipped:MicroShift] label to tests accessing openshift-kube-apiserver/openshift-kube-apiserver-operator namespaces. See custom check for detailed guidance.
Ipv6 And Disconnected Network Test Compatibility ⚠️ Warning test/e2e/user_certs.go contains a Ginkgo test with hardcoded IPv4 localhost address "127.0.0.1" which will fail in IPv6-only disconnected environments. Replace hardcoded "127.0.0.1" with IPv6-compatible loopback address "::1" or dynamically detect IPv4/IPv6 using GetIPAddressFamily() helper function from openshift/origin library.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Enable watch termination grace period (redux)' accurately summarizes the main change of adding a new apiServerArguments setting for shutdown-watch-termination-grace-period to the default configuration.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed This PR contains only configuration changes to defaultconfig.yaml; no Ginkgo tests were added or modified, making the check not applicable to this PR.
Test Structure And Quality ✅ Passed This PR is a configuration-only change (+2/-0 lines to defaultconfig.yaml) per the raw summary. No Ginkgo test code was modified in this PR—the test files present are part of the initial repository...
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR is a configuration-only change to defaultconfig.yaml; no new Ginkgo e2e tests are added, so SNO test compatibility check does not apply.
Topology-Aware Scheduling Compatibility ✅ Passed This PR only adds a configuration parameter (shutdown-watch-termination-grace-period) to defaultconfig.yaml. The check applies to deployment manifests, operator code, or controller modifications—th...
Ote Binary Stdout Contract ✅ Passed OTE main.go uses cli.Run() from k8s.io/component-base for stdout handling, klog.Fatal() for errors (stderr by default), and bindata/assets.go contains only asset utilities with no stdout writes. No...
No-Weak-Crypto ✅ Passed PR adds only a configuration parameter shutdown-watch-termination-grace-period: 5s to defaultconfig.yaml. No code implementing weak cryptographic algorithms (MD5, SHA1, DES, RC4, 3DES, Blowfish,...
Container-Privileges ✅ Passed PR change is only in config file (not K8s manifests); all existing container manifests have strong security: non-root user, allowPrivilegeEscalation:false, no privileged/host/SYS_ADMIN settings.
No-Sensitive-Data-In-Logs ✅ Passed The PR adds a single non-sensitive configuration parameter (shutdown-watch-termination-grace-period=5s) for graceful kube-apiserver termination. No passwords, tokens, API keys, PII, or other sensit...

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci

openshift-ci Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign p0lyn0mial for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@benluddy

Copy link
Copy Markdown
Contributor Author

/payload-aggregate periodic-ci-openshift-release-master-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade 10

@openshift-ci

openshift-ci Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

@benluddy: trigger 0 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

@benluddy

Copy link
Copy Markdown
Contributor Author

/payload 5.0 ci blocking

@openshift-ci

openshift-ci Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

@benluddy: trigger 5 job(s) of type blocking for the ci release of OCP 5.0

  • periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade
  • periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-azure-ovn-upgrade
  • periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-upgrade
  • periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aks
  • periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/50caa6f0-69c8-11f1-94f7-de45d6e2e667-0

@benluddy

Copy link
Copy Markdown
Contributor Author

xref #1862 (comment)

@benluddy

Copy link
Copy Markdown
Contributor Author

From https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-cluster-kube-apiserver-operator-2200-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade/2066992765111111680, it appears the disruption samplers don't honor 429 responses with Retry-After the same way client-go's REST clients do. Since this change is allowing a few seconds to drain watches after the shutdown delay duration has elapsed, we're hitting the WithRetryAfter filter that is meant to shunt clients to other API servers.

@benluddy

Copy link
Copy Markdown
Contributor Author

/payload-with-prs 5.0 ci blocking openshift/origin#31311

@openshift-ci

openshift-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@benluddy: trigger 5 job(s) of type blocking for the ci release of OCP 5.0

  • periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade
  • periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-azure-ovn-upgrade
  • periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-upgrade
  • periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aks
  • periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ac034dc0-6a58-11f1-9862-86893dca838b-0

@sdodson

sdodson commented Jun 17, 2026

Copy link
Copy Markdown
Member

/test images

shutdown-send-retry-after:
- "true"
shutdown-watch-termination-grace-period:
- 5s # shorter than the pod graceful termination period on single-node

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Being higher extends the overall termination period? I guess if we find that 5s is insufficient here we can explore something like the controller that adjusts goaway-chance based on SNO vs HA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants