OCPBUGS-74970: Fix kubelet certificate wait loop in criometricsproxy.yaml by aksjadha · Pull Request #6125 · openshift/machine-config-operator

aksjadha · 2026-06-03T08:22:17Z

Related bug: Fix kubelet certificate wait loop and mount path in criometricsproxy.yaml
The previous condition [ -n "$(test -e ...)" ] always evaluated to false because test -e produces no stdout output — it communicates via exit code only. So -n always evaluated to false, causing the loop to exit immediately instead of waiting for the kubelet certificate to appear.
The init container mounts the host's /var/lib/kubelet at /var. So inside the init container, the host's /var/lib/kubelet/pki/kubelet-server-current.pem appears at /var/pki/kubelet-server-current.pem — but the script checks /var/lib/kubelet/pki/kubelet-server-current.pem, which doesn't exist at that path inside the init container.

- What I did

Replaced with the correct condition [ ! -e /var/lib/kubelet/pki/kubelet-server-current.pem ], which properly loops until the kubelet certificate file exists
Updated init container mount path to /var/lib/kubelet from /var to match main container so the script's path /var/lib/kubelet/pki/kubelet-server-current.pem resolves correctly.

- How to verify it

Check kube-rbac-proxy-crio-ip pod logs in namespace openshift-machine-config-operator to verify the CRI-O metrics proxy init container correctly waits for the kubelet certificate before proceeding

- Description for the changelog

Fixes the kubelet certificate wait loop condition in criometricsproxy.yaml across all node roles (arbiter, master, worker)
The old condition [ -n "$(test -e /var/lib/kubelet/pki/kubelet-server-current.pem)" ] was incorrect. Replaced with the correct condition [ ! -e /var/lib/kubelet/pki/kubelet-server-current.pem ], which properly loops until the kubelet certificate file exists

Summary by CodeRabbit

Bug Fixes
- Fixed kubelet initialization so nodes now wait for the kubelet server certificate to become available before starting dependent pods, ensuring reliable startup across arbiter, master, and worker nodes.

openshift-merge-bot · 2026-06-03T08:22:19Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-06-03T08:22:24Z

@aksjadha: This pull request references Jira Issue OCPBUGS-74970, which is invalid:

expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Related bug: Fix kubelet certificate wait loop in criometricsproxy.yaml

The previous condition [ -n "$(test -e ...)" ] always evaluated to false because test -e produces no stdout output — it communicates via exit code only. So -n always evaluated to false, causing the loop to exit immediately instead of waiting for the kubelet certificate to appear.

- What I did

Replaced with the correct condition [ ! -e /var/lib/kubelet/pki/kubelet-server-current.pem ], which properly loops until the kubelet certificate file exists

- How to verify it

Check kube-rbac-proxy-crio-ip pod logs in namespace openshift-machine-config-operator to verify the CRI-O metrics proxy init container correctly waits for the kubelet certificate before proceeding

- Description for the changelog

Fixes the kubelet certificate wait loop condition in criometricsproxy.yaml across all node roles (arbiter, master, worker)

The old condition [ -n "$(test -e /var/lib/kubelet/pki/kubelet-server-current.pem)" ] was incorrect. Replaced with the correct condition [ ! -e /var/lib/kubelet/pki/kubelet-server-current.pem ], which properly loops until the kubelet certificate file exists

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-06-03T08:22:41Z

Walkthrough

The PR inverts the kubelet server certificate wait condition across three node configuration templates and updates an arbiter volumeMount path. InitContainers now wait until /var/lib/kubelet/pki/kubelet-server-current.pem exists before proceeding.

Changes

Kubelet Certificate Readiness Wait

Layer / File(s)	Summary
Kubelet certificate readiness condition across node types `templates/arbiter/01-arbiter-kubelet/_base/files/criometricsproxy.yaml`, `templates/master/01-master-kubelet/_base/files/criometricsproxy.yaml`, `templates/worker/01-worker-kubelet/_base/files/criometricsproxy.yaml`	The initContainer wait loop condition is inverted from `while [ -e /var/lib/kubelet/pki/kubelet-server-current.pem ]` to `while [ ! -e /var/lib/kubelet/pki/kubelet-server-current.pem ]`, causing the initContainer to wait until the kubelet server certificate file appears. The arbiter template also changes the `var-lib-kubelet` volumeMount target path to `/var/lib/kubelet`.

🎯 3 (Moderate) | ⏱️ ~20 minutes

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Container-Privileges	❌ Error	Kubernetes manifests contain privileged: true, hostNetwork: true, and openshift.io/required-scc: privileged in all three criometricsproxy.yaml files (arbiter, master, worker).	Review if privileged mode is truly required; consider using explicit capabilities instead of privileged: true where feasible.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and clearly summarizes the main change: fixing the kubelet certificate wait loop in criometricsproxy.yaml files across multiple node roles.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR only modifies YAML configuration files (criometricsproxy.yaml templates), not Ginkgo test files. No test names are affected.
Test Structure And Quality	✅ Passed	PR contains no Ginkgo test code changes; check applies only to test code. PR exclusively modifies YAML configuration files for kubelet certificate wait loop fixes.
Microshift Test Compatibility	✅ Passed	PR adds only YAML template files (criometricsproxy.yaml) fixing kubelet cert wait loop; no new Ginkgo e2e tests are introduced, so MicroShift compatibility check does not apply.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No new Ginkgo e2e tests added. PR only fixes kubelet certificate wait loop in YAML pod manifests, making SNO test compatibility check not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	Bug fix to initContainer wait loop in static Pod manifests with no new scheduling constraints that conflict with SNO, Two-Node, or HyperShift topologies.
Ote Binary Stdout Contract	✅ Passed	PR modifies only YAML Kubernetes pod configuration files, not OTE binary or Go source code. OTE stdout check is not applicable to YAML template changes.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR modifies only YAML configuration files (criometricsproxy.yaml), not Ginkgo e2e tests. Custom check applies only when new e2e tests are added.
No-Weak-Crypto	✅ Passed	PR fixes kubelet certificate wait-loop logic. No weak crypto algorithms, custom crypto implementations, or secret comparison code were introduced.
No-Sensitive-Data-In-Logs	✅ Passed	Logging in the changes only outputs generic status messages, progress dots, and timeout alerts; never exposes sensitive data like passwords, tokens, API keys, PII, or certificate contents.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci · 2026-06-03T08:22:59Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aksjadha
Once this PR has been reviewed and has the lgtm label, please assign mrunalp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aksjadha · 2026-06-03T08:27:00Z

/jira refresh

openshift-ci-robot · 2026-06-03T08:27:05Z

@aksjadha: This pull request references Jira Issue OCPBUGS-74970, which is invalid:

expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "4.23" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

aksjadha · 2026-06-03T08:33:27Z

/jira refresh

openshift-ci-robot · 2026-06-03T08:33:34Z

@aksjadha: This pull request references Jira Issue OCPBUGS-74970, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

… init container's mountPath /var/lib/kubelet[C

openshift-ci-robot · 2026-06-03T14:40:58Z

@aksjadha: This pull request references Jira Issue OCPBUGS-74970, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

Details

In response to this:

Related bug: Fix kubelet certificate wait loop in criometricsproxy.yaml

The previous condition [ -n "$(test -e ...)" ] always evaluated to false because test -e produces no stdout output — it communicates via exit code only. So -n always evaluated to false, causing the loop to exit immediately instead of waiting for the kubelet certificate to appear.

- What I did

Replaced with the correct condition [ ! -e /var/lib/kubelet/pki/kubelet-server-current.pem ], which properly loops until the kubelet certificate file exists

- How to verify it

Check kube-rbac-proxy-crio-ip pod logs in namespace openshift-machine-config-operator to verify the CRI-O metrics proxy init container correctly waits for the kubelet certificate before proceeding

- Description for the changelog

Fixes the kubelet certificate wait loop condition in criometricsproxy.yaml across all node roles (arbiter, master, worker)

The old condition [ -n "$(test -e /var/lib/kubelet/pki/kubelet-server-current.pem)" ] was incorrect. Replaced with the correct condition [ ! -e /var/lib/kubelet/pki/kubelet-server-current.pem ], which properly loops until the kubelet certificate file exists

Summary by CodeRabbit

Bug Fixes

Fixed kubelet initialization so nodes now wait for the kubelet server certificate to become available before starting dependent pods, ensuring reliable startup across arbiter, master, and worker nodes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

aksjadha · 2026-06-04T06:32:49Z

Fix is working as expected.

Before fix, there were multiple restarts of pods as init container was not waiting for file to exist.

$ oc describe pod kube-rbac-proxy-crio-ip -n openshift-machine-config-operator
  containerStatuses:
    lastState:
      terminated:
        message: "W0601 01:32:19.053758       1 deprecated.go:66] \n==== Removed Flag
          Warning ======================\n\nlogtostderr is removed in the k8s upstream
          and has no effect any more.\n\n===============================================\n\t\t\nI0601
          01:32:19.053832       1 kube-rbac-proxy.go:532] Reading config file: /etc/kubernetes/crio-metrics-proxy.cfg\nE0601
          01:32:19.055157       1 run.go:72] \"command failed\" err=\"failed to load
          kubeconfig: unable to build rest config based on provided path to kubeconfig
          file: invalid configuration: [unable to read client-cert /var/lib/kubelet/pki/kubelet-client-current.pem
          for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem:
          no such file or directory, unable to read client-key /var/lib/kubelet/pki/kubelet-client-current.pem
          for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem:
          no such file or directory]\"\n"
        reason: Error
    name: kube-rbac-proxy-crio

$ oc logs kube-rbac-proxy-crio-ip -n openshift-machine-config-operator
I0603 08:59:37.507013       1 kube-rbac-proxy.go:532] Reading config file: /etc/kubernetes/crio-metrics-proxy.cfg
I0603 08:59:37.508125       1 kube-rbac-proxy.go:235] Valid token audiences: 
I0603 08:59:37.508369       1 dynamic_cafile_content.go:161] "Starting controller" name="client-ca::/etc/kubernetes/kubelet-ca.crt"
I0603 08:59:37.510016       1 kube-rbac-proxy.go:349] Reading certificate files
E0603 08:59:37.510065       1 run.go:72] "command failed" err="failed to initialize certificate reloader: error loading certificates: error loading certificate: open /var/lib/kubelet/pki/kubelet-server-current.pem: no such file or directory"

With fix, init container checking if file exists and checking correct mount path i,e /var/lib/kubelet/, there are no multiple restart.

NAME                                                             READY   STATUS    RESTARTS   AGE
kube-rbac-proxy-crio-ip-11.us-west-2.compute.internal   1/1     Running   0          67m
kube-rbac-proxy-crio-ip-12.us-west-2.compute.internal    1/1     Running   0          130m
kube-rbac-proxy-crio-ip-13.us-west-2.compute.internal   1/1     Running   0          130m
kube-rbac-proxy-crio-ip-14.us-west-2.compute.internal     1/1     Running   0          130m

openshift-ci · 2026-06-04T06:59:30Z

@aksjadha: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ngopalak-redhat · 2026-06-04T08:25:57Z

        - |
          echo -n "Waiting for kubelet key and certificate to be available"
-          while [ -n "$(test -e /var/lib/kubelet/pki/kubelet-server-current.pem)" ] ; do
+          while [ ! -e /var/lib/kubelet/pki/kubelet-server-current.pem ] ; do


The tries variable is incremented but never checked, so the loop could run forever if the certificate never appears.

ngopalak-redhat · 2026-06-04T09:02:29Z

        volumeMounts:
        - name: var-lib-kubelet
-          mountPath: "/var"
+          mountPath: "/var/lib/kubelet"


Can you also add readOnly:true. In my understanding it just reads the cert

openshift-ci Bot requested review from cheesesashimi and pablintino June 3, 2026 08:23

aksjadha marked this pull request as draft June 3, 2026 08:31

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 3, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 3, 2026

openshift-ci Bot requested a review from sergiordlr June 3, 2026 08:33

Fix kubelet certificate wait loop in criometricsproxy.yaml and update…

ee0dcdd

… init container's mountPath /var/lib/kubelet[C

aksjadha force-pushed the fix-OCPBUGS-74970 branch from af45f24 to ee0dcdd Compare June 3, 2026 14:39

aksjadha marked this pull request as ready for review June 4, 2026 06:33

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 4, 2026

openshift-ci Bot requested review from isabella-janssen and yuqi-zhang June 4, 2026 06:33

ngopalak-redhat suggested changes Jun 4, 2026

View reviewed changes

Conversation

aksjadha commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented Jun 3, 2026

Uh oh!

openshift-ci-robot commented Jun 3, 2026

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Pre-merge checks failed

❌ Failed checks (1 error)

Uh oh!

openshift-ci Bot commented Jun 3, 2026

Uh oh!

aksjadha commented Jun 3, 2026

Uh oh!

openshift-ci-robot commented Jun 3, 2026

Uh oh!

aksjadha commented Jun 3, 2026

Uh oh!

openshift-ci-robot commented Jun 3, 2026

Uh oh!

openshift-ci-robot commented Jun 3, 2026

Summary by CodeRabbit

Uh oh!

aksjadha commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Jun 4, 2026

Uh oh!

ngopalak-redhat Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

ngopalak-redhat Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aksjadha commented Jun 3, 2026 •

edited

Loading

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading

aksjadha commented Jun 4, 2026 •

edited

Loading