EKS Anywhere 0.25.3 upgrade stalls because kubeadm omits kubeadm:apiserver-kubelet-client ClusterRoleBinding on upgraded workload clusters

[mariuszadamski-ca]

What happened:

When upgrading a workload cluster with stacked etcd from EKS Anywhere v0.24.4 to v0.25.3 (Kubernetes v1.34 → v1.35), the upgrade process gets stuck when rolling out new control plane nodes.

All control plane Machines report Unknown status

$ kubectl get ma -A -l cluster.x-k8s.io/cluster-name=eksa-w01
NAMESPACE     NAME                        CLUSTER    NODE NAME                   READY     AVAILABLE   UP-TO-DATE   PHASE     AGE     VERSION
eksa-system   eksa-w01-cppng              eksa-w01   eksa-w01-cppng              Unknown   False       False        Running   69m     v1.34.3-eks-1-34-13
eksa-system   eksa-w01-md-0-8gpmr-crglq   eksa-w01   eksa-w01-md-0-8gpmr-crglq   True      True        True         Running   69m     v1.34.3-eks-1-34-13
eksa-system   eksa-w01-md-0-8gpmr-shfx7   eksa-w01   eksa-w01-md-0-8gpmr-shfx7   True      True        True         Running   69m     v1.34.3-eks-1-34-13
eksa-system   eksa-w01-tn5dz              eksa-w01   eksa-w01-tn5dz              Unknown   False       False        Running   67m     v1.34.3-eks-1-34-13
eksa-system   eksa-w01-zzgfb              eksa-w01   eksa-w01-zzgfb              Unknown   False       True         Running   2m48s   v1.35.6-eks-1-35-11

with condition:

- lastTransitionTime: "2026-08-17T15:28:28Z"
  message: '* EtcdMemberHealthy: Failed to connect to etcd: failed to get etcd status:
    context deadline exceeded'
  observedGeneration: 3
  reason: ReadyUnknown
  status: Unknown
  type: Ready

Simultaneously, I can see in the capi-kubeadm-control-plane-controller-manager logs

{
  "level": "warn",
  "ts": "2026-08-17T15:35:35.672791Z",
  "caller": "v3@v3.6.6/retry_interceptor.go:65",
  "msg": "retrying of unary invoker failed",
  "target": "etcd-endpoints://0xc00141c5a0/etcd-eksa-w01-cppng",
  "method": "/etcdserverpb.Maintenance/Status",
  "attempt": 0,
  "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: error upgrading connection: unable to upgrade connection: Forbidden (user=kube-apiserver-kubelet-client, verb=create, resource=nodes, subresource(s)=[proxy])\""
}

with the authorization issue clearly visible.

Root Cause Analysis

During upgrade to Kubernetes 1.35, apiserver-kubelet-client.crt no longer contains group kubeadm:cluster-admins:

root@eksa-w01-zzgfb:~# openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -subject
subject=CN = kube-apiserver-kubelet-client

(previously it was subject=O = kubeadm:cluster-admins, CN = kube-apiserver-kubelet-client).

Existing upgraded clusters therefore lose authorization through the old kubeadm:cluster-admins binding (note: the Ubuntu image built for Kubernetes v1.35.2 and EKS Anywhere v0.25.2 is not afected by this). As a result, the kube-apiserver-kubelet-client user used by the KubeadmControlPlane controller health checks loses the ability to tunnel the calls through API server proxy:

$ kubectl auth can-i create nodes --subresource proxy --as=kube-apiserver-kubelet-client --all-namespaces
no

Fresh 1.35 clusters receive kubeadm:apiserver-kubelet-client ClusterRoleBinding, which directly binds the user to system:kubelet-api-admin, so the above command returns yes.

This might be related to a Cluster API issue: kubernetes-sigs/cluster-api#13770

What you expected to happen:

Upgrading an existing workload cluster must create or reconcile kubeadm:apiserver-kubelet-client before control-plane health checks run.

How to reproduce it (as minimally and precisely as possible):

1. Deploy an EKS Anywhere management cluster on v0.24.4.
2. Create a workload cluster with stacked etcd (3 control plane nodes).
3. Upgrade the management cluster to v0.25.3.
4. Upgrade the workload cluster spec to v0.25.3 (Kubernetes v1.35).
5. Observe the upgrade stalling while rolling out new control plane nodes.

Anything else we need to know?:

Applying the missing ClusterRoleBinding directly on the workload cluster immediately unblocks CAPI reconciliation and allows the upgrade to finish:

$ kubectl --kubeconfig <workload-cluster-kubeconfig> apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:apiserver-kubelet-client
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kubelet-api-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: kube-apiserver-kubelet-client
EOF

Environment:

• EKS Anywhere Source Version: v0.24.4 (Kubernetes v1.34.3)
• EKS Anywhere Target Version: v0.25.3 (Kubernetes v1.35.6)
• Infrastructure Provider: vSphere
• OS / Node Image: Ubuntu (built with EKS-A imagebuilder, --eksa-release v0.25.3), Bottlerocket (Ova image for EKS-D 1-35-11 release)
• etcd Topology: Stacked