Regression in 36.0.0: in-cluster config no longer attaches Authorization header (every request 401s)

[risjai]

Summary

Starting in kubernetes==36.0.0, clients configured via config.load_incluster_config() send HTTP requests without an Authorization header. Every API call from inside a pod therefore returns 401 Unauthorized. The same code path works correctly in 35.0.0 with the same service account token, the same cluster, and the same network setup.

Reproducer

Run the following inside an in-cluster pod (e.g. an EKS pod with a mounted service account token):

from kubernetes import config, client
import urllib3

# Patch urllib3 to trace requests
orig_urlopen = urllib3.PoolManager.urlopen
def traced(self, method, url, **kw):
    print("method:", method)
    print("url:", url)
    print("headers:", dict(kw.get("headers") or {}))
    return orig_urlopen(self, method, url, **kw)
urllib3.PoolManager.urlopen = traced

config.load_incluster_config()
api_client = client.ApiClient()
client.VersionApi(api_client).get_code()

Observed output (kubernetes 36.0.0)

method: GET
url: https://172.20.0.1:443/version/
headers: {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/36.0.0/python', 'Content-Type': 'application/json'}

Note: no Authorization header is set. The server then responds 401 Unauthorized.

Expected output (kubernetes 35.0.0, same code, same token)

method: GET
url: https://172.20.0.1:443/version/
headers: {'Accept': 'application/json', 'Authorization': 'Bearer <redacted>', 'User-Agent': 'OpenAPI-Generator/35.0.0/python', 'Content-Type': 'application/json'}

Returns 200 OK.

Confirmed not a token / RBAC / network issue

The same SA token works in every other context:

1. curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $TOKEN" https://172.20.0.1:443/version → 200.
2. urllib3.PoolManager().request(...) directly with the same token → 200.
3. Pinning kubernetes<36 (resolves to 35.0.0) with no other changes → 200.

The token itself is valid: decoded JWT shows iat ~15 minutes ago, exp ~365 days out, aud=['https://kubernetes.default.svc']. RBAC is correct — curl with the same token returns 200 against /api/v1/namespaces/{ns}/pods.

Workarounds verified

• Pin kubernetes<36 (resolves to 35.0.0).
• Use urllib3 directly with the bearer token from /var/run/secrets/kubernetes.io/serviceaccount/token.
• Manually set the header on the ApiClient configuration after load_incluster_config().

Suspected area

The BearerToken auth setting plumbing in kubernetes/client/api_client.py (or whatever was touched between 35.0.0 and 36.0.0 along the auth / ApiClient path). It looks like load_incluster_config() is populating the configuration with a key that the generated auth_settings=["BearerToken"] lookup in ApiClient no longer reads, so the header is silently dropped.

Environment

• kubernetes (PyPI): 36.0.0
• urllib3: 1.26.12
• Python: 3.11
• Cluster: EKS, Kubernetes 1.32.13-eks
• Discovered: 2026-05-22

Impact

Any production system using kubernetes-client/python from inside a Kubernetes pod will start failing the moment pip resolves to 36.0.0. This includes operators, controllers, and any in-cluster tooling that calls load_incluster_config(). We hit it on a DynamicClient(api_client) discovery call at startup; consumers calling any other API will hit the same 401.

Note

36.0.0 was released approximately 2026-05-21 (one day before this report). I did a quick gh issue list ... --search "401 in-cluster" and didn't find a clean duplicate, but happy to consolidate if maintainers identify one.

[jbothma]

I was seeing

  File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/kubernetes/client/rest.py", line 232, in request
    raise ForbiddenException(http_resp=r)
kubernetes.client.exceptions.ForbiddenException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '...', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '...', 'X-Kubernetes-Pf-Prioritylevel-Uid': '...', 'Date': 'Fri, 22 May 2026 07:50:38 GMT', 'Content-Length': '300'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"cronjobs.batch is forbidden: User \"system:anonymous\" cannot list resource \"cronjobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"cronjobs"},"code":403}

Pinning to <36 fixed it.

The User \"system:anonymous\" straight after google-github-actions/get-gke-credentials@v3 was what eventually led me here.

[aojea]

Looks like this has been fixed by #2585

/cc @yliaog

[MarquiseRosier]

Hey guys when can we expect the release

[yliaog]

I'm going to cut a new release today

[yliaog]

@roycaihw is not available today, we may need to wait another day or two, sorry for the delay.