Skip to content

fix(opentelemetry-collector): add rbac permissions to create tokenreviews#2159

Open
ekeih wants to merge 1 commit intoopen-telemetry:mainfrom
ekeih:rbac-tokenreviews
Open

fix(opentelemetry-collector): add rbac permissions to create tokenreviews#2159
ekeih wants to merge 1 commit intoopen-telemetry:mainfrom
ekeih:rbac-tokenreviews

Conversation

@ekeih
Copy link
Copy Markdown

@ekeih ekeih commented Apr 20, 2026

Hi 👋

After upgrading to 0.110.0 of the opentelemetry-operator chart the opentelemetry-operator pods started to log an error each time their /metrics endpoint was scraped and a 500 status code is returned:

{"level":"ERROR","timestamp":"2026-04-20T15:37:48Z","logger":"controller-runtime.metrics","message":"Authentication failed","path":"/metrics","error":"tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:opentelemetry:opentelemetry-operator" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope","errorCauses":[{"error":"tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:opentelemetry:opentelemetry-operator" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/metrics/filters.WithAuthenticationAndAuthorization.func1.1\n\tsigs.k8s.io/controller-runtime@v0.22.5/pkg/metrics/filters/filters.go:89\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2861\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:3340\nnet/http.initALPNRequest.ServeHTTP\n\tnet/http/server.go:4013\nnet/http.(*http2serverConn).runHandler\n\tnet/http/h2_bundle.go:6386"}

Looking through the changelogs and PR history I think #2153 might have removed more than intended.

As a workaround I added the following in our setup which fixed the issue:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: opentelemetry-operator-tokenreview
rules:
  - apiGroups: ["authentication.k8s.io"]
    resources: ["tokenreviews"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: opentelemetry-operator-tokenreview
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: opentelemetry-operator-tokenreview
subjects:
  - kind: ServiceAccount
    name: opentelemetry-operator
    namespace: opentelemetry

So as far as I understand the chart and the new authentication logic this PR should add the necessary permissions to allow scraping of the /metrics endpoint.

I see the changelog also mentions RBAC: ClusterRole for /metrics access only created when manager.metrics.secure: true. I think this means

{{ if .Values.manager.metrics.secure }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
{{- include "opentelemetry-operator.labels" . | nindent 4 }}
app.kubernetes.io/component: controller-manager
name: {{ template "opentelemetry-operator.fullname" . }}-metrics
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
{{- end }}
{{ end }}

As far as I can tell there is no RoleBinding or ClusterRoleBinding that actually uses this ClusterRole. During debugging I was thinking that maybe this role is supposed to solve the issue I am seeing. So as an experiment I manually created a ClusterRoleBinding to use this role, but it did not solve the issue.

Looking at #2153 I don't understand how it could have worked withouth the tokenreviews permission for others. So I am wondering if maybe I am doing something wrong entirely?

@ekeih ekeih requested review from a team, Allex1 and jvoravong as code owners April 20, 2026 16:09
@linux-foundation-easycla
Copy link
Copy Markdown

CLA Not Signed

@ekeih
Copy link
Copy Markdown
Author

ekeih commented Apr 20, 2026

Just noticed the CLA warning. I will not go through the CLA process for a 8 line yaml template change, but if you think my change is needed fix the underlying issue, please feel free to take the commit/change and apply in your own PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant