fix(evalhub): bind evalhub-events ClusterRole to server SA per tenant namespace - #857
Conversation
… namespace The evalhub-events ClusterRole (events create/patch) was bound only to the operator SA via a static ClusterRoleBinding. The EvalHub server SA had no events permission in tenant namespaces, so EventRecorder calls from NotifyJobPhaseTransition were silently rejected by the API server. Labels were patched correctly (jobs-writer covers batch/jobs:patch) but no EvaluationRunning or EvaluationFailed events appeared. Fix: create a namespace-scoped RoleBinding for evalhub-events per tenant namespace in reconcileTenantNamespaces, following the same pattern as the existing evalhub-jobs-writer and evalhub-job-config bindings. Because createJobRoleBinding stamps the standard job-resource labels, cleanupStaleTenantResources GCs the binding automatically when a namespace loses the tenant label. Remove the stale static evalhub-events-binding ClusterRoleBinding: the operator SA already has events create/patch via its kubebuilder:rbac marker in evaluation_job_failure_reconciler.go, making the ClusterRole binding to controller-manager redundant. Remove both entries from the OPA rbac.rego allowlist. Fixes: https://redhat.atlassian.net/browse/RHAISTRAT-1923
|
Warning Review limit reached
Next review available in: 44 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughEvalHub replaces its static Events ClusterRoleBinding with tenant-specific RoleBindings. Tenant reconciliation binds the API ServiceAccount to the Events ClusterRole, removes the static resource and policy entries, and adds test coverage. ChangesEvalHub Events RBAC
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@controllers/evalhub/service_accounts.go`:
- Around line 270-273: The EvalHub Events ClusterRole name is inconsistent
between deployment manifests and the controller constant. Update the role
resource in evalhub_events_role.yaml and evalhubEventsClusterRoleName so both
resolve to the same name, preserving the RoleBinding lookup used by
reconcileTenantNamespaces.
In `@controllers/evalhub/tenant_namespaces.go`:
- Around line 108-119: Update reconcileServiceAccount to create an additional
namespace-scoped RoleBinding in instance.Namespace for
generateServiceAccountName(instance), using evalhubEventsClusterRoleName and the
existing events RoleBinding configuration. Preserve the current tenant-namespace
binding and ensure failures follow the method’s existing error-handling pattern.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1fb763c6-d072-47d5-9573-d12e0e3b154f
📒 Files selected for processing (6)
config/components/evalhub/kustomization.yamlconfig/components/evalhub/rbac/evalhub_events_binding.yamlcontrollers/evalhub/service_accounts.gocontrollers/evalhub/tenant_namespaces.gocontrollers/evalhub/unit_test.gopolicy/rbac.rego
💤 Files with no reviewable changes (3)
- config/components/evalhub/rbac/evalhub_events_binding.yaml
- config/components/evalhub/kustomization.yaml
- policy/rbac.rego
…le-tenant mode resolveNamespace falls back to the server's in-cluster namespace (instance.Namespace) when no explicit tenant is set, covering single-tenant deployments where reconcileTenantNamespaces is skipped. Without a RoleBinding there, EventRecorder calls from NotifyJobPhaseTransition are silently rejected in that mode. Add createEventsRoleBinding and call it from createServiceAccount, following the same pattern as createJobsWriterRoleBinding, createJobConfigRoleBinding, and createHardwareProfilesReaderRoleBinding. Per-tenant bindings in reconcileTenantNamespaces are unchanged. Relates to: https://redhat.atlassian.net/browse/RHAISTRAT-1923
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: m-misiura, scheruku-rh The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ddc0b8f
into
trustyai-explainability:main
3 participants
What and why
Kubernetes Events were not appearing for EvalHub evaluation lifecycle transitions (
EvaluationRunning,EvaluationFailed) even though thetrustyai.opendatahub.io/evaluation-phaselabel was being patched correctly.Root cause: the
evalhub-eventsClusterRole (events create/patch) was bound via a staticClusterRoleBindingto the operator SA (controller-manager), not to the EvalHub server SA. The server'sEventRecordercalls inNotifyJobPhaseTransitionwere silently rejected by the API server (the broadcaster is async so no error surfaced). Label patches worked becauseevalhub-jobs-writer(batch/jobs:patch) was correctly bound to the server SA per tenant namespace.A secondary consequence: once the server sets
evaluation-phase=FailedserverAlreadyHandledFailure()=trueand skips its own event, leaving no event emitted from either path.Fix: create a namespace-scoped RoleBinding for
evalhub-eventsin each tenant namespace duringreconcileTenantNamespaces, following the same pattern already used forevalhub-jobs-writerandevalhub-job-config. BecausecreateJobRoleBindingstamps the standard job-resource labels,cleanupStaleTenantResourcesGCs the binding automatically when a namespace loses the tenant label (no additional cleanup code needed).Also removes the stale static
evalhub-events-bindingClusterRoleBinding: the operator SA already hasevents create/patchvia its//+kubebuilder:rbacmarker inevaluation_job_failure_reconciler.go, so that binding was redundant and bound to the wrong subject.Relates to https://redhat.atlassian.net/browse/RHAISTRAT-1923
Type
Testing
TestEvalHubReconciler_reconcileTenantNamespaces/should_provision_resources_in_labelled_tenant_namespaceextended to assert theevalhub-eventsRoleBinding is created with the correct ClusterRole reference and subject (EvalHub server SA in the instance namespace).Summary by CodeRabbit
Enhancements
Bug Fixes