fix: allow tenant owners to create namespaces with create-if-missing clients - #1082
Open
aminmr wants to merge 1 commit into
Open
fix: allow tenant owners to create namespaces with create-if-missing clients#1082aminmr wants to merge 1 commit into
aminmr wants to merge 1 commit into
Conversation
aminmr
force-pushed
the
fix/namespace-creation-tenant-owner-1895
branch
from
July 24, 2026 18:03
5cefb41 to
a4d7ee2
Compare
aminmr
force-pushed
the
fix/namespace-creation-tenant-owner-1895
branch
from
July 24, 2026 18:21
ca1d4b1 to
fbfcd3e
Compare
…clients A tenant owner running `helm install --create-namespace` fails every time with a 403 on a namespace-scoped resource, and the namespace is never created (projectcapsule/capsule#1895). Two distinct problems are at play. First, Kubernetes authorizes namespace-scoped requests via RoleBindings that live inside the target namespace, so a namespace that does not exist yet can never have one. A tenant owner probing a resource there therefore always gets 403 Forbidden, never the 404 Not Found a cluster-wide privileged caller receives for the same request. Clients using the idiomatic "GET, and create it if 404" pattern -- Helm's `--create-namespace` among them -- treat that 403 as fatal and abort before they ever attempt to create the namespace. Second, once creation does proceed, Capsule provisions the owner's RoleBindings asynchronously, after the Namespace object has already been persisted and returned. A client acting immediately inside the namespace it was just handed can still race that provisioning. Both are addressed in the reverse proxy's ModifyResponse hook: - maskForbiddenForMissingNamespace rewrites 403 to 404 when a namespace-scoped GET-by-name failed only because the namespace does not exist. It is deliberately narrow so it cannot become a cross-tenant existence oracle: it applies only when the namespace name falls within the naming space of a tenant the caller actually owns, and only when the namespace is confirmed absent. A caller can thus only learn that a namespace they are entitled to create does not exist yet -- something they can already determine. Every other 403 is passed through untouched. - waitForOwnerCreationResponse holds a successful namespace-create response until the owner's RoleBinding lands, covering both the collection POST and the server-side-apply PATCH that Helm 4 uses by default. Dry-run creates are skipped, since they persist nothing to wait for. The wait is bounded end-to-end by a context deadline and fails open: the namespace is already created by that point, so a timeout releases the response rather than misreporting success as an error. No new RBAC is required; capsule-proxy already has the necessary read access. Refs: projectcapsule/capsule#1895
aminmr
force-pushed
the
fix/namespace-creation-tenant-owner-1895
branch
from
July 24, 2026 18:28
fbfcd3e to
5721772
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's broken
A tenant owner running
helm install --create-namespacefails every time, and the namespace is never created (projectcapsule/capsule#1895):Root cause
1. A tenant owner can never get
404for a missing namespace — only403. Kubernetes authorizes namespace-scoped requests via RoleBindings that live inside the target namespace, so a namespace that doesn't exist yet can't have one. A cluster-admin, authorized cluster-scoped, gets a clean404for the same request.Helm does a pre-flight "does this resource exist?" GET for each chart resource before creating the namespace. It sees the
403, treats it as a real permission error, and aborts — never reaching the create. Every "GET, create if 404" client hits this (kubectl apply, Argo CD, Terraform). This is the deterministic failure, and it's why the namespace is never created.2. Owner RBAC is provisioned asynchronously after creation. Once creation proceeds, Capsule syncs the owner's RoleBindings after the Namespace is persisted and returned — measured at ~70 ms after the create call returns. A client acting immediately inside the namespace it was just handed can still lose that race.
The fix
Both handled in the reverse proxy's
ModifyResponsehook (internal/webserver/webserver.go). No operator changes, no new RBAC.maskForbiddenForMissingNamespacerewrites403→404when a namespace-scoped GET-by-name failed only because the namespace is absent.Deliberately narrow, so it cannot become a cross-tenant existence oracle — it applies only when both:
<tenant>or<tenant>-*, mirroring Capsule'sforceTenantPrefixboundary — the only ownership relationship determinable before the namespace exists, since there's no owner reference or tenant label yet); andA caller can therefore only learn "a namespace I'm entitled to create doesn't exist yet" — which they can already determine by listing their own namespaces. Every other
403(another tenant's namespace, a system namespace, one that genuinely exists) passes through untouched. No authorization decision changes; this only corrects a status code that leaked an artifact of RBAC's authorize-before-existence-check ordering.waitForOwnerCreationResponseholds a successful namespace-create response until the owner's RoleBinding lands, covering both the collectionPOSTand the server-side-applyPATCH(Helm 4 defaults--server-sideto true; K8s returns 201 for both on genuine creation). A short settle buffer lets sibling RoleBindings land, since Capsule creates one per configured ClusterRole and they don't appear atomically. Two safety properties:?dryRun=Allreturns 201 but persists nothing, so there'd be no RoleBinding to wait for.Testing
kind (k8s 1.36), cert-manager, Capsule 0.13.9, this build of capsule-proxy; the exact Tenant from the issue (
forceTenantPrefix: true, owner group withcluster-admin+capsule-namespace-deleter), authenticating as a tenant owner through the proxy. Chart deploys a ServiceAccount, Deployment and Service.helm install --create-namespace, Helm 3.19.0 (version in the report)helm install --create-namespace, Helm 4.2.2 (SSA path)Masking cannot leak across tenants — verified with a second tenant owned by a different identity:
my-tenantonly403403— indistinguishable, no oracle404— create-if-missing workskube-system(exists, not owned)403— unchangedAlso:
--dry-run=serverreturns in ~0.25–0.43 s persisting nothing (would block ~3 s without the skip); gated real creates run ~0.27–0.36 s, well under the 3 s cap; each owner still lists only their own namespaces; a tenant owner readingkube-systemsecrets still gets403.go build ./...,go vetandgofmtare clean.Known limitation
The ownership check keys on tenant-prefixed namespace names. Where
forceTenantPrefixis disabled and an owner creates a namespace not matching their tenant prefix, the masking won't apply and that client still hits the original problem. That isn't safely fixable at the proxy layer: before the namespace exists there's no way to establish it belongs to that tenant, so masking it would reintroduce the cross-tenant oracle. I took the conservative side deliberately — happy to revisit.Refs projectcapsule/capsule#1895