Skip to content

fix: allow tenant owners to create namespaces with create-if-missing clients - #1082

Open
aminmr wants to merge 1 commit into
projectcapsule:mainfrom
aminmr:fix/namespace-creation-tenant-owner-1895
Open

fix: allow tenant owners to create namespaces with create-if-missing clients#1082
aminmr wants to merge 1 commit into
projectcapsule:mainfrom
aminmr:fix/namespace-creation-tenant-owner-1895

Conversation

@aminmr

@aminmr aminmr commented Jul 24, 2026

Copy link
Copy Markdown

What's broken

A tenant owner running helm install --create-namespace fails every time, and the namespace is never created (projectcapsule/capsule#1895):

Error: INSTALLATION FAILED: unable to continue with install: could not get information about
the resource ServiceAccount "my-release-sa" in namespace "my-tenant-test": serviceaccounts
"my-release-sa" is forbidden: User "jane-tenant-owner" cannot get resource "serviceaccounts"
in API group "" in the namespace "my-tenant-test"

Root cause

1. A tenant owner can never get 404 for a missing namespace — only 403. Kubernetes authorizes namespace-scoped requests via RoleBindings that live inside the target namespace, so a namespace that doesn't exist yet can't have one. A cluster-admin, authorized cluster-scoped, gets a clean 404 for the same request.

Helm does a pre-flight "does this resource exist?" GET for each chart resource before creating the namespace. It sees the 403, treats it as a real permission error, and aborts — never reaching the create. Every "GET, create if 404" client hits this (kubectl apply, Argo CD, Terraform). This is the deterministic failure, and it's why the namespace is never created.

2. Owner RBAC is provisioned asynchronously after creation. Once creation proceeds, Capsule syncs the owner's RoleBindings after the Namespace is persisted and returned — measured at ~70 ms after the create call returns. A client acting immediately inside the namespace it was just handed can still lose that race.

The fix

Both handled in the reverse proxy's ModifyResponse hook (internal/webserver/webserver.go). No operator changes, no new RBAC.

maskForbiddenForMissingNamespace rewrites 403404 when a namespace-scoped GET-by-name failed only because the namespace is absent.

Deliberately narrow, so it cannot become a cross-tenant existence oracle — it applies only when both:

  • the namespace name falls within the naming space of a tenant the caller owns (<tenant> or <tenant>-*, mirroring Capsule's forceTenantPrefix boundary — the only ownership relationship determinable before the namespace exists, since there's no owner reference or tenant label yet); and
  • the namespace is confirmed absent.

A caller can therefore only learn "a namespace I'm entitled to create doesn't exist yet" — which they can already determine by listing their own namespaces. Every other 403 (another tenant's namespace, a system namespace, one that genuinely exists) passes through untouched. No authorization decision changes; this only corrects a status code that leaked an artifact of RBAC's authorize-before-existence-check ordering.

waitForOwnerCreationResponse holds a successful namespace-create response until the owner's RoleBinding lands, covering both the collection POST and the server-side-apply PATCH (Helm 4 defaults --server-side to true; K8s returns 201 for both on genuine creation). A short settle buffer lets sibling RoleBindings land, since Capsule creates one per configured ClusterRole and they don't appear atomically. Two safety properties:

  • Dry-run creates are skipped?dryRun=All returns 201 but persists nothing, so there'd be no RoleBinding to wait for.
  • Bounded and fails open — the wait is capped by a context deadline, so a slow/hung API call can't hold the response past it. On timeout the response is released as-is: the namespace is already created, so failing it would misreport success as an error and orphan the namespace.

Testing

kind (k8s 1.36), cert-manager, Capsule 0.13.9, this build of capsule-proxy; the exact Tenant from the issue (forceTenantPrefix: true, owner group with cluster-admin + capsule-namespace-deleter), authenticating as a tenant owner through the proxy. Chart deploys a ServiceAccount, Deployment and Service.

Scenario Before After
helm install --create-namespace, Helm 3.19.0 (version in the report) 0/25 15/15
helm install --create-namespace, Helm 4.2.2 (SSA path) 0/10 10/10
App reaches Running (Deployment, Service, SA, Pod) n/a

Masking cannot leak across tenants — verified with a second tenant owned by a different identity:

Probe as owner of my-tenant only Result
non-owned namespace that exists 403
non-owned namespace that is missing 403 — indistinguishable, no oracle
own namespace that is missing 404 — create-if-missing works
kube-system (exists, not owned) 403 — unchanged

Also: --dry-run=server returns in ~0.25–0.43 s persisting nothing (would block ~3 s without the skip); gated real creates run ~0.27–0.36 s, well under the 3 s cap; each owner still lists only their own namespaces; a tenant owner reading kube-system secrets still gets 403. go build ./..., go vet and gofmt are clean.

go test ./internal/... passes except three assertions in internal/request (TestParseXFCCCert, TestHTTP_processXFCC) expecting capitalised error strings. These fail identically on unmodified main — pre-existing and unrelated, so I left them alone.

Known limitation

The ownership check keys on tenant-prefixed namespace names. Where forceTenantPrefix is disabled and an owner creates a namespace not matching their tenant prefix, the masking won't apply and that client still hits the original problem. That isn't safely fixable at the proxy layer: before the namespace exists there's no way to establish it belongs to that tenant, so masking it would reintroduce the cross-tenant oracle. I took the conservative side deliberately — happy to revisit.

Refs projectcapsule/capsule#1895

…clients

A tenant owner running `helm install --create-namespace` fails every time
with a 403 on a namespace-scoped resource, and the namespace is never
created (projectcapsule/capsule#1895).

Two distinct problems are at play.

First, Kubernetes authorizes namespace-scoped requests via RoleBindings
that live inside the target namespace, so a namespace that does not exist
yet can never have one. A tenant owner probing a resource there therefore
always gets 403 Forbidden, never the 404 Not Found a cluster-wide
privileged caller receives for the same request. Clients using the
idiomatic "GET, and create it if 404" pattern -- Helm's
`--create-namespace` among them -- treat that 403 as fatal and abort
before they ever attempt to create the namespace.

Second, once creation does proceed, Capsule provisions the owner's
RoleBindings asynchronously, after the Namespace object has already been
persisted and returned. A client acting immediately inside the namespace
it was just handed can still race that provisioning.

Both are addressed in the reverse proxy's ModifyResponse hook:

  - maskForbiddenForMissingNamespace rewrites 403 to 404 when a
    namespace-scoped GET-by-name failed only because the namespace does
    not exist. It is deliberately narrow so it cannot become a
    cross-tenant existence oracle: it applies only when the namespace
    name falls within the naming space of a tenant the caller actually
    owns, and only when the namespace is confirmed absent. A caller can
    thus only learn that a namespace they are entitled to create does not
    exist yet -- something they can already determine. Every other 403 is
    passed through untouched.

  - waitForOwnerCreationResponse holds a successful namespace-create
    response until the owner's RoleBinding lands, covering both the
    collection POST and the server-side-apply PATCH that Helm 4 uses by
    default. Dry-run creates are skipped, since they persist nothing to
    wait for. The wait is bounded end-to-end by a context deadline and
    fails open: the namespace is already created by that point, so a
    timeout releases the response rather than misreporting success as an
    error.

No new RBAC is required; capsule-proxy already has the necessary read
access.

Refs: projectcapsule/capsule#1895
@aminmr
aminmr force-pushed the fix/namespace-creation-tenant-owner-1895 branch from fbfcd3e to 5721772 Compare July 24, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant