diff --git a/content/en/docs/_index.md b/content/en/docs/_index.md index 48e0507..70646c2 100644 --- a/content/en/docs/_index.md +++ b/content/en/docs/_index.md @@ -2,3 +2,10 @@ title: Documentation weight: 20 --- + +## Where to start + +| I am a... | Start here | +|---|---| +| **Cluster Administrator** | [Installation](/docs/operating/setup/installation/) then [Quickstart](/docs/quickstart/) | +| **Tenant Owner** | [Quickstart - Tenant Owners section](/docs/quickstart/#tenant-owners) | diff --git a/content/en/docs/operating/architecture.md b/content/en/docs/operating/architecture.md index a1f04c5..b88e8ca 100644 --- a/content/en/docs/operating/architecture.md +++ b/content/en/docs/operating/architecture.md @@ -22,7 +22,7 @@ The answer to this question may be influenced by the following aspects: * _You’ll need robust monitoring that enables Tenant Owners to clearly understand and manage what’s happening inside their own tenant._ * **Are your customers technically capable of working directly with the Kubernetes API?** - * _If not, you may need to build a more user-friendly platform with better UX — for example, a multi-tenant ArgoCD setup, or UI layers like Headlamp._ + * _If not, you may need to build a more user-friendly platform with better UX, for example a multi-tenant ArgoCD setup, or UI layers like Headlamp._ ## Personas diff --git a/content/en/docs/operating/authentication.md b/content/en/docs/operating/authentication.md index b2973c8..8d31f44 100644 --- a/content/en/docs/operating/authentication.md +++ b/content/en/docs/operating/authentication.md @@ -220,9 +220,9 @@ $ kubectl config set-context alice-oidc@mycluster \ As user alice, you should be able to use kubectl to create some namespaces: ```shell -$ kubectl --context alice-oidc@mycluster create namespace oil-production -$ kubectl --context alice-oidc@mycluster create namespace oil-development -$ kubectl --context alice-oidc@mycluster create namespace gas-marketing +$ kubectl --context alice-oidc@mycluster create namespace wind-production +$ kubectl --context alice-oidc@mycluster create namespace wind-development +$ kubectl --context alice-oidc@mycluster create namespace water-marketing ``` **Warning**: once your `ID_TOKEN` expires, the kubectl OIDC Authenticator will attempt to refresh automatically your `ID_TOKEN` using the `REFRESH_TOKEN`. In case the OIDC uses a self signed CA certificate, make sure to specify it with the idp-certificate-authority option in your kubeconfig file, otherwise you'll not able to refresh the tokens. diff --git a/content/en/docs/operating/backup-restore.md b/content/en/docs/operating/backup-restore.md index b031ea3..60f45a9 100644 --- a/content/en/docs/operating/backup-restore.md +++ b/content/en/docs/operating/backup-restore.md @@ -15,7 +15,7 @@ The first requirement aims to backup all the resources stored into etcd database The main limitation of Velero is the multi tenancy. Currently, Velero does not support multi tenancy meaning it can be only used from admin users and so it cannot provided "as a service" to the users. This means that the cluster admin needs to take care of users' backup. -Assuming you have multiple tenants managed by Capsule, for example oil and gas, as cluster admin, you can to take care of scheduling backups for: +Assuming you have multiple tenants managed by Capsule, for example wind and water, as cluster admin, you can to take care of scheduling backups for: * Tenant cluster resources * Namespaces belonging to each tenant @@ -27,10 +27,10 @@ Create a backup of the tenant `solar`. It consists in two different backups: * backup of the tenant resource * backup of all the resources belonging to the tenant -To backup the oil tenant selectively, label the tenant as: +To backup the wind tenant selectively, label the tenant as: ```bash -kubectl label tenant oil capsule.clastix.io/tenant=solar +kubectl label tenant wind capsule.clastix.io/tenant=solar ``` and create the backup @@ -65,7 +65,7 @@ spec: ttl: 720h0m0s ``` -Create a backup of all the resources belonging to the oil tenant namespaces: +Create a backup of all the resources belonging to the wind tenant namespaces: ```bash velero create backup solar-namespaces \ @@ -109,15 +109,15 @@ Using Velero to restore a Capsule tenant can lead to an incomplete recovery of t ```bash kubectl get tnt NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE -gas active 9 5 {"pool":"gas"} 34m -oil active 9 8 {"pool":"oil"} 33m +water active 9 5 {"pool":"water"} 34m +wind active 9 8 {"pool":"wind"} 33m solar active 9 0 # <<< {"pool":"solar"} 54m ``` To avoid this problem you can use the script [velero-restore.sh](https://github.com/projectcapsule/capsule/blob/main/hack/velero-restore.sh) located under the hack/ folder: ```bash -./velero-restore.sh --kubeconfing /path/to/your/kubeconfig --tenant "oil" restore +./velero-restore.sh --kubeconfing /path/to/your/kubeconfig --tenant "wind" restore ``` Running this command, we are going to patch the tenant's namespaces manifests that are actually ownerReferences-less. Once the command has finished its run, you got the tenant back. @@ -125,7 +125,7 @@ Running this command, we are going to patch the tenant's namespaces manifests th ```bash kubectl get tnt NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE -gas active 9 5 {"pool":"gas"} 44m -solar active 9 8 {"pool":"oil"} 43m -oil active 9 3 # <<< {"pool":"solar"} 12s +water active 9 5 {"pool":"water"} 44m +solar active 9 8 {"pool":"wind"} 43m +wind active 9 3 # <<< {"pool":"solar"} 12s ``` diff --git a/content/en/docs/operating/best-practices/_index.md b/content/en/docs/operating/best-practices/_index.md index ad5eaeb..3081c9d 100644 --- a/content/en/docs/operating/best-practices/_index.md +++ b/content/en/docs/operating/best-practices/_index.md @@ -10,13 +10,13 @@ This is general advice you should consider before making Kubernetes Distribution ### Authentication User authentication for the platform should be handled via a central OIDC-compatible identity provider system (e.g., Keycloak, Azure AD, Okta, or any other OIDC-compliant provider). -The rationale is that other central platform components — such as ArgoCD, Grafana, Headlamp, or Harbor — should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions. +The rationale is that other central platform components, such as ArgoCD, Grafana, Headlamp, or Harbor, should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions. -[Capsule relies on native Kubernetes RBAC](/docs/operating/authentication/), so it's important to consider how the Kubernetes API handles user authentication. +[Capsule relies on native Kubernetes RBAC](/docs/operating/setup/authentication/), so it's important to consider how the Kubernetes API handles user authentication. ### OCI Pull-Cache -By default, Kubernetes clusters pull images directly from upstream registries like `docker.io`, `quay.io`, `ghcr.io`, or `gcr.io`. In production environments, this can lead to issues — especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled). +By default, Kubernetes clusters pull images directly from upstream registries like `docker.io`, `quay.io`, `ghcr.io`, or `gcr.io`. In production environments, this can lead to issues, especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled). To ensure availability, performance, and control over container images, it's essential to provide an on-premise OCI mirror. This mirror should be configured via the CRI (Container Runtime Interface) by defining it as a mirror endpoint in registries.conf for default registries (e.g., `docker.io`). @@ -25,7 +25,7 @@ This way, all nodes automatically benefit from caching without requiring develop ### Secrets Management In more complex environments with multiple clusters and applications, managing secrets manually via YAML or Helm is no longer practical. -Instead, a centralized secrets management system should be established — such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project [OpenBao](https://openbao.org/) (formerly the Vault community fork). +Instead, a centralized secrets management system should be established, such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project [OpenBao](https://openbao.org/) (formerly the Vault community fork). To integrate these external secret stores with Kubernetes, the [External Secrets Operator (ESO)](https://external-secrets.io/latest/) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing. diff --git a/content/en/docs/operating/best-practices/workloads.md b/content/en/docs/operating/best-practices/workloads.md index 7f29c17..f1b9908 100644 --- a/content/en/docs/operating/best-practices/workloads.md +++ b/content/en/docs/operating/best-practices/workloads.md @@ -192,7 +192,7 @@ metadata: and the regular Pod Security Admission Controller does the magic: ```yaml -kubectl --kubeconfig alice-oil.kubeconfig apply -f - << EOF +kubectl --kubeconfig alice-wind.kubeconfig apply -f - << EOF apiVersion: v1 kind: Pod metadata: diff --git a/content/en/docs/operating/concepts/_index.md b/content/en/docs/operating/concepts/_index.md new file mode 100644 index 0000000..eca1d28 --- /dev/null +++ b/content/en/docs/operating/concepts/_index.md @@ -0,0 +1,5 @@ +--- +title: Architecture & Concepts +weight: 2 +description: Key design decisions, personas, templating, and metadata reference for Capsule +--- diff --git a/content/en/docs/operating/admission-policies.md b/content/en/docs/operating/concepts/admission-policies.md similarity index 100% rename from content/en/docs/operating/admission-policies.md rename to content/en/docs/operating/concepts/admission-policies.md diff --git a/content/en/docs/operating/concepts/architecture.md b/content/en/docs/operating/concepts/architecture.md new file mode 100644 index 0000000..b88e8ca --- /dev/null +++ b/content/en/docs/operating/concepts/architecture.md @@ -0,0 +1,139 @@ +--- +title: Architecture +weight: 2 +description: Architecture references and considerations +--- + +## Key Decisions + +Introducing a new separation of duties can lead to a significant paradigm shift. This has technical implications and may also impact your organizational structure. Therefore, when designing a multi-tenant platform pattern, carefully consider the following aspects. As **Cluster Administrator**, ask yourself: + + * 🔑 **How much ownership can be delegated to Tenant Owners (Platform Users)?** + +The answer to this question may be influenced by the following aspects: + +* **Are the Cluster Administrators willing to grant permissions to Tenant Owners?** + * _You might have a problem with know-how and probably your organisation is not yet pushing Kubernetes itself enough as a key strategic platform. The key here is enabling Platform Users through good UX and know-how transfers_ + +* **Who is responsible for the deployed workloads within the Tenants?** + * _If Platform Administrators are still handling this, a true “shift left” has not yet been achieved._ + +* **Who gets paged during a production outage within a Tenant’s application?** + * _You’ll need robust monitoring that enables Tenant Owners to clearly understand and manage what’s happening inside their own tenant._ + +* **Are your customers technically capable of working directly with the Kubernetes API?** + * _If not, you may need to build a more user-friendly platform with better UX, for example a multi-tenant ArgoCD setup, or UI layers like Headlamp._ + +## Personas + +In Capsule, we introduce a new persona called the `Tenant Owner`. The goal is to enable Cluster Administrators to delegate tenant management responsibilities to Tenant Owners. Here’s how it works: + +### Capsule Administrators + +[Configure Capsule Administrators](/docs/operating/setup/configuration/#administrators). The ClusterRoles assigned to Administrators can be configured in the [CapsuleConfiguration](/docs/operating/setup/configuration/#rbac) as well. + +They are promoted to [Tenant-Owners](#tenant-owners) for all available tenants. Effectively granting them the ability to manage all namespaces within the cluster, across all tenants. + +**Note**: Granting Capsule Administrator rights should be done with caution, as it provides extensive control over the cluster's multi-tenant environment. **When granting Capsule Administrator rights, the entity gets the privileges to create any namespace (also not part of capsule tenants) and the privileges to delete any tenant namespaces.** + +Capsule Administrators can: + + - Create and manage [namespaces via labels in any tenant](/docs/tenants/namespaces/#label). + - Create namespaces outside of tenants. + - Delete namespaces in any tenant. + +Administrators come in handy in bootstrap scenarios or GitOps scenarios where certain users/serviceaccounts need to be able to manage namespaces for all tenants. + +### Capsule Users + +[Configure Capsule Users](/docs/operating/setup/configuration/#users) + +Any entity which needs to interact with tenants and their namespaces must be defined as a **Capsule User**. This is where the flexibility of Capsule comes into play. You can define users or groups as Capsule Users, allowing them to create and manage namespaces within any tenant they have access to. If they are not defined as Capsule Users, any interactions will be ignored by Capsule. Often a best practice is to define a single group which identifies all your tenant users. This way you can have one generic group for all your users and then use additional groups to separate responsibilities (e.g. administrators vs normal users). + +**Only one entry is needed to identify a Capsule User. This is only important for Namespace Admission.**. + +![Capsule Users Admission](/images/content/capsule-users-admission.drawio.png) + +You can always verify the effective Capsule Users by checking the Configuration Status. As this is variable with [Tenant Owners](#tenant-owners), the status will always show the effective users: + +```bash +kubectl get capsuleconfiguration default -o jsonpath='{.status.users}' | jq + +[ + { + "kind": "Group", + "name": "oidc:kubernetes:admin" + }, + { + "kind": "Group", + "name": "projectcapsule.dev" + }, + { + "kind": "User", + "name": "test-user" + } +] +``` + +### Tenant Owners + +[Configure Tenant Owners](/docs/tenants/permissions/#ownership) + +**Every Tenant Owner must be a [Capsule User](#capsule-users). By using the [TenantOwner CRD](/docs/tenants/permissions/#tenant-owners) this is automatically handeled.** + + +They manage the namespaces within their tenants and perform administrative tasks confined to their tenant boundaries. This delegation allows teams to operate more autonomously while still adhering to organizational policies. Tenant Owners can be used to shift reposnsability of one tenant towards this user group. promoting them to the SPOC of all namespaces within the tenant. + +Tenant Owners can: + + - Create and manage namespaces within their tenant. + - Delete namespaces within their tenant. + +Capsule provides robust tools to strictly enforce tenant boundaries, ensuring that each tenant operates within its defined limits. This separation of duties promotes both security and efficient resource management. + +## Layouts + +Let's discuss different Tenant Layouts which could be used . These are just approaches we have seen, however you might also find a combination of these which fits your use-case. + +### Tenant As A Service + +With this approach you essentially just provide your Customers with the Tenant on your cluster. The rest is their responsibility. This concludes to a shared responsibility model. This can be achieved when also the Tenant Owners are responsible for everything they are provisiong within their Tenant's namespaces. + +![Resourcepool Dashboard](/images/content/architecture/layout-taas.drawio.png) + +## Scheduling + +Workload distribution across your compute infrastructure can be approached in various ways, depending on your specific priorities. Regardless of the use case, it's essential to preserve maximum flexibility for your platform administrators. This means ensuring that: + + - Nodes can be drained or deleted at any time. + - Cluster updates can be performed at any time. + - The number of worker nodes can be scaled up or down as needed. + +If your cluster architecture prevents any of these capabilities, or if certain applications block the enforcement of these policies, you should reconsider your approach. + +### Dedicated + +Strong tenant isolation, ensuring that any noisy neighbor effects remain confined within individual tenants (tenant responsibility). This approach may involve higher administrative overhead and costs compared to shared compute. It also provides enhanced security by dedicating nodes to a single customer/application. It is recommended, at a minimum, to separate the cluster’s operator workload from customer workloads. + +![Dedicated Nodepool](/images/content/scheduling-dedicated.drawio.png) + +### Shared + +With this approach you share the nodes amongst all Tenants, therefore giving you more potential for optimizing resources on a node level. It's a common pattern to separate the controllers needed to power your Distribution (operators) from the actual workload. This ensures smooth operations for the cluster + +**Overview**: + +- ✅ Designed for cost efficiency . +- ✅ Suitable for applications that typically experience low resource fluctuations and run with multiple replicas. +- ❌ Not ideal for applications that are not cloud-native ready, as they may adversely affect the operation of other applications or the maintenance of node pools. +- ❌ Not ideal if strong isolation is required + +![Shared Nodepool](/images/content/scheduling-shared.drawio.png) + + +We provide the concept of [ResourcePools](/docs/resource-management/resourcepools/) or [CustomQuotas](/docs/resource-management/customquotas/) to manage resources cross namespaces. There's some further aspects you must think about with shared approaches: + + * [PriorityClasses](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/) + * [ResourceQuotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) + * [LimitRanges](https://kubernetes.io/docs/concepts/policy/limit-range/) + * [Descheduling/Rebalancing](https://github.com/kubernetes-sigs/descheduler) diff --git a/content/en/docs/operating/known-metadata.md b/content/en/docs/operating/concepts/known-metadata.md similarity index 100% rename from content/en/docs/operating/known-metadata.md rename to content/en/docs/operating/concepts/known-metadata.md diff --git a/content/en/docs/operating/concepts/templating.md b/content/en/docs/operating/concepts/templating.md new file mode 100644 index 0000000..97bb39e --- /dev/null +++ b/content/en/docs/operating/concepts/templating.md @@ -0,0 +1,198 @@ +--- +title: Templating +weight: 10 +description: "Templating in Capsule Items" +--- + +## Fast Templates + +For simple template cases we provide a fast templating engine. With this engine, you can use Go templates syntax to reference Tenant and Namespace fields. There are no operators or anything else supported. + +Available fields are: + + * `{{tenant.name}}`: The Name of the Tenant + * `{{namespace}}`: The Name of the namespace within the tenant (current context) + + +## Sprout Templating + +Our template library is mainly based on the upstream implementation from Sprout. You can find the all available functions here: + +* [https://docs.atom.codes/sprout/registries/list-of-all-registries](https://docs.atom.codes/sprout/registries/list-of-all-registries) + +We have removed certain functions which could exploit runtime information. Therefor the following functions are not available: + + * `env` + * `expandEnv` + +### Data + +You can provide structured data for each `Tenant` which can be used in templating: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + data: + bool: true + foo: bar + list: + - a + - b + number: 123 + obj: + nested: value +``` + +## Function Library + +Custom Functions we provide in our template package. + +### deterministicUUID + +`deterministicUUID` generates a deterministic, RFC-4122–compliant UUID (version 5 + RFC4122 variant) from a set of input strings. It is designed for use in templates where you need stable, repeatable IDs derived from meaningful inputs (e.g. cluster name, tenant, role name), instead of random UUIDs. + +This is especially useful for: + + * Crossplane / IaC resources that must not change IDs across reconciles + +The function takes any number of strings and turns them into a UUID in a fully deterministic way. + +What that means in practice: + +* If you call it twice with the same values, you get the same UUID +* If any input changes, the UUID changes too +* There is no randomness involved +* The output is always a valid UUID + +So from the outside, it behaves just like a normal UUID, just deterministic. + +```go +deterministicUUID(parts ...string) string +``` + +Example usage: + +```yaml +{{ deterministicUUID "cluster-a" "app-123" "tenant-x" "some-role" }} +``` + +### generateAgeKey + +`generateAgeKey` generates a new age X25519 key pair for use with [`age`](https://github.com/FiloSottile/age). It returns both the private identity and the public recipient key. + +This is useful in templates where a resource needs to create an age-compatible encryption identity, for example when generating Kubernetes Secrets that are later used for encrypting or decrypting data. + +This is especially useful for: + + * Bootstrap secrets that need an age identity + * Generating encryption keys during initial provisioning + * Creating age recipient keys for systems that need to encrypt data for a generated identity + +The function does **not** return a plain string. It returns an object with two fields: + +* `Identity`: the private age identity, e.g. `AGE-SECRET-KEY-1...` +* `Recipient`: the public age recipient, e.g. `age1...` + +What that means in practice: + +* Each call generates a new key pair +* The identity is the private key and must be treated as secret +* The recipient is the public key and can be shared with systems that need to encrypt data +* The output is not deterministic +* Calling this function during every reconcile may rotate the generated key unless the result is persisted + +```go +generateAgeKey() any +``` + +Example usage: + +```yaml +{{ $key := generateAgeKey }} +apiVersion: v1 +kind: Secret +metadata: + name: age-key +type: Opaque +stringData: + identity: {{ $key.Identity | quote }} + recipient: {{ $key.Recipient | quote }} +``` + +Output: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: age-key +type: Opaque +stringData: + identity: "AGE-SECRET-KEY-1..." + recipient: "age1..." +``` + +Because this function creates a new random key pair on every call, it should usually only be used when the generated Secret is created once and then reused. For continuously reconciled resources, prefer generating the key in controller logic and persisting it before using it in templates. + + +### generateAgePQKey + +`generateAgePQKey` generates a new age post-quantum hybrid key pair for use with [`age`](https://github.com/FiloSottile/age). It returns both the private identity and the public recipient key. + +This is useful in templates where a resource needs to create an age-compatible encryption identity using the newer hybrid recipient format. + +This is especially useful for: + + * Bootstrap secrets that should use age hybrid keys + * Generating encryption keys during initial provisioning + * Creating public recipient keys for systems that need to encrypt data for a generated hybrid identity + * Future-facing age encryption setups where hybrid keys are preferred + +The function does **not** return a plain string. It returns an object with two fields: + +* `Identity`: the private age hybrid identity, e.g. `AGE-SECRET-KEY-PQ-1...` +* `Recipient`: the public age recipient, e.g. `age1...` + +What that means in practice: + +* Each call generates a new key pair +* The identity is the private key and must be treated as secret +* The recipient is the public key and can be shared with systems that need to encrypt data +* The output is not deterministic +* Calling this function during every reconcile may rotate the generated key unless the result is persisted + +```go +generateAgePQKey() any +``` + +Example Usage: + +```yaml +{{ $key := generateAgePQKey }} +apiVersion: v1 +kind: Secret +metadata: + name: age-pq-key +type: Opaque +stringData: + identity: {{ $key.Identity | quote }} + recipient: {{ $key.Recipient | quote }} +``` + +Output: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: age-pq-key +type: Opaque +stringData: + identity: "AGE-SECRET-KEY-PQ-1..." + recipient: "age1..." +``` + +Because this function creates a new random key pair on every call, it should usually only be used when the generated Secret is created once and then reused. For continuously reconciled resources, prefer generating the key in controller logic and persisting it before using it in templates. diff --git a/content/en/docs/operating/operations/_index.md b/content/en/docs/operating/operations/_index.md new file mode 100644 index 0000000..ea5510c --- /dev/null +++ b/content/en/docs/operating/operations/_index.md @@ -0,0 +1,5 @@ +--- +title: Day-2 Operations +weight: 6 +description: Monitoring, backup and restore, and troubleshooting for production Capsule clusters +--- diff --git a/content/en/docs/operating/operations/backup-restore.md b/content/en/docs/operating/operations/backup-restore.md new file mode 100644 index 0000000..60f45a9 --- /dev/null +++ b/content/en/docs/operating/operations/backup-restore.md @@ -0,0 +1,131 @@ +--- +title: Backup & Restore +description: Run Backups and Restores of Tenants +weight: 6 +--- + +[Velero](https://velero.io/) is a backup and restore solution that performs data protection, disaster recovery and migrates Kubernetes cluster from on-premises to the Cloud or between different Clouds. + +When coming to backup and restore in Kubernetes, we have two main requirements: + +* Configurations backup +* Data backup + +The first requirement aims to backup all the resources stored into etcd database, for example: namespaces, pods, services, deployments, etc. The second is about how to backup stateful application data as volumes. + +The main limitation of Velero is the multi tenancy. Currently, Velero does not support multi tenancy meaning it can be only used from admin users and so it cannot provided "as a service" to the users. This means that the cluster admin needs to take care of users' backup. + +Assuming you have multiple tenants managed by Capsule, for example wind and water, as cluster admin, you can to take care of scheduling backups for: + +* Tenant cluster resources +* Namespaces belonging to each tenant + +## Create backup of a tenant + +Create a backup of the tenant `solar`. It consists in two different backups: + +* backup of the tenant resource +* backup of all the resources belonging to the tenant + +To backup the wind tenant selectively, label the tenant as: + +```bash +kubectl label tenant wind capsule.clastix.io/tenant=solar +``` + +and create the backup + +```bash +velero create backup solar-tenant \ + --include-cluster-resources=true \ + --include-resources=tenants.capsule.clastix.io \ + --selector capsule.clastix.io/tenant=solar +``` + +resulting in the following Velero object: + +```yaml +apiVersion: velero.io/v1 +kind: Backup +metadata: + name: solar-tenant +spec: + defaultVolumesToRestic: false + hooks: {} + includeClusterResources: true + includedNamespaces: + - '*' + includedResources: + - tenants.capsule.clastix.io + labelSelector: + matchLabels: + capsule.clastix.io/tenant: solar + metadata: {} + storageLocation: default + ttl: 720h0m0s +``` + +Create a backup of all the resources belonging to the wind tenant namespaces: + +```bash +velero create backup solar-namespaces \ + --include-cluster-resources=false \ + --include-namespaces solar-production,solar-development,solar-marketing +``` + +resulting in the following Velero object: + +```yaml +apiVersion: velero.io/v1 +kind: Backup +metadata: + name: solar-namespaces +spec: + defaultVolumesToRestic: false + hooks: {} + includeClusterResources: false + includedNamespaces: + - solar-production + - solar-development + - solar-marketing + metadata: {} + storageLocation: default + ttl: 720h0m0s +``` + +> Velero requires an Object Storage backend where to store backups, you should take care of this requirement before using Velero. + +## Restore a tenant from the backup + +To recover the tenant after a disaster, or to migrate it to another cluster, create a restore from the previous backups: + +```bash +velero create restore --from-backup solar-tenant +velero create restore --from-backup solar-namespaces +``` + +Using Velero to restore a Capsule tenant can lead to an incomplete recovery of tenant because the namespaces restored with Velero do not have the `OwnerReference` field used to bind the namespaces to the tenant. For this reason, all restored namespaces are not bound to the tenant: + +```bash +kubectl get tnt +NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE +water active 9 5 {"pool":"water"} 34m +wind active 9 8 {"pool":"wind"} 33m +solar active 9 0 # <<< {"pool":"solar"} 54m +``` + +To avoid this problem you can use the script [velero-restore.sh](https://github.com/projectcapsule/capsule/blob/main/hack/velero-restore.sh) located under the hack/ folder: + +```bash +./velero-restore.sh --kubeconfing /path/to/your/kubeconfig --tenant "wind" restore +``` + +Running this command, we are going to patch the tenant's namespaces manifests that are actually ownerReferences-less. Once the command has finished its run, you got the tenant back. + +```bash +kubectl get tnt +NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE +water active 9 5 {"pool":"water"} 44m +solar active 9 8 {"pool":"wind"} 43m +wind active 9 3 # <<< {"pool":"solar"} 12s +``` diff --git a/content/en/docs/operating/monitoring.md b/content/en/docs/operating/operations/monitoring.md similarity index 99% rename from content/en/docs/operating/monitoring.md rename to content/en/docs/operating/operations/monitoring.md index 6457a7f..5388de2 100644 --- a/content/en/docs/operating/monitoring.md +++ b/content/en/docs/operating/operations/monitoring.md @@ -12,7 +12,7 @@ Capsule can export OpenTelemetry traces for admission webhook requests through t ## Tenants -Instrumentation for [Tenants](../tenants/). +Instrumentation for [Tenants](/docs/tenants/). ### Metrics diff --git a/content/en/docs/operating/operations/troubleshoting.md b/content/en/docs/operating/operations/troubleshoting.md new file mode 100644 index 0000000..5680590 --- /dev/null +++ b/content/en/docs/operating/operations/troubleshoting.md @@ -0,0 +1,172 @@ +--- +title: Troubleshooting +weight: 15 +description: "Different topics when you encounter problems with Capsule" +--- + +This page covers the most common issues encountered when running Capsule. Each section describes the symptom, the likely cause, and how to resolve it. + +## User cannot create namespaces or interact with tenants + +**Symptom**: A user runs `kubectl create namespace my-ns` and gets a generic `Forbidden` error, or Capsule completely ignores the request and does not assign the namespace to any tenant. + +**Cause**: The user is not recognized as a Capsule User. Capsule only acts on namespace creation requests from subjects that are configured as Capsule Users in the `CapsuleConfiguration`. + +**Resolution**: Check which subjects are currently recognized: + +```bash +kubectl get capsuleconfiguration default -o jsonpath='{.status.users}' | jq +``` + +If the user (or any of their groups) is not listed, add them. The recommended approach is to create a `TenantOwner` resource for the user: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantOwner +metadata: + name: alice + labels: + projectcapsule.dev/tenant: "solar" +spec: + kind: User + name: "alice" +``` + +Alternatively, configure a group in the `CapsuleConfiguration`: + +```yaml +manager: + options: + users: + - kind: Group + name: projectcapsule.dev +``` + +## Namespace creation is rejected by the admission webhook + +**Symptom**: `Error from server (Forbidden): admission webhook "namespaces.mutating.projectcapsule.dev" denied the request`. + +Two common reasons: + +### Namespace quota exceeded + +The tenant has reached its namespace quota. The error message will include `Cannot exceed Namespace quota`. + +```bash +# Check the current namespace count vs quota +kubectl get tenant solar -o jsonpath='{.status.namespaceCount} / {.spec.namespaceOptions.quota}' +``` + +Contact your cluster administrator to increase the quota, or delete unused namespaces. + +### Force tenant prefix not respected + +The tenant has `forceTenantPrefix: true` and the namespace name does not start with the tenant name. + +```bash +# Wrong +kubectl create namespace my-app + +# Correct +kubectl create namespace solar-my-app +``` + +If you belong to multiple tenants, Capsule cannot infer which one to use. Prefix the namespace name with the tenant you intend to use. + +## Admission webhook denied a label or annotation change + +**Symptom**: `Error from server (Forbidden): admission webhook "namespaces.validating.projectcapsule.dev" denied the request: metadata label "..." is not allowed`. + +**Cause**: A rule on the tenant restricts which values are allowed for that label. The error message contains the exact label key and the list of allowed values. + +**Resolution**: + +1. Read the error message carefully. It states the allowed values. +2. Use one of the permitted values: + +```bash +# Example: allowed values are restricted and baseline +kubectl label namespace solar-development pod-security.kubernetes.io/enforce=baseline --overwrite +``` + +3. If the label is marked as `managed`, it is controlled entirely by Capsule and cannot be changed by tenant users. Contact your cluster administrator. + +## Cannot list namespaces or cluster-scoped resources + +**Symptom**: `kubectl get namespaces` returns an error such as `Error from server (Forbidden): namespaces is forbidden: User "alice" cannot list resource "namespaces" in API group "" at the cluster scope`. + +**Cause**: The Capsule Proxy is not in use. Without the proxy, Kubernetes RBAC prevents non-admin users from listing cluster-scoped resources. The proxy intercepts those requests and filters results to show only resources belonging to the user's tenants. + +**Resolution**: Verify that the Capsule Proxy is installed and that your kubeconfig points to the proxy endpoint rather than the Kubernetes API server directly. See the [Proxy documentation](/docs/proxy/) for setup instructions. + +```bash +# Confirm which server your kubeconfig is pointing to +kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' +``` + +## TenantOwner resource is not granting access + +**Symptom**: A `TenantOwner` resource was created but the user still cannot interact with the tenant. + +**Cause**: The `TenantOwner` CRD uses [aggregation](/docs/tenants/permissions/#aggregation) to bind owners to tenants. The resource must carry the correct label for implicit assignment, and the tenant must reference it via `matchOwners` or the owner must be listed directly. + +**Checklist**: + +1. Confirm the `TenantOwner` has the correct tenant label: + +```bash +kubectl get tenantowner alice -o jsonpath='{.metadata.labels}' +# Expected: {"projectcapsule.dev/tenant":"solar"} +``` + +2. Confirm the tenant lists the owner in its status: + +```bash +kubectl get tenant solar -o jsonpath='{.status.owners}' | jq +``` + +3. Confirm the user is recognized as a Capsule User (see the first section above). + +4. If using `matchOwners` with label selectors, verify the `TenantOwner` carries the matching labels: + +```bash +kubectl get tenantowner platform-team --show-labels +``` + +## Capsule Proxy is being throttled by the API server + +**Symptom**: Clients talking through the Capsule Proxy receive connection errors such as: + +``` +net/http: abort Handler +``` + +or requests hang and are dropped without a response. + +**Cause**: The Capsule Proxy is sending more requests to the Kubernetes API server than its configured QPS and burst limits allow. Because the proxy sits between clients and the API server, the client is the first to see the resulting aborted connections. + +**Resolution**: Increase the `clientConnectionQPS` and `clientConnectionBurst` values for the Capsule Proxy. See the [Proxy production installation guide](/docs/proxy/setup/installation/#qpsburst) for the recommended Helm values: + +```yaml +options: + clientConnectionQPS: 200 + clientConnectionBurst: 400 +``` + +## Capsule controller fails to start due to timeouts + +**Symptom**: The Capsule controller pod restarts repeatedly or never reaches `Running`. Logs contain errors such as: + +``` +E0707 08:38:18.319041 1 leaderelection.go:452] "Error retrieving lease lock" + err="...net/http: request canceled (Client.Timeout exceeded while awaiting headers)" +``` + +or the controller stalls silently during startup on clusters with many resources. + +**Cause**: Two common sources of startup timeouts: + +- **Cache sync timeout** — On large clusters the controller's informer cache takes longer to populate than the default timeout allows, causing the controller to give up before it is ready to serve. +- **Leader election timeout** — In high-pressure environments the controller cannot renew its leader lease within the default deadline, triggering repeated leader-election failures. + +**Resolution**: Adjust the `cacheSyncTimeout` and `leaderElection` values for the Capsule controller. See the [Production installation guide](/docs/operating/setup/installation/#cache-synchronisation) for the recommended Helm values. diff --git a/content/en/docs/operating/setup/authentication.md b/content/en/docs/operating/setup/authentication.md new file mode 100644 index 0000000..8d31f44 --- /dev/null +++ b/content/en/docs/operating/setup/authentication.md @@ -0,0 +1,228 @@ +--- +title: Authentication +description: Integrate Capsule with Authentication of your Kubernetes cluster +weight: 5 +--- + +Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by userGroups option in the CapsuleConfiguration, which defaults to `projectcapsule.dev`. + +## OIDC + +In the following guide, we'll use [Keycloak](https://www.keycloak.org/) an Open Source Identity and Access Management server capable to authenticate users via OIDC and release JWT tokens as proof of authentication. + +### Configuring OIDC Server + +Configure Keycloak as OIDC server: + + * Add a realm called caas, or use any existing realm instead + * Add a group `projectcapsule.dev` + * Add a user alice assigned to group `projectcapsule.dev` + * Add an OIDC client called `kubernetes` (Public) + * Add an OIDC client called `kubernetes-auth` (Confidential (Client Secret)) + +For the kubernetes client, create protocol mappers called groups and audience +If everything is done correctly, now you should be able to authenticate in Keycloak and see user groups in JWT tokens. Use the following snippet to authenticate in Keycloak as alice user: + +```bash +$ KEYCLOAK=sso.clastix.io +$ REALM=kubernetes-auth +$ OIDC_ISSUER=${KEYCLOAK}/realms/${REALM} + +$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/token \ + -d grant_type=password \ + -d response_type=id_token \ + -d scope=openid \ + -d client_id=${OIDC_CLIENT_ID} \ + -d client_secret=${OIDC_CLIENT_SECRET} \ + -d username=${USERNAME} \ + -d password=${PASSWORD} | jq +``` + +The result will include an `ACCESS_TOKEN`, a `REFRESH_TOKEN`, and an `ID_TOKEN`. The access-token can generally be disregarded for Kubernetes. It would be used if the identity provider was managing roles and permissions for the users but that is done in Kubernetes itself with RBAC. The id-token is short lived while the refresh-token has longer expiration. The refresh-token is used to fetch a new id-token when the id-token expires. + +```json +{ + "access_token":"ACCESS_TOKEN", + "refresh_token":"REFRESH_TOKEN", + "id_token": "ID_TOKEN", + "token_type":"bearer", + "scope": "openid groups profile email" +} +``` + +To introspect the `ID_TOKEN` token run: + +```bash +$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/introspect \ + -d token=${ID_TOKEN} \ + --user kubernetes-auth:${OIDC_CLIENT_SECRET} | jq +``` + +The result will be like the following: + +```json +{ + "exp": 1601323086, + "iat": 1601322186, + "aud": "kubernetes", + "typ": "ID", + "azp": "kubernetes", + "preferred_username": "alice", + "email_verified": false, + "acr": "1", + "groups": [ + "capsule.clastix.io" + ], + "client_id": "kubernetes", + "username": "alice", + "active": true +} +``` + +### Configuring Kubernetes API Server + +Configuring Kubernetes for OIDC Authentication requires adding several parameters to the API Server. Please, refer to the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens) for details and examples. Most likely, your kube-apiserver.yaml manifest will looks like the following: + +#### Authentication Configuration (Recommended) + +The configuration file approach allows you to configure multiple JWT authenticators, each with a unique issuer.url and issuer.discoveryURL. The configuration file even allows you to specify CEL expressions to map claims to user attributes, and to validate claims and user information. + +```yaml +apiVersion: apiserver.config.k8s.io/v1beta1 +kind: AuthenticationConfiguration +jwt: +- issuer: + url: https://${OIDC_ISSUER} + audiences: + - kubernetes + - kubernetes-auth + audienceMatchPolicy: MatchAny + claimMappings: + username: + claim: 'email' + prefix: "" + groups: + claim: 'groups' + prefix: "" + certificateAuthority: +``` + +This file must be present and consistent across all kube-apiserver instances in the cluster. Add the following flag to the kube-apiserver manifest: + +```yaml +spec: + containers: + - command: + - kube-apiserver + ... + - --authentication-configuration-file=/etc/kubernetes/authentication/authentication.yaml +``` + +[Read More](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#using-authentication-configuration) + +#### OIDC Flags (Legacy) + +```yaml +spec: + containers: + - command: + - kube-apiserver + ... + - --oidc-issuer-url=https://${OIDC_ISSUER} + - --oidc-ca-file=/etc/kubernetes/oidc/ca.crt + - --oidc-client-id=kubernetes + - --oidc-username-claim=preferred_username + - --oidc-groups-claim=groups + - --oidc-username-prefix=- +``` + +#### KinD + +As reference, here is an example of a [KinD](https://github.com/kubernetes-sigs/kind) configuration for OIDC Authentication, which can be useful for local testing: + +```yaml +apiVersion: kind.x-k8s.io/v1alpha4 +kind: Cluster +nodes: + - role: control-plane + kubeadmConfigPatches: + - | + kind: ClusterConfiguration + apiServer: + extraArgs: + oidc-issuer-url: https://${OIDC_ISSUER} + oidc-username-claim: preferred_username + oidc-client-id: kubernetes + oidc-username-prefix: "keycloak:" + oidc-groups-claim: groups + oidc-groups-prefix: "keycloak:" + enable-admission-plugins: PodNodeSelector +``` + +### Configuring kubectl + +There are two options to use kubectl with OIDC: + + * OIDC Authenticator + * Use the --token option + +**Plugin** + +One way to use OIDC authentication is the use of a kubectl plugin. The [Kubelogin Plugin](https://github.com/int128/kubelogin) for kubectl simplifies the process of obtaining an OIDC token and configuring kubectl to use it. Follow the link to obtain installation instructions. + +```shell +kubectl oidc-login setup \ + --oidc-issuer-url=https://${OIDC_ISSUER} \ + --oidc-client-id=kubernetes-auth \ + --oidc-client-secret=${OIDC_CLIENT_SECRET} +``` + +**Manual** + +To use the OIDC Authenticator, add an oidc user entry to your kubeconfig file: + +```shell +$ kubectl config set-credentials oidc \ + --auth-provider=oidc \ + --auth-provider-arg=idp-issuer-url=https://${OIDC_ISSUER} \ + --auth-provider-arg=idp-certificate-authority=/path/to/ca.crt \ + --auth-provider-arg=client-id=kubernetes-auth \ + --auth-provider-arg=client-secret=${OIDC_CLIENT_SECRET} \ + --auth-provider-arg=refresh-token=${REFRESH_TOKEN} \ + --auth-provider-arg=id-token=${ID_TOKEN} \ + --auth-provider-arg=extra-scopes=groups +``` + +To use the `--token` option: + +```shell +$ kubectl config set-credentials oidc --token=${ID_TOKEN} +``` + +Point the kubectl to the URL where the Kubernetes APIs Server is reachable: + +```shell +$ kubectl config set-cluster mycluster \ + --server=https://kube.projectcapsule.io:6443 \ + --certificate-authority=~/.kube/ca.crt +``` + +> If your APIs Server is reachable through the capsule-proxy, make sure to use the URL of the capsule-proxy. + +Create a new context for the OIDC authenticated users: + +```shell +$ kubectl config set-context alice-oidc@mycluster \ + --cluster=mycluster \ + --user=oidc +``` + +As user alice, you should be able to use kubectl to create some namespaces: + +```shell +$ kubectl --context alice-oidc@mycluster create namespace wind-production +$ kubectl --context alice-oidc@mycluster create namespace wind-development +$ kubectl --context alice-oidc@mycluster create namespace water-marketing +``` + +**Warning**: once your `ID_TOKEN` expires, the kubectl OIDC Authenticator will attempt to refresh automatically your `ID_TOKEN` using the `REFRESH_TOKEN`. In case the OIDC uses a self signed CA certificate, make sure to specify it with the idp-certificate-authority option in your kubeconfig file, otherwise you'll not able to refresh the tokens. diff --git a/content/en/docs/operating/setup/configuration.md b/content/en/docs/operating/setup/configuration.md index 4ddf49f..aa35711 100644 --- a/content/en/docs/operating/setup/configuration.md +++ b/content/en/docs/operating/setup/configuration.md @@ -19,7 +19,7 @@ kubectl explain capsuleConfiguration.spec These entities are automatically owners for all existing tenants. Meaning they can add namespaces to any tenant. However they must be specific by using the capsule label for interacting with namespaces. Because if that label is not defined, it's assumed that namespace interaction was not targeted towards a tenant and will therefore be ignored by capsule. May also be handy in GitOps scenarios where certain service accounts need to be able to manage namespaces for all tenants. -[Read More](/docs/operating/architecture/#capsule-administrators) +[Read More](/docs/operating/concepts/architecture/#capsule-administrators) ```yaml manager: @@ -33,7 +33,7 @@ manager: These entities are automatically owners for all existing tenants. Meaning they can add namespaces to any tenant. However they must be specific by using the capsule label for interacting with namespaces. Because if that label is not defined, it's assumed that namespace interaction was not targeted towards a tenant and will therefore be ignored by capsule. May also be handy in GitOps scenarios where certain service accounts need to be able to manage namespaces for all tenants. -[Read More](/docs/operating/architecture/#capsule-users) +[Read More](/docs/operating/concepts/architecture/#capsule-users) ```yaml manager: diff --git a/content/en/docs/operating/setup/installation.md b/content/en/docs/operating/setup/installation.md index 4c70059..0aec612 100644 --- a/content/en/docs/operating/setup/installation.md +++ b/content/en/docs/operating/setup/installation.md @@ -200,7 +200,7 @@ manager: rbac: strict: true clusterRole: - extraResources: + extraResources: - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch", "update", "patch"] @@ -231,7 +231,7 @@ Before you enable this option, you must implement the required permissions for y ### Admission Policies -While Capsule provides a robust framework for managing multi-tenancy in Kubernetes, it does not include built-in admission policies for enforcing specific security or operational standards for all possible aspects of a Kubernetes cluster. [We provide additional policy recommendations here](/docs/operating/admission-policies/). +While Capsule provides a robust framework for managing multi-tenancy in Kubernetes, it does not include built-in admission policies for enforcing specific security or operational standards for all possible aspects of a Kubernetes cluster. [We provide additional policy recommendations here](/docs/operating/concepts/admission-policies/). ### Certificate Management diff --git a/content/en/docs/operating/setup/openshift.md b/content/en/docs/operating/setup/openshift.md index b9dea23..9444f2b 100644 --- a/content/en/docs/operating/setup/openshift.md +++ b/content/en/docs/operating/setup/openshift.md @@ -44,7 +44,7 @@ kubectl patch clusterrolebinding.rbac self-provisioners -p '{ "metadata": { "ann ### Extend the admin role This example extends the default Kubernetes `admin` role so tenant owners gain admin privileges on all namespaces within their tenant. The extension adds: - The finalizers required to create/edit resources managed by Capsule -- The SCCs that tenant owners can use — in this example, `restricted-v2` and `nonroot-v2` +- The SCCs that tenant owners can use (in this example, `restricted-v2` and `nonroot-v2`) ```yaml kind: ClusterRole @@ -92,6 +92,26 @@ The following chart values can be used: ``` Deploy the Capsule Helm chart with (at least) these values. +### etcd Encryption + +If etcd encryption is enabled on your OpenShift cluster and you are using `TenantResource` or `GlobalTenantResource` to replicate `ConfigMap`s or `Secret`s, the Capsule replication webhook must be configured to skip requests from OpenShift's storage version migrator. Without this exclusion, the migrator, which iterates over all Secrets and ConfigMaps to rotate the encryption key, will be blocked or cause reconciliation errors, because the webhook intercepts its requests. + +Add the following `matchCondition` to the replication webhook via the Helm chart values: + +```yaml +webhooks: + hooks: + replications: + matchConditions: + - name: "exclude-privileged-users" + expression: > + !( + request.userInfo.username in [ + "system:serviceaccount:openshift-kube-storage-version-migrator:kube-storage-version-migrator-sa" + ] + ) +``` + ### Example Tenant and TenantOwners A minimal example tenant looks like the following: diff --git a/content/en/docs/operating/setup/rancher.md b/content/en/docs/operating/setup/rancher.md index aa5f55b..938afcc 100644 --- a/content/en/docs/operating/setup/rancher.md +++ b/content/en/docs/operating/setup/rancher.md @@ -47,9 +47,9 @@ It then explains how for the tenant user, the access to Kubernetes resources is #### Configure an identity provider for Kubernetes -You can follow [this general guide](/docs/operating/authentication/) to configure an OIDC authentication for Kubernetes. +You can follow [this general guide](/docs/operating/setup/authentication/) to configure an OIDC authentication for Kubernetes. -For a Keycloak specific setup yon can check [this resources list](/docs/operating/authentication/#oidc). +For a Keycloak specific setup yon can check [this resources list](/docs/operating/setup/authentication/#oidc). #### Known issues @@ -99,7 +99,7 @@ When onboarding tenants, the administrator needs to create the following, in ord ``` More on declarative `Project`s [here](https://github.com/rancher/rancher/issues/35631). -- In the identity provider, create a user with [correct OIDC claim](/docs/operating/authentication/#oidc) of the Tenant. +- In the identity provider, create a user with [correct OIDC claim](/docs/operating/setup/authentication/#oidc) of the Tenant. - In Rancher, add the new user to the `Project` with the *Read-only* `Role`. - In Rancher, add the new user to the `Cluster` with the *Tenant Member* `Cluster Role`. @@ -170,28 +170,28 @@ That Project monitoring `Namespace` will be named as `cattle-project- diff --git a/content/en/docs/overview/_index.md b/content/en/docs/overview/_index.md index 3d69a39..8eefb72 100644 --- a/content/en/docs/overview/_index.md +++ b/content/en/docs/overview/_index.md @@ -51,11 +51,11 @@ This shift-left model means Tenant Owners handle day-to-day namespace operations The Capsule controller is a Kubernetes operator that continuously watches Tenant resources and reconciles the desired state across all namespaces that belong to a tenant. When a Tenant is created or updated, the controller automatically propagates the configured policies to every namespace in that tenant. -When a Tenant Owner creates a new namespace, the controller detects it and immediately applies all inherited policies. This means tenants are always in a consistent, compliant state — even as they grow. +When a Tenant Owner creates a new namespace, the controller detects it and immediately applies all inherited policies. This means tenants are always in a consistent, compliant state, even as they grow. ![capsule-operator](/images/content/capsule-operator.png) ## Get Started -- [**Quickstart** - create your first Tenant in minutes](/docs/tenants/quickstart/) +- [**Quickstart** - create your first Tenant in minutes](/docs/quickstart/) - [**Tenant Docs** - explore everything Tenants can do](/docs/tenants/) diff --git a/content/en/docs/proxy/gangplank.md b/content/en/docs/proxy/gangplank.md index 23fbec3..3f5d4e9 100644 --- a/content/en/docs/proxy/gangplank.md +++ b/content/en/docs/proxy/gangplank.md @@ -29,7 +29,7 @@ jwt: prefix: "" ``` -[Read More](/docs/operating/authentication/#configuring-kubernetes-api-server) +[Read More](/docs/operating/setup/authentication/#configuring-kubernetes-api-server) ## Integration diff --git a/content/en/docs/proxy/reflection.md b/content/en/docs/proxy/reflection.md index 34b063e..09768d0 100644 --- a/content/en/docs/proxy/reflection.md +++ b/content/en/docs/proxy/reflection.md @@ -6,7 +6,7 @@ date: 2024-02-20 weight: 3 --- -Namespace RoleBinding reflection allows users and groups to list every Namespace in which they are referenced by a RoleBinding. The user does not need to be a /docs/operating/architecture/#tenant-owners. +Namespace RoleBinding reflection allows users and groups to list every Namespace in which they are referenced by a RoleBinding. The user does not need to be a [Tenant Owner](/docs/operating/concepts/architecture/#tenant-owners). Enable the reflector in the Helm values: @@ -27,7 +27,7 @@ The functionality of allowing access to other `Namespaces` is achieved by enabli For example, the following Role and RoleBinding allow alice to list Pods in the green-test Namespace. Because alice is referenced by the RoleBinding, Namespace reflection also allows her to see green-test when listing Namespaces through Capsule Proxy: ```yaml ---- +--- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: @@ -66,7 +66,7 @@ The RoleBinding does not require a special label for Namespace reflection. All R ### GlobalProxySettings - The same behavior can be achieved by using [`GlobalProxySettings`](/docs/proxy/proxysettings/#globalproxysettings) to enable further listing of `Namespace` resources. Note that this must use the label `"kubernetes.io/metadata.name"`. This provides the user alice the ability to list the `Namespaces` `green-test` and `green-prod` without being [Tenant Owner](/docs/operating/architecture/#tenant-owners) or having any other permissions on the `Tenant`: + The same behavior can be achieved by using [`GlobalProxySettings`](/docs/proxy/proxysettings/#globalproxysettings) to enable further listing of `Namespace` resources. Note that this must use the label `"kubernetes.io/metadata.name"`. This provides the user alice the ability to list the `Namespaces` `green-test` and `green-prod` without being [Tenant Owner](/docs/operating/concepts/architecture/#tenant-owners) or having any other permissions on the `Tenant`: ```yaml apiVersion: capsule.clastix.io/v1beta1 kind: GlobalProxySettings @@ -93,7 +93,7 @@ spec: ## Namespaced Items -For all namespaced items it's possible to grant users `LIST` permissions within any `Tenant` namespace. They don't have to be a [Tenant Owner](/docs/operating/architecture/#tenant-owners) or anything. This is useful for example for operators, where they might also want to see all pods but are not directly owner on any `Tenant`. Reflection is resolved based on **`RoleBindings`** and the associated `Roles` and `ClusterRoles` (recommended). +For all namespaced items it's possible to grant users `LIST` permissions within any `Tenant` namespace. They don't have to be a [Tenant Owner](/docs/operating/concepts/architecture/#tenant-owners) or anything. This is useful for example for operators, where they might also want to see all pods but are not directly owner on any `Tenant`. Reflection is resolved based on **`RoleBindings`** and the associated `Roles` and `ClusterRoles` (recommended). The `Role` or `ClusterRole` must provide `LIST` permissions to the allowed resource(s). The `RoleBindings` considered for reflection must always use the label `reflection.proxy.projectcapsule.dev/enabled: "true"` to be considered for reflection. The following example shows how to grant a user `LIST` permissions on all pods within any `Tenant` namespace: @@ -179,7 +179,7 @@ spec: reflection.proxy.projectcapsule.dev/enabled: "true" ``` - By default, any user can add further `RoleBindings` with the `reflection.proxy.projectcapsule.dev/enabled: "true"` label to any `RoleBinding`. To prevent that, you can deny it with another [Enforcement rule](/docs/tenants/rules/enforcement/#metadata): + By default, any user can add further `RoleBindings` with the `reflection.proxy.projectcapsule.dev/enabled: "true"` label to any `RoleBinding`. To prevent that, you can deny it with another [Enforcement rule](/docs/rules/enforcement/metadata): ```yaml --- apiVersion: capsule.clastix.io/v1beta2 diff --git a/content/en/docs/quickstart/_index.md b/content/en/docs/quickstart/_index.md index 7d471d9..52f4ec4 100644 --- a/content/en/docs/quickstart/_index.md +++ b/content/en/docs/quickstart/_index.md @@ -2,754 +2,91 @@ title: Quickstart 🚀 type: docs weight: 2 -description: "Create your first Capsule Tenant" +description: "Create your first Capsule Tenant and start using it" --- -The following quickstart guide will help you to create your first Capsule Tenant and start using it. The guide assumes that you have already installed Capsule in your cluster and that you have a working Kubernetes cluster. Also it first shows to conceptual side from the Platform (Cluster Administrator) perspective and then from the `TenantOwner` perspective. -## Installation - -Start a local Kubernetes cluster with [KinD](https://kind.sigs.k8s.io/) use the following configuration: - -[Get Here](/docs/quickstart/kind.yaml) -```yaml -# kind.yaml -apiVersion: kind.x-k8s.io/v1alpha4 -kind: Cluster -nodes: - - role: control-plane - - role: worker - extraPortMappings: - - hostPort: 9001 - containerPort: 9001 -``` - -Create dedicated `kind` cluster with the following command: - -```bash -kind create cluster --name capsule --config kind.yaml --wait=120s -``` - -We are using a custom port-mapping for the [Capsule Proxy](/docs/proxy/). After the cluster is up and running, install Capsule with the following command: - -```bash -helm upgrade --install capsule oci://ghcr.io/projectcapsule/charts/capsule --debug --create-namespace -n capsule-system --version {{< capsule_chart_version >}} \ - --set 'proxy.enabled=true' \ - --set 'proxy.certManager.generateCertificates=false' \ - --set 'proxy.options.additionalSANs={localhost}' \ - --set 'proxy.options.generateCertificates=true' \ - --set "proxy.options.leaderElection=true" \ - --set "proxy.options.roleBindingReflector=true" \ - --set "proxy.service.type=NodePort" \ - --set "proxy.kind=DaemonSet" \ - --set "proxy.daemonset.hostNetwork=true" \ - --set "proxy.serviceMonitor.enabled=false" \ - --set "proxy.options.extraArgs={--feature-gates=ProxyClusterScoped=true}" \ - --set 'crds.install=true' \ - --set 'certManager.generateCertificates=false' \ - --set 'tls.enableController=true' \ - --set 'tls.create=true' -``` - -For more information about the installation process, please refer to the [installation guide](/docs/operating/setup/installation/). Verify components are running with the following command: - -```bash -kubectl get pods -n capsule-system - -NAME READY STATUS RESTARTS AGE -capsule-controller-manager-7584dc9546-l6tgl 1/1 Running 1 (21s ago) 29s -capsule-crds-vfq9k 0/1 Completed 0 41s -capsule-post-install-2lm99 0/1 Completed 0 28s -capsule-proxy-fjl5s 0/1 Running 0 29s -capsule-proxy-certgen-5x7d6 0/1 Completed 0 29s -``` - -Great, now we are ready to create our first Capsule Tenant. The following sections will guide you through the process of creating a Tenant and configuring it. - -## Platform - -Perspective of the Cluster Administrator, who is responsible for creating and managing tenants in the cluster. The Cluster Administrator can create tenants and assign them to users or groups of users, who will then be able to manage their own namespaces within the tenant. - -### Tenants - -In Capsule, a Tenant is an abstraction to group multiple namespaces in a single entity within a set of boundaries defined by the Cluster Administrator. - -#### Ownership - -[Read More](/docs/tenants/permissions/#ownership) - -The tenant is then assigned to a user or group of users who is called [`TenantOwner`](/docs/operating/architecture/#tenant-owners). Capsule defines a Tenant as Custom Resource with cluster scope. Create the tenant as cluster admin: - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User -``` - -Here we mention the user `alice`, as of now `alice` is not considered a [**Capsule User**](/docs/operating/architecture/#capsule-users) because she was not defined [**as such in the `CapsuleConfiguration`**](/docs/operating/setup/configuration/#users). We must either define her in the configuration or create a [`TenantOwner`](/docs/tenants/permissions/#tenant-owners) resource for her or any of the groups she may belong to. In this example, we will assume that every user carries the group `projectcapsule.dev` (default setting), so we don't have to configure each user manually. Therefore the configuration should look like this: - -```yaml -manager: - options: - users: - - kind: Group - name: projectcapsule.dev -``` - -The more modern approach would be creating a dedicated [`TenantOwner`](/docs/tenants/permissions/#tenant-owners) resource for `alice`. This makes the step of adding the subject to the [`CapsuleConfiguration`](/docs/operating/setup/configuration/) obsolete, as this is done trough [**aggregation**](/docs/tenants/permissions/#aggregation). With using the label `projectcapsule.dev/tenant: "solar"` we can leverage [implicit assignment](/docs/tenants/permissions/#implicit-tenant-assignment) to the `Tenant` solar. Let's create a `TenantOwner` resource for `joe`: - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: TenantOwner -metadata: - name: joe - labels: - projectcapsule.dev/tenant: "solar" -spec: - kind: User - name: "joe" -``` - -We can always verify all [**Capsule User**](/docs/operating/architecture/#capsule-users) via the `status` of the used [`CapsuleConfiguration`](/docs/operating/setup/configuration/) resource: - -```bash -kubectl get capsuleconfiguration capsule -o jsonpath='{.status.users}' -``` - -If a user or any of their groups is not listed in the `status.users` section of the `CapsuleConfiguration` resource, they will not be able to access any tenant or namespace managed by Capsule. - - -You can check the tenant we created previously with the following command: - -```bash -$ kubectl get tnt -NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STATUS AGE -solar Active 0 True reconciled 13s -``` - -We create dedicated `TenantOwners` who represent cluster administrators. They are matched by labels defined in the `permissions.matchOwners` section of the `Tenant` spec. In our case, any user or group with the label `team: platform` is considered a `TenantOwner` for the `solar` tenant. - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: TenantOwner -metadata: - name: platform-team - labels: - team: platform -spec: - kind: Group - name: "oidc:kubernetes:admin" -``` - -We can now verify all owners of the `solar` tenant: - -```bash -kubectl get tenant solar -o jsonpath='{.status.owners}' -``` - -The result should be similar to: - -```json -[ - { - "kind": "Group", - "name": "oidc:kubernetes:admin", - "clusterRoles": [ - "admin", - "capsule-namespace-deleter" - ] - }, - { - "kind": "User", - "name": "alice", - "clusterRoles": [ - "admin", - "capsule-namespace-deleter" - ] - }, - { - "kind": "User", - "name": "joe", - "clusterRoles": [ - "admin", - "capsule-namespace-deleter" - ] - } -] -``` - -### Namespaces - -From the perspective of the Cluster Administrator, we want to mainly control the format on how namespaces are created. The actual management of the namespaces is delegated to the [`TenantOwner`s](#tenant-owners). We can define a set of rules to control how namespaces are created within a `Tenant`. - -#### Size Quota - -We can restrict the number of namespaces that can be created within a `Tenant` with [namespace quotas](/docs/tenants/quotas/#namespace-quotas). Let's cap this `Tenant` to a maximum of 2 namespaces: - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 -``` - -#### Prefix - -[Read More](/docs/tenants/administration/#force-tenant-prefix) - -We are enforcing the `Namespaces` of the `Tenant` to be prefixed with the `Tenant` name. This keeps the sorting of `Namespaces` clean and directly tells us which `Tenant` a `Namespace` belongs to. This is done with the `forceTenantPrefix` option in the `Tenant` spec: - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true -``` - -### Rules - -[Read More](/docs/tenants/rules/) - -With [Rules](/docs/tenants/rules/) we can apply different policies within a `Tenant` based on their metadata. As previously seen they can also be used to enforce metadata for `Namespaces`. This comes in handy when we have different applications environment in the same `Tenant` and we want to apply different policies to them. For example, we can have a `Tenant` with two namespaces: `solar-production` (`environment=prod`) and `solar-development` (`environment=dev`) . We can apply different rules to each namespace based on their metadata. - -#### Metadata - -[Read More](/docs/rules/enforcement/metadata/) - -Since the `Namespaces` are managed by the `TenantOwners`, we may want to require certain metadata to be present in the namespaces created within a `Tenant`. For this case we want to force the [`TenantOwners`](#tenant-owners) to provide the label `environment` with a value of either `prod`, `test` or `dev` when creating a namespace within the `solar` tenant. This can be done with [namespace metadata](/docs/tenants/metadata/#requiredmetadata): - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - rules: - - enforce: - action: allow - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - environment: - required: true - default: "dev" - values: - - exact: - - dev - - test - - prod -``` - -We may wan't to enforce a set of metadata to be applied to all namespaces created within a `Tenant`. This can be done with [namespace metadata](/docs/tenants/metadata/#additionalmetadatalist). This ensures these labels are always present in the namespaces created within the `Tenant` and can't be removed or modified. - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - rules: - - enforce: - action: allow - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - environment: - required: true - default: "dev" - values: - - exact: - - dev - - test - - prod - - enforce: - action: allow - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - pod-security.kubernetes.io/audit: - required: true - managed: "restricted" - pod-security.kubernetes.io/warn: - required: true - managed: "baseline" - - namespaceSelector: - matchExpressions: - - key: environment - operator: NotIn - values: - - prod - enforce: - action: allow - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - pod-security.kubernetes.io/enforce: - required: true - default: "restricted" - values: - - exact: - - restricted - - baseline - - namespaceSelector: - matchLabels: - environment: prod - enforce: - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - pod-security.kubernetes.io/enforce: - required: true - managed: "restricted" -``` - -We can also enforce that certain labels (also via regexp) can be enforced to only a subset of users. For example we may have labels (`"openshift.io/.*"`) which should be able to be modified by anyone else but [Capsule Users](/docs/operating/architecture/#capsule-users) - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - rules: - - audience: - - kind: Custom - name: "CapsuleUser" - enforce: - action: deny - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - "openshift.io/.*": - required: false - values: - - exp: ".*" - - enforce: - action: allow - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - environment: - required: true - default: "dev" - values: - - exact: - - dev - - test - - prod - - namespaceSelector: - matchExpressions: - - key: environment - operator: NotIn - values: - - prod - enforce: - action: allow - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - pod-security.kubernetes.io/enforce: - required: true - default: "restricted" - values: - - exact: - - restricted - - baseline - - namespaceSelector: - matchLabels: - environment: prod - enforce: - action: allow - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - pod-security.kubernetes.io/enforce: - required: true - managed: "restricted" -``` - -#### Permissions - -[Read More](/docs/tenants/rules/permissions/) - -Often you may have other users with different permissions. These are not [Tenant Owners](/docs/operating/architecture/#tenant-owners) but might be other parties that may interact with the `Tenant` and its `Namespaces`. For example, we may have a group of users that are responsible for monitoring the `Tenant` and its `Namespaces`. We can create a set of rules to allow them to view the `Tenant` and its `Namespaces` but not modify them. This can be done with [permissions rules](/docs/tenants/rules/permissions/): - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - rules: - - namespaceSelector: - matchExpressions: - - key: environment - operator: NotIn - values: - - prod - permissions: - bindings: - - clusterRoleName: 'edit' - subjects: - - kind: Group - name: tenant:{{ .tenant.metadata.name }}:operators - - namespaceSelector: - matchLabels: - environment: prod - permissions: - bindings: - - clusterRoleName: 'view' - subjects: - - kind: Group - name: tenant:{{ .tenant.metadata.name }}:operators -``` - -#### Workloads - -[Read More](/docs/rules/enforcement/workloads/) - -There might also be different requirements for the priority of workloads running in different namespaces. For example, we may want to allow `BestEffort` Pods in the `solar-development` namespace but not in the `solar-production` namespace. This can be done with [Workload Rules](/docs/rules/enforcement/workloads/#best-effort): - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - rules: - - namespaceSelector: - matchExpressions: - - key: environment - operator: NotIn - values: - - prod - enforce: - action: allow - workloads: - qosClasses: - - BestEffort - - namespaceSelector: - matchLabels: - environment: prod - enforce: - action: allow - workloads: - qosClasses: - - Guaranteed -``` - -We can also provide a subset of PriorityClasses for the Tenant to use in their workloads. This is currently only possible for the entire Tenant but will be ported to the rules in the future. For now, we can define a set of allowed PriorityClasses for the entire Tenant: - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - rules: - - namespaceSelector: - matchExpressions: - - key: environment - operator: NotIn - values: - - prod - enforce: - action: allow - workloads: - qosClasses: - - BestEffort - - namespaceSelector: - matchLabels: - environment: prod - enforce: - action: allow - workloads: - qosClasses: - - Guaranteed - priorityClasses: - matchLabels: - env: "production" -``` +This guide gets you from zero to a working multi-tenant cluster in minutes. You will install Capsule, create a Tenant as a cluster administrator, and then immediately switch to the tenant owner's perspective to see what Capsule actually does. -#### Services - -[Read More](/docs/rules/enforcement/services/) +## Installation -Often from a platform perspective, we want to control the type of services that can be created within a `Tenant`. It is possible to restrict the type of services that can be created within a `Tenant` with [Service Rules](/docs/rules/enforcement/services/#service-types). For example, we can allow only `ClusterIP` services: +Start a local Kubernetes cluster with [KinD](https://kind.sigs.k8s.io/): +[Get Here](/docs/quickstart/kind.yaml) ```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - rules: - - enforce: - action: allow - services: - types: - - ClusterIP +# kind.yaml +apiVersion: kind.x-k8s.io/v1alpha4 +kind: Cluster +nodes: + - role: control-plane + - role: worker + extraPortMappings: + - hostPort: 9001 + containerPort: 9001 ``` -You may also allow other `Service` types but be more strict in their configuration. For example, we can allow `ExternalName` services but only if they match a certain hostname pattern (forced to tenant subdomain): - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - rules: - - enforce: - action: allow - services: - types: - - ClusterIP - - ExternalName - externalNames: - hostnames: - - exp: ".*\\.{{ .tenant.metadata.name }}\\.svc\\.company\\.com" +```bash +kind create cluster --name capsule --config kind.yaml --wait=120s ``` -### Resource Quota +The extra port mapping is for the [Capsule Proxy](/docs/proxy/), which lets tenant users issue `kubectl get namespaces` and see only their own. Install Capsule with the Proxy included: + +```bash +helm upgrade --install capsule oci://ghcr.io/projectcapsule/charts/capsule --debug --create-namespace -n capsule-system --version {{< capsule_chart_version >}} \ + --set 'proxy.enabled=true' \ + --set 'proxy.certManager.generateCertificates=false' \ + --set 'proxy.options.additionalSANs={localhost}' \ + --set 'proxy.options.generateCertificates=true' \ + --set "proxy.options.leaderElection=true" \ + --set "proxy.options.roleBindingReflector=true" \ + --set "proxy.service.type=NodePort" \ + --set "proxy.kind=DaemonSet" \ + --set "proxy.daemonset.hostNetwork=true" \ + --set "proxy.serviceMonitor.enabled=false" \ + --set "proxy.options.extraArgs={--feature-gates=ProxyClusterScoped=true}" \ + --set 'crds.install=true' \ + --set 'certManager.generateCertificates=false' \ + --set 'tls.enableController=true' \ + --set 'tls.create=true' +``` -[Read More](/docs/tenants/quotas/) +Verify everything is running: -Another improtant aspect of the `Tenant` is the ability to define a set of [`ResourceQuotas`](https://kubernetes.io/docs/concepts/policy/resource-quotas/) for the entire `Tenant`. This allows the Cluster Administrator to control the amount of resources that can be used by the `Tenant` and its namespaces. For example, we can define a resource quota for the entire `Tenant`: +```bash +kubectl get pods -n capsule-system -```yaml -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - permissions: - matchOwners: - - matchLabels: - team: platform - owners: - - name: alice - kind: User - namespaceOptions: - quota: 2 - forceTenantPrefix: true - resourceQuotas: - scope: Tenant - items: - - hard: - limits.cpu: "8" - limits.memory: 16Gi - requests.cpu: "8" - requests.memory: 16Gi +NAME READY STATUS RESTARTS AGE +capsule-controller-manager-7584dc9546-l6tgl 1/1 Running 1 (21s ago) 29s +capsule-crds-vfq9k 0/1 Completed 0 41s +capsule-post-install-2lm99 0/1 Completed 0 28s +capsule-proxy-fjl5s 0/1 Running 0 29s +capsule-proxy-certgen-5x7d6 0/1 Completed 0 29s ``` -We also provide other mechanisms to control the amount of resources that can be used by the `Tenant` and its namespaces: +For more installation options see the [installation guide](/docs/operating/setup/installation/). -* [Resource Management](/docs/resource-management/) +## Create Your First Tenant -### Full Tenant +A **Tenant** groups one or more namespaces under a shared set of policies and limits. The cluster administrator creates and owns tenants. Users assigned as `TenantOwner` manage namespaces within them, without needing cluster-admin rights. -Here we have two `Tenants` with different rules and permissions. The `solar` tenant is a production tenant with multiple application stages with strict rules and permissions, while the `lunar` tenant is a development tenant with more relaxed rules and permissions. - -[Get Here](/docs/quickstart/full-tenant.yaml) +Apply the following Tenant as cluster admin: ```yaml -# solar.yaml ---- apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: name: solar spec: - permissions: - matchOwners: - - matchLabels: - team: platform owners: - name: alice kind: User - - name: bob - kind: User namespaceOptions: quota: 2 forceTenantPrefix: true - resourceQuotas: - scope: Tenant - items: - - hard: - limits.cpu: "8" - limits.memory: 16Gi - requests.cpu: "8" - requests.memory: 16Gi rules: - - audience: - - kind: Custom - name: "CapsuleUser" - enforce: - action: deny - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - "openshift.io/.*": - required: false - values: - - exp: ".*" - enforce: action: allow metadata: - - apiGroups: + - apiGroups: - "v1" kinds: - "Namespace" @@ -762,521 +99,194 @@ spec: - dev - test - prod - services: - types: - - ClusterIP - - ExternalName - externalNames: - hostnames: - - exp: ".*\\.{{ .tenant.metadata.name }}\\.svc\\.company\\.com" - namespaceSelector: - matchExpressions: - - key: environment - operator: NotIn - values: - - prod - permissions: - bindings: - - clusterRoleName: 'edit' - subjects: - - kind: Group - name: tenant:{{ .tenant.metadata.name }}:operators + matchLabels: + environment: prod enforce: action: allow workloads: qosClasses: - Guaranteed - - BestEffort - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - pod-security.kubernetes.io/enforce: - required: true - default: "restricted" - values: - - exact: - - restricted - - baseline - namespaceSelector: - matchLabels: - environment: prod - permissions: - bindings: - - clusterRoleName: 'view' - subjects: - - kind: Group - name: tenant:{{ .tenant.metadata.name }}:operators + matchExpressions: + - key: environment + operator: NotIn + values: + - prod enforce: action: allow workloads: qosClasses: + - BestEffort + - Burstable - Guaranteed - metadata: - - apiGroups: - - "v1" - kinds: - - "Namespace" - labels: - pod-security.kubernetes.io/enforce: - required: true - managed: "restricted" -``` - -Once applied we can verify the `Tenant` with the following command: - -```bash -kubectl get tnt solar - -NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STATUS AGE -solar Active 2 0 True reconciled 35s ``` -### Replications - -[Read More](/docs/replications/) - -From a platform perspective, we may want to enforce certain objects per `Namespace` of `Tenant's`. With Replications we can enforce certain objects to be present in all `Namespaces` of a `Tenant`. See the following examples for common use cases of [replications](/docs/replications/). - -#### Example: Networkpolicies - -Distribute a [`NetworkPolicy`](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to all `Namespaces` of a `Tenant` to enforce a certain network policy for all workloads within the `Tenant`/`Namespace`. The following `NetworkPolicy` is an attempt to achieve a default deny policy for all `Namespaces` of the `Tenant` but allow intra-namespace communication and allow communication between all `Namespaces` of the same `Tenant`. It also allows communication to system namespaces (eg. monitoring, ingress, etc.). [Read More](https://kubernetes.io/docs/concepts/security/multi-tenancy/#network-isolation) - -[Get Here](/docs/quickstart/gtr-netpol.yaml) - -```yaml ---- -apiVersion: capsule.clastix.io/v1beta2 -kind: GlobalTenantResource -metadata: - name: default-networkpolicies -spec: - resyncPeriod: 60s - resources: - - rawItems: - - apiVersion: networking.k8s.io/v1 - kind: NetworkPolicy - metadata: - name: default-policy - spec: - # Apply to all pods in this namespace - podSelector: {} - policyTypes: - - Ingress - - Egress - ingress: - # Allow traffic from the same namespace (intra-namespace communication) - - from: - - podSelector: {} - - # Allow traffic from all namespaces within the tenant - - from: - - namespaceSelector: - matchLabels: - capsule.clastix.io/tenant: "{{tenant.name}}" - - # Allow ingress from other namespaces labeled (System Namespaces, eg. Monitoring, Ingress) - - from: - - namespaceSelector: - matchLabels: - company.com/system: "true" - - egress: - # Allow DNS to kube-dns service IP (might be different in your setup) - - to: - - ipBlock: - cidr: 10.96.0.10/32 - ports: - - protocol: UDP - port: 53 - - protocol: TCP - port: 53 - - # Allow traffic to all namespaces within the tenant - - to: - - namespaceSelector: - matchLabels: - capsule.clastix.io/tenant: "{{tenant.name}}" -``` +What this configures: -#### Example: LimitRanges +- **`owners`**: `alice` is the Tenant Owner and can create namespaces inside this tenant. +- **`namespaceOptions.quota: 2`**: alice can create at most 2 namespaces. [Read more](/docs/tenants/namespaces/#namespace-quota) +- **`forceTenantPrefix: true`**: every namespace must start with `solar-`. [Read more](/docs/tenants/administration/#force-tenant-prefix) +- **`rules`**: the `environment` label is required on every namespace, with `dev` as the default. Capsule enforces this at admission time. [Read more](/docs/rules/enforcement/metadata/) +- **`QoS rules`**: production namespaces (labeled `environment=prod`) only accept `Guaranteed` pods. Development and test namespaces accept any QoS class. [Read more](/docs/rules/enforcement/workloads/) -[LimitRanges](https://kubernetes.io/docs/concepts/policy/limit-range/) can be used to enforce resource limits and requests for containers in a namespace. The following example enforces different `LimitRanges` for different environments (dev, test, prod) within the same `Tenant`. This ensures that workloads in each environment adhere to the specified resource constraints. +### Tenant Owners -[Get Here](/docs/quickstart/gtr-limitranges.yaml) +Capsule only acts on requests from subjects it recognises as **Capsule Users**. The recommended way to register a user is to create a `TenantOwner` resource. The label `projectcapsule.dev/tenant: "solar"` binds it to the tenant automatically via [aggregation](/docs/tenants/permissions/#aggregation): ```yaml ---- apiVersion: capsule.clastix.io/v1beta2 -kind: GlobalTenantResource +kind: TenantOwner metadata: - name: limitranges + name: alice + labels: + projectcapsule.dev/tenant: "solar" spec: - resyncPeriod: 60s - resources: - - namespaceSelector: - matchLabels: - environment: dev - rawItems: - - apiVersion: v1 - kind: LimitRange - metadata: - name: service-level-bronze - spec: - limits: - - max: - cpu: 0 - memory: "0" - min: - cpu: 0 - memory: "0" - type: Container - - - namespaceSelector: - matchLabels: - environment: test - rawItems: - - apiVersion: v1 - kind: LimitRange - metadata: - name: service-level-silver - spec: - limits: - - default: - memory: "256Mi" - defaultRequest: - cpu: 128m - memory: "256Mi" - type: Container - - - namespaceSelector: - matchLabels: - environment: prod - rawItems: - - apiVersion: v1 - kind: LimitRange - metadata: - name: service-level-gold - spec: - limits: - - default: - cpu: 128m - memory: "256Mi" - defaultRequest: - cpu: 128m - memory: "256Mi" - type: Container -``` - -## Tenant Owners - -Each tenant comes with a delegated user or group of users acting as the tenant admin. In the Capsule jargon, this is called the [`TenantOwner`s](/docs/operating/architecture/#tenant-owners). Other users can operate inside a tenant with different levels of permissions and authorizations assigned directly by the `TenantOwner`. - -Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by --capsule-user-group option, which defaults to `capsule.clastix.io`. - -Assignment to a group depends on the authentication strategy in your cluster. - -For example, if you are using capsule.clastix.io, users authenticated through a X.509 certificate must have capsule.clastix.io as Organization: `-subj "/CN=${USER}/O=capsule.clastix.io"` - -Users authenticated through an OIDC token must have in their token: - -```json -"users_groups": [ - "projectcapsule.dev", - "other_group" -] + kind: User + name: "alice" ``` -### Proxy Access - -The [hack/create-user.sh](https://github.com/projectcapsule/capsule/blob/main/hack/create-user.sh) can help you set up a dummy kubeconfig for the alice user acting as owner of a tenant called solar. +Capsule matches users by the groups they carry on every request. Creating a `TenantOwner` registers the subject automatically - no manual configuration needed. You can verify who is recognised at any time: ```bash -curl -s https://raw.githubusercontent.com/projectcapsule/capsule/main/hack/create-user.sh | bash -s -- alice solar projectcapsule.dev,other_group +kubectl get capsuleconfiguration default -o jsonpath='{.status.users}' | jq ``` -Now we are also injecting the [Capsule Proxy](/docs/proxy/) as Kubernetes API Server. The Capsule Proxy is a Kubernetes API Server that acts as a reverse proxy to the Kubernetes API Server. It mainly allows to issue `LIST` and `GET` requests to Kubernetes API Server across all namespaces and tenants. - -```bash -KUBECONFIG=alice-solar.kubeconfig kubectl config set clusters.kind-capsule.certificate-authority-data $(kubectl -n capsule-system get secret capsule-proxy -o jsonpath='{.data.ca}') -KUBECONFIG=alice-solar.kubeconfig kubectl config set clusters.kind-capsule.server https://localhost:9001 -export KUBECONFIG=alice-solar.kubeconfig -kubectl get ns -A -``` +For the quickstart we use impersonation (`--as-group projectcapsule.dev`) which bypasses the need for a real certificate or token. In production, authentication depends on your cluster setup (X.509 certificates, OIDC tokens, etc.), use the [Gangplank](/docs/proxy/gangplank/) workflow to issue real kubeconfigs. -Alice can now always `LIST` their `Tenants`: +Verify the Tenant is active and alice is listed as an owner: ```bash -kubectl get tnt +kubectl get tnt solar NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STATUS AGE -solar Active 2 0 True reconciled 37m +solar Active 2 0 True reconciled 10s ``` -In production environments this process can be automated with [Gangplank](/docs/proxy/gangplank/) - -### Impersonation - -You can simulate this behavior by using [impersonation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation): - ```bash -kubectl --as alice --as-group projectcapsule.dev ... +kubectl get tenant solar -o jsonpath='{.status.owners}' | jq ``` -However with this you might hit certain limitations regarding `Namespaces`: +## As a Tenant Owner -```shell -kubectl --as alice --as-group projectcapsule.dev label namespace solar-development pod-security.kubernetes.io/enforce=baseline --overwrite -o yaml +Now switch to alice's perspective. Use [impersonation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation) to simulate her identity: -Error from server (Forbidden): namespaces "solar-development" is forbidden: User "alice" cannot get resource "namespaces" in API group "" in the namespace "solar-development" +```bash +# All following commands run as alice +alias kubectl-alice='kubectl --as alice --as-group projectcapsule.dev' ``` -### Manage Namespaces +### Create a namespace -As `TenantOwner` (`alice`), we attempt to create a namespace within the `solar` tenant. The `TenantOwner` can create namespaces within the tenant they own. Let's attempt to create a `Namespace` called `development`: +Try creating a namespace without the required prefix: ```bash -kubectl projectcapsule.dev create namespace development +kubectl-alice create namespace development ``` -You will be denied with the following error: - -```bash +``` Error from server (Forbidden): admission webhook "namespaces.mutating.projectcapsule.dev" denied the request: The Namespace name must start with 'solar-' when ForceTenantPrefix is enabled in the Tenant. ``` -Since we have enabled the `forceTenantPrefix` option in the `Tenant` spec, we must create namespaces with the prefix of the tenant name (`solar-`) . However capsule correctly identified that `alice` belongs to the `solar` `Tenant` and allowed the creation of the namespace with the correct prefix. This works because `alice` currently belongs to a single `Tenant`. If `alice` belonged to multiple tenants, she would have to specify the tenant name in the namespace name ([Read More](/docs/tenants/namespaces/#multiple-tenants)). - -Let's try again with the name `solar-development`: +Capsule immediately enforces the naming rule. Try with the correct prefix: ```bash -kubectl create namespace solar-development -o yaml +kubectl-alice create namespace solar-development -o yaml ``` -This has worked, we can also observe that based on the rules defined in the `Tenant`, the namespace has been automatically labeled with `environment=dev` and the pod security labels have been applied: +The namespace is created and Capsule automatically applies the default label `environment=dev`: ```yaml apiVersion: v1 kind: Namespace metadata: - creationTimestamp: "2026-07-23T08:26:12Z" labels: capsule.clastix.io/tenant: solar environment: dev kubernetes.io/metadata.name: solar-development - pod-security.kubernetes.io/enforce: restricted name: solar-development - ownerReferences: - - apiVersion: capsule.clastix.io/v1beta2 - kind: Tenant - name: solar - uid: 3e95e43d-9ef7-4ba7-bc66-79af19ca8021 - resourceVersion: "68295" - uid: cc49f3ba-ba51-4430-af2d-ab4a40369bce ``` -Maybe `pod-security.kubernetes.io/enforce=restricted` is a bit too strict for a development environment, so let's change it's value to `baseline`: - -```bash -kubectl label namespace solar-development pod-security.kubernetes.io/enforce=baseline --overwrite -o yaml -``` +### Enforce label constraints -This is allowed by the rules defined in the `Tenant` since the namespace is labeled with `environment=dev` and the rule allows `pod-security.kubernetes.io/enforce` to be set to either `restricted` or `baseline`. If we try to set it to `privileged` we will be denied: +The `environment` label can only be set to `dev`, `test`, or `prod`. Try to label the namespace with a value that is not allowed: ```bash -kubectl label namespace solar-development pod-security.kubernetes.io/enforce=privileged --overwrite -o yaml +kubectl-alice label namespace solar-development environment=staging --overwrite ``` -```bash -Error from server (Forbidden): admission webhook "namespaces.validating.projectcapsule.dev" denied the request: metadata label "privileged" at metadata.labels["pod-security.kubernetes.io/enforce"] is not allowed by namespace rule: value did not match any allowed rule. Allowed metadata values: exact: restricted, baseline +``` +Error from server (Forbidden): admission webhook "namespaces.validating.projectcapsule.dev" denied the request: metadata label "staging" at metadata.labels["environment"] is not allowed by namespace rule: value did not match any allowed rule. Allowed metadata values: exact: dev, test, prod ``` -When we are using the [Capsule Proxy](/docs/proxy/) we can now issue `LIST` and `GET` requests to the Kubernetes API Server across all namespaces and tenants and only get the resources from `Tenant` alice is owner of: +Allowed values work fine: ```bash -kubectl get ns -A - -NAME STATUS AGE -solar-development Active 4m45s +kubectl-alice label namespace solar-development environment=test --overwrite ``` -Next up we are trying to create namespace in the `environment=prod` which is only allowed to be created with the `pod-security.kubernetes.io/enforce=restricted` label. Let's try to create a namespace called `solar-production`: +### Namespace quota + +Create a second namespace, this time explicitly as production: ```bash -kubectl apply --server-side=true -o yaml -f - < 4m34s -solar-production default-policy 59s -``` - -Alice is also unable to delete these policies, eventough she has full permissions to manage `Networkpolicies`. This allows here to do here own firewalling (maybe among her `Tenants`/`Namespaces`) but enforces strictness by the platform to ensure that all `Namespaces` of the `Tenant` have a default deny policy in place: - -```bash - kubectl delete netpol -n solar-development --all - -Error from server (Forbidden): admission webhook "replications.validating.projectcapsule.dev" denied the request: resource default-policy is managed by a global capsule replication default-networkpolicies -``` +### List with the Proxy -Since we provide all namespaces with a default network policy, we can see that the `default-policy` has been replicated in all namespaces of the `Tenant`. The same goes for the limitranges, however here we have conditions based on the `environment` label of the namespace: +Without the Proxy, `kubectl get namespaces -A` returns `Forbidden` for non-admin users. Point alice's kubeconfig to the Capsule Proxy to get a filtered view: ```bash -kubectl get limitrange -A - -NAMESPACE NAME CREATED AT -solar-development service-level-bronze 2026-07-23T10:43:07Z -solar-production service-level-gold 2026-07-23T10:46:33Z +curl -s https://raw.githubusercontent.com/projectcapsule/capsule/main/hack/create-user.sh | bash -s -- alice solar projectcapsule.dev +KUBECONFIG=alice-solar.kubeconfig kubectl config set clusters.kind-capsule.certificate-authority-data $(kubectl -n capsule-system get secret capsule-proxy -o jsonpath='{.data.ca}') +KUBECONFIG=alice-solar.kubeconfig kubectl config set clusters.kind-capsule.server https://localhost:9001 +export KUBECONFIG=alice-solar.kubeconfig ``` -### Common Interactions - -Next up we can try to schedule a pod in the `solar-development` `Namespace`: +Now list namespaces; alice sees only hers: ```bash -kubectl -n solar-development run nginx --image=docker.io/nginx -``` - -Here we don't have to do anything special, since QOS class allow BestEffort (no resources) and PSS enforces baseline (no privileged containers) we can schedule a pod without any issues. +kubectl get ns -A -```bash -kubectl -n solar-development run nginx --image=docker.io/nginx -kubectl -n solar-development get pods -w +NAME STATUS AGE +solar-development Active 5m +solar-production Active 2m ``` -If you try to schedule a pod in the `solar-production` `Namespace` you will be denied since the QOS class only allows Guaranteed (resources must be specified) and PSS enforces restricted (no privileged containers): +In production, automate kubeconfig distribution with [Gangplank](/docs/proxy/gangplank/). -```bash -kubectl apply --server-side=true -n solar-production -o yaml -f - < + Optional deep-dive into Capsule rules, permissions, and resource distribution. +--- + +{{% alert title="Optional" color="info" %}} +This page builds on the [basic quickstart](/docs/quickstart/). It shows more of what Capsule can do once you have a working Tenant. +{{% /alert %}} + +The examples below extend the `solar` Tenant from the quickstart. Apply each section on top of the existing Tenant with `kubectl apply`. + +--- + +## Pod Security Standards + +[Read More](/docs/rules/enforcement/metadata/) + +Kubernetes [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) (PSS) control what security contexts are allowed in a namespace. Capsule can enforce a PSS level per environment and even **manage** a label so users cannot override it. + +Add the following rules to the `solar` Tenant: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + owners: + - name: alice + kind: User + namespaceOptions: + quota: 2 + forceTenantPrefix: true + rules: + - enforce: + action: allow + metadata: + - apiGroups: + - "v1" + kinds: + - "Namespace" + labels: + environment: + required: true + default: "dev" + values: + - exact: + - dev + - test + - prod + + # dev and test: allow restricted or baseline, default to restricted + - namespaceSelector: + matchExpressions: + - key: environment + operator: NotIn + values: + - prod + enforce: + action: allow + workloads: + qosClasses: + - BestEffort + - Burstable + - Guaranteed + metadata: + - apiGroups: + - "v1" + kinds: + - "Namespace" + labels: + pod-security.kubernetes.io/enforce: + required: true + default: "restricted" + values: + - exact: + - restricted + - baseline + + # prod: lock pod-security to restricted, users cannot change it + - namespaceSelector: + matchLabels: + environment: prod + enforce: + action: allow + workloads: + qosClasses: + - Guaranteed + metadata: + - apiGroups: + - "v1" + kinds: + - "Namespace" + labels: + pod-security.kubernetes.io/enforce: + required: true + managed: "restricted" +``` + +The key difference between `values` and `managed`: + +- `values` defines what a user is **allowed** to set. +- `managed` means Capsule owns the value. It is applied automatically and any attempt to change it is silently corrected by the webhook. + +### See it in action + +Create a production namespace and try to set a permissive PSS level: + +```bash +kubectl-alice apply -f - < Capsule provides two dedicated Custom Resource Definitions for propagating Kubernetes resources across Tenant Namespaces, covering both the cluster administrator and Tenant owner personas: -- **[GlobalTenantResource](./global/)** — cluster-scoped, managed by cluster administrators. Selects Tenants by label and replicates resources into all matching Tenant Namespaces. -- **[TenantResource](./tenant/)** — namespace-scoped, managed by Tenant owners. Replicates resources across the Namespaces within a single Tenant. +- **[GlobalTenantResource](./global/)**: cluster-scoped, managed by cluster administrators. Selects Tenants by label and replicates resources into all matching Tenant Namespaces. +- **[TenantResource](./tenant/)**: namespace-scoped, managed by Tenant owners. Replicates resources across the Namespaces within a single Tenant. -Both CRDs follow the same structure: resources are defined in `spec.resources` blocks, reconciled on a configurable `resyncPeriod`, and support [Go-template-based generators](/docs/operating/templating/) for dynamic resource creation. +Both CRDs follow the same structure: resources are defined in `spec.resources` blocks, reconciled on a configurable `resyncPeriod`, and support [Go-template-based generators](/docs/operating/concepts/templating/) for dynamic resource creation. diff --git a/content/en/docs/replications/global.md b/content/en/docs/replications/global.md index ac57de9..8b7bb8b 100644 --- a/content/en/docs/replications/global.md +++ b/content/en/docs/replications/global.md @@ -13,6 +13,10 @@ The diagram below shows that an Administrator can create a `GlobalTenantResource ![Global Tenant Resource Replication overview](/images/content/replication-globaltenantresource.png) +{{% alert title="OpenShift: etcd Encryption" color="warning" %}} +If you are running on OpenShift with etcd encryption enabled and replicating `ConfigMap`s or `Secret`s, you must exclude the OpenShift storage version migrator from the replication webhook. Without this, the migrator cannot rotate encryption keys. See the [OpenShift installation guide](/docs/operating/setup/openshift/#etcd-encryption) for the required configuration. +{{% /alert %}} + A common use case is distributing image pull secrets to all Tenants that must use a specific container registry. In the following example, Bill labels two Tenants and then creates a `GlobalTenantResource` to push the corresponding pull secret into each of their Namespaces automatically. ```bash @@ -83,7 +87,7 @@ The `namespaceSelector` field restricts replication to Namespaces matching a lab #### AdditionalMetadata -Use `additionalMetadata` to attach extra `labels` and `annotations` to every generated object. [Fast Template values](/docs/operating/templating/#fast-templates) are supported: +Use `additionalMetadata` to attach extra `labels` and `annotations` to every generated object. [Fast Template values](/docs/operating/concepts/templating/#fast-templates) are supported: ```yaml --- @@ -244,7 +248,7 @@ spec: projectcapsule.dev/replicate: "true" ``` -[Fast Templates](/docs/operating/templating/#fast-templates) are supported for `namespace`: +[Fast Templates](/docs/operating/concepts/templating/#fast-templates) are supported for `namespace`: ```yaml --- @@ -320,7 +324,7 @@ metadata: uid: 5f10a3f3-863e-4f45-9454-cff8f5bce86a ``` -[Fast Templates](/docs/operating/templating/#fast-templates) are supported for `selector`: +[Fast Templates](/docs/operating/concepts/templating/#fast-templates) are supported for `selector`: ```yaml --- @@ -342,7 +346,7 @@ spec: #### Raw -Raw items let you define resources inline as standard Kubernetes manifests. Use this when the resource does not yet exist in the cluster, or when you want to define it directly in the spec. [Fast Templates](/docs/operating/templating/#fast-templates) are supported. +Raw items let you define resources inline as standard Kubernetes manifests. Use this when the resource does not yet exist in the cluster, or when you want to define it directly in the spec. [Fast Templates](/docs/operating/concepts/templating/#fast-templates) are supported. ```yaml --- @@ -405,7 +409,7 @@ For more advanced templating, consider [Generators](#generators). #### Generators -Generators render one or more Kubernetes objects from a Go template string. The template content must be valid YAML; multi-document output separated by `---` is supported. The template engine is based on [go-sprout](https://github.com/go-sprout/sprout) - see [available functions](/docs/operating/templating/#sprout-templating). +Generators render one or more Kubernetes objects from a Go template string. The template content must be valid YAML; multi-document output separated by `---` is supported. The template engine is based on [go-sprout](https://github.com/go-sprout/sprout) - see [available functions](/docs/operating/concepts/templating/#sprout-templating). A simple example that creates a `ClusterRole` per Tenant: @@ -1528,7 +1532,7 @@ spec: ### Collect HTTPRoutes within per Tenant and aggregate to managed gateway -The following example solves a common problem with Gateway-API and Certificate management. Assume you have a managed gateway with a managed cluster-issuer. In this case we can load all the HTTPRoutes and template the corresponding tenant. The following example also allows to modify the EnvoyProxy instance with [Tenant Data](/docs/operating/templating/#data) +The following example solves a common problem with Gateway-API and Certificate management. Assume you have a managed gateway with a managed cluster-issuer. In this case we can load all the HTTPRoutes and template the corresponding tenant. The following example also allows to modify the EnvoyProxy instance with [Tenant Data](/docs/operating/concepts/templating/#data) ```yaml --- diff --git a/content/en/docs/replications/tenant.md b/content/en/docs/replications/tenant.md index 1e20f9f..4b58ed4 100644 --- a/content/en/docs/replications/tenant.md +++ b/content/en/docs/replications/tenant.md @@ -13,6 +13,10 @@ The diagram below shows that an Administrator or a Tenant Owner can create a `Te ![Tenant Resource Replication overview](/images/content/replication-tenantresource.png) +{{% alert title="OpenShift: etcd Encryption" color="warning" %}} +If you are running on OpenShift with etcd encryption enabled and replicating `ConfigMap`s or `Secret`s, you must exclude the OpenShift storage version migrator from the replication webhook. Without this, the migrator cannot rotate encryption keys. See the [OpenShift installation guide](/docs/operating/setup/openshift/#etcd-encryption) for the required configuration. +{{% /alert %}} + ## Prerequisites Tenant owners must have RBAC permission to create, update, and delete `TenantResource` objects. The following `ClusterRole` aggregates to the `admin` role, granting all holders permission to manage `TenantResource` instances: @@ -100,7 +104,7 @@ The `namespaceSelector` field restricts replication to Namespaces matching a lab #### AdditionalMetadata -Use `additionalMetadata` to attach extra `labels` and `annotations` to every generated object. [Fast Template values](/docs/operating/templating/#fast-templates) are supported: +Use `additionalMetadata` to attach extra `labels` and `annotations` to every generated object. [Fast Template values](/docs/operating/concepts/templating/#fast-templates) are supported: ```yaml --- @@ -230,7 +234,7 @@ spec: This distributes the `ConfigMap` named `logging-config` to all other Namespaces of the Tenant that `wind-test` belongs to. -[Fast Templates](/docs/operating/templating/#fast-templates) are supported for `name`, `namespace`, and `selector`. +[Fast Templates](/docs/operating/concepts/templating/#fast-templates) are supported for `name`, `namespace`, and `selector`. ##### Namespace @@ -251,7 +255,7 @@ spec: optional: true ``` -[Fast Templates](/docs/operating/templating/#fast-templates) are supported for the `namespace` property: +[Fast Templates](/docs/operating/concepts/templating/#fast-templates) are supported for the `namespace` property: ```yaml --- @@ -325,7 +329,7 @@ metadata: uid: 5f10a3f3-863e-4f45-9454-cff8f5bce86a ``` -[Fast Templates](/docs/operating/templating/#fast-templates) are supported for `selector`: +[Fast Templates](/docs/operating/concepts/templating/#fast-templates) are supported for `selector`: ```yaml --- @@ -347,7 +351,7 @@ spec: #### Raw -Raw items let you define resources inline as standard Kubernetes manifests. Use this when the resource does not yet exist in the cluster, or when you want to define it directly in the spec. [Fast Templates](/docs/operating/templating/#fast-templates) are supported. +Raw items let you define resources inline as standard Kubernetes manifests. Use this when the resource does not yet exist in the cluster, or when you want to define it directly in the spec. [Fast Templates](/docs/operating/concepts/templating/#fast-templates) are supported. ```yaml --- @@ -410,7 +414,7 @@ For more advanced templating, consider [Generators](#generators). #### Generators -Generators render one or more Kubernetes objects from a Go template string. The template content must be valid YAML; multi-document output separated by `---` is supported. The template engine is based on [go-sprout](https://github.com/go-sprout/sprout) - see [available functions](/docs/operating/templating/#sprout-templating). +Generators render one or more Kubernetes objects from a Go template string. The template content must be valid YAML; multi-document output separated by `---` is supported. The template engine is based on [go-sprout](https://github.com/go-sprout/sprout) - see [available functions](/docs/operating/concepts/templating/#sprout-templating). A simple example that creates a `ClusterRole` per Tenant: diff --git a/content/en/docs/resource-management/resourcepools/_index.md b/content/en/docs/resource-management/resourcepools/_index.md index ef1154f..9f4621e 100644 --- a/content/en/docs/resource-management/resourcepools/_index.md +++ b/content/en/docs/resource-management/resourcepools/_index.md @@ -22,7 +22,7 @@ description: > It is then up to the group of users within those namespaces to manage the resources they consume per namespace. Each ResourcePool provisions a ResourceQuota into all the selected namespaces. Essentially, when `ResourcePoolClaims` are assigned to a ResourcePool, they stack additional resources on top of that `ResourceQuota`, based on the namespace from which the `ResourcePoolClaim` was created. -You can create any number of `ResourcePools` for any kind of namespace — they do not need to be part of a Tenant. Note that the usual ResourceQuota mechanisms apply when, for example, the same resources are defined in multiple `ResourcePools` for the same namespaces (e.g., the lowest defined quota for a resource is always considered). +You can create any number of `ResourcePools` for any kind of namespace; they do not need to be part of a Tenant. Note that the usual ResourceQuota mechanisms apply when, for example, the same resources are defined in multiple `ResourcePools` for the same namespaces (e.g., the lowest defined quota for a resource is always considered). @@ -238,9 +238,9 @@ Options that can be defined on a per-`ResourcePool` basis and influence the gene #### OrderedQueue -When `ResourecePoolClaims` are allocated to a pool, they are placed in a queue. The pool attempts to allocate claims in the order of their [creation timestamps](#priority). However, even if a claim was created earlier, if it requests more resources than are currently available, it will remain in the queue. Meanwhile, a lower-priority claim that fits within the available resources may still be allocated—despite its lower priority. +When `ResourecePoolClaims` are allocated to a pool, they are placed in a queue. The pool attempts to allocate claims in the order of their [creation timestamps](#priority). However, even if a claim was created earlier, if it requests more resources than are currently available, it will remain in the queue. Meanwhile, a lower-priority claim that fits within the available resources may still be allocated, despite its lower priority. -Enabling this option enforces strict ordering: claims cannot be skipped, even if they block other claims from being fulfilled due to resource exhaustion. The `CreationTimestamp` is strictly respected, meaning that once a claim is queued, no subsequent claim can bypass it—even if it requires fewer resources. +Enabling this option enforces strict ordering: claims cannot be skipped, even if they block other claims from being fulfilled due to resource exhaustion. The `CreationTimestamp` is strictly respected, meaning that once a claim is queued, no subsequent claim can bypass it, even if it requires fewer resources. **Default**: `false` @@ -252,7 +252,7 @@ Sets the [default values](#defaults) for the `ResourceQuota` created for the `Re #### DeleteBoundResources -By default, when a `ResourcePool` is deleted, any `ResourcePoolClaims` bound to it are only disassociated—not deleted. Enabling this option ensures that all `ResourcePoolClaims` in a bound state are deleted when the corresponding `ResourcePool` is deleted. +By default, when a `ResourcePool` is deleted, any `ResourcePoolClaims` bound to it are only disassociated, not deleted. Enabling this option ensures that all `ResourcePoolClaims` in a bound state are deleted when the corresponding `ResourcePool` is deleted. **Default**: `false` diff --git a/content/en/docs/rules/_index.md b/content/en/docs/rules/_index.md index 643b0ad..a27d259 100644 --- a/content/en/docs/rules/_index.md +++ b/content/en/docs/rules/_index.md @@ -5,4 +5,9 @@ description: > Configure policies and restrictions on a per-Namespace basis with Rules --- -Enforcement rules allow Bill, the cluster administrator, to set policies and restrictions on a per-`Tenant` basis. These rules are enforced by Capsule admission webhooks when Alice, the `TenantOwner`, creates or modifies resources in her `Namespaces`. With the rule construct, namespaces within the same tenant can be profiled differently depending on their metadata. \ No newline at end of file +Enforcement rules allow Bill, the cluster administrator, to set policies and restrictions on a per-`Tenant` basis. These rules are enforced by Capsule admission webhooks when Alice, the `TenantOwner`, creates or modifies resources in her `Namespaces`. With the rule construct, namespaces within the same tenant can be profiled differently depending on their metadata. + +Rules cover two areas: + +- **[Enforcement](/docs/rules/enforcement/)**: control allowed workloads, ingress hostnames, service types, and namespace metadata. +- **[Permissions](/docs/rules/permissions/)**: distribute RoleBindings and promote ServiceAccounts across Tenant namespaces. diff --git a/content/en/docs/rules/permissions.md b/content/en/docs/rules/permissions.md new file mode 100644 index 0000000..c16aa9d --- /dev/null +++ b/content/en/docs/rules/permissions.md @@ -0,0 +1,466 @@ +--- +title: Permissions +weight: 2 +description: > + Configure policies and restrictions on a per-Tenant basis with Rules +aliases: + - /docs/tenants/rules/permissions/ + - /docs/tenants/rules/ +--- + +Declare permission distribution rules for the selected namespaces. + +## Bindings + +With `Tenant` RoleBindings you can distribute namespaced RoleBindings to all namespaces which are assigned to a `Tenant`. This ensures the defined RoleBindings are present and reconciled in all namespaces of the `Tenant`. This is useful if users should have more insights on a `Tenant` basis. Let's look at an example. + +Assuming a cluster-administrator creates the following clusterRole: + +```yaml +kubectl apply -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: prometheus-servicemonitors-viewer +rules: +- apiGroups: ["monitoring.coreos.com"] + resources: ["servicemonitors"] + verbs: ["get", "list", "watch"] +EOF +``` + + Now the cluster administrator wants to bind this ClusterRole in each namespace of the solar `Tenant`. They can configure this with a `Tenant` manifest: + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + owners: + - name: alice + kind: User + rules: + - permissions: + bindings: + - clusterRoleName: 'prometheus-servicemonitors-viewer' + subjects: + - kind: User + name: alice + labels: + projectcapsule.dev/sample: "true" + annotations: + projectcapsule.dev/sample: "true" +EOF +``` + + As you can see, `subjects` uses the standard [RoleBinding subject](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-subjects) format. This grants permissions to the subject user **alice**, who can get, list, and watch ServiceMonitors in the solar Tenant namespaces, but has no other permissions. + +### Strict + +If you have [strict RBAC enabled for the controller](/docs/operating/setup/installation/#strict-rbac), you need to ensure that the controller ServiceAccount has the permission to create RoleBindings for the specified ClusterRole. The Controller Aggregates ClusterRoles with the labels (OR): + + - `projectcapsule.dev/aggregate-to-controller: "true"` + - `projectcapsule.dev/aggregate-to-controller-instance: {{ .Release.Name }}` + +So for the above example, you need to label the `prometheus-servicemonitors-viewer` ClusterRole like this: + +```yaml +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: prometheus-servicemonitors-viewer + labels: + projectcapsule.dev/aggregate-to-controller: "true" +rules: +- apiGroups: ["monitoring.coreos.com"] + resources: ["servicemonitors"] + verbs: ["get", "list", "watch"] +``` + +### Distribution + + You may have the use-case where you want to distribute different ClusterRoles to different namespaces of the same `Tenant`. For example, you want to give `view` permissions to an operational group in all namespaces of the solar `Tenant` with `environment=prod` label, but you want to give `edit` permissions to the operations group in all other namespaces. You can achieve this by leveraging [GlobalTenantResources](/docs/replications/global/): + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + owners: + - name: alice + kind: User + - name: joe + kind: User + rules: + - namespaceSelector: + matchExpressions: + - key: environment + operator: NotIn + values: + - prod + permissions: + bindings: + - clusterRoleName: 'edit' + subjects: + - kind: Group + name: tenant:{{ .tenant.metadata.name }}:operators + - namespaceSelector: + matchLabels: + environment: prod + permissions: + bindings: + - clusterRoleName: 'view' + subjects: + - kind: Group + name: tenant:{{ .tenant.metadata.name }}:operators +``` + +### Built-in ClusterRoles + +We strongly recommend you use custom ClusterRoles for your `Tenant` rolebindings, but you can also use built-in ClusterRoles (`admin` (default for Tenant Owners), `view` and `edit`). For example, if you want to give the `view` permissions to Joe in all namespaces of the solar `Tenant`, you can use the built-in `view` ClusterRole. + +In that case it also makes sense to use [ClusterRole Aggregation](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles). In the following example we are creating custom aggregated ClusterRoles for these three built-in clusterroles, to allow interactions with the GatewayAPI resources: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: tenant:admins:extension + labels: + rbac.authorization.k8s.io/aggregate-to-admin: "true" +rules: + - apiGroups: ["gateway.networking.k8s.io"] + resources: + - gateways + - httproutes + - grpcroutes + - tlsroutes + - tcproutes + - udproutes + - referencegrants + - backendtlspolicies + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: ["gateway.networking.k8s.io"] + resources: + - gateways/status + - httproutes/status + - grpcroutes/status + - tlsroutes/status + - tcproutes/status + - udproutes/status + - referencegrants/status + - backendtlspolicies/status + verbs: ["get"] + - apiGroups: ["gateway.envoyproxy.io"] + resources: + - clienttrafficpolicies + - backendtrafficpolicies + - securitypolicies + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: ["gateway.envoyproxy.io"] + resources: + - clienttrafficpolicies/status + - backendtrafficpolicies/status + - securitypolicies/status + verbs: ["get"] + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: tenant:members:extension + labels: + rbac.authorization.k8s.io/aggregate-to-edit: "true" +rules: + - apiGroups: ["gateway.networking.k8s.io"] + resources: + - gateways + - httproutes + - grpcroutes + - tlsroutes + - tcproutes + - udproutes + - referencegrants + - backendtlspolicies + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: ["gateway.networking.k8s.io"] + resources: + - gateways/status + - httproutes/status + - grpcroutes/status + - tlsroutes/status + - tcproutes/status + - udproutes/status + - referencegrants/status + - backendtlspolicies/status + verbs: ["get"] + - apiGroups: ["gateway.envoyproxy.io"] + resources: + - clienttrafficpolicies + - backendtrafficpolicies + - securitypolicies + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: ["gateway.envoyproxy.io"] + resources: + - clienttrafficpolicies/status + - backendtrafficpolicies/status + - securitypolicies/status + verbs: ["get"] + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: tenant:viewers:extension + labels: + rbac.authorization.k8s.io/aggregate-to-view: "true" +rules: + - apiGroups: ["gateway.networking.k8s.io"] + resources: + - gateways + - httproutes + - grpcroutes + - tlsroutes + - tcproutes + - udproutes + - referencegrants + - backendtlspolicies + verbs: ["get", "list", "watch"] + - apiGroups: ["gateway.networking.k8s.io"] + resources: + - gateways/status + - httproutes/status + - grpcroutes/status + - tlsroutes/status + - tcproutes/status + - udproutes/status + - referencegrants/status + - backendtlspolicies/status + verbs: ["get"] + - apiGroups: ["gateway.envoyproxy.io"] + resources: + - clienttrafficpolicies + - backendtrafficpolicies + - securitypolicies + verbs: ["get", "list", "watch", "create"] + - apiGroups: ["gateway.envoyproxy.io"] + resources: + - clienttrafficpolicies/status + - backendtrafficpolicies/status + - securitypolicies/status + verbs: ["get"] +``` + +#### Custom Resources + +Capsule grants admin permissions to the `TenantOwners` but is only limited to their namespaces. To achieve that, it assigns the ClusterRole [admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) to the `TenantOwner`. This ClusterRole does not permit the installation of custom resources in the namespaces. + +In order to leave the `TenantOwner` to create Custom Resources in their namespaces, the cluster admin defines a proper Cluster Role. For example: + +```yaml +kubectl apply -f - << EOF +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: argoproj-provisioner +rules: +- apiGroups: + - argoproj.io + resources: + - applications + - appprojects + verbs: + - create + - get + - list + - watch + - update + - patch + - delete +EOF +``` + +Bill can assign this role to any namespace in the Alice's `Tenant` by setting it in the `Tenant` manifest: + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + owners: + - name: alice + kind: User + - name: joe + kind: User + rules: + - permissions: + bindings: + - clusterRoleName: 'argoproj-provisioner' + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: User + name: alice + - apiGroup: rbac.authorization.k8s.io + kind: User + name: joe +``` + +With the given specification, Capsule will ensure that all Alice's namespaces will contain a RoleBinding for the specified Cluster Role. For example, in the `solar-production` namespace, Alice will see: + +```yaml +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: capsule-solar-argoproj-provisioner + namespace: solar-production +subjects: + - kind: User + apiGroup: rbac.authorization.k8s.io + name: alice +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: argoproj-provisioner +``` + +With the above example, Capsule is leaving the `TenantOwner` to create namespaced custom resources. + +> Take Note: a `TenantOwner` having the admin scope on its namespaces only, does not have the permission to create Custom Resources Definitions (CRDs) because this requires a cluster admin permission level. Only Bill, the cluster admin, can create CRDs. This is a known limitation of any multi-tenancy environment based on a single shared control plane. + + + +## Promotions + +As an administrator, you can define promotion rules. A promotion rule selects ServiceAccounts within a Tenant based on specified conditions and assigns them predefined ClusterRoles. + +The selected ClusterRoles are then applied across all namespaces belonging to the Tenant, or a selected subset of namespaces, with the corresponding ServiceAccounts configured as subjects. This allows a ServiceAccount in one namespace to automatically receive equivalent permissions in other namespaces of the same Tenant. + +This feature is particularly useful in scenarios involving [Tenant Replications](/docs/replications/#tenantresource), where consistent permissions across namespaces are required. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + ... + rules: + - permissions: + promotions: + # Every promoted ServiceAccount receives this ClusterRole in all Namespaces of Tenant solar. + - clusterRoles: + - "configmap-replicator" + + # Every promoted ServiceAccount with the matching labels receives this ClusterRole. + - clusterRoles: + - "secret-replicator" + selector: + matchLabels: + super: "account" + + - namespaceSelector: + matchExpressions: + - key: env + operator: In + values: ["prod"] + permissions: + promotions: + # Promoted ServiceAccounts receive this ClusterRole only in namespaces matching env=prod. + - clusterRoles: + - "secret-replicator:prod" +``` + +Make sure the `ClusterRoles` exist. Otherwise, the corresponding `Tenant` reports a reconciliation error: + +```shell +conditions: +- lastTransitionTime: "2026-02-16T23:08:59Z" + message: 'cannot sync rolebindings items: rolebindings.rbac.authorization.k8s.io + "tenant-replicator" not found' +``` + +If you run Capsule in [Strict Mode](/docs/operating/setup/installation/#strict-rbac), the controller must be allowed to grant the corresponding permissions to the `ServiceAccount` in all selected `Namespaces`. You can aggregate the same `ClusterRoles` to the controller: + +```yaml +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: configmap-replicator + labels: + projectcapsule.dev/aggregate-to-controller: "true" +rules: + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create", "patch", "watch", "list", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: secret-replicator + labels: + projectcapsule.dev/aggregate-to-controller: "true" +rules: + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "patch", "watch", "list", "delete"] +``` + +As a [Tenant Owner](#ownership), Alice can promote `ServiceAccounts` by labeling them with `projectcapsule.dev/promote=true`. This feature must be enabled in the [CapsuleConfiguration](/docs/operating/setup/configuration/#allowserviceaccountpromotion). If the feature is disabled, admission fails: + +```shell +Error from server (Forbidden): admission webhook "serviceaccounts.projectcapsule.dev" denied the request: service account promotion is disabled. Contact cluster administrators +``` + +When the feature is enabled, the following command succeeds, assuming `alice` is a Tenant Owner of the `solar` Tenant: + +```shell +kubectl label sa gitops-reconcile -n solar-test projectcapsule.dev/promote=true --as alice --as-group projectcapsule.dev +``` + +Verify the promotion in the `Tenant` status: + +```shell +kubectl get tnt solar -o jsonpath='{.status.promotions}' | jq +``` + +Example status: + +```json +[ + { + "clusterRoles": [ + "tenant-replicator" + ], + "kind": "ServiceAccount", + "name": "system:serviceaccount:solar-test:gitops-reconcile", + "targets": [ + "solar-test", + "solar-prod" + ] + } +] +``` + +You can verify that the RoleBinding was distributed to other namespaces of the `solar` Tenant: + +```shell +kubectl get rolebinding -n solar-prod + +NAME ROLE AGE +.. +capsule:managed:7ad688b586eada40 ClusterRole/configmap-replicator 21s +.. +``` + +To revoke the promotion, Alice can remove the label: + +```shell +kubectl label sa gitops-reconcile -n solar-test projectcapsule.dev/promote- --as alice --as-group projectcapsule.dev +``` diff --git a/content/en/docs/tenants/_index.md b/content/en/docs/tenants/_index.md index b51e3cb..ba38164 100644 --- a/content/en/docs/tenants/_index.md +++ b/content/en/docs/tenants/_index.md @@ -6,7 +6,7 @@ description: > --- Capsule is a framework to implement multi-tenant and policy-driven scenarios in Kubernetes. In this tutorial, we'll focus on a hypothetical case covering the main features of the Capsule Operator. This documentation is styled in a tutorial format, and it's designed to be read in sequence. We'll start with the basics and then move to more advanced topics. -**Acme Corp**, our sample organization, is building a Container as a Service platform (CaaS) to serve multiple lines of business, or departments, e.g. Oil, Gas, Solar, Wind, Water. Each department has its team of engineers that are responsible for the development, deployment, and operating of their digital products. We'll work with the following actors: +**Acme Corp**, our sample organization, is building a Container as a Service platform (CaaS) to serve multiple lines of business, or departments, e.g. Solar, Wind, Water. Each department has its team of engineers that are responsible for the development, deployment, and operating of their digital products. We'll work with the following actors: * **Bill**: the cluster administrator from the operations department of Acme Corp. * **Alice**: the project leader in the Solar & Green departments. She is responsible for a team made of different job responsibilities: e.g. developers, administrators, SRE engineers, etc. diff --git a/content/en/docs/tenants/administration.md b/content/en/docs/tenants/administration.md index 5c19850..3901107 100644 --- a/content/en/docs/tenants/administration.md +++ b/content/en/docs/tenants/administration.md @@ -4,7 +4,7 @@ weight: 6 description: > Administrative controls on tenants --- - + ## Cordoning Bill needs to cordon a `Tenant` and its `Namespaces` for several reasons: @@ -14,7 +14,7 @@ Bill needs to cordon a `Tenant` and its `Namespaces` for several reasons: * During incidents or outages * During planned maintenance of a dedicated nodes pool in a BYOD scenario -With the default installation of Capsule all `CREATE`, `UPDATE` and `DELETE` operations performed by **[Capsule Users](/docs/operating/architecture/#capsule-users)** are dropped. Any Updates to Subresources (i.e. `status` updates) and events are allowed to proceed as usual. If you wish to allow specific Operations, you can change the values for the Cordoning Admission via Values (eg. allow `Pod/DELETE` operations): +With the default installation of Capsule all `CREATE`, `UPDATE` and `DELETE` operations performed by **[Capsule Users](/docs/operating/concepts/architecture/#capsule-users)** are dropped. Any Updates to Subresources (i.e. `status` updates) and events are allowed to proceed as usual. If you wish to allow specific Operations, you can change the values for the Cordoning Admission via Values (eg. allow `Pod/DELETE` operations): ```yaml webhooks: diff --git a/content/en/docs/tenants/enforcement.md b/content/en/docs/tenants/enforcement.md index 033e892..cd3becd 100644 --- a/content/en/docs/tenants/enforcement.md +++ b/content/en/docs/tenants/enforcement.md @@ -1,10 +1,14 @@ --- title: Enforcement -weight: 6 +weight: 999 description: > Configure policies and restrictions on tenant-basis --- +{{% alert title="Deprecated" color="warning" %}} +This page covers legacy enforcement configuration. Use [Rules](/docs/rules/) for all current policy and restriction management. +{{% /alert %}} + ## Scheduling diff --git a/content/en/docs/tenants/namespaces.md b/content/en/docs/tenants/namespaces.md index 03ab00c..eb9b473 100644 --- a/content/en/docs/tenants/namespaces.md +++ b/content/en/docs/tenants/namespaces.md @@ -94,7 +94,7 @@ status: If you feel like these rules are too restrictive, you must implement your own custom logic to handle these cases, for example, with Finalizers for `Namespaces`. -**If namespaces are not correctly assigned to tenants, make sure to evaluate your [Capsule Users Configuration](/docs/operating/architecture/#capsule-users).** +**If namespaces are not correctly assigned to tenants, make sure to evaluate your [Capsule Users Configuration](/docs/operating/concepts/architecture/#capsule-users).** ### Multiple Tenants diff --git a/content/en/docs/tenants/permissions.md b/content/en/docs/tenants/permissions.md index e218773..770e571 100644 --- a/content/en/docs/tenants/permissions.md +++ b/content/en/docs/tenants/permissions.md @@ -12,7 +12,7 @@ Administrators are users that have full control over all `Tenants` and their nam ## Ownership -Capsule introduces the principal, that tenants must have owners ([Tenant Owners](/docs/operating/architecture/#tenant-owners)). The owner of a tenant is a user or a group of users that have the right to create, delete, and manage the [tenant's namespaces](/docs/tenants/namespaces) and other tenant resources. However an owner does not have the permissions to manage the tenants they are owner of. This is still done by cluster-administrators. +Capsule introduces the principal, that tenants must have owners ([Tenant Owners](/docs/operating/concepts/architecture/#tenant-owners)). The owner of a tenant is a user or a group of users that have the right to create, delete, and manage the [tenant's namespaces](/docs/tenants/namespaces) and other tenant resources. However an owner does not have the permissions to manage the tenants they are owner of. This is still done by cluster-administrators. At any time you are able to verify which users or groups are owners of a tenant by checking the `owners` field of the Tenant status subresource: @@ -232,7 +232,7 @@ status: #### Aggregation -All subjects defined in `TenantOwner` resources are automatically considered [Capsule Users](/docs/operating/architecture/#capsule-users) and don't need to mentioned further in the CapsuleConfiguration [User Scope](/docs/operating/setup/configuration/#users). If you don't want this behavior, you can disable it by setting `aggregate: false` in the `TenantOwner` spec: +All subjects defined in `TenantOwner` resources are automatically considered [Capsule Users](/docs/operating/concepts/architecture/#capsule-users) and don't need to mentioned further in the CapsuleConfiguration [User Scope](/docs/operating/setup/configuration/#users). If you don't want this behavior, you can disable it by setting `aggregate: false` in the `TenantOwner` spec: ```yaml apiVersion: capsule.clastix.io/v1beta2 @@ -327,7 +327,7 @@ spec: kind: User ``` -However, it's more likely that Bill assigns the ownership of the solar `Tenant` to a group of users instead of a single one, especially if you use [OIDC Authentication](/docs/operating/authentication/#oidc). Bill creates a new group account solar-users in the Acme Corp. identity management system and then he assigns Alice and Bob identities to the solar-users group. +However, it's more likely that Bill assigns the ownership of the solar `Tenant` to a group of users instead of a single one, especially if you use [OIDC Authentication](/docs/operating/setup/authentication/#oidc). Bill creates a new group account solar-users in the Acme Corp. identity management system and then he assigns Alice and Bob identities to the solar-users group. ```yaml apiVersion: capsule.clastix.io/v1beta2 diff --git a/content/en/docs/tenants/quotas.md b/content/en/docs/tenants/quotas.md index 2c40cd8..c7f63ae 100644 --- a/content/en/docs/tenants/quotas.md +++ b/content/en/docs/tenants/quotas.md @@ -1,10 +1,14 @@ --- title: Quotas -weight: 4 +weight: 999 description: > Strategies on granting quotas on tenant-basis --- +{{% alert title="Deprecated" color="warning" %}} +This page covers the legacy quota approach. It will be removed in a future release. Use [Resource Pools](/docs/resource-management/resourcepools/) or [Custom Quotas](/docs/resource-management/customquotas/) instead. +{{% /alert %}} + With help of Capsule, Bill, the cluster admin, can set and enforce resources quota and limits for Alice's `Tenant`. ## Resource Quota @@ -58,7 +62,7 @@ spec: kind: ResourceQuota apiVersion: v1 metadata: - name: capsule-oil-1 + name: capsule-wind-1 namespace: solar-production labels: tenant: solar @@ -287,5 +291,5 @@ spec: ## Node Pools -Bill, the cluster admin, can dedicate a pool of worker nodes to the oil `Tenant`, to isolate the `Tenant` applications from other noisy neighbors. To achieve this approach use [NodeSelectors](/docs/tenants/enforcement#node-selectors). +Bill, the cluster admin, can dedicate a pool of worker nodes to the wind `Tenant`, to isolate the `Tenant` applications from other noisy neighbors. To achieve this approach use [NodeSelectors](/docs/tenants/enforcement#node-selectors). ```yaml diff --git a/content/en/docs/tenants/rules/_index.md b/content/en/docs/tenants/rules/_index.md index c619be1..d075f74 100644 --- a/content/en/docs/tenants/rules/_index.md +++ b/content/en/docs/tenants/rules/_index.md @@ -3,6 +3,9 @@ title: Rules weight: 8 description: > Configure policies and restrictions on a per-Namespace basis with Rules +_build: + render: never + list: never --- Enforcement rules allow Bill, the cluster administrator, to set policies and restrictions on a per-`Tenant` basis. These rules are enforced by Capsule admission webhooks when Alice, the `TenantOwner`, creates or modifies resources in her `Namespaces`. With the rule construct, namespaces within the same tenant can be profiled differently depending on their metadata. diff --git a/content/en/docs/tenants/rules/permissions.md b/content/en/docs/tenants/rules/permissions.md index 2f11903..a89fd64 100644 --- a/content/en/docs/tenants/rules/permissions.md +++ b/content/en/docs/tenants/rules/permissions.md @@ -3,6 +3,9 @@ title: Permissions weight: 5 description: > Configure policies and restrictions on a per-Tenant basis with Rules +_build: + render: never + list: never --- Declare permission distribution rules for the selected namespaces. diff --git a/content/en/docs/tenants/tenant-owner-guide.md b/content/en/docs/tenants/tenant-owner-guide.md new file mode 100644 index 0000000..0a788cf --- /dev/null +++ b/content/en/docs/tenants/tenant-owner-guide.md @@ -0,0 +1,240 @@ +--- +title: Tenant Owner Guide +weight: 1 +description: > + You have been given a Tenant. Here is what you can do with it. +--- + +This guide is for **Tenant Owners**: users who have been assigned ownership of a Capsule Tenant and are responsible for managing namespaces and team access within it. You do not need cluster-admin rights. Everything here is done with your own kubeconfig. + +If you have not set up your kubeconfig yet, ask your cluster administrator or follow the [Proxy Access section of the Quickstart](/docs/quickstart/#proxy-access). + +## See your Tenant + +Start by confirming which Tenant you own: + +```bash +kubectl get tnt +``` + +Example output: + +``` +NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STATUS AGE +solar Active 5 2 True reconciled 3d +``` + +The `NAMESPACE QUOTA` column shows the maximum number of namespaces you are allowed to create. `NAMESPACE COUNT` shows how many you have used. + +To see the full detail of your Tenant, including its owners, active rules, and resource quotas: + +```bash +kubectl get tenant solar -o yaml +``` + +To see who the current owners are: + +```bash +kubectl get tenant solar -o jsonpath='{.status.owners}' | jq +``` + +## Create namespaces + +As a Tenant Owner, you can create namespaces inside your Tenant without needing cluster-admin rights: + +```bash +kubectl create namespace solar-development +``` + +### Prefix enforcement + +If your cluster administrator has enabled `forceTenantPrefix`, all namespaces must start with your Tenant name. Attempting to create `development` directly will be rejected: + +``` +Error from server (Forbidden): admission webhook "namespaces.mutating.projectcapsule.dev" denied the request: The Namespace name must start with 'solar-' when ForceTenantPrefix is enabled in the Tenant. +``` + +Use `solar-development` instead. + +If you belong to multiple Tenants, Capsule cannot infer which one to use from the namespace name alone. Prefix the namespace explicitly with the correct Tenant name. See [Multiple Tenants](/docs/tenants/namespaces/#multiple-tenants) for details. + +### Required labels + +Your cluster administrator may require certain labels to be present when you create a namespace. If a label is missing or has a disallowed value, the webhook returns an error that states exactly which label is expected and what values are permitted: + +``` +Error from server (Forbidden): admission webhook "namespaces.validating.projectcapsule.dev" denied the request: metadata label "environment" is required +``` + +Some labels may be automatically defaulted or managed (controlled entirely by Capsule). Those cannot be changed. Others can be set to any of the listed allowed values. + +### Track your namespaces + +View all namespaces belonging to your Tenant and their status: + +```bash +kubectl get tnt solar -o jsonpath='{.status.namespaces}' +``` + +## Understand your constraints + +Before deploying workloads, check what limits and rules apply. + +### Resource quotas + +Each namespace may have a `ResourceQuota` that limits CPU, memory, and other resources. Check what quota is in place and how much has been used: + +```bash +kubectl get resourcequota -A +``` + +Example output: + +``` +NAMESPACE NAME REQUEST LIMIT +solar-development capsule-solar-0 requests.cpu: 0/7900m, requests.memory: 0/16Gi limits.cpu: 0/7900m, limits.memory: 0/16Gi +solar-production capsule-solar-0 requests.cpu: 100m/8, requests.memory: 128Mi/16Gi limits.cpu: 100m/8, limits.memory: 128Mi/16Gi +``` + +The quota is shared across all your namespaces. Resources consumed in one namespace reduce what is available in the others. + +### Workload restrictions + +Your cluster administrator may have restricted which types of workloads can run in your namespaces. Common examples: + +- **QoS class**: production namespaces may require `Guaranteed` pods (explicit CPU and memory requests/limits). Development namespaces may allow `BestEffort`. +- **Pod Security Standards**: namespaces may enforce a PSS level (`restricted`, `baseline`, or `privileged`) that controls what security contexts are allowed. + +Check the labels on your namespace to understand what is enforced: + +```bash +kubectl get namespace solar-production --show-labels +``` + +Look for `pod-security.kubernetes.io/enforce`. + +### Allowed services + +Service type restrictions may apply. For example, only `ClusterIP` and `ExternalName` may be allowed, with `ExternalName` hostnames restricted to a specific pattern. Attempting to create a `NodePort` or `LoadBalancer` service in such a Tenant will be denied by the webhook. + +## Grant team members access + +You can grant access to your namespaces by creating `RoleBindings`. Capsule does not prevent you from doing standard Kubernetes RBAC within your own namespaces. + +To give a developer `view` access to a specific namespace: + +```bash +kubectl create rolebinding developer-view \ + --clusterrole=view \ + --user=joe \ + -n solar-development +``` + +To give a group `edit` access: + +```bash +kubectl create rolebinding ops-edit \ + --clusterrole=edit \ + --group=solar:operators \ + -n solar-development +``` + +Your cluster administrator may also configure automatic RoleBinding distribution across all your namespaces via [Permission Rules](/docs/rules/permissions/). These are defined in the Tenant spec and applied to every namespace you create. + +## Distribute resources across namespaces + +Use `TenantResource` to automatically replicate a Kubernetes resource into all namespaces of your Tenant. This is useful for Secrets, ConfigMaps, or any resource that should be present everywhere. + +You need RBAC permission to create `TenantResource` objects. Ask your cluster administrator to apply the [prerequisite ClusterRole](/docs/replications/tenant/#prerequisites) if it is not already in place. + +Example: distribute an image pull Secret to every namespace: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: TenantResource +metadata: + name: registry-credentials + namespace: solar-system +spec: + resyncPeriod: 60s + resources: + - rawItems: + - apiVersion: v1 + kind: Secret + metadata: + name: registry-credentials + type: kubernetes.io/dockerconfigjson + data: + .dockerconfigjson: +``` + +See [TenantResources](/docs/replications/tenant/) for full documentation. + +### Service Account promotion + +As a Tenant Owner you can promote a ServiceAccount within your Tenant so it automatically receives the ClusterRoles your cluster administrator has defined in the Tenant's promotion rules. This gives the ServiceAccount consistent permissions across all Tenant namespaces. A common use case is referencing the promoted ServiceAccount in `spec.serviceAccount.name` of a `TenantResource`, so replication operations run under a scoped identity rather than the Capsule controller. + +Label the ServiceAccount to promote it: + +```bash +kubectl label sa gitops-reconciler -n solar-development projectcapsule.dev/promote=true +``` + +This feature must be enabled by your cluster administrator. If it is not, the webhook will reject the label and return: + +``` +Error from server (Forbidden): admission webhook "serviceaccounts.projectcapsule.dev" denied the request: service account promotion is disabled. Contact cluster administrators +``` + +Once promoted, verify that the expected RoleBindings have been distributed across your namespaces: + +```bash +kubectl get tnt solar -o jsonpath='{.status.promotions}' | jq +``` + +To revoke the promotion, remove the label: + +```bash +kubectl label sa gitops-reconciler -n solar-development projectcapsule.dev/promote- +``` + +See [Promotions](/docs/rules/permissions/#promotions) for full details. + +## The Proxy and kubectl + +When using the [Capsule Proxy](/docs/proxy/), your `kubectl` commands are filtered to show only resources that belong to your Tenant. + +```bash +# Lists only namespaces you own, not all namespaces in the cluster +kubectl get namespaces -A + +# Lists events across all your namespaces +kubectl get events -A +``` + +Without the Proxy, `kubectl get namespaces -A` returns `Forbidden`. If you are hitting this, confirm with your cluster administrator that the Proxy is installed and that your kubeconfig points to the Proxy endpoint: + +```bash +kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' +``` + +## View events and troubleshoot rejections + +Capsule emits events when it blocks or modifies a request. These are visible in the `default` namespace: + +```bash +kubectl get events -A +``` + +Example events: + +``` +NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE +default 2m Warning ForbiddenMetadata namespace/solar-development metadata label "privileged" at metadata.labels["pod-security.kubernetes.io/enforce"] is not allowed +default 5m Normal TenantAssigned namespace/solar-production namespace has been assigned to the desired tenant solar +default 10m Warning Overprovisioned namespace/solar-test namespace cannot be attached, quota exceeded for the elected tenant +``` + +Each admission webhook error message names the exact label, value, or rule that caused the rejection. If the error message references a `managed` label or a rule you cannot change, contact your cluster administrator to adjust the Tenant configuration. + +For more common issues and their solutions, see the [Troubleshooting guide](/docs/operating/operations/troubleshoting/). diff --git a/content/en/docs/whats-new.md b/content/en/docs/whats-new.md index d25ca93..41a0753 100644 --- a/content/en/docs/whats-new.md +++ b/content/en/docs/whats-new.md @@ -24,11 +24,11 @@ weight: 1 * Introducing new rule approach for tenant enforcement [Read More](/docs/tenants/rules/). * Added `RequiredMetadata` for `Namespaces` created in a `Tenant` [Read More](/docs/tenants/metadata/#requiredmetadata). * [Additional Metadata](/docs/tenants/metadata/#additionalmetadata) is now validated at admission. -* Introducing new OCI Registry enforcement [Read More](/docs/tenants/rules/enforcement/#oci-registries) -* Added rule-based promotions for `ServiceAccounts` in `Tenants` [Read More](/docs/tenants/rules/permissions/). +* Introducing new OCI Registry enforcement [Read More](/docs/rules/enforcement/workloads#oci-registries) +* Added rule-based promotions for `ServiceAccounts` in `Tenants` [Read More](/docs/tenants/rules/permissions#promotions). * Added Implicit Assignment of `TenantOwner` [Read More](/docs/tenants/permissions/#implicit-tenant-assignment). * Added Aggregation of `TenantOwner` [Read More](/docs/tenants/permissions/#aggregation). -* Introducing `data` field for `Tenants` [Read More](/docs/operating/templating/#data). +* Introducing `data` field for `Tenants` [Read More](/docs/operating/concepts/templating/#data). * Added new label `projectcapsule.dev/tenant` which is added for all namespaced resources belonging to a `Tenant` [Read More](/docs/tenants/metadata/#managed). * Resources labeled with `projectcapsule.dev/managed-by=controller` can only be created, updated or deleted by the Capsule controller and [administrators](/docs/tenants/permissions/#administrators), and are rejected for all other operations. This prevents deletion of managed resources by users, which are not identified as capsule users (current behavior). * Added configuration options for managed RBAC [Read More](/docs/operating/setup/configuration/#rbac) @@ -42,8 +42,8 @@ weight: 1 ### Bugfix Releases -* Added rule-based QOS enforcement [Read More](/docs/tenants/rules/enforcement/#qos-classes). -* Added rule-based Scheduler enforcement [Read More](/docs/tenants/rules/enforcement/#scheduler-names). +* Added rule-based QOS enforcement [Read More](/docs/rules/enforcement/workloads#qos-classes). +* Added rule-based Scheduler enforcement [Read More](/docs/rules/enforcement/workloads#scheduler-names). ## Fixes 🐛 @@ -71,7 +71,6 @@ Newly added documentation to integrate Capsule with other applications: * [CoreDNS Plugin](https://github.com/CorentinPtrl/capsule_coredns) (Community Contribution) * [Argo CD](/ecosystem/integrations/argocd/) -* [Flux CD](/docs/guides/use-fluxcd.md) ## Project Updates 💫 diff --git a/content/en/ecosystem/integrations/dashboard.md b/content/en/ecosystem/integrations/dashboard.md index 96d4b16..6aa4678 100644 --- a/content/en/ecosystem/integrations/dashboard.md +++ b/content/en/ecosystem/integrations/dashboard.md @@ -17,7 +17,7 @@ This guide describes how to integrate the [Kubernetes Dashboard](https://kuberne ## OIDC Authentication -Your cluster must also be configured to use [OIDC Authentication](/docs/operating/authentication#oidc) for seemless Kubernetes RBAC integration. In a such scenario, you should have in the kube-apiserver.yaml manifest the following content: +Your cluster must also be configured to use [OIDC Authentication](/docs/operating/setup/authentication#oidc) for seemless Kubernetes RBAC integration. In a such scenario, you should have in the kube-apiserver.yaml manifest the following content: ```yaml spec: @@ -78,7 +78,7 @@ helm install oauth2-proxy oauth2-proxy/oauth2-proxy -n ${KUBERNETES_DASHBOARD_NA ## Configuring Keycloak -The Kubernetes cluster must be configured with a valid OIDC provider: for our guide, we're giving for granted that Keycloak is used, if you need more info please follow the [OIDC Authentication](/docs/operating/authentication#oidc) section. +The Kubernetes cluster must be configured with a valid OIDC provider: for our guide, we're giving for granted that Keycloak is used, if you need more info please follow the [OIDC Authentication](/docs/operating/setup/authentication#oidc) section. In a such scenario, you should have in the `kube-apiserver.yaml` manifest the following content: ```yaml diff --git a/content/en/ecosystem/integrations/headlamp.md b/content/en/ecosystem/integrations/headlamp.md index a54fb9d..5eb45b5 100644 --- a/content/en/ecosystem/integrations/headlamp.md +++ b/content/en/ecosystem/integrations/headlamp.md @@ -35,7 +35,7 @@ jwt: prefix: "" ``` -[Read More](/docs/operating/authentication/#configuring-kubernetes-api-server) +[Read More](/docs/operating/setup/authentication/#configuring-kubernetes-api-server) ## Integration diff --git a/content/en/ecosystem/integrations/kyverno.md b/content/en/ecosystem/integrations/kyverno.md index 04bda40..55d2791 100644 --- a/content/en/ecosystem/integrations/kyverno.md +++ b/content/en/ecosystem/integrations/kyverno.md @@ -27,7 +27,7 @@ admissionController: Not all relevant settings are covered by Capsule. We recommend to use Kyverno to enforce additional policies, as their policy implementation is of a very high standard. Here are some policies you might want to consider in multi-tenant environments: -[Moved to new page](/docs/operating/architecture.md) +[Moved to new page](/docs/operating/concepts/architecture/) ## References diff --git a/content/en/ecosystem/integrations/teleport.md b/content/en/ecosystem/integrations/teleport.md index b859f13..9f33a8f 100644 --- a/content/en/ecosystem/integrations/teleport.md +++ b/content/en/ecosystem/integrations/teleport.md @@ -153,7 +153,7 @@ capsule-values.yaml: ```yaml manager: options: - capsuleUserGroups: ["tenant-oil"] + capsuleUserGroups: ["tenant-wind"] forceTenantPrefix: true ``` @@ -163,7 +163,7 @@ tenant.yaml: apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: - name: oil + name: wind spec: owners: - name: alice @@ -174,7 +174,7 @@ Install `capsule` with `tenant-oil` as a capsule user group via helm chart: - `helm repo add projectcapsule https://projectcapsule.github.io/charts` - `helm upgrade --install capsule -n capsule-system --create-namespace projectcapsule/capsule --version 0.10.9 -f capsule-values.yaml` -- Create tenant named `oil`: `kubectl apply -f tenant.yaml` +- Create tenant named `wind`: `kubectl apply -f tenant.yaml` ### Capsule Proxy @@ -185,7 +185,7 @@ Install default `capsule-proxy` via helm chart: ### Teleport -Create teleport role for kubernetes cluster access which adds `tenant-oil` group to user auth token. +Create teleport role for kubernetes cluster access which adds `tenant-wind` group to user auth token. - `docker exec -it teleport bash` @@ -200,7 +200,7 @@ Create teleport role for kubernetes cluster access which adds `tenant-oil` group spec: allow: kubernetes_groups: - - tenant-oil + - tenant-wind kubernetes_labels: capsule: "true" kubernetes_resources: diff --git a/content/en/ecosystem/integrations/velero.md b/content/en/ecosystem/integrations/velero.md index b622efc..1cff749 100644 --- a/content/en/ecosystem/integrations/velero.md +++ b/content/en/ecosystem/integrations/velero.md @@ -1,6 +1,6 @@ --- title: Velero -link: "/docs/operating/backup-restore/" +link: "/docs/operating/operations/backup-restore/" logo: https://velero.io/docs/main/img/velero.png display: true integration: true