Skip to content
Draft
7 changes: 7 additions & 0 deletions content/en/docs/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,10 @@
title: Documentation
weight: 20
---

## Where to start

| I am a... | Start here |
|---|---|
| **Cluster Administrator** | [Installation](/docs/operating/setup/installation/) then [Quickstart](/docs/quickstart/) |
| **Tenant Owner** | [Quickstart - Tenant Owners section](/docs/quickstart/#tenant-owners) |
2 changes: 1 addition & 1 deletion content/en/docs/operating/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The answer to this question may be influenced by the following aspects:
* _You’ll need robust monitoring that enables Tenant Owners to clearly understand and manage what’s happening inside their own tenant._

* **Are your customers technically capable of working directly with the Kubernetes API?**
* _If not, you may need to build a more user-friendly platform with better UXfor example, a multi-tenant ArgoCD setup, or UI layers like Headlamp._
* _If not, you may need to build a more user-friendly platform with better UX, for example a multi-tenant ArgoCD setup, or UI layers like Headlamp._

## Personas

Expand Down
6 changes: 3 additions & 3 deletions content/en/docs/operating/authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,9 +220,9 @@ $ kubectl config set-context alice-oidc@mycluster \
As user alice, you should be able to use kubectl to create some namespaces:

```shell
$ kubectl --context alice-oidc@mycluster create namespace oil-production
$ kubectl --context alice-oidc@mycluster create namespace oil-development
$ kubectl --context alice-oidc@mycluster create namespace gas-marketing
$ kubectl --context alice-oidc@mycluster create namespace wind-production
$ kubectl --context alice-oidc@mycluster create namespace wind-development
$ kubectl --context alice-oidc@mycluster create namespace water-marketing
```

**Warning**: once your `ID_TOKEN` expires, the kubectl OIDC Authenticator will attempt to refresh automatically your `ID_TOKEN` using the `REFRESH_TOKEN`. In case the OIDC uses a self signed CA certificate, make sure to specify it with the idp-certificate-authority option in your kubeconfig file, otherwise you'll not able to refresh the tokens.
20 changes: 10 additions & 10 deletions content/en/docs/operating/backup-restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The first requirement aims to backup all the resources stored into etcd database

The main limitation of Velero is the multi tenancy. Currently, Velero does not support multi tenancy meaning it can be only used from admin users and so it cannot provided "as a service" to the users. This means that the cluster admin needs to take care of users' backup.

Assuming you have multiple tenants managed by Capsule, for example oil and gas, as cluster admin, you can to take care of scheduling backups for:
Assuming you have multiple tenants managed by Capsule, for example wind and water, as cluster admin, you can to take care of scheduling backups for:

* Tenant cluster resources
* Namespaces belonging to each tenant
Expand All @@ -27,10 +27,10 @@ Create a backup of the tenant `solar`. It consists in two different backups:
* backup of the tenant resource
* backup of all the resources belonging to the tenant

To backup the oil tenant selectively, label the tenant as:
To backup the wind tenant selectively, label the tenant as:

```bash
kubectl label tenant oil capsule.clastix.io/tenant=solar
kubectl label tenant wind capsule.clastix.io/tenant=solar
```

and create the backup
Expand Down Expand Up @@ -65,7 +65,7 @@ spec:
ttl: 720h0m0s
```

Create a backup of all the resources belonging to the oil tenant namespaces:
Create a backup of all the resources belonging to the wind tenant namespaces:

```bash
velero create backup solar-namespaces \
Expand Down Expand Up @@ -109,23 +109,23 @@ Using Velero to restore a Capsule tenant can lead to an incomplete recovery of t
```bash
kubectl get tnt
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
gas active 9 5 {"pool":"gas"} 34m
oil active 9 8 {"pool":"oil"} 33m
water active 9 5 {"pool":"water"} 34m
wind active 9 8 {"pool":"wind"} 33m
solar active 9 0 # <<< {"pool":"solar"} 54m
```

To avoid this problem you can use the script [velero-restore.sh](https://github.com/projectcapsule/capsule/blob/main/hack/velero-restore.sh) located under the hack/ folder:

```bash
./velero-restore.sh --kubeconfing /path/to/your/kubeconfig --tenant "oil" restore
./velero-restore.sh --kubeconfing /path/to/your/kubeconfig --tenant "wind" restore
```

Running this command, we are going to patch the tenant's namespaces manifests that are actually ownerReferences-less. Once the command has finished its run, you got the tenant back.

```bash
kubectl get tnt
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
gas active 9 5 {"pool":"gas"} 44m
solar active 9 8 {"pool":"oil"} 43m
oil active 9 3 # <<< {"pool":"solar"} 12s
water active 9 5 {"pool":"water"} 44m
solar active 9 8 {"pool":"wind"} 43m
wind active 9 3 # <<< {"pool":"solar"} 12s
```
8 changes: 4 additions & 4 deletions content/en/docs/operating/best-practices/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ This is general advice you should consider before making Kubernetes Distribution
### Authentication

User authentication for the platform should be handled via a central OIDC-compatible identity provider system (e.g., Keycloak, Azure AD, Okta, or any other OIDC-compliant provider).
The rationale is that other central platform componentssuch as ArgoCD, Grafana, Headlamp, or Harbor should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions.
The rationale is that other central platform components, such as ArgoCD, Grafana, Headlamp, or Harbor, should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions.

[Capsule relies on native Kubernetes RBAC](/docs/operating/authentication/), so it's important to consider how the Kubernetes API handles user authentication.
[Capsule relies on native Kubernetes RBAC](/docs/operating/setup/authentication/), so it's important to consider how the Kubernetes API handles user authentication.

### OCI Pull-Cache

By default, Kubernetes clusters pull images directly from upstream registries like `docker.io`, `quay.io`, `ghcr.io`, or `gcr.io`. In production environments, this can lead to issues especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled).
By default, Kubernetes clusters pull images directly from upstream registries like `docker.io`, `quay.io`, `ghcr.io`, or `gcr.io`. In production environments, this can lead to issues, especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled).

To ensure availability, performance, and control over container images, it's essential to provide an on-premise OCI mirror.
This mirror should be configured via the CRI (Container Runtime Interface) by defining it as a mirror endpoint in registries.conf for default registries (e.g., `docker.io`).
Expand All @@ -25,7 +25,7 @@ This way, all nodes automatically benefit from caching without requiring develop
### Secrets Management

In more complex environments with multiple clusters and applications, managing secrets manually via YAML or Helm is no longer practical.
Instead, a centralized secrets management system should be established such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project [OpenBao](https://openbao.org/) (formerly the Vault community fork).
Instead, a centralized secrets management system should be established, such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project [OpenBao](https://openbao.org/) (formerly the Vault community fork).

To integrate these external secret stores with Kubernetes, the [External Secrets Operator (ESO)](https://external-secrets.io/latest/) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing.

Expand Down
2 changes: 1 addition & 1 deletion content/en/docs/operating/best-practices/workloads.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ metadata:
and the regular Pod Security Admission Controller does the magic:

```yaml
kubectl --kubeconfig alice-oil.kubeconfig apply -f - << EOF
kubectl --kubeconfig alice-wind.kubeconfig apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
Expand Down
5 changes: 5 additions & 0 deletions content/en/docs/operating/concepts/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
title: Architecture & Concepts
weight: 2
description: Key design decisions, personas, templating, and metadata reference for Capsule
---
139 changes: 139 additions & 0 deletions content/en/docs/operating/concepts/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
---
title: Architecture
weight: 2
description: Architecture references and considerations
---

## Key Decisions

Introducing a new separation of duties can lead to a significant paradigm shift. This has technical implications and may also impact your organizational structure. Therefore, when designing a multi-tenant platform pattern, carefully consider the following aspects. As **Cluster Administrator**, ask yourself:

* 🔑 **How much ownership can be delegated to Tenant Owners (Platform Users)?**

The answer to this question may be influenced by the following aspects:

* **Are the Cluster Administrators willing to grant permissions to Tenant Owners?**
* _You might have a problem with know-how and probably your organisation is not yet pushing Kubernetes itself enough as a key strategic platform. The key here is enabling Platform Users through good UX and know-how transfers_

* **Who is responsible for the deployed workloads within the Tenants?**
* _If Platform Administrators are still handling this, a true “shift left” has not yet been achieved._

* **Who gets paged during a production outage within a Tenant’s application?**
* _You’ll need robust monitoring that enables Tenant Owners to clearly understand and manage what’s happening inside their own tenant._

* **Are your customers technically capable of working directly with the Kubernetes API?**
* _If not, you may need to build a more user-friendly platform with better UX, for example a multi-tenant ArgoCD setup, or UI layers like Headlamp._

## Personas

In Capsule, we introduce a new persona called the `Tenant Owner`. The goal is to enable Cluster Administrators to delegate tenant management responsibilities to Tenant Owners. Here’s how it works:

### Capsule Administrators

[Configure Capsule Administrators](/docs/operating/setup/configuration/#administrators). The ClusterRoles assigned to Administrators can be configured in the [CapsuleConfiguration](/docs/operating/setup/configuration/#rbac) as well.

They are promoted to [Tenant-Owners](#tenant-owners) for all available tenants. Effectively granting them the ability to manage all namespaces within the cluster, across all tenants.

**Note**: Granting Capsule Administrator rights should be done with caution, as it provides extensive control over the cluster's multi-tenant environment. **When granting Capsule Administrator rights, the entity gets the privileges to create any namespace (also not part of capsule tenants) and the privileges to delete any tenant namespaces.**

Capsule Administrators can:

- Create and manage [namespaces via labels in any tenant](/docs/tenants/namespaces/#label).
- Create namespaces outside of tenants.
- Delete namespaces in any tenant.

Administrators come in handy in bootstrap scenarios or GitOps scenarios where certain users/serviceaccounts need to be able to manage namespaces for all tenants.

### Capsule Users

[Configure Capsule Users](/docs/operating/setup/configuration/#users)

Any entity which needs to interact with tenants and their namespaces must be defined as a **Capsule User**. This is where the flexibility of Capsule comes into play. You can define users or groups as Capsule Users, allowing them to create and manage namespaces within any tenant they have access to. If they are not defined as Capsule Users, any interactions will be ignored by Capsule. Often a best practice is to define a single group which identifies all your tenant users. This way you can have one generic group for all your users and then use additional groups to separate responsibilities (e.g. administrators vs normal users).

**Only one entry is needed to identify a Capsule User. This is only important for Namespace Admission.**.

![Capsule Users Admission](/images/content/capsule-users-admission.drawio.png)

You can always verify the effective Capsule Users by checking the Configuration Status. As this is variable with [Tenant Owners](#tenant-owners), the status will always show the effective users:

```bash
kubectl get capsuleconfiguration default -o jsonpath='{.status.users}' | jq

[
{
"kind": "Group",
"name": "oidc:kubernetes:admin"
},
{
"kind": "Group",
"name": "projectcapsule.dev"
},
{
"kind": "User",
"name": "test-user"
}
]
```

### Tenant Owners

[Configure Tenant Owners](/docs/tenants/permissions/#ownership)

**Every Tenant Owner must be a [Capsule User](#capsule-users). By using the [TenantOwner CRD](/docs/tenants/permissions/#tenant-owners) this is automatically handeled.**


They manage the namespaces within their tenants and perform administrative tasks confined to their tenant boundaries. This delegation allows teams to operate more autonomously while still adhering to organizational policies. Tenant Owners can be used to shift reposnsability of one tenant towards this user group. promoting them to the SPOC of all namespaces within the tenant.

Tenant Owners can:

- Create and manage namespaces within their tenant.
- Delete namespaces within their tenant.

Capsule provides robust tools to strictly enforce tenant boundaries, ensuring that each tenant operates within its defined limits. This separation of duties promotes both security and efficient resource management.

## Layouts

Let's discuss different Tenant Layouts which could be used . These are just approaches we have seen, however you might also find a combination of these which fits your use-case.

### Tenant As A Service

With this approach you essentially just provide your Customers with the Tenant on your cluster. The rest is their responsibility. This concludes to a shared responsibility model. This can be achieved when also the Tenant Owners are responsible for everything they are provisiong within their Tenant's namespaces.

![Resourcepool Dashboard](/images/content/architecture/layout-taas.drawio.png)

## Scheduling

Workload distribution across your compute infrastructure can be approached in various ways, depending on your specific priorities. Regardless of the use case, it's essential to preserve maximum flexibility for your platform administrators. This means ensuring that:

- Nodes can be drained or deleted at any time.
- Cluster updates can be performed at any time.
- The number of worker nodes can be scaled up or down as needed.

If your cluster architecture prevents any of these capabilities, or if certain applications block the enforcement of these policies, you should reconsider your approach.

### Dedicated

Strong tenant isolation, ensuring that any noisy neighbor effects remain confined within individual tenants (tenant responsibility). This approach may involve higher administrative overhead and costs compared to shared compute. It also provides enhanced security by dedicating nodes to a single customer/application. It is recommended, at a minimum, to separate the cluster’s operator workload from customer workloads.

![Dedicated Nodepool](/images/content/scheduling-dedicated.drawio.png)

### Shared

With this approach you share the nodes amongst all Tenants, therefore giving you more potential for optimizing resources on a node level. It's a common pattern to separate the controllers needed to power your Distribution (operators) from the actual workload. This ensures smooth operations for the cluster

**Overview**:

- ✅ Designed for cost efficiency .
- ✅ Suitable for applications that typically experience low resource fluctuations and run with multiple replicas.
- ❌ Not ideal for applications that are not cloud-native ready, as they may adversely affect the operation of other applications or the maintenance of node pools.
- ❌ Not ideal if strong isolation is required

![Shared Nodepool](/images/content/scheduling-shared.drawio.png)


We provide the concept of [ResourcePools](/docs/resource-management/resourcepools/) or [CustomQuotas](/docs/resource-management/customquotas/) to manage resources cross namespaces. There's some further aspects you must think about with shared approaches:

* [PriorityClasses](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)
* [ResourceQuotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/)
* [LimitRanges](https://kubernetes.io/docs/concepts/policy/limit-range/)
* [Descheduling/Rebalancing](https://github.com/kubernetes-sigs/descheduler)
Loading