feat: CommunityToolkit.Aspire.Hosting.K3s — k3s Kubernetes cluster hosting integration

[edmondshtogu]

Closes #1321

Overview of changes

Adds CommunityToolkit.Aspire.Hosting.K3s, a hosting integration that runs a lightweight Kubernetes cluster as an Aspire resource tree. Developers can declare a local Kubernetes cluster in Program.cs — with Helm charts, manifests, and exposed service endpoints — the same way they add Redis or PostgreSQL. No external tooling beyond Docker is required.

What's included

New package: src/CommunityToolkit.Aspire.Hosting.K3s/

File	Responsibility
K3sClusterResource.cs	ContainerResource — k3s server; holds kubeconfig directory path and image settings
K3sClusterOptions.cs	Configuration (pod/service CIDR, disabled components, k3s image tag, helm/kubectl image overrides)
K3sBuilderExtensions.cs	AddK3sCluster, WithDataVolume, WithLifetime, WithReference(cluster), WithK3sVersion, …
K3sReadinessHealthCheck.cs	File-based health check — polls cluster/kubeconfig.yaml, writes local/ + container/ variants, probes nodes via KubernetesClient
HelmReleaseResource.cs	ContainerResource — runs alpine/helm; child of cluster; exits 0 on success
K3sBuilderExtensions.Helm.cs	AddHelmRelease, WithHelmValue, WithHelmValuesFile
K8sManifestResource.cs	ContainerResource — runs alpine/k8s; child of cluster; exits 0 on success
K3sBuilderExtensions.Manifest.cs	AddK8sManifest with auto-detected Kustomize support
K3sServiceEndpointResource.cs	Resource — in-process port-forward; M1 passive health via IsReady flag
K3sBuilderExtensions.ServiceEndpoint.cs	AddServiceEndpoint, WithReference(endpoint)
K3sInProcessPortForwarder.cs	KubernetesClient WebSocket TCP forwarder; binds 0.0.0.0:{port}
K3sAgentResource.cs	Worker node support (K3sClusterOptions.AgentCount)
HelmContainerImageTags.cs / KubectlContainerImageTags.cs	Pinned image defaults; overridable via K3sClusterOptions

New Tests & Examples:

• Unit Tests: tests/CommunityToolkit.Aspire.Hosting.K3s.Tests/ — 85 unit tests covering resource registration, script generation, kubeconfig variants, Kustomize detection, values file injection, and public API null guards.
• Integration Tests: tests/CommunityToolkit.Aspire.Hosting.K3s.IntegrationTests/ — Linux-only (ubuntu-latest CI), bitnami/nginx as the test chart.
• Examples: examples/k3s/CommunityToolkit.Aspire.Hosting.K3s.AppHost/

Key design decisions

• Health check via bind-mount, not docker exec. k3s writes its kubeconfig to K3S_KUBECONFIG_OUTPUT=/tmp/k3s-kubeconfig/kubeconfig.yaml, bind-mounted to AppHostDirectory/.k3s/{name}/cluster/ on the host. The health check polls File.Exists, rewrites server URLs into local/ and container/ variants, then confirms node readiness via IKubernetes.CoreV1.ListNodeAsync. No shell access, no docker exec, works with any container runtime.
• Helm and kubectl run as containers. HelmReleaseResource and K8sManifestResource extend ContainerResource and are shown as children of the cluster in the Aspire dashboard. The install/apply script is injected via WithContainerFiles. They cannot use WaitFor(cluster) (Aspire forbids a child waiting for its parent), so their scripts poll for /root/.kube/kubeconfig.yaml — which only appears after the cluster health check passes — before proceeding. Consumers use WaitForCompletion(helmRelease) since these are run-to-completion containers.
• Kustomize auto-detected. AddK8sManifest inspects the host path at configuration time: if kustomization.yaml is present, it bind-mounts the directory (preserving relative base references) and uses kubectl apply -k; otherwise it uses Aspire's built-in WithContainerFiles(destinationPath, hostPath) to copy only the YAML files and applies with kubectl apply -f --server-side. The script auto-detects the mode at runtime.
• Service exposure without NodePort. K3sServiceEndpointResource starts an in-process KubernetesClient WebSocket port-forward bound to 0.0.0.0:{hostPort}. Host resources receive services__{name}url=http(s)://localhost:{port}; container resources receive services{name}__url=http(s)://host.docker.internal:{port} with --add-host=host.docker.internal:host-gateway injected automatically.
• Image overrides via K3sClusterOptions. The helm and kubectl container images are configurable via HelmImage/HelmTag/HelmRegistry and KubectlImage/KubectlTag/KubectlRegistry on K3sClusterOptions. Defaults: docker.io/alpine/helm:3.17.3 and docker.io/alpine/k8s:1.32.3.

Usage

var cluster = builder.AddK3sCluster("k8s")
    .WithDataVolume()
    .WithK3sVersion("v1.32.3-k3s1");

var widgetCrd = cluster.AddK8sManifest("widget-crd", "./k8s/crds/");

var argocd = cluster.AddHelmRelease("argocd", "argo-cd",
    repo: "https://argoproj.github.io/argo-helm",
    version: "7.8.0",
    @namespace: "argocd")
    .WithHelmValuesFile("./deploy/argocd-values.yaml");

var ui = cluster.AddServiceEndpoint("argocd-ui", "argocd-server", 443, "argocd")
    .WaitForCompletion(argocd);

builder.AddProject<Projects.WidgetOperator>("operator")
    .WaitForCompletion(widgetCrd)
    .WithReference(cluster);

builder.AddProject<Projects.Api>("api")
    .WaitFor(ui)
    .WithReference(ui);

PR Checklist

• Created a feature/dev branch in your fork (vs. submitting directly from a commit on main)
• Based off latest main branch of toolkit
• PR doesn't include merge commits (always rebase on top of our main, if needed)
• New integration
  • Docs are written
  • Added description of major feature to project description for NuGet package (4000 total character limit, so don't push entire description over that)
• Tests for the changes have been added (for bug fixes / features) (if applicable)
• Contains NO breaking changes
• Every new API (including internal ones) has full XML docs
• Code follows all style conventions

Other information

Security Assumptions
This change assumes local development only. k3s runs in privileged mode (required by k3s/containerd). The bind-mounted kubeconfig directory is readable only by the AppHost user. KubernetesClientConfiguration uses the embedded CA cert from the kubeconfig; no DangerousAcceptAnyServerCertificateValidator is used.
Remaining Follow-up Work
While this PR represents the core ready-to-ship features, follow-ups will be needed for:

• Expanded integration test coverage.
• Polyglot/ATS completeness.
• Publish-time diagnostics for cluster resources.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 1322

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 1322"

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[edmondshtogu]

@dotnet-policy-service agree