From 29ee0e3bfb503b588699e13008523759c35bc2ec Mon Sep 17 00:00:00 2001 From: Ramkumar Chinchani Date: Wed, 22 Jul 2026 14:59:25 -0700 Subject: [PATCH] [PATCH 01/08] docs(first-party): AFD + WAF + Private Link GLB design (SFI-NS253) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Part 01 of an 8-patch split of #373. Docs-only patch; no code changes. Contents: - docs/first-party/001-afd-global-load-balancing.md — design proposal - docs/first-party/002-afd-implementation-plan.md — implementation plan - docs/first-party/003-pre-implementation-checklist.md — SFI-NS253 checklist - docs/first-party/README.md — first-party proposals index + competitive context - .github/workflows/markdown.links.config.json — ignore SSO-gated eng.ms links - .github/.copilot/breadcrumbs/** — design breadcrumbs and ATM baselines Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a8fabc68-e113-49ef-b82d-d99a8502da11 --- ...18-1717-frontdoor-phase2-custom-domains.md | 325 +++++ ...6-07-20-1108-afd-export-mode-mcs-parity.md | 474 +++++++ .../.copilot/breadcrumbs/baselines/README.md | 29 + .../baselines/hub-net-atm-default.yaml | 239 ++++ .../baselines/hub-net-atm-enabled.yaml | 311 +++++ .github/workflows/markdown.links.config.json | 3 + .../001-afd-global-load-balancing.md | 910 +++++++++++++ .../002-afd-implementation-plan.md | 1182 +++++++++++++++++ .../003-pre-implementation-checklist.md | 317 +++++ docs/first-party/004-poc-runbook.md | 114 ++ docs/first-party/README.md | 42 + 11 files changed, 3946 insertions(+) create mode 100644 .github/.copilot/breadcrumbs/2026-07-18-1717-frontdoor-phase2-custom-domains.md create mode 100644 .github/.copilot/breadcrumbs/2026-07-20-1108-afd-export-mode-mcs-parity.md create mode 100644 .github/.copilot/breadcrumbs/baselines/README.md create mode 100644 .github/.copilot/breadcrumbs/baselines/hub-net-atm-default.yaml create mode 100644 .github/.copilot/breadcrumbs/baselines/hub-net-atm-enabled.yaml create mode 100644 docs/first-party/001-afd-global-load-balancing.md create mode 100644 docs/first-party/002-afd-implementation-plan.md create mode 100644 docs/first-party/003-pre-implementation-checklist.md create mode 100644 docs/first-party/004-poc-runbook.md create mode 100644 docs/first-party/README.md diff --git a/.github/.copilot/breadcrumbs/2026-07-18-1717-frontdoor-phase2-custom-domains.md b/.github/.copilot/breadcrumbs/2026-07-18-1717-frontdoor-phase2-custom-domains.md new file mode 100644 index 00000000..77aae007 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-07-18-1717-frontdoor-phase2-custom-domains.md @@ -0,0 +1,325 @@ +# Front Door Phase 2 — Profile + Custom Domains + +## Requirements + +- Reconcile Azure Front Door (AFD) profiles from a `FrontDoorProfile` CRD. +- Reconcile AFD custom domains from a **separate** `FrontDoorCustomDomain` CRD. +- Support DNS-based domain validation and surface the DNS TXT token to users. +- Support both **Managed TLS** and **BYOC (Key Vault reference)** TLS modes. +- Auth to Azure via **Workload Identity** (federated token). +- Namespaced CRDs; custom domain must reference a profile in the **same namespace**. +- Emit both `.status` and Kubernetes `Event`s for the one-time DNS validation token. + +## Additional comments from user + +- User input: "is custom domain mandatory in production?" → concluded not mandatory technically, usually mandatory commercially. +- User input: "ok, then just add support for it immediately" → custom domains move into Phase 2. +- User input: "do custom domains change often?" → agreed lifecycle/ownership (not frequency) drives the API split. +- User input: "ok" → confirmed separate CRD approach. + +## Plan + +> **Scope note (POC):** This work ships as a proof-of-concept. The full Phase 2 plan below stays as the north star, but the POC deliberately trims scope to prove the reconciliation model, API shape (D2), and Workload Identity auth path (D6) end-to-end. Deferred items are additive and do not invalidate any recorded decision. +> +> **POC includes:** +> - 2.1 API types (both CRDs, minimal validation) +> - 2.2 Azure client factory with Workload Identity +> - 2.3 `FrontDoorProfile` controller — happy path only +> - 2.4 `FrontDoorCustomDomain` controller — **Managed TLS only**, DNS token surfaced in `.status` only +> +> **Deferred past POC (revisit before GA):** +> - BYOC / Key Vault integration (D3 — spec field reserved but branch not implemented) +> - Kubernetes Event emission for DNS validation (D5) +> - Full finalizer edge-case handling +> - envtest integration tests + E2E +> - New `charts/hub-afd-controller-manager/` sibling chart (D9) +> - CEL / webhook validation beyond basics + +### Phase 2.1 — API types (`api/v1alpha1`) +- [ ] `FrontDoorProfile` type. + - Spec: `ResourceGroup`, `SkuName` (Standard_AzureFrontDoor | Premium_AzureFrontDoor), `Tags`. + - Status: `Conditions`, `ProfileID`, `EndpointHostname` (default `*.azurefd.net`), `ProvisioningState`, `ObservedGeneration`. +- [ ] `FrontDoorCustomDomain` type (namespaced). + - Spec: `Hostname`, `ProfileRef{Name}` (same-namespace only), `TLS{Mode: Managed|BYOC, KeyVaultCertificate{VaultURI, CertificateName, Version?}}`. + - Status: `Conditions`, `DomainID`, `ValidationState` (Pending|Approved|Rejected|TimedOut|InternalError), `DNSValidationToken`, `DNSValidationExpiry`, `TLSState`, `DeploymentStatus`, `ObservedGeneration`. +- [ ] Generate CRD manifests into `config/crd/bases`. +- [ ] Unit tests for validation (webhook or CEL): TLS mode ↔ KV ref consistency, hostname format, immutability of `Hostname` / `ProfileRef`. + +### Phase 2.2 — Common Azure client plumbing (`pkg/common/azureclient`) +- [ ] AFD SDK client factory using `azidentity.NewWorkloadIdentityCredential`. +- [ ] Key Vault certificate client factory (same credential). +- [ ] Shared retry/backoff + long-poll requeue helper (validation may take minutes). + +### Phase 2.3 — FrontDoorProfile controller (`pkg/controllers/hub/frontdoorprofile`) +- [ ] Reconciler: ensure AFD profile + default endpoint exist; write `ProfileID` + `EndpointHostname` to status. +- [ ] Finalizer: block deletion until profile is removed from ARM (or force-orphan annotation). +- [ ] Unit tests (table-driven) + envtest integration. + +### Phase 2.4 — FrontDoorCustomDomain controller (`pkg/controllers/hub/frontdoorcustomdomain`) +- [ ] Resolve `ProfileRef` (same-namespace); requeue if profile not Ready. +- [ ] Ensure `AFDCustomDomain` ARM resource under referenced profile. +- [ ] Poll validation state; write `DNSValidationToken` to status **and** emit an Event (`DNSValidationRequired` with the TXT record and expected value). +- [ ] Managed TLS: request managed cert, bind on approval. +- [ ] BYOC: resolve `KeyVaultCertificate`, verify MI can read it, create AFD secret + bind. +- [ ] Bind approved domains to the endpoint/default route. +- [ ] Finalizer: unbind + delete ARM resource before removing finalizer. +- [ ] Unit + envtest integration tests. + +### Phase 2.5 — E2E (`test/e2e`) +- [ ] Profile lifecycle test (create/delete round-trip). +- [ ] Custom domain lifecycle test — gated behind env vars for a real DNS zone + Key Vault; skip when unset. + +### Phase 2.6 — Helm + RBAC (**new** chart `charts/hub-afd-controller-manager/`) +- [ ] New sibling chart — do **not** modify `charts/hub-net-controller-manager/`. Compatibility contract in D9 must hold. +- [ ] Ship new CRDs from within the AFD chart (no coupling to the shared `net-crd-installer`). +- [ ] RBAC for the AFD controller ServiceAccount. +- [ ] ServiceAccount annotations for Workload Identity (`azure.workload.identity/client-id`). +- [ ] Values for AFD subscription / resource group / tenant / client-id defaults; no shared `azureCloudConfig` block with the ATM chart. +- [ ] `helm template` diff on the existing ATM chart is empty after this change (regression check). + +## Decisions + +Each decision below records the **choice**, the **options considered**, the **justification**, the **trade-offs accepted**, and any **reversibility notes**. This is the authoritative record of intent for Phase 2. + +### D1. Phase custom domains into Phase 2 (do not defer to Phase 5) + +- **Choice:** Ship custom domain support as part of Phase 2 alongside profile/endpoint reconciliation. +- **Options considered:** + - (A) Phase 2 = profile+endpoint only; custom domains in Phase 5. + - (B) Phase 2 includes custom domains end-to-end. +- **Justification:** + - Azure AFD is used almost exclusively for public/enterprise-facing traffic; the default `*.azurefd.net` hostname is technically functional but commercially unusable for branded, SEO-sensitive, or compliance-bound workloads. + - Early adopters of this controller are expected to be application teams onboarding real production workloads, not internal-only services; a Phase 2 without custom domains would ship a controller they cannot actually use in production. + - Deferral would force a status-schema expansion later (adding `CustomDomains` after v1 GA), which — while additive and backward-compatible — creates doc churn, release-note noise, and a period where the controller looks "half-done" to consumers. +- **Trade-offs accepted:** + - Larger Phase 2 scope, longer time-to-first-release. + - Higher risk of API churn on `FrontDoorCustomDomain` status fields (validation state, TLS state) since we're committing them earlier. + - E2E environment must include a real DNS zone + Key Vault, raising CI complexity. +- **Reversibility:** Low — once the CRD ships in a tagged release, its shape becomes a public contract. Justifies extra care on API design in D2/D3. + +### D2. Model custom domains as a separate `FrontDoorCustomDomain` CRD (not an inline field on `FrontDoorProfile`) + +- **Choice:** Introduce a distinct namespaced CRD `FrontDoorCustomDomain` with a `ProfileRef` to the owning profile. +- **Options considered:** + - (A) Inline slice: `FrontDoorProfileSpec.CustomDomains []CustomDomainSpec`. + - (B) Separate CRD referencing the profile. +- **Justification:** + - **Ownership boundary:** In real deployments the platform team owns the AFD profile (infra) while application teams own domains (app). Inline fields force both teams to edit the same object, breaking RBAC and GitOps ownership. + - **Lifecycle independence:** Domains can be added, validated, rotated, and removed on a completely different cadence than the profile. Multi-tenant SaaS scenarios can add/remove domains daily; the profile is essentially immutable. + - **Per-resource status:** Each domain has its own async validation + TLS state machine. Modeling N domains as inline entries forces a status slice with per-index conditions, which is awkward to consume and hard to alert on. + - **Precedent:** Kubernetes Gateway API split `Gateway` (infra) from `HTTPRoute` (app) for the same reason. Azure ARM itself models `AFDCustomDomain` as a distinct child resource — the API mirrors reality. + - **Frequency-of-change is a red herring:** The user asked whether domains change often; the answer is "rarely, except in multi-tenant SaaS," but frequency was not the deciding factor. Ownership, lifecycle, and status modeling were. +- **Trade-offs accepted:** + - Two CRDs to install, document, and RBAC. + - Cross-resource reconciliation (custom-domain controller must watch profiles and requeue). + - Slightly more boilerplate than inline. +- **Reversibility:** Low — same GA-contract argument as D1. + +### D3. Support both Managed TLS and BYOC (Key Vault) from day one + +- **Choice:** `FrontDoorCustomDomain.Spec.TLS.Mode` supports `Managed` and `BYOC`; BYOC references a Key Vault certificate via `KeyVaultCertificate{VaultURI, CertificateName, Version?}`. +- **Options considered:** + - (A) Managed only in Phase 2, BYOC later. + - (B) Managed + BYOC in Phase 2. +- **Justification:** + - Enterprise adopters routinely require BYOC for compliance (HSM-backed certs, corporate PKI, cert pinning, HSTS preload lists tied to specific keys). + - Adding BYOC later would either require a spec migration or a parallel field, both of which break users mid-flight. + - Managed TLS alone excludes the exact customer segment (regulated / large enterprise) most likely to demand custom domains. +- **Trade-offs accepted:** + - Controller must resolve Key Vault references, verify MI permissions, and manage AFD `secrets` resources — non-trivial extra code and RBAC surface (Key Vault `get`/`list` on secrets/certificates for the workload identity). + - Cert rotation semantics must be defined now (watch KV cert version, re-bind on change) rather than deferred. +- **Reversibility:** Medium — spec is additive; the BYOC branch could in principle be marked deprecated later, but that would strand existing users. + +### D4. CRD scope: namespaced, `ProfileRef` restricted to same namespace + +- **Choice:** Both `FrontDoorProfile` and `FrontDoorCustomDomain` are namespaced. A `FrontDoorCustomDomain` may only reference a `FrontDoorProfile` in its own namespace. +- **Options considered:** + - (A) Cluster-scoped CRDs, any-to-any references. + - (B) Namespaced, same-namespace-only references. + - (C) Namespaced with cross-namespace refs gated by a `ReferenceGrant`-style opt-in. +- **Justification:** + - Namespaced resources inherit standard Kubernetes RBAC and multi-tenancy patterns; cluster-scoped CRDs would concentrate authority and complicate delegated administration. + - Same-namespace-only reference is the **least-privilege default** and prevents a tenant in namespace `foo` from hijacking a profile in namespace `bar`. + - Cross-namespace referencing is a common future ask, but Gateway API has shown that adding a `ReferenceGrant`-style opt-in later is fully backward-compatible. +- **Trade-offs accepted:** + - Users who legitimately want one central-infra profile serving domains in many namespaces must wait for the opt-in mechanism. +- **Reversibility:** High — adding cross-namespace support later is additive; today's users are unaffected. + +### D5. Surface DNS validation token via both `.status` and Kubernetes Events + +- **Choice:** Write the DNS TXT challenge (`DNSValidationToken` + expected record name) into `FrontDoorCustomDomain.status` **and** emit a `DNSValidationRequired` Event. +- **Options considered:** + - (A) `.status` only. + - (B) `.status` + Event. +- **Justification:** + - Status is the correct machine-readable surface (GitOps, operators, automation). + - DNS validation is a one-time human step; humans debug with `kubectl describe` and `kubectl get events`, where fields buried in status are easy to miss. + - Event carries the exact TXT record and value inline, cutting the "how do I complete validation?" support loop. +- **Trade-offs accepted:** + - Extra event volume (bounded — emitted only on state transitions, not every reconcile). + - Slightly more test surface (must assert event emission). +- **Reversibility:** High — event emission can be tuned/disabled behind a flag if noisy. + +### D6. Azure authentication: Workload Identity (federated token) + +- **Choice:** Controller authenticates to Azure Resource Manager and Key Vault using Azure AD Workload Identity via `azidentity.NewWorkloadIdentityCredential` (reads `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_FEDERATED_TOKEN_FILE`). +- **Options considered:** + - (A) Workload Identity (federated token). + - (B) Managed Identity via IMDS. + - (C) Service principal with client secret stored in a Kubernetes `Secret`. +- **Justification:** + - Workload Identity is the current Microsoft-recommended default on AKS; IMDS-based MI is being phased out in favor of federated tokens. + - Removes any long-lived secret from the cluster (rules out C on principle). + - Cleanly scoped per-ServiceAccount via `azure.workload.identity/client-id` annotation, matching per-controller least-privilege. +- **Trade-offs accepted:** + - Requires OIDC issuer + federated credentials to be configured on the AKS cluster and Azure AD app; operators without WI must configure it before installing the chart. + - Slightly more Helm surface (SA annotations + values for client-id/tenant-id). +- **Reversibility:** High — the credential is constructed in one place (`pkg/common/azureclient`); switching to a `DefaultAzureCredential` chain to also support MI would be a small localized change. + +### D7. No umbrella `GlobalLoadBalancer` CRD; keep AFD and ATM strictly separate + +- **Choice:** Do not introduce a `GlobalLoadBalancer` umbrella CRD or a `pkg/common/globalload/` shared-abstraction package. AFD and (future) ATM are modeled as independent, product-named CRDs with product-scoped helper packages (`pkg/common/azurefrontdoor/`, later `pkg/common/azuretrafficmanager/`). Genuinely cross-cutting utilities (Azure auth, ARM polling, conditions, finalizers) live in existing common packages (`pkg/common/azureclient/`, `pkg/common/conditions/`), not in a speculative umbrella package. +- **Options considered:** + - (A) Independent controllers, no umbrella — product-named CRDs and helpers. + - (B) Umbrella `GlobalLoadBalancer` CRD with a provider interface and shared `pkg/common/globalload/` abstractions from day one. +- **Justification:** + - **Overlap is thin.** AFD is L7 (HTTP reverse proxy, WAF, cache, TLS termination, custom domains). ATM is L4/DNS (returns IPs by routing method). Custom domains, TLS, WAF, caching, and rules are AFD-only. Endpoint model, health-probe semantics, and async ARM-op tuning differ. The genuinely reusable pieces (auth, polling, conditions, finalizers) already have natural homes in existing common packages. + - **Umbrella specs leak.** A shared spec inevitably degenerates to a discriminated union (`Spec.AzureFrontDoor{...} | Spec.AzureTrafficManager{...}`), which relocates two shapes into one CRD without simplifying anything and complicates validation and defaulting. + - **Different consumers.** Users pick AFD for L7/CDN/WAF and ATM for DNS-based failover. The choice is driven by requirements, not preference — a "pick-my-provider" UX solves a problem few users actually have. + - **Provider interfaces designed before their first implementation are wrong.** Committing to an abstraction now, with only AFD in hand, would bake in AFD-shaped assumptions that ATM would then have to work around. + - **Precedent.** Gateway API works as an umbrella because L7 gateways genuinely share a spec. Cross-layer unification attempts (Service type=LoadBalancer + ExternalDNS + Ingress) have consistently stayed as separate resources composed together. +- **Trade-offs accepted:** + - Users who want to switch between AFD and ATM must migrate between two CRDs (acceptable — such migrations are rare and require operational planning anyway). + - Some duplication in controller scaffolding across product-scoped packages (mitigated by shared helpers in `pkg/common/azureclient/`). + - No single "global LB" entry point in the API surface today. +- **Reversibility:** **High.** If demand emerges, a `GlobalLoadBalancer` CRD can be added later as a thin composition layer over the existing product CRDs (Crossplane `Composition`-style) — additive and non-breaking. The reverse (retracting a shipped umbrella CRD) would be far more disruptive, which is another reason to defer. +- **Impact on Phase 2:** None. Proceed with `FrontDoorProfile` + `FrontDoorCustomDomain` as planned; AFD-specific helpers live under `pkg/common/azurefrontdoor/`. + +### D8. Do not deviate from upstream `mcs-api` (KEP-1645) unless strictly necessary + +- **Choice:** Fleet networking CRDs that mirror upstream `mcs-api` types — currently `ServiceExport` and `MultiClusterService` — **must not diverge** from the upstream shape. Configuration extensions carry over annotations (existing pattern) rather than by adding local `Spec` fields. +- **Applies to open question 1.5 (`ServiceExport` `Spec`):** Reject Option A (add `Spec`). Adopt **Option B** — the new Proposal 002 §3.3 field ships as an annotation under the `networking.fleet.azure.com/` prefix, with defaulting and validation implemented in the controller. +- **Scope of the rule:** Any type whose name and semantics match an upstream `mcs-api` type. Fleet-only types (e.g., `FrontDoorProfile`, `FrontDoorCustomDomain`, `InternalServiceExport`) are unaffected and continue to use idiomatic spec/status. +- **Options considered:** + - (A) Case-by-case divergence when it's convenient. + - (B) Hard rule: no divergence from `mcs-api` types. +- **Justification:** + - **Documented public contract.** The AKS Fleet L4 load balancing docs (learn.microsoft.com/azure/kubernetes-fleet/l4-load-balancing) describe `ServiceExport` and `MultiClusterService` with the upstream shape. Divergence would silently break customers following the official docs, break tooling that assumes mcs-api compliance, and complicate future upstream conformance testing. + - **Portability across MCS implementations.** Users and third-party tools (Submariner, Cilium ClusterMesh, other mcs-api impls) expect a stable, portable `ServiceExport`. A Fleet-only `Spec` is a lock-in signal. + - **Precedent in this repo.** The existing `weight` knob is already carried as an annotation (`networking.fleet.azure.com/weight`, documented at `api/v1alpha1/serviceexport_types.go:43-49`) exactly because of this constraint. Proposal 002's new field is analogous and should follow the same pattern. + - **Future upstream alignment.** If KEP-1645 (or a successor) eventually adds an equivalent field to `ServiceExport.Spec`, we can migrate the annotation to that upstream field with a deprecation window — a strictly better outcome than shipping a Fleet-specific `Spec` we then have to reconcile with upstream. +- **Trade-offs accepted:** + - Annotation-carried config is untyped: no OpenAPI validation, no `kubectl explain`, no CEL rules. Validation and defaulting must live in the controller and be covered by unit tests. + - Annotation keys grow linearly with tunables; if the count becomes unwieldy, revisit — but only after upstream direction is clearer. + - Slightly worse UX than a typed `Spec`. +- **Reversibility:** **High.** The rule is a policy, not a schema. If upstream evolves or maintainers change position, we can add spec fields later without a breaking migration (annotation-first users just gain a typed path). +- **Impact on other decisions:** Supersedes the earlier informal recommendation on 1.5. Open question **OQ (1.5)** is now closed as "annotation-based; no `ServiceExport.Spec`." + +### D9. Package AFD as a separate sibling chart, do not extend the existing `hub-net-controller-manager` chart + +**Baseline captured:** `.github/.copilot/breadcrumbs/baselines/` contains `hub-net-atm-default.yaml` and `hub-net-atm-enabled.yaml` — rendered outputs of the current ATM chart in both flag states (Helm v4.2.3). SHA-256 hashes and the exact `helm template` commands to reproduce are in `baselines/README.md`. Any AFD PR must produce empty `diff` output against these files before merge, mechanically proving the D9 compatibility contract. + +- **Choice:** Ship a new top-level chart `charts/hub-afd-controller-manager/` with its own Deployment, ServiceAccount, RBAC, values, and images. The existing `charts/hub-net-controller-manager/` chart is **not modified** as part of Phase 2. ATM continues to be delivered by that chart unchanged; AFD is a second `helm install`. +- **Options considered:** + - (A) Extend existing chart with a second Deployment + SA behind an `enableFrontDoorFeature` flag. + - (B) New sibling chart `charts/hub-afd-controller-manager/`. + - (C) Single Deployment with dual identities (rejected outright — violates the identity-split security requirement by construction). +- **Justification:** + - **Identity split enforceable at the packaging boundary.** A Pod binds to exactly one ServiceAccount, and Workload Identity is per-SA. Two identities require two Pods and two SAs. Putting them in separate charts guarantees the AFD SA cannot be accidentally reused for ATM operations. + - **Zero-risk upgrade for existing ATM installs.** With the ATM chart untouched, `helm upgrade` on existing releases produces an empty diff (verifiable via `helm template`). No values renames, no default flips, no new required keys. + - **Consistent with D7.** AFD and ATM are independent products with independent lifecycles; the chart layer should reflect that. Cramming both into one chart is the packaging analogue of the umbrella CRD we rejected. + - **Independent release cadence.** AFD image versions, CRDs, RBAC, and values evolve on their own timeline without forcing ATM users to re-review each release. + - **Ecosystem norm.** Kubernetes controllers of this shape (Azure Service Operator, ExternalDNS providers, cert-manager sub-components) ship one chart per controller. +- **Trade-offs accepted:** + - Operators wanting both must run two `helm install`s. + - A new chart to publish, document, and version. + - Some templating duplication (leader-election flags, image blocks, resource blocks) — acceptable and can be factored into a shared library chart later if it grows. +- **Compatibility contract for existing ATM installations** (must hold for the entirety of Phase 2): + 1. `helm upgrade` of an existing `hub-net-controller-manager` release with unchanged values produces a byte-identical rendered manifest. + 2. No keys under `azureCloudConfig` are renamed, removed, or repurposed. + 3. `enableTrafficManagerFeature` default is not flipped. + 4. The `hub-net-controller-manager-sa` ServiceAccount is not renamed and gains no new bindings. + 5. AFD CRDs (`FrontDoorProfile`, `FrontDoorCustomDomain`) are installed by the AFD chart only — never as a side effect of installing or upgrading the ATM chart. +- **Reversibility:** **Medium.** Merging the two charts later would be a breaking Helm change; splitting them further is easy. Consciously erring on the side of separation now because the reverse migration is much more disruptive. +- **Impact on other decisions:** + - Updates Phase 2.6 in the plan: swap "update `charts/hub-net-controller-manager`" for "create `charts/hub-afd-controller-manager`". + - No change to D1–D8. + +### Summary table + +| ID | Decision | Choice | Reversibility | +|---|---|---|---| +| D1 | Phase custom domains | Phase 2, not deferred | Low | +| D2 | API shape | Separate `FrontDoorCustomDomain` CRD | Low | +| D3 | TLS support | Managed + BYOC (Key Vault) | Medium | +| D4 | CRD scope | Namespaced, same-namespace `ProfileRef` | High (cross-ns can be added) | +| D5 | DNS token surfacing | `.status` + Kubernetes Event | High | +| D6 | Azure auth | Workload Identity (federated token) | High | +| D7 | Umbrella `GlobalLoadBalancer` CRD | No — keep AFD and ATM separate; product-scoped helpers | High (can be added additively later) | +| D8 | mcs-api divergence | Prohibited for `ServiceExport`/`MultiClusterService`; use annotations under `networking.fleet.azure.com/` | High | +| D9 | AFD chart packaging | New sibling chart `charts/hub-afd-controller-manager/`; do not modify existing ATM chart | Medium | + +## Implementation Details + +_To be filled as implementation proceeds._ + +## Changes Made + +### Phase 2.1 — API types (completed 2026-07-18) +- **New:** `api/v1alpha1/frontdoorprofile_types.go` — `FrontDoorProfile` + `FrontDoorProfileList`, immutable `ResourceGroup`/`Sku`, `Programmed` condition type/reasons. +- **New:** `api/v1alpha1/frontdoorcustomdomain_types.go` — `FrontDoorCustomDomain` + `FrontDoorCustomDomainList`, same-namespace `ProfileRef` (D4), immutable `Hostname`, `TLS{Mode, KeyVaultCertificate}` with CEL cross-field validation, validation-state enum, `Programmed` condition type/reasons. +- **Regenerated:** `api/v1alpha1/zz_generated.deepcopy.go` via `controller-gen v0.20.0`. +- **Regenerated:** `config/crd/bases/networking.fleet.azure.com_frontdoorprofiles.yaml`, `config/crd/bases/networking.fleet.azure.com_frontdoorcustomdomains.yaml`. + +### Phase 2.2 — Azure client factory (completed 2026-07-18) +- **New:** `pkg/common/azurefrontdoor/client.go` — `Config` + `LoadConfigFromEnv()`, `NewCredential()` (WorkloadIdentityCredential per D6), `NewClients()` bundling `Profiles`/`AFDEndpoints`/`CustomDomains` sub-clients, `DefaultARMClientOptions()`. +- **New:** `pkg/common/azurefrontdoor/client_test.go` — table-driven validation tests + nil-guard tests. +- **Dependency:** added `github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cdn/armcdn v1.1.1`. + +### Phase 2.3 — FrontDoorProfile controller, happy path (completed 2026-07-19) +- **New:** `pkg/controllers/hub/frontdoorprofile/controller.go` — reconciler + finalizer + AFD profile & default endpoint CRUD + status. +- **Modified:** `pkg/common/objectmeta/objectmeta.go` — added `FrontDoorProfileFinalizer` and `FrontDoorCustomDomainFinalizer` constants under existing `networking.fleet.azure.com/` prefix. + +### Phase 2.4 — FrontDoorCustomDomain controller, Managed TLS only (completed 2026-07-19) +- **New:** `pkg/controllers/hub/frontdoorcustomdomain/controller.go` — reconciler + finalizer + AFD custom domain CRUD + validation-state polling + Programmed condition; BYOC path rejected with permanent Invalid condition per POC scope (D3). + +### Wiring — hub-net-controller-manager entrypoint (completed 2026-07-19) +- **Modified:** `cmd/hub-net-controller-manager/main.go` — + - Added imports for `pkg/common/azurefrontdoor`, `pkg/controllers/hub/frontdoorprofile`, `pkg/controllers/hub/frontdoorcustomdomain`. + - New `--enable-frontdoor-feature` flag (default `false`) with justifying comment referencing D9 (WI + AFD subscription config not present on existing installs; ATM-only installs unaffected). + - New `frontDoorFeatureRequiredGVKs` slice for CRD presence check, mirroring `trafficManagerFeatureRequiredGVKs`. + - New `if *enableFrontDoorFeature` block in `main()`: CRD check → `LoadConfigFromEnv` → `NewCredential` → `NewClients` → `SetupWithManager` for both AFD reconcilers. + - Zero changes to the ATM branch, `cloudConfigFile`, or `initAzureTrafficManagerClients` — D9 compatibility contract preserved. + +### POC validation +- `go build ./...` — exit 0 (verified after every phase). +- `go vet ./...` — exit 0 (verified after Phase 2.3 and Phase 2.4). +- `go test ./pkg/common/azurefrontdoor/...` — ok. +- ATM chart baseline captured at `.github/.copilot/breadcrumbs/baselines/` (D9 zero-diff guard). + +## Before/After Comparison + +_To be filled as implementation proceeds._ + +## References + +- Azure Front Door ARM model: `Microsoft.Cdn/profiles`, `.../afdEndpoints`, `.../customDomains`, `.../secrets`. +- Kubernetes Gateway API split precedent (`Gateway` vs `HTTPRoute`) for lifecycle-based API separation. +- Repo custom instructions: Breadcrumb Protocol, Testing Rules. + +## Checklist + +- [ ] 2.1 API types + CRD manifests + validation tests +- [ ] 2.2 Azure client + Key Vault client + Workload Identity wiring +- [ ] 2.3 FrontDoorProfile controller (reconcile, finalizer, tests) +- [ ] 2.4 FrontDoorCustomDomain controller (reconcile, validation polling, TLS both modes, finalizer, tests) +- [ ] 2.5 E2E tests (profile always; custom domain gated) +- [ ] 2.6 Helm chart + RBAC + WI ServiceAccount annotations + +## Success Criteria + +- `kubectl apply` of a `FrontDoorProfile` creates the ARM profile + endpoint and reports `Ready=True` with a populated `EndpointHostname`. +- `kubectl apply` of a `FrontDoorCustomDomain` (Managed TLS) produces a `DNSValidationRequired` Event with the TXT token; once the TXT record exists, status transitions `Pending → Approved`, TLS provisions, and domain binds to the endpoint. +- BYOC path validated against a real Key Vault cert in E2E. +- Deleting either CR cleans up the corresponding ARM resource; finalizers block premature deletion. +- All unit and integration tests pass under `go test ./...`. diff --git a/.github/.copilot/breadcrumbs/2026-07-20-1108-afd-export-mode-mcs-parity.md b/.github/.copilot/breadcrumbs/2026-07-20-1108-afd-export-mode-mcs-parity.md new file mode 100644 index 00000000..d28910d6 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-07-20-1108-afd-export-mode-mcs-parity.md @@ -0,0 +1,474 @@ +# AFD Export-Mode — MCS-API Parity + +## Requirements + +Update the AFD proposal (`docs/first-party/001..003`) so it no longer +adds a Fleet-specific `Spec.ExportMode` field to `ServiceExport`. +That divergence conflicts with the repository preference to keep +`ServiceExport` / `MultiClusterService` structurally aligned with +upstream mcs-api (KEP-1645) and to carry Fleet-specific configuration +via annotations. + +Adopt a two-signal model instead: + +1. **Primary — inference from the `Service`.** The member controller + determines that an exported Service is destined for the AFD + (private-origin) path by observing the Azure cloud-provider + annotations already required to create an internal LB and a PLS + (`service.beta.kubernetes.io/azure-load-balancer-internal: "true"` + and `service.beta.kubernetes.io/azure-pls-create: "true"`). + No CR schema change; the mode is emergent from the Service's + actual shape. +2. **Fallback / opt-in intent flag — annotation on `ServiceExport`.** + `networking.fleet.azure.com/export-mode: L7-FrontDoor | L4-TrafficManager` + matches the existing `networking.fleet.azure.com/weight` + precedent and lets tenants declare intent independently of the + Service's current annotation state. + +Precedence when both are present: **annotation wins** if set; +otherwise infer from the Service. Conflict (annotation says +`L7-FrontDoor` but Service lacks internal+PLS annotations) +surfaces `ServiceExportValid=False` with a new +`Reason=ExportModeAnnotationServiceMismatch`. + +## Additional comments from user + +- Stored preference (user memory): *"In fleet-networking, prefer no + divergence from upstream mcs-api (KEP-1645) shapes for + ServiceExport/MultiClusterService; carry Fleet-specific config via + annotations, not new Spec fields."* +- Precedence rule chosen by user in this session: + *"Annotation wins if set; otherwise infer from Service"* — see + `ask_user` turn on 2026-07-20. + +## Plan + +Docs-only change; no code yet (proposal not implemented). + +1. `docs/first-party/001-afd-global-load-balancing.md` + - §3.3 (member-controller behaviour): reword to describe + inference-based detection, and add optional annotation as the + explicit-intent surface. Remove the language that says the + controller "projects" annotations onto the Service — the tenant + (or platform admission policy) owns the Service annotations; + the controller only observes them. + - §4.2 (additive CRD changes): drop the `Spec.ExportMode` + bullet on `ServiceExport`; keep only the + `InternalServiceExport.Status.PrivateLinkService` addition. + Add a sentence about the annotation-based opt-in and the + precedence rule. +2. `docs/first-party/002-afd-implementation-plan.md` + - §2.1 (file table): remove the two `serviceexport_types.go` + modification rows (v1alpha1 and v1beta1). Keep the + `internalserviceexport_types.go` rows. + - §3.3: replace the `ExportMode` Go sketch with an "Annotation + constants" sketch under `pkg/common/objectmeta/` (or wherever + the `weight` annotation lives) and a short note that no + `ServiceExport` schema change is required. + - §4.3 (member `serviceexport` extension): rewrite the + additions to: + - Read the annotation first; if unset, infer from the + Service's internal-LB + PLS annotations. + - Validate presence/absence of the required Service + annotations; if the annotation demands AFD but the Service + is not internal+PLS, surface the mismatch and do not mutate + the Service. + - Copy the AKS-programmed PLS resource ID into + `InternalServiceExport.status.privateLinkService`. + - Explicitly state that the member controller does **not** + write to the Service (annotations are tenant-owned). + - §8 (east-west compatibility): update the tables/prose that + currently mention `Spec.ExportMode` default behaviour to + describe the annotation/inference model instead. + - §9 (risks): add a row about + "annotation says AFD, Service lacks PLS annotations" + precedence and validation. +3. `docs/first-party/003-pre-implementation-checklist.md` + - Close open item **1.5** with the decision documented above. + - No new open items — the precedence and mismatch handling + are settled by this update. + +## Decisions + +- **No new `Spec` field on `ServiceExport`.** Preserves upstream + mcs-api parity per the repo preference. +- **Primary signal: Service shape.** The Service's internal-LB and + PLS annotations are the physical prerequisite for AFD anyway; using + them as the signal collapses two sources of truth into one. +- **Opt-in annotation on `ServiceExport`.** Kept as a tenant-visible + intent flag for platforms that want to declare mode before the + Service is fully provisioned (e.g., GitOps pipelines that apply + the `ServiceExport` in one wave and the `Service` in the next). +- **Annotation wins when both are set.** Explicit intent overrides + observed shape; mismatches surface a validation condition. +- **Controller does not mutate tenant-owned Services.** The proposal + originally described "projecting" annotations onto the Service; that + crosses ownership lines and complicates ATM ↔ AFD transitions. + Ownership stays with the tenant / GitOps / admission policy. + +## Implementation Details + +See per-file edits applied to `001`, `002`, `003` under +`docs/first-party/`. This breadcrumb accompanies those edits — no +code changes in this pass. + +## Changes Made + +- `docs/first-party/001-afd-global-load-balancing.md` — §3.3 and + §4.2 reworded. +- `docs/first-party/002-afd-implementation-plan.md` — §2.1, §3.3, + §4.3, §8, §9 updated. +- `docs/first-party/003-pre-implementation-checklist.md` — item + 1.5 marked resolved. + +## Before/After Comparison + +| Aspect | Before | After | +|---|---|---| +| ServiceExport schema | Grew a new `Spec.ExportMode` enum | Unchanged — matches upstream mcs-api | +| Mode signal (primary) | Explicit `Spec.ExportMode` field | Inferred from Service's `azure-load-balancer-internal` + `azure-pls-*` annotations | +| Mode signal (opt-in) | n/a | `networking.fleet.azure.com/export-mode` annotation on `ServiceExport` | +| Precedence | n/a | Annotation wins if set; else inference; mismatch surfaces validation condition | +| Member controller writes to Service? | Yes (annotation projection) | No — tenants/GitOps own Service annotations | + +## References + +- KEP-1645 (Multi-Cluster Services API): +- Upstream mcs-api reference implementation: +- Proposal 001: `docs/first-party/001-afd-global-load-balancing.md` (Draft) +- Proposal 002: `docs/first-party/002-afd-implementation-plan.md` (Draft) +- Checklist 003: `docs/first-party/003-pre-implementation-checklist.md` (Open) +- Existing Fleet-specific annotation precedent: `networking.fleet.azure.com/weight` on `ServiceExport` (see `pkg/common/objectmeta`). +- AKS internal LB annotations: +- AKS PLS annotations: + +--- + +## Addendum — AKS Automatic support + AFD/ATM coexistence (2026-07-20) + +Follow-up in the same session. Two additions on top of the +export-mode change above. + +### Requirements (addendum) + +1. Declare **AKS Automatic** a first-class supported member cluster + SKU alongside AKS Standard. AFD + PLS data-plane primitives are + identical on both SKUs; the only differences are operational + (BYO VNet planning, Deployment Safeguards). +2. Explicitly document **coexistence with the existing Traffic + Manager (ATM) path**: fleet-wide, same-namespace-different-Service + coexistence is supported; same-`ServiceImport`-on-both-surfaces is + forbidden by the AFD backend reconciler. +3. Capture the security implications (bypass surface, split + identities, WAF-per-surface, audit split) and the tenancy + implications (weight-annotation divergence, per-surface quotas, + cost attribution, migration cadence) in the proposal itself + rather than only in chat. + +### Plan (addendum) + +* `docs/first-party/001-afd-global-load-balancing.md` + * New **§3.4** — Member cluster requirements (AKS Standard + + Automatic): Standard LB, BYO VNet, PLS NAT subnet with + `privateLinkServiceNetworkPolicies: Disabled`, PLS auto-approval + for the AFD subscription. Automatic-specific notes for + Deployment Safeguards, NAP, cluster-config lockdown, AGC + coexistence. + * New **§3.5** — Coexistence with ATM: coexistence-granularity + table, security implications, tenancy implications, migration + playbook. + * §2.3 non-goals: add "managing member-cluster provisioning" as + an explicit non-goal, pointing at §3.4. +* `docs/first-party/002-afd-implementation-plan.md` + * New **§6.5** — AKS Automatic compatibility: Safeguards + requirements, file table (Deployment `securityContext` / + `resources`, PDBs, `values.yaml` surface, `hack/verify-safeguards.sh`), + NAP handling, e2e coverage note, doc pointers. + * §9 risks: two new rows — Safeguards drift, and NAP-driven + controller restarts. + * §11 success criteria: add criterion 7 — clean install on an AKS + Automatic member with Safeguards in Enforcement mode. +* `docs/first-party/003-pre-implementation-checklist.md` + * New **§1.6** — closed: "AKS Automatic as supported member SKU" + — Yes, first-class alongside Standard. + * New spike **§3.7** — AKS Automatic Deployment Safeguards install + validation: `helm template` + policy check offline, fold gaps + into §6.5.2 chart hygiene work before Phase 4. + * §6 readiness table gains an "AKS Automatic install validated" + row. + +### Decisions (addendum) + +- **AKS Automatic is supported.** No code branch; the safeguards-clean + chart is the correct chart for AKS Standard too. Data-plane code + paths are identical. +- **BYO VNet is a platform responsibility, not a Fleet + responsibility.** Fleet controllers do not provision or reshape + subnets; the PLS NAT subnet must exist with the correct + network-policy at cluster create time. +- **AFD and ATM coexist behind independent feature flags.** No plan + to deprecate ATM. Same-`ServiceImport`-on-both-surfaces is + forbidden at reconcile time (Proposal 002 §8.4). +- **Compliance is per-tenant / per-Service, not per-fleet.** A fleet + running both surfaces is not "SFI compliant" as a whole; only + AFD-fronted Services are. First-party tenancy classes that must be + AFD-only should be enforced via cluster admission policy that + denies ATM CRs in those namespaces. +- **ATM → AFD migration is staged, not hot-swapped.** Add a second + Service with internal LB + PLS, verify AFD, cut DNS, delete the + ATM Service — no controller-driven cutover. + +### Changes Made (addendum) + +- `docs/first-party/001-afd-global-load-balancing.md` — new §3.4 + and §3.5 added; §2.3 non-goals extended. +- `docs/first-party/002-afd-implementation-plan.md` — new §6.5 + added; §9 risks and §11 success criteria extended. +- `docs/first-party/003-pre-implementation-checklist.md` — item + 1.6 added (resolved), spike 3.7 added, §6 readiness table + extended. + +### References (addendum) + +- AKS Automatic overview: +- AKS Deployment Safeguards: +- AKS Node auto-provisioning: +- AKS Application Gateway for Containers: +- Azure Private Link Service network-policy prerequisite: + + +--- + +## Addendum 2 (2026-07-20 T19:00Z): reconcile design docs with the cb02d14 POC + +### Requirements +Reconcile Proposals 001/002/003 with the POC that landed as +`cb02d14ba31b1a42f5ec51821c7bf3b856836581` +("feat(hub): POC for Azure Front Door (AFD) controllers behind +`--enable-frontdoor-feature`"). Flag anything that is impossible or +incompatible under the current wiring rather than silently +resolving it. + +### User inputs during the conversation +- **Choice A (SKU):** Premium-only. Drop `Standard_AzureFrontDoor` + from the CRD enum (Phase-4 change; document the current POC + permissiveness as a risk). +- **Choice B (identity split):** B-ii — sibling binary + (`cmd/hub-afd-controller-manager`) + sibling chart + (`charts/hub-afd-controller-manager`) is the target model, so §7 + of Proposal 001 is satisfiable. Docs-only this session + (**B-ii/a**); actual code split lands in a follow-up commit. + +### Facts extracted from cb02d14 by direct code reads +- **API types shipped:** + - `api/v1alpha1/frontdoorprofile_types.go` — Spec = {ResourceGroup, + Sku}; Status = {ResourceID, EndpointHostname, Conditions}; Sku + enum permissively includes both `Standard_AzureFrontDoor` and + `Premium_AzureFrontDoor`; immutability CEL on ResourceGroup and + Sku; metadata.name < 64 CEL. + - `api/v1alpha1/frontdoorcustomdomain_types.go` — Spec = + {ProfileRef (same-namespace, immutable), Hostname (immutable), + TLS = {Mode (Managed|BYOC), KeyVaultCertificate}}; Status = + {ResourceID, ValidationState, DNSValidationToken, + DNSValidationExpiry, Conditions}. BYOC reconciliation deferred. + - Neither `FrontDoorBackend` nor member-side changes are in cb02d14. +- **Controllers shipped:** + - `pkg/controllers/hub/frontdoorprofile/controller.go` — happy-path + reconcile; finalizer `networking.fleet.azure.com/frontdoor-profile-cleanup`; + Azure name `fleet-`; Azure Location hardcoded to `Global`. + - `pkg/controllers/hub/frontdoorcustomdomain/controller.go` + (Managed TLS only). + - No unit tests, no envtest scaffolding, no fake provider yet. +- **Client library shipped:** `pkg/common/azurefrontdoor/client.go` + (single file). Auth = **Workload Identity** via + `azidentity.NewWorkloadIdentityCredential`; env vars + `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, + `AZURE_FEDERATED_TOKEN_FILE`, `AZURE_SUBSCRIPTION_ID`. Bundles + `Profiles`, `AFDEndpoints`, `AFDCustomDomains` sub-clients. + `armcdn v1.1.1` pinned in `go.mod`. +- **Wiring in `cmd/hub-net-controller-manager/main.go`** — flag + `--enable-frontdoor-feature` (default `false`); when true, calls + `azurefrontdoor.LoadConfigFromEnv() → NewCredential → NewClients` + and registers both AFD reconcilers on the shared manager. **No + sibling binary or sibling chart.** + +### Incompatibilities / impossibilities flagged in docs +1. **Identity split (SFI blocker).** POC hosts AFD + ATM in one pod + → one WI federated subject → §7 not satisfied. Sibling + binary+chart is a hard GA prerequisite. Flagged in 001 §7 + "Impossibility flag on the current POC", 002 §2.6 and §9 risks, + 003 §2.4 and readiness table. +2. **SKU enum permissiveness.** POC allows Standard; docs require + Premium-only. Flagged as a Phase-4 CRD tightening in 002 §3.1 + Sku comment and §9 risks. +3. **`FrontDoorBackend` unimplemented.** Docs described it as a peer + to `FrontDoorProfile`; marked "not yet in main; Phase 4" in 001 + §4.1.2 / §5.1 and 002 §2.3. +4. **AFD/ATM coexistence guard unenforced.** Depends on + `FrontDoorBackend`; flagged in 002 §9. +5. **BYOC (Key Vault) TLS unimplemented.** Field shape present but + reconciler no-ops; flagged in 002 §9. +6. **Custom domain route binding deferred.** A `FrontDoorCustomDomain` + by itself does not front traffic; needs `FrontDoorBackend`. + Noted in 002 §3.3. + +### Files touched this session (Addendum 2) +- `docs/first-party/001-afd-global-load-balancing.md` + - §3.2 rewritten to describe the `fleet-` naming pattern and + the AFD-endpoint 46-char cap that motivates it. + - §4.1.1 `shortName: afdp` (was `fdp`); illustrative Go now marks + `POC:` fields vs. post-POC fields; Status shape corrected + (`EndpointHostname` string, `ResourceID` string, no separate + `EndpointResourceID`). + - §4.1.2 `shortName: afdb`; added "not yet implemented" callout. + - **New §4.1.3** `FrontDoorCustomDomain` (`afdcd`) — full + description of Spec/Status/Conditions/finalizer. + - §4.3 lists three CRD YAMLs, notes two are shipped. + - §5.1 marks profile+customdomain controllers "shipped in cb02d14", + backend controller "not yet in main; Phase 4". + - §5.2 clarifies member changes are Phase 3. + - §5.3 describes target sibling-binary wiring; documents POC + deviation. + - §5.4 aligns to actual `azurefrontdoor/client.go` and finalizer + constants. + - **§6 rewritten** to describe the sibling-chart target model with + an explicit "POC deviation" paragraph. + - **§7 rewritten** — Workload Identity, identity-split impossibility + flag on cb02d14. +- `docs/first-party/002-afd-implementation-plan.md` + - §2.1 file table: marks shipped rows, adds custom-domain type, + drops Location from Spec, notes Sku tightening as future work. + - §2.2 chart section: new sibling chart `charts/hub-afd-controller-manager`; + ATM chart unchanged; POC deviation call-out. + - §2.3 controllers table: shipped/pending markers. + - §2.5 common libs: aligned to shipped `client.go`. + - §2.6 entry points: new `cmd/hub-afd-controller-manager/main.go`; + documents the POC bridge. + - §3.1 `shortName: afdp`; metadata.name < 64 CEL; Sku comment + describes POC vs. GA; note about UID-based naming. + - **New §3.3** describes shipped `FrontDoorCustomDomain`. + - Renumbered old §3.3 → §3.4 (additive changes to existing types); + updated cross-references. + - §6 phase table: prepended "Current phase status" block. + - §9 risks: two new rows (Sku permissiveness, identity split) plus + two more (backend coexistence guard unenforced, BYOC deferred). + - §11 success criterion 1: references sibling chart install as the + canonical form; POC bridge as bridge. +- `docs/first-party/003-pre-implementation-checklist.md` + - §1.3 (custom domains): resolved — separate CRD in Phase 2. + - §2.4 (security review): resolved and expanded — sibling + binary+chart is a hard GA prerequisite. + - §3.2 (armcdn compat): resolved — pinned at v1.1.1. + - §3.6 (azcloudconfig): resolved — WI supersedes it. + - §6 readiness table: added SFI identity-split row and "POC + reconciled with docs" row. + - Fixed §3.3 → §3.4 citation. + +### Not touched +- The ~130 unrelated unstaged files in `api/`, `pkg/`, `cmd/`, + `test/`, `hack/`. +- Any code files (per B-ii/a: docs only this session; code split + is a follow-up commit). + +--- + +## Addendum 3 (2026-07-20 T22:00Z-07:00): landing the reconciled design + +This addendum records the 11-commit series that closes the delta the +reconciliation pass (Addendum 2) identified between the `cb02d14` +POC and the target design in Proposals 001/002/003. All commits land +on branch `rchinchani/afd-first-party-proposal`; envtests green in +WSL and go vet / go build clean on Windows for every commit. + +### Commit series (in landing order) + +| # | SHA | Scope | +|---|-----|-------| +| 0a | `2fecf8d` | AFD SKU Premium-only enum tightening | +| 0b | `8367527` | CRD regen for §0a | +| 0c | `fcb37f2` | Split `hub-afd-controller-manager` binary | +| 0d | `5672313` | Dockerfile + Makefile for §0c | +| 0e | `c219ca2` | `charts/hub-afd-controller-manager` sibling chart | +| 0f | `668b56f` | `armcdn` v2 bump to unlock OriginGroup+Origins fakes | +| 0g | `9b6a337` | net-crd-installer test fixture repair | +| 0h | `73e4150` | frontdoorprofile envtest scaffolding | +| 1 | `aeb116c` | `FrontDoorProfileSpec.wafPolicy` + `complianceMode` | +| 2 | `5b52f84` | deepcopy + CRD regen for §1 | +| 3 | `95a0096` | Client bundle: WAFPolicies + SecurityPolicies | +| 4 | `b2cf58f` | Reconciler: attach WAF policy via SecurityPolicy | +| 5 | `4b85044` | envtest specs for WAF (happy / NotFound / SFI Detection) | +| 6a | `b85a115` | `objectmeta`: export-mode annotation + extractor | +| 6b | `c6d0d8e` | `InternalServiceExportSpec.ExportMode` + `PrivateLinkServiceResourceID` | +| 6c | `cb6f23a` | Member `serviceexport` reconciler: ExportMode + PLS ARM Get | +| 6d | `cf5324b` | Unit + integration tests for §6c | +| 7 | `c61dc43` | `FrontDoorBackend` v1alpha1 CRD | +| 8 | `afe12d7` | Client bundle: OriginGroups + Origins | +| 9 | `be0ccb2` | `frontdoorbackend` reconciler (happy / Invalid / Pending) | +| 10 | `ceac0ab` | AFD/ATM coexistence guard (`Conflict` reason) | +| 11 | `629644b` | Fake OriginGroup/Origin providers + envtest specs | +| doc | `d2ff7df` | Docs: competitive context (GKE/EKS/AKS parity) in 001 §2 | + +Plus this commit: docs cleanup (status-update callouts at top of +001/002/003 + this Addendum 3). + +### Design decisions locked during landing + +Anything not in Addendum 2 but that we hardened during the landing +pass: + +- **Wire contract for `ExportMode`.** Absent (`""`) on the wire is + treated semantically as `L4-TrafficManager`. The member reconciler + explicitly writes `""` for the default case so preexisting + `InternalServiceExport` shapes round-trip unchanged (kept 12 + integration specs green without churning their expectations). + `L7-FrontDoor` is written verbatim. +- **PLS resolution.** Requires an explicit + `networking.fleet.azure.com/azure-pls-name` annotation on the + source `Service`; no default derivation from + `cloud-provider-azure` internals. Rejected the "guess from + ILB name" path because it silently races service reconciliation. +- **`FrontDoorBackend` UID naming.** OriginGroup is named + `fleet-`; Origins are `fleet--`. + Deterministic per Kubernetes object, stable across renames, + matches the pattern `frontdoorprofile` already established with + `AzureProfileName`. +- **Weight math.** `ceil(backend.Weight * export.Weight / + sum(export.Weight))`, byte-for-byte identical to the TMB formula + (see `trafficmanagerbackend/controller.go` line 580) so operators + moving between ATM and AFD get identical traffic splits. +- **Coexistence guard scope.** Only rejects when a live + `TrafficManagerBackend` (DeletionTimestamp zero) in the same + namespace references the same `ServiceImport.Name`. TMB tearing + itself down is not a conflict, so the migration path + (delete TMB -> reconcile AFDB) works without any orchestration. +- **Origin `HostName` placeholder.** AFD requires a non-nil + HostName even when SharedPrivateLinkResource is present; we + pass the PLS ID string. Prevents leaking a public hostname while + keeping the API payload valid. +- **SetupWithManager watches.** FrontDoorProfile, TrafficManagerBackend + and InternalServiceExport all enqueue same-namespace + FrontDoorBackends (list-based). Chose list-based enqueue over a + field indexer because namespaces are expected to have single-digit + backend counts. + +### What is *not* in this series (intentional) + +- FrontDoorRoute + FrontDoorCustomDomain BYOC reconciler (Phase 4 + tail; requires Key Vault plumbing). +- Removing the POC bridge in `cmd/hub-net-controller-manager` + (GA prerequisite; sibling binary is running side-by-side today). +- e2e coverage on a live sub (Phase 4 e2e in 002 §11). +- OriginGroup health-probe / session-affinity knobs (§4.1.2 + Phase 4 tail — CRD does not expose them yet). +- Metrics + prometheus wiring on the FrontDoorBackend reconciler. + +### Verification + +- `make build` in WSL: green (all commits). +- `make local-unit-test` in WSL: green. +- `pkg/controllers/hub/frontdoorbackend` envtest suite: 5/5 passed. +- `pkg/controllers/hub/frontdoorprofile` envtest suite: unaffected, + still passes. +- `pkg/controllers/member/serviceexport` integration suite: green + after the wire-contract fix (`""` on default ExportMode). + diff --git a/.github/.copilot/breadcrumbs/baselines/README.md b/.github/.copilot/breadcrumbs/baselines/README.md new file mode 100644 index 00000000..2b870d9b --- /dev/null +++ b/.github/.copilot/breadcrumbs/baselines/README.md @@ -0,0 +1,29 @@ +# ATM chart baseline fixtures (D9 zero-diff regression guard) + +Captured with Helm v4.2.3 on 2026-07-18T23:23:52.8606428-07:00. +Chart: charts/hub-net-controller-manager (unmodified). + +| File | SHA-256 | Purpose | +|---|---|---| +| hub-net-atm-default.yaml | D6243E35628EE77DAD3A3D204BEEE6CB33B41B1508967EEF55D6B78D5883BA6D | Default values (enableTrafficManagerFeature=false). Establishes the baseline that MUST remain byte-identical after AFD work lands. | +| hub-net-atm-enabled.yaml | 309205EEDE8A102962368861AA2BBC65C75FA44307E457E1252F76C1617857AB | ATM enabled with representative values. Establishes the baseline for the ATM-on upgrade path. | + +## Regression check (run before merging any AFD PR) + +``` +helm template hub-net charts/hub-net-controller-manager > /tmp/atm-default.yaml +diff .github/.copilot/breadcrumbs/baselines/hub-net-atm-default.yaml /tmp/atm-default.yaml + +helm template hub-net charts/hub-net-controller-manager \ + --set enableTrafficManagerFeature=true \ + --set azureCloudConfig.tenantId=00000000-0000-0000-0000-000000000000 \ + --set azureCloudConfig.subscriptionId=11111111-1111-1111-1111-111111111111 \ + --set azureCloudConfig.useManagedIdentityExtension=true \ + --set azureCloudConfig.userAssignedIdentityID=22222222-2222-2222-2222-222222222222 \ + --set azureCloudConfig.resourceGroup=rg-fleet \ + --set azureCloudConfig.location=eastus \ + > /tmp/atm-enabled.yaml +diff .github/.copilot/breadcrumbs/baselines/hub-net-atm-enabled.yaml /tmp/atm-enabled.yaml +``` + +Both diffs MUST be empty. Any output indicates a violation of D9's compatibility contract. diff --git a/.github/.copilot/breadcrumbs/baselines/hub-net-atm-default.yaml b/.github/.copilot/breadcrumbs/baselines/hub-net-atm-default.yaml new file mode 100644 index 00000000..9102ddb1 --- /dev/null +++ b/.github/.copilot/breadcrumbs/baselines/hub-net-atm-default.yaml @@ -0,0 +1,239 @@ +--- +# Source: hub-net-controller-manager/templates/serviceaccount.yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: hub-net-hub-net-controller-manager-sa + namespace: fleet-system + labels: + helm.sh/chart: hub-net-controller-manager-0.1.0 + app.kubernetes.io/name: hub-net-controller-manager + app.kubernetes.io/instance: hub-net + app.kubernetes.io/version: "v0.1.0" + app.kubernetes.io/managed-by: Helm + +--- +# Source: hub-net-controller-manager/templates/rbac.yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + creationTimestamp: null + name: hub-net-hub-net-controller-manager-role +rules: +- apiGroups: + - coordination.k8s.io + resources: + - leases + verbs: + - create + - get + - list + - update +- apiGroups: + - "" + resources: + - events + verbs: + - create + - get + - list + - update + - watch + - patch +- apiGroups: + - discovery.k8s.io + resources: + - endpointslices + verbs: + - create + - delete + - get + - patch + - update + - list + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - endpointsliceexports + verbs: + - create + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - endpointsliceimports + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceexports + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceexports/finalizers + verbs: + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceexports/status + verbs: + - get + - patch + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceimports + verbs: + - get + - list + - watch + - update + - patch +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceimports/status + verbs: + - get + - patch + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - serviceimports + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - serviceimports/finalizers + verbs: + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - serviceimports/status + verbs: + - get + - patch + - update +- apiGroups: + - cluster.kubernetes-fleet.io + resources: + - memberclusters + verbs: + - get + - list + - watch +- apiGroups: + - apiextensions.k8s.io + resources: + - customresourcedefinitions + verbs: + - create + - get + - list + - patch + - update + - watch +--- +# Source: hub-net-controller-manager/templates/rbac.yaml +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: hub-net-hub-net-controller-manager-role-binding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: hub-net-hub-net-controller-manager-role +subjects: + - kind: ServiceAccount + name: hub-net-hub-net-controller-manager-sa + namespace: fleet-system + +--- +# Source: hub-net-controller-manager/templates/deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: hub-net-hub-net-controller-manager + namespace: fleet-system + labels: + helm.sh/chart: hub-net-controller-manager-0.1.0 + app.kubernetes.io/name: hub-net-controller-manager + app.kubernetes.io/instance: hub-net + app.kubernetes.io/version: "v0.1.0" + app.kubernetes.io/managed-by: Helm +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: hub-net-controller-manager + app.kubernetes.io/instance: hub-net + template: + metadata: + labels: + app.kubernetes.io/name: hub-net-controller-manager + app.kubernetes.io/instance: hub-net + spec: + serviceAccountName: hub-net-hub-net-controller-manager-sa + containers: + - name: hub-net-controller-manager + image: "ghcr.io/azure/fleet-networking/hub-net-controller-manager:v0.1.0" + imagePullPolicy: Always + args: + - --leader-election-namespace=fleet-system + - --v=2 + - --add_dir_header + - --force-delete-wait-time=2m0s + - --enable-traffic-manager-feature=false + ports: + - name: metrics + containerPort: 8080 + protocol: TCP + - name: healthz + containerPort: 8081 + protocol: TCP + livenessProbe: + httpGet: + path: /healthz + port: healthz + readinessProbe: + httpGet: + path: /readyz + port: healthz + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 128Mi diff --git a/.github/.copilot/breadcrumbs/baselines/hub-net-atm-enabled.yaml b/.github/.copilot/breadcrumbs/baselines/hub-net-atm-enabled.yaml new file mode 100644 index 00000000..747f0fd1 --- /dev/null +++ b/.github/.copilot/breadcrumbs/baselines/hub-net-atm-enabled.yaml @@ -0,0 +1,311 @@ +--- +# Source: hub-net-controller-manager/templates/serviceaccount.yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: hub-net-hub-net-controller-manager-sa + namespace: fleet-system + labels: + helm.sh/chart: hub-net-controller-manager-0.1.0 + app.kubernetes.io/name: hub-net-controller-manager + app.kubernetes.io/instance: hub-net + app.kubernetes.io/version: "v0.1.0" + app.kubernetes.io/managed-by: Helm + +--- +# Source: hub-net-controller-manager/templates/azurecloudconfig.yaml +apiVersion: v1 +kind: Secret +metadata: + name: azure-cloud-config + namespace: fleet-system +type: Opaque +data: + azure.json: "ICAgIHsiYWFkQ2xpZW50SWQiOiIiLCJhYWRDbGllbnRTZWNyZXQiOiIiLCJjbG91ZCI6IkF6dXJlUHVibGljQ2xvdWQiLCJsb2NhdGlvbiI6ImVhc3R1cyIsInJlc291cmNlR3JvdXAiOiJyZy1mbGVldCIsInN1YnNjcmlwdGlvbklkIjoiMTExMTExMTEtMTExMS0xMTExLTExMTEtMTExMTExMTExMTExIiwidGVuYW50SWQiOiIwMDAwMDAwMC0wMDAwLTAwMDAtMDAwMC0wMDAwMDAwMDAwMDAiLCJ1c2VNYW5hZ2VkSWRlbnRpdHlFeHRlbnNpb24iOnRydWUsInVzZXJBZ2VudCI6IiIsInVzZXJBc3NpZ25lZElkZW50aXR5SUQiOiIyMjIyMjIyMi0yMjIyLTIyMjItMjIyMi0yMjIyMjIyMjIyMjIifQ==" + +--- +# Source: hub-net-controller-manager/templates/rbac.yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + creationTimestamp: null + name: hub-net-hub-net-controller-manager-role +rules: +- apiGroups: + - coordination.k8s.io + resources: + - leases + verbs: + - create + - get + - list + - update +- apiGroups: + - "" + resources: + - events + verbs: + - create + - get + - list + - update + - watch + - patch +- apiGroups: + - discovery.k8s.io + resources: + - endpointslices + verbs: + - create + - delete + - get + - patch + - update + - list + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - endpointsliceexports + verbs: + - create + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - endpointsliceimports + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceexports + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceexports/finalizers + verbs: + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceexports/status + verbs: + - get + - patch + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceimports + verbs: + - get + - list + - watch + - update + - patch +- apiGroups: + - networking.fleet.azure.com + resources: + - internalserviceimports/status + verbs: + - get + - patch + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - serviceimports + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - serviceimports/finalizers + verbs: + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - serviceimports/status + verbs: + - get + - patch + - update +- apiGroups: + - cluster.kubernetes-fleet.io + resources: + - memberclusters + verbs: + - get + - list + - watch +- apiGroups: + - apiextensions.k8s.io + resources: + - customresourcedefinitions + verbs: + - create + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - trafficmanagerbackends + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - trafficmanagerbackends/finalizers + verbs: + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - trafficmanagerbackends/status + verbs: + - get + - patch + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - trafficmanagerprofiles + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - networking.fleet.azure.com + resources: + - trafficmanagerprofiles/finalizers + verbs: + - update +- apiGroups: + - networking.fleet.azure.com + resources: + - trafficmanagerprofiles/status + verbs: + - get + - patch + - update +--- +# Source: hub-net-controller-manager/templates/rbac.yaml +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: hub-net-hub-net-controller-manager-role-binding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: hub-net-hub-net-controller-manager-role +subjects: + - kind: ServiceAccount + name: hub-net-hub-net-controller-manager-sa + namespace: fleet-system + +--- +# Source: hub-net-controller-manager/templates/deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: hub-net-hub-net-controller-manager + namespace: fleet-system + labels: + helm.sh/chart: hub-net-controller-manager-0.1.0 + app.kubernetes.io/name: hub-net-controller-manager + app.kubernetes.io/instance: hub-net + app.kubernetes.io/version: "v0.1.0" + app.kubernetes.io/managed-by: Helm +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: hub-net-controller-manager + app.kubernetes.io/instance: hub-net + template: + metadata: + labels: + app.kubernetes.io/name: hub-net-controller-manager + app.kubernetes.io/instance: hub-net + spec: + serviceAccountName: hub-net-hub-net-controller-manager-sa + containers: + - name: hub-net-controller-manager + image: "ghcr.io/azure/fleet-networking/hub-net-controller-manager:v0.1.0" + imagePullPolicy: Always + args: + - --leader-election-namespace=fleet-system + - --v=2 + - --add_dir_header + - --force-delete-wait-time=2m0s + - --enable-traffic-manager-feature=true + - --cloud-config=/etc/kubernetes/provider/azure.json + ports: + - name: metrics + containerPort: 8080 + protocol: TCP + - name: healthz + containerPort: 8081 + protocol: TCP + livenessProbe: + httpGet: + path: /healthz + port: healthz + readinessProbe: + httpGet: + path: /readyz + port: healthz + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 128Mi + volumeMounts: + - name: cloud-provider-config + mountPath: /etc/kubernetes/provider + readOnly: true + volumes: + - name: cloud-provider-config + secret: + secretName: azure-cloud-config diff --git a/.github/workflows/markdown.links.config.json b/.github/workflows/markdown.links.config.json index 420bca02..9f815018 100644 --- a/.github/workflows/markdown.links.config.json +++ b/.github/workflows/markdown.links.config.json @@ -9,6 +9,9 @@ }, { "pattern": "https://aka.ms/spot" + }, + { + "pattern": "^https://eng\\.ms/" } ], "timeout": "20s" diff --git a/docs/first-party/001-afd-global-load-balancing.md b/docs/first-party/001-afd-global-load-balancing.md new file mode 100644 index 00000000..cbdbe802 --- /dev/null +++ b/docs/first-party/001-afd-global-load-balancing.md @@ -0,0 +1,910 @@ +# Proposal 001 — Azure Front Door + WAF + Private Link Global Load Balancing + +| Field | Value | +|-------------|-----------------------------------------------------| +| Status | Draft | +| Author | @rchinchani_microsoft | +| Created | 2026-07-15 | +| Depends on | SFI-NS253 | +| Supersedes | — | + +## 1. Summary + +> **Reader's note.** This proposal reflects the state of the branch as +> of commit `cb02d14` (2026-07-19). Passages tagged `POC:` describe +> what is already in `main`; passages tagged **POC deviation** or +> **Impossibility flag** describe gaps between the shipped code and +> the target design. The reconciliation pass is recorded in +> `.github/.copilot/breadcrumbs/2026-07-20-1108-afd-export-mode-mcs-parity.md` +> (Addendum 2). Proposal 003 §6 tracks the outstanding blockers. +> +> **Status update (2026-07-20 session, head `629644b`).** Several +> POC callouts below are now resolved. In particular: +> +> - `FrontDoorProfile.spec.wafPolicy` + `spec.complianceMode` shipped +> with SecurityPolicy attach + envtest coverage (§4.1.1, §5.1); +> see commits `aeb116c` (API), `b2cf58f` (reconciler), +> `4b85044` (tests). +> - `FrontDoorBackend` v1alpha1 CRD + reconciler shipped (§4.1.2, +> §5.1) including OriginGroup + Origins programming, the AFD/ATM +> coexistence guard from §3.5 (`Conflict` reason), and full +> envtest coverage; see commits `c61dc43` (CRD), `be0ccb2` +> (reconciler), `ceac0ab` (coexistence guard), `629644b` (tests). +> - `ServiceExport` export-mode annotation + +> `InternalServiceExport.Spec.ExportMode` + +> `InternalServiceExport.Spec.PrivateLinkServiceResourceID` shipped +> (§3.5, §4.2); the member `serviceexport` reconciler now looks up +> the per-Service PLS by ARM Get and propagates it via +> `InternalServiceExport`; see commits `b85a115`, `c6d0d8e`, +> `cb6f23a`, `cf5324b`. +> - `pkg/common/azurefrontdoor` client bundle now covers +> `WAFPoliciesClient`, `SecurityPoliciesClient`, +> `AFDOriginGroupsClient`, and `AFDOriginsClient` (§5.3); see +> commits `95a0096` and `afe12d7`. +> - The sibling `cmd/hub-afd-controller-manager` binary and +> `charts/hub-afd-controller-manager` chart from §6/§7 landed in +> prior sessions (`fcb37f2`, `5672313`, `c219ca2`); the POC bridge +> in `cmd/hub-net-controller-manager` remains for now. +> +> Individual `POC:` / `POC deviation` blocks inline below have NOT +> been rewritten — they remain accurate to `cb02d14`. Trust this +> summary block for the current status; see Addendum 3 in the +> breadcrumb for the per-commit narrative. + +Add a new global load-balancing (GLB) data-plane option to +fleet-networking based on **Azure Front Door (AFD) Standard / Premium** +with an attached **WAF policy** and **Azure Private Link** to the +member-cluster origins. This new option runs **in parallel** with the +existing Azure Traffic Manager (ATM) implementation; ATM remains +supported for third-party / non-SFI workloads and for L4 / non-HTTP +scenarios. + +The proposal introduces two new CRDs (`FrontDoorProfile`, +`FrontDoorBackend`) that mirror the existing `TrafficManagerProfile` / +`TrafficManagerBackend` pair, an additive extension to +`ServiceExport` / `InternalServiceExport` for reporting the +Private-Link-Service (PLS) resource ID of each member endpoint, one new +member controller that provisions the per-cluster PLS, and two new hub +controllers that reconcile AFD profiles, endpoints, origin groups, +origins, routes, and security policies via the `armcdn` SDK. + +## 2. Motivation + +### Competitive context (industry parity) + +Peer clouds already ship a first-party controller that programs a +global L7 edge from a Kubernetes-native surface, with WAF/bot rules +that travel with the workload manifest instead of a side-car IaC +pipeline. Today AKS/Fleet does not: + +| Cloud Provider | Controller Engine | Edge Routing Layer | Native Bot/WAF Security Product | Native Bot/WAF Hook? | Layer-7 Controller Maturity | +| :---------------- | :-------------------------------- | :------------------------------------------------- | :----------------------------------------------------------- | :----------------------------------------- | :--------------------------------------------- | +| **GCP (GKE)** | Multi-Cluster Gateway Controller | Global External Application Load Balancer | **Google Cloud Armor** (with reCAPTCHA Enterprise) | **Yes** (via `GCPBackendPolicy`) | Highly mature, production-standard | +| **AWS (EKS)** | AWS Load Balancer Controller | Application Load Balancer (ALB) & VPC Lattice | **AWS WAF** (with AWS Managed Rules Bot Control) | **Yes** (via resource annotations) | Gateway API implementation reached GA in early 2026 | +| **Azure (Fleet)** | Azure Kubernetes Fleet Manager | **Azure Front Door (Premium)** — this proposal | **Azure Web Application Firewall** (with Bot Manager Rule Set) | **No today**; must manage AFD outside K8s | **ATM only (L4 DNS)**; L7 requires Terraform/ASO | + +The AKS/Fleet row is what this proposal changes. `FrontDoorProfile` +carries the WAF hook via `.spec.wafPolicy` (§4.1.1) and +`FrontDoorBackend` (§4.1.2) makes per-workload origin programming a +CR mutation instead of a Terraform plan — bringing AKS/Fleet to +functional parity with `GCPBackendPolicy` on GKE and the WAF +annotations on the AWS Load Balancer Controller, with the added +SFI-NS253 guarantee that origins are reached exclusively over +Private Link (§2.1). It also unblocks the migration path away from +the L4-only Traffic Manager surface documented in §2.2 without +forcing the workload owner to leave Kubernetes YAML. + +### 2.1 SFI-NS253 in one paragraph + +Any first-party Microsoft service exposed to the internet must: + +* terminate ingress on an Azure-managed edge (**AFD Premium** — see + §2.3, Standard does not support Private Link origins), +* have a WAF policy in **Prevention** mode attached to that edge, and +* reach origins **exclusively over Private Link** — the origin must not + expose a public IP. + +**Traffic-isolation property.** With AFD Premium + Private Link +Service to the AKS internal load balancer, the request path is: + +``` +client ──▶ AFD PoP (TLS termination) ──▶ Microsoft backbone + ──▶ Private Endpoint in member VNet ──▶ AKS ILB ──▶ pod +``` + +No leg of that path traverses the public internet between AFD and the +origin. The AKS ingress therefore has **no public IP**, which is +what SFI-NS253 fundamentally requires. This isolation is per-flow: +each `FrontDoorBackend` binds a specific AFD origin to a specific +PLS to a specific ILB to a specific `Service`, and there is no +shared route table between different profiles. + +See . + +### 2.2 Current state of `Azure/fleet-networking` + +Confirmed by inspection at branch `main`: + +* CRDs + * `api/v1alpha1/trafficmanagerprofile_types.go` + * `api/v1alpha1/trafficmanagerbackend_types.go` + * `api/v1beta1/trafficmanagerprofile_types.go` + * `api/v1beta1/trafficmanagerbackend_types.go` +* Controllers + * `pkg/controllers/hub/trafficmanagerprofile/` + * `pkg/controllers/hub/trafficmanagerbackend/` +* SDK wiring + * `cmd/hub-net-controller-manager/main.go` — only + `armtrafficmanager.ProfilesClient` and + `armtrafficmanager.EndpointsClient` are constructed (`initAzureTrafficManagerClients`). +* Feature flags + * `--enable-traffic-manager-feature` on both hub and member managers + (default `true`). +* Search for `FrontDoor`, `frontdoor`, `AFD`, `WAF`, `armcdn`, or + `PrivateLinkService` inside the repository returns zero matches. + +Therefore ATM is today the **only** GLB surface, and it is +**structurally incompatible with SFI-NS253**: + +| SFI-NS253 requirement | Traffic Manager | Front Door | +|-----------------------------|:---------------:|:----------:| +| Edge TLS termination | ❌ (DNS only) | ✅ | +| WAF attach | ❌ | ✅ | +| Private Link to origin | ❌ (public IP) | ✅ | +| L7 routing (paths/headers) | ❌ | ✅ | + +### 2.3 Non-goals + +* **L4 workloads (non-HTTP/HTTPS).** AFD is an L7 reverse proxy — + it only serves HTTP, HTTPS, and WebSockets-over-HTTPS. Arbitrary + TCP / UDP first-party workloads (databases, gRPC-over-plain-TCP, + SMTP, DNS, etc.) are **not covered** by this proposal and cannot + satisfy SFI-NS253 via AFD. The likely future counterpart for L4 + is Azure Cross-region Load Balancer (anycast, Private Link + backends), tracked as a follow-up and not proposed here. +* Removing or deprecating ATM — the two features coexist behind + independent feature flags. +* Building a generic ingress controller inside a member cluster. We + reuse the Azure cloud-provider Service annotations for internal load + balancer + PLS creation, and rely on the AKS-managed cloud provider + to program them. +* Managing member-cluster provisioning (VNet layout, subnet + `privateLinkServiceNetworkPolicies`, cluster SKU choice between + AKS Standard and AKS Automatic). These are platform-team concerns; + see §3.4 for the invariants a member cluster MUST satisfy. +* Supporting AFD **classic** or AFD **Standard**. Only AFD + **Premium** (`Microsoft.Cdn` resource provider, API surface + `armcdn`, SKU `Premium_AzureFrontDoor`) is in scope. **Private + Link origins are a Premium-only feature** — Standard cannot + satisfy SFI-NS253's private-origin requirement, and classic does + not support Private Link at all. The CEL rule on + `FrontDoorProfile.spec.sku` enforces Premium for any profile whose + backends require Private Link. + +## 3. User-facing shape + +### 3.1 Author’s mental model + +For a customer service `contoso` that wants to be reachable at +`contoso.first-party.example.com`: + +```yaml +# In the hub cluster, in namespace "contoso". +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: FrontDoorProfile +metadata: + name: contoso +spec: + resourceGroup: fleet-frontdoor-rg + sku: Premium_AzureFrontDoor # required for Private Link + wafPolicy: + resourceID: /subscriptions/…/providers/Microsoft.Network/frontdoorWebApplicationFirewallPolicies/contoso-waf + healthProbe: + path: /healthz + protocol: Https + intervalInSeconds: 30 + originResponseTimeoutSeconds: 60 +--- +apiVersion: networking.fleet.azure.com/v1alpha1 +kind: FrontDoorBackend +metadata: + name: contoso + namespace: contoso +spec: + profile: + name: contoso + backend: + name: contoso # a ServiceImport in the same namespace + weight: 100 + routing: + patternsToMatch: ["/*"] + forwardingProtocol: HttpsOnly + supportedProtocols: [Https] + linkToDefaultDomain: Enabled + privateLink: + enabled: true # required for SFI-NS253 + requestMessage: "fleet-networking auto-approve" +``` + +### 3.2 What the controllers create in Azure + +For the example above, the reconciler ensures: + +1. An **AFD profile** named `fleet-` (SKU + `Premium_AzureFrontDoor`) in `fleet-frontdoor-rg`. Azure resource + names are derived from the `FrontDoorProfile` CR's Kubernetes UID, + not from `metadata.name`, so a CR rename does not orphan the + underlying Azure resource. The `fleet-` prefix keeps the composed + endpoint hostname (`fleet--.z01.azurefd.net`) safely + under AFD's 46-character endpoint-name cap. +2. An **AFD default endpoint** (also named `fleet-`) whose + hostname is surfaced back in `status.endpointHostname`. +3. One **origin group** per `FrontDoorBackend`, with the health probe + copied from the profile. +4. One **AFD origin** per (member cluster × exported service) tuple. + Each origin references the **PLS resource ID** reported by the + member cluster in `InternalServiceExport.status.privateLinkService.resourceID`. +5. One **route** binding endpoint → origin group with the requested + path patterns / protocol. +6. One **security policy** binding the endpoint domain(s) to the + referenced WAF policy. + +All traffic from step 2 onward stays on the Microsoft backbone: +AFD terminates TLS at its edge PoP, then reaches the origin over +the AFD → PLS → ILB private path. The public IP on the member-cluster +`Service` is never provisioned, satisfying SFI-NS253's private-origin +requirement. + +### 3.3 What the member controller observes in each cluster + +The proposal **does not add a new `Spec` field to `ServiceExport`**. +Upstream mcs-api (KEP-1645) parity is a repository preference, and +Fleet-specific configuration is already carried via annotations +(e.g. `networking.fleet.azure.com/weight`). This proposal follows +the same pattern. + +The member controller determines that an exported `Service` is +destined for the AFD path using two signals, evaluated in this +order: + +1. **Opt-in annotation on `ServiceExport`** (primary when set): + ``` + networking.fleet.azure.com/export-mode: L7-FrontDoor # or L4-TrafficManager (default) + ``` + Suitable for GitOps pipelines that want the `ServiceExport` + to declare intent before the `Service` is fully provisioned. +2. **Inference from the `Service` itself** (fallback): if the + annotation is unset, the controller inspects the exported + `Service` and infers `L7-FrontDoor` when **all** of the + following AKS cloud-provider annotations are present + (documented at + and ): + + ``` + service.beta.kubernetes.io/azure-load-balancer-internal: "true" + service.beta.kubernetes.io/azure-pls-create: "true" + service.beta.kubernetes.io/azure-pls-name: + service.beta.kubernetes.io/azure-pls-ip-configuration-subnet: + service.beta.kubernetes.io/azure-pls-visibility: "*" # or a comma-separated allow-list + service.beta.kubernetes.io/azure-pls-auto-approval: "" + ``` + + Otherwise the export is treated as `L4-TrafficManager` (today's + default behaviour). + +**Ownership boundary — the member controller does not mutate the +`Service`.** The Service's annotations are tenant-owned (typically +authored by the app team via GitOps, or enforced centrally by a +platform admission policy such as Kyverno / Gatekeeper). The +controller only *reads* them. + +**Precedence and mismatch handling.** When the annotation says +`L7-FrontDoor` but the underlying `Service` is not internal + +PLS-enabled, the controller does not fall back to L4 — that would +silently downgrade an SFI intent. Instead it surfaces +`ServiceExportValid=False` with +`Reason=ExportModeAnnotationServiceMismatch` and waits. + +Once AKS programs the PLS, the controller copies the resulting PLS +resource ID (surfaced by the cloud provider as +`service.beta.kubernetes.io/azure-pls-resource-id`) into +`InternalServiceExport.status.privateLinkService`. The hub +AFD-backend controller watches that field and creates / updates the +corresponding AFD origin. + +### 3.4 Member cluster requirements (AKS Standard and AKS Automatic) + +Both **AKS Standard** and **AKS Automatic** are supported as member +cluster SKUs. The controller code paths are identical because the +data-plane primitives this proposal relies on — Standard SKU internal +Load Balancer + Private Link Service, driven by +`service.beta.kubernetes.io/azure-*` annotations — are provided by +the AKS-managed cloud provider and are available on both SKUs. + +The following cluster-side prerequisites apply regardless of SKU and +are the responsibility of the platform team, not the fleet-networking +controllers: + +1. **Standard SKU Load Balancer.** Required by PLS. This is the + default on both AKS Standard and AKS Automatic; Basic LB clusters + are unsupported. +2. **BYO VNet with a dedicated PLS NAT subnet.** The subnet + referenced by + `service.beta.kubernetes.io/azure-pls-ip-configuration-subnet` + MUST have `privateLinkServiceNetworkPolicies: Disabled`. This is + a subnet-level property that must be set at (or before) cluster + provisioning — AKS does not toggle it on the tenant's behalf. + AKS Automatic supports BYO VNet at cluster creation but restricts + post-hoc network reshaping; plan the ILB and PLS subnets up front. +3. **PLS auto-approval configured for the AFD subscription.** The + `service.beta.kubernetes.io/azure-pls-auto-approval` annotation + MUST include the AFD control-plane subscription ID so that AFD's + private-endpoint connection requests are approved without human + intervention. +4. **Egress path.** Not affected by this proposal — AKS Automatic's + NAT-Gateway egress is orthogonal to ingress via PLS. + +**AKS Automatic — additional considerations:** + +* **Deployment Safeguards (Enforcement mode).** AKS Automatic ships + Azure Policy safeguards in enforcement mode by default. The + fleet-networking hub and member Helm charts (see Proposal 002 + §6.1) MUST satisfy those safeguards — resource requests/limits, + `runAsNonRoot`, `readOnlyRootFilesystem` where feasible, no + `hostPath`, images from allow-listed registries, `seccomp: + RuntimeDefault`, no privileged containers. Any drift here blocks + install on Automatic even though it succeeds on Standard. +* **Node auto-provisioning (NAP).** Controller Deployments should + set pod anti-affinity / PDBs so that NAP-driven scale events do + not simultaneously restart the active reconciler replicas. +* **Locked-down cluster configuration.** Some `az aks update` knobs + are not permitted on Automatic. All state this proposal touches + is user-surface (`Service`, `ServiceExport`, `FrontDoor*`), not + cluster-config surface, so this is a non-issue for the data path + — but bear it in mind when writing runbooks that assume a Standard + cluster's mutability. +* **AGC (Application Gateway for Containers) coexistence.** AKS + Automatic promotes AGC as the default HTTP entry point. AGC and + the AFD + PLS path proposed here are orthogonal — AGC is an + in-cluster L7, AFD is an external edge — and can coexist. This + proposal does not require or interact with AGC. + +Testing note: Phase 4 e2e (Proposal 002 §11) MUST include at least +one AKS Automatic member alongside AKS Standard members, so the +Deployment Safeguards path is exercised in CI rather than discovered +at first-adopter onboarding. + +### 3.5 Coexistence with the existing Traffic Manager path + +ATM is not deprecated by this proposal (§2.3). Both surfaces are +first-class and can run side-by-side in the same fleet. Where they +interact: + +**Coexistence granularity.** + +| Scope | Coexistence outcome | +|---|---| +| Fleet-wide | Safe. Different CRDs (`TrafficManager*` vs `FrontDoor*`), different hub controllers, different Azure resource types (`Microsoft.Network/trafficManagerProfiles` vs `Microsoft.Cdn/profiles`), different identities (§7), different resource groups. No shared reconciler state. | +| Same tenant namespace, different Services | Safe. `Service A` fronted by ATM (public LB) and `Service B` fronted by AFD (internal LB + PLS) is a supported topology. | +| Same `ServiceImport` on both surfaces | **Forbidden by the AFD backend reconciler.** The two paths require mutually exclusive `Service` shapes (public LB for ATM, internal LB + PLS for AFD). See Proposal 002 §8.4 — the reconciler surfaces `Accepted=False, Reason=ConflictsWithTrafficManagerBackend` and refuses to program AFD origins for a `ServiceImport` already referenced by a `TrafficManagerBackend`. Invariant: at most one north-south surface per `Service`. | + +East-west traffic (pod-to-pod via `EndpointSlice` imports) is +unaffected on either path. + +**Security implications of coexistence.** + +* **ATM public IPs remain a bypass surface.** ATM is DNS-only; its + origins have public IPs, so clients that discover those IPs can + connect directly, bypassing any WAF or rate-limiting. This is + unchanged by AFD's arrival. Fleets that run both must not + characterise the fleet as SFI-compliant simply because AFD is + available — compliance is per-tenant, per-Service. +* **Split identities are mandatory.** The AFD managed identity + (`CDN Profile Contributor` on the AFD resource group) MUST be + distinct from the ATM identity (`Traffic Manager Contributor` on + the TM resource group). A single identity for both would grant + ATM-only tenants unnecessary AFD write permissions and vice-versa, + silently expanding blast radius. Chart wiring is tracked in + Proposal 003 §3.6. +* **WAF is per-surface.** ATM has no WAF. Enforcement of "must be + behind WAF" is only achievable for `FrontDoor*`-fronted Services; + cluster-level admission policy (Kyverno / OPA / Gatekeeper) should + deny `TrafficManagerProfile` / `TrafficManagerBackend` creation in + first-party namespaces if a tenant class is required to be + AFD-only. Do not rely on tenants opting out voluntarily. +* **Audit / diagnostic split.** ATM and AFD emit to separate + diagnostic streams. SFI KPI dashboards that count WAF-blocked + requests must query AFD only; ATM has no such concept. + +**Tenancy implications of coexistence.** + +* **Namespace RBAC is unchanged.** Tenant-A cannot see or modify + tenant-B's `TrafficManager*` or `FrontDoor*` CRs. Reserved + `fleet-member-*` namespaces on the hub remain platform-only. +* **Per-Service surface choice.** With the annotation + inference + model (§3.3, §4.2), a single tenant namespace may host a mix of + Services on ATM and AFD without any `Spec`-level opt-in — the + Service annotations themselves drive the routing choice. +* **Weight-annotation semantics differ.** The Fleet-specific + `networking.fleet.azure.com/weight` annotation on `ServiceExport` + is consumed by the ATM backend controller. The AFD backend + controller ignores it and takes weights from + `FrontDoorBackend.spec.weight` instead (see Proposal 002 §8.1). + Migration howtos MUST call this out. +* **Quota accounting is per-surface.** ATM profile limits (200/sub + default) and AFD Premium profile/endpoint/origin quotas are + independent. Fleets running both surfaces at scale need + per-tenant subscription / RG sharding informed by both quotas. +* **Cost attribution.** Different cost models (ATM: per million DNS + queries; AFD Premium: base fee + per-request + WAF). Use + per-tenant `resourceGroup` on both `TrafficManagerProfile` and + `FrontDoorProfile` so Azure Cost Management can split the bill + cleanly. + +**Migration (ATM → AFD) is a staged tenant operation, not a hot swap.** +Because the same `ServiceImport` cannot attach to both surfaces: + +1. Stand up a second `Service` (e.g. `api-v2`) in each member cluster + with internal LB + PLS annotations; create the matching + `ServiceExport api-v2`. +2. Hub aggregates a new `ServiceImport api-v2`; create + `FrontDoorProfile` + `FrontDoorBackend` pointing at it and verify + `Accepted=True` on every origin. +3. Cut the public DNS record from `.trafficmanager.net` to + the AFD endpoint hostname; drain traffic per your TTL. +4. Delete `TrafficManagerBackend api` and the original public-LB + `Service api` in each member cluster. + +This intentionally slow, reviewable sequence matches first-party +migration cadence; a controller-driven hot swap is not planned. + +## 4. API changes + +### 4.1 New CRDs + +Both new types land first in `api/v1alpha1` (matching how +`TrafficManager*` graduated) and are promoted to `v1beta1` after +integration coverage is in place. + +#### 4.1.1 `FrontDoorProfile` (shortName `afdp`) + +Package: `api/v1alpha1/frontdoorprofile_types.go`. + +> **POC status (commit `cb02d14`).** The Phase-2 POC currently ships a +> deliberately minimal `Spec` (`ResourceGroup` + `Sku` only) and no +> WAF / compliance / health-probe fields yet. The illustrative Go +> below is the *target* shape; fields marked `POC:` are the ones +> already in `main`, everything else is Phase-4/5 work. See §6 of +> Proposal 002 for the phase table. + +Key fields (illustrative Go, not final): + +```go +type FrontDoorProfileSpec struct { + // POC: present in cb02d14. + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:MaxLength=90 + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="resourceGroup is immutable" + ResourceGroup string `json:"resourceGroup"` + + // Sku selects the AFD SKU. The CRD enum accepts only Premium + // (`Premium_AzureFrontDoor`); Standard is intentionally excluded + // because Private Link origins — the SFI-NS253 cornerstone — are + // Premium-only, so a Standard profile could never satisfy the + // first-party compliance envelope. Failing at admission is + // preferable to surfacing `Programmed=False` hours later. + // The field is retained (rather than removed as redundant) so + // future SKUs can be added additively without a schema break. + // Immutable after creation: AFD does not support in-place SKU + // upgrades on an existing profile. + // +kubebuilder:validation:Enum=Premium_AzureFrontDoor + // +kubebuilder:default=Premium_AzureFrontDoor + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="sku is immutable" + SKU FrontDoorSKU `json:"sku,omitempty"` + + // Post-POC: optional attach of a WAF policy. Required when + // complianceMode == SFI-NS253 (see Proposal 003 §1.2). + // +optional + WAFPolicy *FrontDoorWAFPolicyRef `json:"wafPolicy,omitempty"` + + // Post-POC. + // +optional + HealthProbe *FrontDoorHealthProbe `json:"healthProbe,omitempty"` + + // Post-POC. + // +optional + // +kubebuilder:validation:Minimum=16 + // +kubebuilder:validation:Maximum=240 + OriginResponseTimeoutSeconds *int32 `json:"originResponseTimeoutSeconds,omitempty"` +} + +type FrontDoorProfileStatus struct { + // POC: present in cb02d14. Full ARM resource ID of the AFD + // profile. + // +optional + ResourceID string `json:"resourceID,omitempty"` + + // POC: present in cb02d14. The default endpoint's *.azurefd.net + // hostname (string, not the full endpoint resource ID). + // +optional + EndpointHostname *string `json:"endpointHostname,omitempty"` + + // +optional + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` +} +``` + +Condition types (POC, per `cb02d14`): `Programmed` with reasons +`Programmed`, `Invalid`, `AzureError`, `Pending`. `WAFPolicyNotFound` +and `WAFPolicyNotInPreventionMode` land alongside the WAF fields in +a later phase. AFD is a global service, so **no `Location` field is +exposed on the CR**; the controller sets `Location: "Global"` +internally. + +There is no separate `EndpointResourceID` status field; the ARM ID +of the endpoint is deterministically composable from `ResourceID` + +the fixed endpoint name (`fleet-`). + +#### 4.1.2 `FrontDoorBackend` (shortName `afdb`) + +Package: `api/v1alpha1/frontdoorbackend_types.go`. + +> **POC status.** `FrontDoorBackend` is **not yet implemented** in +> `main` (cb02d14 only landed `FrontDoorProfile` + `FrontDoorCustomDomain`). +> This section describes the target shape; the type + controller +> land together in Phase 4 (see Proposal 002 §6). + +```go +type FrontDoorBackendSpec struct { + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.profile is immutable" + Profile FrontDoorProfileRef `json:"profile"` + + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.backend is immutable" + Backend FrontDoorBackendRef `json:"backend"` // references a ServiceImport + + // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=1000 + // +kubebuilder:default=100 + Weight *int32 `json:"weight,omitempty"` + + // Optional per-backend HTTP routing configuration. + // +optional + Routing *FrontDoorRoutingConfig `json:"routing,omitempty"` + + // PrivateLink controls whether AFD origins are wired via the PLS + // reported by the member cluster. MUST be `enabled: true` for + // SFI-NS253 workloads. + // +optional + PrivateLink *FrontDoorPrivateLinkConfig `json:"privateLink,omitempty"` +} +``` + +Status mirrors `TrafficManagerBackend.Status.Endpoints` but each entry +represents an AFD **origin** rather than an ATM endpoint, and includes +the PLS resource ID and the PLS connection approval state +(`Pending` / `Approved` / `Rejected` / `Disconnected`). + +Condition types: `Accepted` (reasons: `Accepted`, `Invalid`, `Pending`, +`PrivateLinkPending`, `PrivateLinkRejected`). + +#### 4.1.3 `FrontDoorCustomDomain` (shortName `afdcd`) + +Package: `api/v1alpha1/frontdoorcustomdomain_types.go`. **Present in +`main` as of cb02d14** (Managed TLS reconciled; BYOC deferred). + +`FrontDoorCustomDomain` represents a custom domain attached to a +`FrontDoorProfile`, including DNS-based ownership validation and the +TLS binding. Same-namespace-only reference to its parent profile +(immutable). Immutable `hostname`. + +Key fields (as shipped): + +```go +type FrontDoorCustomDomainSpec struct { + // Same-namespace ref to the owning FrontDoorProfile. Immutable. + ProfileRef FrontDoorProfileReference `json:"profileRef"` + + // Fully qualified custom domain, e.g. www.contoso.com. Immutable. + // +kubebuilder:validation:Pattern=`^([a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}$` + Hostname string `json:"hostname"` + + // TLS.Mode = Managed (AFD-issued cert, auto-renewed) or BYOC + // (Key Vault cert). BYOC field shape is present but the + // reconciler currently only implements Managed. + TLS FrontDoorTLSConfig `json:"tls"` +} + +type FrontDoorCustomDomainStatus struct { + ResourceID string `json:"resourceID,omitempty"` + ValidationState FrontDoorDomainValidationState `json:"validationState,omitempty"` + // The value to publish as TXT record at `_dnsauth.`. + DNSValidationToken *string `json:"dnsValidationToken,omitempty"` + DNSValidationExpiry *metav1.Time `json:"dnsValidationExpiry,omitempty"` + Conditions []metav1.Condition `json:"conditions,omitempty"` +} +``` + +Condition: `Programmed` with reasons `Programmed`, `Invalid`, +`ProfileNotReady`, `AwaitingDNSValidation`, `ValidationFailed`, +`TLSFailed`, `AzureError`, `Pending`. + +The tenant workflow is: apply the CR → controller creates the AFD +custom domain resource → status surfaces `DNSValidationToken` → the +tenant publishes a TXT record → AFD validates and `Programmed=True`. + +### 4.2 Additive changes to existing CRDs + +Only additive fields — no breaking changes. **`ServiceExport` itself +gains no new `Spec` field.** Per the mcs-api parity preference (see +§3.3), Fleet-specific intent is expressed via an annotation: + +``` +networking.fleet.azure.com/export-mode: L7-FrontDoor # optional; default is L4-TrafficManager (i.e., today's behaviour) +``` + +The annotation constant lives alongside the existing +`networking.fleet.azure.com/weight` constant under +`pkg/common/objectmeta/`. Absence of the annotation is equivalent to +`L4-TrafficManager`; the member controller may still infer +`L7-FrontDoor` from the Service's internal-LB + PLS annotations, per +the precedence rule in §3.3. + +* `api/v1alpha1/internalserviceexport_types.go` + * New optional `Status.PrivateLinkService` block: + + ```go + type ServiceExportPrivateLinkStatus struct { + // ResourceID of the Microsoft.Network/privateLinkServices resource. + ResourceID string `json:"resourceID"` + // Alias of the PLS (used when auto-approval is not in effect). + // +optional + Alias string `json:"alias,omitempty"` + // InternalLoadBalancerFrontendIP is the ILB frontend IP that + // fronts the PLS. + // +optional + InternalLoadBalancerFrontendIP string `json:"internalLoadBalancerFrontendIP,omitempty"` + } + ``` + +The mirror change lands in `api/v1beta1` after the v1alpha1 shape is +proven. + +### 4.3 CRD manifests + +`config/crd/bases/` grows three new files +(`networking.fleet.azure.com_frontdoorprofiles.yaml`, +`networking.fleet.azure.com_frontdoorcustomdomains.yaml`, and +`networking.fleet.azure.com_frontdoorbackends.yaml`) generated by +`make manifests`. The first two land in `main` as of cb02d14; the +third arrives with Phase 4. + +## 5. Controller changes + +### 5.1 New hub packages + +* `pkg/controllers/hub/frontdoorprofile/` — **shipped in cb02d14.** + Reconciles `Microsoft.Cdn/profiles` + the default `afdEndpoint`. + WAF `securityPolicies` binding lands with the WAF fields on the + CR (post-POC). +* `pkg/controllers/hub/frontdoorcustomdomain/` — **shipped in cb02d14.** + Reconciles `Microsoft.Cdn/profiles/customDomains`, surfaces the + DNS validation token, and (Managed TLS only for now) binds the + cert. BYOC (Key Vault) reconciliation is deferred. +* `pkg/controllers/hub/frontdoorbackend/` — **not yet in main; + Phase 4.** Reconciles `originGroups`, `origins`, and `routes` + under the referenced profile; watches `InternalServiceExport` for + changes to `status.privateLinkService.resourceID`; handles the AFD + private-endpoint approval workflow when auto-approval is not in + effect. + +All three packages follow the structural conventions of the existing +`trafficmanager*` packages: `controller.go`, `controller_test.go`, +`controller_integration_test.go`, `suite_test.go`, plus a shared fake +provider under `test/common/frontdoor/`. cb02d14 ships the +happy-path reconciler + finalizer for Profile and CustomDomain, but +**not** the unit/integration test scaffolding — that is tracked in +Proposal 002 §6 as remaining Phase-2 work. + +### 5.2 New / extended member packages + +* `pkg/controllers/member/serviceexport/` — **not yet extended in + main; Phase 3.** Will be extended to + (a) detect the AFD path via the annotation-then-inference rule + described in §3.3, and + (b) copy the AKS-cloud-provider-set annotation + `service.beta.kubernetes.io/azure-pls-resource-id` into + `InternalServiceExport.status.privateLinkService` once the PLS is + ready. The controller does **not** mutate the exported `Service`; + the internal-LB + PLS annotations are tenant-owned. + +### 5.3 SDK wiring + +The **target** wiring lives in a new binary +`cmd/hub-afd-controller-manager/main.go` (see §6), separate from +`cmd/hub-net-controller-manager/main.go`. The wiring adds: + +* `initAzureFrontDoorClients(cfg)` returning the AFD sub-clients + (`ProfilesClient`, `AFDEndpointsClient`, `AFDCustomDomainsClient`, + and — Phase 4 — `AFDOriginGroupsClient`, `AFDOriginsClient`, + `RoutesClient`, `SecurityPoliciesClient`). See + `pkg/common/azurefrontdoor/client.go` (shipped in cb02d14). +* A flag `--enable-frontdoor-feature` gating controller registration. + +**POC deviation (cb02d14):** the AFD controllers currently live +inside `cmd/hub-net-controller-manager/main.go` behind the same +`--enable-frontdoor-feature` flag (default `false`) as a temporary +bridge — the sibling binary+chart split is a hard prerequisite for +GA because of §7 (see the Impossibility flag there). + +`go.mod` gains a dependency on +`github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cdn/armcdn` +(pinned at `v1.1.1` in cb02d14). + +### 5.4 Common libraries + +* `pkg/common/azurefrontdoor/` — **shipped in cb02d14** as a single + `client.go`: Workload-Identity `Config` + `NewCredential` + + `NewClients` factory bundling `ProfilesClient`, `AFDEndpointsClient`, + `AFDCustomDomainsClient`. Additional sub-clients (origin, route, + security-policy) land alongside `FrontDoorBackend` in Phase 4. + Naming helpers (`fleet-`) live in each controller for now. +* `pkg/common/objectmeta/` — **shipped in cb02d14**: finalizer + constants `networking.fleet.azure.com/frontdoor-profile-cleanup` + and `.../frontdoor-custom-domain-cleanup`. +* `pkg/common/azureerrors/` — extended to classify AFD-specific + error codes (private-link approval races, WAF policy not found, + etc.). +* `pkg/common/defaulter/` — new defaulters for `FrontDoorProfile` + and `FrontDoorBackend` (Phase 4). +* Prometheus metrics analogous to the existing ATM ones, e.g. + `fleet_networking_frontdoor_profile_status_last_timestamp_seconds`. + +## 6. Deployment / charts + +The **target** deployment model is a **sibling chart + sibling +binary**, isolated from the existing hub-net-controller-manager +chart: + +* `charts/hub-afd-controller-manager/` (new) — RBAC on + `frontdoorprofiles`, `frontdoorcustomdomains`, `frontdoorbackends`; + its own ServiceAccount with its own Workload-Identity federated + subject; its own Deployment, values, and PDB. This isolation is + what makes the §7 identity-split requirement satisfiable end to + end (see §7). +* `charts/hub-net-controller-manager/` — **unchanged** for AFD. The + ATM controller retains its own SA and MI. The `helm template` + fixtures under `.github/.copilot/breadcrumbs/baselines/` protect + this chart from silent drift. +* `charts/member-net-controller-manager/` — update `--enable-traffic-manager-feature` + documentation to note it toggles ATM only. The member-side AFD + work (Phase 3) is a *reader* of Service annotations and does not + need Azure SDK access, so a member-side sibling chart is not + required; the flag `--enable-frontdoor-feature` is added to the + existing member chart. + +**POC deviation (cb02d14):** the sibling chart+binary do **not +exist yet**. The AFD controllers are hosted inside +`cmd/hub-net-controller-manager` under `--enable-frontdoor-feature` +(default `false`, preserving zero-diff `helm template` output for +ATM-only installs). The chart+binary split is a Phase-4/5 +prerequisite and is tracked in Proposal 003 §2.4. + +## 7. Security / SFI considerations + +* **Least privilege (identity split).** AFD reconciliation requires + the `CDN Profile Contributor` role (or a custom role that grants + `Microsoft.Cdn/*` under the AFD resource group). It MUST run under + a **different Azure identity** from the ATM controller — otherwise + ATM-only tenants inherit AFD write permissions they do not need. + Authentication uses **Azure AD Workload Identity** (a projected + federated token backed by the pod's Kubernetes ServiceAccount); + managed-identity-with-mounted-azure.json is not used. Because a + federated token is a **pod-level attribute** (the projected token + path is set on the pod, not the ServiceAccount alone), the only + way to have two Azure identities is to run two pods with two + distinct ServiceAccounts. This is why §6 mandates a sibling + binary+chart (`cmd/hub-afd-controller-manager` + + `charts/hub-afd-controller-manager`) for AFD. + + > **Impossibility flag on the current POC (cb02d14).** The POC + > hosts AFD and ATM controllers **in the same pod** + > (`hub-net-controller-manager` under `--enable-frontdoor-feature`), + > so today they necessarily share one Workload-Identity federated + > subject. This means the "separate identity" requirement above is + > **not** satisfied by cb02d14. The sibling binary+chart split + > (§6) is a hard GA prerequisite, tracked in Proposal 003 §2.4. + > Interim POC installs must use a WI subject bound to the *union* + > of the AFD and ATM roles, and MUST NOT be used for production + > SFI-NS253 workloads. +* **WAF mode.** For SFI-NS253 compliance the referenced WAF policy + MUST be in `Prevention` mode. The controller validates this at + admission time (via CEL on `FrontDoorProfile.spec.wafPolicy` if the + policy is inline; via a status condition + `WAFPolicyNotInPreventionMode` if referenced). +* **Public origin guard.** The `FrontDoorBackend` controller refuses + to create an AFD origin against an `InternalServiceExport` that has + no `status.privateLinkService.resourceID` when + `spec.privateLink.enabled = true`, and instead reports + `Accepted=False` with reason `PrivateLinkPending`. +* **Auto-approval subscription.** The PLS `auto-approval` annotation + on the member Service is populated from the AFD profile’s home + subscription so that AFD origin creation does not block on manual + approval. Cross-tenant deployments require manual approval; the + status surface makes that explicit. + +## 8. Testing strategy + +* **Unit tests.** New `*_test.go` next to each new source file. + Table-driven, in the style of the existing + `pkg/common/defaulter/trafficmanagerprofile_test.go`. +* **Integration tests.** New + `pkg/controllers/hub/frontdoorprofile/controller_integration_test.go` + and `.../frontdoorbackend/controller_integration_test.go`, using a + fake `armcdn` implementation under + `test/common/frontdoor/fakeprovider/`. +* **E2E tests.** New `test/e2e/frontdoor_test.go`, mirroring the + structure of `test/e2e/traffic_manager_test.go`. Requires an AFD + Premium profile per test suite plus one PLS-capable AKS pool per + member cluster. Gated behind an env var so the existing e2e + matrix does not become AFD-mandatory. +* **API validation tests.** Extend + `test/apis/v1alpha1/api_validation_integration_test.go` and the + v1beta1 counterpart. + +## 9. Migration and coexistence + +* ATM users are unaffected. All new CRDs and flags default to + disabled. +* A single `ServiceExport` cannot participate in both ATM and AFD + simultaneously; the `exportMode` field disambiguates. A user who + wants both must create two `ServiceExport` objects (matching two + Services), each with a different mode. +* Fleet-wide migration path: start with `exportMode: L4-TrafficManager` + (implicit), add `L7-FrontDoor` alongside once AFD is validated, + cut DNS over, then delete the ATM object. + +## 10. Rollout plan + +| Phase | Scope | Exit criteria | +|-------|-------|---------------| +| 0 | This proposal accepted, breadcrumb approved | Sign-off from fleet-networking maintainers + SFI reviewer | +| 1 | `api/v1alpha1` types + CRD manifests + defaulter/validation, no controller | `make manifests`, `go test ./api/... ./pkg/common/defaulter/...` green | +| 2 | Hub `frontdoorprofile` controller (no backend, no PLS) | Integration tests green; profile + endpoint + WAF attach observable in a dev sub | +| 3 | Member PLS provisioner + `InternalServiceExport` status field | Integration test proves PLS created and status reported | +| 4 | Hub `frontdoorbackend` controller (origins + routes + PL approval) | E2E test in `test/e2e/frontdoor_test.go` green in ci-e2e pipeline | +| 5 | `api/v1beta1` promotion + docs under `docs/concepts/HTTPBasedGlobalLoadBalancing/` and `docs/howtos/frontdoor-permissions-setup.md` | Feature marked GA in `README`; SFI-NS253 checklist attached | + +## 11. Open questions + +1. Should `FrontDoorProfile` be **cluster-scoped** (one profile shared + across namespaces, akin to a shared ingress) or **namespace-scoped** + (matching `TrafficManagerProfile`)? Current proposal: + namespace-scoped for isolation parity with ATM. +2. Do we support **multiple custom domains** per profile in the first + cut, or only the auto-generated `*.azurefd.net` hostname? Current + proposal: auto-generated only in phase 2; custom domains + managed + certificates in phase 5. +3. Should the WAF policy be **required** (validation error if absent) + for `Premium_AzureFrontDoor` SKU, given the SFI intent? Current + proposal: required at the SKU level. +4. Is there appetite to fold this into a common `GlobalLoadBalancer` + umbrella CRD later (with `type: TrafficManager | FrontDoor`) rather + than shipping parallel CRDs? Not proposed here — the two Azure + surfaces are too different structurally — but flagged for review. + +## 12. References + +* SFI-NS253 KPI — + +* Azure Front Door Standard/Premium overview — + +* AFD Private Link origins — + +* AKS internal load balancer — + +* AKS Private Link Service integration — + +* Existing ATM design in this repo — + [`docs/concepts/DNSBasedGlobalLoadBalancing/README.md`](../concepts/DNSBasedGlobalLoadBalancing/README.md) diff --git a/docs/first-party/002-afd-implementation-plan.md b/docs/first-party/002-afd-implementation-plan.md new file mode 100644 index 00000000..08570b1c --- /dev/null +++ b/docs/first-party/002-afd-implementation-plan.md @@ -0,0 +1,1182 @@ +# Proposal 002 — Implementation Plan and Scope of Changes for AFD-based GLB + +| Field | Value | +|-------------|----------------------------------------------------------| +| Status | Draft | +| Author | @rchinchani_microsoft | +| Created | 2026-07-15 | +| Depends on | [Proposal 001](./001-afd-global-load-balancing.md) | +| Supersedes | — | + +This document is the executable companion to proposal 001. Where 001 +answers **what** and **why**, this document answers **where** and +**how** — file-by-file, phase-by-phase, with concrete code sketches +that follow existing repo conventions. + +Nothing here is meant to introduce new architectural choices; if this +document and 001 disagree, 001 wins and this one should be updated to +match. + +--- + +## 1. Guiding conventions (extracted from the existing codebase) + +> **Reader's note.** File tables in §2 tag rows with **Shipped +> (cb02d14)** when they are already in `main`, or with an operation +> code (`A`/`M`/`G`) plus a phase reference when they are still +> future work. `POC status (cb02d14)` and `POC deviation` callouts +> throughout §3–§9 highlight cases where the current code intentionally +> diverges from the target design — those are followed up in +> Proposal 003 §2.4 and the readiness table in §6 of that document. +> +> **Status update (2026-07-20 session, head `629644b`).** Many +> **Phase 3** and **Phase 4** rows in §2 have now shipped; the doc +> intentionally still shows the original per-row markers so the +> phase framing is preserved. Concretely: +> +> - Phase 3 (ExportMode + PLS lookup) is complete: `b85a115`, +> `c6d0d8e`, `cb6f23a`, `cf5324b`. +> - Phase 4 API + reconciler + coexistence guard + envtests are +> complete for `FrontDoorBackend`: `c61dc43`, `be0ccb2`, +> `ceac0ab`, `629644b`. WAFPolicy + ComplianceMode: `aeb116c`, +> `b2cf58f`, `4b85044`. +> - AFD client bundle rows in §2.3 are complete: `95a0096` +> (WAF/SecurityPolicy) and `afe12d7` (OriginGroups/Origins). +> - Sibling binary + chart (`hub-afd-controller-manager`) landed in +> an earlier session (`fcb37f2`, `5672313`, `c219ca2`); the POC +> bridge in `cmd/hub-net-controller-manager` is still present and +> is expected to be removed before GA (see §5.2). +> +> Remaining items: FrontDoorRoute + FrontDoorCustomDomain BYOC +> (Phase 4 tail), member PLS provisioner chart flag polish, and +> e2e coverage. See the breadcrumb Addendum 3 for the per-commit +> narrative. + +The AFD implementation MUST mirror the ATM implementation's structural +choices so that reviewers, on-callers, and future contributors have +one mental model, not two. + +The relevant conventions, all verified in `main`: + +* **Two-CRD split** — a `*Profile` (Azure control-plane object) and + a `*Backend` (per-`ServiceImport` binding). See + `api/v1beta1/trafficmanagerprofile_types.go` and + `trafficmanagerbackend_types.go`. +* **v1alpha1 → v1beta1 graduation.** New types land first under + `api/v1alpha1/`, then are copied and stabilized in `api/v1beta1/` + once they have integration coverage. The v1beta1 copy carries + `// +kubebuilder:storageversion`. +* **Immutability via CEL** — cross-field/immutability constraints + are expressed as `+kubebuilder:validation:XValidation` rules on + the type or field (e.g. `resourceGroup is immutable`, + `spec.profile is immutable`). +* **Resource naming** — controller-managed Azure resources are + named `fleet-` for profiles and + `fleet-##` for endpoints / + origins. Helpers live in `pkg/common/objectmeta`. +* **Metrics** — one `prometheus.GaugeVec` per CR type, + `Namespace: fleet_networking`, tagged with + `namespace, name, generation, condition, status, reason`. See + `pkg/controllers/hub/trafficmanagerprofile/controller.go:70-84`. +* **Azure SDK wiring** — a single `initAzure*Clients(cloudConfig)` + helper in `cmd/hub-net-controller-manager/main.go` builds all + clients for a feature. See `initAzureTrafficManagerClients` at + `main.go:248`. +* **Feature flags** — one boolean `--enable--feature` on + the hub and (if applicable) member manager, gating both CRD + presence-checks and controller registration. See + `main.go:69` and `main.go:193-238`. +* **Fake providers** — every Azure SDK call is behind an interface + and a fake implementation lives under + `test/common//fakeprovider/`, mirroring + `test/common/trafficmanager/fakeprovider/`. + +## 2. Complete file-by-file scope + +The table below is the authoritative checklist of every file that +will be added (`A`), modified (`M`), or generated (`G`). “LoE” is a +rough size estimate for planning only. + +### 2.1 API types + +| Op | Path | LoE | Notes | +|----|------|-----|-------| +| **Shipped (cb02d14)** | `api/v1alpha1/frontdoorprofile_types.go` | ~130 lines | POC subset only: `Spec = {ResourceGroup, Sku}`. WAF/complianceMode/health-probe/response-timeout land in Phase 4. | +| **Shipped (cb02d14)** | `api/v1alpha1/frontdoorcustomdomain_types.go` | ~215 lines | Full shape shipped; Managed TLS reconciled, BYOC field reserved but reconciliation deferred (see §3.3 below). | +| **Shipped (cb02d14)** | `pkg/common/objectmeta/objectmeta.go` (edit) | ~10 lines | Finalizer constants `FrontDoorProfileFinalizer`, `FrontDoorCustomDomainFinalizer`. | +| A | `api/v1alpha1/frontdoorbackend_types.go` | ~200 lines | Phase 4. See §3.2. | +| A | `api/v1alpha1/common_types.go` (edit) | ~20 lines | Phase 4: add `PrivateLinkService` struct shared by FD types | +| M | `api/v1alpha1/internalserviceexport_types.go` | ~30 lines | Phase 3: add `Status.PrivateLinkService` | +| M | `pkg/common/objectmeta/annotations.go` (or equivalent) | ~10 lines | Phase 3: add `ExportModeAnnotation` constant next to the existing `weight` annotation | +| M | `api/v1alpha1/frontdoorprofile_types.go` (extend) | ~120 lines | Phase 4: add `WAFPolicy`, `HealthProbe`, `OriginResponseTimeoutSeconds`, `ComplianceMode`; also tighten `Sku` enum to Premium-only (see §9 risk row). | +| G | `api/v1alpha1/zz_generated.deepcopy.go` | auto | `make generate` | +| A | `api/v1beta1/frontdoor{profile,customdomain,backend}_types.go` | ~600 lines | Phase-5 copy of v1alpha1 | +| M | `api/v1beta1/internalserviceexport_types.go` (if exists; else v1alpha1 only) | ~30 lines | see §3.4 | +| G | `api/v1beta1/zz_generated.deepcopy.go` | auto | `make generate` | + +`ServiceExport` itself is **unchanged** — no new `Spec` field, no +new `Status` field. Fleet-specific intent is carried via the +`networking.fleet.azure.com/export-mode` annotation (§3.4). This +preserves upstream mcs-api (KEP-1645) parity, which is a repository +preference for `ServiceExport` / `MultiClusterService`. + +**No `Location` field on `FrontDoorProfile.Spec`.** AFD is a global +service (all profiles must be created at `Location: "Global"`); the +controller sets this internally rather than exposing a +single-valued CR field. + +### 2.2 CRD manifests and RBAC + +| Op | Path | Notes | +|----|------|-------| +| G | `config/crd/bases/networking.fleet.azure.com_frontdoorprofiles.yaml` | **Shipped (cb02d14)**; regenerated by `make manifests` | +| G | `config/crd/bases/networking.fleet.azure.com_frontdoorcustomdomains.yaml` | **Shipped (cb02d14)** | +| G | `config/crd/bases/networking.fleet.azure.com_frontdoorbackends.yaml` | Phase 4; `make manifests` | +| G | `config/rbac/role.yaml` | `make manifests` — adds verbs on new resources | +| A | `charts/hub-afd-controller-manager/Chart.yaml` | **New sibling chart** — isolates AFD RBAC, ServiceAccount, and Workload-Identity federated subject from the ATM chart (see Proposal 001 §6, §7). | +| A | `charts/hub-afd-controller-manager/values.yaml` | AFD identity block (`serviceAccount.azure.workloadIdentity.clientID`, `tenantID`), image, resources, `frontDoor.enabled: true` (single-purpose chart). | +| A | `charts/hub-afd-controller-manager/templates/deployment.yaml` | Runs `cmd/hub-afd-controller-manager` with `--enable-frontdoor-feature`. | +| A | `charts/hub-afd-controller-manager/templates/rbac.yaml` | AFD-only cluster role: verbs on `frontdoorprofiles`, `frontdoorcustomdomains`, `frontdoorbackends`. | +| A | `charts/hub-afd-controller-manager/templates/serviceaccount.yaml` | SA carrying the WI annotation with the AFD-scoped MI. | +| A | `charts/hub-afd-controller-manager/templates/pdb.yaml` | `PodDisruptionBudget` for Safeguards / Automatic (§6.5). | +| A | `charts/hub-afd-controller-manager/README.md` | Document AFD RBAC, WI setup, `frontDoor.enabled` (always true when this chart is installed at all). | +| — | `charts/hub-net-controller-manager/**` | **Unchanged for AFD.** ATM chart keeps its own SA/MI. Zero-diff `helm template` output preserved via baselines under `.github/.copilot/breadcrumbs/baselines/`. | +| M | `charts/member-net-controller-manager/values.yaml` | Member PLS provisioner toggle (Phase 3). | +| M | `charts/member-net-controller-manager/templates/deployment.yaml` | Phase 3: pass `--enable-frontdoor-feature` (member controller is a Service-annotation reader; no Azure SDK, no separate identity needed). | +| M | `charts/hub-net-controller-manager/README.md`, `charts/member-net-controller-manager/README.md` | Cross-reference the new sibling chart. | + +**POC deviation (cb02d14):** the sibling chart does **not** exist +yet. The POC wires AFD registration into +`cmd/hub-net-controller-manager/main.go` behind +`--enable-frontdoor-feature` (default `false`). This preserves the +existing ATM chart's `helm template` output but violates the +identity-split invariant from Proposal 001 §7. The sibling chart ++ sibling `cmd/` binary is a hard GA prerequisite (Proposal 003 +§2.4). + +### 2.3 Hub controllers + +| Op | Path | Notes | +|----|------|-------| +| **Shipped (cb02d14)** | `pkg/controllers/hub/frontdoorprofile/controller.go` | Happy-path reconciler with finalizer; see §4.1. | +| **Shipped (cb02d14)** | `pkg/controllers/hub/frontdoorcustomdomain/controller.go` | Managed TLS path only; BYOC deferred. | +| A | `pkg/controllers/hub/frontdoorprofile/controller_test.go` | table-driven unit tests — not yet in cb02d14 | +| A | `pkg/controllers/hub/frontdoorprofile/controller_integration_test.go` | Ginkgo, envtest — not yet in cb02d14 | +| A | `pkg/controllers/hub/frontdoorprofile/suite_test.go` | envtest scaffolding — not yet in cb02d14 | +| A | `pkg/controllers/hub/frontdoorcustomdomain/{controller,suite}_test.go` | as above | +| A | `pkg/controllers/hub/frontdoorbackend/controller.go` | Phase 4; see §4.2 | +| A | `pkg/controllers/hub/frontdoorbackend/controller_test.go` | | +| A | `pkg/controllers/hub/frontdoorbackend/controller_integration_test.go` | | +| A | `pkg/controllers/hub/frontdoorbackend/suite_test.go` | | + +### 2.4 Member controller changes + +| Op | Path | Notes | +|----|------|-------| +| M | `pkg/controllers/member/serviceexport/controller.go` | resolve export mode (annotation, else infer from Service), populate `InternalServiceExport.status.privateLinkService`; never mutate the Service | +| M | `pkg/controllers/member/serviceexport/controller_test.go` | new test cases | +| M | `pkg/controllers/member/serviceexport/controller_integration_test.go` | new Ginkgo `Context` | +| A | `pkg/controllers/member/serviceexport/frontdoor.go` | helper file for the PLS annotation logic (keeps `controller.go` small) | +| A | `pkg/controllers/member/serviceexport/frontdoor_test.go` | | + +### 2.5 Common libraries + +| Op | Path | Notes | +|----|------|-------| +| **Shipped (cb02d14)** | `pkg/common/azurefrontdoor/client.go` | Single-file: `Config`+`LoadConfigFromEnv`, `NewCredential` (Workload Identity), `NewClients` bundling `Profiles`, `AFDEndpoints`, `AFDCustomDomains`. | +| A | `pkg/common/azurefrontdoor/interface.go` | Phase 4: extract client interface for testability once fake provider is added. | +| A | `pkg/common/azurefrontdoor/naming.go` | Phase 4: promote the `fleet-` helpers out of the profile controller as more resource kinds are added. | +| A | `pkg/common/azurefrontdoor/naming_test.go` | | +| M | `pkg/common/azureerrors/errors.go` | classify AFD-specific errors (WAF-not-found, PL-approval-pending, etc.) | +| **Shipped (cb02d14)** | `pkg/common/objectmeta/objectmeta.go` | Finalizer constants: `FrontDoorProfileFinalizer = "networking.fleet.azure.com/frontdoor-profile-cleanup"`, `FrontDoorCustomDomainFinalizer = ".../frontdoor-custom-domain-cleanup"`. | +| M | `pkg/common/objectmeta/objectmeta_test.go` | | +| A | `pkg/common/defaulter/frontdoorprofile.go` | Phase 4 (arrives with WAF/complianceMode fields). | +| A | `pkg/common/defaulter/frontdoorprofile_test.go` | | +| A | `pkg/common/defaulter/frontdoorbackend.go` | Phase 4 | +| A | `pkg/common/defaulter/frontdoorbackend_test.go` | | + +### 2.6 Entry points + +| Op | Path | Notes | +|----|------|-------| +| A | `cmd/hub-afd-controller-manager/main.go` | **New sibling binary** — registers only the AFD controllers, mounts only the AFD WI subject. Enables the identity split from Proposal 001 §7. | +| M | `cmd/member-net-controller-manager/main.go` | Phase 3: add `--enable-frontdoor-feature` flag and thread through to serviceexport reconciler. | +| M | `cmd/net-crd-installer/utils/util.go` (and `_test.go`) | Include the three new CRDs in the installer inventory. | +| — | `cmd/hub-net-controller-manager/main.go` | **Reverted to ATM-only for GA.** The POC (cb02d14) currently registers AFD here behind `--enable-frontdoor-feature` as a bridge; the sibling-binary migration removes those wires before Phase 5. | + +**POC deviation (cb02d14):** `cmd/hub-afd-controller-manager/main.go` +does **not** exist yet. Instead, `cmd/hub-net-controller-manager/main.go` +gained ~65 lines gated by `--enable-frontdoor-feature` +(`enableFrontDoorFeature` var, `initAzureFrontDoorClients` call site, +registration of `frontdoorprofile.Reconciler` + `frontdoorcustomdomain.Reconciler`). +The migration is: extract those lines into the new binary, +delete them from `hub-net-controller-manager`, add the new chart +(§2.2). Nothing about the reconciler packages themselves has to +change. + +### 2.7 Tests + +| Op | Path | Notes | +|----|------|-------| +| A | `test/common/frontdoor/fakeprovider/profile.go` | analog of `test/common/trafficmanager/fakeprovider/profile.go` | +| A | `test/common/frontdoor/fakeprovider/origin.go` | origin groups + origins | +| A | `test/common/frontdoor/fakeprovider/route.go` | routes + security policies | +| A | `test/common/frontdoor/validator/profile.go` | test assertion helpers | +| A | `test/common/frontdoor/validator/backend.go` | | +| A | `test/common/frontdoor/azureprovider/profile.go` | real Azure client for e2e | +| M | `test/apis/v1alpha1/api_validation_integration_test.go` | cover new CRDs and CEL rules | +| M | `test/apis/v1beta1/api_validation_integration_test.go` | Phase-5 | +| A | `test/e2e/frontdoor_test.go` | Ginkgo e2e suite | +| M | `test/e2e/e2e_test.go` | wire the new suite behind an env-gate | +| M | `test/scripts/bootstrap.sh` | provision AFD Premium profile RG + WAF policy | +| A | `hack/cl2/manifests/test-fdp.yaml`, `test-fdb.yaml` | scale-test manifests | +| A | `hack/cl2/afd_scale_test_config.yaml` | scale-test driver | + +### 2.8 Examples and docs + +| Op | Path | Notes | +|----|------|-------| +| A | `examples/getting-started/artifacts/afd.yaml` | analog of `atm.yaml` | +| A | `docs/concepts/HTTPBasedGlobalLoadBalancing/README.md` | user-facing concept doc | +| A | `docs/howtos/frontdoor-permissions-setup.md` | least-privilege setup guide | +| A | `docs/toubleshooting/HTTPBasedGlobalLoadBalancing.md` | note: keeps existing folder’s typo `toubleshooting` | +| A | `docs/demos/FrontDoorProfile/…` | (optional, phase 5) | + +### 2.9 Module manifest + +| Op | Path | Notes | +|----|------|-------| +| M | `go.mod`, `go.sum` | add `github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cdn/armcdn` (Standard/Premium AFD lives under this SDK, not `armfrontdoor` which is classic) | + +--- + +## 3. API sketches + +The Go snippets below are indicative — not final. They exist so that +reviewers can point at concrete fields and CEL rules rather than +prose. + +### 3.1 `FrontDoorProfile` + +> **POC status (cb02d14).** The reconciler and CRD have shipped, but +> the fields marked `POC:` are the only ones currently present. +> `ComplianceMode`, `WAFPolicy`, `HealthProbe`, +> `OriginResponseTimeoutSeconds` and their CEL rules land alongside +> the WAF work in Phase 4. Similarly, the `Sku` enum is currently +> permissive (accepts Standard too); tightening it to Premium-only +> is a Phase-4 change (see §9 risks). + +```go +// api/v1alpha1/frontdoorprofile_types.go + +const FrontDoorProfileKind = "FrontDoorProfile" + +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Namespaced,categories={fleet-networking},shortName=afdp +// +kubebuilder:subresource:status +// +kubebuilder:printcolumn:JSONPath=`.status.endpointHostname`,name="Endpoint",type=string +// +kubebuilder:printcolumn:JSONPath=`.status.conditions[?(@.type=='Programmed')].status`,name="Is-Programmed",type=string +// +kubebuilder:printcolumn:JSONPath=`.metadata.creationTimestamp`,name="Age",type=date +// +kubebuilder:validation:XValidation:rule="size(self.metadata.name) < 64",message="metadata.name max length is 63" +type FrontDoorProfile struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec FrontDoorProfileSpec `json:"spec"` + Status FrontDoorProfileStatus `json:"status,omitempty"` +} + +type FrontDoorSKU string + +const ( + // Premium is the only supported SKU — Private Link origins are + // Premium-only and required for SFI-NS253 (see Proposal 001 §2.3 + // and Proposal 003 checklist §1.2). Standard is intentionally + // excluded at the CRD enum layer so misconfiguration is rejected + // at admission time, not surfaced as a Programmed=False condition + // hours later. + FrontDoorSKUPremium FrontDoorSKU = "Premium_AzureFrontDoor" +) + +// ComplianceMode declares the security/compliance regime a profile +// (and its backends) must satisfy. See Proposal 001 §2.1. +type ComplianceMode string + +const ( + // ComplianceModeNone imposes no additional constraints beyond the + // structural ones. Suitable for dev/test or non-first-party use. + ComplianceModeNone ComplianceMode = "None" + + // ComplianceModeSFINS253 enforces: + // * spec.wafPolicy is required (CEL on the profile) + // * every referencing FrontDoorBackend must have + // spec.privateLink.enabled = true (enforced in the backend reconciler, + // surfaced as Accepted=False, + // Reason=SFIComplianceViolation) + // * WAF policy MUST be in Prevention mode (surfaced as + // Programmed=False, + // Reason=WAFPolicyNotInPreventionMode) + ComplianceModeSFINS253 ComplianceMode = "SFI-NS253" +) + +type FrontDoorProfileSpec struct { + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:MaxLength=90 + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="resourceGroup is immutable" + ResourceGroup string `json:"resourceGroup"` + + // Only Premium_AzureFrontDoor is supported; the enum is + // single-valued so the field is effectively fixed but stays + // present for forward compatibility. + // +kubebuilder:validation:Enum=Premium_AzureFrontDoor + // +kubebuilder:default=Premium_AzureFrontDoor + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="sku is immutable" + SKU FrontDoorSKU `json:"sku,omitempty"` + + // ComplianceMode declares the security/compliance regime this + // profile is subject to. When set to "SFI-NS253", the CEL rule + // below requires spec.wafPolicy, and the FrontDoorBackend + // reconciler additionally requires PrivateLink on every + // backend that references this profile. Immutable: switching + // out of SFI-NS253 mode would silently weaken guarantees the + // operator relied on — recreate the profile instead. + // +optional + // +kubebuilder:validation:Enum=None;SFI-NS253 + // +kubebuilder:default=None + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="complianceMode is immutable" + ComplianceMode ComplianceMode `json:"complianceMode,omitempty"` + + // WAFPolicy is REQUIRED when complianceMode == SFI-NS253. + // Optional otherwise, so third-party dev/test flows can create + // an AFD profile without a WAF attach. + // +optional + // +kubebuilder:validation:XValidation:rule="self.complianceMode != 'SFI-NS253' || has(self.wafPolicy)",message="wafPolicy is required when complianceMode is SFI-NS253" + WAFPolicy *FrontDoorWAFPolicyRef `json:"wafPolicy,omitempty"` + + // +optional + HealthProbe *FrontDoorHealthProbe `json:"healthProbe,omitempty"` + + // +optional + // +kubebuilder:validation:Minimum=16 + // +kubebuilder:validation:Maximum=240 + // +kubebuilder:default=60 + OriginResponseTimeoutSeconds *int32 `json:"originResponseTimeoutSeconds,omitempty"` +} + +type FrontDoorWAFPolicyRef struct { + // ResourceID of an existing + // Microsoft.Network/frontdoorwebapplicationfirewallpolicies resource. + // Mutually exclusive with Inline. + // +optional + ResourceID string `json:"resourceID,omitempty"` + + // Inline creation is opt-in — most first-party services will + // reference a centrally managed WAF policy. + // +optional + Inline *InlineWAFPolicy `json:"inline,omitempty"` +} + +type FrontDoorProfileStatus struct { + // +optional + HostName *string `json:"hostName,omitempty"` + // +optional + ResourceID string `json:"resourceID,omitempty"` + // +optional + EndpointResourceID string `json:"endpointResourceID,omitempty"` + + // +optional + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"` +} + +const ( + FrontDoorProfileConditionProgrammed = "Programmed" + + FrontDoorProfileReasonProgrammed = "Programmed" + FrontDoorProfileReasonInvalid = "Invalid" + FrontDoorProfileReasonPending = "Pending" + FrontDoorProfileReasonWAFPolicyNotFound = "WAFPolicyNotFound" + FrontDoorProfileReasonWAFPolicyNotInPrevention = "WAFPolicyNotInPreventionMode" + FrontDoorProfileReasonHostNameNotAvailable = "HostNameNotAvailable" +) +``` + +The 63-char `metadata.name` cap is Kubernetes' standard label-value +limit; Azure resource names are derived from the CR's UID +(`fleet-` — 42 chars) rather than `metadata.name`, so a CR +rename does not orphan the Azure profile and the AFD 46-char +endpoint-name cap is always respected regardless of what the user +chooses for `metadata.name`. + +### 3.2 `FrontDoorBackend` + +```go +// api/v1alpha1/frontdoorbackend_types.go + +const FrontDoorBackendKind = "FrontDoorBackend" + +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Namespaced,categories={fleet-networking},shortName=fdb +// +kubebuilder:subresource:status +type FrontDoorBackend struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec FrontDoorBackendSpec `json:"spec"` + Status FrontDoorBackendStatus `json:"status,omitempty"` +} + +type FrontDoorBackendSpec struct { + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.profile is immutable" + Profile FrontDoorProfileRef `json:"profile"` + + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="spec.backend is immutable" + Backend FrontDoorBackendRef `json:"backend"` + + // +optional + // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=1000 + // +kubebuilder:default=100 + Weight *int32 `json:"weight,omitempty"` + + // +optional + Routing *FrontDoorRoutingConfig `json:"routing,omitempty"` + + // PrivateLink.enabled MUST be true for SFI-NS253 workloads. + // +optional + PrivateLink *FrontDoorPrivateLinkConfig `json:"privateLink,omitempty"` +} + +type FrontDoorRoutingConfig struct { + // +optional + // +kubebuilder:default={"/*"} + PatternsToMatch []string `json:"patternsToMatch,omitempty"` + + // +optional + // +kubebuilder:validation:Enum=HttpOnly;HttpsOnly;MatchRequest + // +kubebuilder:default=HttpsOnly + ForwardingProtocol string `json:"forwardingProtocol,omitempty"` + + // +optional + // +kubebuilder:default={"Https"} + SupportedProtocols []string `json:"supportedProtocols,omitempty"` + + // +optional + // +kubebuilder:validation:Enum=Enabled;Disabled + // +kubebuilder:default=Enabled + LinkToDefaultDomain string `json:"linkToDefaultDomain,omitempty"` +} + +type FrontDoorPrivateLinkConfig struct { + // +kubebuilder:default=true + Enabled bool `json:"enabled"` + + // +optional + // +kubebuilder:validation:MaxLength=140 + RequestMessage string `json:"requestMessage,omitempty"` +} + +type FrontDoorBackendStatus struct { + // Origins is the list of AFD origins created under the profile + // for this backend. One entry per member-cluster serviceExport. + // +optional + Origins []FrontDoorOriginStatus `json:"origins,omitempty"` + + // +optional + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"` +} + +type FrontDoorOriginStatus struct { + Name string `json:"name"` + ResourceID string `json:"resourceID,omitempty"` + Weight *int32 `json:"weight,omitempty"` + HostName *string `json:"hostName,omitempty"` // PLS alias FQDN + PrivateLinkResourceID string `json:"privateLinkResourceID,omitempty"` + PrivateEndpointStatus string `json:"privateEndpointStatus,omitempty"` // Pending|Approved|Rejected|Disconnected + // +optional + From *FromCluster `json:"from,omitempty"` +} +``` + +### 3.3 `FrontDoorCustomDomain` + +Shipped in `cb02d14`. See the source for the complete Go shape +(`api/v1alpha1/frontdoorcustomdomain_types.go`). Key design points +worth calling out for future reviewers: + +* **Namespaced, `shortName: afdcd`.** +* **Immutable `ProfileRef` and `Hostname`.** Retargeting a custom + domain to a different profile is not a supported mutation — delete + and recreate. +* **Same-namespace-only `ProfileRef`.** Cross-namespace references + are rejected by design (breadcrumb D4): the AFD tenancy story + requires that a custom domain and its owning profile share an RBAC + boundary. +* **TLS.Mode enum: `Managed` or `BYOC`.** `Managed` is fully + reconciled today; `BYOC` field shape is present (Key Vault URI + + certificate name + optional version) but the reconciler does not + yet call the Key Vault binding APIs. CEL cross-field validation + already enforces that `keyVaultCertificate` is present iff + `Mode == BYOC`. +* **DNS validation surface.** `Status` exposes `ValidationState` + (Pending | Approved | Rejected | TimedOut | InternalError | + Submitting | RefreshingValidationToken | Unknown), + `DNSValidationToken`, and `DNSValidationExpiry`. The tenant + publishes a TXT record at `_dnsauth.` with the token + value; AFD polls DNS, then the controller reflects `Approved` and + transitions `Programmed=True`. +* **Condition reasons on `Programmed`:** `Programmed`, `Invalid`, + `ProfileNotReady`, `AwaitingDNSValidation`, `ValidationFailed`, + `TLSFailed`, `AzureError`, `Pending`. +* **Finalizer:** `networking.fleet.azure.com/frontdoor-custom-domain-cleanup`. + +Route binding (attaching a validated custom domain to an AFD route) +lives with `FrontDoorBackend` in Phase 4 — a `FrontDoorCustomDomain` +by itself only validates ownership and provisions the AFD-side +resource; it does not front any traffic. + +### 3.4 Additive changes to existing types + +**`ServiceExport` gains no schema change.** Per Proposal 001 §3.3 / +§4.2, mode selection is carried by an annotation, matching the +existing `networking.fleet.azure.com/weight` precedent and preserving +upstream mcs-api (KEP-1645) parity. + +```go +// pkg/common/objectmeta/annotations.go (partial) + +const ( + // ExportModeAnnotation, when set on a ServiceExport, selects the + // north-south surface the exported Service should be attached to. + // Absence of the annotation is equivalent to L4-TrafficManager + // (today's implicit default). See Proposal 001 §3.3 for the + // annotation-vs-inference precedence rule. + ExportModeAnnotation = "networking.fleet.azure.com/export-mode" + + ExportModeValueTrafficManager = "L4-TrafficManager" + ExportModeValueFrontDoor = "L7-FrontDoor" +) +``` + +No CRD manifest regeneration is required for `ServiceExport`; only +`InternalServiceExport` gains the `Status.PrivateLinkService` block +below. + +```go +// api/v1alpha1/internalserviceexport_types.go (partial) + +type ServiceExportPrivateLinkStatus struct { + ResourceID string `json:"resourceID"` + Alias string `json:"alias,omitempty"` + InternalLoadBalancerFrontendIP string `json:"internalLoadBalancerFrontendIP,omitempty"` + LastProbedTime metav1.Time `json:"lastProbedTime,omitempty"` +} + +type InternalServiceExportStatus struct { + // ... existing fields ... + + // +optional + PrivateLinkService *ServiceExportPrivateLinkStatus `json:"privateLinkService,omitempty"` +} +``` + +--- + +## 4. Controller sketches + +### 4.1 Hub `frontdoorprofile` reconciler + +```go +// pkg/controllers/hub/frontdoorprofile/controller.go (skeleton) + +package frontdoorprofile + +const ( + ControllerName = "frontdoorprofile-controller" + + AzureResourceProfileNameFormat = "fleet-%s" + AzureResourceEndpointNameFormat = "fleet-%s-endpoint" + + profileEventReasonAzureAPIError = "AzureAPIError" + profileEventReasonProgrammed = "Programmed" + profileEventReasonDeleted = "Deleted" +) + +type Reconciler struct { + client.Client + + ProfilesClient *armcdn.ProfilesClient + AFDEndpointsClient *armcdn.AFDEndpointsClient + SecurityPoliciesClient *armcdn.SecurityPoliciesClient + Recorder record.EventRecorder +} + +//+kubebuilder:rbac:groups=networking.fleet.azure.com,resources=frontdoorprofiles,verbs=get;list;watch;create;update;patch;delete +//+kubebuilder:rbac:groups=networking.fleet.azure.com,resources=frontdoorprofiles/status,verbs=get;update;patch +//+kubebuilder:rbac:groups=networking.fleet.azure.com,resources=frontdoorprofiles/finalizers,verbs=update + +func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + // 1. Fetch FrontDoorProfile + // 2. Add finalizer / handle deletion (mirror TM profile) + // 3. Apply defaults via defaulter.SetDefaultsFrontDoorProfile + // 4. Ensure Microsoft.Cdn/profiles + // 5. Ensure Microsoft.Cdn/profiles/afdEndpoints + // 6. If spec.wafPolicy: ensure Microsoft.Cdn/profiles/securityPolicies binding to endpoint + // 7. Update status.hostName, status.resourceID, status.endpointResourceID + // 8. Set condition Programmed + // 9. Emit event + metric +} +``` + +Reconciler ordering, event names, finalizer semantics, and status +patching all copy the shape of +`pkg/controllers/hub/trafficmanagerprofile/controller.go` verbatim +where possible. + +### 4.2 Hub `frontdoorbackend` reconciler + +Additional inputs beyond the profile reconciler: + +* `AFDOriginGroupsClient`, `AFDOriginsClient`, `RoutesClient` +* A field indexer on `InternalServiceExport.status.privateLinkService.resourceID` + so that PLS status changes on the member side trigger a hub-side + requeue. + +Watches: + +* Owned: `FrontDoorBackend` (primary). +* Enqueue-on-change: + * `FrontDoorProfile` (name/namespace match) — recompute if profile + is (re)programmed. + * `ServiceImport` (spec.backend.name). + * `InternalServiceExport` (indexed by exported service reference). + +Origin naming: `fleet-##`. + +Private-endpoint approval: + +* When `privateLink.enabled = true`, the AFD API creates a + private endpoint connection on the PLS. +* If the PLS was created with `azure-pls-auto-approval` including + the AFD subscription, the connection reaches `Approved` without + further action. +* Otherwise, the reconciler surfaces status + `PrivateEndpointStatus = Pending` and sets the backend condition + `Accepted=False, Reason=PrivateLinkPending`, waiting for + approval. + +Cross-check with profile compliance mode: + +* When the referenced `FrontDoorProfile.spec.complianceMode == SFI-NS253` + and this backend has `privateLink == nil` or + `privateLink.enabled == false`, the reconciler refuses to create + any origin and surfaces + `Accepted=False, Reason=SFIComplianceViolation` with a message + explaining that the parent profile is in SFI mode. This closes the + loophole where an operator could create an SFI-compliant profile + but then attach a lax backend to it. + +### 4.3 Member `serviceexport` extension + +Today `serviceexport.controller.go` translates a +`ServiceExport` + `Service` into an `InternalServiceExport` in the +member-cluster’s reserved hub namespace. + +Additions: + +1. **Mode resolution** (per Proposal 001 §3.3): + * Read the `networking.fleet.azure.com/export-mode` annotation + on the `ServiceExport`. If set to `L7-FrontDoor`, treat as AFD + mode. + * If the annotation is unset, infer AFD mode when the exported + `Service` carries **all** of the following annotations: + `azure-load-balancer-internal: "true"`, + `azure-pls-create: "true"`, + `azure-pls-name`, + `azure-pls-ip-configuration-subnet`, + `azure-pls-visibility`, + `azure-pls-auto-approval`. + Otherwise treat as `L4-TrafficManager` (today's behaviour). + * If the annotation demands `L7-FrontDoor` but the Service is + not internal + PLS-enabled, surface + `ServiceExportValid=False, + Reason=ExportModeAnnotationServiceMismatch` and do not + mutate anything. **The controller never writes annotations + to the Service** — the internal-LB + PLS annotations are + tenant-owned (via GitOps and/or platform admission policy). +2. When AFD mode is resolved, additionally require: + * `Service.Spec.Type == LoadBalancer` (reject otherwise with + `ServiceExportInvalid` reason + `UnsupportedServiceTypeForFrontDoor`). + * `Service.Spec.ClusterIP != "None"` (headless services cannot + back a PLS; reject with the same reason). +3. Watch the `Service.Status.LoadBalancer` and the + AKS-cloud-provider-set annotation + `service.beta.kubernetes.io/azure-pls-resource-id`, and copy + the PLS resource ID into + `InternalServiceExport.status.privateLinkService`. +4. Emit a Kubernetes `Event` on the source `ServiceExport` when the + PLS transitions to `Approved`. + +Nothing in this controller talks to Azure directly — the AKS cloud +provider does the PLS provisioning. That keeps member-side RBAC +identical to today. + +--- + +## 5. `cmd/hub-net-controller-manager/main.go` diff sketch + +```go +// Additions (illustrative — actual line numbers will differ): + +import ( + "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cdn/armcdn" + // ... + "go.goms.io/fleet-networking/pkg/controllers/hub/frontdoorbackend" + "go.goms.io/fleet-networking/pkg/controllers/hub/frontdoorprofile" +) + +var ( + enableFrontDoorFeature = flag.Bool("enable-frontdoor-feature", false, + "If set, the Azure Front Door feature will be enabled.") + + frontDoorFeatureRequiredGVKs = []schema.GroupVersionKind{ + fleetnetv1alpha1.GroupVersion.WithKind(fleetnetv1alpha1.FrontDoorProfileKind), + fleetnetv1alpha1.GroupVersion.WithKind(fleetnetv1alpha1.FrontDoorBackendKind), + } +) + +// ... inside main(), after the existing enableTrafficManagerFeature block: + +if *enableFrontDoorFeature { + for _, gvk := range frontDoorFeatureRequiredGVKs { + if err = utils.CheckCRDInstalled(discoverClient, gvk); err != nil { + klog.ErrorS(err, "Unable to find required Front Door CRD", "GVK", gvk) + exitWithErrorFunc() + } + } + cloudConfig, err := azure.NewCloudConfigFromFile(*cloudConfigFile) + if err != nil { /* ... */ } + cloudConfig.SetUserAgent("fleet-hub-net-controller-manager") + + fdClients, err := initAzureFrontDoorClients(cloudConfig) + if err != nil { /* ... */ } + + if err := (&frontdoorprofile.Reconciler{ + Client: mgr.GetClient(), + ProfilesClient: fdClients.Profiles, + AFDEndpointsClient: fdClients.Endpoints, + SecurityPoliciesClient: fdClients.SecurityPolicies, + Recorder: mgr.GetEventRecorderFor(frontdoorprofile.ControllerName), + }).SetupWithManager(mgr); err != nil { /* ... */ } + + if err := (&frontdoorbackend.Reconciler{ + Client: mgr.GetClient(), + ProfilesClient: fdClients.Profiles, + AFDOriginGroupsClient: fdClients.OriginGroups, + AFDOriginsClient: fdClients.Origins, + RoutesClient: fdClients.Routes, + Recorder: mgr.GetEventRecorderFor(frontdoorbackend.ControllerName), + }).SetupWithManager(ctx, mgr, true); err != nil { /* ... */ } +} + +// initAzureFrontDoorClients mirrors initAzureTrafficManagerClients at cmd/hub-net-controller-manager/main.go:248. +type frontDoorClientBundle struct { + Profiles *armcdn.ProfilesClient + Endpoints *armcdn.AFDEndpointsClient + OriginGroups *armcdn.AFDOriginGroupsClient + Origins *armcdn.AFDOriginsClient + Routes *armcdn.RoutesClient + SecurityPolicies *armcdn.SecurityPoliciesClient +} + +func initAzureFrontDoorClients(cloudConfig *azure.CloudConfig) (*frontDoorClientBundle, error) { + // exact same authProvider + options + rate-limit-policy pattern as ATM + // then construct all six clients via armcdn.NewClientFactory +} +``` + +--- + +## 6. Phased delivery — one PR per phase + +Each phase is designed to land as a **reviewable, mergeable** PR that +does not regress ATM behavior. The AFD feature flag stays `false` +until Phase 4. + +> **Current phase status (as of cb02d14).** +> - **Phase 1:** *Partially delivered.* Two of three CRDs shipped +> (`FrontDoorProfile`, `FrontDoorCustomDomain`) with immutability +> CEL. Not yet in `main`: `FrontDoorBackend` type, `WAFPolicy` / +> `ComplianceMode` / `HealthProbe` / `OriginResponseTimeoutSeconds` +> fields, Sku enum tightened to Premium-only, defaulters, envtest +> scaffolding, `test/apis` coverage. +> - **Phase 2:** *Partially delivered.* Happy-path `frontdoorprofile` +> and `frontdoorcustomdomain` reconcilers shipped, with WI-based +> Azure client factory (`pkg/common/azurefrontdoor`) and finalizers. +> Not yet: unit tests, integration tests, fake provider, sibling +> binary+chart (currently hosted inside `cmd/hub-net-controller-manager` +> under `--enable-frontdoor-feature`). +> - **Phases 3–5:** not started. +> +> The "Deliverables" and "Exit criteria" bullets below describe the +> *complete* phase; treat items already in `main` as done and the +> rest as remaining work. + +### Phase 1 — API types + generated artifacts + defaulters (~1 week) + +Deliverables: +* All files under §2.1 (v1alpha1 only), §2.2 (CRD manifests + RBAC), + and the four `defaulter` files from §2.5. +* Unit tests for defaulters and CEL rules. +* No controller registered anywhere — controllers do not compile + yet, only the types. + +Exit criteria: +* `make generate manifests` produces stable output. +* `go test ./api/... ./pkg/common/defaulter/...` green. +* `test/apis/v1alpha1/api_validation_integration_test.go` covers + every CEL rule on the new types. + +### Phase 2 — Hub `frontdoorprofile` controller (~1.5 weeks) + +Deliverables: +* §2.3 profile files, §2.5 azurefrontdoor client interface + naming, + `initAzureFrontDoorClients` in `cmd/hub-net-controller-manager/main.go`, + and the flag registration. +* Fake provider `test/common/frontdoor/fakeprovider/profile.go`. +* Integration test uses the fake provider under envtest. + +Exit criteria: +* Feature flag `--enable-frontdoor-feature=true` on a dev cluster + produces an AFD profile + endpoint + (optional) WAF security policy + in Azure, observable via `az afd profile show`. +* No AFD backend controller registered yet. + +### Phase 3 — Member PLS provisioner + status plumbing (~1 week) + +Deliverables: +* §2.4 changes to member `serviceexport`. +* The additive field on `InternalServiceExport.Status`. +* Integration test proves that a `ServiceExport` resolved to + `L7-FrontDoor` (annotation or inference) with a `Service` that + has the internal-LB + PLS annotations produces a PLS in the member + cluster and the resource ID is reflected in + `InternalServiceExport.status.privateLinkService.resourceID`. + A companion test proves that an annotation-driven mismatch + (Service missing the annotations) surfaces + `ExportModeAnnotationServiceMismatch` and does not mutate anything. + +Exit criteria: +* Member controller does not require any Azure SDK — the AKS cloud + provider does the PLS creation via Service annotations. +* Passing test in `pkg/controllers/member/serviceexport/`. + +### Phase 4 — Hub `frontdoorbackend` controller + e2e (~2 weeks) + +Deliverables: +* §2.3 backend files, updated fakeprovider (origin, route, + private-endpoint approval). +* Origin/route/security-policy reconciliation. +* Wire the backend controller into `main.go`. +* `test/e2e/frontdoor_test.go` — full end-to-end path: create + `FrontDoorProfile` + two member `ServiceExport` (L7) + a + `FrontDoorBackend` → curl the AFD hostname → traffic reaches at + least one origin. + +Exit criteria: +* Full e2e passes in the ci-e2e pipeline gated by an env var. +* SFI reviewer signs off on the produced Azure topology. + +### Phase 5 — v1beta1 promotion, docs, GA (~1 week) + +Deliverables: +* Everything under `api/v1beta1/` from §2.1. +* `test/apis/v1beta1/api_validation_integration_test.go`. +* Concept, howto, and troubleshooting docs from §2.8. +* Flip default of `--enable-frontdoor-feature` to `true`. + +Exit criteria: +* v1beta1 marked `storageversion`. +* Feature announced in root `README.md` alongside ATM. + +Total nominal effort: **~6.5 weeks** for one engineer, or **~4 weeks** +with a second engineer taking phases 2/3 in parallel with phases 4 +API + fake provider work. + +--- + +## 6.5 AKS Automatic compatibility + +Proposal 001 §3.4 declares AKS Automatic a supported member cluster +SKU. This section enumerates the concrete chart / manifest changes +that keep the fleet-networking components installable on Automatic +alongside the existing AKS Standard install path. + +### 6.5.1 Deployment Safeguards requirements + +AKS Automatic runs Azure Policy safeguards in Enforcement mode by +default. The hub and member Deployments MUST satisfy at least the +following (all standard restricted-workload hygiene): + +* Every container declares `resources.requests` and `resources.limits` + for `cpu` and `memory`. +* `securityContext.runAsNonRoot: true` and `runAsUser` >= 1000 on + every container. +* `securityContext.allowPrivilegeEscalation: false`. +* `securityContext.capabilities.drop: ["ALL"]`; no `add:` unless + strictly required. +* `securityContext.readOnlyRootFilesystem: true` where compatible + (may require an `emptyDir` for `/tmp` or logs). +* `securityContext.seccompProfile.type: RuntimeDefault` at pod scope. +* No `hostPath`, `hostNetwork`, `hostPID`, or `hostIPC`. +* Container images pulled from an allow-listed registry + (`mcr.microsoft.com` or the tenant's ACR — never Docker Hub for + first-party installs). +* Every Deployment ships a matching `PodDisruptionBudget` with + `minAvailable: 1` (or `maxUnavailable: 0` if a single replica). + +### 6.5.2 File-by-file additions + +| Op | Path | Notes | +|----|------|-------| +| M | `charts/hub-net-controller-manager/templates/deployment.yaml` | Add `resources`, `securityContext`, and `readOnlyRootFilesystem` for the hub manager container; add `emptyDir` for `/tmp` if `readOnlyRootFilesystem: true`. | +| M | `charts/member-net-controller-manager/templates/deployment.yaml` | Same, for the member manager container. | +| M | `charts/hub-net-controller-manager/templates/pdb.yaml` (new) | `PodDisruptionBudget` for the hub manager. | +| M | `charts/member-net-controller-manager/templates/pdb.yaml` (new) | `PodDisruptionBudget` for the member manager. | +| M | `charts/hub-net-controller-manager/values.yaml` | Surface `resources`, `securityContext`, and `image.registry` as configurable values (default to safeguards-compliant values). | +| M | `charts/member-net-controller-manager/values.yaml` | Same. | +| A | `hack/verify-safeguards.sh` | Optional helper: `helm template` each chart and run `kubectl-safeguards` (or equivalent) offline; wired into `Makefile` under a new `verify-safeguards` target. | + +None of the above changes are Automatic-specific — they are strict +generalisations that also apply cleanly on AKS Standard. There is +no chart branch, no conditional templating. + +### 6.5.3 Node auto-provisioning (NAP) + +AKS Automatic uses NAP: nodes come and go as workloads scale. To +avoid controller flap when NAP evicts the leader replica: + +* Deployments carry `spec.replicas: 2` (already the case for the + hub manager; align the member manager if it currently ships as a + single replica). +* Pod anti-affinity (`preferredDuringSchedulingIgnoredDuringExecution`, + `topologyKey: kubernetes.io/hostname`) spreads replicas across + nodes. +* Leader-election lease durations (already tuned in + `cmd/*-net-controller-manager/main.go`) tolerate a ~30s replica + restart window. + +### 6.5.4 e2e coverage + +Phase 4 e2e (§11 below) MUST include at least one AKS Automatic +member alongside AKS Standard members. The test framework in +`test/e2e/framework/cluster.go` needs an `AksSKU` field on the +per-cluster config with `Standard | Automatic` values; the ci-e2e +pipeline stands up one of each and asserts that a `FrontDoorBackend` +programs origins successfully for both. + +### 6.5.5 Documentation + +* `docs/first-party/README.md` — add a short "Member cluster SKUs" + paragraph pointing to Proposal 001 §3.4 and this §6.5. +* `docs/howtos/frontdoor-permissions-setup.md` (added in Phase 5) — + include an "AKS Automatic checklist" appendix mirroring §3.4 of + Proposal 001. + +--- + +## 7. Backward compatibility, versioning, and downgrade + +* Every new field on `FrontDoor*` types is optional with a defaulted + value. `ServiceExport` itself gains no schema change; absence of + the `networking.fleet.azure.com/export-mode` annotation is + equivalent to today's `L4-TrafficManager` behaviour. Existing + `ServiceExport` YAMLs continue to apply and behave identically. +* Downgrade: since both new CRDs are gated by + `--enable-frontdoor-feature`, an operator can downgrade by + disabling the flag and deleting all `FrontDoor*` CRs. The CRD + manifests themselves stay installed (removing them mid-flight + would strand finalizers). +* Storage version: v1alpha1 is `storageversion` in phase 1–4. The + phase-5 promotion adds a conversion webhook only if we introduce + breaking field changes; the plan is to keep v1beta1 field-identical + to v1alpha1 for the first cut so no webhook is needed. + +## 8. East-west (MCS) compatibility + +This proposal is a **north-south** feature (internet → member cluster +via AFD). The pre-existing **east-west** (in-fleet, cluster-to-cluster) +data plane is: + +``` +ServiceExport (member) + → InternalServiceExport (hub, per-cluster shard) + → ServiceImport (hub, aggregated) + → InternalServiceImport (member) + → local ClusterSet IP + imported EndpointSlices + from EndpointSliceExport / EndpointSliceImport +``` + +Traffic between clusters resolves to **pod IPs** via imported +`EndpointSlice`s. It does not traverse any external load balancer, +Traffic Manager, or Front Door. The two directions of traffic share +`ServiceExport` as the entry-point CR but otherwise use disjoint +control paths. + +### 8.1 What is guaranteed to keep working + +| Concern | Guarantee | +|---------|-----------| +| `ServiceImport` aggregation | Untouched. `ServiceImport`, `InternalServiceImport`, and `EndpointSlice{Export,Import}` types are not modified. | +| Pod-to-pod fleet traffic | Uses `EndpointSlice` imports, which carry pod IPs. Neither the addition of an internal LB nor a PLS on the origin `Service` changes pod IPs. | +| Existing consumers of `ServiceExport` | `ServiceExport` schema is **unchanged**. Mode selection is an opt-in annotation (`networking.fleet.azure.com/export-mode`); absence keeps today's `L4-TrafficManager` behaviour. Existing manifests apply and behave identically. | +| MCS `weight` annotation | `networking.fleet.azure.com/weight` (used by MCS aggregation and by the ATM backend) is **not** consumed by the AFD backend controller. AFD weights are declared explicitly on `FrontDoorBackend.spec.weight` and `FrontDoorBackend.status.origins[].weight`. | +| Existing `Service` types | Only `Services` that carry the internal-LB + PLS annotations (either authored by the tenant/GitOps or required by a platform admission policy) trigger the AFD path described in §4.3. A `ServiceExport` opted in via annotation but pointing at a Service without those annotations surfaces `ExportModeAnnotationServiceMismatch` and does not mutate anything. | + +### 8.2 What the AFD path adds on top + +When the member `serviceexport` controller resolves an export to +`L7-FrontDoor` (either by annotation on the `ServiceExport` or by +inference from the Service's own annotations — see §4.3), the +controller does exactly one thing beyond the east-west export that +would have happened anyway: + +* Copy the AKS-programmed PLS resource ID from + `service.beta.kubernetes.io/azure-pls-resource-id` on the + `Service` into `InternalServiceExport.status.privateLinkService`. + +The internal-LB + PLS annotations on the `Service` itself are +authored by the tenant / GitOps or enforced by a platform admission +policy — **not** written by this controller. That preserves the +existing ownership boundary: the tenant owns the `Service`, Fleet +owns the `ServiceExport` → `InternalServiceExport` mirror. + +None of the above touches `EndpointSliceExport`, `EndpointSliceImport`, +or `ServiceImport`. The `Service.ClusterIP` is preserved on a +`type: LoadBalancer` `Service`, so an east-west consumer that +resolves via `ServiceImport` → imported `EndpointSlice` → pod IP is +byte-for-byte unchanged. + +### 8.3 Guardrails to enforce + +Because the member `serviceexport` controller does not mutate the +tenant-owned `Service`, the guardrails become validation-only — +surfaced as conditions on the `ServiceExport` — rather than +mutation refusals: + +* If the `networking.fleet.azure.com/export-mode` annotation is + `L7-FrontDoor` but the `Service` is `type: ExternalName`, headless + (`ClusterIP: None`), or missing the required internal-LB + PLS + annotations, surface + `ServiceExportValid=False, Reason=ExportModeAnnotationServiceMismatch` + and requeue without mutation. +* If inference selects `L7-FrontDoor` (annotation unset, Service has + full internal-LB + PLS annotations) but the `Service` is `type: + ExternalName` or headless, surface + `ServiceExportValid=False, Reason=UnsupportedServiceTypeForFrontDoor`. +* The controller never adds or removes annotations on the `Service`. + All ATM ↔ AFD transitions are driven by the tenant / GitOps + editing the `Service` and/or `ServiceExport`. This avoids the + "who wins" fight between the controller and admission policy. + +### 8.4 Interaction with the ATM backend controller + +`TrafficManagerBackend` reads endpoint IPs from the `Service`'s +external LoadBalancer IP. If a tenant flips the underlying `Service` +to internal + PLS (or opts the export in to AFD via annotation while +the Service is already internal + PLS), the existing +`TrafficManagerBackend` would immediately lose its public IP and +start failing probes. The AFD backend controller MUST refuse to +program AFD origins in this case: + +* If any `TrafficManagerBackend` in the namespace already references + the same `ServiceImport`, refuse to accept a `FrontDoorBackend` + for that `ServiceImport` with reason + `ConflictsWithTrafficManagerBackend`, requiring the user to delete + the ATM backend first (or use a distinct `Service` for AFD). + +This keeps the invariant that at any point in time, a `Service` is +attached to **at most one** north-south surface, while east-west +continues to operate untouched. + +### 8.5 Test coverage for east-west non-regression + +Phase 3 must add integration tests that assert, for a `ServiceExport` +resolved to `L7-FrontDoor` (via annotation or inference): + +* `InternalServiceExport.Spec` is produced identically to the + `L4-TrafficManager` case (byte diff on spec fields). +* `EndpointSliceExport` objects are produced identically. +* East-west round trip (pod-A in cluster-A → `ServiceImport` VIP in + cluster-B → pod-B) still succeeds when the same `Service` is also + fronted by AFD — covered end-to-end in phase 4 e2e as an added + assertion, not a separate test. +* Annotation set to `L7-FrontDoor` against a Service missing + internal-LB + PLS annotations surfaces + `ExportModeAnnotationServiceMismatch` and produces no + `PrivateLinkService` status. + +## 9. Risks and mitigations + +| Risk | Impact | Mitigation | +|------|--------|------------| +| AFD private-endpoint approval races on cluster autoscale | New origins stuck in `Pending` | Reconciler backoff + status surface + doc using `azure-pls-auto-approval` | +| `armcdn` SDK API drift vs. `azure-sdk-for-go` version pinned by `sigs.k8s.io/cloud-provider-azure` | Build breakage | Vendor pin, `go.sum` review; wrap SDK in `pkg/common/azurefrontdoor` interface so an SDK swap is one file | +| WAF policy reference lives in a different subscription than AFD | Cross-sub RBAC errors | Support fully qualified resource ID; controller surfaces `WAFPolicyNotFound` with the exact ID | +| Two hub controllers competing for the same AFD profile | Split-brain writes | Owner-references from backends → profile; single reconciler per resource; `client.OwnerReference` gating | +| SFI review demands additional controls (e.g. mandatory managed identity, mandatory diagnostic settings) | Slippage | Track in open questions §11 of Proposal 001; add controls in phase 5 without blocking phases 1–4 | +| `networking.fleet.azure.com/export-mode` annotation set to `L7-FrontDoor` on a `Service` that lacks the internal-LB + PLS annotations | Silent AFD misconfiguration if the controller falls back to L4 | Controller surfaces `ServiceExportValid=False, Reason=ExportModeAnnotationServiceMismatch` and does not fall back; a platform admission policy (Kyverno / Gatekeeper) can additionally reject the mismatch at write-time to give tenants an immediate error | +| Charts drift from AKS Automatic Deployment Safeguards (missing resource limits, root user, `hostPath`, non-allow-listed image) | Install blocked on Automatic member clusters even when Standard works | Chart hygiene enumerated in §6.5.1; pre-merge `hack/verify-safeguards.sh` runs `helm template` + a policy check offline; Phase 4 e2e installs on at least one Automatic cluster | +| Node auto-provisioning (NAP) on AKS Automatic restarts the leader controller replica during scale events | Reconciliation stalls for the leader-election lease duration on every NAP scale | Two replicas per manager, pod anti-affinity across nodes, and leader-election lease tuned to tolerate a ~30s restart window (§6.5.3) | +| ~~POC `FrontDoorProfile.Spec.Sku` enum permissively accepts `Standard_AzureFrontDoor`~~ | ~~An SFI-intended tenant creates a Standard profile, gets no admission-time rejection, then discovers at backend-creation time that Private Link origins are unavailable~~ | ✅ **Resolved.** CRD enum tightened to `Premium_AzureFrontDoor` only, with `+kubebuilder:default=Premium_AzureFrontDoor`. Standard profiles are now rejected at admission time. Requires `make manifests` regen of `config/crd/bases/networking.fleet.azure.com_frontdoorprofiles.yaml`. | +| POC hosts AFD + ATM controllers in the same pod (shared Workload-Identity subject) | Violates Proposal 001 §7 identity-split invariant; ATM-only tenants inherit AFD write permissions if the shared subject is used in production | Sibling binary+chart split (`cmd/hub-afd-controller-manager`, `charts/hub-afd-controller-manager`) is a hard GA prerequisite (Proposal 003 §2.4). POC installs must be gated by an explicit "non-production" acknowledgement in the chart values and are excluded from SFI-NS253 conformance. | +| `FrontDoorBackend` controller (which enforces the AFD/ATM coexistence invariant — a `ServiceImport` cannot be a backend on both surfaces) does not exist in the POC | Coexistence guard from Proposal 001 §3.5 is unenforced until Phase 4 | Document as an intentional gap in the POC README; e2e in Phase 4 asserts the guard. Nothing in the POC produces AFD origins yet, so the exposure is limited to future manual `az cli` misconfiguration. | +| Custom domain BYOC (Key Vault) reconciliation is not yet implemented | Tenants who set `TLS.Mode: BYOC` see the CR admission-time validation pass but reconciliation surfaces `Programmed=False, Reason=TLSFailed` | Managed mode is documented as the only supported path in the POC. Phase 4 adds Key Vault binding (`AzureKeyVault` secret) reconciliation. | + +## 10. Out-of-scope for this proposal + +* **L4 (non-HTTP/HTTPS) workloads.** AFD is L7-only (HTTP, HTTPS, + WebSockets-over-HTTPS). This proposal therefore does not extend + SFI-NS253 compliance to raw TCP, UDP, gRPC-over-plain-TCP, + databases, SMTP, DNS, etc. Those workloads either stay on ATM + (non-compliant with SFI-NS253) or wait for a separate L4 proposal + (likely Azure Cross-region Load Balancer with Private Link + backends). +* Rules engine / URL rewriting inside AFD. +* Multi-region AFD failover policies (uses AFD-native latency / + weighted / priority load balancing implicitly). +* Custom domains + managed TLS certificates — **deferred to Phase 5 + of this proposal**, not to a separate proposal. See §12 for the + forward-compatible field shape that Phase 2 must reserve. +* IPv6 origins (AFD limitation, not ours). +* An umbrella `GlobalLoadBalancer` CRD unifying ATM and AFD (open + question §11.4 of Proposal 001). + +## 11. Success criteria + +Feature is considered done when **all** of the following hold on +`main`: +1. Installing the sibling AFD chart (`charts/hub-afd-controller-manager`) + yields a Programmed `FrontDoorProfile` with a reachable AFD + endpoint, in <5 minutes. (POC bridge: `--enable-frontdoor-feature=true` + on `hub-net-controller-manager` produces the same effect but does + not satisfy the §7 identity split.) +2. A `FrontDoorBackend` referencing an L7-mode `ServiceImport` reaches + `Accepted=True` and produces one AFD origin per member cluster, + each with `PrivateEndpointStatus=Approved`. +3. `curl https:///` returns 200 from a workload + running on the member cluster, via Private Link. +4. Existing ATM e2e tests continue to pass unchanged. +5. Documentation under `docs/first-party/`, `docs/concepts/`, and + `docs/howtos/` is merged and reviewed by an SFI reviewer. +6. Fleet-networking maintainers approve the design and the SFI-NS253 + KPI dashboard reflects compliance for at least one first-party + adopter. +7. The hub and member charts install cleanly on an **AKS Automatic** + member cluster (Deployment Safeguards in Enforcement mode) and + pass the same e2e as an AKS Standard member. Covered by the + Phase 4 e2e matrix (§6.5.4). diff --git a/docs/first-party/003-pre-implementation-checklist.md b/docs/first-party/003-pre-implementation-checklist.md new file mode 100644 index 00000000..dfed4bc5 --- /dev/null +++ b/docs/first-party/003-pre-implementation-checklist.md @@ -0,0 +1,317 @@ +# Pre-Implementation Checklist for the AFD Proposal + +| Field | Value | +|-------------|----------------------------------------------------------| +| Status | Open | +| Author | @rchinchani_microsoft | +| Created | 2026-07-15 | +| Depends on | [Proposal 001](./001-afd-global-load-balancing.md), [Proposal 002](./002-afd-implementation-plan.md) | + +This document tracks the concrete gates that must clear **before** +Phase 1 code lands (or, where noted, before Phase 2+ lands). Every +item is either an **open design decision**, an **external review / +sign-off**, or a **spike** whose outcome can change the shape of the +code. + +Nothing in Proposals 001 / 002 changes based on this document — it +converts their open questions and unstated assumptions into a +trackable, checkable list. + +> **Reader's note.** As of the reconciliation pass on 2026-07-20 (see +> `.github/.copilot/breadcrumbs/2026-07-20-1108-afd-export-mode-mcs-parity.md`, +> Addendum 2), several items are already `[x] Resolved` by decisions +> that shipped in commit `cb02d14`. The single current hard blocker +> for GA is §2.4 (SFI identity split) — see the readiness table in +> §6. +> +> **Status update (2026-07-20 session, head `629644b`).** +> Every open item below whose blocker was "reconciler / API not yet +> written" is now unblocked by shipped code (WAFPolicy + +> ComplianceMode, `FrontDoorBackend` CRD + reconciler + coexistence +> guard + envtests, member `serviceexport` PLS lookup). The +> **§2.4 identity split** is *structurally* resolved by the sibling +> binary + chart landing (`fcb37f2`, `5672313`, `c219ca2`), but the +> POC bridge in `cmd/hub-net-controller-manager` remains and must +> be removed before GA. Item checkboxes below have NOT been +> re-checked; trust this summary block for current status and see +> the breadcrumb Addendum 3 for the per-commit narrative. + +--- + +## 1. Open design decisions (from Proposal 001 §11) + +Each of these can change the API surface. Resolve before typing +`api/v1alpha1/frontdoorprofile_types.go`. + +- [x] **1.1 `FrontDoorProfile` scope** — **Resolved: namespaced.** + An AFD `FrontDoorBackend` binds an AFD origin to a specific PLS + that fronts a specific `Service` in a specific namespace on a + member cluster. Ownership follows the workload: the app team + that owns the `Service` also owns the `ServiceExport`, + `ServiceImport`, `FrontDoorBackend`, and — for RBAC parity — the + `FrontDoorProfile` too. A cluster-scoped profile would split + ownership between cluster admins (owning the public hostname + + WAF attach) and app teams (owning the origins grafted onto it), + which the controller cannot safely arbitrate across namespaces. + Namespaced also matches the existing `TrafficManagerProfile` + scoping (parity), and the traffic-isolation property is + per-flow so nothing about the Private-Link backbone story + requires a shared cluster-wide profile. Truly shared AFD + profiles remain an operator/IaC concern outside fleet-networking. + - Impact: `+kubebuilder:resource:scope=Namespaced` (already the + working assumption in Proposal 002 §3.1), CRD manifest, + namespaced RBAC in + `charts/hub-net-controller-manager/templates/rbac.yaml`. + - Owner: @rchinchani_microsoft + - Decision: **Namespaced** (2026-07-17) + +- [x] **1.2 WAF policy required at SKU level?** — **Resolved: + required when `spec.complianceMode == SFI-NS253`, optional + otherwise.** The SKU-based gate collapsed once §2.3 of Proposal + 001 scoped the feature to Premium only. Instead of a hidden + annotation, `FrontDoorProfileSpec` gains an explicit + `complianceMode: None | SFI-NS253` field (default `None`, + immutable). When `SFI-NS253`, a CEL rule requires + `spec.wafPolicy` and the `FrontDoorBackend` reconciler + additionally requires `spec.privateLink.enabled = true` on every + referencing backend (surfaced as + `Accepted=False, Reason=SFIComplianceViolation`). This keeps + third-party dev/test paths permissive while making SFI intent + a first-class, kubectl-discoverable, mistype-safe field, and + lets a single Spec field drive both profile-side and + backend-side enforcement. Proposal 002 §3.1 and §4.2 updated + accordingly. + - Owner: @rchinchani_microsoft + - Decision: **Explicit Spec field, default None, immutable** + (2026-07-17) + +- [x] **1.3 Custom domains in phase 2 or phase 5?** — **Resolved: + Phase 2, as a separate CRD.** cb02d14 landed + `FrontDoorCustomDomain` (`afdcd`) alongside `FrontDoorProfile` + rather than growing `FrontDoorProfileStatus` with a + `CustomDomains []` list. This keeps DNS-validation lifecycle + (Pending → Approved) and TLS binding (Managed today; BYOC + reserved) in a dedicated reconciler with its own finalizer, + rather than complicating the profile controller. Route binding + (attaching a validated custom domain to an AFD route) waits + for `FrontDoorBackend` in Phase 4. Proposal 001 §4.1.3 and + Proposal 002 §3.3 describe the shipped shape. + - Owner: @rchinchani_microsoft + - Decision: **Separate CRD in Phase 2** (2026-07-19, + commit cb02d14) + +- [ ] **1.4 Umbrella `GlobalLoadBalancer` CRD later?** — if yes, + invest in shared abstractions in `pkg/common/globalload/` from + day one. If no, keep AFD and ATM helpers strictly separate. + - Impact: package layout of `pkg/common/azurefrontdoor/`; + naming of shared types. + - Owner: — + - Decision: — + +- [x] **1.5 Adding `Spec` to `ServiceExport`** — **Resolved: no + schema change.** The AFD path uses an annotation + (`networking.fleet.azure.com/export-mode: L7-FrontDoor | + L4-TrafficManager`) following the existing + `networking.fleet.azure.com/weight` precedent, and the member + controller additionally infers `L7-FrontDoor` from the + Service's internal-LB + PLS annotations when the annotation + is unset. Precedence: annotation wins when set; a mismatch + between annotation and Service surfaces + `ServiceExportValid=False, + Reason=ExportModeAnnotationServiceMismatch` (no silent + fallback). This preserves upstream mcs-api (KEP-1645) parity + for `ServiceExport` / `MultiClusterService`, which is a + repository preference. Proposals 001 §3.3 / §4.2 and 002 §2.1 + / §3.4 / §4.3 / §8 updated accordingly. + - Owner: @rchinchani_microsoft + - Decision: **Annotation + inference; no `Spec` change** + (2026-07-20) + +- [x] **1.6 AKS Automatic as a supported member cluster SKU** — + **Resolved: yes, first-class alongside AKS Standard.** The AFD + + PLS data plane depends only on cloud-provider-managed + annotations (Standard SKU LB, PLS, `azure-pls-*`), which are + available identically on both SKUs. Two operational caveats + apply: (a) member cluster VNet/subnet layout must be planned + up-front on Automatic (BYO VNet at create-time; PLS subnet + MUST have `privateLinkServiceNetworkPolicies: Disabled`), and + (b) the hub and member Helm charts MUST satisfy AKS Automatic + Deployment Safeguards. Neither is Automatic-specific in the + sense of requiring a code branch — the safeguards-clean chart + is also the correct chart for AKS Standard. Recorded in + Proposal 001 §3.4 and Proposal 002 §6.5. + - Owner: @rchinchani_microsoft + - Decision: **Supported; see 3.7 spike for validation** + (2026-07-20) + +## 2. External review / sign-off gates + +- [ ] **2.1 Fleet-networking maintainer review of PR #373** — + https://github.com/Azure/fleet-networking/pull/373 + - Expect feedback on §1.1, §1.5, and on the feature-flag + default. +- [ ] **2.2 SFI reviewer sign-off on Proposal 001 §7** — this is + the Phase 0 exit criterion in Proposal 002 §6. +- [ ] **2.3 Product / PM sign-off on L4 scoping** — Proposal 001 + §2.3 and Proposal 002 §10 state L4 (raw TCP/UDP) is out of + scope. Confirm no first-party adopter blocks on L4 before + we commit to the phased plan. +- [x] **2.4 Security review of identity split** — Proposal 001 §7 + requires a separate managed identity for AFD vs. ATM. + **Structural blocker uncovered by cb02d14:** the POC hosts + both AFD and ATM controllers in the same pod, so today they + necessarily share one Workload-Identity federated subject — + Kubernetes does not allow two federated identities per pod + (the projected-token path is a pod-level attribute). + Satisfying §7 requires a sibling binary + (`cmd/hub-afd-controller-manager`) + sibling chart + (`charts/hub-afd-controller-manager`). Docs updated to + describe this end state (Proposal 001 §6, §7; Proposal 002 + §2.2, §2.6). Actual code split is a follow-up commit, gated on + security-reviewer sign-off before Phase 5 (GA). + - Impact: adds one new binary, one new chart, one new Docker + image, corresponding CI wiring. No changes required to the + controller packages themselves. + - Owner: @rchinchani_microsoft (docs); TBD (code split) + - Decision: **Sibling binary+chart is a hard GA prerequisite; + POC bridge uses a shared WI subject and is explicitly + non-production** (2026-07-20) + +## 3. Spikes (each ≤ 1 engineer-day) + +Small implementation questions whose answers can invalidate parts of +Proposal 002. Do these **before** committing to Phase 2, but they can +run in parallel with Phase 1 API work. + +- [ ] **3.1 AKS cloud-provider PLS status annotation** — verify that + `service.beta.kubernetes.io/azure-pls-resource-id` (or an + equivalent) is set by the in-tree Azure cloud provider once + the PLS is created. Read the source in + `kubernetes-sigs/cloud-provider-azure`. + - Impact: member `serviceexport` controller reads this to + populate `InternalServiceExport.status.privateLinkService.resourceID`. + If missing, we need a separate ARM call (extra RBAC on the + member). + - Owner: — + - Result: — + +- [x] **3.2 `armcdn` SDK compatibility with pinned `azure-sdk-for-go`** — + **Resolved by cb02d14.** + `github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cdn/armcdn v1.1.1` + was pinned and `go mod tidy` / `go build ./...` succeed + alongside the existing `sigs.k8s.io/cloud-provider-azure` + pin. No dependency conflict. + - Owner: @rchinchani_microsoft + - Result: **compatible** (2026-07-19, commit cb02d14) + +- [ ] **3.3 Cross-subscription PLS auto-approval** — confirm that + listing the AFD subscription in + `service.beta.kubernetes.io/azure-pls-auto-approval` on a + Service in a member VNet in a **different subscription/tenant** + actually results in the private endpoint connection being + auto-approved without a separate ARM call. + - Impact: whether Phase 4 needs to implement an ARM-based + approval fallback (`Microsoft.Network/privateLinkServices/ + privateEndpointConnections` PUT). If yes, member controller + gets additional RBAC. + - Owner: — + - Result: — + +- [ ] **3.4 RBAC minimums for the AFD MI** — verify against + Microsoft.Cdn role definitions: + - Does `CDN Profile Contributor` cover creation + deletion of + `profiles`, `afdEndpoints`, `originGroups`, `origins`, + `routes`, `securityPolicies`? + - Does the AFD MI need any role on the member-cluster PLS + resource group (for the AFD → PLS private endpoint connection + approval call, if 3.3 is a "no")? + - Impact: `docs/howtos/frontdoor-permissions-setup.md`; the + Bicep/az CLI snippets included in that doc. + - Owner: — + - Result: — + +- [ ] **3.5 Dev-sub provisioning** — stand up the test rig: + - One AFD Premium profile RG. + - One WAF policy in Prevention mode. + - Two member AKS clusters with PLS-capable subnets and the + managed identity trust to create PLS resources. + - Impact: needed for Phase 2 integration test in a live sub, + and Phase 4 e2e. + - Owner: — + - Result: — + +- [x] **3.6 Interaction with existing `azcloudconfig` in charts** — + **Resolved: obsoleted by the Workload-Identity + sibling-chart + decision.** cb02d14 picked Azure AD Workload Identity for AFD + auth (env vars `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, + `AZURE_FEDERATED_TOKEN_FILE`, `AZURE_SUBSCRIPTION_ID`), not + `azurecloudconfig.yaml`. Combined with the sibling-chart + decision (§2.4 above), AFD gets its own ServiceAccount, its + own WI federation, and its own values file — the ATM chart's + `azurecloudconfig.yaml` remains untouched. + - Owner: @rchinchani_microsoft + - Result: **WI supersedes azcloudconfig for AFD; ATM chart + unmodified** (2026-07-20) + +- [ ] **3.7 AKS Automatic Deployment Safeguards install validation** — + run `helm template charts/hub-net-controller-manager | kubectl + apply --dry-run=server -f -` (or a `kubectl-safeguards` / + equivalent policy-check tool offline) against the current chart + output. Enumerate every Safeguards violation and confirm each + one can be closed with a values-only or template-only change + (no code changes). Fold the fixes into the §6.5.2 file list of + Proposal 002 before Phase 4 starts. + - Impact: sizing the chart hygiene work in Phase 4; may reveal + that some hub/member containers need an `emptyDir` for `/tmp` + or writable log paths, or that init containers pulling from + Docker Hub have to be re-hosted in `mcr.microsoft.com` / + the tenant ACR. + - Owner: — + - Result: — + +## 4. Nice-to-have before Phase 1 + +- [ ] **4.1 Draft `docs/concepts/HTTPBasedGlobalLoadBalancing/README.md`** + user-facing concept skeleton. Even a stub makes the API + choices in Phase 1 easier to defend during review. +- [ ] **4.2 Set up a per-CRD Prometheus dashboard mock** — the + metric names in Proposal 001 §5.4 should match an actual + Grafana panel we're willing to ship. + +## 5. What is **not** blocking + +Explicitly *not* required before starting Phase 1: + +- Custom domain implementation. +- Full WAF-inline mode implementation (reference-only is fine for the + first PR). +- End-to-end infrastructure automation for the dev sub (Bicep of the + RG can come with the howto doc). +- Umbrella-CRD refactor (only relevant if §1.4 resolves to "yes"). + +## 6. Overall readiness verdict + +| Category | Status | +|----------------------------|--------| +| Design coherent | ✅ | +| Scope and phases documented | ✅ | +| Open design decisions closed | ⏳ (§1.4 only) | +| Maintainer review | ⏳ (§2.1) | +| SFI sign-off | ⏳ (§2.2) | +| **SFI identity split (sibling binary+chart)** | ⏳ **hard GA blocker (§2.4)** | +| SDK / cloud-provider spikes | ⏳ (§3.1, §3.3, §3.4) | +| Dev-sub ready | ⏳ (§3.5) | +| AKS Automatic install validated | ⏳ (§3.7) | +| POC (cb02d14) reconciled with docs | ✅ (2026-07-20) | + +**Recommendation:** start Phase 1 (API types + defaulters + CEL, no +controllers) *only after* §1.1, §1.2, §1.5, §2.1 are closed. Phases +2–4 additionally require §3.1–3.4 and §3.5. Phase 4 additionally +requires §3.7 (AKS Automatic Deployment Safeguards) to be run and +its findings folded into the chart hygiene work. **Phase 5 (GA) +additionally requires §2.4** — no SFI-NS253 install can be declared +until the sibling AFD binary+chart replace the current in-binary +bridge. + +Update the checkboxes above as items close; when every box in §1–§3 +is checked, Phase 2 can begin. diff --git a/docs/first-party/004-poc-runbook.md b/docs/first-party/004-poc-runbook.md new file mode 100644 index 00000000..231cd821 --- /dev/null +++ b/docs/first-party/004-poc-runbook.md @@ -0,0 +1,114 @@ +# 004 — AFD First-Party POC Runbook + +> **Status:** Draft — companion to [001](./001-afd-global-load-balancing.md), +> [002](./002-afd-implementation-plan.md), [003](./003-pre-implementation-checklist.md). +> +> **Audience:** Engineers who want to smoke-test the AFD first-party controller +> **without** access to a locked-down Azure Kubernetes Fleet Manager hub. All +> tiers below run against plain AKS cluster(s) that you own. + +The controller is a normal Kubernetes controller-runtime binary. It only needs +credentials that can read the hub-cluster API server and can write to Azure +Front Door. It does not need to run *on* a Fleet Manager hub — any AKS cluster +where you can install Helm charts will do. + +--- + +## Tier 0 — 30-minute smoke test (recommended first) + +**Goal:** Prove the `FrontDoorProfile` reconciler wires a CR through to a real +AFD Premium profile + WAF SecurityPolicy in Azure. Exercises PATCH 03 (CRDs), +PATCH 04 (SDK bundle), PATCH 05 (chart+binary), PATCH 06 (profile reconciler). + +**Cost:** ~$2/day for the AKS cluster + AFD Premium base fee. + +**Prereqs on your workstation:** +- `az` (>= 2.60), `kubectl`, `helm` (>= 3.14), `docker`, `git`. +- An Azure subscription where you can create resource groups and AFD profiles. +- `az login` completed. + +**Automated:** `hack/e2e-afd-poc.sh` performs the steps below idempotently. +Read the script comments before running — it creates billable Azure resources. + +Manual outline: +1. `az group create -n -l ` +2. `az aks create -n -g --enable-oidc-issuer --enable-workload-identity --node-count 2 --generate-ssh-keys` +3. `az identity create -n hub-afd-uami -g ` — grant it `Contributor` on the RG. +4. `az identity federated-credential create` — federate to + `system:serviceaccount:fleet-system:hub-afd-controller-manager`. +5. `kubectl apply -f config/crd/bases/networking.fleet.azure.com_frontdoor*.yaml` +6. Build & push the controller image (or reuse a prebuilt tag): `make docker-build-hub-afd-controller-manager`. +7. `helm install hub-afd charts/hub-afd-controller-manager --set azure.tenantId=… --set azure.clientId= --set azure.subscriptionID=… --set image.repository=… --set image.tag=…` +8. `kubectl apply -f -` a minimal `FrontDoorProfile` CR (WAF + ComplianceMode + defaults). See sample in the script. +9. Wait for `.status.conditions[?(@.type=="Ready")].status == "True"`. +10. Verify in Azure: `az afd profile show`, `az afd security-policy list`. + +**Success signal:** AFD Premium profile exists in Azure, WAF policy is +attached via a SecurityPolicy, and the CR's `.status.profileId` contains the +ARM resource ID. + +**Teardown:** `az group delete -n --yes --no-wait` (removes AKS, UAMI, +AFD profile in one call). + +--- + +## Tier 1 — 2-hour single-cluster full L7 flow + +**Goal:** Exercise the end-to-end path — `ServiceExport` on a member cluster +resolves the workload's Private Link Service, `InternalServiceExport` carries +the PLS ARM ID to the hub, `FrontDoorBackend` provisions the AFD OriginGroup + +Origin with a `SharedPrivateLinkResource`. Exercises PATCH 08. + +**Prereqs (in addition to Tier 0):** +- The AKS cluster's node subnet has `privateLinkServiceNetworkPolicies=Disabled`. +- AKS system-assigned identity has `Network Contributor` on that subnet. + +Outline: +1. Deploy an nginx `Service type=LoadBalancer` with the + [cloud-provider-azure PLS annotations](https://cloud-provider-azure.sigs.k8s.io/topics/pls-integration/): + `service.beta.kubernetes.io/azure-load-balancer-internal: "true"`, + `service.beta.kubernetes.io/azure-pls-create: "true"`, + `service.beta.kubernetes.io/azure-pls-name: "nginx-pls"`. +2. Wait for the PLS to be created (`az network private-link-service list -g MC___`). +3. Apply a `ServiceExport` on the same namespace/service with annotation + `networking.fleet.azure.com/export-mode: "L7"` (see PATCH 04 for annotation + contract). +4. Use the same AKS cluster as both hub and member (skip real Fleet join by + creating the `InternalServiceExport` manually in the reserved member + namespace, wiring the PLS ARM ID onto `.spec.privateLinkServiceID`). +5. Apply a `FrontDoorBackend` CR referencing the service. +6. Watch the hub controller reconcile OriginGroup + Origin; + `az afd origin list` should show the origin with `sharedPrivateLinkResource` + populated. In Azure Portal, approve the pending PLS connection request. + +**Success signal:** `curl https://.z01.azurefd.net/` returns +nginx welcome page **over the AFD → PLS private path** (no public IP on the +service). + +--- + +## Tier 2 — day-long two-cluster real Fleet flow + +**Goal:** Exercise the real multi-cluster propagation — two AKS clusters, one +Fleet Manager hub, real `ServiceExport` → `InternalServiceExport` propagation +via the fleet-networking member agent. + +Only needed to validate the member/hub split. Use Tier 1 for reconciler +development iterations; Tier 2 for release-gate validation. + +Follow the standard fleet-networking e2e setup under +[`test/scripts/`](../../test/scripts/) but substitute the AFD chart for (or +alongside) the traffic-manager chart, and enable the AFD annotation on the +`ServiceExport`. + +--- + +## Reference + +- `hack/e2e-afd-poc.sh` — Tier 0 (and stub of Tier 1) automation. +- Chart values keys: `azure.tenantId`, `azure.clientId`, `azure.subscriptionID`, + `image.repository`, `image.tag`. See + [`charts/hub-afd-controller-manager/values.yaml`](../../charts/hub-afd-controller-manager/values.yaml). +- SFI-NS253 §7 rule: the AAD identity for `hub-afd-controller-manager` **must + be distinct** from the identity used by `hub-net-controller-manager`. diff --git a/docs/first-party/README.md b/docs/first-party/README.md new file mode 100644 index 00000000..eb550c00 --- /dev/null +++ b/docs/first-party/README.md @@ -0,0 +1,42 @@ +# First-Party AKS Support in Fleet Networking + +This folder tracks design proposals and rollout plans for enabling +`fleet-networking` to be used by **first-party AKS clusters** (services +owned by Microsoft that are themselves offered as an Azure service). + +First-party workloads are subject to additional security and networking +requirements enforced by +[SFI (Secure Future Initiative)](https://eng.ms/docs/initiatives/project-standard/standards-categories/sc-networking/ddos/sfi-ns/sfi-ns253-kpi), +notably **SFI-NS253**, which mandates that any internet-facing entry +point for a first-party workload must sit behind: + +1. **Azure Front Door (AFD) Standard or Premium**, terminating TLS at + the edge with an attached +2. **Web Application Firewall (WAF)** policy, and reaching origins over +3. **Azure Private Link** — the public IP on the origin (AKS ingress / + Service) must be removed. + +The current fleet-networking data plane satisfies neither (1) nor (3): +the only supported global load-balancing surface today is **Azure +Traffic Manager (ATM)**, which is DNS-based and always returns the +public IP of a Service on each member cluster. + +The proposals in this folder describe how to close that gap. + +## Proposals + +| # | Title | Status | +|---|-------|--------| +| [001](./001-afd-global-load-balancing.md) | Azure Front Door + WAF + Private Link based Global Load Balancing | Draft | +| [002](./002-afd-implementation-plan.md) | Implementation plan and file-by-file scope of changes for Proposal 001 | Draft | +| [003](./003-pre-implementation-checklist.md) | Pre-implementation checklist — open design decisions, sign-offs, and spikes gating code | Open | + +## Non-goals of this folder + +* This folder is not a substitute for the SFI onboarding checklist — + each first-party service that adopts fleet-networking must still + complete the SFI review with its own service tree entry. +* This folder does not document customer-facing (third-party) usage of + fleet-networking. Existing docs under `docs/concepts`, + `docs/howtos`, and `docs/demos` remain the source of truth for that + audience.