diff --git a/hyperfleet/components/adapter/framework/adapter-label-stamping.md b/hyperfleet/components/adapter/framework/adapter-label-stamping.md new file mode 100644 index 00000000..3b320337 --- /dev/null +++ b/hyperfleet/components/adapter/framework/adapter-label-stamping.md @@ -0,0 +1,57 @@ +--- +Status: Draft +Owner: HyperFleet Team +Last Updated: 2026-06-16 +--- + +# Adapter Automatic Label and Annotation Stamping + +> Part of [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) + +**Current state**: The framework defines a set of standard `hyperfleet.io/*` label and annotation keys as string constants in `pkg/constants/constants.go` (`hyperfleet.io/generation`, `adapter`, `cluster-id`, `created-by`). These are a shared reference for adapter authors — the framework does not inject them. Each adapter's manifest template must add them explicitly by hand. The only exception is `hyperfleet.io/generation`: this annotation is mandatory and validated at apply time by `internal/manifest/generation.go`, which refuses to apply any manifest missing it or carrying a non-numeric value. + +This means label coverage across adapter-created resources is inconsistent: adapters that omit `hyperfleet.io/adapter` or `hyperfleet.io/cluster-id` from their templates are invisible to any tooling that relies on those labels for discovery, ownership tracking, or sweep-based cleanup. + +**Proposal**: The framework injects the standard set of `hyperfleet.io/*` labels and annotations at apply time, before the manifest reaches the transport client. Adapter authors no longer need to add them manually. The merge strategy is fill-gaps-only: labels already present in the adapter's manifest template take precedence, so intentional overrides are preserved and no existing adapter is broken. + +**Standard labels stamped on every resource:** + +| Label / Annotation | Value Source | Notes | +|---|---|---| +| `hyperfleet.io/adapter` | Adapter config: `adapter.name` | Identifies the adapter instance managing this resource | +| `hyperfleet.io/resource-id` | TBD — pending HYPERFLEET-896 alignment | Stable identifier across recreations | +| `hyperfleet.io/generation` | Event param: `generation` | Already enforced; automatic stamping is a no-op if already present | + +**Implementation**: A `stampLabels(manifest, frameworkLabels)` function called in `resource_executor.go` before `ApplyResource()`. Works identically for K8s and Maestro. + +## Alternatives Considered + +### §2 — Automatic Label Stamping + +#### Enforce at apply time + +**What**: Similar to how enforcement of `generation` label, check the other required labels are set before applying + +**Why Rejected**: Reduces the developer experience by adding these HyperFleet specific concerns when creating adapter tasks and pollutes manifests files. + +#### Fail Fast at Config Load + +**What**: Instead of injecting missing labels at runtime, validate at adapter startup that every manifest template includes the required `hyperfleet.io/*` labels. Fail to start if any are missing. + +**Why Rejected**: Manifest templates are Go templates rendered at event time with per-event params — static analysis cannot guarantee the rendered output will contain a label that is computed from a template expression. Validation would have to be done on the rendered manifest at apply time anyway, which is equivalent to the injection approach. Runtime injection is strictly less breaking: adapters that already include the labels are unaffected; adapters that omit them gain coverage automatically. + +#### Kubernetes Admission Webhook + +**What**: Deploy a mutating admission webhook on the management cluster that stamps `hyperfleet.io/*` labels on any resource created by an adapter service account. + +**Why Rejected**: Works only for the direct Kubernetes transport path — Maestro ManifestWork objects are created on the management cluster, but the labels need to appear on the ManifestWork itself, not on the nested spoke-cluster manifests that Maestro eventually applies. The webhook cannot reach those. A webhook also requires cluster-level infrastructure (certificate rotation, webhook registration) that is out of scope for an adapter framework change. Framework-side injection covers both transports uniformly. + +## Related Documentation + +- [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) — Main index document +- [Adapter Lifecycle Gates](./adapter-lifecycle-gates.md) — §1: Lifecycle Gates +- [Adapter Resilience Model](./adapter-resilience-model.md) — §3: Resilience Model +- [Adapter Stuck Detection](./adapter-stuck-detection.md) — §4: Stuck Detection +- [Adapter Periodic Execution](./adapter-periodic-execution.md) — §5: Periodic Execution +- [Adapter Resource Retention](./adapter-resource-retention.md) — §6: Resource Retention +- [Adapter Sweep Controller](./adapter-sweep-controller.md) — §7: Sweep Controller diff --git a/hyperfleet/components/adapter/framework/adapter-lifecycle-gates.md b/hyperfleet/components/adapter/framework/adapter-lifecycle-gates.md new file mode 100644 index 00000000..02704f21 --- /dev/null +++ b/hyperfleet/components/adapter/framework/adapter-lifecycle-gates.md @@ -0,0 +1,219 @@ +--- +Status: Draft +Owner: HyperFleet Team +Last Updated: 2026-06-16 +--- + +# Adapter Lifecycle Gates + +> Part of [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) + +Adapter resources phase allows per resource lifecycle operations determined by CEL expression gates (`lifecycle.recreate.when`, `lifecycle.delete.when`) evaluated against discovered resource state. This proposal introduces new gates following the same pattern for: +- Executing or not an individual resource (vs using `precondition`, which skips the whole resources phase) +- Applying patches to resources instead of full apply +- Orphan resources, so they remain alive but are no longer managed by HyperFleet + +## 1.1 Resource-Level When Gate + +A resource-level `when` expression is evaluated before any lifecycle gate. If it evaluates to false, the executor skips all actions for that resource in the current event — no apply, no patch, no recreate. + +```yaml +resources: + - name: "provisioningJob" + when: + expression: "..." + lifecycle: + ... +``` + +This gate follows the same CEL evaluation context as all other lifecycle expressions: extracted params, discovered resources, and adapter metadata are all available. + +### Discovery always runs + +Discovery is executed before evaluating `when`. The executor still discovers the current state of the resource and makes it available as `resources.provisioningJob` (an optional value, absent if nothing was found). This means: + +- Values can be used to define the `when` expression on the resource itself. + - E.g. providing a "create only" feature by skipping any operation once a resource is discovered + - Provide some "throttling" for rapid events, checking on the last execution report of the same adapter + - E.g. skip adapter1 resource if `last_updated_time` occured within some time window +- Sibling resources can read `resources.provisioningJob` in their own `when` expressions, `lifecycle.*` expressions, and manifest templates, regardless of whether this resource's `when` gate passed. + +### Replace `precondition` + +A precondition failure is event-scoped: if a precondition evaluates to false or errors, the executor aborts the entire resources phase — all remaining resources in the event are skipped, not just the one the precondition was intended to guard. This makes preconditions the right tool for event-wide guards ("abort if the cluster is not yet registered") but the wrong tool for resource-specific guards. + +The resource-level `when` gate is resource-scoped: if it evaluates to false, only that resource's modification actions are skipped. The executor continues to the next resource in the list without interruption. Other resources are discovered, evaluated, and applied normally. This matters when resources in the same event are logically independent — a debounce on a provisioning job should not prevent an annotation patch on a separate namespace resource from running in the same event. + +Resource level `when` conditions can replace the `precondition` feature. There are some tradeoffs: +- For adapter configs with many resources, a shared `precondition` avoids repetition at every resource +- Increase CEL complexity because of mixing at resource level `when` the conditions for a global skip and specific skip. + - This can be alleviated by defining a variable like `skipAll` and then at resource level combining like `skipAll && specific-when-for-resource` + +## 1.1 Gate apply operations `lifecycle.apply.when` + +`lifecycle.apply.when` gates whether the apply step runs for a given resource. If the expression evaluates to false, the apply step is skipped and the executor continues to the next resource. + +**Default is True when absent**: the apply step always runs when there is no explicit `lifecycle.apply` defined. Only when defined and evaluates false, the apply is skipped. + +The existing generation-aware logic determines the specific action — create if the resource does not exist, skip if the generation is unchanged, update or recreate if the generation has advanced. `lifecycle.apply.when` is an additional gate evaluated before that logic; omitting it leaves the generation-aware behavior unchanged. + +The primary use case is create-only behaviour: only write the resource if it does not already exist. + +```yaml +resources: + - name: "bootstrapNamespace" + transport: + client: "kubernetes" + manifest: + apiVersion: v1 + kind: Namespace + metadata: + name: "{{ .clusterId }}" + annotations: + hyperfleet.io/generation: "{{ .generation }}" + discovery: + by_name: "{{ .clusterId }}" + lifecycle: + apply: + when: + expression: "!resources.?bootstrapNamespace.hasValue()" +``` + +The expression has access to the same CEL context as `lifecycle.recreate.when`: extracted params, discovered resources, and adapter metadata. Discovery must be configured so that the current resource state is available in `resources.*` for the expression to evaluate against. + + +## 1.3 Patch resources `lifecycle.patch` + +Patch is an operation on an already-existing resource, not a creation strategy. `lifecycle.patch` is an array — each entry has its own `when` gate and `document`. The executor evaluates every entry independently and applies all whose `when` evaluates to true, in order. + +Use this feature if you need for smaller changes than the apply operation. + +```yaml +resources: + - name: "clusterNamespace" + transport: + client: "kubernetes" + discovery: + by_name: "{{ .clusterId }}" + lifecycle: + patch: + - when: + expression: > + resources.?clusterNamespace.hasValue() + && dig(resources.clusterNamespace.metadata.annotations, "example.io/status") != clusterStatus + document: + metadata: + annotations: + example.io/status: "{{ .clusterStatus }}" + + - when: + expression: > + resources.?clusterNamespace.hasValue() + && dig(resources.clusterNamespace.metadata.labels, "example.io/tier") != tier + document: + metadata: + labels: + example.io/tier: "{{ .tier }}" +``` + +The first entry fires only when the `example.io/status` annotation on the discovered resource differs from the current `clusterStatus` param — the expression reads the live value off the discovered resource and compares it to the desired value. The second entry fires independently when the label drifts. Both can apply in the same event; neither fires if the discovered state already matches. + +Each `document` is the patch body, rendered through the normal Go template pipeline and sent as a JSON merge patch (RFC 7396): fields present are set, fields absent are left untouched on the server, fields explicitly set to `null` are removed. + +`lifecycle.patch` and the normal apply path are mutually exclusive per resource: if any patch entry fires, the standard apply step is skipped for that resource. This is consistent with how `lifecycle.recreate.when` and `lifecycle.delete.when` take precedence over the default path. + +**Transport**: Maestro's `PatchManifestWork` already uses JSON merge patch internally. For K8s, the transport client issues a `Patch` call with `types.MergePatchType`. The `PatchResourceLabels` primitive added for `lifecycle.orphan.when` provides the same transport method. + +**Note on recreation**: `recreate_on_change: true` is superseded by `lifecycle.recreate.when` from [Adapter Recreation Flow Design](./adapter-recreation-flow-design.md) (HYPERFLEET-837). Recreation is a lifecycle gate, not a strategy. + +### 1.3.1. Alternatives + +First of all, using Patch operations is an advance use case, so ideally we should prove the necessity of implementing patch and not select a complex/very flexible solution. + + +#### Strategic Merge Patch + +**What**: Use Kubernetes Strategic Merge Patch (SMP) instead of JSON Merge Patch (RFC 7396) as the patch type for `lifecycle.patch`. + +**Why Rejected**: SMP can append to arrays (e.g., adding a container to a Pod spec) rather than replacing them, which is more powerful for some K8s object types. However, SMP is a Kubernetes-specific extension and is not supported by Maestro's `PatchManifestWork`, which uses JSON merge patch internally. Using SMP would require a per-transport patch type switch, or restricting `lifecycle.patch` to K8s-only. JSON merge patch works for the documented use cases (patching metadata annotations and labels) and keeps transport behavior uniform. + +#### Server-Side Apply (SSA) + +**What**: Replace `lifecycle.patch` with Server-Side Apply using a dedicated field manager, letting the API server track which fields the adapter owns and detect conflicts with other managers. + +**Why Rejected**: SSA provides the strongest ownership semantics and conflict detection, but requires the transport client to maintain field manager state across events and handle conflict responses (409). It also does not map to Maestro's patch API. The use cases for `lifecycle.patch` — conditional annotation and label updates on already-existing resources — do not require conflict detection: the `when` expression reads the live value and only fires when a diff is detected. SSA complexity is not justified by these cases. + +## 1.4 Detach resources (lifecycle.orphan.when) + +An adapter can currently "release" a resource without deleting it: skip the delete step in `lifecycle.delete.when` and report `Finalized=True` in the post-action status conditions. The platform sees `Finalized=True` and stops sending events for that resource. + +But the existing resource is left with HyperFleet specific labels, which can cause problems with cleaning jobs identifying resources as orphans to be deleted. + +`lifecycle.orphan.when` makes this a first-class lifecycle operation. The expression is typically gated on the event being a deletion and the resource still carrying its management labels — confirming the resource is still managed and this event is the handoff point. + +```yaml +resources: + - name: "provisioningJob" + transport: + client: "kubernetes" + discovery: + by_name: "{{ .clusterId }}-job" + lifecycle: + orphan: + when: + expression: > + is_deleting + && resources.?provisioningJob.hasValue() + && "hyperfleet.io/adapter" in resources.provisioningJob.metadata.labels +``` + +When `lifecycle.orphan.when` evaluates to true, the executor: + +1. Discover existing resources +2. Skips the delete step entirely +3. Issues a label-strip patch via the transport client, removing `hyperfleet.io/*` labels and annotations. + +The label presence check in the expression (`"hyperfleet.io/adapter" in resources.provisioningJob.metadata.labels`) ensures the gate is idempotent: if the event is redelivered after labels have already been stripped, the expression evaluates to false and neither delete nor orphan fires. + +**Precedence**: `lifecycle.orphan.when` is evaluated before `lifecycle.delete.when`. If orphan fires, delete is skipped. + +**Implementation**: Add `orphan.when` CEL field to `LifecycleConfig`. Add a `PatchResourceLabels(ctx, gvk, namespace, name, remove []string, target)` method to the `TransportClient` interface for the label-strip operation. + +**Maestro compatibility**: For Maestro ManifestWork resources, the label-strip is applied to the ManifestWork object itself, not to the nested manifests. The sweep controller must account for this when reconciling Maestro-managed resources. + +### 1.4.1 Alternatives + +#### Recreate Without HyperFleet Labels + +**What**: Instead of stripping labels from the existing resource in place, delete it and recreate it without the `hyperfleet.io/*` labels. The new resource carries no HyperFleet ownership markers and is invisible to the sweep controller. + +**Why Rejected**: Recreation contradicts the fundamental semantic of the orphan operation. Orphan means preserve the existing resource without disruption — delete + create does the opposite: + +- **Cascading deletion**: Deleting the resource triggers GC on all children (ownerReferences, finalizers). A Namespace deletion removes everything inside it. For Maestro ManifestWork, deletion propagates to all spoke-managed manifests — destroying exactly what the operation is meant to preserve. +- **Identity break**: Recreation produces a new UID, `creationTimestamp`, and `resourceVersion`. Any external system holding a reference by UID, ownerReference, or watch sees the original as gone and the replacement as an unrelated object. +- **Race window**: Between delete and create, the sweep controller may observe the labeled resource disappearing and treat cleanup as complete, or the new unlabeled resource may arrive before HyperFleet stops sending events and be double-managed. + +The label-strip patch is the correct primitive because it is in-place: the resource keeps its identity, its children, its running state, and its UID. Only HyperFleet's claim is removed. + +#### Alternative Names to orphan + +The term "orphan" carries two conflicting meanings in this domain — Kubernetes uses it for the GC `propagationPolicy: Orphan` (leave children in place when parent is deleted), while this design uses it to mean "release a resource from HyperFleet management." This ambiguity is a readability risk for adapter authors familiar with K8s. + +| Name | Rationale | Concern | +|---|---|---| +| `lifecycle.release.when` | "Release from management" reads naturally in context | `release` is also used for software releases; ambiguous in a CI/CD platform context | +| `lifecycle.disown.when` | Precise: strip ownership, leave in place | Less common English; may read as aggressive/negative | +| `lifecycle.handoff.when` | Conveys transfer of responsibility to another owner | Implies a receiver exists; disownment is unconditional | +| `lifecycle.finalize.when` | Mirrors the `Finalized=True` condition it produces | `Finalized` in K8s means object is being deleted (finalizer cleared) — opposite meaning | + +**Recommendation to revisit**: `lifecycle.release.when` is the least ambiguous in context and avoids the K8s GC collision. If the project glossary settles on "release" for this concept, rename accordingly. Until then, `orphan.when` is retained as the design name with an explicit note in the glossary entry distinguishing it from `propagationPolicy: Orphan`. + +## Related Documentation + +- [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) — Main index document +- [Adapter Label Stamping](./adapter-label-stamping.md) — §2: Automatic Label and Annotation Stamping +- [Adapter Resilience Model](./adapter-resilience-model.md) — §3: Resilience Model +- [Adapter Stuck Detection](./adapter-stuck-detection.md) — §4: Stuck Detection +- [Adapter Periodic Execution](./adapter-periodic-execution.md) — §5: Periodic Execution +- [Adapter Resource Retention](./adapter-resource-retention.md) — §6: Resource Retention +- [Adapter Sweep Controller](./adapter-sweep-controller.md) — §7: Sweep Controller diff --git a/hyperfleet/components/adapter/framework/adapter-lifecycle-management-design.md b/hyperfleet/components/adapter/framework/adapter-lifecycle-management-design.md new file mode 100644 index 00000000..401ae4be --- /dev/null +++ b/hyperfleet/components/adapter/framework/adapter-lifecycle-management-design.md @@ -0,0 +1,87 @@ +--- +Status: Draft +Owner: HyperFleet Team +Last Updated: 2026-06-15 +--- + +# Adapter Resource Lifecycle Management Design + +**Jira**: [HYPERFLEET-827](https://issues.redhat.com/browse/HYPERFLEET-827) · [HYPERFLEET-1065](https://issues.redhat.com/browse/HYPERFLEET-1065) + +## What & Why + +**What**: Add per-resource lifecycle configuration to the adapter framework so adapter authors can control what happens after initial resource creation: how resources are updated, how failures are handled, and how resources are cleaned up. + +**Why**: The adapter framework creates real infrastructure (Kubernetes objects, Maestro ManifestWork resources) but treats each event as a fresh operation with no memory of prior state: +- No configurable update strategy — adapter authors cannot express "recreate on generation change" vs "apply in place" vs "never write once created" +- No declarative way to stop managing a resource — adapter authors cannot express "release this resource from HyperFleet management without deleting it" +- No retry for resource operations — a transient K8s API server error fails the entire event and depends on broker redelivery +- No persistent state — execution context is in-memory per event; no state survives across event executions +- No automatic label stamping — standard `hyperfleet.io/*` labels are convention, not enforcement, so sweep-based cleanup cannot be built reliably + +**Related Documentation:** +- [Adapter Framework Design](./adapter-frame-design.md) — Core executor architecture +- [Adapter Recreation Flow Design](./adapter-recreation-flow-design.md) — Recreation flow specifics (HYPERFLEET-837) +- [Adapter Status Contract](./adapter-status-contract.md) — Status reporting patterns + +### Scope + +This design covers seven proposals: + +1. **Lifecycle gates** (`§1`) — [adapter-lifecycle-gates.md](./adapter-lifecycle-gates.md): four new per-resource CEL-gated operations following the existing `lifecycle.recreate.when` / `lifecycle.delete.when` pattern: + - resource-level `when` (`§1.1`) — evaluated before any lifecycle gate; if false, all modification actions (apply, patch, recreate) are skipped for that resource in the current event; primary use case is debouncing using CEL timestamp arithmetic + - `lifecycle.apply.when` (`§1.2`) — gate whether the apply step runs (create-only use case) + - `lifecycle.patch` (`§1.3`) — array of conditional JSON merge patches on an already-existing resource + - `lifecycle.detach.when` (`§1.4`) — conditional resource disownment: strip `hyperfleet.io/*` management labels and report `Finalized=True` without deleting + +2. **Automatic label and annotation stamping** (`§2`) — [adapter-label-stamping.md](./adapter-label-stamping.md): the framework injects standard `hyperfleet.io/*` labels on all adapter-created resources at apply time, making them consistently discoverable for tooling and sweep-based cleanup + +3. **Resilience model** (`§3`) — [adapter-resilience-model.md](./adapter-resilience-model.md): documents why per-resource retry is not implemented in the adapter and how the Sentinel reconciliation cycle (convergence at ~10s, drift-check at 30 minutes) covers both failure cases, with `Available=False` closing the drift-check corner case + +4. **Stuck detection** (`§4`) — [adapter-stuck-detection.md](./adapter-stuck-detection.md): unified reconciliation metrics at the API layer via [HYPERFLEET-1205](https://redhat.atlassian.net/browse/HYPERFLEET-1205), covering both deletion and create/update flows via `Reconciled=False` condition tracking + +5. **Periodic execution** (`§5`) — [adapter-periodic-execution.md](./adapter-periodic-execution.md): enables detection of adapter config changes (`adapter.version`) and framework binary changes (`adapter.frameworkVersion`) by including both as `adapter_config_version` and `adapter_framework_version` in every status report; values are stored in the raw statuses endpoint and not surfaced in the customer-facing conditions view; adapter config authors capture last-reported values via a precondition API call and use lifecycle gate expressions to decide what runs on a version change; this design also renames `adapter.version` from a framework binary constraint to an author-declared config version, introducing `adapter.frameworkVersion` as its replacement + +6. **Resource retention** (`§6`) — [adapter-resource-retention.md](./adapter-resource-retention.md): per-resource `retention:` configuration that governs how many historical versions of a resource accumulate and for how long; enables recreate-with-history semantics (versus the default delete-before-create replace mode), multi-result label-based discovery, and debounce protection against rapid version creation + +7. **Sweep controller** (`§7`) — [adapter-sweep-controller.md](./adapter-sweep-controller.md): a background controller that periodically detects and removes K8s and Maestro resources orphaned by HyperFleet API force-deletes, using the `hyperfleet.io/adapter` and `hyperfleet.io/resource-id` labels stamped automatically by the framework to identify and verify each managed resource against the API + +--- + +## Current State + +The adapter executes a 4-phase sequential pipeline per CloudEvent: + +1. **Parameter Extraction** — extracts params from event data and env vars +2. **Preconditions** — evaluates CEL or structured conditions; supports API calls with retry/backoff +3. **Resources** — applies, discovers, and deletes resources sequentially +4. **Post-Actions** — CEL-gated HTTP calls or log entries; always runs for error reporting + +### What Already Exists + +| Capability | Location | Notes | +|---|---|---| +| `recreate_on_change: bool` | `internal/configloader/types.go` | Delete+create when generation changes; superseded by `lifecycle.recreate.when` (HYPERFLEET-837) | +| `lifecycle.delete.when` CEL expression | `resource_executor.go` | Gated deletion ordering per resource | +| `lifecycle.delete.propagationPolicy` | `transportclient/types.go` | Background / Foreground / Orphan (K8s GC sense only) | +| Generation-aware idempotent apply | `internal/manifest/generation.go` | Create / Skip (same gen) / Update / Recreate | +| Retry/backoff for precondition API calls | `internal/hyperfleetapi/client.go` | Exponential, linear, constant + ±10% jitter; 3 attempts default | +| `hyperfleet.io/*` label constants | `pkg/constants/constants.go` | String constants for label/annotation keys (`hyperfleet.io/generation`, `managed-by`, `cluster-id`, `created-by`). Used as a shared reference — the framework does **not** inject these automatically; each adapter's YAML manifest template must add them explicitly | +| `hyperfleet.io/generation` enforcement | `internal/manifest/generation.go` | The one exception: this annotation is mandatory and validated at apply time. The adapter refuses to apply any manifest that is missing it or has a non-numeric value | +| Label/annotation stamping | — | **Not automatic.** No framework mechanism injects `hyperfleet.io/*` labels into manifests. Adapter authors must add them to their YAML templates by hand. This design adds automatic stamping — see [Automatic Label and Annotation Stamping](#automatic-label-and-annotation-stamping) below | + +--- + +## Design + +Each proposal is documented in its own file: + +| Proposal | Document | +|---|---| +| §1 — Lifecycle Gates | [adapter-lifecycle-gates.md](./adapter-lifecycle-gates.md) | +| §2 — Automatic Label Stamping | [adapter-label-stamping.md](./adapter-label-stamping.md) | +| §3 — Resilience Model | [adapter-resilience-model.md](./adapter-resilience-model.md) | +| §4 — Stuck Detection | [adapter-stuck-detection.md](./adapter-stuck-detection.md) | +| §5 — Periodic Execution | [adapter-periodic-execution.md](./adapter-periodic-execution.md) | +| §6 — Resource Retention | [adapter-resource-retention.md](./adapter-resource-retention.md) | +| §7 — Sweep Controller | [adapter-sweep-controller.md](./adapter-sweep-controller.md) | diff --git a/hyperfleet/components/adapter/framework/adapter-periodic-execution.md b/hyperfleet/components/adapter/framework/adapter-periodic-execution.md new file mode 100644 index 00000000..bf04550b --- /dev/null +++ b/hyperfleet/components/adapter/framework/adapter-periodic-execution.md @@ -0,0 +1,129 @@ +--- +Status: Draft +Owner: HyperFleet Team +Last Updated: 2026-06-22 +--- + +# Adapter Periodic Execution + +> Part of [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) + +When an adapter config is updated or the framework binary is upgraded, the resources it manages may need to be reconciled again — even if the HyperFleet API spec has not changed. Today there is no mechanism to detect either change: the framework has no memory of which config version or binary version last ran for a given resource, so a drift-check event looks identical to a post-upgrade event and adapter authors cannot distinguish them. + +This proposal enables that detection. The framework tracks the adapter config version across executions and injects both the config version and the framework binary version into the CEL evaluation context on every event. Adapter config authors use lifecycle gate expressions to decide which resources or operations should run in response to a version change — the framework provides the signal, the config author decides the action. + +To support this, the design introduces two config schema changes and one API contract change: `adapter.version` is repurposed as the author-declared config version; `adapter.frameworkVersion` is added as the framework binary constraint; and both the config version (`adapter_config_version`) and the framework binary version (`adapter_framework_version`) are included in every status report sent to the HyperFleet API, stored in the raw statuses and not surfaced in the customer-facing conditions view. + +## Config Schema Changes + +Today `adapter.version` is a check against the framework binary version. This design repurposes it and adds a sibling field: + +| Field | Before | After | +|---|---|---| +| `adapter.version` | Framework binary version constraint | **Author-declared adapter config version** | +| `adapter.frameworkVersion` | — | Framework binary version constraint (replaces old `adapter.version`) | + +```yaml +adapter: + version: "2.0.0" # version of this task config, declared by the author + frameworkVersion: "1.5.0" # minimum required framework binary version +``` + +`adapter.version` is a free-form string the config author increments when they make a meaningful change to the config. The framework does not compute or validate it — authorship and cadence are entirely the author's responsibility. + +## Detection Mechanism + +### API contract + +Both the adapter config version and the framework binary version are reported to the HyperFleet API on every status update as new fields in `AdapterStatusCreateRequest`, added as siblings of `adapter`: + +```json +{ + "adapter": "provisioner", + "adapter_config_version": "2.0.0", + "adapter_framework_version": "1.5.0", + "observed_generation": 4, + "observed_time": "2026-06-22T10:00:00Z", + "conditions": [ + { "type": "Reconciled", "status": "True" } + ] +} +``` + +Both fields are included on every report. They are stored in the raw status records returned by `GET /api/hyperfleet/v1/resources/{id}/statuses` and are not surfaced in `resource.status.conditions`, which is the customer-facing view of the resource. This keeps internal adapter bookkeeping out of the conditions API. + +### Making version values available in CEL + +The framework does not inject version variables automatically. Adapter config authors wire them up using the existing params and precondition mechanisms. + +**Current version params** — sourced from the adapter config and the running binary (e.g., via ConfigMap entry and environment variable respectively): + +```yaml +params: + - name: adapterVersion + source: "configmap.hyperfleet.provisioner-config.adapter\.version" + type: "string" + - name: adapterFrameworkVersion + source: "env.ADAPTER_FRAMEWORK_VERSION" + type: "string" +``` + +**Last reported versions** — captured from the raw statuses endpoint in a precondition. Both fields are extracted from the same API call: + +```yaml +preconditions: + - name: "getAdapterStatuses" + api_call: + method: "GET" + url: "/api/hyperfleet/v1/resources/{{ .resourceId }}/statuses" + capture: + - name: "lastAdapterVersion" + expression: | + getAdapterStatuses.items.filter(s, s.adapter == "provisioner").size() > 0 + ? getAdapterStatuses.items.filter(s, s.adapter == "provisioner")[0].adapter_config_version + : "" + - name: "lastAdapterFrameworkVersion" + expression: | + getAdapterStatuses.items.filter(s, s.adapter == "provisioner").size() > 0 + ? getAdapterStatuses.items.filter(s, s.adapter == "provisioner")[0].adapter_framework_version + : "" +``` + +All four are then available as named variables in lifecycle gate expressions for the rest of the event execution. + +### Execution trigger + +With the params and captures defined, lifecycle gates can compare current vs last-reported values and conditionally skip or run resources: + +| Trigger | How to detect | +|---|---| +| spec.generation changed | Existing generation-aware apply logic | +| `adapter.version` changed | `adapterVersion != lastAdapterVersion` in a `when` expression | +| Framework binary changed | `adapterFrameworkVersion != lastAdapterFrameworkVersion` in a `when` expression | + +## Using version context in lifecycle gates + +The adapter config author controls what happens when a version change is detected. A resource-level `when` gate scopes execution to version-change events: + +```yaml +resources: + - name: "migrationJob" + when: + expression: "adapterVersion != lastAdapterVersion || adapterFrameworkVersion != lastAdapterFrameworkVersion" + lifecycle: + apply: + when: + expression: "!resources.?migrationJob.hasValue()" +``` + +This runs the migration job resource only when the config version or framework binary has changed and the job does not already exist — a one-time resource per upgrade. Resources without a version-checking `when` gate run on every event as normal. + +## Related Documentation + +- [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) — Main index document +- [Adapter Lifecycle Gates](./adapter-lifecycle-gates.md) — §1: Lifecycle Gates +- [Adapter Label Stamping](./adapter-label-stamping.md) — §2: Automatic Label and Annotation Stamping +- [Adapter Resilience Model](./adapter-resilience-model.md) — §3: Resilience Model +- [Adapter Stuck Detection](./adapter-stuck-detection.md) — §4: Stuck Detection +- [Adapter Resource Retention](./adapter-resource-retention.md) — §6: Resource Retention +- [Adapter Sweep Controller](./adapter-sweep-controller.md) — §7: Sweep Controller diff --git a/hyperfleet/components/adapter/framework/adapter-resilience-model.md b/hyperfleet/components/adapter/framework/adapter-resilience-model.md new file mode 100644 index 00000000..560da657 --- /dev/null +++ b/hyperfleet/components/adapter/framework/adapter-resilience-model.md @@ -0,0 +1,104 @@ +--- +Status: Draft +Owner: HyperFleet Team +Last Updated: 2026-06-16 +--- + +# Adapter Resilience Model + +> Part of [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) + +**TL;DR;** Do nothing, HyperFleet resiliency is accomplished by Sentinel retriggering events (keep reading only for more explanation) + +The adapter does not implement per-resource retry for apply and delete operations. The resilience model for HyperFleet is the Sentinel reconciliation cycle: Sentinel continuously compares observed resource state against the desired spec and generates a new event approximately every 10 seconds for as long as the state has not converged. Once the state converges, Sentinel switches to a periodic drift-check cadence (default: every 30 minutes). + +**Reconciliation events**: a transient apply failure leaves the resource state unchanged and unconverged. Sentinel observes continued divergence and sends the next event at the ~10s cadence. No inner retry is needed. + +**Drift-check events**: Sentinel also sends periodic events (default every 30 minutes) to detect and correct drift on resources that were previously converged. This introduces a corner case: if the apply fails during a drift-check event, the resource state is unchanged and still appears converged to Sentinel, which would otherwise not retrigger for another 30 minutes. This gap is closed by the adapter reporting `Available=False` on the API resource whenever an apply fails. `Available=False` signals to Sentinel that the resource is not reconciled, causing it to switch from the drift cadence back to the ~10s convergence cadence. The adapter must always surface apply failures via status conditions — this is what keeps the gap closed regardless of which Sentinel cadence triggered the event. + +Adding an inner retry loop inside a single event execution creates a two-loop problem: + +- **Concurrent execution risk**: a retry with a long `activeDeadline` holds the event execution open while Sentinel generates new events because the resource has not converged (or has reported `Available=False`). Multiple executions attempt the same apply against an already-stressed API server. +- **Redundancy**: the failure classes an inner retry would handle are already covered by the Sentinel cycle, provided the adapter surfaces failures correctly. +- **Backoff amplification**: exponential backoff inside the adapter does not coordinate with Sentinel's cadence. Under sustained API server pressure, both loops fire independently, increasing load rather than reducing it. + +The adapter's responsibility is to drive state convergence and always report failure via `Available=False` and status conditions. Retry is a property of the reconciliation loop, not of a single event execution. + +The existing retry mechanism in `internal/hyperfleetapi/client.go` (used for precondition API calls) is retained as-is. Precondition calls are synchronous blocking checks where a short backoff before concluding failure is semantically correct. Resource apply and delete do not share this property. + +## Alternatives Considered + +### §3 — Resilience Model + +#### hyperfleet.io/last-updated Annotation + +During the design of the Periodic Execution section, a `hyperfleet.io/last-updated` annotation was considered as a framework-stamped RFC3339 timestamp recording when the adapter last successfully applied a resource. This would make time-aware CEL expressions in drift-check cycles straightforward without requiring a precondition API call: + +```yaml +lifecycle: + patch: + - when: + expression: > + resources.?certSecret.hasValue() + && dig(resources.certSecret.metadata.annotations, "hyperfleet.io/last-updated") < now() + document: + metadata: + annotations: + example.io/rotated-at: "{{ .now }}" +``` + +Other uses where this pattern is useful: + +- **Rate-limiting expensive operations** — only trigger a heavyweight recreation or API call if the last successful apply was more than N hours ago, even though Sentinel sends drift events every 30 minutes +- **Audit trail on the resource** — operators can `kubectl describe` a resource and see when the adapter last touched it without querying the HyperFleet API +- **Cross-adapter coordination** — a downstream adapter can read `hyperfleet.io/last-updated` stamped by an upstream adapter on a shared resource to condition its own actions on recency + +**Why this was not adopted at framework level**: no specific use case in the current design requires it. The HyperFleet API already stores `lastTransitionTime` on the `Reconciled` condition, which is the authoritative timestamp for reconciliation events. Stamping an additional annotation on every managed resource adds a write on every apply cycle for a capability that may never be used by most adapters. + +No ecosystem precedent exists for a standalone last-applied timestamp annotation on managed resources — ArgoCD, Flux, and Helm all avoid it, relying on `metadata.managedFields[].time` or status conditions on controller objects instead. Crossplane is the closest precedent, using RFC3339 timestamps on managed resources for creation-lifecycle tracking only. + +**For adapter authors who need this**: add the annotation manually to the manifest template. The framework will not overwrite it (fill-gaps-only merge strategy). A custom key is equally valid — there is no requirement to use the `hyperfleet.io/` prefix for adapter-specific bookkeeping: + +```yaml +manifest: + metadata: + annotations: + hyperfleet.io/generation: "{{ .generation }}" + my-adapter.io/last-updated: "{{ .now }}" # custom key, stamped by adapter author +``` + +The `now()` template function is available in Go templates used for manifest rendering, providing the current RFC3339 timestamp at event execution time. + +#### Per-Resource Retry with Backoff + +An earlier version of this design proposed a per-resource retry configuration for apply and delete operations, modeled on the existing retry mechanism for precondition API calls: + +```yaml +resources: + - name: "managedCluster" + retry: + maxRetries: 3 + retryBackoff: exponential # exponential | linear | constant + activeDeadline: 60s + transport: + client: "kubernetes" + ... +``` + +The proposed behavior: on a transient transport error (connection refused, timeout, 429, 503), the executor would wait for an exponential/linear/constant backoff interval (±10% jitter) and retry up to `maxRetries` times before surfacing the error as a resource failure. The `activeDeadline` would be enforced via `context.WithDeadline`. Implementation would extract the existing retry loop from `internal/hyperfleetapi/client.go` into a new `pkg/retry` package. + +**Why this was not adopted**: The Sentinel reconciliation cycle covers both failure scenarios. For convergence events, a failed apply leaves the resource unconverged and Sentinel retriggers at ~10s. For periodic drift-check events (default every 30 minutes), a failed apply could create a gap — but the adapter closes it by reporting `Available=False` on failure, which signals Sentinel to treat the resource as not reconciled and switch back to the ~10s cadence. Provided the adapter always reports failure correctly via status conditions, no inner retry is needed. An inner retry loop also creates concurrent execution risk: if the resource has not converged (or has reported `Available=False`), Sentinel generates new events while the retry holds the current execution open, causing multiple executions to contend against an already-stressed API server. + +The existing retry in `internal/hyperfleetapi/client.go` is not affected — precondition calls are synchronous blocking checks where a short backoff before concluding failure is semantically correct, and they are not subject to Sentinel retriggering. + +**If this is revisited**: Any inner retry must complete well within the Sentinel cycle (< 5s total backoff). `maxRetries` should be at most 1–2, with no `activeDeadline` longer than a few seconds. It should be framed as "avoid unnecessary Sentinel cycles for sub-second blips," not as a resilience primitive. + +## Related Documentation + +- [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) — Main index document +- [Adapter Lifecycle Gates](./adapter-lifecycle-gates.md) — §1: Lifecycle Gates +- [Adapter Label Stamping](./adapter-label-stamping.md) — §2: Automatic Label and Annotation Stamping +- [Adapter Stuck Detection](./adapter-stuck-detection.md) — §4: Stuck Detection +- [Adapter Periodic Execution](./adapter-periodic-execution.md) — §5: Periodic Execution +- [Adapter Resource Retention](./adapter-resource-retention.md) — §6: Resource Retention +- [Adapter Sweep Controller](./adapter-sweep-controller.md) — §7: Sweep Controller diff --git a/hyperfleet/components/adapter/framework/adapter-resource-retention.md b/hyperfleet/components/adapter/framework/adapter-resource-retention.md new file mode 100644 index 00000000..6362ff61 --- /dev/null +++ b/hyperfleet/components/adapter/framework/adapter-resource-retention.md @@ -0,0 +1,143 @@ +--- +Status: Draft +Owner: HyperFleet Team +Last Updated: 2026-06-16 +--- + +# Adapter Resource Retention + +> Part of [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) + +Some adapter resources should accumulate as history rather than be replaced in-place — provisioning jobs, pipeline runs, audit records. The `retention:` block at the resource level enables this: it configures how many historical versions are kept and for how long. + +**Retention is not a lifecycle gate.** It is a resource-level policy that runs as a post-apply cleanup pass. It is independent of `lifecycle.recreate.when` — the gate decides *whether* to create a new version; retention decides *how many old versions to keep* after the new one exists. + +## Replace mode vs. retain mode + +The presence or absence of `retention:` determines the recreation strategy when `lifecycle.recreate.when` fires: + +- **Replace mode** (no `retention:` block): the old resource is deleted first, then the new manifest is applied with the same name. One resource exists at a time. +- **Retain mode** (`retention:` block present): the new manifest is applied without deleting the old one first. The adapter author controls the naming to ensure no collision — the name should vary per creation (e.g., include a timestamp or generation suffix). Old versions accumulate and are pruned by retention rules. + +This distinction matters because the two modes have incompatible semantics: replace requires the old name to be free before creating; retain requires the old resource to remain alive until pruned. + +## Multi-result discovery + +In retain mode, discovery must find all existing versions of the logical resource, not a single resource by name. The standard mechanism is a label selector on `hyperfleet.io/resource-id`, which is stamped automatically on every resource (see `§2`): + +```yaml +discovery: + by_selectors: + label_selector: + hyperfleet.io/resource-id: "{{ .resourceId }}" +``` + +This returns a list of all versions sharing the same API resource identity. The executor resolves the **current version** — the entry whose `hyperfleet.io/generation` annotation matches the current event's generation param, or the newest by `metadata.creationTimestamp` as fallback — and exposes only that single resource as `resources.myJob` in CEL expressions. The full version list is an internal executor concern. + +**Authors write lifecycle expressions exactly as they do today.** `resources.myJob` is always a single optional resource, never a list. The multi-version behavior (pruning historical versions, applying delete or orphan gates across all versions) is performed by the executor transparently, without the author needing to iterate over versions in CEL. + +## Retention rules + +```yaml +retention: + historyLimit: 3 # keep at most N historical versions; oldest pruned first + ttl: 24h # delete versions older than this duration + expression: > # CEL expression evaluated per item; true = delete + item.status.conditions.exists( + c, c.type == "Failed" && c.status == "True") +``` + +All three rules apply independently. If any enabled rule triggers for an item, it is deleted. The pruning pass runs after the apply step on every event where `retention:` is configured. The current item (the one whose `hyperfleet.io/generation` matches the current event) is never deleted regardless of expression result. + +### `expression` — per-item CEL predicate + +`expression` evaluates a CEL expression independently against each item returned by discovery. If the expression returns `true`, that item is deleted, regardless of whether `historyLimit` or `ttl` would have triggered for it. + +Three variables are available in addition to the standard params and adapter metadata: + +| Variable | Value | +|---|---| +| `item` | The discovered resource currently being evaluated | +| `items` | The full list of discovered resources, as returned by discovery | +| `itemIndex` | Zero-based index of `item` within `items` | + +This is useful for use cases that age- or count-based rules cannot express: + +- **Delete only failed runs**: prune job items that completed with `Failed=True`, retaining successful history even if `historyLimit` would otherwise remove it +- **Delete superseded configs**: prune items whose config hash no longer matches the current adapter config (i.e., created by an old config revision) +- **Delete stale pending items**: prune items stuck in a pending state for longer than a threshold +- **Keep only the N most recent successful runs**: use `itemIndex` and a filter over `items` to count how many successful items precede this one + +```yaml +retention: + historyLimit: 10 + expression: > + item.status.conditions.exists(c, c.type == "Failed" && c.status == "True") + || ( + !item.status.conditions.exists(c, c.type == "Complete" && c.status == "True") + && timestamp(item.metadata.creationTimestamp) < timestamp(now()) - duration("1h") + ) +``` + +This example deletes failed items immediately and also deletes any item that has not completed within 1 hour — treating a stuck job as prunable while keeping successful history up to `historyLimit`. + +All three fields are optional. A `retention:` block with only `expression` and no `historyLimit` or `ttl` relies entirely on the expression to drive pruning, which can leave unbounded history if the expression never returns `true`. + +## Delete and orphan semantics with multiple versions + +When a event arrives for a resource that is marked for deletion and multi-result discovery is active, all discovered versions of the resource should be affected — not only the current version. Leaving historical versions in place after a delete would orphan them with no cleanup path, since the API resource they reference will be deleted when reporting with `Finalized=True`. + +- **`lifecycle.delete.when`**: evaluated once as a gate; if true, the executor deletes all discovered versions. +- **`lifecycle.orphan.when`**: evaluated once as a gate; if true, ownership labels are stripped from all discovered versions before reporting `Finalized=True`. This ensures no version remains visible to the sweep controller. + +The precedence rule is unchanged: `lifecycle.orphan.when` is evaluated before `lifecycle.delete.when`. If orphan fires, the delete step is skipped for all versions. + +## Example — provisioning job with history + +```yaml +resources: + - name: provisioningJob + when: + expression: > + !resources.?provisioningJob.hasValue() + || timestamp(resources.provisioningJob.metadata.creationTimestamp) + < timestamp(now()) - duration("10m") + transport: + client: "kubernetes" + manifest: + apiVersion: batch/v1 + kind: Job + metadata: + name: "{{ .clusterId }}-job-{{ .now | replace \":\" \"-\" }}" + namespace: "{{ .namespace }}" + annotations: + hyperfleet.io/generation: "{{ .generation }}" + discovery: + namespace: "{{ .namespace }}" + by_selectors: + label_selector: + hyperfleet.io/resource-id: "{{ .resourceId }}" + retention: + historyLimit: 5 + ttl: 48h + lifecycle: + recreate: + when: + expression: > + resources.provisioningJob.status.conditions.exists( + c, c.type == "Complete" && c.status == "True") + || resources.provisioningJob.status.conditions.exists( + c, c.type == "Failed" && c.status == "True") +``` + +The resource-level `when` prevents a burst of convergence events from creating multiple jobs: if the current version is younger than 10 minutes the entire resource is skipped. Once the interval elapses, `lifecycle.recreate.when` is evaluated — it fires when the job has completed or failed, triggering a new run. Retention keeps at most 5 historical runs and discards any older than 48 hours. + +## Related Documentation + +- [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) — Main index document +- [Adapter Lifecycle Gates](./adapter-lifecycle-gates.md) — §1: Lifecycle Gates (including the `when` gate used for debouncing) +- [Adapter Label Stamping](./adapter-label-stamping.md) — §2: Automatic Label and Annotation Stamping (provides `hyperfleet.io/resource-id` for multi-result discovery) +- [Adapter Resilience Model](./adapter-resilience-model.md) — §3: Resilience Model +- [Adapter Stuck Detection](./adapter-stuck-detection.md) — §4: Stuck Detection +- [Adapter Periodic Execution](./adapter-periodic-execution.md) — §5: Periodic Execution +- [Adapter Sweep Controller](./adapter-sweep-controller.md) — §7: Sweep Controller diff --git a/hyperfleet/components/adapter/framework/adapter-stuck-detection.md b/hyperfleet/components/adapter/framework/adapter-stuck-detection.md new file mode 100644 index 00000000..eaf01173 --- /dev/null +++ b/hyperfleet/components/adapter/framework/adapter-stuck-detection.md @@ -0,0 +1,36 @@ +--- +Status: Draft +Owner: HyperFleet Team +Last Updated: 2026-06-16 +--- + +# Adapter Stuck Detection + +> Part of [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) + +**Current state**: The HyperFleet API already exposes deletion-specific stuck detection via `hyperfleet_api_resource_pending_deletion_stuck` — a gauge computed on each Prometheus scrape by querying the database for resources with `deleted_time` set beyond a configurable threshold (default 30 minutes). This covers the deletion flow only and has no equivalent for create and update reconciliations. + +**Planned** ([HYPERFLEET-1205](https://redhat.atlassian.net/browse/HYPERFLEET-1205)): A unified reconciliation metrics system that extends the same pattern to all reconciliation flows using the `Reconciled=False` condition already stored on API resources. + +New metrics: + +| Metric | Type | Description | +|---|---|---| +| `hyperfleet_api_reconciliation_requests_total` | Counter | Incremented when `Reconciled` transitions to `False` | +| `hyperfleet_api_resource_pending_reconciliation` | Gauge | Resources currently in `Reconciled=False` state | +| `hyperfleet_api_resource_pending_reconciliation_stuck` | Gauge | Resources where `Reconciled=False` has persisted beyond the stuck threshold | +| `hyperfleet_api_resource_pending_reconciliation_stuck_duration_seconds` | Gauge | Maximum duration any resource has been stuck | + +All metrics carry an `is_delete` label (`true` when `deleted_time IS NOT NULL`) to differentiate deletion flows from create/update flows. The existing deletion metrics become redundant and will be removed once HYPERFLEET-1205 ships. + +**Why no adapter-side annotation is needed**: the API stores the `Reconciled=False` condition with a transition timestamp in its database. Duration and stuck detection are computed from that timestamp via DB queries at scrape time — the API is the authoritative source. The adapter's responsibility is to correctly report `Available=False` on apply failure, which drives `Reconciled=False` on the API resource. Sentinel picks up the signal and retriggers; if `Reconciled=False` persists beyond the threshold, the API metrics surface it. No annotation on the managed K8s resource is required. + +## Related Documentation + +- [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) — Main index document +- [Adapter Lifecycle Gates](./adapter-lifecycle-gates.md) — §1: Lifecycle Gates +- [Adapter Label Stamping](./adapter-label-stamping.md) — §2: Automatic Label and Annotation Stamping +- [Adapter Resilience Model](./adapter-resilience-model.md) — §3: Resilience Model +- [Adapter Periodic Execution](./adapter-periodic-execution.md) — §5: Periodic Execution +- [Adapter Resource Retention](./adapter-resource-retention.md) — §6: Resource Retention +- [Adapter Sweep Controller](./adapter-sweep-controller.md) — §7: Sweep Controller diff --git a/hyperfleet/components/adapter/framework/adapter-sweep-controller.md b/hyperfleet/components/adapter/framework/adapter-sweep-controller.md new file mode 100644 index 00000000..e0d342b4 --- /dev/null +++ b/hyperfleet/components/adapter/framework/adapter-sweep-controller.md @@ -0,0 +1,95 @@ +--- +Status: Draft +Owner: HyperFleet Team +Last Updated: 2026-06-22 +--- + +# Adapter Sweep Controller + +> Part of [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) + +## What & Why + +**What**: A background controller that periodically scans for K8s and Maestro resources carrying HyperFleet management labels whose corresponding HyperFleet API resource no longer exists, and deletes them. + +**Why**: The normal reconciliation flow depends on the HyperFleet API sending a delete event to the adapter. A force-delete of the API resource bypasses that flow — the adapter never receives an event, so the managed K8s or Maestro resources are left running with HyperFleet labels but no live API counterpart. Without a sweep, these orphaned resources accumulate indefinitely. + +The sweep controller cannot use the API as its source of truth: after a force-delete, the API has no record of the deleted resource. It must invert the question — instead of "what resources should exist?", it asks "does this resource that exists on the cluster still have a live API counterpart?" + +## How + +The sweep controller runs on a configurable interval and executes a three-step loop: + +```mermaid +flowchart TD + A[List all K8s and Maestro resources with hyperfleet.io/adapter label] --> B[Extract hyperfleet.io/resource-id from each resource] + B --> C{GET /api/hyperfleet/v1/resources/id} + C -->|200 OK — API resource exists| D[Skip — resource is still managed] + C -->|404 — API resource force-deleted| E[Delete the K8s or Maestro resource] + C -->|Other error| F[Skip — treat as transient; retry next interval] +``` + +1. **Discover** — list all K8s resources and Maestro ManifestWork objects carrying the `hyperfleet.io/adapter` label, across all namespaces and resource kinds +2. **Verify** — for each discovered resource, extract `hyperfleet.io/resource-id` and call `GET /api/hyperfleet/v1/resources/{id}` on the HyperFleet API +3. **Act** — if the API returns 404, delete the K8s or Maestro resource; if the API returns the resource or any non-404 error, skip it + +The generic `/resources` endpoint from [HYPERFLEET-896](https://issues.redhat.com/browse/HYPERFLEET-896) is the key enabler: the sweep controller works across all resource kinds without kind-specific logic. + +### Prerequisites + +| Prerequisite | Provided by | +|---|---| +| `hyperfleet.io/adapter` label on all managed resources | Automatic label stamping (`§2`) | +| `hyperfleet.io/resource-id` carrying the API UUID | Automatic label stamping (`§2`) | +| Labels stripped on intentional disownment | `lifecycle.detach.when` (`§1.4`) — detached resources are invisible to the sweep controller | +| Generic `/resources` endpoint | HYPERFLEET-896 — required for the verification call | + +### Operational considerations + +- **Interval**: configurable; a few hours is sufficient — force-deleted resources are not immediately harmful, just wasteful +- **Kubernetes CronJob**: The sweep controller is created as a kubernetes cron job periodically +- **Maestro**: ManifestWork objects carry the labels on the ManifestWork itself; the sweep controller deletes the ManifestWork, not the nested manifests +- **Dry-run mode**: should support a read-only mode that reports what would be deleted without acting +- **Per adapter**: we can have specific sweep jobs per adapter, in case more fine grained granularity is required for some type of resources + +## Trade-offs + +### What We Gain + +- ✅ Orphaned resource cleanup without coupling to adapter availability — the sweep controller runs independently of adapter instances +- ✅ Works across all resource kinds via the generic `/resources` endpoint — no kind-specific logic required +- ✅ Intentional disownment (`lifecycle.detach.when`) is naturally invisible to the sweep: labels are stripped before the API resource is deleted, so the controller skips detached resources + +### What We Lose / What Gets Harder + +- ❌ Cleanup is eventual — orphaned resources linger for up to one sweep interval before being detected and removed +- ❌ Requires a separately deployed and operational component — if the sweep controller is down, orphaned resources accumulate silently +- ⚠️ Every managed resource requires two labels to be present — if label stamping (`§2`) is missing or stripped unexpectedly, resources become invisible to the sweep controller + +### Acceptable Because + +Force-deleted resources are wasteful but not immediately harmful. A bounded delay (configurable sweep interval, typically minutes) is an acceptable trade-off for decoupling cleanup from adapter and API availability. The alternative — synchronous cleanup via finalizers — creates a harder failure mode: a stuck finalizer blocks resource deletion indefinitely. + +## Alternatives Considered + +### K8s Finalizers + +**What**: The framework adds a K8s finalizer to every managed resource at apply time. The finalizer is cleared only after the adapter confirms the HyperFleet API record is gone (via a pre-delete check). Cleanup is synchronous and guaranteed. + +**Why Rejected**: Finalizers create an availability dependency — if the adapter is down or the HyperFleet API is unreachable, the managed resource is stuck in a terminating state indefinitely and cannot be cleaned up even manually without force-removing the finalizer. For infrastructure resources (Namespaces, ManifestWorks), a stuck finalizer is more harmful than a delayed sweep. + +### Lease / Heartbeat Expiry + +**What**: The adapter stamps a `hyperfleet.io/lease-expires-at` annotation (RFC3339, current time + lease duration) on every resource at apply time. An external controller deletes resources whose lease timestamp has passed. + +**Why Rejected**: Requires a write on every successful apply — even for no-op drift-check events — to renew the lease. For adapters managing many resources at a high Sentinel cadence, this is significant write amplification. It also requires tuning the lease duration: too short and a temporarily unavailable adapter causes healthy resources to be deleted; too long and stale resources linger. The sweep controller achieves equivalent cleanup without per-event writes, at the cost of a sweep interval delay. + +## Related Documentation + +- [Adapter Resource Lifecycle Management Design](./adapter-lifecycle-management-design.md) — Main index document +- [Adapter Lifecycle Gates](./adapter-lifecycle-gates.md) — §1.4: `lifecycle.detach.when` — resources detached via this gate are invisible to the sweep controller +- [Adapter Label Stamping](./adapter-label-stamping.md) — §2: Automatic Label and Annotation Stamping — prerequisite for sweep controller discovery +- [Adapter Resilience Model](./adapter-resilience-model.md) — §3: Resilience Model +- [Adapter Stuck Detection](./adapter-stuck-detection.md) — §4: Stuck Detection +- [Adapter Periodic Execution](./adapter-periodic-execution.md) — §5: Periodic Execution +- [Adapter Resource Retention](./adapter-resource-retention.md) — §6: Resource Retention