Skip to content

feat(agent): add CRD gate implementation and tests#3839

Open
floreks wants to merge 4 commits into
masterfrom
sebastian/prod-4535-services-with-crds-and-cr-instances-need-multiple-passes-to
Open

feat(agent): add CRD gate implementation and tests#3839
floreks wants to merge 4 commits into
masterfrom
sebastian/prod-4535-services-with-crds-and-cr-instances-need-multiple-passes-to

Conversation

@floreks

@floreks floreks commented Jul 13, 2026

Copy link
Copy Markdown
Member
  • Introduce crdGate to enhance CRD processing with conditional resources
  • Implement CRD gate tests including eligibility checks and wait strategies
  • Enhance database store with CRD establishment tracking
  • Update insert and update logic to include CRD establishment field

Test Plan

Test environment: https://console.plrl-dev-aws.onplural.sh/

Checklist

  • I have added a meaningful title and summary to convey the impact of this PR to a user.
  • If required, I have updated the Plural documentation accordingly.
  • I have added tests to cover my changes.
  • I have deployed the agent to a test environment and verified that it works as expected (required only when changing agent code).

Plural Flow: console

- Introduce `crdGate` to enhance CRD processing with conditional resources
- Implement CRD gate tests including eligibility checks and wait strategies
- Enhance database store with CRD establishment tracking
- Update insert and update logic to include CRD establishment field
@floreks floreks self-assigned this Jul 13, 2026
@floreks floreks added the enhancement New feature or request label Jul 13, 2026
@linear

linear Bot commented Jul 13, 2026

Copy link
Copy Markdown

PROD-4535

@floreks
floreks marked this pull request as draft July 13, 2026 12:57
@soffi-ai

soffi-ai Bot commented Jul 13, 2026

Copy link
Copy Markdown

Soffi AI Summary

This PR introduces a CRD gate mechanism in the deployment operator's streamline applier to solve a multi-pass problem: when a service deployment contains both CRD definitions and Custom Resource (CR) instances, the CRs cannot be applied until the CRDs are fully established in the Kubernetes API server. Without gating, a single-pass apply would fail because the API server doesn't yet recognize the new resource types.

The CRD gate (crdGate) intercepts the apply wave for CR resources, checks whether the CRDs they depend on were newly applied in the current run (vs. pre-existing), waits for those CRDs to be established (either via persisted store state or live discovery polling), and resets the REST mapper so the dynamic client can resolve the new GVKs before proceeding. Key design decisions include:

  • Fail-open behavior: if the store is unavailable during eligibility classification, the gate disables itself rather than blocking applies indefinitely.
  • No-op for dry runs and pre-existing CRDs: the gate only activates for newly applied CRDs, avoiding unnecessary waits on re-deploys.
  • CRD establishment tracking is persisted to the database store so the gate can resume correctly across reconcile cycles.

Complementary changes include: a new IsCRDEstablished utility in pkg/common/crd.go, extended store interfaces and DB queries to track the CRD establishment field, discovery cache additions to support Add/ResetRESTMapper flows, and a small refactor to how wave.go integrates pre-apply gates. The change is covered by a comprehensive test suite (~800 lines of new tests) covering eligibility logic, wait/polling strategies, timeout handling, and store error paths.

Commits

Commit Summary
1be9ef7 Core feature commit: introduces crdGate in the streamline applier (crd_gate.go) to gate CR applies behind CRD establishment, extends the wave applier to invoke pre-apply gates, adds CRD establishment tracking to the DB store and its interfaces, adds IsCRDEstablished to pkg/common/crd.go, extends the discovery cache with Add/ResetRESTMapper, and ships a large test suite covering eligibility, wait strategies, and error paths.
bee75d6 Bug fix: corrects a string formatting error in the CRD establishment timeout error message within crd_gate.go (wrong format verb or missing argument).
5573b15 Cleanup: fixes import ordering in the store package and improves variadic argument usage for Go style consistency.
9622ea2 Merge commit syncing the feature branch with the latest master to incorporate upstream changes before the PR is finalized.

Deploy in Soffi


Updated: 2026-07-16 12:50 UTC

@greptile-apps

greptile-apps Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces a crdGate that prevents resources depending on newly-introduced CRDs from being applied until those CRDs are fully established and discoverable. It also adds an established column to the SQLite component store so the gate can poll CRD readiness via the watch stream rather than making live API calls.

  • crd_gate.go: New crdGate skips already-applied CRDs (checked against the store before SyncServiceComponents), uses singleflight.DoChan to deduplicate concurrent waits per GVK, and polls IsCRDEstablished + RestMapping with exponential backoff up to 30 s.
  • db_queries.go / db_store.go: Adds established BOOLEAN NOT NULL DEFAULT 0 to the component table and a new IsCRDEstablished query that requires an exact UID match, ensuring stale watch rows from a previous CRD instance never unblock the gate.
  • db_store_profiled*.go: Fixes the pre-existing variadic-spread bug in SaveComponentAttributes wrappers (argsargs...) alongside the new IsCRDEstablished tracing methods.

Confidence Score: 5/5

Safe to merge; gate logic, schema change, and profiling wrappers are all correct with thorough test coverage.

The new crdGate correctly identifies only newly-applied CRDs as gate candidates, deduplicates concurrent waits, and uses an exact UID match so a watch row from a deleted-and-re-created CRD cannot spuriously unblock the gate. The established column is backward-compatible. The variadic spread fix eliminates a real SQL argument mismatch.

The resolve function in crd_gate.go has a minor error-message cosmetic issue but no logic defects.

Important Files Changed

Filename Overview
go/deployment-operator/pkg/streamline/applier/crd_gate.go New CRD gate that deduplicates REST-mapping waits via singleflight; error message in resolve always says "timed out" even on cancellation, and duplicates the error string when no backoff iterations ran.
go/deployment-operator/pkg/streamline/applier/applier.go Gate wiring added before SyncServiceComponents so GetAppliedComponent correctly reads previous-cycle state; import ordering and gate option building look correct.
go/deployment-operator/pkg/streamline/store/db_store.go Adds IsCRDEstablished with exact UID+name match; SaveComponent and SaveComponents now track the established column consistently.
go/deployment-operator/pkg/streamline/store/db_queries.go Adds established column (NOT NULL DEFAULT 0) and getCRDEstablished query; setComponentWithSHA and setComponentUnsynced updated consistently.
go/deployment-operator/pkg/streamline/applier/wave.go Gate slice, WithWaveGates option, runGates, gatesEnabled, and pre/post-apply gate invocations all added cleanly; clientForMapping refactored out of clientForResource.
go/deployment-operator/pkg/common/crd.go New shared CRD utilities (IsCRD, CRDEstablished, ServedCRDGVKs) extracted to common package; correctly removes the duplicate crdGK var from template/common.go.

Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/mas..." | Re-trigger Greptile

Comment thread go/deployment-operator/pkg/cache/discovery/cache.go
floreks added 2 commits July 13, 2026 16:16
…message

- Adjust error message formatting to properly nest error details
- Reorder imports for better readability and maintainability
- Fix variadic argument usage in `SaveComponentAttributes` function
@floreks

floreks commented Jul 13, 2026

Copy link
Copy Markdown
Member Author

@greptileai

Comment thread go/deployment-operator/pkg/streamline/applier/applier.go
Comment thread go/deployment-operator/pkg/streamline/applier/applier.go
Comment thread go/deployment-operator/pkg/streamline/applier/wave.go
Comment thread go/deployment-operator/pkg/streamline/applier/wave.go
@floreks
floreks marked this pull request as ready for review July 15, 2026 14:41
…-services-with-crds-and-cr-instances-need-multiple-passes-to
@floreks

floreks commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@greptileai ignore Go's new function usage. It has been updated in Go 1.26 and is allowed. Review again with that in mind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant