Skip to content

Tunable sentinel poll intervals#3905

Open
michaeljguarino wants to merge 7 commits into
masterfrom
more-perf-improvements
Open

Tunable sentinel poll intervals#3905
michaeljguarino wants to merge 7 commits into
masterfrom
more-perf-improvements

Conversation

@michaeljguarino

@michaeljguarino michaeljguarino commented Jul 23, 2026

Copy link
Copy Markdown
Member

This apparently never was added to the AgentConfiguration crd, adds it in so we can tune in high cluster deployments

Test Plan

Test environment: https://console.plrldemo.onplural.sh/cd/clusters/a1748282-ce8b-48ab-ae7e-326e74fce04e/services/f3f89a54-d1a7-4bc8-9152-daa07ede918d/components

Checklist

  • I have added a meaningful title and summary to convey the impact of this PR to a user.
  • If required, I have updated the Plural documentation accordingly.
  • I have added tests to cover my changes.
  • I have deployed the agent to a test environment and verified that it works as expected (required only when changing agent code).

Plural Flow: console

@michaeljguarino
michaeljguarino requested a review from a team July 23, 2026 16:07
@michaeljguarino michaeljguarino added the enhancement New feature or request label Jul 23, 2026
@soffi-ai

soffi-ai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Soffi AI Summary

This PR bundles several related improvements across the agent operator, AI workbench, and cluster health visualization, motivated by production tuning needs in high-scale cluster deployments.

Tunable Sentinel Poll Intervals (primary change): The AgentConfiguration CRD was missing a field to configure how frequently sentinels poll for state changes. This adds sentinelPollInterval (and related interval fields) to the CRD spec, its Go type definition, and the generated deepcopy/CRD YAML manifests — allowing operators to reduce polling pressure on large fleets without redeploying the agent binary.

AI Workbench Enhancements:

  • Adds a workbenchPrFollowup mutation so the workbench can trigger follow-up actions on open PRs (e.g., re-evaluate after a review cycle).
  • Introduces a VERIFY activity type and REJECTED status to the workbench job activity model, enabling an explicit verification step in agentic workflows with a distinct "rejected" terminal state.
  • Adds a Verify agent implementation in the deployment operator and wires it to the engine's state machine.
  • Expands WorkbenchToolCategory with Coding, Observability, and Verification categories, improving tool organization in the UI.

Configurable Health Map Limits: Introduces a healthmapClusterCount field on ConsoleConfiguration (exposed via GraphQL) and a corresponding Helm chart value/secret, letting operators cap how many clusters appear in the cluster health scores heatmap. The frontend query and component both consume this new limit, and a bug in the heatmap display is fixed.

Cleanup: Removes boundRoles from the me GraphQL query (reducing payload size for the common auth check), strips redundant role-binding logic from the frontend user/group management components, and deletes a now-unused styled-components type shim.

Commits

Commit Summary
c59c1e9 Adds sentinelPollInterval (and related interval fields) to the AgentConfiguration CRD type, its generated deepcopy, CRD YAML manifest, CLI args, and common config — making sentinel polling frequency tunable at the operator level for high-cluster-count deployments.
4a9865c Removes boundRoles from the me GraphQL query, reducing the size of the primary auth payload, and cleans up the associated role-binding display logic from the user, group, and persona management UI components.
08f96c1 Adds a verifyState transition to the agent engine state machine, wiring it into the execution lifecycle alongside the new verify agent.
c6c84f6 Implements the VerifyAgent in the deployment operator, providing an explicit verification step in agentic workflow runs.
487aac2 Adds the workbenchPrFollowup GraphQL mutation (server schema + Go client models + generated TypeScript types), enabling the workbench to initiate follow-up actions on open pull requests.
8317b67 Introduces a healthmapClusterCount config field (GraphQL schema, Helm chart values/secrets, frontend query variable) to cap the number of clusters shown in the cluster health heatmap, addressing performance in large fleets.
1a1ab21 Fixes the cluster health scores heatmap display and adds the REJECTED WorkbenchJobActivity status and Verify/Coding/Observability/Verification workbench tool categories to the GraphQL schema and frontend types.

Deploy in Soffi


Updated: 2026-07-24 22:03 UTC

@greptile-apps

greptile-apps Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR exposes the sentinel reconciler's poll interval as a tunable setting via the AgentConfiguration CRD, replacing the previously hardcoded 30-second constant in console.go. It also upgrades the sentinel cache to use the cache.Store interface (backed by DynamicCache) so the cache TTL adapts at runtime when the interval changes.

  • Adds sentinelPollInterval to AgentConfigurationSpec, the CRD schema, the CLI flag layer (--sentinel-poll-interval, default 30s), and the ConfigurationManager with full merge/override semantics consistent with all other interval fields.
  • Introduces EffectivePollInterval and ControllerCacheTTLFunc helpers in the sentinel package, plus pollJitterWindow that falls back to 15s when polling is disabled, all covered by a new interval_test.go.
  • Adds deep copy entries for SentinelPollInterval (and incidental StreamingProxy/Memory fields in AgentRuntimeSpec) in the auto-generated file.

Confidence Score: 5/5

Safe to merge — the change is additive, fully backward-compatible (new field is optional with a matching CLI default), and the new dynamic-interval logic is covered by tests.

All changed paths follow the established pattern for configurable intervals used by other reconcilers. The sentinel cache upgrade to DynamicCache mirrors how other caches already work. New test coverage validates disable, jitter window, and cache TTL scenarios. No existing behavior is removed; the hardcoded constant is simply replaced by the same default value flowing through the proper config layer.

No files require special attention.

Important Files Changed

Filename Overview
go/deployment-operator/pkg/controller/sentinel/reconciler.go Upgrades sentinelCache to cache.Store interface, switches to DynamicCache with ControllerCacheTTLFunc, and makes GetPollInterval dynamic via EffectivePollInterval. Adds pollJitterWindow helper with 15s fallback when polling is disabled.
go/deployment-operator/pkg/common/config.go Adds sentinelPollInterval field, its parse/set logic in setValueLocked, override in mergeAgentConfigurationSpec, and GetSentinelPollInterval accessor; consistent with all existing interval fields.
go/deployment-operator/cmd/agent/args/args.go Adds sentinel-poll-interval CLI flag with 30s default, SentinelPollInterval() accessor, and wires it into AgentConfigurationDefaults; pattern is consistent with existing interval flags.
go/deployment-operator/pkg/controller/sentinel/interval_test.go New test file covering disable-by-config, jitter window with custom interval, jitter window fallback when polling is disabled, and dynamic cache TTL calculation.
go/deployment-operator/api/v1alpha1/agentconfig_types.go Adds SentinelPollInterval field to AgentConfigurationSpec with appropriate doc comment and omitempty tag.
go/deployment-operator/api/v1alpha1/zz_generated.deepcopy.go Auto-generated deep copy additions for SentinelPollInterval, StreamingProxy, and Memory fields; standard code-gen pattern.
go/deployment-operator/cmd/agent/console.go Removes the hardcoded 30s sentinelPollInterval constant and replaces it with args.SentinelPollInterval() at the reconciler construction site.
go/deployment-operator/cmd/agent/main_test.go Extends existing AgentConfiguration tests to cover SentinelPollInterval default (30s), override (5m), and reset back to default.
go/deployment-operator/pkg/controller/sentinel/socket_publisher.go Changes sentinelRunCache field type from *cache.Cache to cache.Store interface, consistent with the reconciler change.
go/deployment-operator/config/crd/bases/deployments.plural.sh_agentconfigurations.yaml CRD schema updated to include sentinelPollInterval with accurate description; straightforward generated change.

Reviews (1): Last reviewed commit: "Tunable sentinel poll intervals" | Re-trigger Greptile

@michaeljguarino

Copy link
Copy Markdown
Member Author

plural deploy this

@plural-copilot plural-copilot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This apparently never was added to the AgentConfiguration crd, adds it in so we can tune in high cluster deployments
@michaeljguarino
michaeljguarino force-pushed the more-perf-improvements branch from b9622e3 to c59c1e9 Compare July 23, 2026 16:31
@michaeljguarino

Copy link
Copy Markdown
Member Author

plural deploy this

@plural-copilot plural-copilot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Deployed for testing via https://github.com/pluralsh/plrl-up-demos/pull/2347

Console image tag bumped to sha-c59c1e9 (commit c59c1e986d77963da855765ac9d8107f60d280be) on the mgmt console service.

@michaeljguarino

Copy link
Copy Markdown
Member Author

plural deploy this

@plural-copilot plural-copilot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Deployed! Console image pinned to sha-c59c1e9 (commit c59c1e9) via GitOps update PR: https://github.com/pluralsh/plrl-up-demos/pull/2348

Once that PR merges and syncs, the test environment will be running this build.

@michaeljguarino

Copy link
Copy Markdown
Member Author

plural deploy this

@plural-copilot plural-copilot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deployed via https://github.com/pluralsh/plrl-up-demos/pull/2353 using image tag sha-4a9865c.

@michaeljguarino
michaeljguarino force-pushed the more-perf-improvements branch from 727035d to 08f96c1 Compare July 24, 2026 13:04
@michaeljguarino

Copy link
Copy Markdown
Member Author

plural deploy this

@plural-copilot plural-copilot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deployed for testing: https://github.com/pluralsh/plrl-up-demos/pull/2368

This updates the console service in the test environment to use image tag sha-08f96c1 (head commit of this PR) so the tunable sentinel poll interval change can be verified.

@michaeljguarino
michaeljguarino force-pushed the more-perf-improvements branch from 672c3b5 to 1f22f4b Compare July 24, 2026 16:16
@michaeljguarino
michaeljguarino force-pushed the more-perf-improvements branch from 1f22f4b to c6c84f6 Compare July 24, 2026 16:48
@michaeljguarino
michaeljguarino force-pushed the more-perf-improvements branch from 13e0f7b to 487aac2 Compare July 24, 2026 19:25
@michaeljguarino

Copy link
Copy Markdown
Member Author

plural deploy this

@plural-copilot plural-copilot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljguarino
michaeljguarino force-pushed the more-perf-improvements branch from a823b4e to 8317b67 Compare July 24, 2026 21:25
@michaeljguarino

Copy link
Copy Markdown
Member Author

plural deploy this

@plural-copilot plural-copilot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michaeljguarino

Copy link
Copy Markdown
Member Author

plural deploy this

@plural-copilot plural-copilot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy PR created: https://github.com/pluralsh/plrl-up-demos/pull/2376
Image tag: sha-1a1ab21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants