Skip to content

[stretch] Alert when a veda-backend instance hasn't been deployed in a configurable window #208

Description

@botanical

Summary

We currently have no way of knowing when an instance is running a stale version of veda-backend unless someone manually checks. A lightweight alerting mechanism would help us catch instances that fall behind, particularly after breaking upstream changes (like the recent stac-fastapi 6.3 incompatibility) see more here : NASA-IMPACT/veda-backend#623

When a breaking change lands in a dependency or we cut a release with a critical fix, there's no signal that an instance is still running an older version. This has led to situations where we discover incompatibilities at deploy time rather than proactively.

Possible approaches

  • GitHub Actions scheduled workflow that checks deployed instances against the latest release tag and opens an issue or sends a Slack notification if the gap exceeds a threshold?
  • CloudWatch/EventBridge rule that fires if no deployment event has been recorded for an environment within a configurable window
  • Could we maybe use Grafana and add a widget (not sure if that's the correct term) showing last deploy date per instance (lower effort, no alerting, but makes staleness visible)

Acceptance Criteria

  • Define what "stale" means (time-based, version-based, or both)
  • At least one instance type (starting with veda-backend) is covered
  • Notification is sent to a team-visible channel (Slack, GitHub issue, or email)
  • Threshold is configurable

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions