feat: add datadog-error-monitor skill

[tofarr]

Summary

Adds a new datadog-error-monitor skill — a cron automation that polls Datadog logs every 15 minutes, maintains a self-evolving regex-based error pattern library, and triggers targeted OpenHands investigation conversations when new or spiking errors are detected.

Draft PR for review and discussion — created by an AI agent (OpenHands) on behalf of @tofarr.

---

How it works

Every 15 min (cron script — no LLM)
     │
     ▼
Query Datadog ──► Match against known patterns ──► Update run_history
     │
     ├── Unknown logs detected? ─────────────────────────────────────┐
     │                                                               │
     └── Any pattern spiked (count > 3× rolling baseline)? ─────────┤
                                                                     ▼
                                                            Start one OpenHands
                                                          investigation conversation
                                                                     │
                                                    ┌────────────────┘
                                                    ▼
                                          Agent categorizes unknown errors
                                          into named regex patterns, writes
                                          them back to the state file,
                                          investigates root causes in local
                                          codebases, creates PRs if confident,
                                          posts summary to Slack

Token efficiency: The cron script is 100% deterministic — zero LLM calls on quiet runs. A conversation is only started when triggered, and only one conversation runs at a time.

---

Files

File	Purpose
SKILL.md	11-step setup workflow the agent guides users through
scripts/main.py	Cron script template (657 lines, stdlib-only, customised at automation-creation time)
references/state-schema.md	JSON state file schema, agent write protocol, and lifecycle notes
references/datadog-api.md	Datadog API reference (auth, log search, rate limits)
references/agent-prompt-template.md	Investigation prompt structure, token budget, and workspace notes
README.md	User-facing overview

---

Key design decisions

• Self-evolving pattern library: Starts empty. The agent builds patterns organically from the first run's uncategorized logs and writes them back to the state file. Future runs match deterministically with no LLM cost.
• One global conversation at a time: All triggers from a given run are batched into a single investigation conversation. No new conversation starts while one is running.
• Spike detection: current_count > mean(last 3 runs) × SPIKE_MULTIPLIER — activates only after 3+ history entries to avoid false positives on newly added patterns.
• Multi-host git support: Repos can span GitHub, GitLab, and Bitbucket. The setup skill verifies each repo path and the corresponding token at configuration time.
• PR guardrail: The agent prompt explicitly instructs: create a PR only if highly confident of a code-level root cause. Infrastructure/config issues get a Slack note instead.

---

Open questions / known gaps for review

1. Pattern bootstrapping UX — On first run everything is uncategorized. Users may want to run a one-time manual bootstrap query to see what the first agent conversation will receive. Should the SKILL.md include an optional pre-run step that previews a sample of recent errors?

2. min_cluster_size not yet configurable — Currently hardcoded to 1 (any single unmatched log triggers). Should this be a configurable setup parameter?

3. Investigation conversation workspace isolation — The agent's workspace is set to the first configured repo path. For multi-repo setups this is a minor limitation. A dedicated investigations directory could be used instead. Opinions?

4. No .plugin metadata directory — Other skills have a .plugin/ directory with plugin manifests. This PR doesn't include it yet. Should it follow the same pattern?

Screenshots

Skill appears in beta list:

Setup works as expected:

Monitors create conversations