docs: move agent memory into the repo

AGENTS.md

@@ -17,7 +17,9 @@ Agent instructions for every coding agent (Claude Code, Codex, Cursor, Copilot,
   - The only competitor-facing pages in `web/docs/` are the `migrate-from-*.md` import guides.
 
 ## Development workflow
-Laptop only: if a server already runs on port 4000, apply code changes directly (`make dev` / `make dev-test` hot-reload). `rtk` prefixes, the k8xp VPN and `gopass` are user-global setup, not part of the repo. Sandboxes and Claude cloud sessions: see [Without Docker](#without-docker-cloud-sessions-ci-like-sandboxes).
+Laptop only: if a server already runs on port 4000, apply code changes directly (`make dev` / `make dev-test` hot-reload). Never stop or `pkill` that devloop: run anything else on another port and kill it by PID. `rtk` prefixes, the k8xp VPN and `gopass` are user-global setup, not part of the repo. The maintainer's dev instance (`solidping.k8xp.com`) and the prod fleet are operated from the private `k8xp` repo (`k8s/solidping/OPERATIONS.md`). When a git answer gates a decision (current branch, did a merge land), check it with plain `git`: `rtk git` has reported stale and even invented branch names. Sandboxes and Claude cloud sessions: see [Without Docker](#without-docker-cloud-sessions-ci-like-sandboxes).
+
+Before calling the API of any server, `GET /api/mgmt/version`: it gives the run mode (which credentials apply) and the commit actually serving.
 
 1. Start infrastructure: `docker-compose up -d`
 2. Run everything: `make dev` (backend + dash0 + status0 with hot reload)
@@ -118,6 +120,11 @@ curl -s -H "Authorization: Bearer $TOKEN" 'http://localhost:4000/api/v1/orgs/def
 - **Backend**: table-driven tests + testcontainers for integration (see `server/AGENTS.md`)
 - **Dash0**: Playwright E2E in `web/dash0/e2e/` (see `web/dash0/AGENTS.md`)
 - Comprehensive coverage expected for new features
+- To reproduce or verify UI behaviour, write or run a Playwright spec rather than driving a browser through an MCP tool (much slower). Use the user's browser only when the flow needs their logged-in session.
+- Local E2E runs use a side-car server, never the `:4000` devloop. Recipe and traps: [wiki/testing/local-e2e.md](wiki/testing/local-e2e.md). A full suite run must stay under 1 hour; a slower run is a harness bug to fix, not to wait out.
+- Flaky tests are bugs: root-cause them, never re-run to green ([wiki/testing/flaky-tests.md](wiki/testing/flaky-tests.md)). A test asserting an absence needs a positive control.
+- Fix lint findings in code. Never relax `.golangci.yml` or the eslint config to get green. Local linter vs CI parity, red-CI traps and the release flow: [wiki/runbooks/ci-and-release-triage.md](wiki/runbooks/ci-and-release-triage.md).
+- Server behaviours that mislead when operating a deployment (migrations on rollout, inert env vars, provider callbacks, check diagnosis): [wiki/runbooks/operations-gotchas.md](wiki/runbooks/operations-gotchas.md).
 
 ## Never name a real company — use `acme`
 
@@ -142,6 +149,7 @@ Exceptions:
 ## Specs
 - Filename format: `YYYY-MM-DD-NN-title.md` (`NN` unique per day across `specs/todos/` and `specs/done/YYYY/MM/`)
 - Active: `specs/todos/`, Done: `specs/done/YYYY/MM/`, Backlog: `specs/backlog/`, Cancelled: `specs/cancelled/`
+- Never delete the `specs/todos/` directory itself, even when it is empty.
 
 ## Batch branches
 `/implement-todos` and similar multi-spec runs integrate onto a dated batch branch (e.g. `batch/2026-06-23`).

server/AGENTS.md

@@ -104,8 +104,8 @@ Mechanics worth knowing before you touch an existing file:
   (`make dev` / `dev-test` / `dev-saas`) always runs warn. `solidping migrate repair`
   re-records checksums for applied migrations (no migration runs) to clear a cosmetic-edit
   mismatch. See `wiki/conventions/database.md` for the full guard/repair writeup.
-- **`internal/middleware/`**: Authentication, CORS, logging, and organization context
-- **`internal/config/`**: Configuration management using koanf (YAML + environment variables)
+- **`internal/middleware/`**: Authentication, CORS, logging, and organization context. Adding a middleware to a shared chain (`orgGroup`)? Update the hand-built test fixtures that copy that chain in the same change (`wiki/testing/flaky-tests.md`).
+- **`internal/config/`**: Configuration management using koanf (YAML + environment variables). The env loader maps every `_` after `SP_` to `.`, so a new `SP_*` variable for a multi-word key (`rate_limiting`, `max_open_conns`...) needs a manual reader in `Load()` next to `applyRateLimitingEnv`, or it is silently ignored.
 
 ## Database Schema
 

web/dash0/AGENTS.md

@@ -185,6 +185,13 @@ mirror `incidents.index.tsx` and `jobs.index.tsx`. The primary navigation
 `replace: true` only for incidental refinements (filters, scope toggles)
 so they don't spam history.
 
+Caveat: under a layout route, a cold deep link can run a second
+search-validation pass with an empty search and drop the param. If a deep
+link resets, seed local state from the URL once on mount and write the
+param back on every change (`organization.discovery.new.tsx`). Make
+`validateSearch` omit absent keys (return `{}`), or every `<Link>` to the
+route will require a `search` prop.
+
 ### Row actions: icons, not menus
 
 In list/table rows, prefer two ghost icon buttons (`Pencil` for edit,

wiki/README.md

@@ -102,12 +102,16 @@ Operational procedures for diagnosing the running system.
 - [runbooks/api-testing-with-curl.md](runbooks/api-testing-with-curl.md) — Log in, save a token, and exercise the REST API by hand with curl; default credentials, forced password rotation, troubleshooting.
 - [runbooks/claude-cloud.md](runbooks/claude-cloud.md) — Working in a Claude cloud (remote) sandbox with no Docker, secrets or built frontend: `scripts/cloud-setup.sh`, what a clean clone needs (embed placeholders), SQLite smoke server, CI env parity for E2E, what to run before pushing.
 - [runbooks/observability-toggles.md](runbooks/observability-toggles.md) — `SP_PROMETHEUS_ENABLED`, `SP_METRICS_SCRAPE_TOKEN`, `SP_PROFILER_ENABLED`, `SP_OTEL_ENABLED`: what each switches and how they interact.
+- [runbooks/ci-and-release-triage.md](runbooks/ci-and-release-triage.md) — CI and release states that lie: skipped jobs, conflicting PRs with zero runs, flapping run states, local golangci-lint vs CI parity, the tag-triggered release flow, release-please silent failures and their recovery, and why squash-merged branches look unmerged.
+- [runbooks/operations-gotchas.md](runbooks/operations-gotchas.md) — Server behaviours that misled whoever ran a real deployment: unlocked migrations on rollout, env vars that silently do nothing, one callback URL per provider app, signed calls to a renamed org, check diagnosis, silent realtime, connection-pool budgets.
 
 ## Testing
 
 - [testing/coverage.md](testing/coverage.md) — Backend coverage: how `make test-cover` and the `backend-postgres` gate measure it, what is excluded (generated/vendored), the baseline, the `COVERAGE_MIN` ratchet, and report-only frontend coverage
 - [testing/test-layers.md](testing/test-layers.md) — Which Go test layer runs where (PR CI / nightly / local only): `make test` is `-short` and covers **no** Postgres, `make test-postgres` and the `backend-postgres` job do, `SP_TEST_REQUIRE_POSTGRES` turns a skip into a failure, and the `slowtests` build tag holds the live-network/Docker suites
 - [testing/e2e-ci.md](testing/e2e-ci.md) — E2E test infrastructure: CI environment, Playwright config, local execution
+- [testing/local-e2e.md](testing/local-e2e.md) — Running dash0 E2E on a laptop: the Postgres side-car recipe, six traps that each fake a red run (stale embed, base URL, env parity...), the `make dev-test` fast path, and how to write specs that survive the full suite.
+- [testing/flaky-tests.md](testing/flaky-tests.md) — Flaky tests are bugs: rules, proof techniques (widen the window, wall-clock landmines), and the known shapes in this repo (testcontainer contention, hand-built middleware fixtures).
 - [testing/http-test-checks.md](testing/http-test-checks.md) — Fake API test checks: 5 predefined scenarios (stable, flaky, unstable, slow, 503)
 - [testing/dash0-e2e-typecheck-and-lint.md](testing/dash0-e2e-typecheck-and-lint.md) — Why `e2e/` has its own tsconfig and scoped lint, and the `page.evaluate` closure guard.
 

wiki/conventions/database.md

@@ -193,6 +193,36 @@ shipping a permanent migration. `014_v0_17_0` is the consolidated v0.17.0
 migration, unrelated to that draft; see the consolidation rule above for why
 reusing the number is safe here and what it costs dev databases.)
 
+### Diagnosing a "no such column" 500 on a dev database
+
+The usual cause: a spec appended a SECTION to the unreleased migration after your
+dev database had already applied it, so the new statements never ran.
+
+1. **Read the guard first.** `grep -i checksum logs/backend.log*` (glob the rotated
+   `.1`/`.2` files too: the boot-time warning may only be there). The guard names the
+   migration, both checksums and the repair options. It once logged the answer 67
+   times while someone read migration files instead.
+2. **Measure the blast radius.** Grep `logs/backend.log*` for the missing column, not
+   just the endpoint you noticed. The same missing column once also broke the check
+   worker's active-incident query, so incident detection was dead on every probe.
+3. **Pin the applied version.** `select checksum from migration_checksums where name='NNN'`
+   gives the sha256 of the file as applied. `git log --format=%h -- <migration>` then
+   `git show <sha>:<migration> | shasum -a 256` finds the commit whose version ran. The
+   missing delta is `git diff <sha> HEAD -- <migration>`; no `-` lines means it is purely
+   additive and can be applied by hand.
+4. **Prefer a hand fix over a reset when the data matters.** Apply the missing
+   statements, then sweep the current file object by object (every `create table`,
+   `alter table ... add column`, `create index` against `sqlite_master` /
+   `pragma table_info`, and each backfill `UPDATE`'s WHERE as a `count(*)`). A database
+   can be partially stale after earlier hand fixes, so do not assume. Only then run
+   `solidping migrate repair` (option 3 above). SQLite picks up an `ADD COLUMN` without a
+   restart.
+
+When a consolidation is under way, verify it by replay rather than by the test suite:
+apply every `*.up.sql` in order to an empty SQLite file, assert each folded section's
+objects exist, apply the `.down.sql` and assert they are gone. Before deleting a
+migration file, `grep 'migrations/NNN_' --include=*.go` for tests that read it by name.
+
 **The unreleased migration right now is `025_v0_34_0`** (opened 2026-09-27 by
 spec 2026-09-27-01, the "Capture now" failure columns on `check_jobs`).
 `024_v0_33_0` is the last RELEASED migration (it shipped in v0.33.0), so any

wiki/conventions/specs-workflow.md

@@ -48,3 +48,13 @@ How should we solve it?
 - When searching by slug, match partially (e.g., `credentials` matches `2025-12-06-credentials.md`)
 - Specs in `todos/` are processed oldest-first by `/implement-todos`
 - Archived specs go to `specs/done/YYYY/MM/` based on their date prefix
+- Never delete `specs/todos/` itself, even when emptying it
+- A spec's Proposal and Decisions define its scope. An implementation plan that narrows
+  them does not, and an audit reads the Proposal, not the plan
+- An **investigation** spec (diagnose, not build) names both legitimate outcomes: a
+  product-side cause gets a fix plus a regression test at the layer that was wrong; an
+  infra-side or unreachable cause gets a written finding in `wiki/` ("I could not reach
+  the data, here is what I tried and what to run next" is an honest finding). Restate
+  its non-goals as hard limits, and say whether small writes to a live system are
+  allowed to reach a diagnosis. With no code to commit, the closing
+  `chore: all checks passing for <title>` commit is made with `--allow-empty`

wiki/features/notifications-and-escalation.md

@@ -557,6 +557,15 @@ recorded; they just don't trigger paging. When the window ends, the
 next failing result will open a fresh incident if the failure persists.
 There is no "deferred" incident that fires at window end.
 
+Membership is only what the window lists: its checks, plus the checks of
+its groups (`ListMaintenanceWindowsForCheck`). **A window with nothing
+attached suppresses nothing** and shows no "under maintenance" badge on
+status pages. There is no "empty means org-wide" fallback. An operator
+who forgets to attach checks believes they are covered and gets paged
+during planned work, so dash0 renders a 0 count as an amber warning
+(`ChecksCountCell` in `maintenance-windows.index.tsx`), never as a
+neutral number. Keep it that way on any new surface showing the count.
+
 ### Per-incident silencing (ack / snooze / manual resolve)
 
 Operators can ack, snooze, or manually resolve an incident at any time.