Skip to content

perf: hourly usage aggregate (usage_hourly) to remove RDS IOPS bursts from analytics #1133

Description

@henrypark133

Problem

Prod RDS hits its storage ceiling (gp3 12k IOPS / 500 MiB/s) in scheduled bursts at :17–:24 each hour (nearai-stats sync) and :01 at 00/06/12/18 UTC. ~97% of burst load is admin analytics and customer usage-summary queries scanning raw organization_usage_log (43.6M rows, 12 GB heap, 25 GB indexes); month-wide windows fall into full sequential scans (~100–190 s per 30-day model-revenue call).

Plan

One derived table, usage_hourly (inference usage per UTC hour × org × workspace × key × model × provider), recomputed hourly from raw by a background job (replace semantics, data-derived progress, fast catch-up after deploy, per-day parity logs). Analytics readers switch to it; their raw aggregation SQL is deleted.

Shipping as one release, three stacked PRs:

  • PR B: table + hourly recompute job + parity log (no reader changes)
  • PR C: structural split of crates/database/src/repositories/analytics.rs by report family (no behavior change)
  • PR D: switch admin analytics + customer usage summaries + API-key list to usage_hourly; 504 on statement timeout; lag metric

Stays on raw by design: credit_type-filtered reports, row-level history/exports, /v1/billing/costs, per-minute revenue density, per-key admission.

Context

Success criteria

  • PI read IOPS/throughput in the burst windows < 50% of the ceiling
  • Admin analytics endpoints p95 < 2 s
  • nearai-stats monthly streams succeed with their default deadline

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions