Skip to content

[Ledger] Replace Event#merchants scan with a Ledger-scoped lookup#14356

Open
garyhtou wants to merge 3 commits into
mainfrom
garyhtou/ledger-merchants-perf
Open

[Ledger] Replace Event#merchants scan with a Ledger-scoped lookup#14356
garyhtou wants to merge 3 commits into
mainfrom
garyhtou/ledger-merchants-perf

Conversation

@garyhtou

@garyhtou garyhtou commented Jul 18, 2026

Copy link
Copy Markdown
Member

Note

This PR was written by Claude Code. Gary directed the investigation and reviewed the changes, but the diagnosis, code, and benchmarks below were produced by Claude.

Summary of the problem

The Ledger page's merchant filter — both the @merchant_name label (set_ledger_filters.rb) and the merchants_filter dropdown action — called Event#merchants, which:

  • loads every CanonicalTransaction and CanonicalPendingTransaction the event has ever had, with no date bound or limit,
  • N+1s each one's raw_stripe_transaction/raw_pending_stripe_transaction,
  • calls YellowPages::Merchant.lookup once per transaction rather than deduping first.

Per Gary's rule for this system — rendering the Ledger should only touch Ledger, Ledger::Item, Ledger::Mapping, and linked_object — this also reached directly into the old transaction-engine tables, which the Ledger page shouldn't need to know about at all.

Describe your changes

Replaced with a new ledger_merchants(ledgers) helper that:

  1. Starts from Ledger::Item scoped to the given ledgers, filtered to linked_object_type: "CardCharge" — so it only ever reflects what's actually on this Ledger, via linked_object (a CardCharge), never the raw transaction engine directly.
  2. Extracts just the merchant_data key from the underlying JSONB blob using Postgres's -> operator, instead of loading full RawStripeTransaction/RawPendingStripeTransaction records. merchant_data is a small part of a multi-KB JSONB payload (~2.6KB average per row); loading the whole blob per charge just to read one key turned out to be most of the remaining cost after fixing the N+1.
  3. Falls back from a charge's latest settled transaction to its pending one, mirroring CardCharge#merchant_data's own raw_stripe_transactions.last || raw_pending_stripe_transaction logic.

The legacy (non-Ledger) Transactions page still calls Event#merchants directly for its own filter menu and is untouched — that page isn't bound by the Ledger-only-access rule.

Benchmarks

Medians, before → after, across a few orgs (public IDs shown for the rest):

Org Before After Speedup
Hack Club HQ 33,598ms 88ms 382x
HCB Operations 11,393ms 32ms 353x
org_lbu5qR 16,259ms 46ms 353x
org_XDunKA 9,340ms 40ms 236x
org_MNOuwZ 7,179ms 24ms 303x
org_a7Xunb 5,930ms 26ms 228x

A disclosed behavior difference

The new method returns slightly fewer distinct merchants than the old one on long-running orgs — e.g. 2,187 vs 2,959 on Hack Club HQ. This is because it's scoped to CardCharge-linked Ledger::Items that actually exist; some very old canonical_transactions predate the Ledger migration and were never backfilled with a corresponding Ledger::Item. Since the merchant filter isn't wired up to actually filter query results yet (# TODO: add filtering for merchant in ledger_query), this list is display-only today, and I think matching what's actually navigable on the Ledger page is more correct than the old event-wide scan, not less — but flagging it explicitly since the numbers differ.

Tests

spec/controllers/events_controller_spec.rb covers both call sites through their actual rendered response (render_views + asserting on response.body, not internal controller state): the merchant name resolving into the active-filter badge, the generic fallback when a merchant isn't found, distinct merchants rendering most-transacted-first, and a direct assertion on ledger_merchants' return value for the exact per-merchant counts. The one I'd flag as most important given the scoping change: that another org's ledger doesn't leak into the merchant list.

The ledger page's merchant filter (both the @merchant_name label in
set_ledger_filters and the merchants_filter dropdown action) called
Event#merchants, which loads every CanonicalTransaction and
CanonicalPendingTransaction the event has ever had, N+1s each one's
raw_stripe_transaction, and calls YellowPages::Merchant.lookup per
transaction (not deduped) - all just to list distinct merchants.
Rendering the Ledger should only need Ledger, Ledger::Item,
Ledger::Mapping, and linked_object; this reached around all of that
into the old transaction engine directly.

Replaced with ledger_merchants(ledgers): starts from Ledger::Item
scoped to the given ledgers (linked_object_type: "CardCharge"), then
extracts just the `merchant_data` JSON key via Postgres's `->`
operator instead of loading full RawStripeTransaction/
RawPendingStripeTransaction records - merchant_data is a small part of
a multi-KB JSONB blob, and loading the whole blob per charge (~2.6KB
average) to read one key was most of the remaining cost after fixing
the N+1. Falls back from the latest settled transaction to the
pending one per charge, matching CardCharge#merchant_data's own logic.

The legacy (non-Ledger) Transactions page still uses Event#merchants
directly and is unaffected.

Benchmarked medians across several orgs (before -> after):
- Hack Club HQ: 33,598ms -> 88ms (382x)
- HCB Operations: 11,393ms -> 32ms (353x)
- 4 other orgs (org 5516, 2924, 1692, 1362): 5,930-16,259ms -> 24-46ms
  (228-353x)

The new method returns slightly fewer distinct merchants than the old
one on long-running orgs (e.g. 2,187 vs 2,959 on Hack Club HQ) because
it's scoped to CardCharge items that actually exist on the Ledger -
some very old canonical_transactions predate the Ledger migration and
were never backfilled with a corresponding Ledger::Item. Since the
merchant filter isn't wired to actually filter query results yet (see
the "TODO: add filtering for merchant" in ledger_query), this is a
display-only list, and matching what's actually in the Ledger is more
correct for a Ledger page than the old event-wide scan.

spec/controllers/events_controller_spec.rb covers both call sites,
including that another org's ledger doesn't leak into the merchant
list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@garyhtou
garyhtou marked this pull request as ready for review July 19, 2026 01:26
@garyhtou
garyhtou requested a review from a team July 19, 2026 01:26
garyhtou and others added 2 commits July 18, 2026 18:36
…ontroller internals

The merchant filter tests read state off the controller with
instance_variable_get(:@merchant_name / :@merchants) after firing an
unrelated action - testing internal state a real caller never sees,
rather than the response the controller actually produces. A
Rails-testing-focused review flagged this.

Added render_views and switched to asserting on response.body: the
merchant name appears in the active-filter badge (#ledger), and
distinct merchant names appear in the dropdown, most-transacted-first
(#merchants_filter, tested via rendered order since the count itself
isn't displayed). Kept one direct assertion on ledger_merchants'
return value for the exact per-merchant counts, since that's real
grouping/counting logic worth pinning precisely, called directly
rather than inferred from unrelated state.

Verified the cross-ledger test is not vacuous: temporarily removed
the ledger_mappings scoping from ledger_merchants and confirmed it
fails, then restored.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant