[DEV-15813] Spending explorer limit applied only after full result materialization - #4726
[DEV-15813] Spending explorer limit applied only after full result materialization#4726james-at-kc wants to merge 8 commits into
Conversation
james-at-kc
changed the title
There was a problem hiding this comment.
Since we have the limit hard coded on this endpoint do you mind creating a test that patches the limit to be lower (e.g, 2) and validate that the resulting endpoint call returns only 2 results? Other comments aren't worth blocking over, more just a suggestion and a question about performance.
| exp = Explorer(alt_set, queryset) | ||
| if _type == "award": | ||
| # Cheap full-set total (single aggregate row) so response total matches prior behavior | ||
| actual_total = ( |
There was a problem hiding this comment.
Good catch! Not something that was picked up in refinement, but definitely some logic that needs to stay the same. Have you noticed any drop in performance with two queries? I would imagine not since it is now doing much less work overall.
| ) -> dict[str, Any]: | ||
| """Process award, award_category, and recipient types. | ||
|
|
||
| Mythos - apply early limit - 8/2026 |
There was a problem hiding this comment.
Small and won't block over it, but I would remove the part about Mythos here since someone could find the ticket associated with the commit if they needed. The comment below tells me everything I need to know.
2 participants
Description:
The /api/v2/spendingendpoint pulls more data from the DB than necessary when searching on a type of “award”. Django querysets won’t hit the DB until an action is taken on the queryset (e.g., converted to a list, retrieving a count, etc.). However, once that action is taken the query will pull all relevant data without a limit; unless a limit has been specified. The endpoint is currently putting more pressure on the DB than necessary by converting a queryset to a list prior to applying a limit via a slice: usaspending-api/usaspending_api/spending_explorer/v2/filters/type_filter.py at 055c58f48fb210568e1bf826356e75409faa9559 · fedspendingtransparency/usaspending-api
Technical Details:
The Explorer class supports a limit parameter which has a default value and is applied to the methods of the class
The default and maximum number of records that can be retrieved is currently defined in SPENDING_EXPLORER_LIMIT (see usaspending-api/usaspending_api/spending_explorer/v2/views/spending_explorer.py at 055c58f48fb210568e1bf826356e75409faa9559 · fedspendingtransparency/usaspending-api)
The filters/type_filter.py file continues to function as expected with the new limit parameter in place.
Rationale for changes:
The spending explorer's /api/v2/spending/ endpoint was asking the database for ALL matching award rows, loading them into the app, then keeping only the top 1000.
This change asks the database to consider at most 1000 rows (default limit) while keeping the same behavior for other explorer types (non award types).
The existing code only capped award types so this behavior was brought forward.
We moved the Spending Explorer award cap from "fetch everything then slice in Python" to "SQL limit on the explorer queryset".
We kept non-award and unreported behavior unchanged (per original Mythos finding).
We refactored type_filter.py to conform to ruff's complexity flag.
Requirements for PR Merge:
Explain N/A in above checklist:
2 - there was no change to the API
3 - there was no change to the data returned so there was no impact on the frontend
4 - no ops involvement other than to deploy the merged branch