Skip to content

Fix broken db seed and silent test discovery in the scraper - #199

Merged
ThatXliner merged 1 commit into
mainfrom
worktree-seed-and-test-fixes
Jul 26, 2026
Merged

Fix broken db seed and silent test discovery in the scraper#199
ThatXliner merged 1 commit into
mainfrom
worktree-seed-and-test-fixes

Conversation

@ThatXliner

Copy link
Copy Markdown
Collaborator

Two independent bugs, both live on main today. Split out of #198 so they can land on their own.

1. pnpm db:seed fails outright

Every one of the five fixture bill descriptions runs 119–135 characters, and bill.description is capped at 100 by the bill_description_max_100_chars check constraint:

H.R. 1001   132 chars  OVER LIMIT
S. 502      135 chars  OVER LIMIT
H.R. 2200   119 chars  OVER LIMIT
S. 789      124 chars  OVER LIMIT
H.R. 3456   129 chars  OVER LIMIT

The bills insert is the first statement in seed(), so it aborts the whole run — nothing gets seeded. Not bills, not government content, not court cases, not the video feed rows.

This is on the contributor onboarding path. pnpm onboard asks "Seed sample feed content for local development?", and everyone who says yes gets Sample-data seed failed. and an empty database.

Fix: clamp with clampBillDescription — the same helper the scraper and API already use — rather than hand-trimming the five strings. The fixtures stay readable as prose in the file, what lands in the table matches what a real scraped row looks like, and the invariant now holds by construction: you can't reintroduce the bug without deleting the clamp.

Result (truncates on word boundaries, as intended):

H.R. 1001 | 99 | A bill to authorize funding for the repair and modernization of roads, bridges, and public transit…
S. 502    | 98 | A bill to establish comprehensive federal data privacy protections for consumers and regulate the…

2. pnpm test in the scraper silently skips half the suite

The glob in tsx --test src/**/*.test.ts is unquoted, so the shell expands it before tsx ever sees it — and sh's ** behaves like *, matching exactly one directory level:

$ sh -c 'echo src/**/*.test.ts'
src/utils/bill-description.test.ts src/utils/reprocessing-policy.test.ts

src/env.test.ts and src/scraper-contracts.test.ts were never collected, and anything nested deeper than src/utils/ never would be. 2 of 4 files ran, and the suite reported green the whole time — which is the dangerous part: a broken test in an uncollected file looks identical to a passing one.

Fix: quote the pattern so globbing is handled by Node's test runner, which walks recursively.

before after
files collected 2 of 4 4 of 4
tests discovered 7 17

Verification

Both reproduced on a clean checkout of main first, then re-run after the fix:

  • Seed: applied migrations to a throwaway local Postgres, confirmed the constraint violation and zero rows inserted; after the fix, 5 bills / 4 gov content / 3 court cases / 12 videos insert cleanly. Inspected the clamped descriptions directly in the table.
  • Tests: 7 → 17 discovered, all passing.
  • Full monorepo typecheck, lint, and format pass.

Note on #198

#198 currently carries both of these fixes too — it needed the seed working to verify its own changes. Once this merges, I can rebase #198 to drop them, or leave it and resolve the (identical-content) conflict at merge time. Happy to do whichever you prefer.

🤖 Generated with Claude Code

Two independent bugs, both on main today.

`pnpm db:seed` fails outright. Every one of the five fixture bill descriptions
runs 119–135 characters, and `bill.description` is capped at 100 by the
`bill_description_max_100_chars` check constraint. The first insert aborts, so
nothing is seeded at all — not bills, not government content, not court cases,
not the video feed rows. `pnpm onboard` offers "Seed sample feed content for
local development?", so every contributor who says yes hits a hard failure and
ends up with an empty database.

Fixed by clamping with `clampBillDescription`, the same helper the scraper and
API already use, rather than hand-trimming the five strings. The fixtures stay
readable as prose, what lands in the table matches what a real scraped row
looks like, and the invariant now holds by construction — you cannot
reintroduce the bug without deleting the clamp.

`pnpm test` in the scraper silently skips half the suite. The glob in
`tsx --test src/**/*.test.ts` is unquoted, so the shell expands it before tsx
sees it, and sh's `**` behaves like `*` — matching exactly one directory level.
`src/utils/*.test.ts` was collected; `src/env.test.ts`, `src/*.test.ts`, and
anything nested deeper was not. 2 of 4 files ran, and the suite reported green
the whole time.

Quoting the pattern hands globbing to Node's test runner, which walks
recursively: 17 tests discovered instead of 7.

Verified by reproducing both on a clean checkout of main, then re-running:
the seed inserts 5 bills / 4 gov content / 3 court cases / 12 videos against a
throwaway local Postgres, and the clamped descriptions truncate on word
boundaries as intended.

Co-Authored-By: Claude <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
billion-nextjs Ready Ready Preview, Comment Jul 26, 2026 10:31pm

@ThatXliner
ThatXliner merged commit 99d7fef into main Jul 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant