Prefetch Hermes+Lazer feeds at build time for the docs search index - #3917
Prefetch Hermes+Lazer feeds at build time for the docs search index#3917aditya520 wants to merge 4 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
5 Skipped Deployments
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 637287c22e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "generate:price-feeds": { | ||
| "dependsOn": ["//#install:modules"], | ||
| "outputs": ["src/generated/**"] | ||
| }, |
There was a problem hiding this comment.
When only the upstream Hermes/Lazer feed lists change (or a transient API outage produced an incomplete snapshot), this task can be served from Turbo cache instead of re-fetching: Turborepo's config docs state that cache defaults to true and task inputs default to source-controlled package files, while src/generated/ is gitignored and the API responses are external (https://turborepo.dev/docs/reference/configuration#cache). In the inspected Vercel path (apps/developer-hub/vercel.json runs turbo run build --filter @pythnetwork/developer-hub), a cache hit replays the old snapshot/.next output, so search can stay stale until a tracked file changes or the cache is forced; please make the snapshot contents part of the build cache key or disable caching for the relevant generation/build path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Both halves of this are now addressed:
- Generation caching:
generate:price-feedsiscache: falsesince3b18e659d, so the snapshot is re-fetched on every build. - Build cache key: fixed in
ab3b14b4f.src/generated/price-feeds.jsonis gitignored, so it wasn't in the build task's default inputs ($TURBO_DEFAULT$hashes tracked files only) — a warm cache could replay a stale.nextover a fresh snapshot. The snapshot is now listed inbuild.inputs.
Verified: the file appears in the hashed inputs via --dry=json; removing a single feed from the snapshot flips the build hash (147893e6… → 8feb5d3b…); and two consecutive builds with unchanged feed data still cache-hit, since Turbo hashes inputs after dependency tasks run — so the key always reflects what the generator just wrote, and caching stays useful when feed data hasn't changed.
3da01a6 to
3cddf64
Compare
…cs search The docs search route fetched ~6.2 MB across three external APIs (hermes.pyth.network, hermes-beta.pyth.network, and the Lazer symbols API) on every cold-start lambda before building the Orama index. Move those fetches to a build-time generator that writes a static `src/generated/price-feeds.json` snapshot, which the route now imports synchronously. - New `scripts/generate-price-feeds-snapshot.ts`: fetches the three sources in parallel, validates with the shared zod schemas, and writes the snapshot. Best-effort per source — a failed fetch warns and falls back to the previously-generated file (or an empty set on a cold build) and never fails the build. - Extract the Hermes/Lazer zod schemas and types into a shared `feed-schemas.ts` consumed by both the route and the generator. - The `/api/search` route performs zero external HTTP calls at runtime; its `indexes` callback is now fully synchronous. Feed titles, IDs, URLs, and `price-feed-core` / `price-feed-pro` tags are unchanged. - Wire `generate:price-feeds` into package.json and turbo.json (build dependency + cached `src/generated/**` output); gitignore the output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The generate:price-feeds task fetches feed lists from external Hermes/Lazer APIs, whose contents are not reflected in any source-controlled input. With Turbo's default cache:true, a cache hit replays a stale (or transient-outage incomplete) snapshot until an unrelated tracked file changes. Mirror the generate:changelog precedent for gitignored, externally-fetched build artifacts: set cache:false so the snapshot is always regenerated. The task still runs before build via build.dependsOn, so the route's static-import target is always present. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The search route returned every match with no cap: "e" produced 16,802
results / 8.1 MB and pasting a feed symbol took 4.66s against production.
fumadocs groups Orama hits by page_id with no limit on group count, and
every one of the ~9.4k feeds is its own group, so any common token
("usd", "price", "e") matched thousands of them. Note that Orama's
`limit` does not bound group count, so the cap has to be applied in the
route handler.
- Cap responses at 24 results with per-family quotas (12 docs / 6 Core /
6 Pro, unused slots backfilled). Without quotas "USD" pushed the first
documentation hit to rank 624 and "solana" surfaced no feeds at all.
- Drop `structuredData.contents` from feed entries. fumadocs already
indexes `title` and `description` as their own documents, so those
entries duplicated text at 2-3x the document count. Symbol, name,
description and feed ID all remain searchable.
- Deduplicate Core feeds by symbol. hermes-beta republishes 99.4% of the
mainnet symbol list (2,873 of 2,890) into rows with identical titles
and identical URLs, i.e. literal duplicate entries in the dialog.
Mainnet wins ties; the 17 beta-only symbols stay searchable.
- Add a deterministic exact-match pass over feed symbols and names.
BM25 over the remaining documents does not favour the feed a user
typed: searching "Crypto.BTC/USD" returned 24 results without that
feed among them.
- Apply the biome fixes CI misses (`biome ci --changed` reports
"Checked 0 files" on this repo's PR runs).
Measured over 6 interleaved A/B rounds of local production builds
(15-query set, 9 iterations each): 542.9ms -> 117.5ms median total
latency (4.62x) and 38.57 MB -> 0.12 MB total payload. Worst pairing of
fastest-baseline against slowest-optimized round is 3.23x. Loopback
makes payload nearly free, so this understates the production gain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3cddf64 to
3b18e65
Compare
Added scope: search performance and relevance (
|
| before | after | |
|---|---|---|
| median total latency | 542.9ms | 117.5ms |
| total payload | 38.57 MB | 0.12 MB |
4.62x median. Per-round: 4.75x, 4.27x, 3.23x, 4.44x, 4.86x, 5.32x. Pairing the fastest baseline round against the slowest optimized round still gives 3.23x. Loopback makes an 8 MB response nearly free, so this understates the production gain.
Verification. 20/20 correctness checks — Core by symbol / bare ticker / 64-char feed ID, Pro by name / symbol / numeric Lazer ID / description word, BTC+ETH+USD each returning both families, docs still reachable on prose queries, no duplicate feed rows. Build passes 45/45, tsc and biome check clean.
Still open on this PR
Two items from review that are not addressed here:
- An upstream outage at build time silently ships an empty feed index. The fallback to "the previously-generated snapshot" cannot work on Vercel:
src/generated/is gitignored and the task iscache: false, so there is never a previous snapshot to read. The build goes green and search quietly loses thousands of entries. - No refresh path. No cron, no ISR, so new feeds only appear on the next deploy. Lazer added 64 feeds in the 5 days after this PR was opened.
generate:price-feeds is cache:false, but src/generated/price-feeds.json is gitignored and therefore absent from the build task's default inputs ($TURBO_DEFAULT$ hashes tracked files only). A warm build cache could replay a stale .next over a freshly generated snapshot, so search served old feeds until an unrelated tracked file changed. List the snapshot in the build task's inputs so its content is part of the cache key. Package-level inputs replace the root list, so the root excludes are restated. Verified: the file now appears in the hashed inputs (--dry=json), removing a single feed flips the build hash, and two consecutive builds with unchanged feed data still produce a cache hit -- Turbo hashes inputs after dependency tasks run, so the key always reflects what the generator just wrote. Addresses the remaining half of the Codex review comment on turbo.json (the cache:false half landed earlier). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
The developer-hub docs search route (
apps/developer-hub/src/app/api/search/route.ts) fetched ~6.2 MB across three external APIs —hermes.pyth.network,hermes-beta.pyth.network, and the Lazer symbols API — on every cold-start lambda before building the Fumadocs/Orama index. This moves those fetches to build time so the route serves the feed list from a locally-generated static JSON snapshot imported synchronously.No behavior change from the user's perspective: feed titles, IDs, links, and
price-feed-core/price-feed-protags stay identical, and global docs search still surfaces feed entries.Changes
apps/developer-hub/scripts/generate-price-feeds-snapshot.ts— fetches the three sources in parallel, validates them with the shared zod schemas, and writessrc/generated/price-feeds.json. Best-effort per source: a failed fetch/validation logs a warning and falls back to the previously-generated snapshot (or an empty set on a cold build). A top-leveltry/catchguarantees the import target exists so the build never fails.apps/developer-hub/src/app/api/search/feed-schemas.ts— extracts the Hermes/Lazer zod schemas, their inferred types, and thePriceFeedsSnapshotshape into a module shared by both the route and the generator (avoids schema drift).route.ts— replaces thegetHermesFeeds()/getLazerFeeds()runtime fetches with a static import of the snapshot. Theindexescallback is now fully synchronous and performs zero external HTTP calls. The AdvancedIndex transforms (title / description / url / id / tag / structuredData) are byte-identical to before.package.json— addsgenerate:price-feedsalongsidegenerate:docs/generate:changelog.turbo.json— adds thegenerate:price-feedstask and wires it intobuild.dependsOn. The task iscache: falseand itssrc/generated/**output is intentionally kept out of thebuildtask outputs, mirroring the existinggenerate:changelogprecedent for gitignored, externally-fetched build artifacts. Because the feed lists come from external APIs (not source-controlled inputs), a cacheable task would replay a stale — or transient-outage incomplete — snapshot until an unrelated tracked file changed;cache: falsemakes it regenerate on every build, and keeping it out of the build outputs means a warm build cache never restores a stale snapshot over the freshly generated one. The task still runs beforebuildviabuild.dependsOn, so the route's static-import target is always present..gitignore— ignoressrc/generated/(regenerated at each build, never committed).Verification
pnpm turbo run test:types --filter=@pythnetwork/developer-hub— pass (full build rangenerate:price-feedsfresh, thentsc).pnpm turbo run test:lint --filter=@pythnetwork/developer-hub— pass.pnpm generate:price-feedson a deletedsrc/generated/— recreates the dir and writes the snapshot (hermes 3056,hermes-beta 2890,lazer 3435).turbo run generate:price-feeds --dry=jsonconfirmscache.local: false/cache.remote: falsefor the task.git merge-tree origin/main HEAD— no conflicts.