Skip to content

Update minimizing-costs guide: rank estimates are now accurate - #1379

Merged
max-ostapenko merged 1 commit into
mainfrom
docs-rank-dry-run-estimates
Oct 23, 2026
Merged

max-ostapenko merged 1 commit into
mainfrom
docs-rank-dry-run-estimates

Conversation

@max-ostapenko

Copy link
Copy Markdown
Contributor

BigQuery's bytes-processed estimates now account for cluster pruning on rank, so the guide's warning about the estimate bug is out of date.

Checked on the guide's own example (crawl.pages, 2023-05-01, desktop, rank <= 1000): the dry run estimated 92.9 MB and the run processed 91.2 MB. On the 2026-09 crawl, estimates matched processed bytes exactly for rank filters on both crawl.pages and crawl.requests, with and without client/is_root_page.

Changes:

  • Clustering caution: keep the "no guarantee" link, say estimates now reflect rank.
  • New tip: check the console estimate or bq query --dry_run (both free) instead of relying on figures in the guide.
  • "Use RANK": replace the "leap of faith" paragraph with advice to widen rank step by step, plus a note about the old bug.
  • sample_data: reposition as a small fixed set for developing query logic; note it's rebuilt each crawl.
  • Comparison table: rank now ✅ for accurate estimates; closing line recommends rank, with TABLESAMPLE for random samples.
  • Remove hardcoded scan sizes (329 MB, 10 MB, 6.56 TB, 680 MB). They no longer matched even for the fixed 2023-05-01 month (e.g. a11y is now ~3 MB, the full-month query ~0.7 TiB), so the text now compares relatively.

@max-ostapenko
max-ostapenko merged commit 85dd616 into main Oct 23, 2026
4 of 5 checks passed
@max-ostapenko
max-ostapenko deleted the docs-rank-dry-run-estimates branch October 23, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants