Make import-wikidata WDQS queries resilient with retries and backoff - #474
Open
arnekaiser wants to merge 1 commit into
Open
Make import-wikidata WDQS queries resilient with retries and backoff#474arnekaiser wants to merge 1 commit into
arnekaiser wants to merge 1 commit into
Conversation
WDQS (Wikidata Query Service) requests can fail transiently with HTTP 429/5xx responses, connection errors, timeouts, or truncated/garbled JSON bodies (e.g. JSONDecodeError at char 12881888). Previously a single such failure aborted the entire import run, losing all progress. Changes to bin/import-wikidata: * wd_query() rewritten with a retry loop (up to WDQS_MAX_RETRIES=5) using exponential backoff (WDQS_BACKOFF_BASE=5s -> 5,10,20,40,80s) and honoring the Retry-After header. Handles connection errors, timeouts, retryable HTTP statuses (429,500,502,503,504), unexpected Content-Type, empty response bodies, and unparseable JSON. * Add _backoff_sleep() helper computing the retry delay. * Add an explicit request timeout (WDQS_TIMEOUT=120s). * Reduce batch sizes to be gentler on WDQS: create_ids_from_wdqs 5000->1000, resolve_redirects 1000->500. * Add a 1s inter-batch delay (WDQS_INTER_BATCH_DELAY) between consecutive WDQS batches to respect rate limits. * Factor out save_cache() and persist the cache on permanent WDQS failure (in main) so a re-run can skip already-retrieved IDs. * Save cache whenever use_wdqs and new_ids are set (not only when cache is non-None).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WDQS (Wikidata Query Service) requests can fail transiently with HTTP 429/5xx responses, connection errors, timeouts, or truncated/garbled JSON bodies (e.g. JSONDecodeError at char 12881888). Previously a single such failure aborted the entire import run, losing all progress.
Fixes #468
Changes to bin/import-wikidata: