Skip to content

Make import-wikidata WDQS queries resilient with retries and backoff - #474

Open
arnekaiser wants to merge 1 commit into
openmaptiles:masterfrom
arnekaiser:fix/import-wikidata-wdqs-retries
Open

Make import-wikidata WDQS queries resilient with retries and backoff#474
arnekaiser wants to merge 1 commit into
openmaptiles:masterfrom
arnekaiser:fix/import-wikidata-wdqs-retries

Conversation

@arnekaiser

Copy link
Copy Markdown

WDQS (Wikidata Query Service) requests can fail transiently with HTTP 429/5xx responses, connection errors, timeouts, or truncated/garbled JSON bodies (e.g. JSONDecodeError at char 12881888). Previously a single such failure aborted the entire import run, losing all progress.

Fixes #468

Changes to bin/import-wikidata:

  • wd_query() rewritten with a retry loop (up to WDQS_MAX_RETRIES=5) using exponential backoff (WDQS_BACKOFF_BASE=5s -> 5,10,20,40,80s) and honoring the Retry-After header. Handles connection errors, timeouts, retryable HTTP statuses (429,500,502,503,504), unexpected Content-Type, empty response bodies, and unparseable JSON.
  • Add _backoff_sleep() helper computing the retry delay.
  • Add an explicit request timeout (WDQS_TIMEOUT=120s).
  • Reduce batch sizes to be gentler on WDQS: create_ids_from_wdqs 5000->1000, resolve_redirects 1000->500.
  • Add a 1s inter-batch delay (WDQS_INTER_BATCH_DELAY) between consecutive WDQS batches to respect rate limits.
  • Factor out save_cache() and persist the cache on permanent WDQS failure (in main) so a re-run can skip already-retrieved IDs.
  • Save cache whenever use_wdqs and new_ids are set (not only when cache is non-None).

WDQS (Wikidata Query Service) requests can fail transiently with HTTP
429/5xx responses, connection errors, timeouts, or truncated/garbled
JSON bodies (e.g. JSONDecodeError at char 12881888). Previously a single
such failure aborted the entire import run, losing all progress.

Changes to bin/import-wikidata:

* wd_query() rewritten with a retry loop (up to WDQS_MAX_RETRIES=5)
  using exponential backoff (WDQS_BACKOFF_BASE=5s -> 5,10,20,40,80s)
  and honoring the Retry-After header. Handles connection errors,
  timeouts, retryable HTTP statuses (429,500,502,503,504), unexpected
  Content-Type, empty response bodies, and unparseable JSON.
* Add _backoff_sleep() helper computing the retry delay.
* Add an explicit request timeout (WDQS_TIMEOUT=120s).
* Reduce batch sizes to be gentler on WDQS: create_ids_from_wdqs
  5000->1000, resolve_redirects 1000->500.
* Add a 1s inter-batch delay (WDQS_INTER_BATCH_DELAY) between
  consecutive WDQS batches to respect rate limits.
* Factor out save_cache() and persist the cache on permanent WDQS
  failure (in main) so a re-run can skip already-retrieved IDs.
* Save cache whenever use_wdqs and new_ids are set (not only when
  cache is non-None).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

import-wikidata stopped working

1 participant