[ebeirne]

Base: dev · Branch: feat/conditional-requests (2 commits, be5769c, 9260867)

Why

Spiders that run on a schedule, such as one that checks a set of feeds every few minutes, mostly re-download pages that haven't changed. Most servers support conditional requests, but nothing in scrapling/spiders/ sends If-None-Match / If-Modified-Since or understands a 304. Doing it by hand in a spider is awkward: headers set on a request are inherited by response.follow(), and the validators have to be kept somewhere between runs.

What changes

• New Spider attributes conditional_requests = False and conditional_requests_dir = None (default .scrapling_validators/{name}/, mirroring .scrapling_cache/).
• New ConditionalRequestManager (spiders/conditional.py) with the same on-disk layout and atomic writes as ResponseCacheManager: one small JSON file per request fingerprint holding the ETag / Last-Modified values.
• In the engine:
  • Before fetching a GET/HEAD request, the saved validators are sent as If-None-Match / If-Modified-Since.
  • A 304 answer to a conditional request skips the callback and calls a new Spider.on_not_modified(request, response) hook (a no-op by default). It counts in a new not_modified_count stat, and AutoThrottle records it as healthy. Without that, AutoThrottle would back off on every unchanged page.
  • Validators are saved only after a 2xx response's callback finished without raising, so a page that failed (or was blocked) is downloaded in full next time. A response with no validators deletes the old entry.
  • Requests that set their own conditional headers, and non-GET/HEAD requests, are left alone.
• SessionManager.fetch() takes optional per-fetch headers that are merged into a copy of the kwargs. The request is never modified, so response.follow() never inherits the validators. Browser sessions read extra_headers, so they ignore these and simply fetch normally.
• Development mode: conditional requests are disabled, with a warning, while development_mode is on. A cache miss needs the full body to cache, and a 304 would leave nothing to replay. cache.py behaves exactly as before.

Tests

22 new cases in tests/spiders/test_conditional.py. Most drive a real Spider against a fake site that implements RFC 9110 conditional logic, plus one end-to-end case through the default FetcherSession against pytest-httpbin's /etag/{etag}. They cover:

• the 304 skip and hook, changed pages, Last-Modified alone

• failed callbacks and blocked responses not being remembered

• followed requests not inheriting validators, and shared header dicts staying untouched

• user-set validators, POST requests, disabled-by-default

• the AutoThrottle health flag, and development mode replaying from the cache unchanged

• pytest tests/spiders -m "not browser": 528 passed (506 on dev)

• ruff, mypy scrapling/, vermin -t=3.10-, bandit: clean. pyright: no new errors.

Notes

• Overlaps with feat(spiders): add an opt-in persistent seen-set that survives between runs #535 (persistent seen-set): both change the end of CrawlerEngine._run_callbacks. Whichever lands second needs a small rebase: keep return False in the except, then the seen-store add, then return True. I checked that combination locally: 577 passed.
• _run_callbacks now returns bool (whether the callback finished without raising). It's private.
• Docs: new "Conditional Requests" section in docs/spiders/advanced.md (English only).
• No new dependencies.

AI disclosure

This PR was written primarily by Claude Code, Anthropic's coding agent: the code, the tests, the docs and this description. I asked for these changes while building a news-monitoring project on top of Scrapling's spiders, where the gaps came up, and I chose to submit them. The test results above come from local runs on Windows with Python 3.12. Happy to make any changes.

🤖 Generated with Claude Code