Fix/3187 tvtv us scraper - #3243
Open
Sicilykill wants to merge 3 commits into
Open
Conversation
- Update API endpoint from /api/v1/lineup to /partial/source - Implement Playwright adapter to bypass Cloudflare protection - Move playwright-adapter to scripts/helpers for reusability - Change parser from JSON to HTML using cheerio - Fix timestamp generation to use midnight UTC - Add HTMX-specific headers - Update tests for new HTML structure - Add Playwright integration to grab.ts via axios interceptors - Mark tvtv.us as working in SITES.md Fixes iptv-org#3187 Fixes iptv-org#3147
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes the tvtv.us scraper by migrating to the new HTMX-based API and implementing Cloudflare bypass using Playwright.
Problem
The tvtv.us site changed their API structure:
/api/v1/lineup/.../grid/...) returns 404 errorsSolution
/api/v1/lineup/.../grid/...to/partial/source/{timestamp}/{channel_id}dayjs(timestamp)todayjs.utc(timestamp)to correctly handle UTC timesChanges
scripts/helpers/playwright-adapter.jsShared Playwright adapter (reusable by other sites if needed)Additional Notes
The Playwright adapter is generic and reusable for any site requiring Cloudflare bypass. It's in a shared location (
scripts/helpers/playwright-adapter.js) so other scrapers can easily use it by:sitesNeedingPlaywrightarray in [grab.ts]adapter: playwrightAdapterin their site configSetup Requirements
After merging, users will need to install Chromium browser binaries. I guess we could add this as a post script? Not sure on how this should be handled.
Fixes issues #3187 and partially #3147