Skip to content

Fix/3187 tvtv us scraper - #3243

Open
Sicilykill wants to merge 3 commits into
iptv-org:masterfrom
Sicilykill:fix/3187-tvtv-us-scraper
Open

Fix/3187 tvtv us scraper#3243
Sicilykill wants to merge 3 commits into
iptv-org:masterfrom
Sicilykill:fix/3187-tvtv-us-scraper

Conversation

@Sicilykill

@Sicilykill Sicilykill commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes the tvtv.us scraper by migrating to the new HTMX-based API and implementing Cloudflare bypass using Playwright.

Problem

The tvtv.us site changed their API structure:

  • Old API endpoint (/api/v1/lineup/.../grid/...) returns 404 errors
  • Timestamps were incorrectly treated as local time instead of UTC, causing a 4-hour offset for users

Solution

  • API Migration: Updated from /api/v1/lineup/.../grid/... to /partial/source/{timestamp}/{channel_id}
  • Cloudflare Bypass: Implemented Playwright headless browser adapter in shared location
  • Timezone Fix: Changed from dayjs(timestamp) to dayjs.utc(timestamp) to correctly handle UTC times
  • Parser Update: Switched from JSON to HTML parsing using cheerio
  • Integration: Added automatic Playwright detection in [grab.ts]

Changes

  • Updated sites/tvtv.us/tvtv.us.config.js New API endpoint, fixed timestamps, HTML parser
  • Created scripts/helpers/playwright-adapter.js Shared Playwright adapter (reusable by other sites if needed)
  • Updated sites/tvtv.us/tvtv.us.test.js Tests for new HTML structure
  • Created sites/tvtv.us/README.md Documentation for the scraper
  • Modified scripts/commands/epg/grab.ts Auto-loads Playwright for sites that need it
  • Updated SITES.md Marked tvtv.us as working (🟢)
  • Updated package.json Added playwright dependency

Additional Notes

The Playwright adapter is generic and reusable for any site requiring Cloudflare bypass. It's in a shared location (scripts/helpers/playwright-adapter.js) so other scrapers can easily use it by:

  1. Adding to the sitesNeedingPlaywright array in [grab.ts]
  2. Using adapter: playwrightAdapter in their site config

Setup Requirements

After merging, users will need to install Chromium browser binaries. I guess we could add this as a post script? Not sure on how this should be handled.

npm install
npx playwright install chromium

Fixes issues #3187 and partially #3147

- Update API endpoint from /api/v1/lineup to /partial/source
- Implement Playwright adapter to bypass Cloudflare protection
- Move playwright-adapter to scripts/helpers for reusability
- Change parser from JSON to HTML using cheerio
- Fix timestamp generation to use midnight UTC
- Add HTMX-specific headers
- Update tests for new HTML structure
- Add Playwright integration to grab.ts via axios interceptors
- Mark tvtv.us as working in SITES.md

Fixes iptv-org#3187
Fixes iptv-org#3147
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant