Skip to content

fix: update ondemand.s regex for new webpack chunk format #411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ryanstoic wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: update ondemand.s regex for new webpack chunk format #411

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 14 additions & 11 deletions twikit/x_client_transaction/transaction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@
from .utils import float_to_hex, is_odd, base64_encode, handle_x_migration

ON_DEMAND_FILE_REGEX = re.compile(
r"""['|\"]{1}ondemand\.s['|\"]{1}:\s*['|\"]{1}([\w]*)['|\"]{1}""", flags=(re.VERBOSE | re.MULTILINE))
INDICES_REGEX = re.compile(
r"""(\(\w{1}\[(\d{1,2})\],\s*16\))+""", flags=(re.VERBOSE | re.MULTILINE))
r',(\d+):["\']ondemand\.s["\']', flags=(re.VERBOSE | re.MULTILINE))
ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"'
INDICES_REGEX = re.compile(r'\[(\d+)\],\s*16')
Comment on lines 15 to +18

@coderabbitai coderabbitai Bot Mar 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Quote character inconsistency between regex patterns.

ON_DEMAND_FILE_REGEX correctly handles both single and double quotes around ondemand.s, but ON_DEMAND_HASH_PATTERN only matches double quotes around the hash value. If the webpack output uses single quotes for hashes (as it might for chunk names), the second lookup will fail silently.

Proposed fix to handle both quote types 
-ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"'
+ON_DEMAND_HASH_PATTERN = r',{}:["\']([0-9a-f]+)["\']'
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
ON_DEMAND_FILE_REGEX = re.compile(
r"""['|\"]{1}ondemand\.s['|\"]{1}:\s*['|\"]{1}([\w]*)['|\"]{1}""", flags=(re.VERBOSE | re.MULTILINE))
INDICES_REGEX = re.compile(
r"""(\(\w{1}\[(\d{1,2})\],\s*16\))+""", flags=(re.VERBOSE | re.MULTILINE))
r',(\d+):["\']ondemand\.s["\']', flags=(re.VERBOSE | re.MULTILINE))
ON_DEMAND_HASH_PATTERN = r',{}:"([0-9a-f]+)"'
INDICES_REGEX = re.compile(r'\[(\d+)\],\s*16')
ON_DEMAND_FILE_REGEX = re.compile(
r',(\d+):["\']ondemand\.s["\']', flags=(re.VERBOSE | re.MULTILINE))
ON_DEMAND_HASH_PATTERN = r',{}:["\']([0-9a-f]+)["\']'
INDICES_REGEX = re.compile(r'\[(\d+)\],\s*16')
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@twikit/x_client_transaction/transaction.py` around lines 15 - 18, The
ON_DEMAND_HASH_PATTERN currently only matches double-quoted hashes which can
miss single-quoted webpack outputs; update ON_DEMAND_HASH_PATTERN to accept
either single or double quotes around the hash (similar to ON_DEMAND_FILE_REGEX)
so both quote styles are handled consistently—modify the pattern referenced as
ON_DEMAND_HASH_PATTERN to use a quote-capturing alternative like ["'] around the
hash value and ensure any backreference logic matches that capture.



class ClientTransaction:
Comment on lines +17 to 21

@sourcery-ai sourcery-ai Bot Mar 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: Hash pattern only matches double-quoted hashes while the ondemand.s key allows both quote styles.

Because ON_DEMAND_HASH_PATTERN only matches double-quoted hashes, re.search will fail whenever the hash is single-quoted (e.g. ,123:'abcd'), so on_demand_file_url is never constructed and you hit Couldn't get KEY_BYTE indices. To avoid this, update the hash pattern to accept both quote styles, e.g. r',{}:["\']([0-9a-f]+)["\']' so it stays consistent with ON_DEMAND_FILE_REGEX.

Expand All @@ -42,14 +42,17 @@ async def get_indices(self, home_page_response, session, headers):
key_byte_indices = []
response = self.validate_response(
home_page_response) or self.home_page_response
on_demand_file = ON_DEMAND_FILE_REGEX.search(str(response))
if on_demand_file:
on_demand_file_url = f"https://abs.twimg.com/responsive-web/client-web/ondemand.s.{on_demand_file.group(1)}a.js"
on_demand_file_response = await session.request(method="GET", url=on_demand_file_url, headers=headers)
key_byte_indices_match = INDICES_REGEX.finditer(
str(on_demand_file_response.text))
for item in key_byte_indices_match:
key_byte_indices.append(item.group(2))
on_demand_match = ON_DEMAND_FILE_REGEX.search(str(response))
if on_demand_match:
chunk_index = on_demand_match.group(1)
hash_match = re.search(
ON_DEMAND_HASH_PATTERN.format(chunk_index), str(response))
if hash_match:
file_hash = hash_match.group(1)
on_demand_file_url = f"https://abs.twimg.com/responsive-web/client-web/ondemand.s.{file_hash}a.js"
on_demand_file_response = await session.request(method="GET", url=on_demand_file_url, headers=headers)
for item in INDICES_REGEX.finditer(on_demand_file_response.text):
key_byte_indices.append(item.group(1))
if not key_byte_indices:
raise Exception("Couldn't get KEY_BYTE indices")
key_byte_indices = list(map(int, key_byte_indices))
Expand Down