Skip to content

fix: handle NUL and vertical tab characters in source - #425

Draft
toddr-bot wants to merge 2 commits into
masterfrom
koan.toddr.bot/fix-issue-69
Draft

fix: handle NUL and vertical tab characters in source#425
toddr-bot wants to merge 2 commits into
masterfrom
koan.toddr.bot/fix-issue-69

Conversation

@toddr-bot

@toddr-bot toddr-bot commented Apr 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

PPI threw "Encountered unexpected character '0'" when parsing Perl source containing NUL (ASCII 0) or vertical tab (ASCII 11) characters, even though perl itself runs such code fine. This adds both characters to the tokenizer's whitespace character map so they are parsed as PPI::Token::Whitespace, preserving round-trip fidelity.

Fixes #69

Changes

  • Add NUL (ordinal 0) and vertical tab (ordinal 11) to %CLASSMAP in PPI::Token::Whitespace as whitespace characters
  • Add \0 and \v escaping in PPI::Dumper output for readability
  • Add regression tests for NUL and VT parsing, round-trip, and whitespace classification

Test plan

  • Tests verify parsing source with NUL between statements, NUL at start of source, and vertical tab between statements
  • Each case checks: no parse errors, round-trip serialization, whitespace token classification
  • Full test suite (53,095 tests across 69 files) passes with no regressions

Generated by Kōan /fix


Quality Report

Changes: 3 files changed, 34 insertions(+), 1 deletion(-)

Code scan: clean

Tests: skipped

Branch hygiene: clean

Generated by Kōan post-mission quality pipeline

toddr-bot and others added 2 commits April 27, 2026 06:40
PPI throws "Encountered unexpected character '0'" when parsing Perl
source containing NUL (ASCII 0) or vertical tab (ASCII 11) characters,
even though Perl itself handles them fine. These tests verify that PPI
can parse such source code, preserve the characters as whitespace tokens,
and round-trip them correctly.

Tests are marked $TODO since the fix is not yet implemented.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Perl tolerates NUL (ASCII 0) and vertical tab (ASCII 11) characters in
source code, but PPI threw "Encountered unexpected character" for both.
Add them to the tokenizer's whitespace character map so they are parsed
as PPI::Token::Whitespace, preserving round-trip fidelity.

Also escape \0 and \v in Dumper output for readability.

Fixes #69

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RT 36384: PPI won't parse source containing NUL

1 participant