Repository navigation
Conversation
The comment scanner consumed the newline but never emitted the Newline token. The next line's words were appended to the current command. A backslash inside a comment is literal and does not continue the line.
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 3 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Two inputs work on main only because the comment swallows its newline. With this change alone they fail: # 1. a comment after && or |
echo a && # note
echo b
# main: prints a, b. this PR: expected a command or assignment
# 2. a comment before a closing backtick
echo `echo a # note
`
# main: prints a. this PR: expected a command or assignment but got: "CmdSubstEnd"
#34901 includes this branch, because its statement rule needs the |
|
One more note, from a review of #34901 (which includes this branch): the |
This change is the word loop again: words after a redirection stay in the command, an fd duplication takes no target, and a second redirection is an error. The rule that a statement holds one command turns every place where the lexer emits no line end into a parse error. A comment did that, and so does a CRLF line that ends in `)` or `]]`: main runs such a script and prints one `command not found` line, the rule stopped the whole script. The rule belongs on top of the lexer change that ends comment, CRLF and tab lines with a Newline token (#29670). So this commit removes from this branch: the rule in parse_stmt, the comment change merged from #41468, the newline after `&&`, `||` and `|`, the shared closer of command substitutions, and their tests. Tests: an interpolated array after `>` only asserts that its tail does not run, so the case holds when #45780 makes that input an ambiguous redirect. A quoted word that touches an fd duplication has a case.
|
Correction to my two notes above: #34901 no longer includes this branch. It is only the word loop again (bd91e39), so nothing there depends on this PR now. The two inputs this change breaks alone still stand ( |
…l-comment-newline
… operator The comment scanner stops at the newline that ends the comment. The newline arm of the lexer stays the only producer of the Newline token and of the word break. Inside backquotes a comment also ends at the closing backquote, and a backslash still quotes the next character. Three places parsed a comment at the end of the line only because the comment consumed its newline. They now accept a line break: - after `&&`, `||` and `|` (POSIX `linebreak`). A `then`, `elif`, `else` or `fi` on the next line stays a parse error. - after `[[` and before `]]`. - before a closing backquote, as before the `)` of `$( )`.
…l-comment-newline
…he next command Three results of the change to comments, found in review: - `then # note` + newline + `else` ran `else` as a command. The body loop of `parse_if_body` now skips line breaks before it tests for the word that closes the body, so an empty body over several lines parses like `then else` on one line. - A backquote inside a backquoted comment closed the substitution, and the text after it started the first word of the next substitution. `eat_subshell` now flushes the pending word for every kind of subshell (the change of #45295). - The two closers of a command substitution share one delimiter rule.
With the pending word flushed before a backquote, `` a=`` `` has an empty body. The closer pushed a delimiter after `CmdSubstBegin`, and the parser took it for a command. `$()` and an empty pair of backquotes now parse, as in bash.
…l-comment-newline
|
Updated 4:11 AM PT - Oct 21st, 2026
✅ @robobun, your commit 67f72f31cadb71b33740e4878b027d1bd3bbdfba passed in 🧪 To try this PR locally: bunx bun-pr 41468That installs a local version of the PR into your bun-41468 --bun |
robobun
changed the title
&&, ||, | and inside [[ ]]
|
Status of this PR Reproduced on bun 1.4.3 and on a release build of main (9b4284f). Save this as if false; then
if true; then
echo both
fi # inner
else
echo none
fimain prints nothing and exits 0. bash and dash print The rework is pushed as 67f72f3. It covers the two inputs from my note of 2026-10-03 ( PR: #41468 |
There was a problem hiding this comment.
I re-reviewed the latest revision and found no bugs; both points from my earlier review are now handled (operators accept a following linebreak via skip_linebreak_after_operator, and the backslash-newline tests now feed the lexer a real \ + LF). Because this reshapes lexer token boundaries and parser statement/clause termination for every Bun shell script, a human look would still be worthwhile.
What was reviewed:
eat_commentcursor arithmetic in both the ASCII and Unicode lexers:cursor_pos()is the byte index of the next unread char in both, and the end-of-input case lands on the existing past-the-end sentinel, so no out-of-range cursor.backquoted_comment_len: theat + 2skip after a backslash is safe becauseindex_of_any_posreturns None oncestart >= len; the "backtick is comment text inside$( )under backquotes" rule knowingly departs from bash but matches how this lexer already nests backticks there, and the test states that.- The
break_wordmove intoeat_subshellkeeps the prior ordering relative to the depth check for$(and(, and the new Backtick-open call fixesx\...`gluing;delimit_before_cmd_subst_endnow suppressing Delimit for empty$()`/ parse_if_bodyhoistingskip_newlinesbefore the loop andparse_cond_exprskipping newlines only after[[and before]](not between operands) both have tests pinning the exact new behavior and error messages.
Extended reasoning...
The diff is confined to src/shell_parser/parse.rs (+94/-25) plus four test files: the lexer now stops a # comment before its newline (and, inside backquotes, at the closing backtick), moves the pre-subshell break_word into eat_subshell, and dedups the delimiter-before-close logic; the parser gains POSIX linebreak handling after &&/||/|, newline skipping inside [[ ]], and an empty-clause-body fix. No security-sensitive surface such as auth or crypto is touched, though the original bug could make a trailing comment feed the next line to rm -rf, so correctness of the lexer here matters for users. Deferring rather than approving because the change alters token boundaries and statement termination in a stateful hand-written lexer/parser used by every Bun shell script, and the backquote-comment rules are a deliberate divergence from bash that a maintainer should sign off on; no third-party objections are outstanding and both prior inline findings from this bot are addressed by the new commits.
Problem
#comment at the end of a line swallows the newline, so the next line joins the command.rm -rf build # clean+ newline +cp -R src builddeletessrc, andfi # inner+ newline +elseskips theelsebranch. Both exit 0.Lexer::eat_comment(src/shell_parser/parse.rs:3482) consumes the newline and pushes noToken::Newline.Fix
eat_commentstops before the newline or the closing backquote. The\narm oflexstays the only producer ofToken::Newline.&&,||,|, inside[[ ]], before a closing backquote, in an emptythenbody.lex.test.ts,parse.test.ts,bunshell.test.ts,run-shell.test.tsfail on main. 15 other shell test files pass.Background
Token::NewlineandToken::Semicolonend a statement.linebreakallows a newline after&&,||and|. The parser had no such rule.Token::Newlinefromeat_comment(this PR's first version, Shell: handle CRLF, tab, comment, and backslash-newline line endings in the lexer #29670). That is a second producer, and it skips the word break.Downsides
NAME=value # noteis no longer a prefix of the next command. For an exported name Bun then passes the old value (Bun Shell: prefix assignment of an existing env var duplicates the environ entry (child sees the old value) #32202)..textgrows 1,024 bytes. Instruction counts not measured:perfandvalgrindare not installed.Notes
How it was found, and reach. No GitHub issue reports this. A fuzz comparison of the Bun shell with bash and dash found it, and the request was to complete this PR in place. Every entry that parses shell text has it:
bun file.sh,bun exec, aBun.$template with more than one line, and a package.json script that the Bun shell runs (--shell=bun, the default on Windows).Before and after. Release builds of main (9b4284f) and of this PR.
⏎is a newline. bash 5.2.37 and dash 0.5.12 give the last column (dash has no[[ ]]).if false; then⏎if true; then⏎echo both⏎fi # inner⏎else⏎echo none⏎finoneboth,none,bun: command not found: elsebothelif true; then⏎echo elifbeforeelsenoneelifif true; then⏎echo a # c⏎else⏎echo b⏎fia else,baif true; then⏎echo a # c⏎fiExpected "else", "elif", or "fi" but got: Eofaif true # c⏎then echo a; fiExpected "then" but got: Eofaecho one # note⏎echo twoone echo twoone,twoFOO=bar # c⏎echo "[$FOO]"[][bar]echo one # note \⏎echo twooneone,twoecho `echo 1 # c`⏎echo `echo 2 # c`1 echoecho 21,2echo `echo a # c`Unclosed command substitutionaecho a &&⏎echo bexpected a command or assignmenta,b[[ a == a⏎]] && echo yesExpected "]]" but got: `\n`yesecho `echo a⏎`expected a command or assignment but got: "CmdSubstEnd"aecho x`echo y`bun: command not found: xechoxyecho $()expected a command or assignment but got: "CmdSubstEnd"What the diff carries. The comment scanner is the fix. Each other part keeps a form that works on main only because the comment took its newline, or its closing backquote, with it.
eat_commentfinds the newline withstrings::index_of_char_usizeand moves the cursor to it. Outside backquotes a backslash is comment text. Inside backquotes (backquoted_comment_len, and the inheritedin_backquoteflag) a backslash still quotes the next character, as on main and in bash, and the closing backquote ends the comment.skip_linebreak_after_operator: line breaks after&&,||,|. Athen,elif,elseorfion the next line stays the parse error of main. Without this guard,cmd &&⏎elsewould runelseas a command.parse_cond_expr: line breaks after[[and before]]. A line break around the operator stays an error, as in bash.delimit_before_cmd_subst_end: the)of$( )and the closing backquote share one rule. No delimiter follows a newline, a;or an empty body.parse_if_body: line breaks come before the test for the word that closes the body.then # note⏎elsekeeps an emptythenbody, likethen elseon one line.eat_subshellflushes the pending word for every kind of subshell. This is the same text as the source change of shell: flush the pending word in eat_subshell, so text before a backtick substitution stays in its word #45295. The comment rule needs it: a comment that ends at a backquote leaves its other words in front of the next backquote, and on main those words become the first word of that substitution. Withecho `echo a # remove it with `rm` -rf build⏎`,rm -rf buildwould run. bash and this PR printarmand report-rf: command not found.Scripts that ran on main and now fail to parse. bash rejects the first six.
echo a # c⏎&& echo ba,bexpected a command or assignment but got: "DoubleAmpersand"echo a # c⏎| cataexpected a command or assignment but got: "Pipe"echo a > # c⏎fileatofileRedirection with no file[[ a # c⏎== a ]]Expected a conditional expression operator, but got: `\n`[[ a == # c⏎a ]]Expected a word, but got: `\n`[[ -n # c⏎a ]]Expected a word, but got: `\n`echo a # c⏎> out.log echo bb, writesatoout.logexpected a command or assignment but got: "Redirect"The last script is valid. bash prints
aand writesb. A line that starts with a redirection does not parse on main either (related to #9455). The comment only hid that.Other changes at run time. Each is the result of bash and dash, except where the text says so.
cmd \⏎# note⏎argran ascmd arg. The comment line now ends the command andargruns as a command.NAME=value # note⏎cmdwas a prefix assignment forcmd. It is now a plain assignment. For a name that is not in the environment,cmddoes not see it, as in bash. For an exported name (PATH,NODE_ENV) bash passes the new value and Bun passes the old one. That gap is Bun Shell: prefix assignment of an existing env var duplicates the environ entry (child sees the old value) #32202, and shell: keep variables in one table with an export attribute, and prefix assignments on their command #45371 is the draft for it. The same lines without the comment behave this way on main.then⏎elsewith an empty body ranelseas a command. It is now an empty body, with or without comments between the two words. bash rejects an empty body.a &&⏎for i in 1 2; do echo $i; donewas a parse error. It now runs as its one-line spelling does: the Bun shell has nofor, sofor,doanddoneare commands that are not found.$()and an empty pair of backquotes parse.Not changed here.
bunshell.test.tshas atodotest with the result of bash for each of the first three.#directly after;,|,&&,(,$(or an opening backquote runs a command named#. A#after a tab does not start a comment, because a tab does not end a word (shell: break words on a tab in the lexer #43861, Shell: handle CRLF, tab, comment, and backslash-newline line endings in the lexer #29670).echo a\⏎# notestarts a comment, and bash printsa# note. One script of this kind becomes reachable:echo `# note ; cmd⏎echo a⏎`was a parse error on main and now runscmd, as its one-line spelling does on main.fi elseon one line, and a word after a compound command (shell: end a statement at the then/elif/else/fi that follows a compound command #45904). A reserved word after&&on one line (shell: reject!, brace groups, here-documents and stray then/elif/else/fi at parse time #43922).echo a "$(true)" bprintsa b, on main too. bash prints two spaces.Other PRs.
eat_commentchange and itsskip_newlines()after&&,||and|. The two do not merge: its#arm expectseat_commentto return abooland pushes a secondToken::Newline. Shell: handle CRLF, tab, comment, and backslash-newline line endings in the lexer #29670 keeps CRLF (for Bun Shell on Windows fails to handle CRLF line endings in .sh files, causing command resolution failure #29669), the tab, and the rule for where a comment starts.break_word, give$((...))its own error #38479: this PR takes over the case of a newline or a;before a closing backquote, and the empty body. shell: lex both ends of a command substitution throughbreak_word, give$((...))its own error #38479 needs a rebase on the shared helper.fi # inner⏎else. The tests of the one that lands second no longer fail before.!, brace groups, here-documents and stray then/elif/else/fi at parse time #43922 rejects a straythen,elif,elseorfi. With it, the guard inskip_linebreak_after_operatorcan go.Measurements. Release builds of main (9b4284f) and of this PR.
.text81,904,485 to 81,905,509 bytes (+1,024,llvm-size -A). File 107,493,088 to 107,497,184 bytes (+4,096, one page).bun_shell_parsersymbols 80,774 to 81,991 bytes (nm -S):Lexer::lex+245 (ASCII) and +421 (UTF-8),parse_compound_cmd+181,skip_linebreak_after_operator171 (new),delimit_before_cmd_subst_end98 twice (new),Parser::match+37,parse_stmt+35,parse_pipeline+4,parse_if_body-73.size_ofthe lexer: 200 to 200 bytes. The newboolfits in padding.Token::Newlinepush sites: 1 to 1.skip_newlines()call sites: 4 to 8.ifwith nothen.shellInternals.lexor.parsecall. Each process takes the minimum of 60 calls, and each cell is the minimum over 15 processes. main and PR runs are interleaved, and main is sampled twice to show the spread.echo aa && b | c || decho aifwith 2000 statements[[ -n a ]]echo `echo a`echo $(echo a)A second run of the same table moved single cells by up to 5% in both directions, on both builds. I found no stable slowdown. Instruction, branch and allocation counts would settle it, and
perf,valgrindandstraceare not installed here. By a read of the diff, a script with no comment pays: one call that tests one token per&&,||or|, three token tests per[[ ]], oneboolper sublexer, and onematchper closer. The diff adds no allocation.Tests.
lex.test.ts: a table of 36 scripts, each with a comment. The tokens must equal those of the same script with the comment text removed. Each row runs a second time with a non-ASCII character in the comment, which selects the other lexer. More tables cover the closers, the flush, and the empty substitution.parse.test.ts: 60 pairs (5 things in front of the comment, 12 next lines) whose tree must equal the tree without the comment, the line breaks after each operator,[[ ]], empty bodies, and each parse error with its exact message.bunshell.test.ts: 28 tests throughBun.$. 25 print what bash prints. 3 have an empty body, which bash rejects. 6 more aretodotests for the rules that this PR does not change.run-shell.test.ts: a script file, a package.json script andbun exec.lex.test.ts, 98 inparse.test.ts, 25 inbunshell.test.ts, 7 inrun-shell.test.ts). The new tests that pass on main pin what must stay.lex,parse,bunshell,exec,run-shell, and 14 more files oftest/js/bun/shell/pass.leak.test.tsandshell-load.test.tsreach their time limits on this host in a debug build. They run no comment and no line break after an operator.Considered and not taken.
#, and words such as--color=#fffand$(x)#fragdepend on the result. It is a second change with its own tests, and Shell: handle CRLF, tab, comment, and backslash-newline line endings in the lexer #29670 has the code for it.