Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/runtime/shell.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,8 @@ await $`
`;
```

Bun Shell removes the newlines at the end of the output, and any `\r` directly before them. A quoted substitution in a command (`"$(...)"`) keeps the rest of the output. An unquoted substitution also drops trailing spaces, tabs and `\r`. A variable assignment drops them too, also when the substitution is quoted.

<Note>

Because Bun internally uses the special [`raw`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#raw_strings) property on the input template literal, using the backtick syntax for command substitution doesn't work:
Expand Down
5 changes: 3 additions & 2 deletions src/runtime/shell/builtin/exit.rs
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ impl Exit {
}

fn parse_exit_code(s: &[u8]) -> Option<crate::shell::ExitCode> {
// %256 is bash semantics — keep wrapper fn.
bun_core::fmt::parse_decimal::<u64>(s).map(|n| (n % 256) as crate::shell::ExitCode)
// Blanks around the number and %256 are bash semantics — keep wrapper fn.
bun_core::fmt::parse_decimal::<u64>(bun_core::strings::trim(s, b" \t"))
.map(|n| (n % 256) as crate::shell::ExitCode)
}
24 changes: 20 additions & 4 deletions src/runtime/shell/states/Expansion.rs
Original file line number Diff line number Diff line change
Expand Up @@ -553,6 +553,16 @@ impl Expansion {
me.meta_offsets.clear();
}

/// POSIX 2.6.3: a command substitution ends before its trailing newlines.
fn strip_trailing_line_ends(stdout: &mut Vec<u8>) {
if stdout.last() == Some(&b'\n') {
// bash keeps a CR before a trailing LF. It goes here so CRLF output gives a clean word.
while matches!(stdout.last(), Some(b'\n' | b'\r')) {
stdout.pop();
}
}
}

/// Newlines→spaces, trim, then split on whitespace runs into separate
/// argv words.
fn post_subshell_expansion(me: &mut Expansion, mut stdout: Vec<u8>) {
Expand Down Expand Up @@ -654,11 +664,17 @@ impl Expansion {
me.out_exit_code = exit_code;
}
if no_split {
let mut hi = stdout.len();
while hi > 0 && matches!(stdout[hi - 1], b' ' | b'\n' | b'\r' | b'\t') {
hi -= 1;
if me.assign_ctx {
// `$X` is never field split, so a value still loses these trailing bytes.
let mut hi = stdout.len();
while hi > 0 && matches!(stdout[hi - 1], b' ' | b'\n' | b'\r' | b'\t') {
hi -= 1;
}
stdout.truncate(hi);
} else {
Self::strip_trailing_line_ends(&mut stdout);
}
Comment on lines +667 to 676

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Scripts that store a command's output in a variable and later compare it to, or reuse it beside, a fresh quoted "$(...)" of the same command now get a mismatch when the output ends in a blank, where the base branch and bash both match. In child_done at src/runtime/shell/states/Expansion.rs:667-676 the assign_ctx arm trims \t\r\n while the word arm keeps the blank, so X="$(docker ps)"; [[ "$(docker ps)" == "$X" ]] is false and F="$(cmd)"; echo hi > "$(cmd)"; cat "$F" misses the file. Fix: give both forms the same bytes while keeping ping $IP working, e.g. store the exact value and drop trailing blanks only when an unquoted $X is expanded (SimpleAtom::Var at Expansion.rs:492).

Why this was flagged

A shell script runs a command whose output ends in a blank before its newline, which hostname -I, docker ps (tabwriter-padded columns) and Windows table tools such as wmic always produce, and uses the output both through an assignment and through a quoted substitution in a word. Expansion::child_done takes the assign_ctx arm at src/runtime/shell/states/Expansion.rs:667-673 for X="$(cmd)" and truncates , \t, \r, \n, but for "$(cmd)" in a command word, redirect target or [[ ]] operand it takes the arm at Expansion.rs:674-676 and strip_trailing_line_ends at Expansion.rs:557-564 keeps the blank. On the base branch both arms ran the same trim loop (the deleted lines in this hunk), so the words were equal; bash keeps the blank in both, so they are equal there too. No existing safeguard applies: SimpleAtom::Var at Expansion.rs:492-502 copies the stored bytes verbatim and has no quoted flag to trim on.

Verification: In src/runtime/shell/states/Expansion.rs:666-677 (child_done), under no_split, the if me.assign_ctx arm (lines 667-673) truncates every trailing , \n, \r, \t, while the else arm (line 675) calls strip_trailing_line_ends (lines 557-564), which only pops \n/\r and only when the last byte is \n, so a trailing space/tab survives.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. The PR body lists this mismatch under Downsides, and the Notes show both cells. I did not change it, for these reasons.

The exact value in the variable was the first version of this PR. The earlier review thread found what it breaks: IP=$(hostname -I); ping $IP passes 172.30.0.2 with the space, where main and bash pass none.

The rule proposed here (exact value, and drop the blanks at an unquoted $X) must know that $X is unquoted. SimpleAtom::Var has no such flag: the lexer pushes the same Token::Var for $X and "$X" (src/shell_parser/parse.rs:2659). #44904 adds that flag (QuotedVar) and is a draft. This PR does not copy its parser change.

With the flag, that rule still has a case that matches neither main nor bash. I checked both on main and on bash 5.2:

  • Trim at every unquoted $X: X="a "; Y=b; echo $X$Y prints a b on main and in bash. It would print ab.
  • Trim only at the end of a word: X=$(echo 'a '); touch $X.txt creates a.txt on main, and a and .txt in bash. It would create a .txt.

So each partial rule keeps one such case: the exact value (an unquoted $X), this version (a variable next to a fresh quoted substitution), and the proposed rule (a padded variable followed by text in an unquoted word). Only field splitting of unquoted expansions removes all three. That is a larger change, and #44904, #45968 and #43861 each hold a part of it.

This PR keeps the version whose case is the rarest: it needs output that ends in a blank, and a script that uses that output through a variable and through a quoted substitution. If a maintainer prefers the exact value in variables, remove the assign_ctx arm here. Then the todo test passes, and the test above it pins what breaks.

me.current_out.extend_from_slice(&stdout[..hi]);
me.current_out.extend_from_slice(&stdout);
// A quoted substitution that printed only NUL bytes expands to one empty word.
if had_nul && me.cmd_subst_quoted {
me.has_quoted_empty = true;
Expand Down
49 changes: 49 additions & 0 deletions test/js/bun/shell/bunshell.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -410,6 +410,42 @@ describe("bunshell", () => {
.ensureTempDir()
.runAsTest("make sure its two separate arguments");
});

// POSIX 2.6.3: only the newlines at the end of the output are removed.
describe("quoted substitution keeps trailing whitespace", () => {
TestBuilder.command`echo "[$(echo 'a ')]"`.stdout("[a ]\n").runAsTest("space");
TestBuilder.command`echo "[$(echo -e 'a\t')]"`.stdout("[a\t]\n").runAsTest("tab");
TestBuilder.command`echo "[$(echo -n -e 'a\r')]"`.stdout("[a\r]\n").runAsTest("CR with no LF after it");
TestBuilder.command`echo "a$(echo -n ' ')b"`.stdout("a b\n").runAsTest("only spaces");
TestBuilder.command`echo "[$(echo a; echo ' ')]"`.stdout("[a\n ]\n").runAsTest("space on the last line");
TestBuilder.command`echo "$(echo 'a ')$(echo 'b ')c"`
.stdout("a b c\n")
.runAsTest("two substitutions in one word");
TestBuilder.command`echo "[$(echo "$(echo 'a ')b")]"`.stdout("[a b]\n").runAsTest("nested substitution");
TestBuilder.command`[[ -n "$(echo -n ' ')" ]] && echo yes`.stdout("yes\n").runAsTest("[[ -n ]] operand");

TestBuilder.command`${BUN} run ./code.ts a "$(echo -n ' ')" b`
.ensureTempDir()
.file("code.ts", `console.log(JSON.stringify(process.argv.slice(2)))`)
.stdout('["a"," ","b"]\n')
.runAsTest("output of one space is one argument");

// Windows removes a trailing space from a file name.
test.skipIf(isWindows)("redirect target", async () => {
await TestBuilder.command`echo hi > "$(echo 'out ')"`.ensureTempDir().fileEquals("out ", "hi\n").run();
});
});

// A CR directly before a trailing LF goes with it on every platform. bash keeps that CR.
describe("quoted substitution removes trailing line ends", () => {
TestBuilder.command`echo "[$(echo a; echo; echo)]"`.stdout("[a]\n").runAsTest("every LF");
TestBuilder.command`echo "[$(echo -n -e 'a\r\n\r\n')]"`.stdout("[a]\n").runAsTest("every CRLF");
TestBuilder.command`echo "[$(echo -n -e 'a\r\r\n')]"`.stdout("[a]\n").runAsTest("every CR before the LF");
TestBuilder.command`echo "[$(echo -n -e 'a \r\n')]"`.stdout("[a ]\n").runAsTest("CRLF after a space");
TestBuilder.command`echo "[$(echo -n -e 'a \r\r\n\r\r\n')]"`
.stdout("[a ]\n")
.runAsTest("two line ends after a space");
});
});

describe("unicode", () => {
Expand Down Expand Up @@ -1645,6 +1681,19 @@ describe("deno_task", () => {
TestBuilder.command`VAR=$(echo a && echo b) && echo "$VAR"`
.stdout("a\nb\n")
.runAsTest("assignment keeps command substitution newlines");

// An unquoted $VAR is not split into fields, so an assignment value drops the trailing whitespace.
TestBuilder.command`A=$(echo 'f '); B="$(echo 'g ')"; export C="$(echo 'h ')"; ${BUN} run ./code.ts $A $B $C`
.ensureTempDir()
.file("code.ts", `console.log(JSON.stringify(process.argv.slice(2)))`)
.stdout('["f","g","h"]\n')
.runAsTest("unquoted variable assigned a command substitution with a trailing space");

// bash keeps the space in the value and drops it when it splits an unquoted $VAR.
TestBuilder.command`VAR="$(echo 'a ')" && echo "[$VAR]"`
.stdout("[a ]\n")
.todo("an unquoted variable is not split into fields")
.runAsTest("assignment keeps the trailing space of a command substitution");
});

describe("env variables", async () => {
Expand Down
17 changes: 17 additions & 0 deletions test/js/bun/shell/commands/exit.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,23 @@ describe("exit", async () => {

TestBuilder.command`exit 62757836`.exitCode(204).runAsTest("exit code wraps u8");

// bash accepts a space or a tab around the number, and no other byte after it.
describe("blanks around the number", () => {
for (const arg of ["3 ", " 3", "3\t"]) {
TestBuilder.command`exit ${arg}`.exitCode(3).runAsTest(`accepts ${JSON.stringify(arg)}`);
}

for (const arg of ["3\r", "3\n", "3\f"]) {
TestBuilder.command`exit ${arg}`
.exitCode(1)
.stderr("exit: numeric argument required\n")
.runAsTest(`rejects ${JSON.stringify(arg)}`);
}

// A quoted substitution keeps its trailing space, so the status reaches exit with it.
TestBuilder.command`exit "$(echo '3 ')"`.exitCode(3).runAsTest("accepts a quoted command substitution");
});

// prettier-ignore
TestBuilder.command`exit abc`.exitCode(1).stderr("exit: numeric argument required\n").runAsTest("numeric argument required");
});
5 changes: 5 additions & 0 deletions test/js/bun/shell/exec.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ describe("bun exec", () => {
)
.runAsTest("write a lot of data");

TestBuilder.command`${BUN} exec ${`echo "[$(echo 'a ')]"`}`
.env(bunEnv)
.stdout("[a ]\n")
.runAsTest("quoted command substitution keeps a trailing space");

describe("--help works", () => {
// prettier-ignore
const programs = [
Expand Down
Loading