xtrace is too easily redirected
magicant Thank you for reporting!
I will study the behavior of other shells to decide what yash should do in more complex cases, especially when assignments, redirections, and command invocation are combined in a single simple command.
magicant What I find the most interesting is that existing shells behave differently for sh -cx 'foo=$(echo bar 2>&1) printenv foo 2>/dev/null'.
Bash behaves the same as if 2>/dev/null is omitted. The fact that the xtrace for assignments is not affected by redirections implies that redirections are applied after assignments are expanded. This behavior looks non-conforming because POSIX explicitly requires that redirections should be performed before assignments are expanded.
++ echo bar + foo=bar + printenv foo bar
Dash looks more conforming. The redirection 2>/dev/null affects the subshell environment so the xtrace for echo bar 2>&1 is sent to /dev/null. However, the xtrace for the expanded assignment for=bar is still shown, which means xtrace is not being sent to /dev/null even after the redirection 2>/dev/null has been performed.
+ foo=bar printenv foo bar
Ksh does not show the xtrace for foo=$(echo bar 2>&1) at all. The xtrace for 2>/dev/null, however, is printed.
+ printenv foo + 2> /dev/null bar
Mksh traces everything. This seems the most useful, but obviously xtrace is not simply being sent to the standard error.
+ 2>/dev/null + 2>&1 + echo bar + foo=bar printenv foo bar
Zsh seems doing some weird trick since it prints foo= twice. Besides, all the expanded words are shown including assignments.
+ foo=+ echo bar + foo=bar printenv foo bar
To sum up, dash, mksh and zsh seem doing some trick so that xtrace is sent to the terminal even after the standard error is redirected to /dev/null. Next, I should study what trick they are doing.
magicant After the standard error (file descriptor 2) has been redirected, dash, mksh and zsh print xtrace to a file descriptor saved before the redirection. The file descriptor that was initially the standard error keeps receiving xtrace outputs. I don't know whether this behavior can be considered POSIXly correct.
Ksh always prints xtrace to the standard error, therefore xtrace can be possibly affected by redirections. Ksh prints xtrace as soon as a word is expanded, so redirections do not affect the xtrace for command words in a single simple command. Xtrace for assignments, however, is affected by the redirections in the same simple command. That means, ksh -cx 'foo=$(echo bar 2>&1) printenv foo' is okay, but ksh -cx 'foo=$(bar=$(echo baz) printenv bar 2>&1) printenv foo' does not work.
magicant)
magicant The way dash, mksh and zsh handles the redirected standard error is definitely not required by POSIX. I'm not willing to take this course unless POSIX is changed in favor of it.
Ksh print xtrace for assignments and command words separately. This helps the shell protecting xtrace from redirections, and I suppose this is still POSIXly correct. However, this style of printing can be confusing.
Bash also prints separately, but does not reflect the order of expansion. I don't think this is very useful.
Yash was printing separately, too, until version 2.11. In the current behavior, which has been introduced in #16562, expanded assignments and command words are printed in a single line of xtrace. To do so, the shell have to wait until all the assignments have been expanded, which is after redirections are performed.
So..., different shells have quite different behaviors, and none of them seems perfect to me. Should yash really change anything now?
What mksh does is save an internal copy of the standard error file descriptor on initialisation in main.c:
Fork shf_fdopen(1, SHF_WR, shl_stdout);
shf_fdopen(2, SHF_WR, shl_out);
shf_fdopen(2, SHF_WR, shl_xtrace);
Then it prints xtrace to shl_xtrace. So yes, xtrace *is* sent to FD 2, standard error, as specified by POSIX. The script itself cannot redirect xtrace output, but programs invoking the script can redirect it as part of standard error, as normal.
In fact, the mksh behaviour seems perfect to me, as it completely avoids the possibility of accidentally corrupting command substitutions. It also avoids the xtrace disappearing if a command or block suppresses error messages with 2>/dev/null.
POSIX might not require mksh's behaviour, but it doesn't prohibit it either. I also think mksh's behaviour is the only one that isn't inherently broken. Bash's is slightly less broken as the problem only occurs when redirecting blocks.
To me this also looks like yash's current behaviour could be non-conformant, as yash executes redirections that are part of the command before printing the xtrace.
Maybe I need to start another thread on the Austin group...
My apologies – I have to revise my earlier statement re mksh being POSIX compliant. Looks like, if you redirect stderr for a block, then a set -x within that block becomes local to that block. I really like that behaviour, because it means you can still redirect xtrace for certain commands within a script without causing any problems such as corrupted command substitutions – but it's also clearly not POSIX.
$ mksh -o posix -c '{ set -x; echo foo; } 2>/dev/null; echo end'
foo
end
magicant
$ yash -x -c 'echo hi 2>/dev/null' hi
The xtrace is not printed. All other shells print the xtrace.
The cause is that the redirection is not only parsed, but applied before the xtrace is printed. This behaviour is different from that of all other shells.
POSIX only says (for set -x) that “the shell shall write to standard error a trace for each command after it expands the command and before it executes it”. If the redirection is considered part of the command, you could read this to say that the shell should print the xtrace before applying the redirection. This is also what all other shells do.
Apart from that, I believe the current behaviour is undesirable. Suppose you want to catch both stdout and stderr in a variable using a command substitution like:
variable=$(some command 2>&1)
With xtrace active on yash, the trace becomes part of the value of the variable, which is clearly not desirable. On other shells, the trace is printed to stderr as normal.