Bash Strict Mode: What set -euo pipefail Really Does

Almost every Bash style guide now tells you to start scripts with set -euo pipefail. It is good advice, and I use it in most of my own scripts, but it does not solve every problem. Each of the three options has rules about when it does and does not fire, and those rules surprise people. This post goes through each option, what it really does, and the cases where it surprises you. Every example below was run on Bash 4.4.

set -e: exit when a command fails

set -e (or set -o errexit) makes the shell exit as soon as a command returns a non-zero status. Without it, a failing cd or cp is ignored and the script continues with the next line. That is how a cleanup script ends up running rm in the wrong directory.

#!/bin/bash
set -e
echo "before"
false
echo "after"

$ bash t1.sh; echo "exit=$?"
before
exit=1

That is the easy part. The hard part is the list of places where a failure does not stop the script.

Ignored in if, while, && and ||

A command whose status is being tested cannot trigger set -e: the condition of if, while and until, anything after !, and every command in an && or || list except the last one. That is reasonable, since a test is supposed to be allowed to fail. The surprise is that the rule is inherited by everything called from that context, including whole functions.

#!/bin/bash
set -e
check() {
    false
    echo "check: still running after false"
}
if check; then
    echo "check succeeded"
fi
check && echo "and-list ok"
check
echo "not reached"

$ bash t2.sh; echo "exit=$?"
check: still running after false
check succeeded
check: still running after false
and-list ok
exit=1

Inside if check, the false in the function body is ignored completely. Because the function's last command (echo) succeeds, the if reports success. Only the third, bare call to check exits. So you cannot write a function that relies on set -e for its error handling and then also use it in a condition. If a function can fail, make it check its own commands and return 1 explicitly.

A trailing && can end the script

The reverse case is just as common. A [ test ] && action line on its own is harmless, because the failed test is not the last command in the list. But if that line is the last thing in a function, the function returns the test's status, and the caller sees a failure.

#!/bin/bash
set -e
f() {
    [ -f /nonexistent ] && echo "found"
}
f
echo "not reached"

$ bash t12.sh; echo "exit=$?"
exit=1

No error message, just a silent exit. Use a real if for anything that is not meant to be a test, or end the function with return 0.

(( i++ )) returns 1 when i was 0

The arithmetic command (( )) returns 1 when the expression evaluates to zero. A post-increment evaluates to the old value, so the first (( i++ )) on a counter that starts at 0 "fails".

#!/bin/bash
set -e
i=0
(( i++ ))
echo "i is now $i"

$ bash t3.sh; echo "exit=$?"
exit=1

The increment happened, but the script exited before it could say so. Use i=$(( i + 1 )), which is an assignment and always succeeds. Or use (( ++i )) when the counter starts at 0. It still fails if the new value is 0, for example when counting up from -1.

Command substitution and local

Two more problems. First, local var=$(cmd) returns the status of local, not of cmd, so the failure is lost. Declare the variable on one line and assign it on the next. Second, by default the shell inside $( ) does not inherit set -e, so a failing command in the middle of a substitution is ignored as long as the last one succeeds. Bash 4.4 added shopt -s inherit_errexit to fix that.

#!/bin/bash
set -e
out=$(false; echo "inside: kept going")
echo "out=$out"
shopt -s inherit_errexit
out=$(false; echo "inside: kept going")
echo "not reached"

$ bash t13.sh; echo "exit=$?"
out=inside: kept going
exit=1

Keep in mind that inherit_errexit does not exist in Bash 3.2 on macOS, and shopt -s with an unknown option name is an error there.

set -o pipefail: a pipeline fails if any part fails

Normally the exit status of a pipeline is the status of its last command. false | true succeeds, and so does mysqldump ... | gzip > backup.gz when the dump fails halfway, because gzip succeeded. With pipefail, the pipeline returns the status of the last command that failed, so set -e can see it.

#!/bin/bash
set -e
false | true
echo "pipe status ignored: $?"
set -o pipefail
false | true
echo "not reached"

$ bash t4.sh; echo "exit=$?"
pipe status ignored: 0
exit=1

head, SIGPIPE and exit status 141

The most common surprise is a pipeline that stops reading early. When head -n 1 exits after one line, the writer gets SIGPIPE the next time it writes, and a process killed by a signal has status 128 + signal number: 141 for SIGPIPE. With pipefail, that counts as a failure.

#!/bin/bash
set -eo pipefail
seq 1 1000000 | head -n 1
echo "exit statuses: ${PIPESTATUS[*]}"

$ bash t5.sh; echo "exit=$?"
1
exit=141

The worst part is that it depends on timing and size. seq 1 10 | head -n 1 succeeds, because seq writes all its output into the pipe buffer before head exits. So the bug appears only when the input grows, in production. When you stop reading early on purpose, make that explicit:

first=$(seq 1 1000000 | head -n 1) || [[ $? -eq 141 ]]

That accepts SIGPIPE and still exits on any other error. grep -q and awk '{ ...; exit }' on large input behave the same way, because they also stop reading early.

set -u: unset variables are errors

set -u (set -o nounset) turns a reference to an unset variable into an error that exits a non-interactive shell. It catches typos like $fiel for $file, and it stops rm -rf "$prefix/" from becoming rm -rf / when prefix was never set.

Optional values need a default, which is good practice anyway. ${1:-} means "the first argument, or empty". ${DEBUG:-off} gives a fallback.

#!/bin/bash
set -u
name=${1:-world}
echo "hello $name"
echo "debug is ${DEBUG:-off}"
echo "$1"

$ bash t7.sh; echo "exit=$?"
hello world
debug is off
t7.sh: line 6: $1: unbound variable
exit=1

Empty arrays before Bash 4.4

In Bash 4.4 and later, expanding an empty array with "${files[@]}" is fine under set -u. In older versions, including the Bash 3.2 that macOS ships as /bin/bash, the same line fails with "unbound variable". Indexing an element that does not exist is an error in every version:

#!/bin/bash
set -u
files=()
echo "count: ${#files[@]}"
for f in "${files[@]}"; do echo "$f"; done
echo "empty array expanded fine"
echo "${files[0]}"

$ bash t6.sh; echo "exit=$?"
count: 0
empty array expanded fine
t6.sh: line 7: files[0]: unbound variable
exit=1

If the script must also run on old Bash, check ${#files[@]} before expanding, or use the ${files[@]+"${files[@]}"} idiom, which expands to nothing when the array is empty. It is ugly, but it works everywhere.

What about IFS=$'\n\t'?

The "unofficial strict mode" article that popularised set -euo pipefail also sets IFS=$'\n\t', so unquoted variables split on newlines and tabs but not spaces. It does stop for f in $(ls) breaking on file names with spaces. I do not recommend it as a default, for two reasons.

First, it hides the real bug, which is the missing quotes. The same script breaks again as soon as someone removes the IFS line or copies a function elsewhere. Second, it breaks code that splits on spaces on purpose, and it changes how "${array[*]}" joins elements, because the join uses the first character of IFS:

#!/bin/bash
IFS=$'\n\t'
files="a.txt b.txt"
for f in $files; do printf '[%s]\n' "$f"; done
args=(-l -a)
printf '<%s>\n' "${args[*]}"

$ bash t8.sh
[a.txt b.txt]
<-l
-a>

Quote your expansions, use arrays for lists, and set IFS locally when you need it (IFS=, read -r a b), as described in explode() and implode().

A template that respects the rules

This is the structure I use. Optional arguments get defaults, arrays are checked before use, nothing important is inside a condition, and the exit status is set explicitly.

#!/bin/bash
set -euo pipefail
main() {
    local dir=${1:-}
    if [[ -z $dir ]]; then
        printf 'usage: %s DIR\n' "${0##*/}" >&2
        return 2
    fi
    local -a files=()
    local f
    for f in "$dir"/*.log; do
        [[ -f $f ]] || continue
        files+=("$f")
    done
    printf 'found %d log files\n' "${#files[@]}"
    if (( ${#files[@]} > 0 )); then
        printf '  %s\n' "${files[@]##*/}"
    fi
}
main "$@"

$ mkdir demo && touch demo/app.log 'demo/error log.log' demo/notes.txt
$ ./count-logs demo
found 2 log files
  app.log
  error log.log
$ ./count-logs; echo $?
usage: count-logs DIR
2

Strict mode is extra protection, not error handling. It catches the failures you forgot about. The ones you expect still need an explicit check and a clear message. The set builtin section of the Bash manual lists the exact rules for -e, and they are worth reading once.

Related

• trap and cleanup: removing temp files on exit
• Parsing script options with getopts
• Reading a file line by line
• is_num(): check that input is a number