Bash trap: Clean Up Temp Files and Locks on Exit

Every script that creates a temporary file has the same problem: if it fails halfway, or someone presses Ctrl+C, the file stays behind. On a busy server that turns into thousands of tmp.XXXXXXXXXX files in /tmp, or a stale lock that stops the next cron run. The fix is the trap builtin, which runs a command when the shell receives a signal or exits. This post covers the patterns I use, all tested on Bash 4.4.

trap EXIT and mktemp

The basic pattern is short. Create the temporary file or directory with mktemp, which picks an unused name safely, and register a cleanup function on the EXIT pseudo-signal. Bash runs the EXIT trap whenever the shell exits: at the end of the script, on an explicit exit, and when set -e aborts it.

#!/bin/bash
set -euo pipefail
tmpdir=$(mktemp -d)
cleanup() {
    rm -rf -- "$tmpdir"
}
trap cleanup EXIT
date > "$tmpdir/stamp"
ls "$tmpdir"
false
echo "not reached"

$ bash t1.sh; echo "exit=$?"
stamp
exit=1

The false killed the script through set -e, and the directory was gone afterwards. Two habits matter here. I use a temporary directory rather than a file, because a script usually ends up needing more than one scratch file, and one rm -rf cleans them all. And I register the trap straight after mktemp, before anything that can fail.

Use single quotes when you write the trap command inline: trap 'rm -f -- "$tmpfile"' EXIT. With double quotes, the variable is expanded when trap runs, not when the trap fires, and a path containing a space is then split into two words. A function avoids the quoting question entirely.

INT and TERM

In Bash, the EXIT trap also runs when the script is killed by SIGINT (Ctrl+C), SIGTERM (a plain kill) or SIGHUP. I checked all three on Bash 4.4 and the temp file was removed each time. That is a Bash feature, though: dash and some other sh implementations do not run the EXIT trap on a signal. Nothing runs on SIGKILL (kill -9). No trap can catch it.

The surprise is what happens when you trap a signal yourself. A trap replaces the default action, and the default action for SIGINT was "exit". If your handler does not exit, the script continues:

#!/bin/bash
trap 'echo "caught INT"' INT
sleep 30
echo "still running after ^C"

caught INT
still running after ^C

Sometimes that is what you want, but usually it is not. If you trap INT or TERM, end the handler with an explicit exit and the conventional status, 128 plus the signal number: 130 for SIGINT, 143 for SIGTERM. The EXIT trap still runs after that exit, so the cleanup stays in one place:

trap cleanup EXIT
trap 'echo "interrupted" >&2; exit 130' INT
trap 'echo "terminated" >&2; exit 143' TERM

Pressing Ctrl+C in the middle of a sleep 30 then printed interrupted, the script exited with status 130, and the temp file was removed.

Preserving the exit status

When the EXIT trap starts, $? holds the status the script is exiting with. The first command inside the handler overwrites it, so if you want to log it or decide anything based on it, save it on the first line. Then end the handler with exit "$status", so the cleanup can never change the result your caller sees.

cleanup() {
    local status=$?
    rm -f -- "$tmpfile"
    echo "cleanup: exiting with status $status" >&2
    exit "$status"
}

In my tests, Bash 4.4 already kept the original status when the last command in the handler failed, and a bare exit in the handler kept it too. The explicit exit "$status" costs nothing and documents the intent. What does change the status is an exit with a different number inside the handler, so never write exit 0 in a cleanup function.

ERR and set -E

The ERR trap runs whenever a command fails, under the same rules as set -e: not for commands in an if condition, not for the left side of && or ||. It is the place to print which command failed, which set -e on its own never tells you. $LINENO and $BASH_COMMAND inside the trap give the line and the command text.

There is one problem: by default the ERR trap is not inherited by shell functions, command substitutions or subshells. With plain set -e, a failure inside a function aborts the script without the trap ever firing:

#!/bin/bash
set -e
on_error() {
    local status=$?
    printf 'error: line %s: %s (status %s)\n' \
        "$1" "$BASH_COMMAND" "$status" >&2
}
trap 'on_error $LINENO' ERR
load_config() {
    grep -q '^port=' /nonexistent/app.conf
}
load_config
echo "not reached"

$ bash t6.sh; echo "exit=$?"
grep: /nonexistent/app.conf: No such file or directory
exit=2
$ sed 's/^set -e$/set -eE/' t6.sh > t7.sh
$ bash t7.sh; echo "exit=$?"
grep: /nonexistent/app.conf: No such file or directory
error: line 11: grep -q '^port=' /nonexistent/app.conf (status 2)
exit=2

set -E (set -o errtrace) makes functions inherit the trap. If you use an ERR trap at all, use set -E with it. The trap is quoted with single quotes so that $LINENO is expanded when it fires, giving the line of the failing command.

Resetting and ignoring traps

trap -p lists the traps that are set, and trap - SIGNAL resets one to its default action:

$ bash t9.sh
trap -- 'echo "cleanup"' EXIT
trap -- 'echo "got INT"' SIGINT
after trap - INT:
trap -- 'echo "cleanup"' EXIT
cleanup

An empty string is different from a dash. trap '' INT makes the shell, and the commands it starts, ignore SIGINT completely. I use it to protect a short critical section, such as swapping two files into place, and then restore normal behaviour with trap - INT. When I tested it, a Ctrl+C during the protected sleep 2 did nothing, and one sent after trap - INT stopped the script. Keep those sections short. A script that cannot be interrupted is unpleasant to use.

Lock files

Cron jobs that overlap are a common source of trouble. The simplest portable lock is a directory, because mkdir either creates it or fails, in one step, so two copies of the script cannot both succeed. Checking for a file with [ -e ] and then creating it has a race between the two commands.

#!/bin/bash
set -euo pipefail
lockdir=/var/lock/nightly.lock
if ! mkdir -- "$lockdir" 2>/dev/null; then
    printf '%s: already running\n' "${0##*/}" >&2
    exit 1
fi
trap 'rmdir -- "$lockdir"' EXIT
echo "got the lock, working..."
sleep 3
echo "done"

$ ./nightly.sh & sleep 1; ./nightly.sh; echo "second exit=$?"; wait
got the lock, working...
nightly.sh: already running
second exit=1
done

The trap is set only after mkdir succeeds. Otherwise the second copy would remove the first copy's lock when it exits. The weakness of this approach is SIGKILL or a power cut, which leave the directory behind and need a manual rmdir.

On Linux, flock from util-linux avoids that, because the kernel releases the lock when the process dies, whatever the cause. It is not installed on macOS by default.

exec 9>/var/lock/nightly.lock
if ! flock -n 9; then
    printf '%s: already running\n' "${0##*/}" >&2
    exit 1
fi

No trap is needed for the lock itself: file descriptor 9 stays open until the script exits, and the lock goes with it.

Putting it together

This script sorts a file in a temporary directory and moves the result into place, with cleanup, an error report and signal handling. It is the template I start from when a script writes anything.

#!/bin/bash
# rotate-report -- build a report in a temp dir, publish it atomically
set -eEuo pipefail
workdir=
cleanup() {
    local status=$?
    if [[ -n $workdir ]]; then
        rm -rf -- "$workdir"
    fi
    exit "$status"
}
on_error() {
    printf '%s: line %s: %s failed (status %s)\n' \
        "${0##*/}" "$1" "$BASH_COMMAND" "$2" >&2
}
trap cleanup EXIT
trap 'on_error "$LINENO" "$?"' ERR
trap 'exit 130' INT
trap 'exit 143' TERM
workdir=$(mktemp -d)
src=${1:?usage: rotate-report FILE}
sort -- "$src" > "$workdir/sorted"
uniq -c "$workdir/sorted" | sort -rn > "$workdir/report"
mv -- "$workdir/report" "./report.txt"
echo "report written"

$ printf 'beta\nalpha\nbeta\ngamma\nbeta\nalpha\n' > hits.txt
$ ./rotate-report hits.txt
report written
$ cat report.txt
      3 beta
      2 alpha
      1 gamma
$ ./rotate-report missing.txt; echo "exit=$?"
sort: cannot read: missing.txt: No such file or directory
rotate-report: line 26: sort -- "$src" > "$workdir/sorted" failed (status 2)
exit=2

In both runs the temporary directory was removed. workdir is set to an empty value before the trap is registered. So the cleanup function is safe even if the script dies before mktemp runs, and it never calls rm -rf on an empty path. The trap entry in the Bash manual has the full list of pseudo-signals, including DEBUG and RETURN, which I rarely need.

Related

• Bash strict mode: what set -euo pipefail really does
• Parsing script options with getopts
• __pgrepf(): find a running process
• emptydir(): check whether a directory is empty