Read a File Line by Line in Bash (the Right Way)

Reading a file one line at a time is something almost every shell script does: a list of hosts, a config file, a CSV export. The correct loop is short, but every part of it is there for a reason, and the versions that look simpler lose data silently. This post goes through the standard idiom, the four traps I see most often, and a small helper that puts it all together.

All the examples were run on Bash 4.4 with these test files. notes.txt has leading spaces, a backslash, and no newline at the end of the last line, which is common for files written by editors and other programs:

terminal

$ printf '  indented line\nback\\slash\nlast line no newline' > notes.txt
$ printf 'one two\n*\nthree\n' > words.txt

The idiom

bash

while IFS= read -r line || [[ -n $line ]]; do
    printf '[%s]\n' "$line"
done < notes.txt

output

[  indented line]
[back\slash]
[last line no newline]

Each piece does one job:

IFS=: an empty IFS, set only for read, so leading and trailing whitespace is kept.
-r: raw mode, so a backslash is an ordinary character and not an escape or a line continuation.
|| [[ -n $line ]]: keeps the loop going for a final line without a newline.
< notes.txt: the redirect is on the whole loop, so the file is opened once.

Here is what you get without the first two:

bash

while read line; do printf '[%s]\n' "$line"; done < notes.txt

output

[indented line]
[backslash]

The indentation is gone, the backslash is gone, and so is the last line. Three bugs in a one-line loop, and no error message.

The last line without a newline

read returns non-zero when it reaches end of file before a newline, but it still stores whatever it read in line. The plain while read loop sees the failure and stops, so the data in line is never processed. Adding || [[ -n $line ]] says: continue if read failed but did get some text. On the next pass read fails with an empty line, and the loop ends.

The IFS= read -r loop without that test printed only the first two lines of notes.txt. If you control the files, end them with a newline (POSIX says a text file should). If you do not, keep the extra test.

Why for line in $(cat file) is wrong

This is the loop people write first:

bash

for line in $(cat words.txt); do printf '[%s]\n' "$line"; done

output

[one]
[two]
[a.conf]
[b.conf]
[crlf.txt]
[notes.txt]
[passwd.txt]
[run.sh]
[words.txt]
[three]

The file has three lines (the directory also held my other test files). The unquoted $(cat ...) is split into words on spaces, tabs and newlines, so one two becomes two items. Then each word goes through filename expansion, so the line containing * turned into every file in the directory. Blank lines vanish too. Quoting it as "$(cat words.txt)" gives one item, the whole file. There is no way to make for iterate over lines correctly without changing IFS and turning globbing off, at which point the while loop is simpler. It also reads the entire file into memory first.

Pipe vs redirect: where did my variable go?

bash

count=0
cat words.txt | while IFS= read -r line; do count=$((count+1)); done
echo "pipe: count=$count"
count=0
while IFS= read -r line; do count=$((count+1)); done < words.txt
echo "redirect: count=$count"
count=0
while IFS= read -r line; do
    count=$((count+1))
done < <(grep -v '^\*' words.txt)
echo "process substitution: count=$count"

output

pipe: count=0
redirect: count=3
process substitution: count=2

Each part of a pipeline runs in its own subshell. The loop after the | counted to three, but in a child process, and the parent's count was never touched. With a redirect, the loop runs in the current shell and the variable survives. When the input comes from a command rather than a file, use process substitution, < <(command), as in the third loop. The space between the two < is required.

Bash 4.2 added shopt -s lastpipe, which runs the last part of a pipeline in the current shell. In a script it fixes the pipe version (I got count=3 with it). But it only takes effect when job control is off, so it does nothing at an interactive prompt. I prefer the redirect, because it works everywhere and does not depend on a setting.

Commands that read your input

The loop's stdin is the file. Any command inside the loop that reads stdin, such as ssh, mysql or ffmpeg, will read the rest of the file instead. Here wc -l takes the place of ssh:

bash

while IFS= read -r host; do
    echo "checking $host"
    wc -l > /dev/null
done < hosts.txt

output

checking web01

The file has three hosts. The loop ran once. The fix is to read the file on a different file descriptor so stdin is left alone:

bash

while IFS= read -r host <&3; do
    echo "checking $host"
    wc -l > /dev/null
done 3< hosts.txt

output

checking web01
checking web02
checking web03

For ssh specifically, ssh -n also works, but descriptor 3 fixes it for every command.

A reusable helper

Here is all of that in one function:

bash

# each_line() -- run a command once per line of a file
# Usage: each_line file command [args...]
each_line() {
    [ $# -ge 2 ] || return 1
    local file="$1" line
    shift
    [ -r "$file" ] || return 1
    while IFS= read -r line <&3 || [[ -n $line ]]; do
        "$@" "$line"
    done 3< "$file"
}

terminal

$ show() { printf '[%s]\n' "$1"; }
$ each_line notes.txt show
[  indented line]
[back\slash]
[last line no newline]
$ each_line hosts.txt printf 'host: %s\n'
host: web01
host: web02
host: web03
$ each_line missing.txt show; echo "rc=$?"
rc=1
$ n=0; count() { n=$((n+1)); }; each_line hosts.txt count; echo "n=$n"
n=3

The command gets the line as its last argument. Because there is no pipe, a function you pass in can change global variables, as the counter shows.

mapfile: the whole file into an array

If you need all the lines at once, mapfile (Bash 4.0+, not in the macOS 3.2 shell) reads them into an array in one builtin call:

terminal

$ mapfile -t lines < notes.txt
$ declare -p lines
declare -a lines=([0]="  indented line" [1]="back\\slash" [2]="last line no newline")

-t strips the newline from each element. Whitespace and backslashes are kept, and the unterminated last line is included without any extra test. It is also much faster: on a 100,000-line file, the while read loop took 1.47s on this machine and mapfile took 0.013s. Of course it holds the whole file in memory, so for a multi-gigabyte log, stream it with while read or let awk do the work.

Reading fields

Give read more than one variable and it splits the line on IFS. The last variable gets the rest of the line. Here it parses a copy of a passwd file, using _ for fields I do not care about:

bash

while IFS=: read -r user _ uid _ gecos home shell; do
    printf '%-6s uid=%-5s shell=%s\n' "$user" "$uid" "$shell"
done < passwd.txt

output

alice  uid=1000  shell=/bin/bash
bob    uid=1001  shell=/bin/sh

An empty field (Bob has no GECOS entry) is kept as an empty variable, because : is not whitespace. With the default whitespace IFS, runs of spaces count as one separator instead. To skip comments and blank lines, add [[ $line =~ ^[[:space:]]*(#|$) ]] && continue at the top of the loop.

CRLF files from Windows

A file saved on Windows ends each line with \r\n. read stops at the \n, so every line keeps an invisible \r, which breaks comparisons and file names:

terminal

$ printf 'host=web01\r\nport=8080\r\n' > crlf.txt
$ while IFS= read -r line; do printf '[%s]\n' "$line"; done < crlf.txt | cat -A
[host=web01^M]$
[port=8080^M]$
$ while IFS= read -r line; do
>     line=${line%$'\r'}; printf '[%s]\n' "$line"
> done < crlf.txt | cat -A
[host=web01]$
[port=8080]$

${line%$'\r'} removes one trailing carriage return if there is one, so the same loop handles Unix and Windows files. When splitting fields you can add \r to IFS instead: IFS=$'=\r' read -r key value gave host -> [web01] and port -> [8080]. For an array, "${lines[@]%$'\r'}" strips it from every element after mapfile. If you can change the file, dos2unix or sed -i 's/\r$//' fixes it once.

The Bash builtins chapter documents every option of read and mapfile.

trim(): strip whitespace from a line you have read
explode() and implode(): split a line into an array
emptydir(): another file helper
Bash strict mode: what set -euo pipefail really does
trap and cleanup: removing temporary files when the loop is done

The idiom

The last line without a newline

Why for line in $(cat file) is wrong

Pipe vs redirect: where did my variable go?

Commands that read your input

A reusable helper

mapfile: the whole file into an array

Reading fields

CRLF files from Windows

Related functions

Related articles

Bash trap: Clean Up Temp Files and Locks on Exit

getopts in Bash: Parse Command-Line Options Properly

in_array()