rsync Recipes: Trailing Slashes, --delete and Filters

rsync is the tool I use most and trust least. It copies only what changed, it resumes, it works over SSH, and with one wrong character it nests a directory inside itself or wipes a backup. This is my set of recipes, each one run locally between scratch directories with rsync 3.1.3, so the output is real.

The demo source is a small website directory:

site/
  README.txt
  index.html
  cache/page.tmp
  css/style.css
  images/logo.png
  images/empty/
  js/app.js

The trailing slash on the source

This is the one rule everybody has to learn. A trailing slash on the source means "the contents of this directory". No slash means "this directory itself".

$ rsync -a site backup1
$ find backup1 -maxdepth 2 | sort
backup1
backup1/site
backup1/site/cache
backup1/site/css
backup1/site/images
backup1/site/index.html
backup1/site/js
backup1/site/README.txt
$ rsync -a site/ backup2
$ find backup2 -maxdepth 1 | sort
backup2
backup2/cache
backup2/css
backup2/images
backup2/index.html
backup2/js
backup2/README.txt

Without the slash, you get backup1/site/.... With it, the files land directly in backup2. A slash on the destination makes no difference:

$ rsync -a site/ backup3/
$ diff -r backup2 backup3 && echo same
same

My habit: always put a slash on both the source and the destination when I mean "make this directory look like that one". Then the command reads the same every time and I never have to think about it. If you run a sync twice and the second run creates backup/site/site, you have dropped a slash somewhere.

What -a actually means

-a (archive) is shorthand for -rlptgoD: recurse into directories, copy symlinks as symlinks, and preserve permissions, modification times, group, owner and device/special files. Two things are not included: hard links (-H) and ACLs and extended attributes (-A, -X). Add those for full system backups.

Owner preservation only works when the receiving side runs as root. As a normal user every file ends up owned by you. And preserving modification times is not cosmetic. As you will see below, rsync uses them to decide what needs copying, so without -t (or -a) every run copies every file again.

--delete, safely

By default rsync never removes anything from the destination. Files deleted from the source stay in the copy forever. --delete makes the destination an exact mirror, and it is the option that destroys data when the command is wrong. So the rule is: run it with --dry-run (-n) first, every time.

$ rm site/README.txt
$ printf "new\n" > site/about.html
$ rsync -av --delete --dry-run site/ backup2/
sending incremental file list
deleting README.txt
about.html

sent 323 bytes  received 38 bytes  722.00 bytes/sec
total size is 34  speedup is 0.09 (DRY RUN)
$ ls backup2
cache
css
images
index.html
js
README.txt

The dry run listed one deletion and one new file and changed nothing. Happy with that, I ran it for real. -i (itemize changes) gives a compact one-line-per-change log that I find easier to scan than -v:

$ rsync -a --delete -i site/ backup2/
*deleting   README.txt
>f+++++++++ about.html

Interestingly, the classic missing-slash mistake is fairly harmless with --delete: rsync only deletes inside the directories it is sending, so rsync -a --delete site backup2/ just creates a nested backup2/site/ and removes nothing. The real disaster is an empty source. Imagine a backup script whose source is a mount point, on a night the disk failed to mount:

$ mkdir mnt
$ rsync -a --delete --dry-run -i mnt/ backup2/
*deleting   js/app.js
*deleting   js/
*deleting   images/empty/
*deleting   images/logo.png
*deleting   images/
*deleting   css/style.css
*deleting   css/
*deleting   cache/page.tmp
*deleting   cache/
*deleting   index.html
*deleting   about.html

An empty source and --delete means "delete everything". In unattended scripts I add --max-delete as a safety limit. When the limit is hit, rsync stops deleting and exits with code 25, which your script or monitoring can catch:

$ rsync -a --delete --max-delete=3 --dry-run -i mnt/ backup2/
*deleting   js/app.js
*deleting   js/
*deleting   images/empty/
cannot delete non-empty directory: images
cannot delete non-empty directory: css
cannot delete non-empty directory: cache
Deletions stopped due to --max-delete limit (5 skipped)
rsync error: the --max-delete limit stopped deletions (code 25) at main.c(1187) [sender=3.1.3]

Checking that the source is actually mounted (mountpoint -q /mnt/data || exit 1) before running the sync is even better.

--exclude and --include: order matters

Excluding a directory is simple:

$ rsync -av --exclude='cache/' site/ deploy/
sending incremental file list
created directory deploy
./
about.html
index.html
css/
css/style.css
images/
images/logo.png
images/empty/
js/
js/app.js

sent 539 bytes  received 167 bytes  1,412.00 bytes/sec
total size is 30  speedup is 0.04

The trailing slash in the pattern means "directories only", and an unanchored pattern matches at any depth, so cache/ would also exclude js/cache/. Write /cache/ to match only at the top of the transfer.

Copying only certain files is where people get stuck. rsync checks each path against the rules in order and the first match wins. To copy just CSS and JavaScript, you need three things: include every directory (otherwise rsync never descends into them), include the files you want, then exclude everything else:

$ rsync -a --include='*/' --include='*.css' --include='*.js' --exclude='*' site/ assets/
$ find assets | sort
assets
assets/cache
assets/css
assets/css/style.css
assets/images
assets/images/empty
assets/js
assets/js/app.js

It worked, but it copied every directory too, including cache and images with nothing in them. -m (--prune-empty-dirs) drops directories that end up empty:

$ rsync -a -m --include='*/' --include='*.css' --include='*.js' --exclude='*' site/ assets/
$ find assets | sort
assets
assets/css
assets/css/style.css
assets/js
assets/js/app.js

Note that -m also removes directories that were empty in the source, such as images/empty. Put the exclude first and nothing gets through at all, because * matches every path before the includes are reached:

$ rsync -a --exclude='*' --include='*/' --include='*.css' site/ assets/
$ find assets | sort
assets

Size and mtime vs --checksum

By default rsync skips a file when its size and modification time match on both sides. This "quick check" is why rsync is fast: it does not read unchanged files at all. It can be fooled, though. Here I changed one byte in the copy (same length) and then copied the source's timestamp onto it:

$ rsync -a site/ copy/
$ printf "Body{}\n" > copy/css/style.css
$ touch -r site/css/style.css copy/css/style.css
$ rsync -ai site/ copy/
$ cat copy/css/style.css
Body{}
$ rsync -aic site/ copy/
>fc........ css/style.css
$ cat copy/css/style.css
body{}

The normal run saw nothing to do. -c (--checksum) compares a checksum of every file, spotted the difference (the c in >fc) and fixed it. The price is that both sides read every byte of every file on every run, which on a large tree is slow and puts a heavy load on disks. I use -c to verify a copy once, or when a tool rewrites files while keeping their timestamps, not for routine syncs.

--partial and --progress for big files

--progress shows per-file progress, and --partial keeps a partly transferred file if the connection drops, so the next run can reuse what already arrived instead of starting from zero. Without it, rsync deletes the partial file. -P means both:

$ rsync -aP ztest/random.bin big/
     10,000,000 100%  633.70MB/s    0:00:00 (xfr#1, to-chk=0/1)

(The real output redraws that line as it goes. I am showing the final state.) For a whole tree, --info=progress2 gives one overall progress line instead of one per file.

When -z helps

-z compresses data on the wire. It helps for compressible data (text, logs, SQL dumps) over a slow link and does nothing for data that is already compressed. (rsync 3.1.3 even compresses a purely local copy if you ask, which only wastes CPU.) I wanted to measure it over a real protocol connection without a remote machine. So I gave rsync a replacement for ssh via -e: a three-line script that drops the host name and starts the rsync server locally. The real network protocol then runs over a pipe. The bytes sent for each file, measured on this one machine:

The log shrank to about a sixth. The gzip file and random data got slightly bigger. Compression also costs CPU on both ends, so on a fast LAN -z can make a transfer slower. Use it over the internet for text-heavy data and leave it off otherwise. If your SSH connection already compresses (Compression yes), do not compress twice.

Over SSH

A colon in a path means "remote host". rsync then runs over SSH, starting a second rsync on the other side, which must be installed there too. I cannot run these from my test machine, so here is the syntax rather than output:

# local to remote
rsync -avz site/ deploy@web1.example.com:/var/www/site/
# remote to local
rsync -avz deploy@web1.example.com:/var/www/site/ site-backup/
# non-standard port or key: pass options to ssh with -e
rsync -avz -e 'ssh -p 2222 -i ~/.ssh/deploy_key' site/ deploy@web1.example.com:/var/www/site/
# a deployment: mirror, but never touch uploads, dry run first
rsync -avn --delete --exclude='/uploads/' site/ deploy@web1.example.com:/var/www/site/

The trailing-slash rules are identical on remote paths. A double colon (host::module) is different: it talks to an rsync daemon on port 873, not SSH.

My default commands

• Mirror a directory: rsync -a --delete -i src/ dst/, always after the same command with -n.
• Full backup of a system tree: rsync -aHAX --delete src/ dst/, run as root.
• Big files over an unreliable link: rsync -aP src/ host:dst/.
• Verify a copy: rsync -ainc src/ dst/ (prints only real differences, changes nothing).

The rsync manual page is long but worth reading, particularly the "FILTER RULES" section once you need more than the patterns above.

Related reading

• systemd timers: a practical replacement for cron jobs
• Bash trap: clean up temp files and locks on exit
• emptydir(): check whether a directory is empty