~/ronggeng
Linux

Dockerfile Layer Caching: Order Instructions for Fast Rebuilds

By Nawawi JamiliPublished 9 min read

You change one line of Python, run docker build, and watch pip download and install every dependency again. Nothing about the dependencies changed. The cause is almost always the order of the instructions in the Dockerfile, and the fix is quick once you know how Docker decides what it can reuse.

Everything below was run on one machine with Docker 26.1.3. That machine only has the legacy builder (the buildx plugin is not installed), so every build prints a deprecation note, which I have removed from the output. The basic cache rules are the same under BuildKit, but its progress output looks different.

The demo app

A tiny Flask app with five pinned dependencies in requirements.txt:

output
flask==3.0.3
gunicorn==22.0.0
requests==2.32.3
SQLAlchemy==2.0.36
PyYAML==6.0.2
python
from flask import Flask, jsonify
app = Flask(__name__)
@app.get("/")
def index():
    return jsonify(status="ok", version="1.0")

The build container has network access, so pip really installs from PyPI. Here is the Dockerfile most people write first, saved as Dockerfile.slow:

output
FROM python:3.12-alpine
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_ROOT_USER_ACTION=ignore
WORKDIR /app
COPY . .
RUN pip install --no-cache-dir -r requirements.txt
CMD ["gunicorn", "-b", "0.0.0.0:8000", "app:app"]

Every instruction is a layer

Each instruction produces a new image on top of the previous one. RUN and COPY add filesystem changes. ENV, WORKDIR and CMD only change metadata and add a 0-byte layer. docker history shows the stack, newest first:

terminal
$ docker history rg-cache-demo:fast
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
6866aa024a86   14 seconds ago   /bin/sh -c #(nop)  CMD ["gunicorn" "-b" "0.0…   0B
0638fe72e2b3   14 seconds ago   /bin/sh -c #(nop) COPY dir:f7aafcec8689ded6d…   658B
fcdc46c35570   15 seconds ago   /bin/sh -c pip install --no-cache-dir -r req…   47.3MB
9e577d1a0961   22 seconds ago   /bin/sh -c #(nop) COPY file:ecf9997738b5810a…   80B
d343a7d0d386   55 seconds ago   /bin/sh -c #(nop) WORKDIR /app                  0B
f25595a7dc9d   55 seconds ago   /bin/sh -c #(nop)  ENV PIP_DISABLE_PIP_VERSI…   0B
979e5a8a5684   2 weeks ago      CMD ["python3"]                                 0B        buildkit.dockerfile.v0
(base image layers trimmed)

When it builds, Docker reads the Dockerfile from the top. For each instruction it looks for an existing image that has the same parent and was made by the same instruction. If it finds one, it prints Using cache and moves on. The comparison depends on the instruction:

  • RUN: only the command string is compared. Docker does not check what the command downloads.
  • COPY and ADD: Docker checksums the content of the copied files. Modification times are ignored. I touched both files and rebuilt, and both COPY steps still said Using cache.
  • Everything else: the instruction text itself.

The rule that matters most: once one step misses the cache, every step after it is rebuilt, because their parent image is new.

The slow order, timed

A second build with nothing changed reuses every layer:

terminal
$ time docker build -f Dockerfile.slow -t rg-cache-demo:slow .
Sending build context to Docker daemon   5.12kB
Step 1/6 : FROM python:3.12-alpine
 ---> 979e5a8a5684
Step 2/6 : ENV PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_ROOT_USER_ACTION=ignore
 ---> Using cache
 ---> f25595a7dc9d
Step 3/6 : WORKDIR /app
 ---> Using cache
 ---> d343a7d0d386
Step 4/6 : COPY . .
 ---> Using cache
 ---> 785b5fbcb836
Step 5/6 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Using cache
 ---> f2da1e57f806
Step 6/6 : CMD ["gunicorn", "-b", "0.0.0.0:8000", "app:app"]
 ---> Using cache
 ---> b9b400a2b8e7
Successfully built b9b400a2b8e7
Successfully tagged rg-cache-demo:slow

real	0m0.018s
user	0m0.007s
sys	0m0.005s

Now change the version string in app.py from 1.0 to 1.1 and build again:

terminal
$ sed -i 's/version="1.0"/version="1.1"/' app.py
$ time docker build -f Dockerfile.slow -t rg-cache-demo:slow .
Sending build context to Docker daemon   5.12kB
Step 1/6 : FROM python:3.12-alpine
 ---> 979e5a8a5684
Step 2/6 : ENV PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_ROOT_USER_ACTION=ignore
 ---> Using cache
 ---> f25595a7dc9d
Step 3/6 : WORKDIR /app
 ---> Using cache
 ---> d343a7d0d386
Step 4/6 : COPY . .
 ---> c29f25d6be3a
Step 5/6 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Running in 2b5aef76ce84
(pip output trimmed)
 ---> Removed intermediate container 2b5aef76ce84
 ---> 0491a6b3a31f
Step 6/6 : CMD ["gunicorn", "-b", "0.0.0.0:8000", "app:app"]
 ---> Running in 15cd0a2f4315
 ---> Removed intermediate container 15cd0a2f4315
 ---> b476917b7b67
Successfully built b476917b7b67
Successfully tagged rg-cache-demo:slow

real	0m6.660s

COPY . . copies app.py, so its checksum changed and step 4 missed the cache. Step 5 has a new parent, so pip runs again. Six and a half seconds is not much because this is a small project on a fast connection. With a few hundred npm packages or a dependency that compiles C code, the same mistake costs minutes on every edit.

Copy the dependency list first

The fix is to copy only the file that describes the dependencies, install them, and copy the rest of the source afterwards. Dockerfile.fast:

output
FROM python:3.12-alpine
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_ROOT_USER_ACTION=ignore
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["gunicorn", "-b", "0.0.0.0:8000", "app:app"]

The first build took 7.7 seconds because the dependencies had to be installed once. Then another edit to app.py:

terminal
$ sed -i 's/version="1.1"/version="1.2"/' app.py
$ time docker build -f Dockerfile.fast -t rg-cache-demo:fast .
Sending build context to Docker daemon   5.12kB
(steps 1 to 3 trimmed, all "Using cache")
Step 4/7 : COPY requirements.txt .
 ---> Using cache
 ---> 9e577d1a0961
Step 5/7 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Using cache
 ---> fcdc46c35570
Step 6/7 : COPY . .
 ---> 0638fe72e2b3
Step 7/7 : CMD ["gunicorn", "-b", "0.0.0.0:8000", "app:app"]
 ---> Running in a1aa7177ee1e
 ---> Removed intermediate container a1aa7177ee1e
 ---> 6866aa024a86
Successfully built 6866aa024a86
Successfully tagged rg-cache-demo:fast

real	0m0.116s

From 6.66 seconds to 0.116 seconds, measured on this one machine. The pip layer is rebuilt only when requirements.txt itself changes, which is exactly when you want it rebuilt. The same pattern works for every ecosystem: package.json and package-lock.json before npm ci, composer.json and composer.lock before composer install, go.mod and go.sum before go mod download.

The general rule: order instructions from the least frequently changed to the most frequently changed. Base image and system packages first, dependency manifests next, your own code last.

.dockerignore: a smaller context and fewer cache misses

A real project directory has more than source code in it. I added a Git repository, a local virtualenv, a 150 MB database dump in data/ and a .env file, then built again:

terminal
$ du -sh .git .venv data
128K	.git
9.7M	.venv
150M	data
$ time docker build -f Dockerfile.fast -t rg-cache-demo:fast .
Sending build context to Docker daemon  166.3MB
(steps trimmed)
real	0m3.325s
$ docker image ls rg-cache-demo:fast
REPOSITORY      TAG       IMAGE ID       CREATED         SIZE
rg-cache-demo   fast      ab73c646c926   6 seconds ago   268MB
$ docker run --rm --name rg-cache-ls rg-cache-demo:fast ls -A /app
.env
.git
.venv
Dockerfile.fast
Dockerfile.slow
app.py
data
requirements.txt

This causes three problems. The whole directory is sent to the daemon before step 1 even starts. COPY . . put the dump and the secrets file into the image, which grew from 102 MB to 268 MB. And because .git is part of what COPY . . checksums, any commit or fetch invalidates that layer even when no source file changed.

A .dockerignore file next to the Dockerfile fixes all three:

output
.git
.venv
data/
.env
**/__pycache__
Dockerfile*
.dockerignore
terminal
$ time docker build -f Dockerfile.fast -t rg-cache-demo:fast .
Sending build context to Docker daemon   5.12kB
(steps trimmed)
real	0m0.150s
$ docker image ls rg-cache-demo:fast
REPOSITORY      TAG       IMAGE ID       CREATED                  SIZE
rg-cache-demo   fast      11ff9d4d2c8f   Less than a second ago   102MB
$ docker run --rm --name rg-cache-ls rg-cache-demo:fast ls -A /app
app.py
requirements.txt

The context went from 166.3 MB to 5.12 kB. Excluding the Dockerfile itself is safe because the builder still reads it. The benefit is that editing a comment in the Dockerfile no longer changes what COPY . . copies, so it no longer invalidates that layer and everything after it.

apt-get update and install belong in one RUN

Because RUN is cached by its command string, splitting the package index update from the install causes a problem:

output
FROM debian:bookworm-slim
RUN apt-get update
RUN apt-get install -y --no-install-recommends curl
RUN rm -rf /var/lib/apt/lists/*

Weeks later you add jq to the install line. The install step misses the cache, but the update step does not:

terminal
$ docker build -f Dockerfile.split -t rg-apt:split .
Sending build context to Docker daemon  3.072kB
Step 1/4 : FROM debian:bookworm-slim
 ---> d4c45f66efc5
Step 2/4 : RUN apt-get update
 ---> Using cache
 ---> 83666cc08567
Step 3/4 : RUN apt-get install -y --no-install-recommends curl jq
 ---> Running in c6d1420b978f
(apt output trimmed)

The install runs against a package index from the day that layer was cached. Once the mirror has changed, you get 404 errors for package versions that no longer exist, or old packages without any warning. The split version also wastes space, because deleting the lists in a later layer cannot shrink an earlier one. I built both forms with only curl:

output
FROM debian:bookworm-slim
RUN apt-get update \
    && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*
terminal
$ docker image ls rg-apt
REPOSITORY   TAG       IMAGE ID       CREATED                  SIZE
rg-apt       one       ff5235e5759a   Less than a second ago   86.9MB
rg-apt       split     4b9f7bbc4b9d   4 seconds ago            107MB

The 19.8 MB package index stays in the split image forever. In the single RUN the update, install and cleanup are one layer, and changing the package list re-runs all three.

ARG and ENV can invalidate the cache too

Build arguments are a less obvious way to lose the cache. I declared ARG APP_VERSION=dev near the top, right after the first ENV, and only used it at the end in ENV APP_VERSION=$APP_VERSION. Building with --build-arg APP_VERSION=1.2 and then 1.3:

terminal
$ time docker build -f Dockerfile.arg --build-arg APP_VERSION=1.3 \
    -t rg-cache-demo:arg .
(steps 1 to 5 trimmed, all "Using cache")
Step 6/9 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Running in 12ec23d6e97b
(pip output trimmed)
real	0m7.261s

The pip step never mentions APP_VERSION, yet it re-ran. Every RUN after an ARG sees the argument as an environment variable, so a new value counts as a different command. I then moved the ARG down below the COPY . ., right before the ENV that uses it. After that, the same change took only 0.127 seconds, with every step up to the ENV taken from cache.

ENV works the same way, but with a bigger effect: an ENV BUILD_DATE=... filled from a build argument near the top of the file means a full rebuild every time. Put values that change on every build as late as possible.

Forcing a rebuild

Sometimes the cached result is out of date. pip install -r requirements.txt with unpinned versions stays cached forever, and so does apt-get upgrade, because the command text never changes. --no-cache ignores all cached layers:

terminal
$ time docker build --no-cache -f Dockerfile.fast -t rg-cache-demo:fast .
(output trimmed)
real	0m7.106s

It does not fetch a newer base image, though. --pull does that, and it is worth having both in a nightly CI build so security updates are applied even when nobody touched the code.

BuildKit adds cache mounts (RUN --mount=type=cache,target=/root/.cache/pip) that keep pip or npm download caches between builds even when the layer itself is rebuilt. The legacy builder on this machine does not support them, so I could not test them here. See the Docker build cache documentation for the details.

Topics: cachingdockerperformance

Related articles