~/ronggeng
Linux

Docker Multi-Stage Builds: Smaller Images, Same Result

By Nawawi JamiliPublished 8 min read

To build a program inside Docker you need a compiler, headers, a package manager and often many build tools. To run it you need none of that. A single-stage Dockerfile includes all of it anyway. A multi-stage build compiles in one image and copies only the result into another. The difference is large. In the test below, the same 147 kB binary is in a 257 MB image one way and an 8.44 MB image the other.

All builds here ran on Docker 26.1.3 with the legacy builder, because the buildx plugin (and with it BuildKit) is not installed on this machine. I have removed the deprecation note it prints from the output. Where BuildKit behaves differently I say so, based on the Docker documentation.

The program

A small C version of wc that counts lines, words and bytes on standard input. Any compiled program would do. C keeps the example free of downloads apart from the compiler itself.

output
/* rgwc: count lines, words and bytes on stdin */
#include <ctype.h>
#include <stdio.h>
int main(void)
{
    long lines = 0, words = 0, bytes = 0;
    int c, inword = 0;
    while ((c = getchar()) != EOF) {
        bytes++;
        if (c == '\n')
            lines++;
        if (isspace(c)) {
            inword = 0;
        } else if (!inword) {
            inword = 1;
            words++;
        }
    }
    printf("%ld %ld %ld\n", lines, words, bytes);
    return 0;
}

Single stage: build and run in one image

output
FROM alpine:3.22
LABEL rg.demo=multistage
RUN apk add --no-cache build-base
WORKDIR /src
COPY wc.c .
RUN gcc -O2 -static -o /usr/local/bin/rgwc wc.c
ENTRYPOINT ["/usr/local/bin/rgwc"]

The label is only there so I can find every image from this test later. build-base is Alpine's meta package for gcc, make, the C library headers and related tools. The build output tells you what that costs before the image even exists:

terminal
$ docker build -f Dockerfile.single -t rg-wc:single .
(output trimmed)
(21/21) Installing build-base (0.5-r3)
Executing busybox-1.37.0-r20.trigger
OK: 245 MiB in 37 packages
(output trimmed)
Successfully built c1f2e8982899
Successfully tagged rg-wc:single

The program works, and so does everything else that was installed with it:

terminal
$ printf 'one two three\nfour five\n' | docker run --rm -i --name rg-wc-run rg-wc:single
2 5 24
$ docker run --rm --name rg-wc-run \
    --entrypoint sh rg-wc:single -c 'which gcc; id; du -sh /usr/libexec/gcc'
/usr/bin/gcc
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
116.2M	/usr/libexec/gcc

A full compiler toolchain, running as root, in an image whose only job is to count words. Anyone who gets a shell in that container can build new tools right there.

Multi-stage: name the stages, copy the artefact

A Dockerfile can have several FROM lines. Each one starts a new stage with a fresh filesystem. AS name gives the stage a name, and COPY --from=name copies files out of an earlier stage. Only the last stage (or the one you pick with --target) becomes the image you tag.

output
FROM alpine:3.22 AS build
LABEL rg.demo=multistage
RUN apk add --no-cache build-base
WORKDIR /src
COPY wc.c .
RUN gcc -O2 -static -o /usr/local/bin/rgwc wc.c
FROM build AS debug
RUN apk add --no-cache gdb strace
RUN gcc -O0 -g -static -o /usr/local/bin/rgwc wc.c
FROM alpine:3.22 AS runtime
LABEL rg.demo=multistage
COPY --from=build /usr/local/bin/rgwc /usr/local/bin/rgwc
USER nobody
ENTRYPOINT ["/usr/local/bin/rgwc"]

Three stages. build is the same as the single-stage file. debug starts from build and adds gdb, strace and an unoptimised binary with debug symbols. runtime starts again from plain Alpine and takes nothing but the binary. Name stages instead of using their numbers (COPY --from=0 also works). Adding a stage at the top later would silently change what 0 means.

I linked statically with -static so the binary has no shared library dependencies and runs on any Linux base. If you link dynamically, the runtime image needs the same C library as the build image. A binary built against glibc on Debian will not run on musl-based Alpine.

The sizes

I also built a fourth variant whose final stage is FROM scratch, an empty image, with the binary copied to /rgwc and USER 65534:65534 (a numeric user, because scratch has no /etc/passwd). Then:

terminal
$ docker images rg-wc
REPOSITORY   TAG       IMAGE ID       CREATED                  SIZE
rg-wc        scratch   c9305d208195   Less than a second ago   147kB
rg-wc        multi     2c19993b6598   10 seconds ago           8.44MB
rg-wc        debug     9d36b8dd139f   11 seconds ago           312MB
rg-wc        single    c1f2e8982899   23 seconds ago           257MB
$ printf 'one two three\nfour five\n' | docker run --rm -i --name rg-wc-run rg-wc:multi
2 5 24
$ printf 'one two three\nfour five\n' | docker run --rm -i --name rg-wc-run rg-wc:scratch
2 5 24
$ printf 'one two three\nfour five\n' | wc
      2       5      24

Same output from all of them, and it agrees with the real wc. The multi-stage image is the 8.3 MB Alpine base plus one 147 kB layer:

terminal
$ docker history rg-wc:multi
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
2c19993b6598   32 seconds ago   /bin/sh -c #(nop)  ENTRYPOINT ["/usr/local/b…   0B
ab370077172e   32 seconds ago   /bin/sh -c #(nop)  USER nobody                  0B
24f1d18b9be1   33 seconds ago   /bin/sh -c #(nop) COPY file:5d78389f0f31db3a…   147kB
6699a4e54ec8   52 seconds ago   /bin/sh -c #(nop)  LABEL rg.demo=multistage     0B
c83674e19990   4 weeks ago      CMD ["/bin/sh"]                                 0B        buildkit.dockerfile.v0
<missing>      4 weeks ago      ADD alpine-minirootfs-3.22.6-x86_64.tar.gz /…   8.3MB     buildkit.dockerfile.v0
$ docker run --rm --name rg-wc-run \
    --entrypoint sh rg-wc:multi -c 'which gcc; id; ls -l /usr/local/bin/rgwc'
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)
-rwxr-xr-x    1 root     root        146864 Oct 17 19:10 /usr/local/bin/rgwc

which gcc printed nothing because there is no compiler, and the process runs as nobody. Smaller images pull faster, start faster on a fresh host and give a vulnerability scanner fewer things to report.

Which runtime base to pick is a trade-off. scratch is the smallest but has no shell, no CA certificates, no time zone data and no user database. So debugging means copying in tools or using a debug image. Alpine gives you a shell for 8 MB. A slim Debian base is the safe choice for dynamically linked programs built against glibc.

--target for a debug image

--target stops the build at a named stage and tags that stage as the image:

terminal
$ docker build --target debug -t rg-wc:debug .
Step 1/9 : FROM alpine:3.22 AS build
Step 2/9 : LABEL rg.demo=multistage
Step 3/9 : RUN apk add --no-cache build-base
Step 4/9 : WORKDIR /src
Step 5/9 : COPY wc.c .
Step 6/9 : RUN gcc -O2 -static -o /usr/local/bin/rgwc wc.c
Step 7/9 : FROM build AS debug
Step 8/9 : RUN apk add --no-cache gdb strace
Step 9/9 : RUN gcc -O0 -g -static -o /usr/local/bin/rgwc wc.c
Successfully built 9d36b8dd139f
Successfully tagged rg-wc:debug
$ docker run --rm --name rg-wc-run \
    --entrypoint sh rg-wc:debug -c 'which gdb strace'
/usr/bin/gdb
/usr/bin/strace

(I filtered that build output down to the step lines and the result.) One Dockerfile now gives a 312 MB image with debuggers for when something goes wrong and an 8.44 MB image for production. CI can also use --target build to run tests inside the build stage before the runtime stage is produced.

The legacy builder builds stages you do not need

Here is the plain docker build -t rg-wc:multi . from above, filtered to the step lines:

terminal
$ docker build -t rg-wc:multi .
Step 1/14 : FROM alpine:3.22 AS build
(steps 2 to 6 trimmed, all "Using cache")
Step 7/14 : FROM build AS debug
 ---> ca1a07009bc8
Step 8/14 : RUN apk add --no-cache gdb strace
 ---> Running in 1ebd10879e28
 ---> Removed intermediate container 1ebd10879e28
 ---> 54292c457c70
Step 9/14 : RUN gcc -O0 -g -static -o /usr/local/bin/rgwc wc.c
 ---> Running in 65a12c402bd3
 ---> Removed intermediate container 65a12c402bd3
 ---> 9d36b8dd139f
Step 10/14 : FROM alpine:3.22 AS runtime
(output trimmed)
Successfully tagged rg-wc:multi

The runtime stage never refers to debug, yet steps 8 and 9 ran: the legacy builder installed gdb and strace and compiled the debug binary without any need. It processes every stage in file order up to the target. The build stage came fully from the cache, because its instructions are identical to the single-stage file I had built a minute earlier.

BuildKit, according to the Docker documentation, finds out which stages the target depends on and skips the rest. It can also build independent stages in parallel. I could not verify that on this machine. If you must use the legacy builder, put optional stages such as debug after the production stage. Then select the production stage with --target runtime, because the build stops at the target. I moved the debug stage to the end of the file and tried it:

terminal
$ docker build --target runtime -f Dockerfile.reordered -t rg-wc:reordered .
(filtered to the step lines)
Step 1/11 : FROM alpine:3.22 AS build
Step 2/11 : LABEL rg.demo=multistage
Step 3/11 : RUN apk add --no-cache build-base
Step 4/11 : WORKDIR /src
Step 5/11 : COPY wc.c .
Step 6/11 : RUN gcc -O2 -static -o /usr/local/bin/rgwc wc.c
Step 7/11 : FROM alpine:3.22 AS runtime
Step 8/11 : LABEL rg.demo=multistage
Step 9/11 : COPY --from=build /usr/local/bin/rgwc /usr/local/bin/rgwc
Step 10/11 : USER nobody
Step 11/11 : ENTRYPOINT ["/usr/local/bin/rgwc"]
Successfully built 2c19993b6598
Successfully tagged rg-wc:reordered

The debug stage was skipped, and the image ID is the same as rg-wc:multi. The problem is that a plain docker build without --target now produces the debug image, because it is the last stage. So the --target has to be in your build script.

What must not reach the final image

  • Compilers and build tools: gcc, make, -dev header packages, node-gyp, Go toolchains.
  • Package caches: /var/cache/apk, /var/lib/apt/lists, ~/.npm, ~/.cache/pip, ~/.composer/cache.
  • Source code and test files the program does not read at run time.
  • Secrets used during the build: tokens for private registries, SSH keys, .npmrc with credentials.

Multi-stage builds help with secrets, but less than people think. I passed a fake token as a build argument in the first stage and copied only a log file into the second:

output
FROM alpine:3.22 AS build
LABEL rg.demo=multistage
ARG API_TOKEN
RUN test -n "$API_TOKEN" && echo "token used" > /build.log
FROM alpine:3.22
LABEL rg.demo=multistage
COPY --from=build /build.log /build.log
terminal
$ docker build -q -f Dockerfile.secret --build-arg API_TOKEN=s3cr3t-demo \
    -t rg-secret:multi .
sha256:933f8a8013a06517a29ea12d69d0713c2888e38ac7c230b0aa14be56a347dc07
$ docker history --no-trunc rg-secret:multi | grep -c s3cr3t
0
$ docker images -f dangling=true -f label=rg.demo=multistage
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
<none>       <none>    83814d3530f6   7 seconds ago   8.3MB
$ docker history --no-trunc --format '{{.CreatedBy}}' 83814d3530f6 | grep s3cr3t
|1 API_TOKEN=s3cr3t-demo /bin/sh -c test -n "$API_TOKEN" && echo "token used" > /build.log

The pushed image is clean. But the legacy builder keeps the build stage as an untagged image on the build host, and its history records the build argument in plain text. On a shared CI runner, anyone who can run docker images can read it. A secret written to a file in the build stage is also present in that stage's layers. BuildKit has proper secret mounts (RUN --mount=type=secret) that never write the value into a layer. They are not available with the legacy builder, so I could not test them here. The multi-stage builds documentation covers the BuildKit side.

Topics: dockerperformancesecurity

Related articles