Optimizing Rust container builds 🔗
I’m a Rust newbie, and one of the things that I’ve found frustrating is that the default docker build experience is extremely slow. As it downloads crates, then dependencies, then finally my app - I often get distracted, start doing something else, then come back several minutes later and forget what I was doing
Recently, I had the idea to make it a little better by combining multistage builds with some of the amazing features from BuildKit. Specifically, cache mounts, which let a build container cache directories for compilers & package managers. Here’s a quick annotated before & after from a real app I encountered.
Before 🔗
This is a standard enough multistage Dockerfile. Nothing seemingly terrible or great here - just a normal build stage, and a smaller runtime stage.
FROM rust:1.55 AS build
WORKDIR /app
# The app has 2 parts: an application named "api", and a lib named "game"
# Copy the sources
COPY ./api ./api
COPY ./game ./game
# Build the app
WORKDIR /app/api
RUN cargo build --release
# Use a slim Dockerfile with just our app to publish
FROM debian:buster-slim AS app
COPY --from=build /app/target/release/my-app /
CMD ["/my-app"]
This corresponds to the following build times
# Let's pre-pull the bases so we don't unnecessarily penalize the first build
docker pull rust:1.55
docker pull debian:buster-slim
# First build from scratch
time docker build .
real 5m43.506s
user 0m1.239s
sys 0m0.872s
# Change a file in api/src, and build again
time docker build .
real 5m44.731s
user 0m1.199s
sys 0m0.938s
Wow, 5 minutes. Yes, I’m probably doing cargo build outside of Docker and the real effects aren’t this drastic, but this is an eternity for my short attention span. This is our baseline - let’s see if we can improve it.
After 🔗
Here we’re going to keep multistage builds, but we’ll make a few changes:
- Split layers so that we cache compiled dependencies. Turns out this is harder in Rust than other languages.
- Use BuildKit + cache mounts. This will save us some download time when we have to rebuild dependencies
# syntax=docker/dockerfile:1.3-labs
# The above line is so we can use can use heredocs in Dockerfiles. No more && and \!
# https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/
FROM rust:1.55 AS build
# Capture dependencies
COPY Cargo.toml Cargo.lock /app/
# We create a new lib and then use our own Cargo.toml
RUN cargo new --lib /app/game
COPY game/Cargo.toml /app/game/
# We do the same for our app
RUN cargo new /app/api
COPY api/Cargo.toml /app/api/
# This step compiles only our dependencies and saves them in a layer. This is the most impactful time savings
# Note the use of --mount=type=cache. On subsequent runs, we'll have the crates already downloaded
WORKDIR /app/api
RUN --mount=type=cache,target=/usr/local/cargo/registry cargo build --release
# Copy our sources
COPY ./api /app/api
COPY ./game /app/game
# A bit of magic here!
# * We're mounting that cache again to use during the build, otherwise it's not present and we'll have to download those again - bad!
# * EOF syntax is neat but not without its drawbacks. We need to `set -e`, otherwise a failing command is going to continue on
# * Rust here is a bit fiddly, so we'll touch the files (even though we copied over them) to force a new build
RUN --mount=type=cache,target=/usr/local/cargo/registry <<EOF
set -e
# update timestamps to force a new build
touch /app/game/src/lib.rs /app/api/src/main.rs
cargo build --release
EOF
CMD ["/app/target/release/my-app"]
# Again, our final image is the same - a slim base and just our app
FROM debian:buster-slim AS app
COPY --from=build /app/target/release/my-app /my-app
CMD ["/my-app"]
And the big test - did it help at all? Let’s see
# We have rust / debian pulled from before
# We need to use BuildKit for these features, so let's turn that on
export DOCKER_BUILDKIT=1
# Build from scratch!
time docker build .
real 5m51.538s
user 0m1.209s
sys 0m0.933s
# The big moment - change a file in src and rebuild
time docker build .
real 0m36.053s
user 0m0.148s
sys 0m0.145s
Great success! Container build times dropped from 5m44s to 0m36s!