Docker Multi Architecture Builds (with Gitlab Runner)

With more and more ARM development boards like the Raspberry Pi reaching the mainstream, with projects like pine64 getting significant drive, with notebook manufacturers experimenting with ARM and with Apple announcing that they would use their own in-house developed ARM chips as their next generation CPUs for the upcoming Macbooks and iMac I guess we can all agree that ARM is going to be big.

For end users this is very good news, better performance, less heat and less energy consumed for the same amount of power output.

For developers, this was never really something the majority had to think about, most of the time it was x86 or AMD64 that we had to keep track of. AMD64 > x86... done deal.

Now there is ARM64... is this a typo? No.

With ARM on the advance, we developers should be thinking about making our software ready for the new era, the ARM64 era.

Thankfully, using Docker already, this is quite easy, it has been possible since, but the Docker guys added an easier way of doing things, which at the moment is still an experimental feature.

Nonetheless, we'd like to introduce buildx, dockers go-to build tool for multi-architecture builds.

Docker Buildx is a CLI plugin that extends the docker command with the full support of the features provided by Moby BuildKit builder toolkit.

Since this is an experimental feature, we'll also later discuss how to use buildxin our build pipeline with docker:dind.

Setting up buildx on our local machine

Docker Buildx is included in Docker 19.03. Note that you must enable the ‘Experimental features’ option to use Docker Buildx.

That's what they state in the official docker docs regarding buildx. If you are running a docker version below 19.03, please upgrade and continue.

On macOS

Using Docker Desktop on a Mac, it should be as simple as that, just click the docker icon in the taskbar, click Preferences, go to Command Line and Enable experimental features.

On Linux (Ubuntu)

On Linux, it's also quite simple, just not as GUIish.

Open up a terminal and execute sudo nano /etc/docker/daemon.json.

Paste in the following content:

{ 
    "experimental": true 
} 

After a restart of the docker daemon using sudo service docker restart, we are ready to rock!

Creating a builder instance

In order to be able to create builds for each platform, we ought to create a builder instance first.

docker buildx create --use

Buildx allows you to create new instances of isolated builders. You can use this to get a scoped environment for your CI builds that does not change the state of the shared daemon, or for isolating builds for different projects.

Also, we can create a new instance for a set of remote nodes, forming a build farm, and quickly switch between them. But this is something we aren't interested anyways at the moment.

A quick docker psshould reveal that our new builder instance is up and running, which means we are only one command away from executing our first docker multi architecture build. Exciting already!

Building multi-architecture docker images

As our builder instance is up and running, we can now build multi-architecture docker images by executing:

docker buildx build --platform linux/amd64,linux/arm64 --push -t PRIVATE_CONTAINER_REPOSITORY_URL/MY_PROJECT/MY_CONTAINER:1.0.0 .

Woohoo! But what is it doing? As you might have guessed already, this command is building docker images for amd64, but at the same time also for arm64.

For convenience, we also added the --push and the -tflag to automatically tag and push the images after they have been built.

Verification & Testing

Since everything went so smoothly, you might be wondering how we can verify our supposedly multi-architecture docker image.

For starters, you can check what docker has to say about our image:

docker buildx imagetools inspect PRIVATE_CONTAINER_REPOSITORY_URL/MY_PROJECT/MY_CONTAINER:1.0.0

If the output tells you about 2 manifests, one for platform linux/amd64 and one for linux/arm64 then things are looking pretty good.

Second test would be, depending on where you pushed your image, to check the platform tags on the UI.

And as the last and most obvious step: Just try to run it on an ARM device, like f.e. a Raspberry Pi (running 64-Bit Ubuntu) or any other piece of hardware already running an ARM CPU and a 64-Bit operating system.

Using buildx with Gitlab Docker Runners

Now that we were able to build multi-architecture images on our local machine, it would be cheesy as fu** to have all this integrated in our build pipeline, and never have to worry about it again.

And... it's possible... but since it's still an experimental feature it's also still a bit clumsy.

See by yourself:

stages:
  - buildx
  - docker

buildx:
  image: docker:git
  stage: buildx
  variables:
    GIT_STRATEGY: none
  artifacts:
    paths:
      - buildx
    expire_in: 1 hour
  services:
    - docker:dind
  script:
    - export DOCKER_BUILDKIT=1
    - git clone git://github.com/docker/buildx ./docker-buildx
    - docker build --platform=local -o . ./docker-buildx
  only:
    - master

containerize:
  image: docker
  services:
    - name: docker:dind
      command: ["--experimental"]
  stage: docker
  before_script:
    - mkdir -p ~/.docker/cli-plugins
    - mv buildx ~/.docker/cli-plugins/docker-buildx
    - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
  script:
    - docker buildx create --use
    - docker buildx build --platform linux/amd64,linux/arm64 --push -t PRIVATE_CONTAINER_REPOSITORY_URL/MY_PROJECT/MY_CONTAINER:1.0.0 .
  only:
    - master

We need to utilize one full build step, the buildxbuild step, to build the buildx docker plugin. Then we pass the built plugin as an artifact on the containerizebuild step.

In this build step, we move the built plugin into the right folder to be utilized by docker. This way our docker cli will know about buildx.

We also need to start the docker:dinddaemon with the --experimentalflag, this way the daemon itself knows about buildx.

Third, we need to run qemu in privileged mode as a way to virtualize our builds.

And only now we can finally execute our two docker buildx create and docker buildx buildstatements to have our multi-architecture build up and running.

As you can see, it ain't as pretty as we'd like it to be, but it works great and is a stable solution for the time of buildx being an experimental feature.

Congratulations!

You made it! You set up your project to be bullet proof for all the next CPU generations to come.

You leveraged the power of buildxand integrated it right into your build pipeline for the joy of everybody in your team working on that project!

Well done!