Drastically Reduce and Improve the CI/CD Feedback Loop by Going Local with Docker Compose

For this 4th part, I'll cover Docker image building, scanning and publishing.

Prerequisites:

• Read Part 1: Tests & Build

• Read Part 2: Run and Stress Tests

• Read Part 3: Modularizing CI Jobs

I'll first create a Dockerfile to "dockerize" my Go application, then I'll create Docker Compose services to build, test, scan, and publish the multi-architecture Docker image to Docker Hub.

Dockerizing the Go Application

The Dockerfile is simple. I use the multi-stage build technique to first build the Go application in a golang image, then copy the resulting binary into a lightweight alpine image for execution.

FROM --platform=$BUILDPLATFORM golang:1.25.2-alpine AS builder
ARG TARGETOS
ARG TARGETARCH

WORKDIR /app

COPY go.mod ./
RUN go mod download

COPY . .

RUN <<EOF
go mod tidy
GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -o hello .
EOF

FROM alpine:latest
RUN apk --no-cache add ca-certificates wget
WORKDIR /app
COPY --from=builder /app/hello .

✋ Notes:

• TARGETOS and TARGETARCH are build arguments that allow specifying the target operating system and architecture, and are extrapolated from BUILDPLATFORM by Docker Buildx during multi-architecture builds.

• More information on multi-arch build with Docker Buildx here: https://docs.docker.com/build/building/multi-platform/ and on multi-stage build here: https://docs.docker.com/build/building/multi-stage/

Creating the Local Build Service

Let's now create a new file compose.ci.build-local-image.yml:

services:

  # Create a multi architecture image
  build-local-image:
    image: hello-local-ci:${TAG:-demo-from-ci}
    build:
      context: .
      platforms:
        - "linux/amd64"
        - "linux/arm64"
      dockerfile: Dockerfile

  start-container-from-local-image:
    labels:
      local.service: start-container-from-local-image
    image: hello-local-ci:${TAG:-demo-from-ci}
    command: ["./hello"]
    depends_on:
      build-local-image:
        condition: service_completed_successfully

  stop-container-from-local-image:
    image: docker:cli
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command:
      - /bin/sh
      - -c
      - |
        docker stop $(docker ps -q --filter "label=local.service=start-container-from-local-image")
    depends_on:
      start-container-from-local-image:
        condition: service_started

This file contains three services:

• build-local-image: This service uses the Dockerfile to build a multi-architecture Docker image named hello-local-ci.

• start-container-from-local-image: This service starts a container from the built Docker image and executes the Go application.

• stop-container-from-local-image: This service stops the container started by the previous service.

Without having to restart the entire CI/CD pipeline, you can now test the build and execution of the Docker image locally with the following command:

docker compose -f compose.ci.build-local-image.yml up --build

Then if you open Docker Desktop and go to the "Builds" section, you can verify that the image has been built for 2 architectures (linux/amd64 and linux/arm64):

And now to add these services to the CI/CD pipeline, add this section at the end of the main compose.ci.yml file:

 # Create a multi architecture image
  build-local-image:
    extends:
      file: compose.ci.build-local-image.yml
      service: build-local-image

    depends_on:
      stop-web-service:
        condition: service_completed_successfully

  start-container-from-local-image:
    extends:
      file: compose.ci.build-local-image.yml
      service: start-container-from-local-image
    # dependencies are defined in the extended file
  stop-container-from-local-image:
    extends:
      file: compose.ci.build-local-image.yml
      service: stop-container-from-local-image
    # dependencies are defined in the extended file

Now that I have a local image, I'd like to scan it for potential vulnerabilities before publishing it to Docker Hub. I'll use Docker Scout for this.

Scanning the Docker Image with Docker Scout

To work, Docker Scout requires authentication with Docker Hub. I'll therefore add two new environment variables to my .env file:

DOCKER_HUB_USERNAME="your_docker_hub_username"
DOCKER_HUB_PAT="your_docker_hub_personal_access_token"

How to create a Docker Hub Personal Access Token (PAT): https://docs.docker.com/security/access-tokens/

I'll now create a new file compose.ci.image-analysis.yml for the scan service:

services:
  # Scan the image for vulnerabilities
  image-analysis:
    # Use Docker Scout CLI image
    image: docker/scout-cli
    user: root  # Required to access Docker socket
    environment:
      # Your Docker Hub username
      DOCKER_SCOUT_HUB_USER: ${DOCKER_HUB_USERNAME}
      # Your Docker Hub Personal Access Token (PAT)
      DOCKER_SCOUT_HUB_PASSWORD: ${DOCKER_HUB_PAT}

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock  # Mount Docker socket
      - ./reports:/reports
    command:
      - cves
      - hello-local-ci:${TAG:-demo-from-ci}
      - --output
      - /reports/cves.report.text

✋ Note: If vulnerabilities are detected and you want the service to fail, you can add the --exit-code option to the command:

    command:
      - cves
      - --exit-code
      - hello-local-ci:${TAG:-demo-from-ci}
      - --output
      - /reports/cves.report.text

And now, I add this service once again at the end of the main compose.ci.yml file:

  # Scan the image for vulnerabilities
  image-analysis:
    extends:
      file: compose.ci.image-analysis.yml
      service: image-analysis

    depends_on:
      stop-container-from-local-image:
        condition: service_completed_successfully

If you now run the complete CI pipeline, you'll get a vulnerability scan report in the ./reports/cves.report.text folder:

Excerpt:

## Overview

                    │                Analyzed Image
────────────────────┼────────────────────────────────────────────────
  Target            │  hello-local-ci:demo-from-ci
    digest          │  6543f9770f62
    platform        │ linux/arm64
    provenance      │ git@github.com:Triton-CI/hello-compose-ci.git
                    │  https://github.com/Triton-CI/hello-compose-ci/blob/b55f7c869434d004de70ff2ae7feaa348e8ed017
    vulnerabilities │    0C     1H     1M     2L
    size            │ 10 MB
    packages        │ 26

... (truncated) ...

4 vulnerabilities found in 3 packages
  CRITICAL  0
  HIGH      1
  MEDIUM    1
  LOW       2

Note: Within the scope of this article series, we won't cover vulnerability remediation actions, but Docker Scout provides suggestions for fixing found vulnerabilities.

Let's assume everything is good and I now want to publish the image to Docker Hub.

Publishing the Docker Image to Docker Hub

Once I've determined that the image is ready to be published, I want to rebuild it in multi-architecture mode, tag it and push it to Docker Hub. For this I'll use docker buildx which allows building and pushing multi-architecture images in a single command.

More info on Docker Buildx: https://docs.docker.com/reference/cli/docker/buildx/

I'll therefore create yet another new file compose.ci.build-publish-image.yml for the publishing service:

services:
  build-publish-image:
    image: docker:cli
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./:/workspace
    working_dir: /workspace
    # allow to access the Dockerfile in the current directory

    command:
      - /bin/sh
      - -c
      - |
        docker login -u ${DOCKER_HUB_USERNAME} -p ${DOCKER_HUB_PAT} &&
        docker buildx build \
        --platform linux/amd64,linux/arm64 \
        --push -t ${DOCKER_HUB_USERNAME}/hello-local-ci:${TAG:-demo-from-ci} .

And finally, I add this service at the end of the main compose.ci.yml file:

  build-publish-image:
    profiles: [ publish-image ]
    extends:
      file: compose.ci.build-publish-image.yml
      service: build-publish-image
    depends_on:
      image-analysis:
        condition: service_completed_successfully

✋ Important note: notice the profiles: [ publish-image ] line. With this configuration, the build-publish-image service will only run if the publish-image profile is activated when running Docker Compose. This allows you to control when you want to publish the image.

To run the complete pipeline with image publishing, use the following command:

docker compose -f compose.ci.yml --profile publish-image up --build

And you can then verify on Docker Hub that the image has been published for both architectures:

Conclusion

Our pipeline is starting to be really complete with builds, tests, scans and publishing of multi-architecture Docker images.

CI Pipeline DAG

Pipeline Stages:

1. unit-tests - Run unit tests

2. multi-arch-build-application - Build multi-architecture application

3. start-service-for-local-testing - Deploy the service locally for testing

4. test-home-endpoint - Test the endpoint

5. stress-test - Perform stress testing on the endpoint

6. stop-web-service - Stop the web service after testing

7. build-local-image - Create multi-architecture Docker image

8. start-container-from-local-image - Start container from local image

9. stop-container-from-local-image - Stop the container

10. image-analysis - Analyze the Docker image

11. build-publish-image - Build and publish image (profile: publish-image, optional)

Being able to run all this locally with Docker Compose drastically reduces the feedback loop and improves developer productivity.

You can find the complete code for this project on GitHub, right here: https://github.com/Triton-CI/hello-compose-ci/tree/v0.0.4

In the next episode, we'll see how to add some useful AI to our pipeline with small models to generate automatic reports. Stay tuned!