Drastically Reduce and Improve the CI/CD Feedback Loop by Going Local with Docker Compose

This Part 3 of the series focuses on modularizing the different CI jobs by using separate Docker Compose files for each step of the CI/CD pipeline. This allows for better organization, reusability, and maintainability of CI configurations.

Prerequisites:

• Read Part 1: Tests & Build
• Read Part 2: Run and Stress Tests

Let's Start with the First Two Services: unit-tests and multi-arch-build-application

We'll move the unit-tests and multi-arch-build-application services into their own files compose.ci.unit-tests.yml and compose.ci.multi-arch-build-application.yml thanks to Docker Compose's extends functionality.

✋ You should know that we have some constraints with this approach:

The extensions principle I used to have common environment variables will no longer work, or we'd need to duplicate the x-common-variables section in each compose.ci.*.yml file, which doesn't make much sense. Indeed, extensions cannot be shared between multiple Compose files.

Nevertheless, by using the .env file, Docker Compose will automatically load the environment variables defined in this file and make them available for all Compose files used. Thus, you can define PROJECT_NAME and PROJECT_DESCRIPTION in the .env file and use them in all your Compose files without having to duplicate the common variables section:

# .env
PROJECT_NAME: "hello-compose-ci"
PROJECT_DESCRIPTION: "A simple web service application"

And we won't need the double dollar sign $$ anymore to reference environment variables in Compose files, a simple $ will suffice.

👋 Good to Know: you can specify a different .env file per compose service using the env-file key in the service definition: https://docs.docker.com/compose/how-tos/environment-variables/set-environment-variables/#use-the-env_file-attribute

Modularization of the Two CI Services

Here are the modifications made:

Creating compose.ci.unit-tests.yml with the following content (the same as before but with adjustments for environment variables):

services:
  # Run unit tests
  unit-tests:
    image: golang:1.25.2-alpine

    working_dir: /project
    command:
      - /bin/sh
      - -c
      - |
        echo "🧪 Running unit tests for ${PROJECT_NAME}"
        echo "📝 Description: ${PROJECT_DESCRIPTION}"

        go mod download
        if go test -v > ./reports/test-output.txt; then
          echo "✅ Tests completed"
          echo "📄 Test report generated at ./reports/test-output.txt"
          go test -cover > ./reports/test-coverage.txt
          exit 0
        else
          echo "❌ Tests failed"
          exit 1
        fi

    volumes:
      - ./:/project

Then, we create the following file for the multi-architecture build service.

Creating compose.ci.multi-arch-build-application.yml (the same as before but with adjustments for environment variables):

services:
  multi-arch-build-application:
    image: golang:1.25.2-alpine
    working_dir: /project
    command:
      - /bin/sh
      - -c
      - |
        echo "🏗️ Building application for ${PROJECT_NAME}"
        go mod download

        for OS_ARCH in "linux/amd64" "linux/arm64" "darwin/amd64" "darwin/arm64"; do
          OS=$${OS_ARCH%/*}
          ARCH=$${OS_ARCH#*/}
          echo "Building for $$OS/$$ARCH..."
          CGO_ENABLED=0 GOOS=$$OS GOARCH=$$ARCH go build \
              -ldflags="-s -w" \
              -o ./builds/hello-$$OS-$$ARCH main.go
          if [ $? -ne 0 ]; then
            echo "❌ Build failed for $$OS/$$ARCH"
            exit 1
          fi
        done

        echo "✅ All builds succeeded"
        echo "📦 Artifacts generated at ./builds"
    volumes:
      - ./:/project

✋ Note: You'll notice that I didn't keep the dependency notion between the unit-tests and multi-arch-build-application services. I'll specify it in the main compose.ci.yml file.

Updating compose.ci.yml

For these 2 services, we'll now use Docker Compose's extends functionality to include them in the main compose.ci.yml file as follows:

services:
  # Run unit tests
  unit-tests:
    extends:
      file: compose.ci.unit-tests.yml
      service: unit-tests

  multi-arch-build-application:
    extends:
      file: compose.ci.multi-arch-build-application.yml
      service: multi-arch-build-application

    depends_on:
      unit-tests:
        condition: service_completed_successfully

With this method, we simplify the reading of the CI pipeline while preserving the jobs' execution logic (dependencies between services).

Modularizing Other CI Services

I used the same approach for the other CI services, like start-service-for-local-testing, test-home-endpoint, stress-test, and stop-web-service. Each service can be moved to its own Compose file, then included in the main compose.ci.yml file using extends.

The final result will be as follows:

# Compose CI pipeline
x-documentation: >
  Project: hello-compose-ci
  Start the CI pipeline:
  docker compose -f compose.ci.yml up --build

services:
  # Run unit tests
  unit-tests:
    extends:
      file: compose.ci.unit-tests.yml
      service: unit-tests

  multi-arch-build-application:
    extends:
      file: compose.ci.multi-arch-build-application.yml
      service: multi-arch-build-application

    depends_on:
      unit-tests:
        condition: service_completed_successfully

  # Deploy the function (locally) for testing
  start-service-for-local-testing:
    extends:
      file: compose.ci.start-service-for-local-testing.yml
      service: start-service-for-local-testing

    depends_on:
      multi-arch-build-application:
        condition: service_completed_successfully

  # Test the endpoint
  test-home-endpoint:
    extends:
      file: compose.ci.test-home-endpoint.yml
      service: test-home-endpoint

    depends_on:
      start-service-for-local-testing:
        condition: service_healthy

  # Stress test the endpoint
  stress-test:
    extends:
      file: compose.ci.stress-test.yml
      service: stress-test
    depends_on:
      test-home-endpoint:
        condition: service_completed_successfully

  # Stop the function after testing
  stop-web-service:
    extends:
      file: compose.ci.stop-web-service.yml
      service: stop-web-service

    depends_on:
      stress-test:
        condition: service_completed_successfully

Regarding the details of other components, you can see the complete source codes of other jobs in the GitHub repository: https://github.com/Triton-CI/hello-compose-ci/tree/v0.0.3:

• compose.ci.start-service-for-local-testing.yml
• compose.ci.test-home-endpoint.yml
• compose.ci.stress-test.yml
• compose.ci.stop-web-service.yml

Conclusion

By modularizing the different CI jobs (Compose services) into separate Docker Compose files, we've improved the organization and maintainability of our local CI/CD pipeline. Each step of the pipeline is now clearly defined in its own file, which makes future modifications and additions easier.

Regarding the reusability of configurations, there's another Docker Compose feature called include that allows referencing remote Compose files. This is useful for sharing/reusing "job components" across multiple projects.

This concludes Part 3 of this series. In the upcoming parts, we'll cover the following topics (normally in this order):

• Part 4: Creating a Multi-architecture Docker Image and Publishing on Docker Hub.
• Part 5: Analyzing CVE Vulnerabilities in Docker Images.
• Part 6: Using AI Agents to Generate Reports.
• Part 7: Deployment.