Significantly Reduce and Improve the CI/CD Feedback Loop by Going Local with Docker Compose

Prerequisites: Read Part 1: Tests & Build

For this second part of the series on how to "do your CI locally" with Docker Compose, I'll add the following steps (jobs) to our CI/CD pipeline:

• Start the web service (only if the application build succeeded)

• Launch a first HTTP connection to verify that the web service is properly started (only if the web service startup succeeded)

• Execute stress tests against the running web service and generate a report (if the first HTTP connection succeeded)

• Stop the web service (once the stress tests are completed)

👋 Before we start, I've added a healthcheck handler to the service that I'll be able to use in the CI to verify that the web service is properly started and ready to receive HTTP requests:

func healthCheckHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")

    w.WriteHeader(http.StatusOK)
    response := map[string]any{
        "status": "healthy",
    }
    json.NewEncoder(w).Encode(response)
}

func main() {
    http.HandleFunc("/", rootHandler)
    http.HandleFunc("/text", textHandler)
    http.HandleFunc("/html", htmlHandler)
    http.HandleFunc("/json", jsonHandler)
    http.HandleFunc("/health", healthCheckHandler)

    log.Println("Server starting on :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

And compared to the first version of the pipeline, I've replaced the build-application job with a multi-architecture build job multi-arch-build-application (already presented in the previous blog post):

  multi-arch-build-application:
    image: golang:1.25.2-alpine
    environment:
      <<: *common-variables
    working_dir: /project
    command:
      - /bin/sh
      - -c
      - |
        echo "🏗️ Building application for $${PROJECT_NAME}"
        go mod download

        for OS_ARCH in "linux/amd64" "linux/arm64" "darwin/amd64" "darwin/arm64"; do
          OS=$${OS_ARCH%/*}
          ARCH=$${OS_ARCH#*/}
          echo "Building for $$OS/$$ARCH..."
          CGO_ENABLED=0 GOOS=$$OS GOARCH=$$ARCH go build \
              -ldflags="-s -w" \
              -o ./builds/hello-$$OS-$$ARCH main.go
          if [ $? -ne 0 ]; then
            echo "❌ Build failed for $$OS/$$ARCH"
            exit 1
          fi
        done

        echo "✅ All builds succeeded"
        echo "📦 Artifacts generated at ./builds"
    volumes:
      - ./:/project
    depends_on:
      unit-tests:
        condition: service_completed_successfully

It's time to start our web service!

CI Job to Start the Web Service

Here's the code for the start-service-for-local-testing service to add to the compose.ci.yml file:

  start-service-for-local-testing:
    labels:
      com.docker.compose.service: start-service-for-local-testing
    image: ubuntu:22.04
    environment:
      TARGETOS: linux
      TARGETARCH: arm64
    volumes:
      - ./builds:/builds
    command:
      - /bin/sh
      - -c
      - |
        apt-get update && apt-get install -y wget
        echo "🚀 Starting the service locally on port 8080"
        chmod +x ./builds/hello-$${TARGETOS}-$${TARGETARCH}
        ./builds/hello-$${TARGETOS}-$${TARGETARCH}

    depends_on:
      multi-arch-build-application:
        condition: service_completed_successfully

    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/health"]
      interval: 30s
      timeout: 30s
      retries: 5
      start_period: 40s

Explanation of the start-service-for-local-testing job:

This service is responsible for starting our compiled application (from the previous job) in a local test environment:

1. Base image: Uses Ubuntu 22.04 as the operating system (👋 you should of course choose an image suited to your needs, lighter for example, but similar to production conditions)

2. 👋 Label: Adds a label local.service: start-service-for-local-testing to easily identify this container (useful for stopping it later)

3. Environment variables: Defines TARGETOS=linux and TARGETARCH=arm64 to select the right binary

4. Volume: Mounts the ./builds folder to access the compiled executables

5. Execution command:

   • Installs wget (necessary for the healthcheck)

   • Makes the executable executable with chmod +x

   • Launches the hello-linux-arm64 application which listens on port 8080

6. Dependency: Waits for multi-arch-build-application to complete successfully before starting

7. Healthcheck: Automatically verifies the service health:

   • Tests the /health endpoint every 30 seconds

   • Waits 40 seconds before the first check (grace period after startup before failures are counted)

   • Makes up to 5 attempts in case of failure

   • The service is considered "healthy" once the endpoint responds correctly

Once the service is "healthy", the other pipeline services (HTTP tests, stress tests) can execute.

CI Job to Test a First HTTP Connection

This job is very simple, it uses the official curlimages/curl image to make an HTTP GET request to the root endpoint of our web service:

  test-home-endpoint:
    image: curlimages/curl:latest
    command: >
      curl --fail http://start-service-for-local-testing:8080

    depends_on:
      start-service-for-local-testing:
        condition: service_healthy

Note 1: I'm using Docker Compose's internal DNS system to target the start-service-for-local-testing service by its service name.

Note 2: The --fail option in curl makes the command fail if the returned HTTP code is an error.

Note 3: This job depends on the start-service-for-local-testing service and will only execute the HTTP request if the service is "healthy" (hence the health check system from the previous job).

CI Job to Execute Stress Tests

I really like the hey tool for simple and effective stress testing. Here's the stress-test job that uses hey to send 1000 concurrent HTTP requests to our web service:

  stress-test:
    image: ubuntu:22.04
    command:
      - /bin/bash
      - -c
      - |
        apt-get update
        apt-get -y install hey
        hey -n 1000 -c 10 -m GET \
        http://start-service-for-local-testing:8080/json > \
        /reports/hey.report.$$(date +%Y%m%d_%H%M%S).txt

    volumes:
      - ./reports:/reports
    depends_on:
      test-home-endpoint:
        condition: service_completed_successfully

Explanation of the stress-test job:

This service performs a load test:

1. Tool installation: Installs hey, a lightweight HTTP benchmarking tool.

2. Test execution: Launches a stress test with the following parameters:

   • -n 1000: Sends 1000 requests in total

   • -c 10: Uses 10 simultaneous connections (concurrency)

   • -m GET: Uses the HTTP GET method

   • Targets the /json endpoint of the start-service-for-local-testing service on port 8080

3. Report generation: Saves the results in a file with a unique timestamp (e.g., hey.report.20251105_174122.txt)

4. Volume: Mounts the ./reports folder to persist test reports

5. Dependency: Waits for test-home-endpoint to complete successfully before launching the stress test

CI Job to Stop the Web Service

Once the stress test is complete, it's important to stop the web service to finish the pipeline. Here's the stop-web-service job:

  stop-web-service:
    image: docker:cli
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command:
      - /bin/sh
      - -c
      - |
        echo '🚦 Stopping container service...'
        docker stop $(docker ps -q --filter "label=local.service=start-service-for-local-testing")

    depends_on:
      stress-test:
        condition: service_completed_successfully

Explanation of the stop-web-service job:

1. Base image: Uses docker:cli which contains the Docker client

2. Docker socket volume: Mounts /var/run/docker.sock to allow the container to communicate with the host's Docker daemon

3. Stop command:

   • Displays a message indicating the service is stopping

   • Uses docker ps -q with a filter on the label local.service=start-service-for-local-testing to find the container ID to stop

   • Executes docker stop on this container

4. Dependency: Waits for stress-test to complete successfully before stopping the service

This job ensures that the web service remains active during all tests and is properly stopped at the end of the pipeline.

What Does Our Complete CI/CD Pipeline Look Like?

Here's the complete diagram of the CI/CD pipeline based on the compose.ci.yml file:

And now, all we have to do is run our CI/CD pipeline locally with Docker Compose!

docker compose -f compose.ci.yml up

And we can follow the progress of the different jobs in Docker Desktop with different views (resource consumption, logs, etc.):

A Mandatory Best Practice

Every time you launch a CI pipeline, you must start from scratch to avoid any side effects from artifacts or containers left behind by a previous execution.

So make sure you've stopped everything cleanly before restarting with the following command:

docker compose -f compose.ci.yml down

Useful options:

• docker compose -f compose.ci.yml down -v : Also removes volumes

• docker compose -f compose.ci.yml down --rmi all : Also removes images

• docker compose -f compose.ci.yml down --remove-orphans : Removes orphaned containers

Conclusion

We've successfully added service startup, initial HTTP testing, load testing, and service shutdown steps to our Docker Compose-based local CI/CD pipeline. These steps are clearly reproducible on demand (which is the essence of CI), allowing you to run them as many times as necessary to validate your code changes quickly and ultimately feel confident before pushing your changes to the remote repository.

In the next article in this series, we'll see how to organize our CI pipeline to make it easier to read and maintain.

Then, the following article will cover Docker image creation, vulnerability scanning, and publishing to Docker Hub, all while using Docker Compose to orchestrate everything locally. Stay tuned!

👋 You can find the complete code for this part of the series on GitHub: https://github.com/Triton-CI/hello-compose-ci/tree/v0.0.2