GitLab Self-Hosted Pipeline for Docker Images

In this blog post, I’ll walk you through the process of setting up a GitLab self-hosted pipeline to build customized or upgraded Docker images. We’ll specifically focus on how to deploy a GitLab Runner and integrate it into your GitLab group projects, allowing you to automate the building and deployment of Docker images based on custom requirements. The example Pipeline code should work for most any Dockerfile that uses a base image to customize, and this base image would be the ‘SOURCE_IMAGE_NAME’ value you store in a Project CI Variable (covered later).

This is a unique how-to as we not only track the base source image version and build new when that changes, but we also track a core added component that has an easy to parse release version that we can track as well. If the latest release changes tag versions we also build a new version and update the current used version in the CI Variable for the Project via a Gitlab API CURL call. This will only work if you have scheduled pipelines and more or less only run 1 pipeline at a time – an updated CI Variable will not affect any currently running Pipeline and in this case, we do not expect or want it to.

Introduction
What You’ll Need
Adding the GitLab Runner to Your Group
Setting Up the GitLab Runner
Configuring the GitLab CI/CD Pipeline
Automating Docker Image Builds with Custom Logic
Updating CI Variables on Successful Build
Conclusion

Introduction

GitLab provides a robust Continuous Integration/Continuous Delivery (CI/CD) platform that allows you to automate your workflows, including building and deploying Docker images. In this guide, we’ll focus on setting up a self-hosted GitLab Runner and configuring it to build customized Docker images for your projects.

Each project will have its own Dockerfile and pipeline configuration (.gitlab-ci.yml). The goal is to check if the base image or any dependent components (like BedrockConnect) have been updated, and if so, rebuild and deploy the new image to your GitLab registry.

What You’ll Need

A self-hosted GitLab instance.
Basic knowledge of GitLab CI/CD concepts.
Docker installed on your system.
Familiarity with shell scripting and YAML configuration.

Adding the GitLab Runner to Your Group

Create a New Runner in GitLab:

Log into your GitLab instance.
Navigate to “Runner” in the sidebar and click “Add runner”.
Configure the runner with the following settings:
- Name: Provide a name for the runner (e.g., docker-builder).
- Tags: Optionally add tags for easier identification.
- Executor: Select the executor type of Shell.
Track, or copy, the string it gives you to ‘register’ your Gitlab Runner

Assign Runner to Your Group:

Under the “Runner” settings, ensure that the runner is assigned to your desired group or project.

Verify Runner Status:

Check the runner status to confirm it’s online and ready to execute jobs. You will only be able to do this once you have deployed the Gitlab Runner using the Token that you generate by adding the Runner to Gitlab

Setting Up the GitLab Runner

Install Docker: Ensure Docker is installed on your system, as it’s required for building images.
Download and Install GitLab Runner:

Download GitLab Runner from https://docs.gitlab.com/runner/install/.
Follow the installation instructions based on your operating system.

Register the Runner with GitLab:

Run the register command that you got when you created the Gitlab Runner in your instance

Start or Enable the Runner:

Follow the directions in the guide linked above to enable and start, and further correctly register your Runner to your Gitlab instance

Configuring the GitLab CI/CD Pipeline

Configure CI Variables in the Project and/or the Group:

If you browse to the Project (or Group) you will find a CI feature on the left side, open it, then open Variables:
Now you can create Variables for use in the Pipeline
In our example we have BUILD_IMAGE_NAME, SOURCE_IMAGE_NAME, EXISTING_JAR_VERSION, GITLAB_RUNNER_API_TOKEN, and REGISTRY_URL as Project CI Variables that store the respective name or value to be used in execution (the GITLAB_RUNNER_API_TOKEN and REGISTRY_URL could likely be stored on the Group CI Variables instead of the Project for better use across many Projects/Pipelines – you could do the same with the REGISTRY_USERNAME and REGISTRY_PASSWORD for Docker Registry login actions if you need to add them to the Pipeline)
The URL we check for the release tag version on latest could also be a Project CI Variable, and I might release and update to this blog entry demonstrating that – just know that if you have a different sub-component, you are going to have to work out how to get the version of it as a var as we have with our URL call and JQ use

Each project will need a .gitlab-ci.yml file in its repository root. Below is an example configuration tailored for building Docker images:

default:
  before_script:
    - docker info

variables:
  CURRENT_HASH: "0000000"
  LATEST_JAR_VERSION: ""
  GROUP_NAME: "mygroup"

stages:
  - check_build_release

check_build_release:
  stage: check_build_release
  when: always
  script:
    - |
      # Start fresh
      docker image prune -f
      docker buildx prune --all --force
      docker builder prune --all --force

      # Extract the digest of the existing image
      CURRENT_HASH=$(docker images --digests sourcehashcheck/$BUILD_IMAGE_NAME | tail -n 1 | awk '{print $4}' | grep -v IMAGE || true)
      echo "Current base image hash: $CURRENT_HASH"

      # Pull the latest base image to get its current hash
      docker pull $SOURCE_IMAGE_NAME

      # Extract the digest of the pulled image
      EXISTING_HASH=$(docker images --digests $SOURCE_IMAGE_NAME | tail -n 1 | awk '{print $4}' | grep -v IMAGE)
      echo "Pulled base image hash: $EXISTING_HASH"

      # Compare hashes to determine if a rebuild is needed
      if [ "$CURRENT_HASH" != "$EXISTING_HASH" ]; then
          echo "Base image has changed. Build required."
          BUILD_REQUIRED=true
      else
          echo "No changes detected to the base image."
          BUILD_REQUIRED=false
      fi

      # Check for BedrockConnect JAR update
      if [ "$EXISTING_JAR_VERSION" == "" ]; then
          # If it's the first time, get the latest version
          LATEST_JAR_VERSION=$(curl -s https://api.github.com/repos/Pugmatt/BedrockConnect/releases | jq -r '.[0].tag_name')
          JAR_UPDATE_REQUIRED=true
      else
          # Get the current latest version to compare
          LATEST_JAR_VERSION=$(curl -s https://api.github.com/repos/Pugmatt/BedrockConnect/releases | jq -r '.[0].tag_name')

          if [ "$EXISTING_JAR_VERSION" != "$LATEST_JAR_VERSION" ]; then
              echo "New BedrockConnect version detected: $LATEST_JAR_VERSION - was $EXISTING_JAR_VERSION"
              JAR_UPDATE_REQUIRED=true
          else
              echo "No changes to BedrockConnect detected."
              JAR_UPDATE_REQUIRED=false
          fi
      fi

      # Set JAR_UPDATE_REQUIRED flag if not already set
      if [ -z "$JAR_UPDATE_REQUIRED" ]; then
          JAR_UPDATE_REQUIRED=false
      fi

      if [[ $BUILD_REQUIRED == "true" || $JAR_UPDATE_REQUIRED == "true" ]]; then
        # Pull the latest base image again to ensure it's up-to-date
        docker pull $SOURCE_IMAGE_NAME

        # Build our customized image using the Dockerfile
        echo "Building '$GROUP_NAME/$BUILD_IMAGE_NAME'..."
        docker build -t $REGISTRY_URL/$GROUP_NAME/$BUILD_IMAGE_NAME .
        RELEASE_REQUIRED=true
      else
        echo "Skipping build, $BUILD_IMAGE_NAME is same version..."
        RELEASE_REQUIRED=false
      fi

      if [[ $RELEASE_REQUIRED == "true" ]]; then
        # Push the new image to the GitLab registry and capture success status
        echo "Attempting to release new version of '$GROUP_NAME/$BUILD_IMAGE_NAME'"
        MAX_RETRIES=3  # Set the maximum number of retries
        PUSH_SUCCESSFUL=false
        for i in {1..3}
        do
            docker login $REGISTRY_URL
            docker push "$REGISTRY_URL/$GROUP_NAME/$BUILD_IMAGE_NAME" && PUSH_SUCCESSFUL=true && break || true
            echo "Docker push failed. Attempt $i of $MAX_RETRIES."
            sleep 10  # Wait for 10 seconds before retrying
        done
        # End fresh
        docker image prune -f
        docker buildx prune --all --force
        docker builder prune --all --force
        # Output status update
        if $PUSH_SUCCESSFUL; then
            echo "Successfully pushed '$GROUP_NAME/$BUILD_IMAGE_NAME' to registry."
            # Tag our Source Hash Check image for a future run
            docker tag $SOURCE_IMAGE_NAME sourcehashcheck/$BUILD_IMAGE_NAME

            # Update the CI variable for this project for a future run
            curl --request PUT --header "PRIVATE-TOKEN: $GITLAB_RUNNER_API_TOKEN" \
                --header "Content-Type: application/json" \
                --data "{ \"value\": \"$LATEST_JAR_VERSION\" }" \
                "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/variables/EXISTING_JAR_VERSION"

            sleep 30 # keep out of cleanup
            exit 0
        else
            echo "Failed to push '$GROUP_NAME/$BUILD_IMAGE_NAME' to registry. Artifact not updated."
            exit 1
        fi
      else
        echo "Skipping release, nothing new built..."
      fi

Automating Docker Image Builds with Custom Logic

The provided YAML configuration includes the following key features:

Base Image Check:

Compares the current base image hash with the latest available hash.
If the hashes differ, it triggers a rebuild.

BedrockConnect JAR Version Update:

Fetches the latest version of BedrockConnect from GitHub using curl and jq.
Checks if the existing version in your project is outdated and updates accordingly.

Build and Deploy:

If either the base image or BedrockConnect requires an update, the pipeline rebuilds the Docker image.
The new image is pushed to the GitLab registry with a retry mechanism for reliability.

Updating CI Variables on Successful Build

After a successful build, the pipeline updates the EXISTING_JAR_VERSION CI variable with the latest version of BedrockConnect. This ensures that future builds only trigger when there are actual changes to either the base image or BedrockConnect.

The update is performed using a curl request to the GitLab API:

curl --request PUT --header "PRIVATE-TOKEN: $GITLAB_RUNNER_API_TOKEN" \
    --header "Content-Type: application/json" \
    --data "{ \"value\": \"$LATEST_JAR_VERSION\" }" \
    "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/variables/EXISTING_JAR_VERSION"

Conclusion

By setting up a GitLab self-hosted pipeline with the provided configuration, you can automate the process of building and deploying customized Docker images. The pipeline checks for updates to your base image and dependent components, rebuilds the image when necessary, and deploys it to your GitLab registry.

This approach ensures that your Docker images are always up-to-date and aligned with the latest changes in your development workflow.

Absolutely Andy, thank you for the notes - I will update as I can to clean up any items mentioned…

Very nice work! Thanks for posting this. I don't use CrowdSec (yet), and both shell scripts complained about no /root/suricata/classification.config…

Posted this on Reddit and I'm discovering that (still need to prove it to myself it works as intended) if…