Buildkit 

The preferred method for building container images is Buildkit since it provides the most caching resulting in the best performance.

When setting up a container build from scratch the follow is preferred (or use Gitlab Component).

container:
  stage: build
  variables:
    CEDARCI_INCREMENTAL: "true"
    CEDARCI_SERVICE_BUILDKIT: "true"
  script:
    - |
      docker buildx build \
        --builder fast \
        --cache-from "type=local,src=.cache" \
        --cache-to "type=local,mode=max,dest=.cache" \
        --tag registry.example.com/foo:bar \
        --push \
        .

Ensure the .dockerignore file excludes the .cache directory.

The combination of the local .cache and CEDARCI_INCREMENTAL from the Intelligent Cache generally provides the best performance. Alternatively, a registry or inline based cache may be used to great effect, especially in combination with a relevant private registry mirror.

When using Buildkit without DinD the docker CLI likes to fall back to the default buildx instance if any docker commands are run outside of buildx build. As such it is best, but not always required, to include --builder fast to ensure the utilization of the optimized builder.

Docker 

Existing Docker-in-Docker builds will have a tuned Buildkit configuration injected. The following .gitlab-ci.yml file demonstrates a container build using the common DinD technique that would be upgraded.

container:
  stage: build
  variables:
    DOCKER_VERSION: 28.1.1
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  services:
    - name: docker:${DOCKER_VERSION}-dind
      alias: docker
      command: ["--tls=false"]
  image: docker:${DOCKER_VERSION}
  script: docker buildx build -t ... --push .

Alternatively, the variable CEDARCI_SERVICE_DOCKER_IN_DOCKER may be set to true to inject the Docker-in-Docker service without additional configuration.

To achieve maximum caching the --cache-{from,to} flags will need to be added as shown in the Buildkit section.

If problems are encountered with the injected configuration it may be disabled by setting CEDARCI_SERVICE_AUTO_DETECT_DISABLE to true.

Builds using the DOCKER_BUILDKIT=1 docker build . style cannot benefit from the tuned configuration and should be adjusted to docker buildx build ..

Gitlab Auto DevOps 

Container builds performed by Gitlab Auto DevOps will automatically be configured to utilize local caching. The existing registry based caching will remain which may reduce performance when compared to only a warm local cache, but is still better than without.

Local caching will be disabled by any of the following:

• CEDARCI_AUTO_DEVOPS_CACHE_DISABLE is set to true
• AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS is configured since it may conflict

Registry Mirror 

Cluster internal container registry mirroring improves the performance and reliability of subsequent uses by eliminating pulls from external registries.

Public Mirror 

Internal container registry mirrors are configured for public images on the following repositories.

• docker.io
• gcr.io
• ghcr.io
• public.ecr.aws
• registry.gitlab.com
• registry.k8s.io

A private mirror configured for the same domain will take precedence over the public mirror.

Private Mirror 

Private internal container registry mirrors may be configured for images requiring authentication. See enhanced config keys:

• registry_mirror_pair
• registry_mirror

Private registry mirrors will automatically be used by Buildkit and for CI job/service images, but not for commands such as docker pull due to technical limitations.

The registry will maintain up to 100Gi of data and will purge images after 5 days of inactivity. Manual flushing may be accomplished by removing the registry configuration and reapplying it.

Workflow 

Instead of the common approach of building everything in a Dockerfile it is generally preferred to build the application outside the container build process and copy the result into the container. Builds on the outside will benefit from the Intelligent Cache and thus be more performant and reliable.

For those already avoiding the build layers ending up in the final image using a multistage build this approach simply moves the first build into the CI job and copies the result.

For example, a multistage build like the following:

FROM toolchain:1.2.3 as build
RUN install_dependencies
RUN compile

FROM scratch
COPY --from=build /app /app

Becomes:

container:
  image: toolchain:1.2.3
  script:
    - install_dependencies
    - compile
    - docker buildx build -t ... --push .

FROM scratch
COPY app /app

Ignore 

The scope of files considered during the build process can be controlled via a .dockerignore file. The less files considered the better the performance.

While irrelevant directories and files can be excluded, it is often easier to maintain an allow list.

*
!vendor/
!src/
!dep.list
!dep.lock

Memory 

All services default to 2Gi of memory unless configured via service_memory. An increase can be applied directly to a service or the whole job using a variable.

container:
  variables:
    KUBERNETES_SERVICE_MEMORY_REQUEST: "4Gi"

Storage 

By default, the Buildkit and Docker services are configured with 15 gigabytes of storage. The value may adjusted using the following variables to a maximum of 40.

• CEDARCI_SERVICE_BUILDKIT_STORAGE
• CEDARCI_SERVICE_DOCKER_IN_DOCKER_STORAGE

container:
  variables:
    CEDARCI_SERVICE_BUILDKIT_STORAGE: "20"

Gitlab Component 

Our Gitlab component can be used to build a container with a local cache backed by the Intelligent Cache.

include:
  - component: $CI_SERVER_FQDN/cedarci/integration/gitlab/container@<VERSION>