How do I set memory swappiness for job containers when using the Docker executor for a Gitlab runner?

General
1

I have an AWS VM running a Docker executor Gitlab runner. The VM’s OS is Amazon Linux 2023. I am having issues with some of my Gitlab jobs running out of memory and want to configure memory swapping to mitigate this. I have memory swappiness setup in my config.toml file, but I when I run a docker inspect command on the job container, I see the memory swappiness is set to null. Why are my job containers not using memory swappiness?

config.toml:

[[runners]]
  name = "aws-runner-1"
  url = "https://gitlab.com/"
  id = <id>
  token = "<runner-token>"
  token_obtained_at = 2024-09-12T13:24:37Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:3.20"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    memory = "1792m"
    memory_swap = "4096m"
    memory_swappiness = 60
    memory_reservation = "256m"
    cpus = "0.9"
    cpuset_cpus = "1"
    shm_size = 0
    network_mtu = 0
2

To configure memory swappiness for job containers when using the Docker executor with a GitLab Runner, you’ll need to set the appropriate Docker runtime options in your config.toml file. However, the memory swappiness setting is not directly supported as a Docker option, and it may not be effective if set incorrectly or not at all.

Here’s a step-by-step guide to help you set up memory swappiness for your Docker containers:

1. Set Up Swappiness on the Host

First, ensure that the swappiness setting is configured on your AWS VM itself. You can check the current swappiness value by running:

cat /proc/sys/vm/swappiness

To temporarily set the swappiness value (for example, to 10):

sudo sysctl vm.swappiness=10

To make this change permanent, add or modify the following line in your /etc/sysctl.conf:

vm.swappiness = 10

Then, apply the changes:

sudo sysctl -p

2. Configure Docker Resource Limits

Since you are using the Docker executor, you can specify memory limits for the containers in the config.toml. While swappiness is not directly set in Docker, you can limit memory to control when swapping occurs.

Here’s how to set it:

[runners.docker]
  ...
  memory = "2g" # Adjust based on your needs
  memory_swap = "3g" # This should be greater than memory, which allows for swapping

3. Use a Custom Entry Point or Command

If your application is sensitive to memory pressure, consider running your jobs with a custom entry point that adjusts the swappiness value within the container. This can be done by creating a small script that sets swappiness inside the container.

Create a script (e.g., set_swappiness.sh):

#!/bin/sh
echo 10 > /proc/sys/vm/swappiness # Adjust the value as needed
exec "$@"

Make it executable:

chmod +x set_swappiness.sh

Then update your config.toml to use this script:

[runners.docker]
  ...
  entrypoint = ["/path/to/set_swappiness.sh"]

4. Restart the GitLab Runner

After making changes to your config.toml, restart the GitLab Runner to apply the new configuration:

sudo gitlab-runner restart

5. Validate the Configuration

Run a job and inspect the Docker container to ensure the swappiness setting is being applied. You can check the swappiness value inside the container with:

docker exec -it <container_id> cat /proc/sys/vm/swappiness