Images Sent to Harbor locally with Size X and via CI/CD Pipeline with Size Y

General
1

After setting up a CI/CD pipeline using GitLab, Harbor, and Rancher, I identified an issue with the deployment of my application where it didn’t function properly.

However, when deploying manually (building the image on my local machine, pushing it manually to Harbor, and applying it to the Rancher deployment), the application works perfectly. One thing I noticed is that the image in question has different sizes when I push it locally versus when it’s built and sent through the pipeline. Through the pipeline, it’s always smaller. This leads me to believe there might be some kind of caching issue.

Pipeline:

variables:
  HARBOR_URL: "harbor.domain"
  PROJECT_NAME: "ciencia_de_dados/cde"
  KUBECONFIG: "${KUBECONFIG}"

stages:
  - prepare
  - build
  - push
  - deploy

prepare_kubeconfig:
  stage: prepare
  image: bitnami/minideb:latest
  tags:
    - docker
    - rancher
  script:
    - apt-get update && apt-get install -y curl
    - echo "$KUBECONFIG" | base64 -d > temp_kubeconfig

.build_and_push_before_script: &build_and_push_before_script
  - echo "$HARBOR_PASSWORD" | docker login -u "$HARBOR_USER" --password-stdin "$HARBOR_URL"

build_image:
  stage: build
  image: docker:latest
  tags:
    - docker
    - rancher
  before_script:
    *build_and_push_before_script
  script:
    - ls -R
    - docker build --no-cache -t "$HARBOR_URL/$PROJECT_NAME:latest" -f Dockerfile .
  only:
    - develop

push_image:
  stage: push
  image: docker:latest
  tags:
    - docker
  before_script:
    *build_and_push_before_script
  script:
    - docker push "$HARBOR_URL/$PROJECT_NAME:latest"
  only:
    - develop

deploy_to_rancher:
  stage: deploy
  image: bitnami/minideb:latest
  tags:
    - rancher
  before_script:
    - apt-get update && apt-get install -y curl
    - curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
    - install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
    - echo "$KUBECONFIG" | base64 -d > temp_kubeconfig
  script:
    - curl -v -k "https://rancherhml.domain:443/k8s/clusters/c-m-cn8gmtcs/version"
    - KUBECONFIG=temp_kubeconfig kubectl create namespace cde --dry-run=client -o yaml | KUBECONFIG=temp_kubeconfig kubectl apply -f -
    - KUBECONFIG=temp_kubeconfig kubectl apply -f deployment.yaml --validate=false
  only:
    - develop

Dockerfile:

FROM python:3.9.19-bullseye
RUN apt-get update && \
    apt-get install -y libaio1 && \
    rm -rf /var/lib/apt/lists/*
WORKDIR /app/cde
COPY requirements.txt .
RUN pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --no-cache-dir --upgrade pip && \
    pip install --trusted-host pypi.org --trusted-host pypi.pythonhosted.org --trusted-host files.pythonhosted.org -r requirements.txt
COPY instantclient_21_12 /opt/oracle/instantclient_21_12
ENV LD_LIBRARY_PATH=/opt/oracle/instantclient_21_12
ENV TNS_ADMIN=/opt/oracle/instantclient_21_12/network/admin
COPY . .
EXPOSE 8000
CMD ["bash", "-c", "python manage.py runserver 0.0.0.0:8000"]

I checked the .dockerignore file to ensure there was nothing in my repository that could cause build failures. I reviewed the Dockerfile and validated its configurations. I added the --no-cache flag to the build image step. I also included a no-cache step in the runner that executes this job. Additionally, I checked for configuration and authentication issues with Harbor.

2

The issue you’re describing, where the image built in your CI/CD pipeline has a smaller size than the one built manually and doesn’t function as expected, is most likely due to missing files or dependencies during the pipeline build process. Let’s systematically address potential causes and solutions:

Potential Causes and Solutions

  1. Files Missing During Build in CI
  • The CI environment might not include all necessary files because of:
    • A misconfigured .dockerignore.
    • Files being ignored in .gitignore (if the build context is uploaded directly from the Git repo).
  • Action: Double-check your .dockerignore and .gitignore to ensure no required files are excluded.
cat .dockerignore
cat .gitignore
  • Log the build context in the pipeline to verify:
ls -laR .  # List all files being used in the build context
  1. Environmental Differences Between Local and CI
  • The docker build in CI uses a clean, isolated environment that might lack dependencies available locally.
  • Action: Log the CI build process to capture any missing packages or errors during the RUN steps in the Dockerfile.
docker build --no-cache --progress=plain -t my-image .
  1. Harbor Authentication or Push Issues
  • If the image is partially built or there are issues during the push step, the resulting image in Harbor may be incomplete.
  • Action:
    • Validate the image layers locally and in Harbor using:
docker image inspect my-image:tag

* Check the logs during the `docker push` step for errors.
  1. Caching in the Docker Build
  • Even with --no-cache, layers may behave differently in CI due to intermediate layers or the way Docker resolves dependencies.
  • Action: Use --pull to ensure the base image (python:3.9.19-bullseye) is the latest version:
docker build --no-cache --pull -t my-image .
  1. Size Differences in the Image
  • Compare the size of the problematic image with the manually built one:
docker images | grep my-image
  • Check which layers differ using:
docker history my-image:tag
  1. Validation of Required Files in the Image
  • After the build, verify the presence of critical files in the image:
docker run --rm -it my-image:tag bash
ls /opt/oracle/instantclient_21_12
ls /app/cde

Suggested Modifications to Your Pipeline

  1. Add File Verification Steps
  • Before building the image, list the files available in the CI workspace:
script:
  - ls -laR .
  - docker build --no-cache -t "$HARBOR_URL/$PROJECT_NAME:latest" -f Dockerfile .
  1. Use Debugging in Docker Build
  • Add --progress=plain to the docker build step to output more detailed logs:
script:
  - docker build --no-cache --progress=plain -t "$HARBOR_URL/$PROJECT_NAME:latest" -f Dockerfile .
  1. Validate Image After Push
  • Add a step to pull and validate the image after pushing to Harbor:
script:
  - docker pull "$HARBOR_URL/$PROJECT_NAME:latest"
  - docker run --rm "$HARBOR_URL/$PROJECT_NAME:latest" ls /app/cde
  1. Check for Large Files
  • Ensure that any large files (e.g., Oracle Instant Client) are present in the build context:
script:
  - du -sh /opt/oracle/instantclient_21_12
  1. Sync Base Images
  • Ensure both local and CI builds use the same base image by explicitly pulling it:
script:
  - docker pull python:3.9.19-bullseye
  - docker build --no-cache -t "$HARBOR_URL/$PROJECT_NAME:latest" -f Dockerfile .

Debugging Next Steps

  1. Build and push the image locally with identical commands to the pipeline:
docker build --no-cache -t harbor.domain/ciencia_de_dados/cde:latest -f Dockerfile .
docker push harbor.domain/ciencia_de_dados/cde:latest
  1. Compare the sizes of local and pipeline-built images:
docker images | grep cde
  1. If the problem persists, inspect the pushed image from Harbor in detail:
docker pull harbor.domain/ciencia_de_dados/cde:latest
docker run -it harbor.domain/ciencia_de_dados/cde:latest bash

By systematically verifying each step and logging detailed outputs, you’ll identify where the pipeline diverges from your local workflow. Let me know how these steps work out or if further assistance is needed!