On an air-gapped network, I am setting up a podman gitlab runner which needs to run podman-in-podman to build additional containers I’ll need on the network. I have created a container file to run rootless podman-in-podman on the gitlab runner. I had an admin build the podman-in-podman image for me because they won’t give me permission to run podman on my own (even though I can create gitlab pipelines to run the same thing, but whatever).
The podman runner works great and I can use the podman image just fine to build other images, but I can’t seem to get it to work for building another working podman-in-podman image. I want to be able to build my own podman-in-podman image in my gitlab pipelines so that I don’t have to ask the admins to build it for me every time they bring over an updated UBI 8 base image. The idea is that once a month when they bring over an updated UBI 8 image, my pipeline will build a new podman-in-podman image based on it, do some testing, and push it to the gitlab registry where it will be the image for a bunch of other jobs.
I’m able to build a podman image using the same file I asked the admin to use, and I can run that image too, but as soon as I try to use podman within that image, I get error messages stating that /usr/bin/newuidmap needs the setuid file capability. I believe this error is a common one if you don’t reinstall shadow-utils, but I do. Stranger still, I check the the filecaps for that file right after I build it, and it looks good, but then when I try to run that image, the filecaps are removed.
Here’s a simplified version of my pipeline so far (copied from the air-gapped network so please forgive any typos):
podman:build
image ${CI_REGISTRY_IMAGE}/podman:latest_built_by_IT_team
script:
- podman login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}
- podman build . -f PodmanContainerFile -t ${CI_REGISTRY_IMAGE}/podman:test
- podman push ${CI_REGISTRY)_IMAGE}/podman:test
# Some debug statements. I'm putting the output of each statement in the comment below it
- podman run ${CI_REGISTRY_IMAGE}/podman:test getcap /usr/bin/newuidmap
# /usr/bin/newuidmap cap_setuid=ep
- podman run ${CI_REGISTRY_IMAGE}/podman:test getfattr -d -m '' -- /usr/bin/newuidmap
# # file: usr/bin/newuidmap
# security.capability=0sAQAAAoAAAAAAAAAAAAAAAAAAAAA=
# security.selinux="system_u:object_r:container_file_t:s0"
# getfattr: Removing leading '/' from absolute path names
podman:test
image: ${CI_REGISTRY_IMAGE}/podman:test
needs:
- podman:build
script:
# Note the two debug commands below indicate that the cap_setuid is not set anymore
- getcap /usr/bin/newuidmap
# No output
- getfattr -d -m '' -- /usr/bind/newuidmap
# getfattr: Removing leading '/' from absolute path names
# # file: usr/bin/newuidmap
# security.selinux="system_u:object_r:container_file_t:s0:c6,c945
- podman login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}
# time="some time" level=error msg="running `/usr/bin/newuidmap 33 0 1000 1 1 10000 55536`: newuidmap: write to uid_map failed: Operation not permitted\n"
# Error: cannot set up namespace using "/usr/bin/newuidmap": should have setuid or have filecaps setuid: exit status 1
And here’s the PodmanContainerFile:
FROM <the UBI 8 image that our security team transferred over>
USER 0
# <some stuff to set up repos and certificates and such on our airgapped network ...>
RUN yum -y update && yum install -y podman crun attr && yum reinstall -y shadow-utils
RUN useradd podman && \
echo podman:10000:55536 > /etc/subuid && \
echo podman:10000:55536 > /etc/subgid
COPY --chmod=0644 podman-containers.conf /home/podman/.config/containers/containers.conf
COPY --chmod=0644 containers.conf /etc/containers/containers.conf
RUN mkdir -p /home/podman/.local/share/containers && \
chown podman:podman -R /home/podman
VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers
RUN sed -i -e 's/driver = "overlay"/driver = "vfs"/g' \
-e 's|^#mount_program|mount_program|g' \
-e '/additionalimage.*/a "var/lib/shared",' \
-e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' \
/etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images \
/var/lib/shared/overlay-layers \
/var/lib/shred/vfs-images \
/var/lib/shared/vfs-layers && \
touch /var/lib/shared/overlay-images/images.lock \
/var/lib/shared/overlay-layers/layers.lock \
/var/lib/shared/vfs-images/images.lock \
/var/lib/shared/vfs-layers/layers.lock && \
chmod 0755 /usr/bin/fusermount3
ENV _CONTAINERS_USERNS_CONFIGURED=""
USER 1000
WORKDIR /home/podman
podman-containers.conf is:
[containers]
volumes = [
"/proc:/proc",
]
And containers.conf is:
[containers]
netns="host"
userns="host"
ipcns="host"
utsns="host"
cgroupns="host"
cgroups="diabled"
log_driver="k8s-file"
[engine]
cgroup_manager="cgroupfs"
events_logger="file"
runtime="crun"