Summary: I deployed a new GitLab CI runner backed by Podman instead of Docker, we hope it will improve the stability and our capacity at building images, but I need help testing it.

Background

We've been having stability issues with the Docker runners for a while now. We also started looking again at container image builds, which are currently failing without Kaniko.

Proposal

Testers needed

I need help testing the new runner. Right now it's marked as not running "untagged jobs", so it's unlikely to pick your CI jobs and run them. It would be great if people could test the new runner.

See the GitLab tag documentation for how to add tags to your configuration. It's basically done by adding a tags field to the .gitlab-ci.yml file.

Note that in TPA's ci-test gitlab-ci.yaml file, we use a TPA_TAG_VALUE variable to be able to pass arbitrary tags down into the jobs without having to constantly change the .yaml file, which might be a useful addition to your workflow.

The tag to use is podman.

You can send any job you want to the podman runner, but we'd like to test a broad variety of things before we put it in production, but especially image buildings. Upstream even has a set of instructions to build packages inside podman.

Long term plan

If this goes well, we'd like to converge towards using podman for all workloads. It's better packaged in Debian, and better designed, than Docker. It also allows us to run containers as non-root.

That, however, is not part of this proposal. We're already running Podman for another service (MinIO) but we're not proposing to convert all existing services to podman. If things work well enough for a long enough period (say 30 days), we might turn off the older Docker running instead.

Alternatives considered

To fix the stability issues in Docker, it might be possible to upgrade to the latest upstream package and abandon the packages from Debian.org. We're hoping that will not be necessary thanks to Podman.

To build images, we could create a "privileged" runner. For now, we're hoping Podman will make building container images easier. If we do create a privileged runner, it needs to take into account the long term tiered runner approach.

Deadline