Background

As more and more Tor projects moved to GitLab and embraced its continuous integration features, managing the ensuing storage requirements has been a challenge.

We regularly deal with near filesystem saturation incidents on the GitLab server, especially involving CI artifact storage, such as tpo/tpa/team#41402 and recently, tpo/tpa/team#41861

Previously, TPA-RFC-14 was implemented to reduce the default artifact retention period from 30 to 14 days. This, and CI optimization of individual projects has provided relief, but the long-term issue has not been definitively addressed since the retention period doesn't apply to some artifacts such as job logs, which are kept indefinitely by default.

Proposal

Implement a daily GitLab maintenance task to delete CI pipelines older than 1 year in all projects hosted on our instance. This will:

Purge old CI pipeline and job records for the GitLab database
Delete associated CI job artifacts, even those "kept" either:
- When manually prevented from expiring ("Keep" button on CI job pages)
- When they're the latest successful pipeline artifact
Delete old CI job log artifacts

Goals

This is expected to significantly reduce the growth rate of CI-related storage usage, and of the GitLab service in general.

Affected users

All users of GitLab CI will be impacted by this change.

But more specifically, some projects have "kept" artifacts, which were manually set not to expire. We'll ensure the concerned users and projects will be notified of this proposal. GitLab's documentation has the instructions to extract this list of non-expiring artifacts.

Timeline

Barring the need to further discussion, this will be implemented on Monday, December 16.

Costs estimates

Hardware

This is expected to reduce future requirements in terms of storage hardware.

Staff

This will reduce the amount of TPA labor needed to deal with filesystem saturation incidents.

Alternatives considered

A "CI housekeeping" script is already in place, which scrubs job logs daily in a hard-coded list of key projects such as c-tor packaging, which runs an elaborate CI pipeline on a daily basis, and triage-bot, which runs it CI pipeline on a schedule, every 15 minutes.

Although it has helped up until now, this approach is not able to deal with the increasing use of personal fork projects which are used for development.

It's possible to define a different retention policy based on a project's namespace. For example, projects under the tpo namespace could have a longer retention period, while others (personal projects) could have a shorter one. This isn't part of the proposal currently as it could violate the principle of least surprise.

References

Discussion ticket: tpo/tpa/team#41874
Make It Ephemeral: Software Should Decay and Lose Data

Keyboard shortcuts