Summary: migrate all Git storage to the new gitaly-01 back-end, each
Git repository read-only during its migration, in the coming week.
Proposal
Move all Git repositories to the new Gitaly server during Week 29, progressively, which means it will be impossible to push new commits to a repository while it is migrated.
This should be a series of short (seconds to minutes), scoped outage, as each repository is marked as read-only one at a time when it's migrated, see "impact" below on what that means more precisely.
The Gitaly migration procedure seems well test and robust, as each repository is checkedsummed before and after migration.
We are hoping this will improve overall performance on the GitLab server, and is part of the design upstream GitLab suggests in scaling an installation of our size.
Affected projects
We plan on migrating the following name spaces in order:
alpha phase, day one (2025-07-14)
This is mostly dogfooding and automation:
anarcat(already done)tpo/tpatpo/web
beta phase, day two (2025-07-15)
This is to include testers outside of TPA yet on projects that are less mission critical and could survive some issues with their Git repositories.
tpo/communitytpo/onion-servicestpo/anti-censorshiptpo/network-health
production phase, day two or three (2025-07-15+)
This is essentially all remaining projects:
tpo/core(includes c-tor and Arti!)tpo/applications(includes Tor Browser and Mullvad Browser)- all remaining projects
Objections and exceptions
If you do not want any such disruption in your project, please let us know before the deadline (2025-07-15) so we can skip your project. But we would rather migrate all projects off of the server to simplify the architecture and better understand the impact of the change.
We would like, in particular, to migrate all of tpo/applications
repositories in the coming week.
Inversely, if you want your project to be prioritized (it might mean a performance improvement!), let us know and you can jump the queue!
Impact
Projects read-only during migration
While a project is migrated, it is "read-only", that is no change can be done to the Git repository.
We believe that other features in projects (like issues and comments) should still work, but the upstream documentation on this is not exactly clear:
To ensure data integrity, projects are put in a temporary read-only state for the duration of the move. During this time, users receive a The repository is temporarily read-only. Please try again later. message if they try to push new commits.
So far our test migrations have been so fast (a couple of seconds per project) that we have not really been able to test this properly.
Effectively, we don't expect users to actually notice this migration. In our tests, a 120MB repository was migrated in a couple of seconds, so apart from very large repositories, most read-only situations should be limited to less than a minute.
It is estimated that our largest repositories (the Firefox forks) will take a 5 to 10 minutes to migrate, and that the entire migration will take, in total, less than 2 hours to shift between the two servers if it would performed in one shot.
Additional complexity for TPA
TPA will need to get familiar with this new service. Installation documentation is available and all the code developed to deploy the service is visible in an internal merge request.
I understand this is a big change right before going on vacation, so any TPA member can veto this and switch to the alternative, a partial or on-demand migration.
Timeline
We plan on starting this work on July 15th, the coming Tuesday.
Hardware
Like the current git repositories on gitlab-02 the git repositories
on gitaly-01 will be hosted on NVMe disks.
Background
GitLab has been having performance problems for a long time now. And for almost as long, we've had the project to "scale GitLab to 2,000 users" (tpo/tpa/team#40479). And while we believe bots (and now, in particular Large Language Models (LLM) bot nets) are responsible for a lot of that load, our last performance incident concluded by observing that there seems to be a correlation between real usage and performance issues.
Indeed, during the July break, GitLab's performance was stellar and, on Monday, as soon as Europe woke up from the break, GitLab's performance collapsed again. And while it's possible that bots are driven by the same schedule as Tor people, we now feel it's simply time to scale the resources associated with one of our most important services.
Gitaly is GitLab's implementation of a Git server. It's basically a web interface to translate (GRPC) requests into Git. It's currently running on the same server as the main GitLab app, but a new server has been built. New servers could be built as needed as well.
Anarcat performed benchmarks showing equivalent or better performance of the new Gitaly server, even when influenced by the load of the current GitLab server. It is expected the new server should reduce the load on the main GitLab server, but it's not clear by how much just yet.
We're hoping this new architecture will give us more flexibility to deploy new such backends in the future and isolate performance issues to improve diagnostics. It's part of the normal roadmap in scaling a large GitLab installation such as ours.
Alternatives considered
Full read-only backups
We have considered performing a full backup of the entire git repositories before the migration. Unfortunately, this would require setting a read-only mode on all of GitLab for the duration of the backup which, according to our test, could take anywhere from 20 to 60 minutes, which seemed like an unacceptable downtime.
Note that we have nightly backups of the GitLab server of course, which is also backed by RAID-10 disk arrays on two different servers. We're only talking about a fully-consistent Git backup here, our normal backups (which, rarely, can be inconsistent and require manual work to reconnect some refs) are typically sufficient anyways. See tpo/tpa/team#40518 for a discussion on GitLab backups.
Partial or on-demand migration
We have also considered doing a more piecemeal approach and just migrating some repositories. We worry that this approach would lead to confusion about the real impact of the migration.
Still, if any TPA member feels strongly enough about this to put a veto on this proposal, we can take this path and instead migrate a few repositories instead.
We could, for example, migrate only the "alpha" targets and a few key
repositories in the tpo/applications and tpo/core groups (since
they're prime crawler targets), and leave the mass migration to a
later time, with a longer test period.
References and discussions
See the discussion issue for comments and more background.