Summary: Gitolite (git-rw.torproject.org) and GitWeb (git.torproject.org and https://gitweb.torproject.org) will be fully retired within 9 to 12 months (by the end of Q2 2024). TPA will implement redirections on the web interfaces to maintain limited backwards compatibility for the old URLs. Start migrating your repositories now by following the migration procedure.

• Background
  • Why migrate?
  • Gitolite and GitWeb inventory
• Proposal
  • Migration procedure
  • Redirections
  • Per-repository particularities
    • idle repositories
    • user repositories
    • "mirror" and "extern" repositories
    • Applications team repositories
    • TPA repositories
  • Hooks
  • Timeline
    • T: proposal adopted, voluntary migration encouraged
    • T+6 months: evaluation and idle repositories locked down
    • T+9 months: TPA enforces migration
    • T+12 months: complete Gitolite and GitWeb server retirement
    • T+24 months: Gitolite and GitWeb backups retirement
  • Requirements
• Personas
  • Arthur, the user
  • Barbara, the drive-by contributor
  • Charlie, the old-timer
• Alternatives considered
  • Keeping Gitolite and GitWeb
  • Keeping Gitolite only for problem repositories
  • Concerns about GitLab's security
  • git:// protocol redirections
  • GitWeb to cgit redirections
  • GitLab hosting improvement plans
  • Other forges
• References

Background

We migrated from Trac to GitLab in June 2020. Since then, we have progressively mirrored or migrated repositories from Gitolite to GitLab. Now, after 3 years, it's time to migrate from Gitolite and GitWeb to GitLab as well.

Why migrate?

As a reminder, we migrated from Trac to GitLab because:

• GitLab allowed us to consolidate engineering tools into a single application: Git repository handling, wiki, issue tracking, code reviews, and project management tooling.

• GitLab is well-maintained, while Trac is not as actively maintained; Trac itself hadn't seen a release for over a year (in 2020; there has been a stable release in 2021 and a preview in 2023).

• GitLab enabled us to build a more modern CI platform.

The migration was a resounding success: no one misses Jenkins, for example and people have naturally transitioned to GitLab. It currently host 1,468 projects, including 888 forks, with 76,244 issues, 8,029 merge requests, and 2,686 users (including 325 "Owners," 152 "Maintainers," 18 "Developers," and 15 "Reporters"). GitLab stores a total of 100 GiB of git repositories.

Besides, the migration is currently underway regardless of this proposal but in a disorganized manner. Some repositories have been mirrored, others have been moved, and too many repositories exist on both servers. Locating the canonical copy can be challenging in some cases. There are very few references from Gitolite to GitLab, and virtually no redirection exists between the two. As a result, downstream projects like Debian have missed new releases produced on GitLab for projects that still existed on Gitolite.

Finally, when we launched GitLab, we agreed that:

It is understood that if one of those features gets used more heavily in GitLab, the original service MUST be eventually migrated into GitLab and turned off. We do not want to run multiple similar services at the same time (for example, run both Gitolite and gitaly on all git repositories, or run Jenkins and GitLab runners).

We have been running Gitolite and GitLab in parallel for over three years now, so it's time to move forward.

Gitolite and GitWeb inventory

As of 2023-05-11, there are 566 Git repositories on disk on the Gitolite server (cupani), but oddly only 539 in the Gitolite configuration file. 358 of those repositories are in the user/ namespace, which leaves us 208 "normal" repositories. Out of those, 65 are in the Attic category, which gives us a remaining 143 active repositories on Gitolite.

All the Gitolite repositories take up 32.4GiB of disk space on the Gitolite server, 23.7GiB occupied by user/ repositories, and tor-browser.git taking another 4.2GiB. We suspect Tor Browser and its user forks are using a crushing majority of disk space on the Gitolite server.

The last repository was created in January 2021 (project/web/status-site.git), over two years ago.

Another server (vineale) handles the Git web interface, colloquially called GitWeb (https://gitweb.torproject.org) but which actually runs cgit. That server has a copy of all the repositories on the main Gitolite server, synchronized through Git hooks running over SSH.

For the purposes of this proposal, we put aside the distinction between "GitWeb" and "cgit". So we refer to the "GitWeb" service unless we explicitly need to refer to "cgit" (the software), even though we do not technically run the actual gitweb software anymore.

Proposal

TPA proposes an organized retreat from Gitolite to GitLab, to conclude in Q2 2024. At first, we encourage users to migrate on their own, with TPA assisting by creating redirections from Gitolite to GitLab. In the last stages of the migration (Q1-Q2 2024), TPA will migrate the remaining repositories itself. Then the old Gitolite and GitWeb services will be shutdown and destroyed.

Migration procedure

Owners migrate their repositories using GitLab to import the repositories from Gitolite. TPA then takes over and creates redirections on the Gitolite side, as detailed in the full migration procedure.

Any given repository will have one of three state after the migration:

• migrated: the repository is fully migrated from Gitolite to GitLab, redirections send users to GitLab and the repository is active on GitLab

• archived: like migrated, but "archived" in GitLab, which means the repository hidden in a different tab and immutable

• destroyed: the repository is not worth migrating at all and will be permanently destroyed

Unless requested otherwise in the next 9 months, TPA will migrate all remaining repositories.

As of May 2023, no new repository may be created on Gitolite infrastructure, all new repositories MUST be created on GitLab.

Redirections

For backwards compatibility, web redirections will permanently set in the static mirror system.

This will include a limited set of URLs that GitLab can support in a meaningful way, but some URLs will break. The following cgit URLs notably do not have an equivalence in GitLab:

The supported URLs are:

Redirections also do not include SSH (ssh://) remotes, which will start failing at the end of the migration.

Per-repository particularities

This section documents the fate of some repositories we are aware of. If you can think of specific changes that need to happen to repositories that are unusual, please do report them to TPA so they can be included in this proposal.

idle repositories

Repositories that did not have any new commit in the last two years are considered "idled" and should be migrated or archived to GitLab by their owners. Failing that, TPA will archive the repositories in the GitLab legacy/ namespace before final deadline.

user repositories

There are 358 repositories under the user/ namespace, owned by 70 distinct users.

Those repositories must be migrated to their corresponding user on the GitLab side.

If the Gitolite user does not have a matching user on GitLab, their repositories will be moved under the legacy/gitolite/user/ namespace in GitLab, owned by the GitLab admin doing the migration.

"mirror" and "extern" repositories

Those repositories will be migrated to, and archived in, GitLab within a month of the adoption of this proposal.

Applications team repositories

In December 2022, the applications team announced, and that "all future code updates will only be pushed to our various gitlab.torproject.org (Gitlab) repos."

The following redirections will be deployed shortly:

See tpo/tpa/team#41181 for the ticket tracking this work.

This is a good example of how a team can migrate to GitLab and submit a list of redirections to TPA.

TPA repositories

Note: this section is only relevant to TPA.

TPA is still a heavy user of Gitolite, with most (24) of its repositories still hosted there at the time of writing (2023-05-11).

Many of those repositories have hooks that trigger all sorts of actions on the infrastructure and will need to be converted in GitLab CI actions or similar.

The following repositories are particularly problematic and will need special work to migrate. Here's the list of repositories and their proposed fate.

The most critical repositories are the ones marked security. A solution will be decided on a case-by-case basis. In general, the approach taken will be to pull changes from GitLab (maybe with a webhook to kick the pull) and check the integrity of the repository with OpenPGP signatures as a trust anchor.

Note that TPA also has Git repositories on the Puppet server (tor-puppet.git) and LDAP server (account-keyring.git), but those are not managed by Gitolite and are out of scope for this proposal.

Hooks

There are 11 Git hooks are currently deployed on the Gitolite server.

Timeline

The migration will happen in four stages:

1. now and for the next 6 months: voluntary migration
2. 6 months later: evaluation and idle repositories locked down
3. 9 months later: TPA enforced migration
4. 12 months later: Gitolite and GitWeb server retirements

T: proposal adopted, voluntary migration encouraged

Once this proposal is standard (see the deadline below), Gitolite users are strongly advised to migrate to GitLab, following the migration procedure (#41212, #41219 for TPA repositories, old service retirement 2023 milestone for the others).

Some modification will be done on the gitweb interface to announce its deprecation. Ideally, a warning would also show up in a global pre-receive hook to warn people on push as well (#41211).

T+6 months: evaluation and idle repositories locked down

After 6 months, TPA will evaluate the migration progress and send reminders to users still needing to migrate (#41214).

TPA will lock Gitolite repositories without any changes in the last two years, preventing any further change (#41213).

T+9 months: TPA enforces migration

After 9 months, the migration will be progressively enforced: repositories will be moved or archived to GitLab by TPA itself, with a completion after 12 months (#41215).

Once all repositories are migrated, the redirections will be moved to the static mirror system (#41216).

The retirement procedure for the two hosts (cupani for Gitolite and vineale for GitWeb) will be started which involves shutting down the machines and removing them from monitoring (#41217, #41218). Disks will not be destroyed for three more months.

T+12 months: complete Gitolite and GitWeb server retirement

After 12 months, the Gitolite (cupani) and GitWeb (vineale) servers will be fully retired which implies physical destruction of the disks.

T+24 months: Gitolite and GitWeb backups retirement

Server backups will be destroyed another 12 months later.

Requirements

In July 2022, TPA requested feedback from tor-internal about requirements for the GitLab migration. Out of this, only one hard requirement came out:

• HTTPS-level redirections for .git URLs. For example, https://git.torproject.org/tor.git MUST redirect to https://gitlab.torproject.org/tpo/core/tor.git

Personas

Here we collect some "personas", fictitious characters that try to cover most of the current use cases. The goal is to see how the changes will affect them. If you are not represented by one of those personas, please let us know and describe your use case.

Arthur, the user

Arthur, the average Tor user, will likely not notice any change from this migration.

Arthur rarely interacts with our Git servers: if at all, it would be through some link to a specification hidden deep inside one of our applications documentation or a website. Redirections will ensure those will keep working at least partially.

Barbara, the drive-by contributor

Barbara is a drive-by contributor, who finds and reports bugs in our software or our documentation. Previously, Barbara would sometimes get lost when she would find Git repositories, because it was not clear where or how to contribute to those projects.

Now, if Barbara finds the old Git repositories, she will be redirected to GitLab where she can make awesome contributions, by reporting issues or merge requests in the right projects.

Charlie, the old-timer

Charlie has been around the Tor project since before it was called Tor. He knows by heart proposal numbers and magic redirections like https://spec.torproject.org/.

Charlie will be slightly disappointed because some deep links to line numbers in GitWeb will break. In particular, line number anchors might not work correctly. Charlie is also concerned about the attack surface in GitLab, but will look at the mitigation strategies to see if something might solve that concern.

Otherwise Charlie should be generally unaffected by the change.

Alternatives considered

Those are other alternatives to this proposal that were discussed but rejected in the process.

Keeping Gitolite and GitWeb

One alternative is to keep Gitolite and GitWeb running indefinitely. This has been the de-facto solution for almost three years now.

In a October 2020 tools meeting, it was actually decided to replace Gitolite with GitLab by 2021 or 2022. The alternative of keeping both services running forever is simply not possible as it imposes too much burden on the TPA team while draining valuable resources away from improving GitLab hosting, all the while providing a false sense of security.

That said, we want to extend a warm thank you to the good people who setup and managed those (c)git(web) and Gitolite servers for all that time: thanks!

Keeping Gitolite only for problem repositories

One suggestion is to keep Gitolite for problematic repositories and keep a mirror to avoid having to migrate those to GitLab.

It seems like only TPA is affected by those problems. We're taking it upon ourselves to cleanup this legacy and pivot to a more neutral, less powerful Git hosting system that relies less on custom (and legacy) Git hooks. Instead, we'll design a more standard system based on web hooks or other existing solutions (e.g. Puppet).

Concerns about GitLab's security

During the discussions surrounding the GitLab migration, one of the concerns raised was, in general terms, "how do we protect our code against the larger attack surface of GitLab?

A summary of those discussions that happened in tpo/tpa/gitlab#36 and tpo/tpa/gitlab#81 was posted in the Security Concerns section of our internal Gitolite documentation.

The conclusion of that discussion was:

In the end, it came up to a trade-off: GitLab is much easier to use. Convenience won over hardened security, especially considering the cost of running two services in parallel. Or, as Nick Mathewson put it:

I'm proposing that, since this is an area where the developers would need to shoulder most of the burden, the development teams should be responsible for coming up with solutions that work for them on some reasonable timeframe, and that this shouldn't be admin's problem assuming that the timeframe is long enough.

For now, the result of that discussion is a summary of git repository integrity solutions, which is therefore delegated to teams.

git:// protocol redirections

We do not currently support cloning repositories over the git:// protocol and therefore do not have to worry about redirecting those, thankfully.

GitWeb to cgit redirections

Once a upon a time, the web interface to the Git repositories was running GitWeb. It was, at some point, migrated to cgit, which changed a bunch of URLs and broke many URLs in the process. See this discussion for examples.

Those URLs have been broken for years and will not be fixed in this migration. TPA is not opposed to fixing them, but we find our energy is best spent redirecting currently working URLs to GitLab than already broken ones.

GitLab hosting improvement plans

This proposal explicitly does not cover possible improvements to GitLab hosting.

That said, GitLab will need more resources, both in terms of hardware and staff. The retirement of the old Git infrastructure might provide a little slack for exactly that purpose.

Other forges

There are many other "forges" like GitLab around. We have used Trac in the past (see our Trac documentation) and projects like Gitea or Sourcehut are around as well.

Other than Trac, no serious evaluation of alternative Git forges was performed before we migrated to GitLab in 2020. Now, we feel it's too late to put that put that into question.

Migrating to other forges is therefore considered out of scope as far as Gitolite's retirement is concerned. But TPA doesn't permanently exclude evaluating other solutions than GitLab in the future.

References

This proposal was established in issue tpo/tpa/team#40472 but further discussions should happen in tpo/tpa/team#41180.