Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Summary: implement a mechanism to enforce signed commits verification and switch to GitLab as a canonical source, for Puppet repositories

Background

Checking the authenticity of Git commits has been considered before in the context of the switch from Gitolite to GitLab, when the attack surface of Tor's Git repositories increased significantly (1, 2). With the upcoming merge of Tor and Tails Puppet codebases and servers, allowing for the verification of commit signatures becomes more important, as the Tails backup server relies on that to resist potential compromise of the Puppet server.

TPA will take this opportunity to implement code signing and verification more broadly. This will not only allow TPA to continue using the Tails backup infra as-is after the merge of Puppet codebases and servers but will also help to create strategies to mitigate potential issues with GitLab or attempts to tamper with our code in general.

Proposal

The general goal is to allow for the verification of authenticity of commits in Git repositories. In particular, we want to ease and increase the use of GitLab CI and merge request workflows without having to increase our attack surface on critical infrastructure (for Puppet, in particular).

The system will be based in sequoia-git, so:

  • Authorization info and OpenPGP certificates will be stored in a policy file.
  • Authentication can be checked against either an openpgp-policy.toml policy file stored in the root of repositories (default) or some other external file.
  • Updates to remote refs will be accepted when there exists an authenticated path from a designated "trust-root" to the tip of the reference being updated (a.k.a. the "Gerwitz method").

On the server side, TPA will initially deploy:

The verification mechanism will be available for deployment to any other Git repository upon request.

On the client-side, users can use different Git hooks to get notification about authentication status.

See Verifying commits for more details on client and server-side implementations.

Scope

TPA

Phase 1: Puppet

TPA will initially deploy this mechanism to protect all references of its Puppet Git repositories, which currently means:

  • puppet.torproject.org:/srv/puppet.torproject.org/git/tor-puppet.git
  • puppet.torproject.org:/srv/puppet.torproject.org/git/tor-puppet-hiera-enc.git

The reason for enforcing verification for all references in the TPA Puppet repositories is that all branches are automatically deployed as Puppet environments, so any branch/tag can end up being used to compile a catalog that is applied to a node.

Phase 2: Other TPA repositories

With the mechanism in place, TPA may implement, in the future, authentication of ref updates to some or all branches of:

  • repositories under the tpo/tpa namespace in GitLab
  • repositories in the TPA infrastructure that are managed via Puppet

Other teams

Any team can request deployment of the authentication mechanism to repositories owned by them and managed by TPA.

For each repository in which the mechanism is to be deployed, the following information is needed:

  • the list of references (branches/tags) to be protected (can be all)
  • a commit ID that represents the trust-root against which authentication will be checked

Known issues

Reference rebinding vulnerability

This mechanism does not bind signatures to references, so it can't verify, by itself, whether a commit is authorized to be referenced by a specific branch or tag. This means that reference updates will be accepted for any commit that is successfully authenticated, and repository reference structure/hierarchy is not verified by this mechanism. We may introduce other mechanisms to deal with this later on (for example, signed pushes).

Also, enforcing signed commits can (and most probably will) result in users signing every commit they produce, which then generates lots of signed commits that should not end up in production. Again, we will not deal with this issue in this proposal.

To be clear, this mechanism allows one to verify whether a commit was produced by an authorized certificate, but does not allow one to verify whether a specific reference to a commit is intended.

See git signed commits are a bad idea for more context on these and other issues.

Concretely, this would allow a hostile GitLab server to block updates to references, deploy draft changes to production, or roll-back changes to previous versions. This is considered to be an acceptable compromise, given that GitLab does not support signed pushes and we do not regularly use (sign) tags on TPA repositories.

Possible authentication improvements

This proposal also does not integrate with LDAP or the future authentication system.

Tasks

This is a draft of the steps required to implement this policy:

  1. Add a policy file to the TPA Puppet repository and deploy it to the GitLab and Puppet Server nodes
  2. Create a Git update hook using sq-git update-hook that can be pinned to a policy file and a trust root
  3. For each of the repositories in scope, find the longest path that can be authenticated with the TPA policy file and store that as that repo's trust root
  4. Deploy the update hook to the repositories in scope
  5. Add a CI job that checks the existence of an authenticated path between the trust root and HEAD. This job should always pass, as we protect all reference updates in the Puppet repositories.
  6. Switch the "canonical" Puppet repository to be the one in GitLab, and configure mirroring to the repository in the Puppet server
  7. Provide instructions and templates of client-side hooks so that users can authenticate changes on the client-side.

This should be done for each of the repositories listed in the Scope section.

Affected users

Initially, affected users are only TPA members, as the initial deployment will be made only to some TPA repositories.

In the future, the server-side hook can be deployed by TPA to any other repositories, upon request from the team owning the repository. Then more and more users would be subject to commit-signing enforcement.

Timeline

Starting from November 2025, other team's repositories can be protected upon request.

Alternatives considered

  • Signed pushes. GitLab does not support signed pushes out of the box and does its own authorization checks using SSH keys and user permissions. Even if it would, signed push checks would be stored and enforced by GitLab, which wouldn't resolve our attack surface broadening issue.
  • Signed tags. In the case of the TPA Puppet repositories, which this proposal initially aims to address, enforcing signed tags would be impractical as several changes are pushed all the time and we rarely publish tags on our repositories.
  • Enforcing signatures in all commits. This option would create a usability issue for repositories that allow for external contributions, as third-party commits would have to be (re-)signed by authorized users, thus breaking Merge Requests and adding churn for our developers.
  • GitLab push rules. Relying on this mechanism would increase our trust in GitLab even more, which is contrary to what this proposal intends. It's also a non-free feature which we generally try to avoid depending on, particularly for security-critical, load-bearing policies.

Appendix

This section expands on how verification works in sequoia-git.

Bootstrap

Trust root

It is always necessary to bootstrap trust in a given repository by defining a "trust root", which is a commit that is considered trusted. The trust root info can't be distributed in the repository itself, otherwise an attacker that can modify the repository can also modify the trust root, and then no real authentication is provided.

The trust root can be passed in the command line for sq-git log (using the --trust-root param) or set as a Git configuration, like this:

git config sequoia.trustRoot $COMMIT_ID

Policy file

The default behavior of sq-git is to authenticate changes using an openpgp-policy.toml policy file that lives in the root of the repository itself: each commit is verified against authorization set in the policy file of its parent(s). If this is the case, just define a trust root and run sq-git log.

Alternatively, repositories can be authenticated against an external arbitrary policy file. In this case, the same policy file is used to authenticate all commits.

In the case of TPA, changes for all repositories are authenticated against one unique policy file, which lives in the Puppet repository. On the client side, the tpo/tpa/repos> repository can be used to bootstrap trust in all other repositories. For that, one needs to define a trust root for the tpo/tpa/repos> repository, and then follow the bootstrap instructions in the repository to automatically set trust roots for all other repositories. If needed, confirm a sane trust root with your team mates.

Important: when using a policy file external to a repository, revoking privileges requires updating trust roots for all repositories, because changes that were valid with the old policy may fail to authenticate under the new policy.

Verifying commits

An openpgp-policy.toml file in a repository contains the OpenPGP certificates allowed to perform operations in the repository and the list of authorized operations each certificate is able to perform.

A user can verify the path between a "trust root" and the current HEAD by running:

sq-git log --trust-root $COMMIT_ID

The tree will be traversed and commits will be checked one by one against the policy file of its parents. Verification succeeds if there is an authenticated path between the trust root and the HEAD.

Note that the definition of the trust root is delegated to each user and not stored in the policy file (otherwise any new commit could point to itself as a trust root).

Alternatively, a commit range can be passed. See sq-git log --help for more info.

Server-side

We will leverage the sq-git update-hook subcommand to implement server-side hooks to prevent the update of refs when authentication fails. Info about trust-roots and OpenPGP policy files will be stored in Git config.

Client-side

Even though authentication of updates is enforced on the server side, being able to authenticate on the client side is also useful to help with auditing and detecting tampering.

First, make sure to configure a trust root for each of your repositories:

git config sequoia.trustRoot $COMMIT_ID

Git doesn't provide a general way to reject commits when pulling from remotes, but we can use Git hooks to, at least, get notified about authentication status of the incoming changes.

For example, a pull generally consists of a fetch followed by a merge, so we can use something like the following post-merge hook:

cat > .git/hooks/post-merge <<EOF
#!/bin/sh
sq-git log
EOF
chmod a+x .git/hooks/post-merge

Note that this runs after a successful merge and will not prevent the merge from happening.

Example of successful pull with merge:

$ git pull origin openpgp-policy
From puppet.torproject.org:/srv/puppet.torproject.org/git/tor-puppet
 * branch                openpgp-policy -> FETCH_HEAD
Updating 95929f769..a4a5430c0
Fast-forward
 .gitlab-ci.yml | 7 +++++++
 1 file changed, 7 insertions(+)
95929f7691d214d45adb70a4f43c7a1879d16db4..a4a5430c09c156815b7c275a15c836c5258b6596:
  Cached positive verification
Verified that there is an authenticated path from the trust root
95929f7691d214d45adb70a4f43c7a1879d16db4 to a4a5430c09c156815b7c275a15c836c5258b6596.

Example of unsuccessful pull with merge:

$ git pull origin openpgp-policy
From puppet.torproject.org:/srv/puppet.torproject.org/git/tor-puppet
 * branch                openpgp-policy -> FETCH_HEAD
Updating 95929f769..a4a5430c0
Fast-forward
 .gitlab-ci.yml | 7 +++++++
 1 file changed, 7 insertions(+)
95929f7691d214d45adb70a4f43c7a1879d16db4..a4a5430c09c156815b7c275a15c836c5258b6596:
  Cached positive verification
Error: Authenticating 95929f7691d214d45adb70a4f43c7a1879d16db4 with 2a3753442fc31c23e6fa9cd7aee4074b07c78a8d

Caused by:
    Commit 2a3753442fc31c23e6fa9cd7aee4074b07c78a8d has no policy

TPA will provide templates and automatic configuration where possible, for example by adding "fixups" to the .mrconfig file where appropriate.

Handling external contributions

For repositories that allow some branches to be pushed without enforcement of signed commits, external contributions can be merged by signing the merge commit, which creates an authenticated path from the trust root to tip of the branch.

In those cases, signing of the merge commit must be done locally and merging must be done by pushing to the repository, as opposed to clicking the "Merge" button in the GitLab interface.

References