A password manager is a service that securely stores multiple passwords without the user having to remember them all. TPA uses password-store to keep its secrets, and this page aims at documenting how that works.

Other teams use their own password managers, see issue 29677 for a discussion on that. In particular, we're slowly adopting Bitwarden as a company-wide password manager, see the vault documentation about this.

Tutorial

Basic usage

Once you have a local copy of the repository and have properly configured your environment (see installation), you should be able to list passwords, for example:

pass ls

or, if you are in a subdirectory:

pass ls tor

To copy a password to the clipboard, use:

pass -c tor/services/rt.torproject.org

Passwords are sorted in different folders, see the folder organisation section for details.

One-time passwords

To access certain sites, you'll need a one-time password which is stored in the password manager. This can be done with the pass-otp extension. Once that is installed, you should use the "clipboard" feature to copy-paste the one time code, with:

pass otp -c tor/services/example.com

Adding a new secret

To add a new secret, use the generate command:

pass generate -c services/SECRETNAME

That will generate a strong password and store it in the services/ folder, under the name SECRETNAME. It will also copy it to the clipboard so you can paste it in a password field elsewhere, for example when creating a new account.

If you cannot change the secret and simply need to store it, use the insert command instead:

pass insert services

That will ask you to confirm the password, and supports only entering a single line. To enter multiple lines, use the -m switch.

Passwords are sorted in different folders, see the folder organisation section for details.

Make sure you push after making your changes! By default, pass doesn't synchronize your changes upstream:

pass git push

Rotating a secret

To regenerate a password, you can reuse the same mechanism as the adding a new secret procedure, but be warned that this will completely overwrite the entry, including possible comments or extra fields that might be present.

How-to

On-boarding new staff

When a new person comes in, their key needs to be added to the .gpg-id file. The easiest way to do this is with the init command. This, for example, will add a new fingerprint to the file:

cd ~/.password-store
pass init $(cat .gpg-id) 0000000000000000000000000000000000000000

The new fingerprint must also be allowed to sign the key store:

echo "export PASSWORD_STORE_SIGNING_KEY=\"$(cat ~/.password-store/.gpg-id)\"" >> ~/.bashrc

The will re-encrypt the password file which will require a lot of touching on your cryptographic token, at just the right time. Most humans can't manage that level of concentration and, anyways, it's a waste of time. So it's actually better to disable touch confirmation for this operation, then re-enable it after, for example:

cd ~/.password-store &&
ykman openpgp keys set-touch sig off &&
ykman openpgp keys set-touch enc off &&
pass init $(cat .gpg-id 0000000000000000000000000000000000000000) &&
printf "reconnect your YubiKey, then press enter: " &&
read _ &&
ykman openpgp keys set-touch sig cached &&
ykman openpgp keys set-touch enc cached

The above assumes ~/.password-store is the TPA password manager, if it is stored elsewhere, you will need to use the PASSWORD_STORE_DIR environment for the init to apply to the right store:

env PASSWORD_STORE_DIR=~/src/tor/tor-passwords pass init ...

Off boarding

When staff that has access to the password store leaves, access to the password manager needs to be removed. This is equivalent to the on boarding procedure except instead of adding a person, you remove them. This, for example, will remove an existing user:

pass init $(grep -v 0000000000000000000000000000000000000000 .gpg-id)

See the above notes for YubiKey usage and non-standard locations.

But that might not be sufficient to protect the passwords, as the person will still have a local copy of the passwords (and could have copied them elsewhere anyway). If the person left on good terms, it might be acceptable to avoid the costly rotation procedure, and the above re-encryption procedure is sufficient, provided that the person who left removes all copies of the password manager.

Otherwise, if we're dealing with a bumpy retirement or layoff, all passwords the person had access to must be rotated. See mass password rotation procedures.

Re-encrypting

This typically happens when onboarding or offboarding people, see the on boarding procedure. You shouldn't need to re-encrypt the store if the keys stay the same, and password store doesn't actually support this (although there is a patch available to force re-encryption).

Migrating passwords to the vault

See converting from pass to bitwarden.

Mass password rotation

It's possible (but very time consuming) to rotate multiple passwords in the store. For this, the pass-update tool is useful, as it automates part of the process. It will:

for all (or a subset of) passwords
copy the current password to the clipboard (or show it)
wait for the operator to copy-paste it to the site
generate and save a new password, and copy it to the clipboard

So a bulk update procedure looks like this:

pass update -c

That will take a long time to proceed those, so it's probably better to do it one service at a time. Here's documentation specific to each section of the password manager. You should prioritize the dns and hosting sections.

See issue 41530 for a mass-password rotation run. It took at least 8h of work, spread over a week, to complete the rotation, and it didn't rotate OOB access, LUKS passwords, GitLab secrets, or Trocla passwords. It is estimated it would take at least double that time to complete a full rotation, at the current level of automation.

DNS and hosting

Those two are similar and give access to critical parts of the infrastructure, so they are worth processing first. Start with current hosting and DNS providers:

pass update -c dns/joker dns/portal.netnod.se hosting/accounts.hetzner.com hosting/app.fastly.com

Then the rest of them:

pass update -c hosting

Services

Those are generally websites with special accesses. They are of a lesser priority, but should nevertheless be processed:

pass update -c services

It might be worth examining the service list to prioritize some of them.

Note that it's impossible to change the following passwords:

DNSwl: they specifically refuse to allow users to change their passwords (!) ("To avoid any risks of (reused) passwords leaking as the result of a security incident, the dnswl.org team preferred to use passwords generated server-side which can not be set by the user.")

The following need coordination with other teams:

anti-censorship: archive.org-gettor, google.com-gettor

root

Next, the root passwords should be rotated. This can be automated with a Fabric task, and should be tested with a single host first:

fab -H survey-01.torproject.org host.password-change --pass-dir=tor/root

Then go on the host and try the generated password:

ssh survey-01.torproject.org

then:

login root

Typing the password should just work there. If you're confident in the procedure, this can be done for all hosts with the delicious:

fab -H $(
  echo $(
    ssh puppetdb-01.torproject.org curl -s -G http://localhost:8080/pdb/query/v4/facts \
    | jq -r ".[].certname" | sort -u \
  ) | sed 's/ /,/g'
) host.password-change --pass-dir=tor/root

If it fails on one of the host (e.g. typically dal-rescue-02), you can skip past that host with:

fab -H $(
  echo $(
    ssh puppetdb-01.torproject.org curl -s -G http://localhost:8080/pdb/query/v4/facts \
    | jq -r ".[].certname" | sort -u \
    | sed '0,/dal-rescue-02/d'
  ) | sed 's/ /,/g'
) host.password-change --pass-dir=tor/root

Then the password needs to be reset on that host by hand.

OOB

Similarly, out-of band access need to be reset. This involves logging in to each server's BIOS and changing the password. pass update, again, should help, but instead of going through a web browser, it's likely more efficient to do this over SSH:

pass update -c oob

There is a REST API for the Supermicro servers that should make it easier to automate this. We currently only have 7 hosts with such password and it is currently considered more time-consuming to automate this than to manually perform each reset using the above.

LUKS

Next, full disk encryption keys. Those are currently handled manually (with pass update) as well, but we are hoping to automate this as well, see issue 41537 for details.

lists

Individual list passwords may be rotated, but that's a lot of trouble and coordination. The site password should be changed, at least. When Mailman 3 is deployed, all those will go away anyway.

misc

Those can probably be left alone; it's unclear if they have any relevance left and should probably be removed.

Trocla

Some passwords are stored in Trocla, on the Puppet server (currently pauli.torproject.org). If we worry about lateral movement of an hostile attacker or a major compromise, it might be worth resetting all some of Trocla's password.

This is currently not automated. In theory, deleting the entire Trocla database (its path is configured in /etc/troclarc.yaml) and running Puppet everywhere should reset all passwords, but this hides a lot of complexity, namely:

IPSec tunnels will collapse until Puppet is ran on both ends, which could break lots of things (e.g. CiviCRM, Ganeti)
application passwords are sometimes manually set, for example the CiviCRM IMAP and MySQL passwords are not managed by Puppet and would need to be reset by hand

Here's a non-exhaustive list of passwords that need manual resets:

CiviCRM IMAP and MySQL
Dangerzone WebDAV
Grafana user accounts
KGB bot password (used in GitLab)
Prometheus CI password (used in GitLab's prometheus-alerts CI)
metrics DB, Tagtor, victoria metrics, weather
network health relay
probetelemetry/v2ray
rdsys frontend/backend

Run git grep trocla in tor-puppet.git for the list. Note that it will match secrets that are correctly managed by Puppet.

Automation could be built to incrementally perform those rotations, interactively. Alternatively, some password expiry mechanism could be used, especially for secrets that are managed in one Puppet run (e.g. the Dovecot mail passwords in GitLab).

GitLab secrets

In case of a full compromise, an attacker could have sucked the secrets out of GitLab projects. The gitlab-tokens-audit.py script in gitlab-tools provides a view of all the group and project access tokens and CI/CD variables in a set of groups or projects.

Those tokens are currently rotated manually, but there could be more automation here as well: the above Python script could be improved to allow rotating tokens and resetting the associated CI/CD variable. A lot of CI/CD secret variables are SSH deploy keys, those would need coordination with the Puppet repository, maybe simply modifying the YAML files at first, but eventually those could be generated by Trocla and (why not) automatically populated in GitLab as well.

S3

Object storage uses secrets extensively to provide access to buckets. In case of a compromise, some or all of those tokens need to be reset. The authentication section of the object storage documentation has some more information.

Basically, all access keys need to be rotated, which means expiring the existing one and creating a new one, then copying the configuration over to the right place, typically Puppet, but GitLab runners need manual configuration.

The bearer token also needs to be reset for Prometheus monitoring.

Other services

Each item in the service list is also probably affected and might warrant a review. In particular, you may want to rotate the CRM keys.

This service is likely not going to alert or require emergency interventions.

Signature invalid

If you get an error like:

Signature for /home/user/.password-store/tor/.gpg-id is invalid.

... that is because the signature in the .gpg-id.sig file is, well, invalid. This can be verified with gpg --verify, for example in this case:

$ gpg --verify .gpg-id.sig 
gpg: assuming signed data in '.gpg-id'
gpg: Signature made lun 15 avr 2024 11:51:18 EDT
gpg:                using EDDSA key BBB6CD4C98D74E1358A752A602293A6FA4E53473
gpg: BAD signature from "Antoine Beaupré <anarcat@orangeseeds.org>" [ultimate]

This is indeed "BAD" because it means the .gpg-id file was changed without a new signature being made. This could be done by an attacker to inject their own key in the store to force you to encrypt passwords to a key under their control.

The first step is to check when the .gpg-id files were changed last, with git log --stat -p .gpg-id .gpg-id.sig. In this case, we had this commit on top:

commit 5b12f7f1e140293e20056569dcd7f8b52c426d90
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Mon Apr 15 12:53:59 2024 -0400

    sort gpg-id files
    
    This will make them easier to merge and manage
---
 .gpg-id | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.gpg-id b/.gpg-id
index 62c4af1..f2fd10c 100644
--- a/.gpg-id
+++ b/.gpg-id
@@ -1,4 +1,4 @@
-BBB6CD4C98D74E1358A752A602293A6FA4E53473
 95F341D746CF1FC8B05A0ED5D3F900749268E55E
-E3ED482E44A53F5BBE585032D50F9EBC09E69937
+BBB6CD4C98D74E1358A752A602293A6FA4E53473
 DC399D73B442F609261F126D2B4075479596D580
+E3ED482E44A53F5BBE585032D50F9EBC09E69937

That is actually a legitimate change! I just sorted the file and forgot to re-sign it. The fix was simply to re-sign the file manually:

gpg --detach-sign .gpg-id

But a safer approach would be to simply revert that commit:

git revert 5b12f7f1e140293e20056569dcd7f8b52c426d90

Disaster recovery

A total server loss should be relatively easy to recover from. Because the password manager is backed by git, it's "simply" a matter of finding another secure location for the repository, where only the TPA admins have access to the server.

TODO: document a step-by-step procedure to recreate a minimal git server or exchange updates to the store. Or Syncthing or Nextcloud maybe?

If the pass command somehow fails to find passwords, you should be able to decrypt the passwords with GnuPG directly. Assuming you are in the password store (e.g. ~/.password-store/tor), this should work:

gpg -d < luks/servername

If that fails, it should tell you which key the file is encrypted to. You need to find a copy of that private key, somehow.

Reference

Installation

The upstream download instructions should get you started with installing pass itself. But then you need a local copy of the repository, and configure your environment.

First, you need to get access to the password manager which is currently hosted on the legacy Git repository:

git clone git@puppet.torproject.org:/srv/puppet.torproject.org/git/tor-passwords.git ~/.password-store

If you do not have access, it's because your onboarding didn't happen correctly, or that this guide is not for you.

Note that the above clones the password manager directly under the default password-store path, in ~/.password-store. If you are already using pass, there's likely already things there, so you will probably want to clone it in a subdirectory, like this:

git clone git@puppet.torproject.org:/srv/puppet.torproject.org/git/tor-passwords.git ~/.password-store/tor

You can also clone the password store elsewhere and use a symbolic link to ~/.password-store to reference it.

If you have such a setup, you will probably want to add a pre-push (sorry, there's no post-push, which would be more appropriate) hook so that pass git push will also push to the sub-repository:

cd ~/.password-store &&
printf '#!/bin/sh\nprintf "echo pushing tor repository first... "\ngit -C tor push || true\n' > .git/hooks/pre-push &&
chmod +x .git/hooks/pre-push

Make sure you configure pass to verify signatures. This can be done by adding a PASSWORD_STORE_SIGNING_KEY to your environment, for example, in bash:

echo "export PASSWORD_STORE_SIGNING_KEY=\"$(cat ~/.password-store/.gpg-id)\"" >> ~/.bashrc

Note that this takes the signing key from the .gpg-id file. You should verify those key fingerprints and definitely not automatically pull them from the .gpg-id file regularly. The above command will actually write the fingerprints (as opposed to using cat .gpg-id) to the configuration file, which is safer as an attacker would need to modify your configuration to take over the repository.

Migration from pwstore

The password store was initialized with this:

export PASSWORD_STORE_DIR=$PWD/tor-passwords
export PASSWORD_STORE_SIGNING_KEY="BBB6CD4C98D74E1358A752A602293A6FA4E53473 95F341D746CF1FC8B05A0ED5D3F900749268E55E E3ED482E44A53F5BBE585032D50F9EBC09E69937"
pass init $PASSWORD_STORE_SIGNING_KEY

This created the .gpg-id metadata file that indicates which keys to use to encrypt the files. It also signed the file (in .gpg-id.sig).

Then the basic categories were created:

mkdir dns hosting lists luks misc root services

misc files were moved in place:

git mv entroy-key.pgp misc/entropy-key.gpg
git mv ssl-contingency-keys.pgp misc/ssl-contingency-keep.gpg
git mv win7-keys.pgp misc/win7-keys.gpg

Note that those files were renamed to .gpg because pass relies on that unfortunate naming convention (.pgp is the standard file extension for encrypted files).

The root passwords were converted with:

gpg -d < hosts.pgp | sed '0,/^host/d'| while read host pass date; do 
    pass insert -m root/$host <<EOF
    $pass
    date: $date
    EOF
done

Integrity was verified with:

anarcat@angela:tor-passwords$ gpg -d < hosts.pgp | sed '0,/^host/d'| wc -l 
gpg: encrypted with 2048-bit RSA key, ID 41D1C6D1D746A14F, created 2020-08-31
      "Peter Palfrader"
gpg: encrypted with 255-bit ECDH key, ID 16ABD08E8129F596, created 2022-08-16
      "Jérôme Charaoui <jerome@riseup.net>"
gpg: encrypted with 255-bit ECDH key, ID 9456BA69685EAFFB, created 2023-05-30
      "Antoine Beaupré <anarcat@torproject.org>"
88
anarcat@angela:tor-passwords$ ls root/| wc -l
88
anarcat@angela:tor-passwords$ for p in $(ls root/* | sed 's/.gpg//') ; do if ! pass $p | grep -q date:; then echo $p has no date; fi ; if ! pass $p | wc -l | grep -q '^2$'; then echo $p does not have 2 lines; fi ; done
anarcat@angela:tor-passwords$

The lists passwords were converted by first going through the YAML to fix lots of syntax errors, then doing the conversion with a Python script written for the purpose, in lists/parse-lists.py.

The passwords in all the other stores were converted using a mix of manual creation and rewriting the files to turn them into a shell script. For example, an entry like:

foo:
  access: example.com
  username: root
  password: REDACTED
bar:
  access: bar.example.com
  username: root
  password: REDACTED

would be rewritten, either by hand or with a macro (to deal with multiple entries more easily), into:

pass inert -m services/foo <<EOF
REDACTED
url: example.com
user: root
EOF
pass inert -m services/bar <<EOF
REDACTED
url: bar.example.com
user: root
EOF

In the process, fields were reordered and renamed. The following changes were performed manually:

url instead of access
user instead of username
password: was stripped and the password was put alone on a the first line, as pass would expect
TOTP passwords were turned into otpauth:// URLs, but the previous incantation was kept as a backup, as that wasn't tested with pass-otp

The OOB passwords were split from the LUKS passwords, so that we can have only the LUKS password on its own in a file. This will also possibly allow layered accesses there where some operators could have access to the BIOS but not the LUKS encryption key. It will also make it easier to move the encryption key elsewhere if needed.

History was retained, for now, as it seemed safer that way. The pwstore tag was laid on the last commit before the migration, if we ever need an easy way to roll back.

Upgrades

Pass is managed client side, and packaged widely. Upgrades have so far not included any breaking changes and should be safe to automate using normal upgrade mechanisms.

SLA

No specific SLA for this service.

Design and architecture

The password manager is based on passwordstore which itself relies on GnuPG for encrypting secrets. The actual encryption varies, but currently data is encrypted with a AES256 session key itself encrypted with ECDH and RSA keys.

Passwords are stored in a git repository, currently Gitolite. Clients pull and push content from said repository and decrypt and encrypt the files with GnuPG/pass.

Services

No long-running service is necessary for this service, although a Git server is used for sharing the encrypted files.

Storage

Files are stored, encrypted, one password per file, on disk. It's preferable to store those files on a fully-encrypted filesystem as well.

Server-side, files are stored in a Git repository, on a private server (currently the Puppet server).

Queues

N/A.

Interfaces

Mainly interface through the pass commandline client. Decryption is possible with the plain gpg -d command, but direct operation is discouraged because it's likely going to miss some pass-specific constructs like checking signatures or encrypting to the right key.

Authentication

Relies on OpenPGP and Git.

Implementation

Pass is written in bash.

Git and OpenPGP.

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker with the label ~Security.

Maintainer

This service is maintained by TPA and specifically managed by @anarcat.

Users

Pass is used by TPA.

Upstream

pass was written by Jason A. Donenfeld of Wireguard fame.

Monitoring and metrics

There's no monitoring of the password manager.

Tests

N/A.

Logs

No logs are held, although the Git history keeps track of changes to the password store.

Backups

Backups are performed using our normal backup system, with the caveat that it requires a decryption key to operate, see also the OpenPGP docs in that regard.

Discussion

Historically, TPA password have been managed in a tool called pwstore, written by weasel. We switched to pass in February 2024 in TPA-RFC-62.

Overview

The main issues with the password manager as it stands right now are that it lives on the legacy Git infrastructure, it's based on GnuPG, it doesn't properly hide the account list, and keeps old entries forever.

Security and risk assessment

No audit was performed on pass, as far as we know. OpenPGP itself is a battle-hardened standard but that has seen more and more criticism in the past few years, particularly in terms of usability. An alternative implementation like gopass could be interesting, especially since it supports an alternative backend called age. The age authors have also forked pass to make it work with age directly.

A major risk with the automation work that was done is that an attacker with inside access to the password manager could hijack large parts of the organisation by quickly rotating other operators out of the password store and key services. This could be mitigated by using some sort of secret sharing scheme where two operators would be required to decrypt some secrets.

There are other issues with pass:

optional store verification: it's possible that operators forget to set the PASSWORD_STORE_SIGNING_KEY variable which will make pass accept unsigned changes to the gpg-id file which could lead a compromise on the Git server be leveraged to extract secrets
limited multi-store support: the PASSWORD_STORE_SIGNING_KEY is global and therefore makes it complicated to have multiple, independent key stores
global, uncontrolled trust store: pass relies on the global GnuPG key store although in theory it should be possible to rely on another keyring by passing different options to GnuPG
account names disclosure: by splitting secrets into different files, we disclose which accounts we have access to, but this is considered a reasonable tradeoff for the benefits it brings
mandatory client use: if another, incompatible, client (e.g. Emacs) is used to decrypt and re-encrypt the secrets, it might not use the right keys
GnuPG/OpenPGP: pass delegates cryptography to OpenPGP, and more specifically GnuPG, which is suffering from major usability and security issues
permanent history: using git leverages our existing infrastructure for file-sharing, but means that secrets are kept in history forever, which makes revocation harder
difficult revocation: a consequence of having client-side copies of passwords means that revoking passwords is more difficult as they need to be rotated at the source
file renaming attack (CVE-2020-28086): an attacker controlling server bar could rename file foo to bar to get an operator accessing bar to reveal the password to foo, low probability and low impact for us

At the time of writing (2025-02-11), there is a single CVE filed against pass, see cvedetails.com.

Technical debt and next steps

The password manager is designed squarely for use by TPA and doesn't aim at providing services to non-technical users. As such, this is a flaw that should be remedied, probably by providing a more intuitive interface organization-wide, see tpo/tpa/team#29677 for that discussion.

The password manager is currently hosted in the legacy Gitolite server and need to be moved out of there. It's unclear where; GitLab is probably too big of an attack surface, with too many operators with global access, to host the repository, so it might move to another virtual machine instead.

Proposed Solution

TPA-RFC-62 documents when we switched to pass and why.

Other alternatives

TPA-RFC-62 lists a few alternatives to pass that were evaluated during the migration. The rest of this section lists other alternatives that were added later.

Himitsu: key-value store with optional encryption for some fields (like passwords), SSH agent, Firefox plugin, GUI, written in Hare
Passbolt: PHP, web-based, open core, PGP based, MFA (closed source), audited by Cure53
redoctober: is a two-person encryption system that could be useful for more critical services (see also blog post).

Keyboard shortcuts