Torproject Sysadmin Team

The Torproject System Administration Team is the team that keeps torproject.org's infrastructure going. This is the internal team wiki. It has mostly documentation mainly targeted for the team members, but also has useful information for people with torproject.org accounts.

The documentation is split into the following sections:

• Introduction to the team, what it does, key services and policies
• Support - in case of fire, press this button
• User documentation - aimed primarily at non-technical users and the general public
• Sysadmin how-to's - procedures specifically written for sysadmins
• Service list - service list and documentation
• Machine list - the full list of machines managed by TPA (in LDAP)
• Policies - major decisions and how they are made
• Providers - info about service and infrastructure providers
• Meetings - minutes from our formal meetings
• Roadmaps - documents our plans for the future (and past successes of course)

Our source code is all hosted on GitLab.

This is a wiki. We welcome changes to the content! If you have the right permissions -- which is actually unlikely, unfortunately -- you can edit the wiki in GitLab directly. Otherwise you can submit a pull request on the wiki replica. You can also clone the git repository and send us a patch by email.

To implement a similar merge request workflow on your GitLab wiki, see TPA's documentation about Accepting merge requests on wikis.

This documentation is primarily aimed at users.

Note: most of this documentation is a little chaotic and needs to be merged with the service listing. You might interested in one of the following quick links instead:

• Email
• Forum (Discourse)
• GitLab
• Nextcloud
• Matrix
• IRC
• Yubikeys
• How to get help?

Other documentation:

• accounts
• admins
• bits-and-pieces
• extra
• hardware-requirements
• lektor-dev-macos
• naming-scheme
• reporting-email-problems
• services
• ssh-jump-host
• static-sites
• svn-accounts

• torproject.org Accounts
  • Decision tree: LDAP account or email alias?
    • LDAP account reasons
    • Email alias reasons
  • New LDAP accounts
    • Step 1
      • username policy
    • Step n+1
  • Retiring accounts
  • Getting added to an existing group/Getting access to a specific host
  • Changing email aliases
    • Adding a new email alias
      • Personal Email Aliases
      • Group Email Aliases
    • Getting added to an existing email alias
  • Changing/Resetting your passwords
    • LDAP
    • Host specific passwords / sudo passwords
  • Changing/Updating your OpenPGP key
    • Revoked or lost old key
    • Actually updating the keyring

Note that this documentation needs work, as it overlaps with user creation procedures, see issue 40129.

torproject.org Accounts

The Tor project keeps all user information in a central LDAP database which governs access to shell accounts, git (write) access and lets users configure their email forwards.

It also stores group memberships which in turn affects which users can log into which hosts.

This document should be consistent with the Tor membership policy, in case of discrepancy between the two documents, the membership policy overrules this document.

Decision tree: LDAP account or email alias?

Here is a simple decision tree to help you decide if a new contributor needs an LDAP account, or if an email alias will do. (All things being equal, it's better to set people up with only an email alias if that's all they need, since it reduces surface area which is better for security.)

LDAP account reasons

Regardless of whether they are a Core Contributor:

• Are they a maintainer for one of our official software projects, meaning they need to push commits (write) to one of our git repos?
  • They should get an LDAP account.
• Do they need to access (read) a private git repo, like "dirauth-conf"?
  • They should get an LDAP account.

Are they a Core Contributor?

• Do they want to make their own personal clones of our git repos, for example to propose patches and changes?
  • They don't need an LDAP account for just this case anymore, since gitlab can host git repos. (They are also welcome to put their personal git repos on external sites if they prefer.)
• Do they need to log in to our servers to use our shared irc host?
  • They should get an LDAP account.
  • If they're not a Core Contributor, they should put their IRC somewhere else, like pastly's server.
• Do they need to log in to our servers to maintain one of our websites or services?
  • An existing Core Contributor should request an LDAP account.
  • If they're not a Core Contributor, but they are a staff member who needs to maintain services, then Tor Project Inc should request an LDAP account.
  • If they are not a staff member, then an existing Core Contributor should request an LDAP account, and explain why they need access.
• Do they need to be able to send email using an @torproject.org address?
  • In our 2022/2023 process of locking down email, it's increasingly necessary for people to have a full ldap account in order to deliver their tor mail to the internet properly.

See New LDAP accounts for details.

Email alias reasons

If none of the above cases apply:

• Are they a Core Contributor?
  • An existing Core Contributor should request an email alias.
• Are they a staff member?
  • Tor Project Inc should request an email alias.

See Changing email aliases for details.

New LDAP accounts

New accounts have to be sponsored by somebody who already has a torproject.org account. If you need an account created, please find somebody in the project who you are working with and ask them to request an account for you.

Step 1

The sponsor will collect all required information:

• name,
• initial forwarding email address (the user can change that themselves later),
• OpenPGP key fingerprint,
• desired username.

The sponsor is responsible for verifying the information's accuracy, in particular establishing some confidence that the key in question actually belongs to the person that they want to have access.

The user's OpenPGP key should be available from the public keyserver network.

The sponsor will create a ticket in GitLab:

• The ticket should include a short rationale as to why the account is required,
• contain all the pieces of information listed above, and
• should be OpenPGP signed by the sponsor using the OpenPGP key we have on file for them. Please enclose the OpenPGP clearsigned blob using {{{ and }}}.

username policy

Usernames are allocated on a first-come, first-served basis. Usernames should be checked for conflict with commonly used administrative aliases (root, abuse, ...) or abusive names (killall*, ...). In particular, the following have special meaning for various services and should be avoided:

root
abuse
arin-admin
certmaster
domainadmin
hostmaster
mailer-daemon
postmaster
security
webmaster

That list, taken from the leap project is not exhaustive and your own judgement should be used to spot possibly problematic aliases. See also those other possible lists:

• systemli
• LEAP
• immerda

Step n+1

Once the request has been filed it will be reviewed by Roger or Nick and either approved or rejected.

If the board indicates their assent, the sysadmin team will then create the account as requested.

Retiring accounts

If you won't be using your LDAP account for a while, it's good security hygiene to have it disabled. Disabling an LDAP account is a simple operation, and reenabling an account is also simple, so we shouldn't be shy about disabling accounts when people stop needing them.

To simplify the review process for disable requests, and because disabling by mistake has less impact than creating a new LDAP account by mistake, the policy here is "any two of {Roger, Nick, Shari, Isabela, Erin, Damian} are sufficient to confirm a disable request."

(When we disable an LDAP account, we should be sure to either realize and accept that email forwarding for the person will stop working too, or add a new line in the email alias so email keeps working.)

Getting added to an existing group/Getting access to a specific host

Almost all privileges in our infrastructure, such as account on a particular host, sudo access to a role account, or write permissions to a specific directory, come from group memberships.

To know which group has access to an specific host, FIXME.

To get added to some unix group, it has to be requested by a member of that group. This member has to create a new ticket in GitLab, OpenPGP-signed (as above in the new account creation section), requesting who to add to the group.

If a new group needs to be created, FIXME.

The reasons why a new group might need to be created are: FIXME.

Should the group be orphaned or have no remaining active members, the same set of people who can approve new account requests can request you be added.

To find out who is on a specific group you can ssh to perdulce:

ssh perdulce.torproject.org

Then you can run:

getent group

See also: the "Host specific passwords" section below

Changing email aliases

Create a ticket specifying the alias, the new address to add, and a brief motivation for the change.

For specifics, see the "The sponsor will create a ticket" section above.

Adding a new email alias

Personal Email Aliases

Tor Project Inc can request new email aliases for staff.

An existing Core Contributor can request new email aliases for new Core Contributors.

Group Email Aliases

Tor Project Inc and Core Contributors can request group email aliases for new functions or projects.

Getting added to an existing email alias

Similar to being added to an LDAP group, the right way to get added to an existing email alias is by getting somebody who is already on that alias to file a ticket asking for you to be added.

Changing/Resetting your passwords

LDAP

If you've lost your LDAP password, you can request that a new one be generated. This is done by sending the phrase "Please change my Debian password" to chpasswd@db.torproject.org. The phrase is required to prevent the daemon from triggering on arbitrary signed email. The best way to invoke this feature is with

echo "Please change my Debian password" | gpg --armor --sign | mail chpasswd@db.torproject.org

After validating the request the daemon will generate a new random password, set it in the directory and respond with an encrypted message containing the new password. This new password can then be used to login (click the "Update my info" button), and use the "Change password" fields to create a new LDAP password.

Note that LDAP (and sudo passwords, below) changes are not instantaneous: they can take between 5 to 8 minutes to propagate to any given host.

More specifically, the password files are generated on the master LDAP server every five minutes, starting at the third minute of the hour, with a cron schedule like this:

 3,8,13,18,23,28,33,38,43,48,53,58

Then those files are synchronized on a more standard 5 minutes schedule to all hosts.

There are also delays involved in the mail loop, of course.

Host specific passwords / sudo passwords

Your LDAP password can not be used to authenticate to sudo on servers. It can only allow to log you in through SSH, but you need a different password to get sudo access, which we call the "sudo password".

To set the sudo password:

1. go to the user management website
2. pick "Update my info"
3. set a new (strong) sudo password

If you want, you can set a password that works for all the hosts that are managed by torproject-admin, by using the "wildcard ("*"). Alternatively, or additionally, you can have per-host sudo passwords -- just select the appropriate host in the pull-down box.

Once set on the web interface, you will have to confirm the new settings by sending a signed challenge to the mail interface. The challenge is a single line, without line breaks, provided by the web interface. With the challenge first you will need to produce an openpgp signature:

echo 'confirm sudopassword ...' | gpg --armor --sign

With it you can compose an email to changes@db.torproject.org, sending the challenge in the body followed by the openpgp signature.

Note that setting a sudo password will only enable you to use sudo to configured accounts on configured hosts. Consult the output of "sudo -l" if you don't know what you may do. (If you don't know, chances are you don't need to nor can use sudo.)

Do mind the delays in LDAP and sudo passwords change, mentioned in the previous section.

Changing/Updating your OpenPGP key

If you are planning on migrating to a new OpenPGP key and you also want to change your key in LDAP, or if you just want to update the copy of your key we have on file, you need to create a new ticket in GitLab:

• The ticket should include your username, your old OpenPGP fingerprint and your new OpenPGP fingerprint (if you're changing keys).
• The ticket should be OpenPGP signed with your OpenPGP key that is currently stored in LDAP.

Revoked or lost old key

If you already revoked or lost your old OpenPGP key and you migrated to a new one before updating LDAP, you need to find a sponsor to create a ticket for you. The sponsor should create a new ticket in GitLab:

• The ticket should include your username, your old OpenPGP fingerprint and your new OpenPGP fingerprint.
• Your OpenPGP key needs to be on a public keyserver and be signed by at least one Tor person other than your sponsor.
• The ticket should be OpenPGP signed with the current valid OpenPGP key of your sponsor.

Actually updating the keyring

See the new-user HOWTO.

Moved to policy/tpa-rfc-2-support.

Bits and pieces of Tor Project infrastructure information

A collection of information looking for a better place, perhaps after being expanded a bit to deserve their own page.

Backups

• We use Bacula to make backups, with one host running a director (currently bacula-director-01.tpo) and another host for storage (currently brulloi.tpo).
• There are BASE files and WAL files, the latter for incremental backups.
• The logs found in /var/log/bacula-main.log and /var/log/bacula/ seem mostly empty, just like the systemd journals.

Servers

• There's one director and one storage node.

• The director runs /usr/local/sbin/dsa-bacula-scheduler which reads /etc/bacula/dsa-clients for a list of clients to back up. This file is populated by puppet (puppetdb $bacula::tag_bacula_dsa_client_list) and will list clients until they're being deactivated in puppet.

Clients

• tor-puppet/modules/bacula/manifests/client.pp gives an idea of where things are at on backup clients.
• Clients run the Bacula File Daemon, bacula-fd(8).

Onion sites

• Example from a vhost template

    <% if scope.function_onion_global_service_hostname(['crm-2018.torproject.org']) -%>
    <Virtualhost *:80>
        ServerName <%= scope.function_onion_global_service_hostname(['crm-2018.torproject.org']) %>
        Use vhost-inner-crm-2018.torproject.org
    </VirtualHost>
    <% end -%>
  

• Function defined in tor-puppet/modules/puppetmaster/lib/puppet/parser/functions/onion_global_service_hostname.rb parses /srv/puppet.torproject.org/puppet-facts/onionbalance-services.yaml.

• onionbalance-services.yaml is populated through onion::balance (tor-puppet/modules/onion/manifests/balance.pp)

• onion:balance uses the onion_balance_service_hostname fact from tor-puppet/modules/torproject_org/lib/facter/onion-services.rb

Puppet

See service/puppet.

Extra is one of the sites hosted by "the www rotation". The www rotation uses several computers to host its websites and it is used within tpo for redundancy.

Extra is used to host images that can be linked in blog posts and the like. The idea is that you do not need to link images from your own computer or people.tpo.

Extra is used like other static sites within tpo. Learn how to write to extra

So you want to give us hardware? Great! Here's what we need...

Physical hardware requirements

If you want to donate hardware, there are specific requirements for machine we manage that you should follow. For other donations, please see the donation site.

This list is not final, and if you have questions, please contact us. Also note that we also accept virtual machine "donations" now, for which requirements are different, see below.

Must have

• Out of band management with dedicated network port, preferably a something standard (like serial-over-ssh, with BIOS redirection), or failing that, serial console and networked power bars
• No human intervention to power on or reboot
• Warranty or post-warranty hardware support, preferably provided by the sponsor
• Under the 'ownership' of Tor, although long-term loans can also work
• Rescue system (PXE bootable OS or remotely loadable ISO image)

Nice to have

• Production quality rather than pre-production hardware
• Support for multiple drives (so we can do RAID) although this can be waived for disposable servers like build boxes
• Hosting for the machine: we do not run our own data centers or rack, so it would be preferable if you can also find a hosting location for the machine, see the hosting requirements below for details

To avoid

• proprietary Java/ActiveX remote consoles
• hardware RAID, unless supported with open drivers in the mainline Linux kernel and userland utilities

Hosting requirements

Those are requirements that apply to actual physical / virtual hosting of machines.

Must have

• 100-400W per unit density, depending on workload
• 1-10gbit, unmetered
• dual stack (IPv4 and IPv6)
• IPv4 address space (at least one per unit, typically 4-8 per unit)
• out of band access (IPMI or serial)
• rescue systems (e.g. PXE booting)
• remote hands SLA ("how long to replace a broken hard drive?")

Nice to have

• "clean" IP addresses (for mail delivery, etc)
• complete /24 IPv4, donated to the Tor project
• private VLANs with local network
• BGP announcement capabilities
• not in europe or northern america
• free, or ~ 150$/unit

Virtual machines requirements

Must have

Without those, we will have to be basically convinced to accept those machines:

• Debian OS
• Shell access (over SSH)
• Unattended reboots or upgrades

The latter might require more explanations. It means the machine can be rebooted without intervention of an operator. It seems trivial, but some setups make that difficult. This is essential so that we can apply Linux kernel upgrades. Alternatively, manual reboots are acceptable if such security upgrades are automatically applied.

Nice to have

Those we would have in an ideal world, but are not deal breakers:

• Full disk encryption
• Rescue OS boot to install our own OS
• Remote console
• Provisioning API (cloud-init, OpenStack, etc)
• Reverse DNS
• Real IP address (no NAT)

To avoid

Those are basically deal breakers, but we have been known to accept those situations as well, in extreme cases:

• No control over the running kernel
• Proprietary drivers

Overview

The aim of this document is to explain the steps required to set up a local Lektor development environment suitable for working on Tor Project websites based on the Lektor platform.

We'll be using the Sourcetree git GUI to provide a user-friendly method of working with the various website's git repositories.

Prerequisites

First we'll install a few prerequisite packages, including Sourcetree.

You must have administrator privileges to install these software packages.

First we'll install the Xcode package.

Open the Terminal app and enter:

xcode-select --install

Click Install on the dialog that appears.

Now, we'll install the brew package manager, again via the Terminal:

/bin/bash -c "$(curl -fsSL
https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Now we're ready to install a few more tools:

brew install coreutils git git-lfs python3.8

And lastly we need to download and install Sourcetree. This can be done from the app's website: https://www.sourcetreeapp.com/

Follow the installer prompts, entering name and email address so that the git commits are created with adequate identifying information.

Connect GitLab account

This step is only required if you want to create Merge Requests in GitLab.

Next, we'll create a GitLab token to allow Sourcetree to retrieve and update projects.

• Navigate to https://gitlab.torproject.org/-/profile/personal_access_tokens
• Enter sourcetree under Token name
• Choose an expiration date, ideally not more than a few months
• Check the box next to api
• Click Create personal access token
• Copy the token into your clipboard

Now, open Sourcetree and click the Connect... button on the main windows, then Add..., and fill in the dialog as below. Paste the token in the Password field.

Click the Save button.

The Remote tab on the main window should now show a list of git repositories available on the Tor Project GitLab.

To clone a project, enter its name (eg. tpo or blog) in the Filter repositories input box and click the Clone link next to it.

Depending on the project, a dialog titled Git LFS: install required may then appear. If so, click Yes to ensure all the files in the project are downloaded from GitLab.

Page moved to TPA-RFC-6: Naming Convention.

Email delivery problems are unfortunately quite common but there are often simple solutions to the problems once we know exactly what is going on.

When reporting delivery problems on Email infrastructure, make sure you include at least the following information in your report:

1. originating email address (e.g. Alice <alice@torproject.org>)
2. destination email address (e.g. Bob <bob@torproject.org>)
3. date and time the email was sent, with timezone, to the second (e.g. 2019-06-03 13:52:30 +0400)
4. how the email was sent (e.g. from my laptop, over SMTP+TLS to my email provider, riseup.net)
5. what error did you get (e.g. a bounce, message not delivered)

Ideally, if you can, provide us with the Message-ID header, if you know what that is and can find it. Otherwise, don't worry about it and provide us with the above details.

If you do get a bounced message, do include the entire bounce, with headers. The simplest way to do so is forward it as an attachment or "view source" and copy-paste it somewhere safe (like https://share.riseup.net/).

Ideally, also include a copy of the original message in your report, also with full headers.

If you can't send a copy of the original message for privacy reasons, at least include the headers of the email.

Send us the message using the regular methods, as appropriate, see the support guide for details.

Service on TPO machines are often run as regular users, from normal sessions, instead of the usual /etc/init.d or systemd configuration provided by Debian packages. This is part of our service vs system admin distinction.

This page aims at documenting how such services are started and managed. There are many ways this can be done: many services have been started as a @reboot cronjob in the past, but we're looking at using systemd --user as a more reasonable way to do this in the future.

systemd startup

Most Debian machines now run systemd which allows all sorts of neat tricks. In particular, it allows us to start programs as a normal user through a systemd --user session that gets started automatically at boot.

Adding a new service

User-level services are deployed in ~/.config/systemd/user/. Let's say we're deploying a service called $SERVICE. You'd need to craft a .service file and drop it in ~/.config/systemd/user/$SERVICE.service:

[Unit]
Description=Run a program forever that does not fork

[Service]
Type=simple
ExecStart=/home/role/bin/service start

[Install]
WantedBy=default.target

Then you can run:

systemctl --user daemon-reload

For the new file to be notified.

If you're getting an error like this:

Failed to connect to bus: No such file or directory

It's because your environment is not setup correctly and systemctl can't find the correct sockets. Try to set the XDG_RUNTIME_DIR environment to the right user directory:

export XDG_RUNTIME_DIR=/run/user/$(id -u)

Then the service can be enabled:

systemctl --user enable $SERVICE

And then started:

systemctl --user start $SERVICE

sysadmin stuff

On the sysadmin side, to enable systemd --user session, we need to run loginctl enable-linger $USER. For example, this will enable the session for the user $USER:

loginctl_user { $USER: linger => enabled }

This will create an empty file for the user in /var/lib/systemd/linger/ but it will also start the systemd --user session immediately, which can already be used to start other processes.

cron startup

This method is now discouraged, but is still in use for older services.

Failing systemd or admin support, you might be able to start services at boot time with a cron job.

The trick is to edit the role account crontab with sudo -u role crontab -e and then adding a line like:

@reboot /home/role/bin/service start

It is deprecated because cron is not a service manager and has no way to restart the service easily on upgrades. It also lacks features like socket activation or restart on failure that systemd provides. Plus it won't actually start the service until the machine is rebooted, that's just plain silly.

The correct way to start the above service is to use the .service file documented in the previous section.

You need to use an ssh jump host to access internal machines at tpo. If you have a recent enough ssh (>= 2016 or so), then you can use the ProxyJump directive. Else, use ProxyCommand. ProxyCommand automatically executes the ssh command on the host to jump to the next host and forward all traffic through.

With recent ssh versions:

Host *.torproject.org !ssh.torproject.org !people.torproject.org !gitlab.torproject.org
  ProxyJump ssh.torproject.org

Or with old ssh versions (before OpenSSH 7.3, or Debian 10 "buster"):

Host *.torproject.org !ssh.torproject.org !people.torproject.org !gitlab.torproject.org
  ProxyCommand ssh -l %r -W %h:%p ssh.torproject.org

Note that there are multiple ssh-like aliases that you can use, depending on your location (or the location of the target host). Right now there are two:

• ssh-dal.torproject.org - in Dallas, TX, USA
• ssh-fsn.torproject.org - in Falkenstein, Saxony, Germany

The canonical list for this is searching for ssh in the purpose field on the machines database.

Note: It is perfectly acceptable to run ping against the server to determine the closest to your location, and you can also run ping from the server to a target server as well. The shortest path will be the one that has the lowest sum for those two, naturally.

This naming convention was announced in TPA-RFC-59.

Host authentication

It is also worth keeping the known_hosts file in sync to avoid server authentication warnings. The server's public keys are also available in DNS. So add this to your .ssh/config:

Host *.torproject.org
  UserKnownHostsFile ~/.ssh/known_hosts.torproject.org
  VerifyHostKeyDNS ask

And keep the ~/.ssh/known_hosts.torproject.org file up to date by regularly pulling it from a TPO host, so that new hosts are automatically added, for example:

rsync -ctvLP ssh.torproject.org:/etc/ssh/ssh_known_hosts ~/.ssh/known_hosts.torproject.org

Note: if you would prefer the above file to not contain the shorthand hostname notation (i.e. alberti for alberti.torproject.org), you can get rid of those with the following command after the file is on your computer:

sed -i 's/,[^,.: ]\+\([, ]\)/\1/g' .ssh/known_hosts.torproject.org

Different usernames

If your local username is different from your TPO username, also set it in your .ssh/config:

Host *.torproject.org
  User USERNAME

Root access

Members of TPA might have a different configuration to login as root by default, but keep their normal user for key services:

# interact as a normal user with Puppet, LDAP, jump and gitlab servers by default
Host puppet.torproject.org db.torproject.org ssh.people.torproject.org people.torproject.org gitlab.torproject.org
  User USERNAME

Host *.torproject.org
  User root

Note that git hosts are not strictly necessary as you should normally specify a git@ user in your git remotes, but it's a good practice nevertheless to catch those scenarios where that might have been forgotten.

When not to use the jump host

If you're going to do a lot of batch operations on all hosts (for example with Cumin), you definitely want to add yourself to the adding yourself to the allow list so that you can skip using the jump host.

For this, anarcat uses a special trusted-network command that fails unless the network is on that allow list. Therefore, the above jump host exception list becomes:

# use jump host if the network is not in the trusted whitelist
Match host *.torproject.org, !host ssh.torproject.org, !host ssh-dal.torproject.org, !host ssh-fsn.torproject.org, !host people.torproject.org, !host gitlab.torproject.org, !exec trusted-network
  ProxyJump anarcat@ssh-dal.torproject.org

The trusted-network command checks for the default gateway on the local machine and checks if it matches an allow list. It could also just poke at the internet to see "what is my IP address", like:

• https://check.torproject.org/
• https://wtfismyip.com/text
• https://ifconfig.me/ip
• https://ip.me/
• https://test.anarc.at/

Sample configuration

Here is a redacted copy of anarcat's ~/.ssh/config file:

Host *
     # disable known_hosts hashing. it provides little security and
     # raises the maintenance cost significantly because the file
     # becomes inscrutable
     HashKnownHosts no
     # this defaults to yes in Debian
     GSSAPIAuthentication no
     # set a path for the multiplexing stuff, but do not enable it by
     # default. this is so we can more easily control the socket later,
     # for processes that *do* use it, for example git-annex uses this.
     ControlPath ~/.ssh/control-%h-%p-%r
     ControlMaster no
     # ~C was disabled in newer OpenSSH to facilitate sandboxing, bypass
     EnableEscapeCommandline yes

# taken from https://trac.torproject.org/projects/tor/wiki/doc/TorifyHOWTO/ssh
Host *-tor *.onion
    # this is with netcat-openbsd
    ProxyCommand nc -x 127.0.0.1:9050 -X 5 %h %p
    # if anonymity is important (as opposed to just restrictions bypass), you also want this:
    # VerifyHostKeyDNS no

# interact as a normal user with certain symbolic names for services (e.g. gitlab for push, people, irc bouncer, etc)
Host db.torproject.org git.torproject.org git-rw.torproject.org gitlab.torproject.org ircbouncer.torproject.org people.torproject.org puppet.torproject.org ssh.torproject.org ssh-dal.torproject.org ssh-fsn.torproject.org
  User anarcat

# forward puppetdb for cumin by default
Host puppetdb-01.torproject.org
  LocalForward 8080 127.0.0.1:8080

Host minio*.torproject.org
  LocalForward 9090 127.0.0.1:9090

Host prometheus2.torproject.org
  # Prometheus
  LocalForward 9090 localhost:9090
  # Prometheus Pushgateway
  LocalForward 9091 localhost:9091
  # Prometheus Alertmanager
  LocalForward 9093 localhost:9093
  # Node exporter is 9100, but likely running locally
  # Prometheus blackbox exporter
  LocalForward 9115 localhost:9115

Host dal-rescue-02.torproject.org
  Port 4622

Host *.torproject.org
  UserKnownHostsFile ~/.ssh/known_hosts.d/torproject.org
  VerifyHostKeyDNS ask
  User root

# use jump host if the network is not in the trusted whitelist
Match host *.torproject.org, !host ssh.torproject.org, !host ssh-dal.torproject.org, !host ssh-fsn.torproject.org, !host people.torproject.org, !host gitlab.torproject.org, !exec trusted-network
  ProxyJump anarcat@ssh-dal.torproject.org

How to change the main website

The Tor website is managed via its git repository.

It is usually advised to get changes validated via a merge request on the project.

Once changes are merged to the main branch, , if the changes pass validation checks they get deployed automatically to staging.

If after the auto-deploy to staging everything looks as expected, changes can be deployed to prod by manually launching the CI job deploy prod.

How to change other static websites

A handful of other static websites -- like extra.tp.o, dist.tp.o, and more -- are hosted at several computers for redundancy, and these computers are together called "the www rotation".

How do you edit one of these websites? Let's say you want to edit extra.

• First you ssh in to staticiforme (using an ssh jump host if needed)

• Then you make your edits as desired to /srv/extra-master.torproject.org/htdocs/

• When you're ready, you run this command to sync your changes to the www rotation:

    sudo -u mirroradm static-update-component extra.torproject.org
  

Example: You want to copy image.png from your Desktop to your blog post indexed as 2017-01-01-new-blog-post:

scp /home/user/Desktop/image.png staticiforme.torproject.org:/srv/extra-master.torproject.org/htdocs/blog/2017-01-01-new-blog-post/
ssh staticiforme.torproject.org sudo -u mirroradm static-update-component extra.torproject.org

Which sites are static?

The complete list of websites served by the www rotation is not easy to figure out, because we move some of the static sites around from time to time. But you can learn which websites are considered "static", i.e. you can use the above steps to edit them, via:

ssh staticiforme cat /etc/static-components.conf

How does this work?

If you're a sysadmin and wondering how that stuff work or do anything back there, look at service/static-component.

SVN accounts

We still use SVN in some places. All public SVN repositories are available at svn.torproject.org. We host our presentations, check.torproject.org, website, and an number of older codebases in it. The most frequently updated directories are the website and presentations. SVN is not tied to LDAP in any way.

SVN Repositories available

The following SVN repositories are available:

• android
• arm
• blossom
• check
• projects
• todo
• torctl
• torflow
• torperf
• translation
• weather
• website

Steps to SVN bliss

1. Open a trac ticket per user account desired.

2. The user needs to pick a username and which repository to access (see list above)

3. SVN access requires output from the following command:

    htdigest -c password.tmp "Tor subversion repository" <username>
   

4. The output should be mailed to the subversion service maintainer (See Infrastructure Page on trac) with Trac ticket reference contained in the email.

5. The user will be added and emailed when access is granted.

6. The trac ticket is updated and closed.

This documentation is primarily aimed at sysadmins and establishes various procedures not necessarily associated with a specific service.

Pages are grouped by some themes to make them easier to find in this page.

Accessing servers:

• cumin
• fabric
• yubikey

User access management:

• create-a-new-user
• new-person
• rename-a-user
• retire-a-user

Machine management:

• DRBD
• incident-response
• lvm
• nftables
• raid
• rename-a-host
• retire-a-host
• new-machine-cymru
• new-machine-hetzner-cloud
• new-machine-hetzner-robot
• new-machine
• new-machine-mandos
• new-machine-ovh-cloud
• reboots
• upgrades

Other misc. documentation:

• apu
• benchmark
• build_and_upload_debs
• git
• ipv6
• iscsi
• keyboard
• lektor
• openpgp
• time
• vacation

The APUs are neat little devices from PC Engines. We use them as jump hosts and, generally, low-power servers where we need them.

This documentation was written with a APU3D4, some details may vary with other models.

• Tutorial
• How to
  • Console access
  • BIOS
  • Memory test
  • Pager playbook
  • Disaster recovery
• Reference
  • Installation
  • Upgrades
  • SLA
  • Design and architecture
  • Services
  • Storage
  • Queues
  • Interfaces
    • Serial console
    • LEDs
    • Network
  • Authentication
  • Implementation
  • Related services
  • Issues
  • Maintainer
  • Users
  • Upstream
  • Monitoring and metrics
  • Tests
  • Logs
  • Backups
  • Other documentation
• Discussion
  • Overview
  • Security and risk assessment
  • Technical debt and next steps
  • Proposed Solution
  • Other alternatives
    • APU hardware
    • APU EOL and alternatives

Tutorial

How to

Console access

The APU comes with a DB-9 serial port. You can connect to that port using, typically, a null modem cable and a serial-to-USB adapter. Once properly connected, the device will show up as /dev/ttyUSB0 on Linux. You can connect to it with GNU screen with:

screen /dev/ttyUSB0 115200

... or with plain cu(1):

cu -l /dev/ttyUSB0 -s 115200

If you fail to connect, PC Engines actually has minimalist but good documentation on the serial port.

BIOS

When booting, you should be able to see the APU's BIOS on the serial console. It looks something like this after a few seconds:

PCEngines apu3
coreboot build 20170302
4080 MB ECC DRAM

SeaBIOS (version rel-1.10.0.1)

Press F10 key now for boot menu

The boot menu then looks something like that:

Select boot device:

1. USB MSC Drive Kingston DataTraveler 3.0 
2. SD card SD04G 3796MiB
3. ata0-0: SATA SSD ATA-9 Hard-Disk (111 GiBytes)
4. Payload [memtest]
5. Payload [setup]

Hitting 4 puts you in a Memtest86 memory test (below). The setup screen looks like this:

### PC Engines apu2 setup v4.0.4 ###
Boot order - type letter to move device to top.

  a USB 1 / USB 2 SS and HS 
  b SDCARD 
  c mSATA 
  d SATA 
  e iPXE (disabled)


  r Restore boot order defaults
  n Network/PXE boot - Currently Disabled
  t Serial console - Currently Enabled
  l Serial console redirection - Currently Enabled
  u USB boot - Currently Enabled
  o UART C - Currently Disabled
  p UART D - Currently Disabled
  x Exit setup without save
  s Save configuration and exit

i.e. it basically allows you to change the boot order, enable network booting, disable USB booting, disable the serial console (probably ill-advised), and mess with the other UART ports.

The network boot actually drops you in iPXE which is nice (version 1.0.0+ (f8e167) from 2016) as it allows you to bootstrap one rescue host with another (see the installation section below).

Memory test

The boot menu (F10 then 4) provides a built-in memory test which runs Memtest86 5.01+ and looks something like this:

Memtest86+ 5.01 coreboot 001| AMD GX-412TC SOC                               
CLK: 998.3MHz  (X64 Mode)   | Pass  6% ##
L1 Cache:   32K  15126 MB/s | Test 67% ##########################              
L2 Cache: 2048K   5016 MB/s | Test #5  [Moving inversions, 8 bit pattern]     
L3 Cache:  None             | Testing: 2048M - 3584M   1536M of 4079M
Memory  : 4079M   1524 MB/s | Pattern:   dfdfdfdf           | Time:   0:03:49
------------------------------------------------------------------------------
Core#: 0 (SMP: Disabled)  |  CPU Temp  | RAM: 666 MHz (DDR3-1333) - BCLK: 100
State: - Running...       |    48 C    | Timings: CAS 9-9-10-24 @ 64-bit Mode
Cores:  1 Active /  1 Total (Run: All) | Pass:       0        Errors:      0  
------------------------------------------------------------------------------







                                                            
                                                               



                                PC Engines APU3
(ESC)exit  (c)configuration  (SP)scroll_lock  (CR)scroll_unlock (l)refresh

Pager playbook

Disaster recovery

Reference

Installation

The current APUs were ordered directly from the PC Engines shop, specifically the USD section. The build was:

    2 apu3d4   144.00 USD 288.00  HTS 8471.5000     TW Weight    470g
      APU.3D4 system board 4GB

    2 case1d2redu 10.70 USD  21.40  HTS 8473.3000     CN Weight    502g
      Enclosure 3 LAN, red, USB

    2 ac12vus2 4.40 USD   8.80  HTS 8504.4000     KH Weight    266g
      AC adapter 12V US plug for IT equipment

    2 msata120c 15.50 USD  31.00  HTS 8523.5100     CN Weight     14g
      SSD M-Sata 120GB TLC

    2 sd4b     6.90 USD  13.80  HTS 8523.5100     TW Weight      4g
      SD card 4GB pSLC Phison

    2 assy2    7.50 USD  15.00  HTS 8471.5000     CH Weight    120g
      assembly + box

Shipping TBD !!!       USD   0.00    Weight   1376g
VAT                    USD   0.00

Total                  USD 378.00

Note how the price is for two complete models. The devices shipped promptly; it was basically shipped in 3 days, but customs added an additional day of delay over the weekend, which led to a 6 days (4 business days) shipping time.

One of the machine was connected over serial (see above) and booted with a GRML "96" (64 and 32 bit) image over USB. Booting GRML from USB is tricky, however, because you need to switch from 115200 to 9600 once grub finishes loading, as GRML still defaults to 9600 baud instead of 115200. It may be possible to tweak the GRUB commandline to change the speed, but since it's in the middle of the kernel commandline and that the serial console editing capabilities are limited, it's actually pretty hard to get there.

The other box was chain-loaded with iPXE from the first box, as a stress-test. This was done by enabling the network boot in the BIOS (F10 to enter the BIOS in the serial console, then 5 to enter setup and n to enable network boot and s to save). Then hit n to enable network boot and choose "iPXE shell" when prompted. Assuming both hosts are connected over their eth1 storage interfaces, you should then do:

iPXE> dhcp net1
iPXE> chain autoexec.ipxe

This will drop you in another DHCP sequence, which will try to configure each interface. You can control-c to skip net0 and then the net1 interface will self-configure and chain-load the kernel and GRML. Because the autoexec.ipxe stores the kernel parameters, it will load the proper serial console settings and doesn't suffer from the 9600 bug mentioned earlier.

From there, SSH was setup and key was added. We had DHCP in the lab so we just reused that IP configuration.

service ssh restart
cat > ~/.ssh/authorized_keys
...

Then the automated installer was fired:

./install -H root@192.168.0.145 \
          --fingerprint 3a:4d:dd:91:79:af:4e:c4:17:e5:c8:d2:d6:b5:92:51   \
          hetzner-robot \
          --fqdn=dal-rescue-01.torproject.org \
          --fai-disk-config=installer/disk-config/dal-rescue \
          --package-list=installer/packages \
          --post-scripts-dir=installer/post-scripts/ \
          --ipv4-address 204.8.99.100 \
          --ipv4-subnet 24 \
          --ipv4-gateway 204.8.99.1

WARNING: the dal-rescue disk configuration is incorrect. The 120GB disk gets partitioned incorrectly, as its RAID-1 partition is bigger than the smaller SD card.

Note that IP configuration was actually performed manually on the node, the above is just an example of the IP address used by the box.

Next, the new-machine procedure was followed.

Finally, the following steps need to be performed to populate /srv:

• GRML image, note that we won't be using the grml.ipxe file, so:

   apt install debian-keyring &&
   wget https://download.grml.org/grml64-small_2022.11.iso &&
   wget https://download.grml.org/grml64-small_2022.11.iso.asc &&
   gpg --verify --keyring /usr/share/keyrings/debian-keyring.gpg grml64-small_2022.11.iso.asc &&
   echo extracting vmlinuz and initrd from ISO... &&
   mount grml64-small_2022.11.iso /mnt -o loop &&
   cp /mnt/boot/grml64small/* . &&
   umount /mnt &&
   ln grml64-small_2022.11.iso grml.iso
  

• build the iPXE image but without the floppy stuff, basically:

apt install build-essential &&
git clone git://git.ipxe.org/ipxe.git &&
cd ipxe/src &&
mkdir config/local/tpa/ &&
cat > config/local/tpa/general.h <<EOF
#define DOWNLOAD_PROTO_HTTPS	/* Secure Hypertext Transfer Protocol */
#undef NET_PROTO_STP		/* Spanning Tree protocol */
#undef NET_PROTO_LACP		/* Link Aggregation control protocol */
#undef NET_PROTO_EAPOL		/* EAP over LAN protocol */
#undef CRYPTO_80211_WEP	/* WEP encryption (deprecated and insecure!) */
#undef CRYPTO_80211_WPA	/* WPA Personal, authenticating with passphrase */
#undef CRYPTO_80211_WPA2	/* Add support for stronger WPA cryptography */
#define NSLOOKUP_CMD		/* DNS resolving command */
#define TIME_CMD		/* Time commands */
#define REBOOT_CMD		/* Reboot command */
#define POWEROFF_CMD		/* Power off command */
#define PING_CMD		/* Ping command */
#define IPSTAT_CMD		/* IP statistics commands */
#define NTP_CMD		/* NTP commands */
#define CERT_CMD		/* Certificate management commands */
EOF
make -j4 CONFIG=tpa bin-x86_64-efi/ipxe.efi bin-x86_64-pcbios/undionly.kpxe

• copy the iPXE files in /srv/tftp:

   cp bin-x86_64-efi/ipxe.efi bin-x86_64-pcbios/undionly.kpxe /srv/tftp/
  

• create a /srv/tftp/autoexec.ipxe:

#!ipxe

dhcp
kernel http://172.30.131.1/vmlinuz
initrd http://172.30.131.1/initrd.img
initrd http://172.30.131.1/grml.iso /grml.iso
imgargs vmlinuz initrd=initrd.magic boot=live config fromiso=/grml.iso live-media-path=/live/grml64-small noprompt noquick noswap console=tty0 console=ttyS1,115200n8
boot

Upgrades

SLA

Design and architecture

Services

Storage

Queues

Interfaces

Serial console

The APU should provide a serial console access over the DB-9 serial port, standard 115200 baud. The install is configured to offer the bootloader and a login prompt over the serial console, and a basic BIOS is also available.

LEDs

The APU has no graphical interface (only serial, see above), but there are LEDs in the front that have been configured from Puppet to make systemd light them up in a certain way.

From left to right, when looking at the front panel of the APU (not the one with the power outlets and RJ-45 jacks):

1. The first LED lights up when the machine boots, and should be on when the LUKS prompt waits. then it briefly turns off when the kernel module loads and almost immediately turns back on when filesystems are mounted (DefaultDependencies=no and After=local-fs.target)
2. The second LED lights up when systemd has booted and has quieted (After=multi-user.target and Type=idle)
3. The third LED should blink according to the "activity" trigger which is defined in ledtrig_activity kernel module

Network

The three network ports should be labeled according to which VLAN they are supposed to be configured for, see the Quintex network layout for details on that configuration.

From left to right, when looking at the back panel of the APU (the one with the network ports, after the DB-9 serial port):

1. eth0 public: public network interface, to be hooked up to the public VLAN, mapped to eth0 in Linux

2. eth1 storage: private network interface, to be hooked up to the storage VLAN and where DHCP and TFTP is offered, mapped to eth1 in Linux

3. eth2 OOB: private network interface, to be hooked up to the OOB ("Out Of Band" management) VLAN, to allow operators to access the OOB interfaces of the other servers

Authentication

Implementation

Related services

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker with the label ~Foo.

Maintainer

Users

Upstream

Monitoring and metrics

Tests

Logs

Backups

Other documentation

Discussion

Overview

Security and risk assessment

Technical debt and next steps

Proposed Solution

Other alternatives

APU hardware

We also considered a full 1U case but that seemed really costly. We have also considered a HDD enclosure but that didn't seem necessary either.

APU EOL and alternatives

As of 2023-04-18, the PC Engines website has a stronger EOL page that explicitly states that "The end is near!" and that:

Despite having used considerable quantities of AMD processors and Intel NICs, we don't get adequate design support for new projects. In addition, the x86 silicon currently offered is not very appealing for our niche of passively cooled boards. After about 20 years of WRAP, ALIX and APU, it is time for me to move on to different things.

It therefore seems unlikely that new PC Engines product will be made in the future, and that platform should be considered dead.

In our initial research (tpo/tpa/team#41058) we found two other options (the SolidRun and Turris, below), but since then we've expanded the search and we're keeping a list of alternatives here.

The specification is, must have:

• small (should fit in a 1U)
• low power (10-50W max)
• serial port or keyboard and monitor support
• at least three network ports
• 3-4GB storage for system (dal-rescue-02 uses 2.1GB as of this writing)
• 1-5GB storage for system images (dal-rescue-02 uses 1GB)

Nice to have:

• faster than the APU3 (AMD GX-412TC SOC 600MHz)
• rack-mountable
• coreboot
• "open hardware"
• 12-24 network ports (yes, that means it's a switch, and that we don't need an extra OOB switch)

Other possibilities:

• SolidRun Honeycomb: can fit two ARM servers in a 1U case, SFP ports, 64GB RAM, 16-core NXP 2GHz, a bit overkill
• Turris Shield: SOHO firewall appliance, not sure about Debian compatibility
• Qotom: rugged devices, has a 1U form factor, no coreboot, no price listing
• Protectli: rugged, coreboot, fanless, 2-6 2.5gbps NICs, Intel quad core, 8-64GB RAM, DP port option
• OPNsense: "network appliances", rackmountable, costly (700$+)
• Fitlet: really rugged, miniature, fanless, lock-in power input, 4x 2.5gbps, 2 mini-HDMI, serial port, quad core Intel Atom, mSATA SSD, coreboot, 400$ without RAM (DDR3L-1600)
• Ten64: crowdfunding project, shipping, 8 gigabit ports, 2 SFP, 8 core ARM, 10W, up to 32GB DDR SO-DIMM, 256MB onboard flash, 700$ before RAM 2xNVMe, mini PCIe for wifi, LTE, SATA, SIM tray, 700$USD before RAM
• NUCs like the Gigabyte's Brix, Beelink could be an option as well, no coreboot, slightly thicker (more than 1U?)
• there's of course a whole cornucopia of SBC (Single Board Computers) out there, e.g. Minnowboard, EspressoBIN, Banana PI, MACCHIATObin, O-DROID, Olimex, Pine64, protectli, UP, and many more (see hackboards.com for a database and this HN discussion for other comments)
• see also this list from anarcat

This procedure documents various benchmarking procedures in use inside TPA.

• HTTP load testing
  • Common procedure
  • Siege
  • apachebench
  • bombardier
  • wrk
  • Other tools

HTTP load testing

Those procedures were quickly established to compare various caching software as part of the cache service setup.

Common procedure

1. punch a hole in the firewall to allow the test server to access tested server, in case it is not public yet

   iptables -I INPUT -s 78.47.61.104 -j ACCEPT
   ip6tables -I INPUT -s 2a01:4f8:c010:25ff::1 -j ACCEPT
   

2. point the test site (e.g. blog.torproject.org) to the tested server on the test server, in /etc/hosts:

   116.202.120.172	blog.torproject.org
   2a01:4f8:fff0:4f:266:37ff:fe26:d6e1 blog.torproject.org
   

3. disable Puppet on the test server:

   puppet agent --disable 'benchmarking requires /etc/hosts override'
   

4. launch the benchmark on the test server

Siege

Siege configuration sample:

verbose = false
fullurl = true
concurrent = 100
time = 2M
url = http://www.example.com/
delay = 1
internet = false
benchmark = true

Might require this, which might work only with varnish:

proxy-host = 209.44.112.101
proxy-port = 80

Alternative is to hack /etc/hosts.

apachebench

Classic commandline:

ab2 -n 1000 -c 100 -X cache01.torproject.org https://example.com/

-X also doesn't work with ATS, modify /etc/hosts instead.

bombardier

We tested bombardier as an alternative to go-wrk in previous benchmarks. The goal of using go-wrk was that it supported HTTP/2 (while wrk didn't), but go-wrk had performance issues, so we went with the next best (and similar) thing.

Unfortunately, the bombardier package in Debian is not the HTTP benchmarking tool but a commandline game. It's still possible to install it in Debian with:

export GOPATH=$HOME/go
apt install golang
go get -v github.com/codesenberg/bombardier

Then running the benchmark is as simple as:

./go/bin/bombardier --duration=2m --latencies https://blog.torproject.org/

wrk

Note that wrk works similarly to bombardier, sampled above, and has the advantage of being already packaged in Debian. Simple cheat sheet:

sudo apt install wrk
echo "10.0.0.0 target.example.com" >> /etc/hosts
wrk --latency -c 100 --duration 2m https://target.example.com/

The main disadvantage is that it doesn't (seem to) support HTTP/2 or similarly advanced protocols.

Other tools

Siege has trouble going above ~100 concurrent clients because of its design (and ulimit) limitations. Its interactive features are also limited, here's a set of interesting alternatives:

Note that the Proto(col) and Features columns are not exhaustive: a tool might support (say) HTTPS, HTTP/2, or HTTP/3 even if it doesn't explicitly mention it, although it's unlikely.

It should be noted that very few (if any) benchmarking tools seem to support HTTP/3 (or even QUIC) at this point. Even HTTP/2 support is spotty: for example, while bombardier supports HTTP/2, it only does so with the slower net/http library at the time of writing (2021). It's unclear how many (if any) other projects to support HTTP/2 as well.

More tools, unreviewed:

• https://github.com/ddosify/ddosify

Builds can be performed on dixie.torproject.org.

Uploads must be go to palmeri.torproject.org.

Preliminary setup

In ~/.ssh/config:

Host dixie.torproject.org
        ProxyCommand ssh -4 perdulce.torproject.org -W %h:%p

In ~/.dput.cf:

[tor]
login = *
fqdn = palmeri.torproject.org
method = scp
incoming = /srv/deb.torproject.org/incoming

Currently available distributions

• Debian:
  • lenny-backport
  • experimental-lenny-backport
  • squeeze-backport
  • experimental-squeeze-backport
  • wheezy-backport
  • experimental-wheezy-backport
  • unstable
  • experimental
• Ubuntu:
  • hardy-backport
  • lucid-backport
  • experimental-lucid-backport
  • natty-backport
  • experimental-natty-backport
  • oneiric-backport
  • experimental-oneiric-backport
  • precise-backport
  • experimental-precise-backport
  • quantal-backport
  • experimental-quantal-backport
  • raring-backport
  • experimental-raring-backport

Create source packages

Source packages must be created for the right distributions.

Helper scripts:

• backports
• build-tor-sources

Build packages

Upload source packages to dixie:

dcmd rsync -v *.dsc dixie.torproject.org:

Build arch any packages:

ssh dixie.torproject.org
for i in *.dsc; do ~weasel/bin/sbuild-stuff $i && linux32 ~weasel/bin/sbuild-stuff --binary-only $i || break; done

Or build arch all packages:

ssh dixie.torproject.org
for i in *.dsc; do ~weasel/bin/sbuild-stuff $i || break; done

Packages with dependencies in deb.torproject.org must be built using $suite-debtpo-$arch-sbuild, e.g. by running:

DIST=wheezy-debtpo ~weasel/bin/sbuild-stuff $DSC

Retrieve build results:

rsync -v $(ssh dixie.torproject.org dcmd '*.changes' | sed -e 's/^/dixie.torproject.org:/') .

Upload first package with source

Pick the first changes file and stick the source in:

changestool $CHANGES_FILE includeallsources

Sign it:

debsign $CHANGES_FILE

Upload:

dput tor $CHANGES_FILE

Start a first dinstall:

ssh -t palmeri.torproject.org sudo -u tordeb /srv/deb.torproject.org/bin/dinstall

Move changes file out of the way:

dcmd mv $CHANGES_FILE archives/

Upload other builds

Sign the remaining changes files:

debsign *.changes

Upload them:

dput tor *.changes

Run dinstall:

ssh -t palmeri.torproject.org sudo -u tordeb /srv/deb.torproject.org/bin/dinstall

Archive remaining build products:

dcmd mv *.changes archives/

Uploading admin packages

There is a separate Debian archive, on db.torproject.org, which can be used to upload packages specifically designed to run on torproject.org infrastructure. The following .dput.cf should allow you to upload built packages to the server, provided you have the required accesses:

[tpo-admin]
fqdn = db.torproject.org
incoming = /srv/db.torproject.org/ftp-archive/archive/pool/tpo-all/
method = sftp
post_upload_command = ssh root@db.torproject.org make -C /srv/db.torproject.org/ftp-archive

This might require fixing some permissions. Do a chmod g+w on the broken directories if this happens. See also ticket 34371 for plans to turn this into a properly managed Debian archive.

• Configuration
• Creating a new user
  • on the LDAP server
  • on other servers
  • Creating a user without a PGP key
  • troubleshooting
• Creating a role
• Sudo configuration
  • Sudo primer
  • Granting access to a role account
• Update a user's GPG key
• Other documentation

This document explains how to create new shell (and email) accounts. See also doc/accounts to evaluate new account requests.

Note that this documentation needs work, as it overlaps with user-facing user management procedures (doc/accounts), see issue 40129.

Configuration

This should be done only once.

git clone db.torproject.org:/srv/db.torproject.org/keyrings/keyring.git account-keyring

It downloads the git repository that manages the OpenPGP keyring. This keyring is essential as it allows users to interact with the LDAP database securely to perform password changes and is also used to send the initial password for new accounts.

When cloning, you may get the following message (see tpo/tpa/team#41785):

fatal: detected dubious ownership in repository at '/srv/db.torproject.org/keyrings/keyring.git'

If this happens, you need to run the following command as your user on db.torproject.org:

git config --global --add safe.directory /srv/db.torproject.org/keyrings/keyring.git

Creating a new user

This procedure can be used to create a real account for a human being. If this is for a machine or another automated thing, create a role account (see below).

To create a new user, specific information need to be provided by the requester, as detailed in doc/accounts.

The short version is:

1. Import the provided key to your keyring. That is necessary for the script in the next point to work.

2. Verify the provided OpenPGP key

   It should be signed by a trusted key in the keyring or in a message signed by a trusted key. See doc/accounts when unsure.

3. Add the OpenPGP key to the account-keyring.git repository and create the LDAP account:

   FINGERPRINT=0123456789ABCDEF0123456789ABCDEF01234567 &&
   NEW_USER=alice &&
   REQUESTER="bob in ticket #..." &&
   ./NEW "$FINGERPRINT" "$NEW_USER" &&
   git add torproject-keyring/"${NEW_USER}-${FINGERPRINT}.gpg" &&
   git commit -m"new user ${NEW_USER} requested by ${REQUESTER}" &&
   git push &&
   ssh -tt $USER@alberti.torproject.org "ud-useradd -n && sudo -u sshdist ud-generate && sudo -H ud-replicate"
   

The last line will create the user on the LDAP server. See below for detailed information on that magic instruction line, including troubleshooting.

Note that $USER, in the above, shouldn't be explicitly expanded unless your local user is different from your alberti user. In my case, $USER, locally, is anarcat and that is how I login to alberti as well.

Notice that when prompted for whom to add (a GPG search), enter the full $FINGERPRINT verified above

What followed are detailed, step-by-step instructions, to be performed after the key was added to the account-keyring.git repository (up to the git push step above).

on the LDAP server

Those instructions are a copy of the last step of the above instructions, provided to clarify what each step does. Do not follow this procedure and instead follow the above.

The LDAP server is currently alberti. Those steps are supposed to be ran as a regular user with LDAP write access.

1. create the user:

   ud-useradd -n
   

   This command asks a bunch of questions interactively that have good defaults, mostly taken from the OpenPGP key material, but it's important to review them anyways. in particular:

   • when prompted for whom to add (a GPG search), enter the full $FINGERPRINT verified above

   • the email forward is likely to be incorrect if the key has multiple email address as UIDs

   • the user might already be present in the Postfix alias file (tor-puppet/modules/postfix/files/virtual) - in that case, use that email as the Email forwarding address if present and remove it from Puppet

2. synchronize the change:

    sudo -u sshdist ud-generate && sudo -H ud-replicate
   

on other servers

This step is optional and can be used to force replication of the change to another server manually.

1. synchronize the change:

   sudo -H ud-replicate
   

2. run puppet:

   sudo puppet agent -t
   

Creating a user without a PGP key

In most cases we want to use the person's PGP key to associate with their new LDAP account, but in some cases it may be difficult to get a person to generate a PGP key (and most importantly, keep managing that key effectively afterwards) and we might still want to grant the person an email account.

For those cases, it's possible to create an LDAP account without associating it to a PGP key.

First, generate a password and note it down somewhere safe temporarily. Then generate a hash for that password and noted it down. If you don't have this command on your computer, you can run that on alberti:

mkpasswd -m bcrypt-a

On alberti, find a free user ID with fab user.list-gaps (more information on that command in the creating a role section)

Then, on alberti, connect to ldapvi and at the end of the file add something like the following. Make sure to modify uid=[...] and all UID and GID numbers and then the user's cn and sn fields to values that make sense for your case and replace the value of mailPassword with the password hash you noted down earlier. Keep the userPassword as-is since it will tell LDAP to lock the LDAP account:

add gid=exampleuser,ou=users,dc=torproject,dc=org
gid: exampleuser
gidNumber: 15xx
objectClass: top
objectClass: debianGroup

add uid=exampleuser,ou=users,dc=torproject,dc=org
uid: exampleuser
objectClass: top
objectClass: inetOrgPerson
objectClass: debianAccount
objectClass: shadowAccount
objectClass: debianDeveloper
uidNumber: 15xx
gidNumber: 15xx
gecos: exampleuser,,,,
cn: Example
sn: User
userPassword: {crypt}$LK$
mailPassword: <REDACTED>
emailForward: <address>
loginShell: /bin/bash
mailCallout: FALSE
mailContentInspectionAction: reject
mailGreylisting: FALSE
mailDefaultOptions: FALSE

Save and exit and you should get prompted about adding two entries.

Lastly, refresh and resync the user database:

• On alberti: sudo -u sshdist ud-generate && sudo -H ud-replicate
• On submit-01 as root: ud-replicate

The final step is then to contact the person on Signal and send them the password in a disappearing message.

troubleshooting

If the ud-useradd command fails with this horrible backtrace:

Updating LDAP directory..Traceback (most recent call last):
  File "/usr/bin/ud-useradd", line 360, in <module>
    lc.add_s(Dn, Details)
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 236, in add_s
    return self.add_ext_s(dn,modlist,None,None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 222, in add_ext_s
    resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout)
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 543, in result3
    resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4(
                                                                           ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 553, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 128, in _ldap_call
    result = func(*args,**kwargs)
             ^^^^^^^^^^^^^^^^^^^^
ldap.INVALID_SYNTAX: {'msgtype': 105, 'msgid': 6, 'result': 21, 'desc': 'Invalid syntax', 'ctrls': [], 'info': 'sn: value #0 invalid per syntax'}

... it's because you didn't fill the form properly. In this case, the sn field ("Last name" in the form) was empty. If you don't have a second name, just reuse the first name.

Creating a role

A "role" account is like a normal user, except it's for machines or services, not real people. It's useful to run different services with different privileges and isolation.

Here's how to create a role account:

0. Do not use ud-groupadd and ud-roleadd. They are partly broken.

1. Run fab user.list-gaps from a clone of the fabric-tasks repository on alberti.tpo to find an unused uidNumber/gidNumber pair.

   • Make sure the numbers match. If you are unsure, find the highest uidNumber / gidNumber pair, increment that and use it as a number. You must absolutely make sure the number is not already in use.
   • the fabric task connects directly to ldap, which is firewalled from the exterior, so you won't be able to run the task from your computer.

2. On LDAP host (currently alberti.tpo), as a user with LDAP write access, do:

    ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D uid=${USER},ou=users,dc=torproject,dc=org
   

3. Create a new group role for the new account:

   • Copy-paste a previous gid that is also a debianGroup
   • Change the first word of the copy-pasted block to add instead of the integer
   • Change the cn (first line) to the new group name
   • Change the gid: field (last line) to the new group name
   • Set the gidNumber to the number found in step 2

4. Create the actual user role:

   • Copy-paste a previous uid role entry (with a objectClass: debianRoleAccount).
   • Change the first word of the copy-pasted block to add instead of the integer
   • Change the uid=, uid:, gecos: and cn: lines.
   • Set the gidNumber and uidNumber to the number found in step 2
   • If you need to set a mail password you can generate a blowcrypt password with python (search for example of how to do this). Change the hash identifier to $2y$ instead of $2b$.

5. Add the role to the right host:

   • Add a allowedGroups: NEW-GROUP line to host entries that should have this role account deployed.
   • If the role account will only be used for sending out email by connecting to submission.torproject.org, the account does not need to be added to a host.

6. Save the file, and accept the changes

7. propagate the changes from the LDAP host:

    sudo -u sshdist ud-generate && sudo -H ud-replicate
   

8. (sometimes) create the home directory on the server, in Puppet:

    file { '/home/bridgescan':
      ensure => 'directory',
      mode   => '0755',
      owner  => 'bridgescan',
      group  => 'bridgescan';
    }
   

Sometimes a role account is made to start services, see the doc/services page for instructions on how to do that.

Sudo configuration

A user will often need to more permissions than its regular scope. For example, a user might need to be able to access a specific role account, as above, or run certain commands as root.

We have sudo configuration that enable us to give piecemeal accesses like this. We often give accesses to groups instead of specific users for easier maintenance.

Entries should be added by declaring a sudo::conf resource in the relevant profile class in Puppet. For example:

sudo::conf { 'onbasca':
  content =>  @(EOT)
	# This file is managed by Puppet.
	%onbasca     ALL=(onbasca)      ALL
	| EOT
}

An alternative to this which avoids the need to create a profile class containing a single sudo::conf resource is to add the configuration to Hiera data. The equivalent for the above would be placing this YAML snippet at the role (preferably) or node hierarchy:

profile::sudo::configs:
  onbasca:
    content: |
      # This file is managed by Puppet.
      %onbasca     ALL=(onbasca)      ALL

Sudo primer

As a reminder, the sudoers file syntax can be distilled to this:

FROMWHO HOST=(TOWHO) COMMAND

For example, this allows the group wheel (FROMWHO) to run the service apache reload COMMAND as root (TOWHO) on the HOST example:

%wheel example=(root) service apache reload

The HOST, TOWHO and COMMAND entries can be set to ALL. Aliases can also be defined and many more keywords. In particular, the NOPASSWD: prefix before a COMMAND will allow users to sudo without entering their password.

Granting access to a role account

That being said, you can simply grant access to a role account by adding users in the role account's group (through LDAP) then adding a line like this in the sudoers file:

%roleGroup example=(roleAccount) ALL

Multiple role accounts can be specified. This is a real-world example of the users in the bridgedb group having full access to the bridgedb and bridgescan user accounts:

%bridgedb		polyanthum=(bridgedb,bridgescan)			ALL

Another real-world example, where members of the %metrics group can run two different commands, without password, on the STATICMASTER group of machines, as the mirroradm user:

%metrics		STATICMASTER=(mirroradm)	NOPASSWD: /usr/local/bin/static-master-update-component onionperf.torproject.org, /usr/local/bin/static-update-component onionperf.torproject.org

Update a user's GPG key

The account-keyring repository contains an update script ./UPDATE which takes the ldap username as argument and automatically updates the key.

If you /change/ a user's key (to a new primary key), you also need to update the user's keyFingerPrint attribute in LDAP.

After updating a key in the repository, the changes must be pushed to the remote hosted on the LDAP server.

Other documentation

Note that a lot more documentation about how to manage users is available in the LDAP documentation.

Cumin

Cumin is a tool to operate arbitrary shell commands on service/puppet hosts that match a certain criteria. It can match classes, facts and other things stored in the PuppetDB.

It is useful to do adhoc or emergency changes on a bunch of machines at once. It is especially useful to run Puppet itself on multiple machines at once to do progressive deployments.

It should not be used as a replacement for Puppet itself: most configuration on server should not be done manually and should instead be done in Puppet manifests so they can be reproduced and documented.

• Installation
  • Debian package
  • Initial configuration
  • Automatic tunneling to puppetdb with bash + systemd unit
  • Virtualenv / pip
• Avoiding spurious connection errors by limiting batch size
• Example commands
• Mangling host lists for Cumin consumption
• Disabling touch confirmation
• Discussion
  • Alternatives considered

Installation

Debian package

cumin has been available through debian archives since boorkworm, so you can simply:

sudo apt install cumin

If your distro does not have packages available, you can also install with a python virtualenv. See the section below for how to achieve this.

Initial configuration

cumin is relatively useless for us if it doesn't poke puppetdb to resolve which hosts to run commands on. So we want to get it to talk to puppetdb. Also, it gets pretty annoying to have to manually setup the ssh tunnel after getting an error printed out by cumin, so we can get the tunnel setup automatically.

Once cumin is installed drop the following configuration in ~/.config/cumin/config.yaml:

transport: clustershell
puppetdb:
    host: localhost
    scheme: http
    port: 6785
    api_version: 4  # Supported versions are v3 and v4. If not specified, v4 will be used.
clustershell:
    ssh_options:
        - '-o User=root'
log_file: cumin.log
default_backend: puppetdb

Now you can simply use an alias like the following:

alias cumin="cumin --config ~/.config/cumin/config.yaml"

while making sure that you setup an ssh tunnel manually before calling cumin like the following:

ssh -L6785:localhost:8080 puppetdb-01.torproject.org

Or instead of the alias and the ssh command, you can try setting up an automatic tunnel upon calling cumin. See the following section to set that up.

Automatic tunneling to puppetdb with bash + systemd unit

This trick makes sure that you never forget to setup the ssh tunnel to puppedb before running cumin. This section will replace cumin by a bash function, so if you created a simple alias like mentioned in the previous section, you should start by getting rid of that alias. Lastly, this trick requires nc in order to verify if the tunnel port is open so, install it with:

sudo apt install nc

To get the automatic tunnel, we'll create a systemd unit that can bring the tunnel up for us. Create the file ~/.config/systemd/user/puppetdb-tunnel@.service, making sure to create the missing directories in the path:

[Unit]
Description=Setup port forward to puppetdb
After=network.target

[Service]
ExecStart=-/usr/bin/ssh -W localhost:8080 puppetdb-01.torproject.org
StandardInput=socket
StandardError=journal
Environment=SSH_AUTH_SOCK=%t/gnupg/S.gpg-agent.ssh

The Environment variable is necessary for the ssh command to be able to request the key from your YubiKey, this may vary according to your authentication system. It's only there because systemd might not have the right variables from your environment, depending on how it's started.

And you'll need the following for socket activation, in ~/.config/systemd/user/puppetdb-tunnel.socket:

[Unit]
Description=Socket activation for PuppetDB tunnel
After=network.target

[Socket]
ListenStream=127.0.0.1:6785
Accept=yes

[Install]
WantedBy=graphical-session.target

With this in place, make sure that systemd has loaded this unit file:

systemctl --user daemon-reload
systemctl --user enable --now puppetdb-tunnel.socket

Note: if you already have a line like LocalForward 8080 127.0.0.1:8080 under a block for host puppetdb-01.torproject.org in your ssh configuration, it will cause problem as ssh will try to bind to the same socket as systemd. That configuration should be removed.

The above can be tested by hand without creating any systemd configuration with:

systemd-socket-activate -a --inetd  -E SSH_AUTH_SOCK=/run/user/1000/gnupg/S.gpg-agent.ssh -l 127.0.0.1:6785 \
    ssh -o BatchMode=yes -W localhost:8080 puppetdb-01.torproject.org

The tunnel will be shutdown as soon as it's done, and fired up as needed. You will need to tap your YubiKey, as normal, to get it to work of course.

This is different from a -N "daemon" configuration where the daemon stays around for a long-lived connection. This is the only way we've found to make it work with socket activation. The alternative to that is to use a "normal" service that is not socket activated and start it by hand:

[Unit]
Description=Setup port forward to puppetdb
After=network.target

[Service]
ExecStart=/usr/bin/ssh -nNT -o ExitOnForwardFailure=yes -o BatchMode=yes -L 6785:localhost:8080 puppetdb-01.torproject.org
Environment=SSH_AUTH_SOCK=/run/user/1003/gnupg/S.gpg-agent.ssh

Virtualenv / pip

If Cumin is not available from your normal packages (see bug 924685 for Debian), you must install it in a Python virtualenv.

First, install dependencies, Cumin and some patches:

sudo apt install python3-clustershell python3-pyparsing python3-requests python3-tqdm python3-yaml
python3 -m venv --system-site-packages ~/.virtualenvs/cumin
~/.virtualenvs/cumin/bin/pip3 install cumin
~/.virtualenvs/cumin/bin/pip3 uninstall tqdm pyparsing clustershell # force using trusted system packages

Now if you follow the initial setup section above, then you can either create an alias in the following way:

alias cumin="~/.virtualenvs/cumin/bin/cumin --config ~/.config/cumin/config.yaml"

Or you can instead use the automatic ssh tunnel trick above, making sure to change the path to cumin in the bash function.

Avoiding spurious connection errors by limiting batch size

If you use cumin to run ad-hoc commands on many hosts at once, you'll most probably want to look into setting yourself up for direct connection to the hosts, instead of passing through a jump host.

Without the above-mentioned setup, you'll quickly hit a problem where hosts give you seemingly random ssh connection errors for a variable percentage of the host list. This is because you are hitting ssh server limitations imposed on you on the jump host. The ssh server uses the default value for its MaxStartups option, which means once you have 10 simultaneous open connections you'll start seeing connections dropped with a 30% chance.

Again, it's recommended in this case to set yourself up for direct ssh connection to all of the hosts. But if you are not in a position where this is possible and you still need to go through the jump host, you can avoid weird issues by limiting your batch size to 10 or lower, e.g.:

cumin -b 10 'F:os.distro.codename=bookworm' 'apt update'

Note however that doing this will have the following effects:

• execution of the command on all hosts will be much slower
• if some hosts see command failures, cumin will stop processing your requested commands after reaching the batch size. so your command will possibly only run on 10 of all of the hosts.

Example commands

This will run the uptime command on all hosts:

cumin '*' uptime

To run against only a subset, you need to use the Cumin grammar, which is briefly described in the Wikimedia docs. For example, this will run the same command only on physical hosts:

cumin 'F:virtual=physical' uptime

You can invert a condition by placing 'not ' in front of it. Also for facts, you can retrieve structured facts using puppet's dot notation (e.g. 'networking.fqdn' to check the fqdn fact). Using these two techniques the following example will run a command on all hosts that have not yet been upgraded to bookworm:

cumin 'not F:os.distro.codename=bookworm' uptime

To run against all hosts that have an ssl::service resource in their latest built catalog:

cumin 'R:ssl::service' uptime

To run against only the dal ganeti cluster nodes:

cumin 'C:role::ganeti::dal' uptime

Or, the same command using the O: shortcut:

cumin 'O:ganeti::dal' uptime

To query any host that applies a certain profile:

cumin 'P:opendkim' uptime

And to query hosts that apply a certain profile with specific parameters:

cumin 'P:opendkim%mode = sv' uptime

Any Puppet fact or class can be queried that way. This also serves as a ad-hoc interface to query PuppetDB for certain facts, as you don't have to provide a command. In that case, cumin runs in "dry mode" and will simply show which hosts match the request:

$ cumin 'F:virtual=physical'
16 hosts will be targeted:
[...]

Mangling host lists for Cumin consumption

Say you have a list of hosts, separated by newlines. You want to run a command on all those hosts. You need to pass the list as comma-separated words instead.

Use the paste command:

cumin "$(paste -sd, < host-list.txt)" "uptime"

Disabling touch confirmation

If running a command that takes longer than a few seconds, the cryptographic token will eventually block future connections and prompt for physical confirmation. This typically is not too much of a problem for short commands, but for long-running jobs, this can lead to timeouts if the operator is distracted.

The best way to workaround this problem is to temporarily disable touch confirmation, for example with:

ykman openpgp keys set-touch aut off
cumin '*' ': some long running command'
ykman openpgp keys set-touch aut on

Discussion

Alternatives considered

• Choria - to be evaluated
• Ansible?
• Puppet mcollective?
• simple SSH loop from LDAP output
• parallel-ssh
• see also the list in the Puppet docs

See also fabric.

DRBD is basically "RAID over the network", the ability to replicate block devices over multiple machines. It's used extensively in our service/ganeti configuration to replicate virtual machines across multiple hosts.

• How-to
  • Checking status
  • Finding device associated with host
  • Finding device associated with traffic
  • Deleting a stray device
  • Deleting a device after it was manually detached
  • Pager playbook
    • Resyncing disks
  • Upstream documentation
• Reference
  • Installation

How-to

Checking status

Just like mdadm, there's a device in /proc which shows the status of the RAID configuration. This is a healthy configuration:

# cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 9B4D87C5E865DF526864868 
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:10821208 dw:10821208 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:10485760 dw:10485760 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:1048580 dw:1048580 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Keyword: UpToDate. This is a configuration that is being resync'd:

version: 8.4.10 (api:1/proto:86-101)
srcversion: 9B4D87C5E865DF526864868 
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:9352840 dw:9352840 dr:0 al:8 bm:0 lo:1 pe:3 ua:0 ap:0 ep:1 wo:f oos:1468352
	[================>...] sync'ed: 86.1% (1432/10240)M
	finish: 0:00:36 speed: 40,436 (38,368) want: 61,440 K/sec
 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:8439808 dw:8439808 dr:0 al:8 bm:0 lo:1 pe:3 ua:0 ap:0 ep:1 wo:f oos:2045952
	[===============>....] sync'ed: 80.6% (1996/10240)M
	finish: 0:00:52 speed: 39,056 (37,508) want: 61,440 K/sec
 2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:1048580 dw:1048580 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

See the upstream documentation for details on this output.

The drbdmon command also provides a similar view but, in my opinion, less readable.

Because DRBD is built with kernel modules, you can also see activity in the dmesg logs

Finding device associated with host

In the drbd status, devices are shown by their minor identifier. For example, this is device minor id 18 having a trouble of some sort:

18: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:1237956 nr:0 dw:11489220 dr:341910 al:177 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
	[===================>] sync'ed:100.0% (0/10240)M
	finish: 0:00:00 speed: 764 (768) K/sec (stalled)

Finding which host is associated with this device is easy: just call list-drbd:

root@fsn-node-01:~# gnt-node list-drbd fsn-node-01 | grep 18
fsn-node-01.torproject.org    18 gettor-01.torproject.org          disk/0 primary   fsn-node-02.torproject.org

It's the host gettor-01. In this specific case, you can either try to figure out what's wrong with DRBD or (more easily) just change the secondary with:

gnt-instance replace-disks -n fsn-node-03 gettor-01

Finding device associated with traffic

If there's a lot of I/O (either disk or network) on a host and you're looking for the device (and therefore virtual machine, see above) associated with it, look in the DRBD dashboard, in the "Disk I/O device details" row, which will show the exact device associated with the I/O.

Then you can use the device number to find the associated virtual machine, see above.

Deleting a stray device

If Ganeti tried to create a device on one node but couldn't reach the other node (for example if the secondary IP on the other node wasn't set correctly), you will see this error in Ganeti:

   - ERROR: node chi-node-03.torproject.org: unallocated drbd minor 0 is in use

You can confirm this by looking at the /proc/drbd there:

root@chi-node-03:~# cat /proc/drbd 
version: 8.4.10 (api:1/proto:86-101)
srcversion: 473968AD625BA317874A57E 
 0: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown   r-----
    ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10485504

And confirm the device does not exist on the other side:

root@chi-node-04:~# cat /proc/drbd 
version: 8.4.10 (api:1/proto:86-101)
srcversion: 473968AD625BA317874A57E

The device can therefore be deleted on the chi-node-03 side. First detach it:

drbdsetup detach /dev/drbd0

Then delete it:

drbdsetup del-minor 0

If you get errors because the device is busy, see if you can see what is holding on to it /sys/devices/virtual/block/drbd0/holders, for example:

# ls -l /sys/devices/virtual/block/drbd3/holders/
total 0
lrwxrwxrwx 1 root root 0 Aug 26 16:03 dm-34 -> ../../dm-34

Then that device map can be removed with:

# dmsetup remove dm-34

Deleting a device after it was manually detached

After manually detaching a disk from a Ganeti instance, Prometheus alerts something like this: "DRBD has 2 out of date disks on dal-node-01.torproject.org". If you really don't need that disk anymore, you can manually delete it from drbd.

First, query Prometheus to learn the device number. In my case, device="drbd34".

After making sure that that device really corresponds to the one you want to delete, run:

drbdsetup detach --force=yes 34
drbdsetup down resource34

Pager playbook

Resyncing disks

A DRBDDegraded alert looks like this:

DRBD has 1 out of date disks on fsn-node-04.torproject.org

It means that, on that host (in this case fsn-node-04.torproject.org), disks are desynchronized for some reason. You can confirm that on the host:

# ssh fsn-node-04.torproject.org cat /proc/drbd
[...]
 9: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:13799284 nr:0 dw:272704248 dr:15512933 al:1331 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:8343096
10: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:2097152 nr:0 dw:2097192 dr:2102652 al:9 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:40
[...]

You need to find which instance this disk is associated with (see also above), by asking the Ganeti master for the DRBD disk listing with gnt-node list-drbd $NODE:

$ ssh fsn-node-01.torproject.org gnt-node list-drbd fsn-node-04
[...]
Node                       Minor Instance                            Disk   Role      PeerNode
[...]
fsn-node-04.torproject.org     9 onionoo-frontend-01.torproject.org  disk/0 primary   fsn-node-03.torproject.org
fsn-node-04.torproject.org    10 onionoo-frontend-01.torproject.org  disk/1 primary   fsn-node-03.torproject.org
[...]

Then you can "reactivate" the disks simply by telling ganeti:

ssh fsn-node-01.torproject.org gnt-instance activate-disks onionoo-frontend-01.torproject.org

And then the disk will resync.

It's also possible a disk was detached and improperly removed. In that case, you might want to delete a device after it was manually detached.

Upstream documentation

• User guide
• upstream intro
• troubleshooting

Reference

Installation

The ganeti Puppet module takes care of basic DRBD configuration, by installing the right software (drbd-utils) and kernel modules. Everything else is handled automatically by Ganeti itself.

TODO: this section is out of date since the Icinga replacement, see tpo/tpa/prometheus-alerts#16.

There's a Nagios check for the DRBD service that ensures devices are synchronized. It will yield an UNKNOWN status when no device is created, so it's expected that new nodes are flagged until they host some content. The check is shipped as part of tor-nagios-checks, as dsa-check-drbd, see dsa-check-drbd.

Fabric is a Python module, built on top of Invoke that could be described as "make for sysadmins". It allows us to establish "best practices" for routine tasks like:

• installing a server (TODO)
• retiring a server
• migrating machines (e.g. ganeti)
• retiring a user (TODO)
• reboots
• ... etc

Fabric makes easy things reproducible and hard things possible. It is not designed to handle larger-scale configuration management, for which we use service/puppet.

• Tutorial
  • Running a command on hosts
  • Listing tasks and self-documentation
• How-to
  • A simple Fabric function
  • Pager playbook
  • Disaster recovery
• Reference
  • Installation
    • Installing Fabric on Debian buster
  • SLA
  • Design
  • Issues
  • Monitoring and testing
• Discussion
  • Problem overview
  • LDAP notes
  • Goals
    • Must have
    • Nice to have
    • Non-Goals
  • Approvals required
  • Proposed Solution
  • Cost
  • Alternatives considered
    • ansible
    • mcollective
    • bolt
    • Doing things by hand
    • Another custom Python script
    • Shell scripts
    • Perl
    • mitogen
    • transilience
    • spicerack and cumin
    • Other Python tools
    • Other discarded alternatives
    • Other ideas

Tutorial

All of the instructions below assume you have a copy of the TPA fabric library, fetch it with:

git clone https://gitlab.torproject.org/tpo/tpa/fabric-tasks.git &&
cd fabric-tasks

Don't trust the GitLab server! This should be done only once, in TOFU (Trust On First Use) mode: further uses of the repository should verify OpenPGP signatures or Git hashes from a known source.

Normally, this is done on your laptop, not on the servers. Servers including the profile::fabric will have the code deployed globally (/usr/local/lib/fabric-tasks as of this writing), with the actual fabric package (and fab binary) available if manage_package is true. See tpo/tpa/team#41484 for the plans with that (currently progressive) deployment.

Running a command on hosts

Fabric can be used from the commandline to run arbitrary commands on servers, like this:

fab -H hostname.example.com -- COMMAND

For example:

$ fab -H perdulce.torproject.org -- uptime
 17:53:22 up 24 days, 19:34,  1 user,  load average: 0.00, 0.00, 0.07

This is equivalent to:

ssh hostname.example.com COMMAND

... except that you can run it on multiple servers:

$ fab -H perdulce.torproject.org,chives.torproject.org -- uptime
 17:54:48 up 24 days, 19:36,  1 user,  load average: 0.00, 0.00, 0.06
 17:54:52 up 24 days, 17:35, 21 users,  load average: 0.00, 0.00, 0.00

Listing tasks and self-documentation

The fabric-tasks repository has a good library of tasks that can be ran from the commandline. To show the list, use:

fab -l

Help for individual tasks can also be inspected with --help, for example:

$ fab -h host.fetch-ssh-host-pubkey
Usage: fab [--core-opts] host.fetch-ssh-host-pubkey [--options] [other tasks here ...]

Docstring:
  fetch public host key from server

Options:
  -t STRING, --type=STRING

The name of the server to run the command against is implicit in the usage: it must be passed with the -H (short for --hosts) argument. For example:

$ fab -H perdulce.torproject.org host.fetch-ssh-host-pubkey
b'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGOnZX95ZQ0mliL0++Enm4oXMdf1caZrGEgMjw5Ykuwp root@perdulce\n'

How-to

A simple Fabric function

Each procedure mentioned in the introduction above has its own documentation. This tutorial aims more to show how to make a simple Fabric program inside TPA. Here we will create a uptime task which will simply run the uptime command on the provided hosts. It's a trivial example that shouldn't be implemented (it is easier to just tell fab to run the shell command) but it should give you an idea of how to write new tasks.

1. edit the source

   $EDITOR fabric_tpa/host.py
   

   we pick the "generic" host library (host.py) here, but there are other libraries that might be more appropriate, for example ganeti, libvirt or reboot. Fabric-specific extensions, monkeypatching and other hacks should live in __init__.py.

2. add a task, which is simply a Python function:

   @task
   def uptime(con):
       return con.run('uptime')
   

   The @task string is a decorator which indicates to Fabric the function should be exposed as a command-line task. In that case, it gets a Connection object passed which we can run stuff from. In this case, we run the uptime command over SSH.

3. the task will automatically be loaded as it is part of the host module, but if this is a new module, add it to fabfile.py in the parent directory

4. the task should now be available:

   $ fab -H perdulce.torproject.org host.uptime
    18:06:56 up 24 days, 19:48,  1 user,  load average: 0.00, 0.00, 0.02
   

Pager playbook

N/A for now. Fabric is an ad-hoc tool and, as such, doesn't have monitoring that should trigger a response. It could however be used for some oncall work, which remains to be determined.

Disaster recovery

N/A.

Reference

Installation

Fabric is available as a Debian package:

apt install fabric

See also the upstream instructions for other platforms (e.g. Pip).

To use tpa's fabric code, you will most likely also need at least python ldap support:

apt install python3-ldap

Fabric code grew out of the installer and reboot scripts in the fabric-tasks repository. To get access to the code, simply clone the repository and run from the top level directory:

git clone https://gitlab.torproject.org/tpo/tpa/fabric-tasks.git &&
cd fabric-tasks &&
fab -l

This code could also be moved to its own repository altogether.

Installing Fabric on Debian buster

Fabric has been part of Debian since at least Debian jessie, but you should install the newer, 2.x version that is only available in bullseye and later. The bullseye version is a "trivial backport" which means it can be installed directly in stable with:

apt install fabric/buster-backports

This will also pull invoke (from unstable) and paramiko (from stable). The latter will show a lot of warnings when running by default, however, so you might want to upgrade to backports as well:

apt install python3-paramiko/buster-backports

SLA

N/A

Design

TPA's fabric library lives in the fabric-tasks repository and consists of multiple Python modules, at the time of writing:

anarcat@curie:fabric-tasks(master)$ wc -l fabric_tpa/*.py
  463 fabric_tpa/ganeti.py
  297 fabric_tpa/host.py
   46 fabric_tpa/__init__.py
  262 fabric_tpa/libvirt.py
  224 fabric_tpa/reboot.py
  125 fabric_tpa/retire.py
 1417 total

Each module encompasses Fabric tasks that can be called from the commandline fab tool or Python functions, both of which can be reused in other modules as well. There are also wrapper scripts for certain jobs that are a poor fit for the fab tool, especially reboot which requires particular host scheduling.

The fabric functions currently only communicate with the rest of the infrastructure through SSH. It is assumed the operator will have direct root access on all the affected servers. Server lists are provided by the operator but should eventually be extracted from PuppetDB or LDAP. It's also possible scripts will eventually edit existing (but local) git repositories.

Most of the TPA-specific code was written and is maintained by anarcat. The Fabric project itself is headed by Jeff Forcier AKA bitprophet it is, obviously, a much smaller community than Ansible but still active. There is a mailing list, IRC channel, and GitHub issues for upstream support (see contact) along with commercial support through Tidelift.

There are no formal releases of the code for now.

Those are the main jobs being automated by fabric:

• automate installs
• automate reboots
• automate retirement

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker component.

Monitoring and testing

There is no monitoring of this service, as it's not running continuously.

Fabric tasks should implement some form of unit testing. Ideally, we would have 100% test coverage.

We use pytest to write unit tests. To run the test suite, use:

pytest-3 fabric_tpa

Discussion

Problem overview

There are multiple tasks in TPA that require manual copy-pasting of code from documentation to the shell or, worse, to grep backwards in history to find the magic command (e.g. ldapvi). A lot of those jobs are error-prone and hard to do correctly.

In case of the installer, this leads to significant variation and chaos in the installs, which results in instability and inconsistencies between servers. It was determined that the installs would be automated as part of ticket 31239 and that analysis and work is being done in new-machine.

It was later realised that other areas were suffering from a similar problem. The upgrade process, for example, has mostly been manual until adhoc shell scripts were written. But unfortunately now we have many shell scripts, none of which work correctly. So work started on automating reboots as part of ticket 33406.

And then it was time to migrate the second libvirt server to service/ganeti (unifolium/kvm2, ticket 33085) and by then it was clear some more generic solution was required. An attempt to implement this work in Ansible only led to frustration at the complexity of the task and tests were started on Fabric instead, which were positive. A few weeks later, a library of functions was available and the migration procedure was almost entirely automated.

LDAP notes

LDAP integration might be something we could consider, because it's a large part of the automation that's required in a lot of our work. One alternative is to talk with ldapvi or commandline tools, the other is to implement some things natively in Python:

• Python LDAP could be used to automate talking with ud-ldap, see in particular the Python LDAP functions, in particular add and delete
• The above docs are very limited, and they suggest external resources also:
  • https://hub.packtpub.com/python-ldap-applications-extra-ldap-operations-and-ldap-url-library/
  • https://hub.packtpub.com/configuring-and-securing-python-ldap-applications-part-2/
  • https://www.linuxjournal.com/article/6988

Goals

Must have

• ease of use - it should be easy to write new tasks and to understand existing ones

• operation on multiple servers - many of the tricky tasks we need to do operate on multiple servers synchronously something that, for example, is hard to do in Puppet

• lifecycle management in an heterogeneous environment: we need to be able to:

  • provision bare-metal on our leased machines at Cymru, on rented machines at Hetzner, on Hetzner cloud, in Openstack (currently done by hand, with shell scripts, and Fabric)

  • reboot the entire infrastructure, considering mirrors and ganeti clusters (currently done with Fabric)

  • do ad-hoc operations like "where is php-fpm running?" (currently done with Cumin) or "grub exploded, i need to load a rescue and rebuild the boot loader" (currently done by hand) or "i need to resize a filesystem" (currently done by copy-pasting from the wiki)

  • retire machines (currently done by hand and Fabric)

Nice to have

• long term maintenance - this should not be Legacy Code and must be unit tested, at least for parts that are designed to stay in the long term (e.g. not the libvirt importer)

Non-Goals

• sharing with the community - it is assumed that those are tasks too site-specific to be reused by other groups, although the code is still shared publicly. shared code belongs to Puppet.

• performance - this does not need to be high performance, as those tasks are done rarely

Approvals required

TPA. Approved in meeting/2020-03-09.

Proposed Solution

We are testing Fabric.

Fabric was picked mostly over Ansible because it allowed more flexibility in processing data from remote hosts. The YAML templating language of Ansible was seen as too limiting and difficult to use for the particular things we needed to do (such as host migration).

Furthermore, we did not want to introduce another configuration management system. Using Ansible could have led to a parallel configuration management interface "creeping in" next to Puppet. The intention of this deployment is to have the absolute minimal amount of code needed to do things Puppet cannot do, not to replace it.

One major problem with Fabric is that it creates pretty terrible code: it is basically a glorified Makefile, because we cannot actually run Python code on the remote servers directly. (Well, we could, but we'd first need to upload the code and call it a shell command, so it is not real IPC.) In that sense, Mitogen is a real eye-opener and game-changer.

Cost

Time and labor.

Alternatives considered

ansible

Ansible makes easy things easy, but it can make it hard to do hard stuff.

For example, how would you do a disk inventory and pass it to another host to recreate those disk? for an Ansible ignorant like me, it's far from trivial, but in Fabric, it's:

json.loads(con.run('qemu-img info --output=json %s' % disk_path).stdout)

Any person somewhat familiar with Python can probably tell what this does. In Ansible, you'll need to first run the command and have a second task to parse the result, both of which involves slow round-trips with the server:

- name: gather information about disk
  shell: "qemu-img info --output=json {{disk_path}}"
  register: result

- name: parse disk information as JSON
  set_fact:
    disk_info: "{{ result.stdout | from_json }}"

That is much more verbose and harder to discover unless you're already deeply familiar with Ansible's processes and data structures.

Compared with Puppet, Ansible's "collections" look pretty chaotic. The official collections index is weirdly disparate and incomplete while Ansible Galaxy is a wild jungle.

For example, there are 677 different Prometheus collections at the time of writing. The most popular Prometheus collection one has lots of issues, namely:

• no support for installation through Debian packages (but you can "skip installation")

• even if you do, incompatible service names for exporters (e.g. blackbox-exporter), arguably a common problem that was also plaguing the Puppet module until @anarcat worked on it

• the module's documentation is kind of hidden inside the source code, for example here is the source docs which show use cases and actual configurations, compared to the actual role docs, which just lists supported variables

Another example is the nginx collection. In general, collections are pretty confusing coming from Puppet, where everything is united under a "module". A Collection is actually closer to a module than a role is, but collections and roles are sometimes, as is the case for nginx, split in separate git repositories, which can be confusing (see the nginx role.

Taking a look at the language in general, Ansible's variable are all globals, which means they all get "scoped" by using a prefix (e.g. prometheus_alert_rules).

Documentation is sparse and confusing. For example, I eventually figured out how to pull data from a host using a lookup function, but that wasn't because of the lookup documentation or the pipe plugin documentation, neither of which show this simple example:

- name: debug list hosts
  debug: msg="{{ lookup('pipe', '/home/anarcat/src/prometheus.debian.net/list-debian.net.sh')}}"

YAML is hell. I could not find a way to put the following shell pipeline in a pipe lookup above, hence the shell script:

ldapsearch -u -x -H ldap://db.debian.org -b dc=debian,dc=org '(dnsZoneEntry=*)' dnsZoneEntry | grep ^dnsZoneEntry | grep -e ' A ' -e ' AAAA ' -e ' CNAME ' | sed -s 's/dnsZoneEntry: //;s/ .*/.debian.net/' | sort -u

For a first time user, the distinction between a lookup() function and a shell task is really not obvious, and the documentation doesn't make it exactly clear that the former runs on the "client" and the latter runs on the "server" (although even the latter can be fuzzy, through delegation).

And since this is becoming a "Ansible crash course for Puppet developers", might as well add a few key references:

• the working with playbooks section is possibly the most important and useful part of the Ansible documentation

• that includes variables and filters, critical and powerful functions that allow processing data from variables, files, etc

• tags can be used to run a subset of a playbook but also skip certain parts

Finally, Ansible is notoriously slow. A relatively simple Ansible playbook to deploy Prometheus runs in 44 seconds while a fully-fledged Puppet configuration of a production server runs in 20 seconds, and this includes a collection of slow facts that takes 10 of those 18 seconds, actual execution is nearer to 7 seconds. The Puppet configuration manages 757 resources while the Ansible configuration manages 115 resources. And that is with ansible-mitogen: without that hack, the playbook takes nearly two minutes to run.

In the end, the main reason we use Fabric instead of Ansible is that we use Puppet for high-level configuration management, and Ansible conflicts with that problem space, leading to higher cognitive load. It's also easier to just program custom processes in Python than in Ansible. So far, however, Fabric has effectively been creating more legacy code as it has been proven hard to effectively unit test unless a lot of care is given to keeping functions small and locally testable.

mcollective

• MCollective was (it's deprecated) a tool that could be used to fire jobs on Puppet nodes from the Puppet master

• Not relevant for our use case because we want to bootstrap Puppet (in which case Puppet is not available yet) or retire Puppet (in which case it will go away).

bolt

• Bolt is interesting because it can be used to bootstrap Puppet

• Unfortunately, it does not reuse the Puppet primitives and instead Bolt "tasks" are just arbitrary commands, usually shell commands (e.g. this task) along with a copious amount of JSON metadata

• does not have much privileged access to PuppetDB or the Puppet CA infrastructure, that needs to be bolted on by hand

Doing things by hand

• timing is sometimes critical
• sets best practices in code instead of in documentation
• makes recipes easily reusable

Another custom Python script

• is it subprocess.check_output? or check_call? or run? what if you want both the output and the status code? can you remember?
• argument parsing code built-in, self-documenting code
• exposes Python functions as commandline jobs

Shell scripts

• hard to reuse
• hard to read, audit
• missing a lot of basic programming primitives (hashes, objects, etc)
• no unit testing out of the box

Perl

• notoriously hard to read

mitogen

A late-comer to the "alternatives considered" section, I actually found out about the mitogen project after the choice of Fabric was made, and a significant amount of code written for it (about 2000 SLOC).

A major problem with Fabric, I discovered, is that it only allows executing commands on remote servers. That is, it's a glorified shell script. Yes, it allows things like SFTP file transfers, but that's about it: it's not possible to directly execute Python code on the remote node. This limitation makes it hard to implement more complex business logic on the remote server. It also makes error control in Fabric less intuitive as normal Python code reflexes (like exception handling) cannot be used. Exception handling, in Fabric, is particularly tricky, see for example issue 2061 but generally: exceptions don't work well inside Fabric.

Basically, I wish I had found out about mitogen before I wrote all this code. It would make code like the LDAP connector much easier to write (as it could run directly on the LDAP server, bypassing the firewall issues). A rewrite of the post-install grml-deboostrap hooks would also be easier to implement than right now.

Considering there isn't that much code written, it's still possible to switch to Mitogen. The major downside of mitogen is that it doesn't have a commandline interface: it's "only" a Python library and everything needs to be written on top of that. In fact, it seems like Mitogen is primarily written as an Ansible backend, so it is possible that non-Ansible use cases might be less supported.

The "makefile" (fabfile, really) approach is also not supported at all by mitogen. So all the nice "self-documentation" and "automatic usage" goodness brought to use by the Fabric decorator would need to be rebuilt by hand. There are existing dispatchers (say like click or fire) which could be used to work around that.

And obviously, the dispatcher (say: run this command on all those hosts) is not directly usable from the commandline, out of the box. But it seems like a minor annoyance considering we're generally rewriting that on top of Fabric right now because of serious limitations in the current scheduler.

Finally, mitogen seems to be better maintained than fabric: at the time of writing:

Those numbers are based on the GitHub current statistics. Another comparison is the openhub dashboard comparing Fabric, Mitogen and pyinvoke (the Fabric backend). It should be noted that:

• all three projects have "decreasing" activity
• the code size is in a similar range: when added together, Fabric and invoke are about 26k SLOC, while mitogen is 36k SLOC. but this does show that mitogen is more complex than Fabric
• there has been more activity in mitogen in the past 12 months
• but more contributors in Fabric (pyinvoke, specifically) over time

The Fabric author also posted a request for help with his projects, which doesn't bid well for the project in the long term. A few people offered help, but so far no major change has happened in the issue queue (lots of duplicates and trivial PRs remain open).

On the other hand, the Mitogen author seems to have moved onto other things. He hasn't committed to the project in over a year, shortly after announcing a "private-source" (GPL, but no public code release) rewrite of the Ansible engine, called Operon. So it's unclear what the fate of mitogen will be.

transilience

Enrico Zini has created something called transilience which sites on top of Mitogen that is somewhat of a Ansible replacement, but without the templatized YAML. Fast, declarative, yet Python. Might be exactly what we need, and certainly better than starting on top of mitogen only.

The biggest advantage of transiliance is that it builds on top of mitogen, because we can run Python code remotely, transparently. Zini was also especially careful about creating a somewhat simple API.

The biggest flaw is that it is basically just a prototype with limited documentation and no stability promises. It's not exactly clear how to write new actions, for example, unless you count this series of blog posts. It might also suffer second-system syndrome in the sense that it might become also complicated as it tries to replicate more of Ansible's features. It could still offer a good source of library items to do common tasks like install packages and so on.

spicerack and cumin

The Wikimedia Foundation (WMF, the organisation running Wikipedia) created a set of tools called spicerack (source code). It is a framework of Python code built on top of Cumin, on top of which they wrote a set of cookbooks to automate various ad-hoc operations on the cluster.

Like Fabric, it doesn't ship Python code on the remote servers: it merely executes shell commands. The advantage over Fabric is that it bridges with the Cumin inventory system to target servers based on the domain-specific language (DSL) available there.

It is also very WMF-specific, and could be difficult to use outside of that context. Specifically, there might be a lot of hardcoded assumptions in the code that we'd need to patch out (example, Ganeti instance creation code, which would then therefore require a fork. Fortunately, spicerack has regular releases which makes tracking forks easier. Collaboration with upstream is possible, but requires registering and contributing to their Gerrit instance (see for example the work anarcat did on Cumin).

It does have good examples of how Cumin can be used as a library for certain operations, however.

One major limitation of Spicerack is that it uses Cumin as a transport, which implies that it can only execute shell commands on the remote server: no complex business logic can be carried over to the remote side, or, in other words, we can't run Python code remotely.

Other Python tools

This article reviews a bunch of Ansible alternatives in Python, let's take a look:

• Bundlewrap: Python-based DSL, push over SSH, needs password-less sudo over SSH for localhost operation, defers to SSH multiplexing for performance (!), uses mako templates, unclear how to write new extend with new "items", active

• Pulumi: lots of YAML, somewhat language agnostic (support for TypeScript, JavaScript, Python, Golang, C#), lots of YAML, requires a backend, too complicated, unclear how to write new backends, active

• Nuka: asyncio + SSH, unclear scoping ("how does shell.command know which host to talk with?"), minimal documentation, not active

• pyinfra: lots of facts, operations, control flow can be unclear, performance close to Fabric, popular, active

• Nornir: no DSL: just Python, plugins, YAML inventory, active

Other discarded alternatives

• FAI: might resolve installer scenario (and maybe not in all cases), but does not resolve ad-hoc tasks or host retirement. we can still use it for parts of the installer, as we currently do, obviously.

Other ideas

One thing that all of those solutions could try to do is the do nothing scripting approach. The idea behind this is that, to reduce toil in complex task, you break it down in individual steps that are documented in a script, split in many functions. This way it becomes possible to automate parts of that script, possibly with reusable code across many tasks.

That, in turns, make automating really complex tasks possible in an incremental fashion...

Git

• Commit signature verification
  • Terminology
  • sequoia-git basics
  • The TPA setup
  • Authentication in the Puppet Server
  • Certificate updates
  • Expired certificates
  • Manual override
  • Other repositories

TPA uses Git in several places in its infra. Several services are managed via repos hosted in GitLab, but some services are managed by repos stored directly in the target systems, such as Puppet, LDAP, DNS, TLS, and probably others.

Commit signature verification

In order to resist tampering attempts such as GitLab compromise, some key repositories are configured to verify commit signatures before accepting ref updates. For that, TPA uses sequoia-git to authenticate operations against certificates and permissions stored in a centralized OpenPGP policy file. See TPA-RFC-90: Signed commits for the initial proposal.

Terminology

Throughout this section, we use the term "certificate" to refer to OpenPGP Transferable Public Keys (see section 11.1 of RFC 4880).

sequoia-git basics

In order to authenticate changes in a Git repository, sequoia-git uses two pieces of information:

• an OpenPGP policy file, containing authorized certificates and a list of permissions for each certificate, and
• a "trust-root", which is the ID of a commit that is considered trusted.

With these, sequoia-git goes through commit by commit checking whether the signature is valid and authorized to perform operations.

By default, sequoia-git uses the openpgp-policy.toml file in the root of the repo being checked, but a path to an external policy file can be passed instead. In TPA, we do the former on the client side and the latter on the server side, as we'll see in the next section.

The TPA setup

In TPA we use one OpenPGP policy file to authenticate changes for all our repositories, namely the openpgp-policy.toml file in the root of the Puppet repository. Using one centralized file allows for updating certificates and permissions in only one place and have it deployed to the relevant places.

For authenticating changes on the server-side:

• the TPA OpenPGP policy file is deployed to /etc/openpgp-policy/policies/tpa.toml,
• trust-roots for the Puppet repos (stored in hiera data for the puppetserver role in the Puppet repo) are deployed to /etc/openpgp-policy/gitconfig/${REPO}.conf, and
• per-repo Git hooks use the above info to authenticate changes.

On the client-side:

• we use the TPA OpenPGP policy file in the root of the Puppet repo,
• trust-roots are stored in the .mrconfig file in tpo/tpa/repos> and set as Git configs in the relevant repos by mr update (see doc on repos.git), and
• per-repo Git hooks use the above info to authenticate changes.

Note: When the trust-root for a repository changes, it needs to be updated in the hiera data for the puppetserver role and/or the .mrconfig file, depending on whether it's supposed to be authenticated on server and/or client side.

Authentication in the Puppet Server

The Puppet repositories stored in the Puppet server are configured with hooks to verify authentication of the incoming commits before performing ref updates.

Puppet deploys in the Puppet server:

• the TPA OpenPGP policy file (openpgp-policy.toml) to /etc/openpgp-policy/policies/tpa.toml,
• global Git configuration containing per-repo policy file and trust-root configs to /etc/openpgp-policy/gitoconfig/, and
• Git update-hooks to the Puppet repositories that only allow ref updates if authentication is valid

See the profile::openpgp_policy Puppet profile for the implementation.

With this, ref updates in the Puppet Git repos are only performed if all commits since the trust-root are signed with authorized certificates contained in the installed TPA OpenPGP policy file.

Certificate updates

While a certificate is still valid and has the sign_commit capability, it's allowed to update any certificate contained in the openpgp-policy.toml file.

To update one or more certificates, first make sure you have up-to-date versions in your local store. One way to do that is by using sq to import the certificate from Tor's Web Key Directory:

sq network wkd search <ADDRESS>

Then use sq-git to update the OpenPGP policy file with certificates from your local store:

sq-git policy sync --disable-keyservers

Note that, if you don't use --disable-keyservers, expired subkeys may end up being included by a sync, and you may think that there are updates to the key when there really aren't. So it's better to just do as suggested above.

You can also edit the openpgp-policy.toml file manually and perform the needed changes.

Note that, because we use a centralized OpenPGP policy file, when permissions are removed for a certificate, we may need to update the trust-root, otherwise old commits may fail to be authenticated against the new policy file.

Expired certificates

If a certificate expires before it's been updated in the openpgp-policy.toml file, changes signed by that certificate will not be accepted, and you'll need to (1) ask another sysadmin with a valid certificate to perform the needed changes and (2) wait for or force deployment of the new file in the server.

See the above section for instructions on how to update the OpenPGP policy file.

Manual override

There may be extreme situations in which you need to override the authentication check, for example if your certificate expired and you're the only sysadmin in duty. In these cases, you can manually remove/update the corresponding Git hooks in the server and push the needed changes. If you do this, make sure to:

• update the trust root both in the hiera data for the puppetserver role and in tpo/tpa/repos>.
• instruct the other sysadmins to pull tpo/tpa/repos> and run mr update, so their local Git configs for trust-roots is automatically updated. If you don't do that, their local checks will start failing when they pull commits that can't be authenticated.

Other repositories

Even though we initially deployed this mechanism to Pupper repositories only, the current implementation of the OpenPGP policy profile allows for configuration of the same setup for arbitrary repositories, which can be configured via hiera. See the hiera data for the puppetserver role for an example.

Setting trust-roots is mandatory, while policy files are optional. If no policy file is explicitly set, the Git hook will perform the authentication checks against the policy file in the root of the repository itself.

878 packets transmitted, 0 received, 100% packet loss, time 14031ms

(See tpo/tpa/team#41654 for a discussion and further analysis of that specific issue.)

MTR can help diagnose issues in this case. Vary parameters like IPv6 (-6) or TCP (--tcp). In the above case, the problem could be reproduced with mtr --tcp -6 -c 10 -w maven.mozilla.org.

Tools like curl can also be useful for quick diagnostics, but note that it supports the happy eyeballs standard so it might hide (e.g. IPv6) issues that might otherwise be affecting other clients.

Unexpected reboot

If a host reboots without a manual intervention, there might be different causes for the reboot to happen. Identifying exactly what happened after the fact can be challenging or even in some cases impossible since logs might not have been updated with information about the issues.

But in some cases the logs do have some information. Some things that can be investigated:

• syslog. look particularly for disk errors, OOM kill messages close to the reboot, kernel oops messages
• dmesg from previous boots, e.g. journaltcl -k -b -1, or see journalctl --list-boots for a list of boot IDs available
• smartctl -t long and smartctl -A / nvme [device-self-test|self-test-log] on all devices
• /proc/mdadm and /proc/drbd: make sure that replication is still all right

Also note that it's possible this is a spurious warning, or that a host took longer than expected to reboot. Normally, our Fabric reboot procedures issue a silence for the monitoring system to ignore those warnings. It's possible those delays are not appropriate for this host, for example, and might need to be tweaked upwards.

Network-level attacks

This section should guide you through network availability issues.

Confirming network-level attacks with Grafana

In case of degraded service availability over the network, it's a good idea to start by looking at metrics in Grafana. Denial of service attacks against a service over the network will often cause a noticeable bump in network activity, both in terms of ingress and egress traffic.

The traffic per class dashboard is a good place to start.

Finding traffic source with iftop

Once you have found there is indeed a spike of traffic, you should try to figure out what it consists of exactly.

A useful tool to investigate this is iftop, which displays network activity in realtime via the console. Here are some useful keyboard shortcuts when using it:

• n toggle DNS resolution
• D toggle destination port
• T toggle cumulative totals
• o freeze current order
• P pause display

In addition, the -f command-line argument can be used to filter network activity. For example, use iftop -f 'port 443' to only monitor HTTPS network traffic.

Firewall blocking

If you are sure that a specific $IP is mounting a Denial of Service attack on a server, you can block it with:

iptables -I INPUT -s $IP -j DROP

$IP can also be a network in CIDR notation, e.g. the following drops a whole Google /16 from the host:

iptables -I INPUT -s 74.125.0.0/16 -j DROP

Note that the above inserts (-I) a rule into the rule chain, which puts it before other rules. This is most likely what you want, as it's often possible there's an already existing rule that will allow the traffic through, making a rule appended (-A) to the chain ineffective.

This only blocks one network or host, and quite brutally, at the network level. From a user's perspective, it will look like an outage. A gentler way way is to use -j REJECT to actually send a reset packet to let the user know they're blocked.

See also our nftables documentation.

Note that those changes are gone after reboot or firewall reloads, for permanent blocking, see below.

Server blocking

An even "gentler" approach is to block clients at the server level. That way the client application can provide feedback to the user that the connection has been denied, more clearly. Typically, this is done with a web server level block list.

We don't have a uniform way to do this right now. In profile::nginx, there's a blocked_hosts list that can be used to add CIDR entries which are passed to the Nginx deny directive. Typically, you would define an entry in Hiera with something like this (example from data/roles/gitlab.yaml):

profile::nginx::blocked_hosts:
  # alibaba, tpo/tpa/team#42152
  - "47.74.0.0/15"

For Apache servers, it's even less standardized. A couple servers (currently donate and crm) have a blocklist.txt file that's used in a RewriteMap to deny individual IP addresses.

Extracting IP range lists

A command like this will extract the IP addresses from a webserver log file and group them by number of hits:

awk '{print $1}' /var/log/nginx/gitlab_access.log | grep -v '0.0.0.0' | sort | uniq -c | sort -n

This assumes log redaction has been disabled on the virtual host, of course, which can be done in emergencies like this. The most frequent hosts will show up first.

You can lookup which netblock the relevant IP addresses belong to a command like ip-info (part of the libnet-abuse-utils-perl Debian package) or asn (part of the asn package). Or this can be done by asking the asn.cymru.com service, with, for example:

nc whois.cymru.com 43 <<EOF
begin
verbose
216.90.108.31
192.0.2.1
198.51.100.0/24
203.0.113.42
end
EOF

This can be used to group IP addresses by netblock and AS number, roughly. A much more sophisticated approach is the asncounter project developed by anarcat, which allows AS and CIDR-level counting and can be used to establish a set of networks or entire ASNs to block.

The asncounter(1) manual page has detailed examples for this. That tool has been accepted in Debian unstable as of 2025-05-28 and should slowly make its way down to stable (probably Debian 14 "forky" or later). It's currently installed on gitlab-02 in /root/asncounter but may eventually be deployed site-wide through Puppet.

Filesystem set to readonly

If a filesystem is switched to readonly, it prevents any process from writing to the concerned disk, which can have consequences of differing magnitude depending on which volume is readonly.

If Linux automatically changes a filesystem to readonly, it usually indicates that some serious issues were detected with the disk or filesystem. Those can be:

• physical drive errors
• bad sectors or other detected ongoing data corruption
• hard drive driver errors
• filesystem corruption

Look out for disk- or filesystem-related errors in:

• syslog
• dmesg
• physical console (e.g. IMPI console)

In some cases with ext4, running fsck can fix issues. However, watch out for files disappearing or being moved to lost+found if the filesystem encounters serious enough inconsistencies.

If the hard disk seems to be showing signs of breakage. Usually that disk will get ejected from the RAID array without blocking the filesystem. However if disk breakage did impact the filesystem consistency and caused it to switch to readonly, migrate the data away from that drive ASAP for example by moving the instance to its secondary node or by rsync'ing it to another machine.

In such a case, you'll also want to review what other instances are currently using the same drive and possibly move all of those instances as well before replacing the drive.

Web server down

Apache web server diagnostics

If you get an alert like ApacheDown, that is:

Apache web server down on test.example.com

It means the apache exporter cannot contact the local web server over its control address http://localhost/server-status/?auto. First, confirm whether this is a problem with the exporter or the entire service, by checking the main service on this host to see if users are affected. If that's the case, prioritize that.

It's possible, for example, that the webserver has crashed for some reason. The best way to figure that out is to check the service status with:

service apache2 status

You should see something like this if the server is running correctly:

● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-09-10 14:56:49 UTC; 1 day 5h ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 475367 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
   Main PID: 338774 (apache2)
      Tasks: 53 (limit: 4653)
     Memory: 28.6M
        CPU: 11min 30.297s
     CGroup: /system.slice/apache2.service
             ├─338774 /usr/sbin/apache2 -k start
             └─475411 /usr/sbin/apache2 -k start

Sep 10 17:51:50 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 10 17:51:50 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
Sep 10 19:53:00 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 10 19:53:00 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
Sep 11 00:00:01 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 11 00:00:01 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
Sep 11 01:29:29 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 11 01:29:29 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
Sep 11 19:50:51 donate-01 systemd[1]: Reloading apache2.service - The Apache HTTP Server...
Sep 11 19:50:51 donate-01 systemd[1]: Reloaded apache2.service - The Apache HTTP Server.

With the first dot (●) in green and the line Active saying active (running). If it isn't, the logs should show why it failed to start.

It's possible you don't see the right logs in there if the service is stuck in a restart loop. In this case, that use this command instead to see the service logs:

journalctl -b -u apache2

That shows the logs for the server from the last boot.

If the main service is online and it's only the exporter having trouble, try to reproduce the issue with curl from the affected server, for example:

root@test.example.com:~# curl http://localhost/server-status/?auto

Normally, this should work, but it's possible Apache is misconfigured and doesn't listen to localhost for some reason. Look at the apache2ctl -S output, and the rest of the Apache configuration in /etc/apache2, particularly the Ports and Listen directives.

See also the Apache exporter scraping failed instructions in the Prometheus documentation, a related alert.

Disk is full or nearly full

When a disk is filled up to 100% of its capacity, some processes can have issues with continuing to work normally. For example PostgreSQL will purposefully exit when that happens in order to avoid the risk of data corruption. MySQL is not so graceful and it can end up with data corruption in some of its databases.

The first step is to check how long you have. For this, a good tool is the Grafana disk usage dashboard. Select the affected instance, and look at the "change rate" panel, it should show you how much time is left per partition.

To clear up this situation, there are two approaches that can be used in succession:

• find what's using disk space and clear out some files
• grow the disk

The first thing that should be attempted is to identify where disk space is used and remove some big files that occupy too much space. For example, if the root partition is full, this will show you what is taking up space:

ncdu -x /

Examples

Maybe the syslog grew to ridiculous sizes? Try:

logrotate -f /etc/logrotate.d/syslog-ng

Maybe some users have huge DB dumps laying around in their home directory. After confirming that those files can be deleted:

rm /home/flagada/huge_dump.sql

Maybe the systemd journal has grown too big. This will keep only 500MB:

journalctl --vacuum-size=500M

If in the cleanup phase you can't identify files that can be removed, you'll need to grow the disk. See how to grow disks with ganeti.

Note that it's possible a suddenly growing disk might be a symptom of a larger problem, for example bots crawling a website abusively or an attacker running a denial of service attack. This warrants further (and more complex) investigation, of course, but can be delegated to after the disk usage alert has been handled.

Other documentation:

• PostgreSQL backups filling up
• MinIO server filling up

Host clock desynchronized

If a host's clock has drifted and is no longer in sync with the rest of the internet, some really strange things can start happening, like TLS connections failing even though the certificate is still valid.

If a host has time synchronization issues, check that the ntpd service is still running:

systemctl status ntpd.service

You can gather information about which peer servers are drifting:

ntpq -pun

Logs for this service are sent to syslog, so you can take a look there to see if some errors were mentioned.

If restarting the ntpd service does not work, verify that a firewall is not blocking port 123 UDP.

Support policies

Please see TPA-RFC-2: support.

Creating a tunnel with HE.net

https://tunnelbroker.net/

https://tunnelbroker.net/new_tunnel.php

enter the IP address of your endpoint (your current IP address is shown and can be copy-pasted if you're already on site)

pick a location and hit "Create tunnel"

then you can add the description (optional)

you can copy the configuration which, for Debian, looks like:

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
        address 2001:470:1c:81::2
        netmask 64
        endpoint 216.66.38.58
        local 216.137.119.51
        ttl 255
        gateway 2001:470:1c:81::1

TODO: replace the above with sample IP addresses

Note that, in the above configuration, you do not have access to the entire /64 the gateway and address live under. They use a /64 for a point to point link because of RFC2627. The network you will announce locally will be different, under the "Routed IPv6 Prefixes" section. For example, in my case it is 2001:470:1d:81::/64 and I have the option to add a /48 if I need more networks.

If you have a dynamic IP address, you will need to setup a dynamic update of your IP address, so that your endpoint gets update correctly on their end. Information about those parameters is in the "Advanced" tab of your tunnel configuration. There you can also unblock IRC and SMTP access.

Reference

Installation

First, install the iSCSI support tools. This requires loading new kernel modules, so we might need to reboot first to clear the module loading protection:

reboot
apt install open-iscsi

Dealing with messed up consoles

For various reasons, it's possible that, during a rescue operation, you end up on a virtual console that has a keymap set differently than what you might expect.

For excellent and logical historical reasons, different countries have different keyboard layouts and while that's usually not a problem on daily operations using SSH, when you hit a serial console, the remote configuration actually takes effect.

This will manifest itself as you failing to enter the root password on a console, for example. This is especially present on some hosts configured with a German keyboard layout (QWERTZ), or inversely, if you're used to such a keyboard (or the french AZERTY layout), most hosts configured with the english QWERTY layout.

A few tips, for QWERTY users landing on a QWERTZ layout:

• Y and Z are reversed, otherwise most letters are in the same place

• - (dash) is left of the right shift key, i.e. in place of / (slash)

• / (slash) is above 7 (so shift-seven)

Resetting a system to a US keyboard

Most systems should generally have a US layout, but if you find a system with a German keyboard layout, you can reset it with the following procedure:

dpkg-reconfigure keyboard-configuration
setupcon -k -f

See also the Debian wiki Keyboard page.

Lektor is a static website generator written in Python, we use it to generate most of the websites of the Tor Project.

• Tutorial
  • Build a Lektor project on your machine
  • Build a basic Lektor website in GitLab CI
• How-to
  • Submit a website contribution
    • As an occasional contributor
    • As a regular contributor
  • Pager playbook
  • Disaster recovery
• Reference
  • Installation
  • SLA
  • Design
    • CI build/test pipelines
    • CD pipelines and environments
      • Staging and production
    • Review apps
    • Localization staging
  • Issues
  • Maintainer, users, and upstream
  • Monitoring and testing
  • Logs and metrics
  • Backups
  • Other documentation
• Discussion
  • Overview
  • Goals
    • Must have
    • Nice to have
    • Non-Goals
  • Approvals required
  • Proposed Solution
  • Cost
  • Alternatives considered

Tutorial

Build a Lektor project on your machine

See this page on the Web team wiki.

Build a basic Lektor website in GitLab CI

To enable automatic builds of a Lektor project in GitLab CI, add this snippet in .gitlab.ci-yml, at the root of the project:

include:
  - project: tpo/tpa/ci-templates
	file:
	  - lektor.yml
	  - pages-deploy.yml

The jobs defined in lektor.yml will spawn a container to build the site, and pages-deploy.yml will deploy the build artifacts to GitLab Pages.

See service/gitlab for more details on publishing to GitLab Pages.

How-to

Submit a website contribution

As an occasional contributor

The first step is to get a GitLab account.

This will allow you to fork the Lektor project in your personal GitLab namespace, where you can push commits with your changes.

As you do this, GitLab CI will continuously build a copy of the website with your changes and publish it to GitLab Pages. The location where these Pages are hosted can be displayed by navigation to the project Settings > Pages.

When you are satisfied, you can submit a Merge Request and one of the website maintainers will evaluate the proposed changes.

As a regular contributor

As someone who expects to submit contributions on a regular basis to one of the Tor Project websites, the first step is to request access. This can be done by joining the #tor-www channel on IRC and asking!

The access level granted for website content contributors is normally Developer. This role grants the ability to push new branches to the GitLab project and submit Merge Requests to the default main branch.

When a Merge Request is created, a CI pipeline status widget will appear under the description, above the discussion threads. If GitLab CI succeeds building the branch, it will publish the build artifacts and display a View app button. Clicking the button will navigate to the build result hosted on review.torproject.net.

Project members with the Developer role on the TPO blog and main website have the permission to accept Merge Requests.

Once the branch is deleted, after the Merge Request is accepted, for example, the build artifacts are automatically unpublished.

Pager playbook

Disaster recovery

See #revert-a-deployment-mistake for instruction on how to roll-back an environment to its previous state after an accidental deployment.

Reference

Installation

Creating a new Lektor website is out of scope for this document.

Check out the Quickstart page in the Lektor documentation to get started.

SLA

Design

The workflows around Lektor websites is heavily dependent on GitLab CI: it handles building the sites, running tests and deploying them to various environments including staging and production.

See service/ci for general documentation about GitLab CI.

CI build/test pipelines

The lektor.yml CI template is used to configure pipelines for build and testing Lektor website projects. Including this into the project's gitlab-ci.yml is usually sufficient for GitLab CI to "do the right thing".

There are several elements that can be used to customize the build process:

• LEKTOR_BUILD_FLAGS: this variable accepts a space separated list of flags to append to the lektor build command. For example, setting this variable to npm will cause -f npm to be appended to the build command.

• LEKTOR_PARTIAL_BUILD: this variable can be used to alter the build process occurring on non-default branches and l10n-staging jobs. When set (to anything), it will append commands defined in .setup-lektor-partial-build to the job's before_script. Its main purpose is to pre-process website sources to reduce the build times by trimming less-essential content which contribute a lot to build duration. See the web/tpo project CI for an example.

• TRANSLATION_BRANCH: this variable must contain the name of the translation repository branch used to store localization files. If this variable is absent, the website will be built without l10n.

Another method of customizing the build process is by overriding keys from the .lektor hash (defined in the lektor.yml template) from their own .gitlab-ci.yml file.

For example, this hash, added to gitlab-ci.yml will cause the jobs defined in the template to use a different image, and set GIT_STRATEGY to clone.

.lektor:
  image: ubuntu:latest
  variables:
    GIT_STRATEGY: clone

This is in addition to the ability to override the named job parameters directly in .gitlab-ci.yml.

CD pipelines and environments

The Tor Project Lektor websites are deployed automatically by GitLab by a process of continuous deployment (CD).

Staging and production

Deployments to staging and production environments are handled by the static-shim-deploy.yml CI template. The service/static-shim wiki page describes the prerequisites for GitLab to be able to upload websites to the static mirror system.

A basic Lektor project that deploys to production would have a .gitlab-ci.yml set up like this:

---
variables:
  SITE_URL: example.torproject.org

include:
  project: tpo/tpa/ci-templates
  file:
    - lektor.yml
    - static-shim-deploy.yml

See the #template-variables documentation for details about the variables involved in the deployment process.

See the #working-with-a-staging-environments documentation for details about adding a staging environment to a project's deployment workflow.

Review apps

Lektor projects which include static-shim-deploy.yml and have access to the REVIEW_STATIC_GITLAB_SHIM_SSH_PRIVATE_KEY CI variable (this includes all projects in the tpo/web namespace) have Review apps automatically enabled.

See the #working-with-review-apps documentation for details about how to use Review apps.

Localization staging

To support the work of translation contributors who work on the Tor Project websites, we automatically build and deploy special localized versions of the projects to reviews.torproject.net.

The workflow can be described as follows:

1. Translations are contributed on Transifex

2. Every 30 minutes, these changes are merged to the corresponding branches in the translation repository and pushed to tpo/translation

3. A project pipeline is triggered and runs the jobs from the lektor-l10n-staging-trigger.yml CI template

4. If the changed files include any .po which is >15% translated, a pipeline will be triggered in the Lektor project with the special L10N_STAGING variable added

5. In the Lektor project, the presence of the L10N_STAGING variable alters the regular build job: all languages >15% translated are built instead of only the officially supported languages for that project. The result is deployed to reviews.torproject.net/tpo/web/<project-name>/l10n

To enable localization staging for a Lektor project, it's sufficient to add this snippet in .gitlab-ci.yml in the relevant tpo/translation branch

variables:
  TRANSLATION_BRANCH : $CI_COMMIT_REF_NAME
  LEKTOR_PROJECT: tpo/web/<project-name>

include:
  - project: tpo/tpa/ci-templates
    file: lektor-l10n-staging-trigger.yml

Replace <project-name> with the name of the Lektor GitLab project.

Issues

Lektor website projects on GitLab have individual issue trackers, so problems related to specific websites such as typos, bad links, missing content or build problems should be filed in the relevant tracker.

For problems related to deployments or CI templates specifically, File or search for issues in the ci-templates issue tracker.

Maintainer, users, and upstream

Lektor websites are maintained in collaboration by the Web team and TPA.

Monitoring and testing

Currently there is no monitoring beyond the supporting infrastructure (eg. DNS, host servers, httpd, etc.).

Logs and metrics

Backups

There are no backups specific to Lektor.

Source code of our Lektor projects is backed up along with GitLab itself, and the production build artifacts themselves are picked up with those of the hosts comprising the static mirror system.

Other documentation

• Lektor documentation

Discussion

Overview

Goals

Must have

Nice to have

Non-Goals

Approvals required

Proposed Solution

Cost

Alternatives considered

  PV Name               /dev/sdb
  VG Name               vg_vineale
  PV Size               40,00 GiB / not usable 4,00 MiB
  Allocatable           yes 
  PE Size               4,00 MiB
  Total PE              10239
  Free PE               1279
  Allocated PE          8960
  PV UUID               CXKO15-Wze1-xY6y-rOO6-Tfzj-cDSs-V41mwe

Extend the volume group

The procedures below assume there is free space on the volume group for the operation. If there isn't you will need to add disks to the volume group, and grow the physical volume. For example:

pvcreate /dev/md123
vgextend vg_vineale /dev/md123

If the underlying disk was grown magically without your intervention, which happens in virtual hosting environments, you can also just extend the physical volume:

pvresize /dev/sdb

Note that if there's an underlying crypto layer, it needs to be resized as well:

cryptsetup resize $DEVICE_LABEL

In this case, the $DEVICE_LABEL is the device's name in /etc/crypttab, not the device name. For example, it would be /dev/mapper/crypt_sdb, not /dev/sdb.

Note that striping occurs at the logical volume level, not at the volume group level, see those instructions from RedHat and this definition.

Also note that you cannot mix physical volumes with different block sizes in the same volume group. This can between older and newer drives, and will yield a warning like:

Devices have inconsistent logical block sizes (512 and 4096).

This can, technically, be worked around with allow_mixed_block_sizes=1 in /etc/lvm/lvm.conf, but this can lead to data loss. It's possible to reformat the underlying LUKS volume with the --sector-size argument, see this answer as well.

See also the upstream documentation.

online procedure (ext3 and later)

Online resize has been possible ever since ext3 came out and it considered reliable enough for use. If you are unsure that you can trust that procedure, or if you have an ext2 filesystem, do not use this procedure and see the ext2 procedure below instead.

To resize the partition to take up all available free space, you should do the following:

1. extend the partition, in case of a logical volume:

   lvextend vg_vineale/srv -L +5G
   

   This might miss some extents, however. You can use the extent notation to take up all free space instead:

   lvextend vg_vineale/srv -l +1279
   

   If the partition sits directly on disk, use parted's resizepart command or fdisk to resize that first.

   To resize to take all available free space:

   lvextend vg_vineale/srv -l '+100%FREE'
   

2. resize the filesystem:

   resize2fs /dev/mapper/vg_vineale-srv
   

That's it! The resize2fs program automatically determines the size of the underlying "partition" (the logical volume, in most cases) and fixes the filesystem to fill the space.

Note that the resize process can take a while. Growing an active 20TB partition to 30TB took about 5 minutes, for example. The -p flag that could show progress only works in the "offline" procedure (below).

If the above fails because of the following error:

  Unable to resize logical volumes of cache type.

It's because the logical volume has a cache attached. Follow the above procedure to "uncache" the logical volume and then re-enable the cache.

WARNING: Make sure you remove the physical volume cache from the volume group before you resize, otherwise the logical volume will be extended to also cover that and re-enabling the cache won't be possible! A typical, incorrect session looks like:

root@materculae:~# lvextend -l '+100%FREE' vg_materculae/srv
  Unable to resize logical volumes of cache type.
root@materculae:~# lvconvert --uncache vg_materculae/srv
  Logical volume "srv_cache" successfully removed
  Logical volume vg_materculae/srv is not cached.
root@materculae:~# lvextend -l '+100%FREE' vg_materculae/srv
  Size of logical volume vg_materculae/srv changed from <150.00 GiB (38399 extents) to 309.99 GiB (79358 extents).
  Logical volume vg_materculae/srv successfully resized.
root@materculae:~# lvs
  LV   VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  srv  vg_materculae -wi-ao---- 309.99g
root@materculae:~# vgs
  VG            #PV #LV #SN Attr   VSize   VFree
  vg_materculae   2   1   0 wz--n- 309.99g    0
root@materculae:~# pvs
  PV         VG            Fmt  Attr PSize    PFree
  /dev/sdc   vg_materculae lvm2 a--   <10.00g    0
  /dev/sdd   vg_materculae lvm2 a--  <300.00g    0

A proper procedure is:

VG=vg_$(hostname)
FAST=/dev/sdc
lvconvert --uncache $VG/srv
vgreduce $VG/srv $FAST # remove the cache volume
lvextend -l '+100%FREE' $VG/srv # resize the volume
vgextend $VG $FAST # re-add the cache volume
lvcreate -n cache -l '100%FREE' $VG $FAST
lvconvert --type cache --cachevol cache $VG

And here's a successful run:

root@materculae:~# VG=vg_$(hostname)
root@materculae:~# FAST=/dev/sdc
root@materculae:~# vgreduce $VG $FAST
  Removed "/dev/sdc" from volume group "vg_materculae"
root@materculae:~# vgs
  VG            #PV #LV #SN Attr   VSize    VFree  
  vg_materculae   1   1   0 wz--n- <300.00g <10.00g
root@materculae:~# lvextend -l '+100%FREE' $VG
  Size of logical volume vg_materculae/srv changed from 150.00 GiB (38400 extents) to <300.00 GiB (76799 extents).
  Logical volume vg_materculae/srv successfully resized.
root@materculae:~# vgextend $VG $FAST
  Volume group "vg_materculae" successfully extended
root@materculae:~# lvcreate -n cache -l '100%FREE' $VG $FAST
  Logical volume "cache" created.
root@materculae:~# lvconvert --type cache --cachevol cache vg_materculae
Erase all existing data on vg_materculae/cache? [y/n]: y
  Logical volume vg_materculae/srv is now cached.
  Command on LV vg_materculae/cache_cvol requires LV with properties: lv_is_visible .

Note that the above output was edited for correctness: the actual run was much bumpier and involved shrinking the logical volume as the "incorrect" run was actually done in tpo/tpa/team#41258.

offline procedure (ext2)

To resize the partition to take up all available free space, you should do the following:

1. stop services and processes using the partition (will obviously vary):

   service apache2 stop
   

2. unmount the filesystem:

   umount /srv
   

3. check the filesystem:

   fsck -y -f /dev/mapper/vg_vineale-srv
   

4. extend the filesystem using the extent notation to take up all available space:

   lvextend vg_vineale/srv -l +1279
   

5. grow the filesystem (-p is for "show progress"):

   resize2fs -p /dev/mapper/vg_vineale-srv
   

6. recheck the filesystem:

   fsck  -f -y /dev/mapper/vg_vineale-srv
   

7. remount the filesystem and start processes:

   mount /srv
   service apache2 start
   

Shrinking

Shrinking the filesystem is also possible, but is more risky. Making an error in the commands in this section could incur data corruption or, more likely, data loss.

It is very important to reduce the size of the filesystem before resizing the size of the logical volume, so the order of the steps is critical. In the procedure below, we're enforcing this order by using lvm's ability to also resize ext4 filesystems to the requested size automatically.

1. First, identify which volume needs to be worked on.

   WARNING: this step is the most crucial one in the procedure. Make sure to verify what you've typed 3 times to be very certain you'll be launching commands on the correct volume before moving on (i.e. "measure twice, cut once")

   VG_NAME=vg_name
   LV_NAME=lv_name
   DEV_NAME=/dev/${VG_NAME}/${LV_NAME}
   

2. Unmount the filesystem:

   umount "$DEV_NAME"
   

   If the above command is not failing because the filesystem is in use, you'll need to stop processes using it. If that's impossible (for example when resizing /), you'll need to reboot in a separate operating system first, or shutdown the VM and work from the physical node below.

3. Forcibly check the filesystem:

   e2fsck -fy "$DEV_NAME"
   

4. Shrink both the filesystem and the logical volume at once:

   WARNING: make sure you get the size right here before launching the command

   Here we reduce to 5G (new absolute size for the volume):

   lvreduce -L 5G --resizefs "${VG_NAME}/${LV_NAME}"
   

   To reduce by 5G instead:

   lvreduce -L -5G --resizefs "${VG_NAME}/${LV_NAME}"
   

   TIP: You might want to ask a coworker to check your command right here, because this is a really risky command!

5. check the filesystem again:

   e2fsck -fy "$DEV_NAME"
   

6. If you want to resize the underlying device (for example, if this is a LVM inside a virtual machine on top of another LVM), you can also shrink the parent logical volume, physical volume, and crypto device (if relevant) at this point.

   lvreduce -L 5G vg/hostname
   pvresize /dev/sdY
   cryptsetup resize DEVICE_LABEL
   

   WARNING: this last step has not been tested.

Renaming

Rename volume group containing root

Assuming a situation where a machine was deployed successfully but the volume group name is not adequate and should be changed. In this example, we'll change vg_ganeti to vg_tbbuild05.

This operation requires at least one reboot, and a live rescue system if the root filesystem is encrypted.

First, rename the LVM volume group:

vgrename vg_ganeti vg_tbbuild05

Then adjust some configuration files and regenerate the initramfs to replace the old name:

sed -i 's/vg_ganeti/vg_tbbuild05/g' /etc/fstab
sed -i 's/vg_ganeti/vg_tbbuild05/g' /boot/grub/grub.cfg
update-initramfs -u -k all

The next step depends on whether the root volume is encrypted or not. If it's encrypted, the last command will output an error like:

update-initramfs: Generating /boot/initrd.img-5.10.0-14-amd64
cryptsetup: ERROR: Couldn't resolve device /dev/mapper/vg_ganeti-root
cryptsetup: WARNING: Couldn't determine root device

If this happens, boot the live rescue system and follow the remount procedure to chroot into the root filesystem of the machine. Then, inside the chroot, execute these two commands to ensure GRUB and the initramfs use the new root LV path/name:

update-grub
update-initramfs -u -k all

Then exit the chroot, cleanup and reboot back into the normal system.

If the root volume is not encrypted, the last steps should be enough to ensure the system boots. To ensure everything works as expected, run the update-grub command after rebooting and ensure grub.cfg retains the new volume group name.

Snapshots

This creates a snapshot for the "root" logical volume, with a 1G capacity:

lvcreate -s -L1G vg/root -n root-snapshot

Note that the "size" here needs to take into account not just the data written to the snapshot, but also data written to the parent logical volume. You can also specify the size as a percentage of the parent volume, for example this assumes you'll only rewrite 10% of the parent:

lvcreate -s -l 10%ORIGIN vg/root -n root-snapshot

If you're performing, for example, a major upgrade, you might want to have that be a fully replica of the parent volume:

lvcreate -s -l 100%ORIGIN vg/root -n root-snapshot

Make sure you destroy the snapshot when you're done with it, as keeping a snapshot around has an impact on performance and will cause issues when full:

lvremove vg/root-snapshot

You can also roll back to a previous snapshot:

lvconvert --merge vg/root-snapshot

Caching

WARNING: those instructions are deprecated. There's a newer, simpler way of setting up the cache that doesn't require two logical volumes, see the rebuild instructions for instructions that need to be adapted here. See also the lvmcache(7) manual page for further instructions.

Create the VG consisting of 2 block devices (a slow and a fast)

apt install lvm2 &&
vg="vg_$(hostname)_cache" &&
lsblk &&
echo -n 'slow disk: ' && read slow &&
echo -n 'fast disk: ' && read fast &&
vgcreate "$vg" "$slow" "$fast"

Create the srv LV, but leave a few (like 50?) extents empty on the slow disk. (lvconvert needs this extra free space later. That's probably a bug.)

pvdisplay &&
echo -n "#extents: " && read extents &&
lvcreate -l "$extents" -n srv "$vg" "$slow"

The -cache-meta disk should be 1/1000 the size of the -cache LV. (if it is slightly more that also shouldn't hurt.)

lvcreate -L 100MB -n srv-cache-meta "$vg" "$fast" &&
lvcreate -l '100%FREE' -n srv-cache "$vg" "$fast"

setup caching

lvconvert --type cache-pool --cachemode writethrough --poolmetadata "$vg"/srv-cache-meta "$vg"/srv-cache

lvconvert --type cache --cachepool "$vg"/srv-cache "$vg"/srv

Disabling / Recovering from a cache failure

If for some reason the cache LV is destroyed or lost (typically by naive operator error), it might be possible to restore the original LV functionality with:

lvconvert --uncache vg_colchicifolium/srv

Rebuilding the cache after removal

If you've just --uncached a volume, for example to resize it, you might want to re-establish the cache. For this, you can't follow the same procedure above, as that requires recreating a VG from scratch. Instead, you need to extend the VG and then create new volumes for the cache. It should look something like this:

1. extend the VG with the fast storage:

   VG=vg_$(hostname)
   FAST=/dev/sdc
   vgextend $VG $FAST
   

2. create a LV for the cache:

   lvcreate -n cache -l '100%FREE' $VG $FAST
   

3. add the cache to the existing LV to be cached:

   lvconvert --type cache --cachevol cache $VG
   

Example run:

root@colchicifolium:~# vgextend vg_colchicifolium /dev/sdc
  Volume group "vg_colchicifolium" successfully extended
root@colchicifolium:~# lvcreate -n cache -l '100%FREE' vg_colchicifolium /dev/sdc 
  Logical volume "cache" created.
root@colchicifolium:~# lvconvert --type cache --cachevol cache vg_colchicifolium
Erase all existing data on vg_colchicifolium/cache? [y/n]: y
  Logical volume vg_colchicifolium/srv is now cached.
  Command on LV vg_colchicifolium/cache_cvol requires LV with properties: lv_is_visible .

You can see the cache in action with the lvs command:

root@colchicifolium:~# lvs
  LV   VG                Attr       LSize  Pool         Origin      Data%  Meta%  Move Log Cpy%Sync Convert
  srv  vg_colchicifolium Cwi-aoC--- <1.68t [cache_cvol] [srv_corig] 0.01   13.03           0.00

You might get a modprobe error on the last command:

    root@colchicifolium:~# lvconvert --type cache --cachevol cache vg_colchicifolium
    Erase all existing data on vg_colchicifolium/cache? [y/n]: y
    modprobe: ERROR: could not insert 'dm_cache_smq': Operation not permitted
      /sbin/modprobe failed: 1
    modprobe: ERROR: could not insert 'dm_cache_smq': Operation not permitted
      /sbin/modprobe failed: 1
      device-mapper: reload ioctl on  (254:0) failed: Invalid argument
      Failed to suspend logical volume vg_colchicifolium/srv.
      Command on LV vg_colchicifolium/cache_cvol requires LV with properties: lv_is_visible .

That's because the kernel module can't be loaded. Reboot and try again.

See also the lvmcache(7) manual page for further instructions.

Troubleshooting

Recover previous lv configuration after wrong operation

You've just made a mistake and resized the wrong LV, or maybe resized the LV without resizing the filesystem first. Here's what you can do:

1. Stop all processes reading and writing from the volume that was mistakenly resized as soon as possible

   • Note that you might need to forcibly kill the processes. However, forcibly killing a database is generally not a good idea.

2. Look into /etc/lvm/archive and find the latest archive. Inspect the file in that latest archive to confirm that the sizes and names of all LVs are correct and match the state prior to the modification.

3. Unmount all volumes from all LVs in the volume group if that's possible. Don't forget bind mounts as well.

   • If your "/" partition is in one of the LVs you might need to reboot into a rescue system to perform the recovery.

4. Deactivate all volumes in the group:

   vgchange -a n vg_name
   

5. Restore the lvm config archive:

   vgcfgrestore -f /etc/lvm/archive/vg_name_00007-745337126.vg vg_name
   

6. Re-enable the LVs:

   vgchange -a y vg_name
   

7. You'll probably want to run a filesystem check on the volume that was wrongly resized. Watch out for what errors happen during the fsck: if it's encountering many issues and especially with unknown or erroneous files, you might want to consider restoring data from backup.

   fsck /dev/vg_name/lv-that-was-mistakenly-resized
   

8. Once that's done, if the state of all things seems ok, you can mount all of the volumes back up:

   mount -a
   

9. Finally, you can now start the processes that use the LVs.

This page documents the Cymru machines we have and how to (re)install them.

• How-to
  • Creating a new machine
  • Bootstrapping installer
  • Automated install procedure
    • remount procedure
    • Extra firmware
    • IP address
    • serial console
    • initramfs boot config
    • iSCSI access
  • SSH RACDM access
  • iDRAC password reset
    • Through the RACDM SSH interface
    • Through the web interface
  • Other BIOS configuration
  • idrac consoles
    • Management network access
      • IPsec
      • SOCKS5
    • Virtual console
      • Fixing arrow keys in the virtual console
    • Virtual machine basics
    • Booting a rescue image
    • Boot timings
    • Serial console access
      • BIOS configuration
      • Usage
    • Power management
    • Resetting the iDRAC
    • IP address change
    • Other documentation
  • Hardware RAID
  • Storage servers
    • Dump all information about a server
    • Listing disks
    • Host and group management
    • CHAP authentication
    • Creating a disk
    • Resizing a disk
    • Deleting a disk
    • Password change
    • Health check
    • IP address dump
    • Other random commands
    • iSCSI manual commands
• Reference
  • Points of presence
  • Hardware inventory
    • Servers
    • SAN cluster specifications
    • SAN management tools setup
    • iSCSI initiator setup
  • Benchmarks
    • Onboard SAS disks, chi-node-01
    • iSCSI load testing, chi-node-01
    • Raw DD test, iSCSI disks, chi-node-04
    • Comparison, NVMe disks, fsn-node-07
    • Comparison, SATA disks, fsn-node-02
  • Glossary
  • Network topoloy
• Discussion
  • Disabling the builtin RAID controller
  • Design
    • Storage
    • multipath configuration
    • Ganeti iSCSI integration
    • Private network access considerations
    • VLAN allocations

How-to

Creating a new machine

If you need to create a new machine (from metal) inside the cluster, you should probably follow this procedure:

1. Get access to the virtual console by:
   1. getting Management network access
   2. get the nasty Java-based Virtual console running
   3. boot a rescue image, typically grml
2. Bootstrap the installer
3. Follow the automated install procedure - be careful to follow all the extra steps as the installer is not fully automated and still somewhat flaky

If you want to create a Ganeti instance, you should really just follow the Ganeti documentation instead, as this page mostly talks about Cymru- and metal-specific things.

Bootstrapping installer

To get Debian installed, you need to bootstrap some Debian SSH server to allow our installer to proceed. This must be done by loading a grml live image through the Virtual console (booting a rescue image, below).

Once an image is loaded, you should do a "quick network configuration" in the grml menu (n key, or type grml-network in a shell). This will fire up a dialog interface to enter the server's IP address, netmask, gateway, and DNS. The first three should be allocated from DNS (in the 82.229.38.in-addr.arpa. file of the dns/domains.git repository). The latter should be set to some public nameserver for now (e.g. Google's 8.8.8.8).

Alternatively, you can use this one-liner to set IP address, DNS servers and start SSH with your SSH key in root's list:

echo nameserver 8.8.8.8 >> /etc/resolv.conf &&
ip link set dev eth0 up &&
ip addr add dev eth0 $address/$prefix &&
ip route add default via $gateway &&
mkdir -p /root/.ssh/ &&
echo "$PUBLIC_KEY" >> /root/.ssh/authorized_keys &&
service ssh restart

If you have booted with a serial console (which you should have), you should also be able to extract the SSH public keys at this point, with:

sed "s/^/$address /" < /etc/ssh/ssh_host_*.pub

This can be copy-pasted into your ~/.ssh/known_hosts file, or, to be compatible with the installer script below, you should instead use:

for key in /etc/ssh/ssh_host_*_key; do
    ssh-keygen -E md5 -l -f $key
done

TODO: make the fabric installer accept non-md5 keys.

Phew! Now you have a shell you can use to bootstrap your installer.

Automated install procedure

To install a new machine in the Cymru cluster, you first need to:

1. configure the BIOS to display in the serial console (see Serial console access)
2. get SSH access to the RACDM
3. change the admin iDRAC password
4. bootstrap the installer through the virtual console and (optionally, because it's easier to copy-paste and debug) through the serial console

From there on, the machine can be bootstrapped with a basic Debian installer with the Fabric code in the fabric-tasks git repository. Here's an example of a commandline:

./install -H root@38.229.82.112 \
          --fingerprint c4:6c:ea:73:eb:94:59:f2:c6:fb:f3:be:9d:dc:17:99 \
          hetzner-robot \
          --fqdn=chi-node-09.torproject.org \
          --fai-disk-config=installer/disk-config/gnt-chi-noraid \
          --package-list=installer/packages \
          --post-scripts-dir=installer/post-scripts/

Taking that apart:

• -H root@IP: the IP address picked from the zonefile
• --fingerprint: the ed25519 MD5 fingerprint from the previous setup
• hetzner-robot: the install job type (only robot supported for now)
• --fqdn=HOSTNAME.torproject.org: the Fully Qualified Domain Name to set on the machine, it is used in a few places, but the hostname is correctly set to the HOSTNAME part only
• --fai-disk-config=installer/disk-config/gnt-chi-noraid: the disk configuration, in fai-setup-storage(8) format
• --package-list=installer/packages: the base packages to install
• --post-scripts-dir=installer/post-scripts/: post-install scripts, magic glue that does everything

The last two are passed to grml-debootstrap and should rarely be changed (although they could be converted in to Fabric tasks themselves).

Note that the script will show you lines like:

STEP 1: SSH into server with fingerprint ...

Those correspond to the manual install procedure, below. If the procedure stops before the last step (currently STEP 12), there was a problem in the procedure, but the remaining steps can still be performed by hand.

If a problem occurs in the install, you can login to the rescue shell with:

ssh -o FingerprintHash=md5 -o UserKnownHostsFile=~/.ssh/authorized_keys.hetzner-rescue root@88.99.194.57

... and check the fingerprint against the previous one.

See new-machine for post-install configuration steps, then follow new-machine-mandos for setting up the mandos client on this host.

IMPORTANT: Do not forget the extra configuration steps, below.

Note that it might be possible to run this installer over an existing, on-disk install. But in my last attempts, it failed during setup-storage while attempting to wipe the filesystems. Maybe a pivot_root and unmounting everything would fix this, but at that point it becomes a bit too complicated.

remount procedure

If you need to do something post-install, this should bring you a working shell in the chroot.

First, set some variables according to the current environment:

export BOOTDEV=/dev/sda2 CRYPTDEV=/dev/sda3 ROOTFS=/dev/mapper/vg_ganeti-root

Then setup and enter the chroot:

cryptsetup luksOpen "$CRYPTDEV" "crypt_dev_${CRYPTDEV##*/}" &&
vgchange -a y ; \
mount "$ROOTFS" /mnt &&
for fs in /run /sys /dev /proc; do mount -o bind $fs "/mnt${fs}"; done &&
mount "$BOOTDEV" /mnt/boot &&
chroot /mnt /bin/bash

This will rebuild grub from within the chroot:

update-grub &&
grub-install /dev/sda

And this will cleanup after exiting chroot:

umount /mnt/boot &&
for fs in /dev /sys /run /proc; do umount "/mnt${fs}"; done &&
umount /mnt &&
vgchange -a n &&
cryptsetup luksClose "crypt_dev_${CRYPTDEV##*/}"

Extra firmware

TODO: make sure this is automated somehow?

If you're getting this error on reboot:

failed to load bnx2-mips-09-6.2.1b.fw firmware

Make sure firmware-bnx2 is installed.

IP address

TODO: in the last setup, the IP address had to be set in /etc/network/interfaces by hand. The automated install assumes DHCP works, which is not the case here.

TODO: IPv6 configuration also needs to be done by hand. hints in new-machine.

serial console

Add this to the grub config to get the serial console working, in (say) /etc/default/grub.d/serial.cfg:

# enable kernel's serial console on port 1 (or 0, if you count from there)
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX console=tty0 console=ttyS1,115200n8"
# same with grub itself
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

initramfs boot config

TODO: figure out the best way to setup the initramfs. So far we've dumped the IP address in /etc/default/grub.d/local-ipaddress.cfg like so:

# for dropbear-initramfs because we don't have dhcp
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX ip=38.229.82.111::38.229.82.1:255.255.255.0::eth0:off"

... but it seems it's also possible to specify the IP by configuring the initramfs itself, in /etc/initramfs-tools/conf.d/ip, for example with:

echo 'IP="${ip_address}::${gateway_ip}:${netmask}:${optional_fqdn}:${interface_name}:none"'

Then rebuild grub:

update-grub

iSCSI access

Make sure the node has access to the iSCSI cluster. For this, you need to add the node on the SANs, using SMcli, using this magic script:

create host userLabel="chi-node-0X" hostType=1 hostGroup="gnt-chi";
create iscsiInitiator iscsiName="iqn.1993-08.org.debian:01:chi-node-0X" userLabel="chi-node-0X-iscsi" host="chi-node-0X" chapSecret="[REDACTED]";

Make sure you set a strong password in [REDACTED]! That password should already be set by Puppet (from Trocla) in /etc/iscsi/iscsid.conf, on the client. See:

grep node.session.auth.password /etc/iscsi/iscsid.conf

You might also need to actually login to the SAN. First make sure you can see the SAN controllers on the network, with, for example, chi-san-01:

iscsiadm -m discovery -t st -p chi-san-01.priv.chignt.torproject.org

Then you need to login on all of those targets:

for s in chi-san-01 chi-san-03 chi-san-03; do
    iscsiadm -m discovery -t st -p ${s}.priv.chignt.torproject.org | head -n1 | grep -Po "iqn.\S+" | xargs -n1 iscsiadm -m node --login -T
done

TODO: shouldn't this be done by Puppet?

Then you should see the devices in lsblk and multipath -ll, for example, here's one disk on multiple controllers:

root@chi-node-08:~# multipath -ll
tb-build-03-srv (36782bcb00063c6a500000f88605b0aac) dm-6 DELL,MD32xxi
size=600G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='service-time 0' prio=14 status=active
| |- 9:0:0:7  sds  65:32  active ready running
| |- 6:0:0:7  sdaa 65:160 active ready running
| `- 4:0:0:7  sdz  65:144 active ready running
`-+- policy='service-time 0' prio=9 status=enabled
  |- 3:0:0:7  sdg  8:96   active ready running
  |- 10:0:0:7 sdw  65:96  active ready running
  `- 11:0:0:7 sdx  65:112 active ready running

See the storage servers section for more information.

SSH RACDM access

Note: this might already be enabled. Try to connect to the host over SSH before trying this.

Note that this requires console access, see the idrac consoles section below for more information.

It is important to enable the SSH server in the iDRAC so we have a more reasonable serial console interface than the outdated Java-based virtual console. (The SSH server is probably also outdated, but at least copy-paste works without running an old Ubuntu virtual machine.) To enable the SSH server, head for the management web interface and then:

1. in iDRAC settings, choose Network
2. pick the Services tab in the top menu
3. make sure the Enabled checkmark is ticked in the SSH section

Then you can access the RACDM interface over SSH.

iDRAC password reset

WARNING: note that the password length is arbitrarily limited, and the limit is not constant across different iDRAC interfaces. Some have 20 characters, some less (16 seems to work).

Through the RACDM SSH interface

1. locate the root user:

   racadm getconfig -u root
   

2. modify its password, changing $INDEX with the index value found above, in the cfgUserAdminIndex=$INDEX field

   racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i $INDEX newpassword
   

An example session:

/admin1-> racadm getconfig -u root
# cfgUserAdminIndex=2
cfgUserAdminUserName=root
# cfgUserAdminPassword=******** (Write-Only)
cfgUserAdminEnable=1
cfgUserAdminPrivilege=0x000001ff
cfgUserAdminIpmiLanPrivilege=4
cfgUserAdminIpmiSerialPrivilege=4
cfgUserAdminSolEnable=1


RAC1168: The RACADM "getconfig" command will be deprecated in a
future version of iDRAC firmware. Run the RACADM 
"racadm get" command to retrieve the iDRAC configuration parameters.
For more information on the get command, run the RACADM command
"racadm help get".

/admin1-> racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 2 [REDACTED]
Object value modified successfully


RAC1169: The RACADM "config" command will be deprecated in a
future version of iDRAC firmware. Run the RACADM 
"racadm set" command to configure the iDRAC configuration parameters.
For more information on the set command, run the RACADM command
"racadm help set".

Through the web interface

Before doing anything, the password should be reset in the iDRAC. Head for the management interface, then:

1. in iDRAC settings, choose User Authentication
2. click the number next to the root user (normally 2)
3. click Next
4. tick the Change password box and set a strong password, saved in the password manager
5. click Apply

Note that this requires console access, see the idrac consoles section below for more information.

Other BIOS configuration

• disable F1/F2 Prompt on Error in System BIOS Settings > Miscellaneous Settings

This can be done via SSH on a relatively recent version of iDRAC:

racadm set BIOS.MiscSettings.ErrPrompt Disabled
racadm jobqueue create BIOS.Setup.1-1

See also the serial console access documentation.

idrac consoles

"Consoles", in this context, are interfaces that allows you to connect to a server as if you you were there. They are sometimes called "out of band management", "idrac" (Dell), IPMI (SuperMicro and others), "KVM" (Keyboard, Video, Monitor) switches, or "serial console" (made of serial ports).

Dell servers have a management interface called "IDRAC" or DRAC ("Dell Remote Access Controller"). Servers at Cymru use iDRAC 7 which has upstream documentation (PDF, web archive).

There is a Python client for DRAC which allows for changing BIOS settings, but not much more.

Management network access

Before doing anything, we need access to the management network, which is isolated from the regular internet (see the network topology for more information).

IPsec

This can be done by configuring a "client" (i.e. a roaming IPsec node) inside the cluster. Anarcat did so with such a config in the Puppet profile::ganeti::chi class with a configuration detailed in the IPsec docs.

The TL;DR: once configured, this is, client side:

ip a add 172.30.141.242/32 dev br0
ipsec restart

On the server side (chi-node-01):

sysctl net.ipv4.ip_forward=1

Those are the two settings that are not permanent and might not have survived a reboot or a network disconnect.

Once that configuration is enabled, you should be able to ping inside 172.30.140.0/24 from the client, for example:

ping 172.30.140.110

Note that this configuration only works between chi-node-13 and chi-node-01. The IP 172.30.140.101 (currently eth2 on chi-node-01) is special and configured as a router only for the iDRAC of chi-node-13. The router on the other nodes is 172.30.140.1 which is incorrect, as it's the iDRAC of chi-node-01. All this needs to be cleaned up and put in Puppet more cleanly, see issue 40128.

An alternative to this is to use sshuttle to setup routing, which avoids the need to setup a router (net.ipv4.ip_forward=1 - although that might be tightened up a bit to restrict to some interfaces?).

SOCKS5

Another alternative that was investigated in the setup (in issue 40097) is to "simply" use ssh -D to setup a SOCKS proxy, which works for most of the web interface, but obviously might not work with the Java consoles. This simply works:

ssh -D 9099 chi-node-03.torproject.org

Then setup localhost:9099 as a SOCKS5 proxy in Firefox, that makes the web interface directly accessible. For newer iDRAC consoles, there is no Java stuff, so that works as well, which removes the need for IPsec altogether.

Obviously, it's possible to SSH directly into the RACADM management interfaces from the chi-node-X machines as well,.

Virtual console

Typically, users will connect to the "virtual console" over a web server. The "old" iDRAC 7 version we have deployed uses a Java applet or ActiveX. In practice, the former Java applets just totally fail in my experiments (even after bypassing security twice) so it's somewhat of a dead end. Apparently, this actually works on Internet Explorer 11, presumably on Windows.

Note: newer iDRAC versions (e.g. on chi-node-14) work natively in the web browser, so you do not need the following procedure at all.

An alternative is to boot an older Ubuntu release (e.g. 12.04, archive) and run a web browser inside of that session. On Linux distributions, the GNOME Boxes application provides an easy, no-brainer way to run such images. Alternatives include VirtualBox, virt-manager and others, of course. (Vagrant might be an option, but only has a 12.04 image (hashicorp/precise64) for VirtualBox, which isn't in Debian (anymore).

1. When booted in the VM, do this:

   sudo apt-get update
   sudo apt-get install icedtea-plugin
   

2. start Firefox and connect to the management interface.

3. You will be prompted for a username and password, then you will see the "Integrated Dell Remote Access Controller 7" page.

4. Pick the Console tab, and hit the Launch virtual console button

5. If all goes well, this should launch the "Java Web Start" command which will launch the Java applet.

6. This will prompt you for a zillion security warnings, accept them all

7. If all the stars align correctly, you should get a window with a copy of the graphical display of the computer.

Note that in my experience, the window starts off being minuscule. Hit the "maximize" button (a square icon) to make it bigger.

Fixing arrow keys in the virtual console

Now, it's possible that an annoying bug will manifest itself at this stage: because the Java applet was conceived to work with an old X11 version, the keycodes for the arrow keys may not work. Without these keys, choosing an alternative boot option cannot be done.

To fix this we can use a custom library designed to fix this exact problem with iDRAC web console:

https://github.com/anchor/idrac-kvm-keyboard-fix

The steps are:

1. First install some dependencies:

   sudo apt-get install build-essential git libx11-dev
   

2. Clone the repository:

   cd ~
   git clone https://github.com/anchor/idrac-kvm-keyboard-fix.git
   cd idrac-kvm-keyboard-fix
   

3. Review the contents of the repository.

4. Compile and install:

   make
   PATH="${PATH}:${HOME}/bin" make install
   

5. In Firefox, open about:preferences#applications

6. Next to "JNLP File" click the dropdown menu and select "Use other..."

7. Select the executable at ~/bin/javaws-idrac

8. Close and launch the Virtual Console again

Virtual machine basics

TODO: move this section (and the libvirt stuff above) to another page, maybe service/kvm?

TODO: automate this setup.

Using the virt-manager is a fairly straightforward way to get a Ubuntu Precise box up and running.

It might also be good to keep an installed Ubuntu release inside a virtual machine, because the "boot from live image" approach works only insofar as the machine doesn't crash.

Somehow the Precise installer is broken and tries to setup a 2GB partition for /, which fails during the install. You may have to redo the partitioning by hand to fix that.

You will also need to change the sources.list to point all hosts at old-releases.ubuntu.com instead of (say) ca.archive.ubuntu.com or security.ubuntu.com to be able to get the "latest" packages (including spice-vdagent, below). This may get you there, untested:

sed -i 's/\([a-z]*\.archive\|security\)\.ubuntu\.com/old-releases.ubuntu.com/' /etc/apt/sources.list

Note that you should install the spice-vdagent (or is it xserver-xorg-video-qxl?) package to get proper resolution. In practice, I couldn't make this work and instead hardcoded the resolution in /etc/default/grub with:

GRUB_GFXMODE=1280x720
GRUB_GFXPAYLOAD_LINUX=keep

Thanks to Louis-Philippe Veronneau for the tip.

If using virt-manager, make sure the gir1.2-spiceclientgtk-3.0 (package name may have changed) is installed otherwise you will get the error "SpiceClientGtk missing".

Finally, note that libvirt and virt-manager do not seem to properly configure NAT to be compatible with ipsec. The symptom of that problem is that the other end of the IPsec tunnel can be pinged from the host, but not the guest. A tcpdump will show that packets do not come out of the external host interface with the right IP address, for example here they come out of 192.168.0.177 instead of 172.30.141.244:

16:13:28.370324 IP 192.168.0.117 > 172.30.140.100: ICMP echo request, id 1779, seq 19, length 64

It's unclear why this is happening: it seems that the wrong IP is being chosen by the MASQUERADE rule. Normally, it should pick the ip that ip route get shows and that does show the right route:

# ip route get 172.30.140.100
172.30.140.100 via 192.168.0.1 dev eth1 table 220 src 172.30.141.244 uid 0 
    cache 

But somehow it doesn't. A workaround is to add a SNAT rule like this:

iptables -t nat -I LIBVIRT_PRT 2 -s 192.168.122.0/24 '!' -d '192.168.122.0/24' -j SNAT --to-source 172.30.141.244

Note that the rules are stateful, so this won't take effect for an existing route (e.g. for the IP you were pinging). Change to a different target to confirm it works.

It might have been able to hack at ip xfrm policy instead, to be researched further. Note that those problems somehow do not occur in GNOME Boxes.

Booting a rescue image

Using the virtual console, it's possible to boot the machine using an ISO or floppy image. This is useful for example when attempting to boot the Memtest86 program, when the usual Memtest86+ crashes or is unable to complete tests.

Note: It is also possible to load an ISO or floppy image (say for rescue) through the DRAC interface directly, in Overview -> Server -> Attached media. Unfortunately, only NFS and CIFS shares are supported, which is... not great. But we could, in theory, leverage this to rescue machines from each other on the network, but that would require setting up redundant NFS servers on the management interface, which is hardly practical.

It is possible to load an ISO through the virtual console, however.

This assumes you already have an ISO image to boot from locally (that means inside the VM if that is how you got the virtual console above). If not, try this:

wget https://download.grml.org/grml64-full_2021.07.iso

PRO TIP: you can mount an ISO image through the virtual image by presenting it as a CD/DVD driver. Then the Java virtual console will notice it and that will save you from copying this file into the virtual machine.

First, get a virtual console going (above). Then, you need to navigate the menus:

1. Choose the Launch Virtual Media option from the Virtual Media menu in the top left

2. Click the Add image button

3. Select the ISO or IMG image you have downloaded above

4. Tick the checkbox of the image in the Mapped column

5. Keep that window open! Bring the console back into focus

6. If available, choose the Virtual CD/DVD/ISO option in the Next Boot menu

7. Choose the Reset system (warm boot) option in the Power menu

If you haven't been able to change the Next Boot above, press F11 during boot to bring up the boot menu. Then choose Virtual CD if you mapped an ISO, or Virtual Floppy for a IMG.

If those menus are not familiar, you might have a different iDRAC version. Try those:

1. Choose the Map CD/DVD from the Virtual media menu

2. Choose the Virtual CD/DVD/ISO option in the Next Boot menu

3. Choose the Reset system (warm boot) option in the Power menu

The BIOS should find the ISO image and download it from your computer (or, rather, you'll upload it to the server) which will be slow as hell, yes.

If you are booting a grml image, you should probably add the following options to the Linux commandline (to save some typing, select the Boot options for grml64-full -> grml64-full: Serial console:

console=tty1 console=ttyS0,115200n8 ssh grml2ram

This will:

1. activate the serial console
2. start an SSH server with a random password
3. load the grml squashfs image to RAM

Some of those arguments (like ssh grml2ram) are in the grml cheatcodes page, others (like console) are builtin to the Linux kernel.

Once the system boots (and it will take a while, as parts of the disk image will need to be transferred): you should be able to login through the serial console instead. It should look something like this after a few minutes:

[  OK  ] Found device /dev/ttyS0.
[  OK  ] Started Serial Getty on ttyS0.
[  OK  ] Started D-Bus System Message Bus.


grml64-full 2020.06 grml ttyS0

grml login: root (automatic login)

Linux grml 5.6.0-2-amd64 #1 SMP Debian 5.6.14-2 (2020-06-09) x86_64
Grml - Linux for geeks

root@grml ~ # 

From there, you have a shell and can do magic stuff. Note that the ISO is still necessary to load some programs: only a minimal squashfs is loaded. To load the entire image, use toram instead of grml2ram, but note this will transfer the entire ISO image to the remote server's core memory, which can take a long time depending on your local bandwidth. On a 25/10mbps cable connection, it took over 90 minutes to sync the image which, clearly, is not as practical as loading the image on the fly.

Boot timings

It takes about 4 minutes for the Cymru machines to reboot and get to the LUKS password prompt.

1. POST check ("Checking memory..."): 0s
2. iDRAC setup: 45s
3. BIOS loading: 55s
4. PXE initialization: 70s
5. RAID controller: 75s
6. CPLD: 1m25s
7. Device scan ("Initializing firmware interfaces..."): 1m45
8. Lifecycle controller: 2m45
9. Scanning devices: 3m20
10. Starting bootloader: 3m25
11. Linux loading: 3m33
12. LUKS prompt: 3m50

This is the time it takes to reach each step in the boot with a "virtual media" (a grml ISO) loaded:

1. POST check ("Checking memory..."): 0s
2. iDRAC setup: 35s
3. BIOS loading: 45s
4. PXE initialization: 60s
5. RAID controller: 67s
6. CPLD: 1m20s
7. Device scan ("Initializing firmware interfaces..."): 1m37
8. Lifecycle controller: 2m44
9. Scanning devices: 3m15
10. Starting bootloader: 3m30

Those timings were calculated in "wall clock" time, using a manually operated stopwatch. The error is estimated to be around plus or minus 5 seconds.

Serial console access

It's possible to connect to DRAC over SSH, telnet, with IPMItool (see all the interfaces). Note that documentation refers to VNC access as well, but it seems that feature is missing from our firmware.

BIOS configuration

The BIOS needs to be configured to allow serial redirection to the iDRAC BMC.

On recent versions on iDRAC:

racadm set BIOS.SerialCommSettings.SerialComm OnConRedirCom2
racadm jobqueue create BIOS.Setup.1-1

On older versions, eg. PowerEdge R610 systems:

racadm config -g cfgSerial -o cfgSerialConsoleEnable 1
racadm config -g cfgSerial -o cfgSerialCom2RedirEnable 1
racadm config -g cfgSerial -o cfgSerialBaudRate 115200

See also the Other BIOS configuration section.

Usage

Typing connect in the SSH interface connects to the serial port. Another port can be picked with the console command, and the -h option will also show backlog (limited to 8kb by default):

console -h com2

That size can be changed with this command on the console:

racadm config -g cfgSerial -o cfgSerialHistorySize 8192

There are many more interesting "RAC" commands visible in the racadm help output.

The BIOS can also display in the serial console by entering the console (F2 in the BIOS splash screen) and picking System BIOS settings -> Serial communications -> Serial communication -> On with serial redirection via COM2 and Serial Port Address: Serial Device1=COM1,Serial Device2=COM2.

Pro tip. When the machine reboots, the following screen flashes really quickly:

Press the spacebar to pause...

KEY MAPPING FOR CONSOLE REDIRECTION:

Use the <ESC><1> key sequence for <F1>
Use the <ESC><2> key sequence for <F2>
Use the <ESC><0> key sequence for <F10>
Use the <ESC><!> key sequence for <F11>
Use the <ESC><@> key sequence for <F12>

Use the <ESC><Ctrl><M> key sequence for <Ctrl><M>
Use the <ESC><Ctrl><H> key sequence for <Ctrl><H>
Use the <ESC><Ctrl><I> key sequence for <Ctrl><I>
Use the <ESC><Ctrl><J> key sequence for <Ctrl><J>

Use the <ESC><X><X> key sequence for <Alt><x>, where x is any letter
key, and X is the upper case of that key

Use the <ESC><R><ESC><r><ESC><R> key sequence for <Ctrl><Alt><Del>

So this can be useful to send the dreaded F2 key through the serial console, for example.

To end the console session, type ^\ (Control-backslash).

Power management

The next boot device can be changed with the cfgServerBootOnce. To reboot a server, use racadm serveraction, for example:

racadm serveraction hardreset
racadm serveraction powercycle

Current status is shown with:

racadm serveraction powerstatus

This should be good enough to get us started. See also the upstream documentation.

Resetting the iDRAC

It can happen that the management interface gets hung. In my case it happened when I left a virtual machine disappear while connected to the iDRAC console overnight. The problem was that the web console login would just hang on "Verifying credentials".

The workaround is to reset the RAC with:

racadm racreset soft

If that's not good enough, try hard instead of soft, see also the (rather not much more helpful, I'm afraid) upstream documentation.

IP address change

To change the IP address of the iDRAC itself, you can use the racadm setniccfg command:

racadm setniccfg -s 172.30.140.13 255.255.255.0 172.30.140.101

It takes a while for the changes to take effect. In the latest change we actually lost access to the RACADM interface after 30 seconds, but it's unclear if that is because the VLAN was changed or it is because the change took 30 seconds to take effect.

More practically, it could be useful to use IPv6 instead of renumbering that interface, since access is likely to be over link-local addresses anyways. This will enable IPv6 on the iDRAC interface and set a link-local address:

racadm config -g cfgIPv6LanNetworking -o cfgIPv6Enable 1

The current network configuration (including the IPv6 link-local address) can be found in:

racadm getniccfg

See also this helpful guide for more network settings, as the official documentation is rather hard to parse.

Other documentation

• Integrated Dell Remote Access Controller (PDF)
• iDRAC 8/7 v2.50.50.50 RACADM CLI Guide (PDF)
• DSA also has some tools to talk to DRAC externally, but they are not public

Hardware RAID

The hardware RAID documentation lives in raid, see that document on how to recover from RAID failures and so on.

Storage servers

To talk to the storage servers, you'll need first to install the SMcli commandline tool, see the install instructions for more information on that.

In general, commands are in the form of:

SMcli $ADDRESS -c -S "$SCRIPT;"

Where:

• $ADDRESS is the management address (in 172.30.40.0/24) of the storage server
• $SCRIPT is a command, with a trailing semi-colon

All the commands are documented in the upstream manual (chapter 12 has all the commands listed alphabetically, but earlier chapters have topical instructions as well). What follows is a subset of those, with only the $SCRIPT part. So, for example, this script:

show storageArray profile;

Would be executed with something like:

SMcli 172.30.140.16 -c 'show storageArray profile;'

Be careful with quoting here: some scripts expect certain arguments to be quoted, and those quotes should be properly escaped (or quoted) in the shell.

Some scripts will require a password (for example to modify disks). That should be provided with the -p argument. Make sure you prefix the command with a "space" so it does not end up in the shell history:

 SMcli 172.30.140.16 -p $PASSWORD -c 'create virtualDisk [...];'

Note the leading space. A safer approach is to use the set session password command inside a script. For example, the equivalent command to the above, with a script, would be this script:

set session password $PASSWORD;
create virtualDisk [...];

And then call this script:

SMcli 172.30.140.16 -f script

Dump all information about a server

This will dump a lot of information about a server.

show storageArray profile;

Listing disks

Listing virtual disks, which are the ones visible from other nodes:

show allVirtualDisks;

Listing physical disks:

show allPhysicalDisks summary;

Details (like speed in RPMs) can also be seen with:

show allPhysicalDisks;

Host and group management

The existing machines in the gnt-chi cluster were all added at once, alongside a group, with this script:

show "Creating Host Group gnt-chi.";
create hostGroup userLabel="gnt-chi";

show "Creating Host chi-node-01 with Host Type Index 1 (Linux) on Host Group gnt-chi.";
create host userLabel="chi-node-01" hostType=1 hostGroup="gnt-chi";
show "Creating Host chi-node-02 with Host Type Index 1 (Linux) on Host Group gnt-chi.";
create host userLabel="chi-node-02" hostType=1 hostGroup="gnt-chi";
show "Creating Host chi-node-03 with Host Type Index 1 (Linux) on Host Group gnt-chi.";
create host userLabel="chi-node-03" hostType=1 hostGroup="gnt-chi";
show "Creating Host chi-node-04 with Host Type Index 1 (Linux) on Host Group gnt-chi.";
create host userLabel="chi-node-04" hostType=1 hostGroup="gnt-chi";

show "Creating iSCSI Initiator iqn.1993-08.org.debian:01:chi-node-01 with User Label chi-node-01-iscsi on host chi-node-01";
create iscsiInitiator iscsiName="iqn.1993-08.org.debian:01:chi-node-01" userLabel="chi-node-01-iscsi" host="chi-node-01";
show "Creating iSCSI Initiator iqn.1993-08.org.debian:01:chi-node-02 with User Label chi-node-02-iscsi on host chi-node-02";
create iscsiInitiator iscsiName="iqn.1993-08.org.debian:01:chi-node-02" userLabel="chi-node-02-iscsi" host="chi-node-02";
show "Creating iSCSI Initiator iqn.1993-08.org.debian:01:chi-node-03 with User Label chi-node-03-iscsi on host chi-node-03";
create iscsiInitiator iscsiName="iqn.1993-08.org.debian:01:chi-node-03" userLabel="chi-node-03-iscsi" host="chi-node-03";
show "Creating iSCSI Initiator iqn.1993-08.org.debian:01:chi-node-04 with User Label chi-node-04-iscsi on host chi-node-04";
create iscsiInitiator iscsiName="iqn.1993-08.org.debian:01:chi-node-04" userLabel="chi-node-04-iscsi" host="chi-node-04";

For new machines, only this should be necessary:

create host userLabel="chi-node-0X" hostType=1 hostGroup="gnt-chi";
create iscsiInitiator iscsiName="iqn.1993-08.org.debian:01:chi-node-04" userLabel="chi-node-0X-iscsi" host="chi-node-0X";

The iscsiName setting is in /etc/iscsi/initiatorname.iscsi, which is configured by Puppet to be derived from the hostname, so it can be reliably guessed above.

To confirm the iSCSI initiator name, you can run this command on the host:

iscsiadm -m session -P 1 | grep 'Iface Initiatorname' | sort -u 

Note that the above doesn't take into account CHAP authentication, covered below.

CHAP authentication

While we trust the local network (iSCSI is, after all, in the clear), as a safety precaution, we do have password-based (CHAP) authentication between the clients and the server. This is configured on the iscsiInitiator object on the SAN, with a setting like:

set iscsiInitiator ["chi-node-01-iscsi"] chapSecret="[REDACTED]";

The password comes from Trocla, in Puppet. It can be found in:

grep node.session.auth.password /etc/iscsi/iscsid.conf

The client's "username" is the iSCSI initiator identifier, which maps to the iscsiName setting on the SAN side. For chi-node-01, it looks something like:

iqn.1993-08.org.debian:01:chi-node-01

See above for details on the iSCSI initiator setup.

We do one way CHAP authentication (the clients authenticate to the server). We do not do it both ways, because we have multiple SAN servers and we haven't figured out how to make iscsid talk to multiple SANs at once (there's only one node.session.auth.username_in, and it's the iSCSI target identifier, so it can't be the same across SANs).

Creating a disk

This will create a disk:

create virtualDisk physicalDiskCount=3 raidLevel=5 userLabel="anarcat-test" capacity=20GB;

Map that group to a Logical Unit Number (LUN):

set virtualDisk ["anarcat-test"] logicalUnitNumber=3 hostGroup="gnt-chi";

Important: the LUN needs to be greater than 1, LUNs 0 and 1 are special. It should be the current highest LUN plus one.

TODO: we should figure out if the LUN can be assigned automatically, or how to find what the highest LUN currently is.

At this point, the device should show up on hosts in the hostGroup, as multiple /dev/sdX (for example, sdb, sdc, ..., sdg, if there are 6 "portals"). To work around that problem (and ensure high availability), the device needs to be added with multipath -a on the host:

root@chi-node-01:~# multipath -a /dev/sdb && sleep 3 && multipath -r
wwid '36782bcb00063c6a500000aa36036318d' added

To find the actual path to the device, given the LUN above, look into /dev/disk/by-path/ip-$ADDRESS-iscsi-$TARGET-lun-$LUN, for example:

root@chi-node-02:~# ls -al /dev/disk/by-path/*lun-3
lrwxrwxrwx 1 root root 9 Mar  4 20:18 /dev/disk/by-path/ip-172.30.130.22:3260-iscsi-iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655-lun-3 -> ../../sde
lrwxrwxrwx 1 root root 9 Mar  4 20:18 /dev/disk/by-path/ip-172.30.130.23:3260-iscsi-iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655-lun-3 -> ../../sdg
lrwxrwxrwx 1 root root 9 Mar  4 20:18 /dev/disk/by-path/ip-172.30.130.24:3260-iscsi-iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655-lun-3 -> ../../sdf
lrwxrwxrwx 1 root root 9 Mar  4 20:18 /dev/disk/by-path/ip-172.30.130.26:3260-iscsi-iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655-lun-3 -> ../../sdc
lrwxrwxrwx 1 root root 9 Mar  4 20:18 /dev/disk/by-path/ip-172.30.130.27:3260-iscsi-iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655-lun-3 -> ../../sdb
lrwxrwxrwx 1 root root 9 Mar  4 20:18 /dev/disk/by-path/ip-172.30.130.28:3260-iscsi-iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655-lun-3 -> ../../sdd

Then the device can be formatted, read and written to as a normal device, in:

/dev/mapper/36782bcb00063c6a500000aa36036318d

For example:

mkfs.ext4 -j /dev/mapper/36782bcb00063c6a500000aa36036318d
mount /dev/mapper/36782bcb00063c6a500000aa36036318d /mnt

To have a meaningful name in the device mapper, we need to add an alias in the multipath daemon. First, you need to find the device wwid:

root@chi-node-01:~# /lib/udev/scsi_id -g -u -d /dev/sdl
36782bcb00063c6a500000d67603f7abf

Then add this to the multipath configuration, with an alias, say in /etc/multipath/conf.d/web-chi-03-srv.conf:

multipaths {
        multipath {
                wwid 36782bcb00063c6a500000d67603f7abf
                alias web-chi-03-srv
        }
}

Then reload the multipath configuration:

multipath -r

Then add the device:

multipath -a /dev/sdl

Then reload the multipathd configuration (yes, again):

multipath -r

You should see the new device name in multipath -ll:

root@chi-node-01:~# multipath -ll
36782bcb00063c6a500000bfe603f465a dm-15 DELL,MD32xxi
size=20G features='5 queue_if_no_path pg_init_retries 50 queue_mode mq' hwhandler='1 rdac' wp=rw
web-chi-03-srv (36782bcb00063c6a500000d67603f7abf) dm-20 DELL,MD32xxi
size=500G features='5 queue_if_no_path pg_init_retries 50 queue_mode mq' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=6 status=active
| |- 11:0:0:4 sdi 8:128 active ready running
| |- 12:0:0:4 sdj 8:144 active ready running
| `- 9:0:0:4  sdh 8:112 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  |- 10:0:0:4 sdk 8:160 active ghost running
  |- 7:0:0:4  sdl 8:176 active ghost running
  `- 8:0:0:4  sdm 8:192 active ghost running
root@chi-node-01:~#

And lsblk:

# lsblk
[...]
sdh                                                                   8:112  0   500G  0 disk  
└─web-chi-03-srv                                                    254:20   0   500G  0 mpath 
sdi                                                                   8:128  0   500G  0 disk  
└─web-chi-03-srv                                                    254:20   0   500G  0 mpath 
sdj                                                                   8:144  0   500G  0 disk  
└─web-chi-03-srv                                                    254:20   0   500G  0 mpath 
sdk                                                                   8:160  0   500G  0 disk  
└─web-chi-03-srv                                                    254:20   0   500G  0 mpath 
sdl                                                                   8:176  0   500G  0 disk  
└─web-chi-03-srv                                                    254:20   0   500G  0 mpath 
sdm                                                                   8:192  0   500G  0 disk  
└─web-chi-03-srv                                                    254:20   0   500G  0 mpath 

See issue 40131.

Resizing a disk

To resize a disk, see the documentation at service/ganeti#resizing-an-iscsi-lun.

Deleting a disk

Before you delete a disk, you should make sure nothing uses it anymore. Where $ALIAS is the name of the device as seen from the Linux nodes (either a multipath alias or WWID):

gnt-cluster command "ls -l /dev/mapper/$ALIAS*"
# and maybe:
gnt-cluster command "kpartx -v -p -part -d /dev/mapper/$ALIAS"

Then you need to flush the multipath device somehow. The DSA ganeti install docs have ideas, grep for "Remove LUNs". They basically do blockdev --flushbufs on the multipath device, then multipath -f the device, then blockdev --flushbufs on each underlying device. And then they rescan the SCSI bus, using a sysfs file we don't have, great.

TODO: see how (or if?) we need to run blockdev --flushbufs on the multipath device, and how to guess the underlying block devices for flushing.

To Unmap a LUN, which will stop making a disk available to a specific host group:

remove virtualDisks ["anarcat-test"] lunMapping;

This will actually not show up on the clients until they run:

iscsiadm -m node --rescan

TODO: last time we tried this, the devices disappeared from lsblk, but they were still in /dev. Only a --logout cleanly removed the devices, which is obviously not practical.

To actually delete a disk:

delete virtualDisk ["anarcat-test"];

... this will obviously complete the catastrophe, and lose all data associated with the disk.

Password change

This will set the password for the admin interface to password:

set storageArray password="password";

Health check

show storageArray healthStatus;

IP address dump

This will show the IP address configuration of all the controllers:

show allControllers

A configured entry looks like this:

   RAID Controller Module in Enclosure 0, Slot 0
    

      Status:                      Online                                    
                                                                             
      Current configuration                                                  
         Firmware version:         07.80.41.60                               
            Appware version:       07.80.41.60                               
            Bootware version:      07.80.41.60                               
         NVSRAM version:           N26X0-780890-001                          
      Pending configuration                                                  
         Firmware version:         None                                      
            Appware version:       None                                      
            Bootware version:      None                                      
         NVSRAM version:           None                                      
         Transferred on:           None                                      
      Model name:                  2650                                      
      Board ID:                    2660                                      
      Submodel ID:                 143                                       
      Product ID:                  MD32xxi                                   
      Revision:                    0780                                      
      Replacement part number:     A00                                       
      Part number:                 0770D8                                    
      Serial number:               1A5009H                                   
      Vendor:                      DELL                                      
      Date of manufacture:         October 5, 2011                           
      Trunking supported:          No                                        
                                                                             
      Data Cache                                                             
         Total present:            1744 MB                                   
         Total used:               1744 MB                                   
      Processor cache:                                                       
         Total present:            304 MB                                    
      Cache Backup Device                                                    
         Status:                   Optimal                                   
         Type:                     SD flash physical disk                    
         Location:                 RAID Controller Module 0, Connector SD 1  
         Capacity:                 7,639 MB                                  
         Product ID:               Not Available                             
         Part number:              Not Available                             
         Serial number:            a0106234                                  
         Revision level:           10                                        
         Manufacturer:             Lexar                                     
         Date of manufacture:      August 1, 2011                            
      Host Interface Board                                                   
         Status:                   Optimal                                   
         Location:                 Slot 1                                    
         Type:                     iSCSI                                     
         Number of ports:          4                                         
         Board ID:                 0501                                      
         Replacement part number:  PN 0770D8A00                              
         Part number:              PN 0770D8                                 
         Serial number:            SN 1A5009H                                
         Vendor:                   VN 13740                                  
         Date of manufacture:      Not available                             
      Date/Time:                   Thu Feb 25 19:52:53 UTC 2021              

      Associated Virtual Disks (* = Preferred Owner): None


      RAID Controller Module DNS/Network name:   6MWKWR1   
         Remote login:                           Disabled  


      Ethernet port:              1                  
         Link status:             Up                 
         MAC address:             78:2b:cb:67:35:fd  
         Negotiation mode:        Auto-negotiate     
            Port speed:           1000 Mbps          
            Duplex mode:          Full duplex        
         Network configuration:   Static             
         IP address:              172.30.140.15      
         Subnet mask:             255.255.255.0      
         Gateway:                 172.30.140.1       
                                                     


      Physical Disk interface:  SAS     
         Channel:               1       
         Port:                  Out     
            Status:             Up      
         Maximum data rate:     6 Gbps  
         Current data rate:     6 Gbps  

      Physical Disk interface:  SAS     
         Channel:               2       
         Port:                  Out     
            Status:             Up      
         Maximum data rate:     6 Gbps  
         Current data rate:     6 Gbps  

      Host Interface(s): Unable to retrieve latest data; using last known state.


      Host interface:                                       iSCSI                           
         Host Interface Card(HIC):                          1                               
         Channel:                                           1                               
         Port:                                              0                               
         Link status:                                       Connected                       
         MAC address:                                       78:2b:cb:67:35:fe               
         Duplex mode:                                       Full duplex                     
         Current port speed:                                1000 Mbps                       
         Maximum port speed:                                1000 Mbps                       
         iSCSI RAID controller module                                                       
            Vendor:                                         ServerEngines Corporation       
            Part number:                                    ServerEngines SE-BE4210-S01     
            Serial number:                                  782bcb6735fe                    
            Firmware revision:                              2.300.310.15                    
         TCP listening port:                                3260                            
         Maximum transmission unit:                         9000 bytes/frame                
         ICMP PING responses:                               Enabled                         
         IPv4:                                              Enabled                         
            Network configuration:                          Static                          
            IP address:                                     172.30.130.22                   
               Configuration status:                        Configured                      
            Subnet mask:                                    255.255.255.0                   
            Gateway:                                        0.0.0.0                         
            Ethernet priority:                              Disabled                        
               Priority:                                    0                               
            Virtual LAN (VLAN):                             Disabled                        
               VLAN ID:                                     1                               
         IPv6:                                              Disabled                        
            Auto-configuration:                             Enabled                         
            Local IP address:                               fe80:0:0:0:7a2b:cbff:fe67:35fe  
               Configuration status:                        Unconfigured                    
            Routable IP address 1:                          0:0:0:0:0:0:0:0                 
               Configuration status:                        Unconfigured                    
            Routable IP address 2:                          0:0:0:0:0:0:0:0                 
               Configuration status:                        Unconfigured                    
            Router IP address:                              0:0:0:0:0:0:0:0                 
            Ethernet priority:                              Disabled                        
               Priority:                                    0                               
            Virtual LAN (VLAN):                             Disabled                        
               VLAN ID:                                     1                               
            Hop limit:                                      64                              
            Neighbor discovery                                                              
               Reachable time:                              30000 ms                        
               Retransmit time:                             1000 ms                         
               Stale timeout:                               30000 ms                        
               Duplicate address detection transmit count:  1                               

A disabled port would looks like:

      Host interface:                                       iSCSI                           
         Host Interface Card(HIC):                          1                               
         Channel:                                           4                               
         Port:                                              3                               
         Link status:                                       Disconnected                    
         MAC address:                                       78:2b:cb:67:36:01               
         Duplex mode:                                       Full duplex                     
         Current port speed:                                UNKNOWN                         
         Maximum port speed:                                1000 Mbps                       
         iSCSI RAID controller module                                                       
            Vendor:                                         ServerEngines Corporation       
            Part number:                                    ServerEngines SE-BE4210-S01     
            Serial number:                                  782bcb6735fe                    
            Firmware revision:                              2.300.310.15                    
         TCP listening port:                                3260                            
         Maximum transmission unit:                         9000 bytes/frame                
         ICMP PING responses:                               Enabled                         
         IPv4:                                              Enabled                         
            Network configuration:                          Static                          
            IP address:                                     172.30.130.25                   
               Configuration status:                        Unconfigured                    
            Subnet mask:                                    255.255.255.0                   
            Gateway:                                        0.0.0.0                         
            Ethernet priority:                              Disabled                        
               Priority:                                    0                               
            Virtual LAN (VLAN):                             Disabled                        
               VLAN ID:                                     1                               
         IPv6:                                              Disabled                        
            Auto-configuration:                             Enabled                         
            Local IP address:                               fe80:0:0:0:7a2b:cbff:fe67:3601  
               Configuration status:                        Unconfigured                    
            Routable IP address 1:                          0:0:0:0:0:0:0:0                 
               Configuration status:                        Unconfigured                    
            Routable IP address 2:                          0:0:0:0:0:0:0:0                 
               Configuration status:                        Unconfigured                    
            Router IP address:                              0:0:0:0:0:0:0:0                 
            Ethernet priority:                              Disabled                        
               Priority:                                    0                               
            Virtual LAN (VLAN):                             Disabled                        
               VLAN ID:                                     1                               
            Hop limit:                                      64                              
            Neighbor discovery                                                              
               Reachable time:                              30000 ms                        
               Retransmit time:                             1000 ms                         
               Stale timeout:                               30000 ms                        
               Duplicate address detection transmit count:  1                               

Other random commands

Show how virtual drives map to specific LUN mappings:

show storageArray lunmappings;

Save config to (local) disk:

save storageArray configuration file="raid-01.conf" allconfig;

iSCSI manual commands

Those are debugging commands that were used to test the system, and should normally not be necessary. Those are basically managed automatically by iscsid.

Discover storage units interfaces:

iscsiadm -m discovery -t st -p 172.30.130.22

Pick one of those targets, then login:

iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655 -p 172.30.130.22 --login

This will show details about the connection, including your iSCSI initiator name:

iscsiadm -m session -P 1

This will also show recognized devices:

iscsiadm -m session -P 3

This will disconnect from the iSCSI host:

iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655 -p 172.30.130.22 --logout

And this will... rescan the host? Not actually sure what this does:

iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3200i.6782bcb00063c6a5000000004ed6d655 -p 172.30.130.22 --rescan

Some of those commands were cargo-culted from this guide.

Note that the deployment guide has more information about network topology and such configuration.

Reference

Points of presence

We actually have two points of presence at cymru: wherever the moly machine is (and is deprecated, see issue 29974) and the gnt-chi cluster. This documentation mostly concerns the latter.

Hardware inventory

There are two main cluster of machines at the main PoP:

• 13 old servers (mostly Dell R610 or R620 2xXeon with a maximum of 386GB RAM per node and 2x500GB SAS disks)
• 8 storage arrays (Dell MD1220 or MD3200 21TB)
• 1 "newer" server(Dell PowerEdge R640 2 Xeon Gold 6230 CPU @ 2.10GHz (40 cores total), 1536 GB of RAM, 2x900GB SSD Intel(R) X550 4-port 10G Ethernet NIC)

Servers

The "servers" are named chi-node-X, where X is a digit from 01 to 13. They are generally used for the gnt-chi Ganeti cluster, except for the last machine(s), assigned to bare-metal GitLab services (see issue 40095 and CI documentation).

• chi-node-01: Ganeti node (#40065) (typically master)
• chi-node-02: Ganeti node (#40066)
• chi-node-03: Ganeti node (#40067)
• chi-node-04: Ganeti node (#40068)
• chi-node-05: kept for spare parts because of hardware issues (#40377)
• chi-node-06: Ganeti node (#40390)
• chi-node-07: Ganeti node (#40670)
• chi-node-08: Ganeti node (#40410)
• chi-node-09: Ganeti node (#40528)
• chi-node-10: Ganeti node (#40671)
• chi-node-11: Ganeti node (#40672)
• chi-node-12: shadow-small simulator node (tpo/tpa/team#40557)
• chi-node-13: first test CI node (tpo/tpa/team#40095)
• chi-node-14: shadow simulator node (tpo/tpa/team#40279)

Memory capacity varies between nodes:

• Nodes 1-4: 384GB (24x16GB)
• 5-7: 96GB (12x8GB)
• 8-13: 192GB (12x16GB)

SAN cluster specifications

There are 4 Dell MD3220i iscsi hardware raid units. Each MD3220i has a MD1220 expansion unit attached for a total of 48 900GB disks per unit (head unit + expansion unit). This provides roughly 172 TB of raw storage ((900GB x 192 disk)/1000) = 172 TB. These storage arrays are quite flexible and provide the ability to create numerous independent volume groups per unit. They also are capable of tagging spare disks for auto disk replacement of failed hard drives.

Upstream has a technical guide book with more complete specifications.

The machines do not run a regular operating system (like, say Linux), or at least does not provide traditional commandline-based interfaces like telnet, SSH or even a web interface. Operations are performed through a proprietary tool called "SMcli", detailed below.

Here's the exhaustive list of the hardware RAID units -- which we call SAN:

• chi-san-01: ~28TiB total: 28 1TB 7200 RPM drives
• chi-san-02: ~40TiB total: 40 1TB 7200 RPM drives
• chi-san-03: ~36TiB total: 47 800GB 10000 RPM drives
• chi-san-04: ~38TiB total, 48 800GB 10000 RPM drives
• Total: 144TiB, not counting mirrors (around 72TiB total in RAID-1, 96TiB in RAID-5)

A node that is correctly setup has the correct host groups, hosts, and iSCSI initiators setup, with CHAP passwords.

All SANs were checked for the following during the original setup:

• batteries status ("optimal")
• correct labeling (chi-san-0X)
• disk inventory (replace or disable all failing disks)
• setup spares

Spare disks can easily be found at harddrivesdirect.com, but are fairly expensive for this platform (115$USD for 1TB 7.2k RPM, 145$USD for 10kRPM). It seems like the highest density per drive they have available is 2TB, which would give us about 80TiB per server, but at the whopping cost of 12,440$USD ($255 per unit in a 5-pack)!

It must be said that this site takes a heavy markup... The typical drive used in the array (Seagate ST9900805SS, 1TB 7.2k RPM) sells for 186$USD right now, while it's 154$USD at NewEgg and 90$USD at Amazon. Worse, a typical Seagate IronWolf 8TB SATA sells for 516$USD while Newegg lists them at 290$USD. That "same day delivery" has a cost... And it's actually fairly hard to find those old drives in other sites, so we probably pay a premium there as well.

SAN management tools setup

The access the iSCSI servers, you need to setup the (proprietary) SMCli utilities from Dell. First, you need to extract the software from a ISO:

apt install xorriso
curl -o dell.iso https://downloads.dell.com/FOLDER04066625M/1/DELL_MDSS_Consolidated_RDVD_6_5_0_1.iso
osirrox -indev dell.iso -extract /linux/mdsm/SMIA-LINUXX64.bin dell.bin
./dell.bin

Click through the installer, which will throw a bunch of junk (including RPM files and a Java runtime!) inside /opt. To generate and install a Debian package:

alien --scripts /opt/dell/mdstoragemanager/*.rpm
dpkg -i smruntime* smclient*

The scripts shipped by Dell assume that /bin/sh is a bash shell (or, more precisely, that the source command exists, which is not POSIX). So we need to patch that:

sed -i '1s,#!/bin/sh,#!/bin/bash,' /opt/dell/mdstoragemanager/client/*

Then, if the tool works, at all, a command like this should yield some output:

SMcli 172.30.140.16 -c "show storageArray profile;"

... assuming there's a server on the other side, of course.

Note that those instructions derive partially from the upstream documentation. The ISO can also be found from the download site. See also those instructions.

iSCSI initiator setup

The iSCSI setup on the Linux side of things is handled automatically by Puppet, in the profile::iscsi class, which is included in the profile::ganeti::chi class. That will setup packages, configuration, and passwords for iSCSI clients.

There still needs to be some manual configuration for the SANs to be found.

Discover the array:

iscsiadm -m discovery -t sendtargets -p 172.30.130.22

From there on, the devices exported to this initiator should show up in lsblk, fdisk -l, /proc/partitions, or lsscsi, for example:

root@chi-node-01:~# lsscsi  | grep /dev/
[0:2:0:0]    disk    DELL     PERC H710P       3.13  /dev/sda 
[5:0:0:0]    cd/dvd  HL-DT-ST DVD-ROM DU70N    D300  /dev/sr0 
[7:0:0:3]    disk    DELL     MD32xxi          0780  /dev/sde 
[8:0:0:3]    disk    DELL     MD32xxi          0780  /dev/sdg 
[9:0:0:3]    disk    DELL     MD32xxi          0780  /dev/sdb 
[10:0:0:3]   disk    DELL     MD32xxi          0780  /dev/sdd 
[11:0:0:3]   disk    DELL     MD32xxi          0780  /dev/sdf 
[12:0:0:3]   disk    DELL     MD32xxi          0780  /dev/sdc 

Next you need to actually add the disk to multipath, with:

multipath -a /dev/sdb

For example:

# multipath -a /dev/sdb
wwid '36782bcb00063c6a500000aa36036318d' added

Then the device is available as a unique device in:

/dev/mapper/36782bcb00063c6a500000aa36036318d

... even though there are multiple underlying devices.

Benchmarks

Overall, the hardware in the gnt-chi cluster is dated, mainly because it lacks fast SSD disks. It can still get respectable performance, because the disks were top of the line when they were setup. In general, you should expect:

• local (small) disks:
  • read: IOPS=1148, BW=4595KiB/s (4706kB/s)
  • write: IOPS=2213, BW=8854KiB/s (9067kB/s)
• iSCSI (network, large) disks:
  • read: IOPS=26.9k, BW=105MiB/s (110MB/s) (gigabit network saturation, probably cached by the SAN)
  • random write: IOPS=264, BW=1059KiB/s (1085kB/s)
  • sequential write: 11MB/s (dd)

In other words, local disks can't quite saturate network (far from it: they don't even saturate a 100mbps link). Network disks seem to be able to saturate gigabit at first glance, but that's probably a limitation of the benchmark. Writes are much slower, somewhere around 8mbps.

Compare this with a more modern setup:

• NVMe:
  • read: IOPS=138k, BW=541MiB/s (567MB/s)
  • write: IOPS=115k, BW=448MiB/s (470MB/s)
• SATA:
  • read: IOPS=5550, BW=21.7MiB/s (22.7MB/s)
  • write: IOPS=199, BW=796KiB/s (815kB/s)

Notice how the large disk writes are actually lower than the iSCSI store in this case, but this could be a fluke because of the existing load on the gnt-fsn cluster.

Onboard SAS disks, chi-node-01

root@chi-node-01:~/bench# fio --name=stressant --group_reporting <(sed /^filename=/d /usr/share/doc/fio/examples/basic-verify.fio) --runtime=1m  --filename=test --size=100m
stressant: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.12
Starting 2 processes
stressant: Laying out IO file (1 file / 100MiB)
Jobs: 1 (f=1): [_(1),V(1)][94.3%][r=104MiB/s][r=26.6k IOPS][eta 00m:21s]                
stressant: (groupid=0, jobs=1): err= 0: pid=13409: Wed Mar 24 17:40:23 2021
  read: IOPS=150k, BW=585MiB/s (613MB/s)(100MiB/171msec)
    clat (nsec): min=980, max=1033.1k, avg=6290.36, stdev=46177.07
     lat (nsec): min=1015, max=1033.1k, avg=6329.40, stdev=46177.22
    clat percentiles (nsec):
     |  1.00th=[  1032],  5.00th=[  1048], 10.00th=[  1064], 20.00th=[  1096],
     | 30.00th=[  1128], 40.00th=[  1144], 50.00th=[  1176], 60.00th=[  1192],
     | 70.00th=[  1224], 80.00th=[  1272], 90.00th=[  1432], 95.00th=[  1816],
     | 99.00th=[244736], 99.50th=[428032], 99.90th=[618496], 99.95th=[692224],
     | 99.99th=[774144]
  lat (nsec)   : 1000=0.03%
  lat (usec)   : 2=97.01%, 4=0.84%, 10=0.07%, 20=0.47%, 50=0.13%
  lat (usec)   : 100=0.12%, 250=0.35%, 500=0.68%, 750=0.29%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=8.82%, sys=27.65%, ctx=372, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
write-and-verify: (groupid=0, jobs=1): err= 0: pid=13410: Wed Mar 24 17:40:23 2021
  read: IOPS=1148, BW=4595KiB/s (4706kB/s)(1024MiB/228181msec)
    slat (usec): min=5, max=547, avg=21.60, stdev= 8.70
    clat (usec): min=22, max=767720, avg=13899.92, stdev=26025.10
     lat (usec): min=42, max=767773, avg=13921.96, stdev=26027.93
    clat percentiles (usec):
     |  1.00th=[    42],  5.00th=[    51], 10.00th=[    56], 20.00th=[    65],
     | 30.00th=[   117], 40.00th=[   200], 50.00th=[  4146], 60.00th=[  8029],
     | 70.00th=[ 13566], 80.00th=[ 21890], 90.00th=[ 39060], 95.00th=[ 60031],
     | 99.00th=[123208], 99.50th=[156238], 99.90th=[244319], 99.95th=[287310],
     | 99.99th=[400557]
  write: IOPS=2213, BW=8854KiB/s (9067kB/s)(1024MiB/118428msec); 0 zone resets
    slat (usec): min=6, max=104014, avg=36.98, stdev=364.05
    clat (usec): min=62, max=887491, avg=7187.20, stdev=7152.34
     lat (usec): min=72, max=887519, avg=7224.67, stdev=7165.15
    clat percentiles (usec):
     |  1.00th=[  157],  5.00th=[  383], 10.00th=[  922], 20.00th=[ 1909],
     | 30.00th=[ 2606], 40.00th=[ 3261], 50.00th=[ 4146], 60.00th=[ 7111],
     | 70.00th=[10421], 80.00th=[13042], 90.00th=[15795], 95.00th=[18220],
     | 99.00th=[25822], 99.50th=[32900], 99.90th=[65274], 99.95th=[72877],
     | 99.99th=[94897]
   bw (  KiB/s): min= 4704, max=95944, per=99.93%, avg=8847.51, stdev=6512.44, samples=237
   iops        : min= 1176, max=23986, avg=2211.85, stdev=1628.11, samples=237
  lat (usec)   : 50=2.27%, 100=11.27%, 250=10.32%, 500=1.76%, 750=1.19%
  lat (usec)   : 1000=0.86%
  lat (msec)   : 2=5.57%, 4=15.85%, 10=17.35%, 20=21.25%, 50=8.72%
  lat (msec)   : 100=2.72%, 250=0.82%, 500=0.04%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.67%, sys=4.52%, ctx=296808, majf=0, minf=7562
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5044KiB/s (5165kB/s), 4595KiB/s-585MiB/s (4706kB/s-613MB/s), io=1124MiB (1179MB), run=171-228181msec
  WRITE: bw=8854KiB/s (9067kB/s), 8854KiB/s-8854KiB/s (9067kB/s-9067kB/s), io=1024MiB (1074MB), run=118428-118428msec

Disk stats (read/write):
    dm-1: ios=262548/275002, merge=0/0, ticks=3635324/2162480, in_queue=5799708, util=100.00%, aggrios=262642/276055, aggrmerge=0/0, aggrticks=3640764/2166784, aggrin_queue=5807820, aggrutil=100.00%
    dm-0: ios=262642/276055, merge=0/0, ticks=3640764/2166784, in_queue=5807820, util=100.00%, aggrios=262642/267970, aggrmerge=0/8085, aggrticks=3633173/1921094, aggrin_queue=5507676, aggrutil=99.16%
  sda: ios=262642/267970, merge=0/8085, ticks=3633173/1921094, in_queue=5507676, util=99.16%

iSCSI load testing, chi-node-01

root@chi-node-01:/mnt# fio --name=stressant --group_reporting <(sed /^filename=/d /usr/share/doc/fio/examples/basic-verify.fio; echo size=100m) --runtime=1m  --filename=test --size=100m
stressant: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.12
Starting 2 processes
write-and-verify: Laying out IO file (1 file / 100MiB)
Jobs: 1 (f=0): [_(1),f(1)][100.0%][r=88.9MiB/s][r=22.8k IOPS][eta 00m:00s]               
stressant: (groupid=0, jobs=1): err= 0: pid=18332: Wed Mar 24 17:56:02 2021
  read: IOPS=26.9k, BW=105MiB/s (110MB/s)(100MiB/952msec)
    clat (nsec): min=1214, max=7423.1k, avg=35799.85, stdev=324182.56
     lat (nsec): min=1252, max=7423.2k, avg=35889.53, stdev=324181.89
    clat percentiles (nsec):
     |  1.00th=[   1400],  5.00th=[   2128], 10.00th=[   2288],
     | 20.00th=[   2512], 30.00th=[   2576], 40.00th=[   2608],
     | 50.00th=[   2608], 60.00th=[   2640], 70.00th=[   2672],
     | 80.00th=[   2704], 90.00th=[   2800], 95.00th=[   3440],
     | 99.00th=[ 782336], 99.50th=[3391488], 99.90th=[4227072],
     | 99.95th=[4358144], 99.99th=[4620288]
   bw (  KiB/s): min=105440, max=105440, per=55.81%, avg=105440.00, stdev= 0.00, samples=1
   iops        : min=26360, max=26360, avg=26360.00, stdev= 0.00, samples=1
  lat (usec)   : 2=3.30%, 4=92.34%, 10=2.05%, 20=0.65%, 50=0.08%
  lat (usec)   : 100=0.01%, 250=0.11%, 500=0.16%, 750=0.28%, 1000=0.11%
  lat (msec)   : 2=0.11%, 4=0.67%, 10=0.13%
  cpu          : usr=4.94%, sys=12.83%, ctx=382, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
write-and-verify: (groupid=0, jobs=1): err= 0: pid=18333: Wed Mar 24 17:56:02 2021
  read: IOPS=23.6k, BW=92.2MiB/s (96.7MB/s)(100MiB/1084msec)
    slat (nsec): min=6524, max=66741, avg=15619.91, stdev=6159.27
    clat (usec): min=331, max=52833, avg=658.14, stdev=1305.45
     lat (usec): min=355, max=52852, avg=674.08, stdev=1305.57
    clat percentiles (usec):
     |  1.00th=[  420],  5.00th=[  469], 10.00th=[  502], 20.00th=[  537],
     | 30.00th=[  570], 40.00th=[  594], 50.00th=[  619], 60.00th=[  644],
     | 70.00th=[  676], 80.00th=[  709], 90.00th=[  758], 95.00th=[  799],
     | 99.00th=[  881], 99.50th=[  914], 99.90th=[ 1188], 99.95th=[52691],
     | 99.99th=[52691]
  write: IOPS=264, BW=1059KiB/s (1085kB/s)(100MiB/96682msec); 0 zone resets
    slat (usec): min=15, max=110293, avg=112.91, stdev=1199.05
    clat (msec): min=3, max=593, avg=60.30, stdev=52.88
     lat (msec): min=3, max=594, avg=60.41, stdev=52.90
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   15], 10.00th=[   17], 20.00th=[   23],
     | 30.00th=[   29], 40.00th=[   35], 50.00th=[   44], 60.00th=[   54],
     | 70.00th=[   68], 80.00th=[   89], 90.00th=[  126], 95.00th=[  165],
     | 99.00th=[  259], 99.50th=[  300], 99.90th=[  426], 99.95th=[  489],
     | 99.99th=[  592]
   bw (  KiB/s): min=  176, max= 1328, per=99.67%, avg=1055.51, stdev=127.13, samples=194
   iops        : min=   44, max=  332, avg=263.86, stdev=31.78, samples=194
  lat (usec)   : 500=4.96%, 750=39.50%, 1000=5.42%
  lat (msec)   : 2=0.08%, 4=0.01%, 10=0.27%, 20=7.64%, 50=20.38%
  lat (msec)   : 100=13.81%, 250=7.34%, 500=0.56%, 750=0.02%
  cpu          : usr=0.88%, sys=3.13%, ctx=34211, majf=0, minf=628
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,25600,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=185MiB/s (193MB/s), 92.2MiB/s-105MiB/s (96.7MB/s-110MB/s), io=200MiB (210MB), run=952-1084msec
  WRITE: bw=1059KiB/s (1085kB/s), 1059KiB/s-1059KiB/s (1085kB/s-1085kB/s), io=100MiB (105MB), run=96682-96682msec

Disk stats (read/write):
    dm-28: ios=22019/25723, merge=0/1157, ticks=16070/1557068, in_queue=1572636, util=99.98%, aggrios=4341/4288, aggrmerge=0/0, aggrticks=3089/259432, aggrin_queue=262419, aggrutil=99.79%
  sdm: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
  sdk: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
  sdi: ios=8686/8573, merge=0/0, ticks=6409/526657, in_queue=532844, util=99.79%
  sdl: ios=8683/8576, merge=0/0, ticks=6091/513333, in_queue=519120, util=99.75%
  sdj: ios=8678/8580, merge=0/0, ticks=6036/516604, in_queue=522552, util=99.77%
  sdh: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

Raw DD test, iSCSI disks, chi-node-04

dd fares much better, possibly because we're doing sequential writing:

root@chi-node-04:/var/log/ganeti/os# dd if=/dev/zero of=/dev/disk/by-id/dm-name-tb-builder-03-root status=progress
10735108608 bytes (11 GB, 10 GiB) copied, 911 s, 11.8 MB/s
dd: writing to '/dev/disk/by-id/dm-name-tb-builder-03-root': No space left on device
20971521+0 records in
20971520+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 914.376 s, 11.7 MB/s

Comparison, NVMe disks, fsn-node-07

root@fsn-node-07:~# fio --name=stressant --group_reporting <(sed /^filename=/d /usr/share/doc/fio/examples/basic-verify.fio; echo size=100m) --runtime=1m  --filename=test --size=100m
stressant: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.12
Starting 2 processes
write-and-verify: Laying out IO file (1 file / 100MiB)

stressant: (groupid=0, jobs=1): err= 0: pid=31809: Wed Mar 24 17:49:48 2021
  read: IOPS=138k, BW=541MiB/s (567MB/s)(100MiB/185msec)
    clat (nsec): min=522, max=2651.8k, avg=6848.59, stdev=57695.32
     lat (nsec): min=539, max=2651.8k, avg=6871.47, stdev=57695.33
    clat percentiles (nsec):
     |  1.00th=[    540],  5.00th=[    556], 10.00th=[    572],
     | 20.00th=[    588], 30.00th=[    596], 40.00th=[    612],
     | 50.00th=[    628], 60.00th=[    644], 70.00th=[    692],
     | 80.00th=[    764], 90.00th=[    828], 95.00th=[    996],
     | 99.00th=[ 292864], 99.50th=[ 456704], 99.90th=[ 708608],
     | 99.95th=[ 864256], 99.99th=[1531904]
  lat (nsec)   : 750=77.95%, 1000=17.12%
  lat (usec)   : 2=2.91%, 4=0.09%, 10=0.21%, 20=0.12%, 50=0.09%
  lat (usec)   : 100=0.04%, 250=0.32%, 500=0.77%, 750=0.28%, 1000=0.06%
  lat (msec)   : 2=0.03%, 4=0.01%
  cpu          : usr=10.33%, sys=10.33%, ctx=459, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
write-and-verify: (groupid=0, jobs=1): err= 0: pid=31810: Wed Mar 24 17:49:48 2021
  read: IOPS=145k, BW=565MiB/s (592MB/s)(100MiB/177msec)
    slat (usec): min=2, max=153, avg= 3.28, stdev= 1.95
    clat (usec): min=23, max=740, avg=106.23, stdev=44.45
     lat (usec): min=25, max=743, avg=109.56, stdev=44.52
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   70], 10.00th=[   73], 20.00th=[   77],
     | 30.00th=[   82], 40.00th=[   87], 50.00th=[   93], 60.00th=[  101],
     | 70.00th=[  115], 80.00th=[  130], 90.00th=[  155], 95.00th=[  182],
     | 99.00th=[  269], 99.50th=[  343], 99.90th=[  486], 99.95th=[  537],
     | 99.99th=[  717]
  write: IOPS=115k, BW=448MiB/s (470MB/s)(100MiB/223msec); 0 zone resets
    slat (usec): min=4, max=160, avg= 6.10, stdev= 2.02
    clat (usec): min=31, max=15535, avg=132.13, stdev=232.65
     lat (usec): min=37, max=15546, avg=138.27, stdev=232.65
    clat percentiles (usec):
     |  1.00th=[   76],  5.00th=[   90], 10.00th=[   97], 20.00th=[  102],
     | 30.00th=[  106], 40.00th=[  113], 50.00th=[  118], 60.00th=[  123],
     | 70.00th=[  128], 80.00th=[  137], 90.00th=[  161], 95.00th=[  184],
     | 99.00th=[  243], 99.50th=[  302], 99.90th=[ 4293], 99.95th=[ 6915],
     | 99.99th=[ 6980]
  lat (usec)   : 50=0.28%, 100=36.99%, 250=61.67%, 500=0.89%, 750=0.04%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.03%, 10=0.06%, 20=0.01%
  cpu          : usr=22.11%, sys=57.79%, ctx=8799, majf=0, minf=623
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,25600,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1081MiB/s (1134MB/s), 541MiB/s-565MiB/s (567MB/s-592MB/s), io=200MiB (210MB), run=177-185msec
  WRITE: bw=448MiB/s (470MB/s), 448MiB/s-448MiB/s (470MB/s-470MB/s), io=100MiB (105MB), run=223-223msec

Disk stats (read/write):
    dm-1: ios=25869/25600, merge=0/0, ticks=2856/2388, in_queue=5248, util=80.32%, aggrios=26004/25712, aggrmerge=0/0, aggrticks=2852/2380, aggrin_queue=5228, aggrutil=69.81%
    dm-0: ios=26004/25712, merge=0/0, ticks=2852/2380, in_queue=5228, util=69.81%, aggrios=26005/25712, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    md1: ios=26005/25712, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=13002/25628, aggrmerge=0/85, aggrticks=1328/1147, aggrin_queue=2752, aggrutil=89.35%
  nvme0n1: ios=12671/25628, merge=0/85, ticks=1176/496, in_queue=1896, util=89.35%
  nvme1n1: ios=13333/25628, merge=1/85, ticks=1481/1798, in_queue=3608, util=89.35%

Comparison, SATA disks, fsn-node-02

root@fsn-node-02:/mnt# fio --name=stressant --group_reporting <(sed /^filename=/d /usr/share/doc/fio/examples/basic-verify.fio; echo size=100m) --runtime=1m  --filename=test --size=100m
stressant: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.12
Starting 2 processes
write-and-verify: Laying out IO file (1 file / 100MiB)
Jobs: 1 (f=0): [_(1),f(1)][100.0%][r=348KiB/s][r=87 IOPS][eta 00m:00s]                   
stressant: (groupid=0, jobs=1): err= 0: pid=9635: Wed Mar 24 17:50:32 2021
  read: IOPS=5550, BW=21.7MiB/s (22.7MB/s)(100MiB/4612msec)
    clat (nsec): min=500, max=273948k, avg=179390.97, stdev=4673600.03
     lat (nsec): min=515, max=273948k, avg=179471.70, stdev=4673600.38
    clat percentiles (nsec):
     |  1.00th=[      524],  5.00th=[      580], 10.00th=[      692],
     | 20.00th=[     1240], 30.00th=[     1496], 40.00th=[     2320],
     | 50.00th=[     2352], 60.00th=[     2896], 70.00th=[     2960],
     | 80.00th=[     3024], 90.00th=[     3472], 95.00th=[     3824],
     | 99.00th=[   806912], 99.50th=[   978944], 99.90th=[ 60030976],
     | 99.95th=[110624768], 99.99th=[244318208]
   bw (  KiB/s): min= 2048, max=82944, per=100.00%, avg=22296.89, stdev=26433.89, samples=9
   iops        : min=  512, max=20736, avg=5574.22, stdev=6608.47, samples=9
  lat (nsec)   : 750=11.57%, 1000=3.11%
  lat (usec)   : 2=23.35%, 4=58.17%, 10=1.90%, 20=0.16%, 50=0.15%
  lat (usec)   : 100=0.03%, 250=0.03%, 500=0.04%, 750=0.32%, 1000=0.69%
  lat (msec)   : 2=0.17%, 4=0.04%, 10=0.11%, 20=0.02%, 50=0.02%
  lat (msec)   : 100=0.05%, 250=0.07%, 500=0.01%
  cpu          : usr=1.41%, sys=1.52%, ctx=397, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
write-and-verify: (groupid=0, jobs=1): err= 0: pid=9636: Wed Mar 24 17:50:32 2021
  read: IOPS=363, BW=1455KiB/s (1490kB/s)(100MiB/70368msec)
    slat (usec): min=2, max=4401, avg=46.08, stdev=38.17
    clat (usec): min=101, max=1002.5k, avg=43920.61, stdev=49423.03
     lat (usec): min=106, max=1002.5k, avg=43967.49, stdev=49419.62
    clat percentiles (usec):
     |  1.00th=[   188],  5.00th=[   273], 10.00th=[   383], 20.00th=[  3752],
     | 30.00th=[  8586], 40.00th=[ 16319], 50.00th=[ 28967], 60.00th=[ 45351],
     | 70.00th=[ 62129], 80.00th=[ 80217], 90.00th=[106431], 95.00th=[129500],
     | 99.00th=[181404], 99.50th=[200279], 99.90th=[308282], 99.95th=[884999],
     | 99.99th=[943719]
  write: IOPS=199, BW=796KiB/s (815kB/s)(100MiB/128642msec); 0 zone resets
    slat (usec): min=4, max=136984, avg=101.20, stdev=2123.50
    clat (usec): min=561, max=1314.6k, avg=80287.04, stdev=105685.87
     lat (usec): min=574, max=1314.7k, avg=80388.86, stdev=105724.12
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    5], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[   12], 40.00th=[   45], 50.00th=[   51], 60.00th=[   68],
     | 70.00th=[  111], 80.00th=[  136], 90.00th=[  167], 95.00th=[  207],
     | 99.00th=[  460], 99.50th=[  600], 99.90th=[ 1250], 99.95th=[ 1318],
     | 99.99th=[ 1318]
   bw (  KiB/s): min=  104, max= 1576, per=100.00%, avg=822.39, stdev=297.05, samples=249
   iops        : min=   26, max=  394, avg=205.57, stdev=74.29, samples=249
  lat (usec)   : 250=1.95%, 500=4.63%, 750=0.69%, 1000=0.40%
  lat (msec)   : 2=1.15%, 4=3.47%, 10=18.34%, 20=7.79%, 50=17.56%
  lat (msec)   : 100=20.45%, 250=21.82%, 500=1.27%, 750=0.23%, 1000=0.10%
  cpu          : usr=0.60%, sys=1.79%, ctx=46722, majf=0, minf=627
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25600,25600,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2910KiB/s (2980kB/s), 1455KiB/s-21.7MiB/s (1490kB/s-22.7MB/s), io=200MiB (210MB), run=4612-70368msec
  WRITE: bw=796KiB/s (815kB/s), 796KiB/s-796KiB/s (815kB/s-815kB/s), io=100MiB (105MB), run=128642-128642msec

Disk stats (read/write):
    dm-48: ios=26004/27330, merge=0/0, ticks=1132284/2233896, in_queue=3366684, util=100.00%, aggrios=28026/41435, aggrmerge=0/0, aggrticks=1292636/3986932, aggrin_queue=5288484, aggrutil=100.00%
    dm-56: ios=28026/41435, merge=0/0, ticks=1292636/3986932, in_queue=5288484, util=100.00%, aggrios=28027/41436, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    md125: ios=28027/41436, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=13768/36599, aggrmerge=220/4980, aggrticks=622303/1259843, aggrin_queue=859540, aggrutil=61.10%
  sdb: ios=13271/36574, merge=193/5009, ticks=703823/1612782, in_queue=1077576, util=61.10%
  sda: ios=14265/36624, merge=248/4951, ticks=540784/906905, in_queue=641504, util=51.08%

Keep in mind the machine was not idle at the time of testing, quite the contrary (under about 4-5 load).

Glossary

• SAN: storage area network
• iSCSI: SCSI over "internet", allows block devices to be mounted over TCP/IP
• iSCSI initiator: an iSCSI "client"
• iSCSI target: an iSCSI "server", typically a SAN with redundant disks, exposing block devices to iSCSI initiators
• multipath: "a technique whereby there is more than one physical path between the server and the storage", typically this means multiple network interfaces on the initiator, target, running over distinct network switches (or at least VLANs)

Network topoloy

The network at Cymru is split into different VLANs:

• "public": VLAN 82 - 38.229.82.0/24, directly on the internet (behind the cymru router), eth0 on all nodes.

• "storage": VLAN 801 - 172.30.130.0/24. access to the iSCSI servers and also used by Ganeti and DRBD for inter-node communications. not directly accessible by the router, eth1 on all nodes.

• "management": VLAN 802 - 172.30.140.0/24, access to the iDRACs and IPMI management interfaces, not directly accessible by the router, but accessible from eth2 on all the nodes but normally not configured.

This is summarized by this diagram:

Note that the bastion host configured above is not currently configured: it can be configured by hand on one of the chi-node-X machines since they have access to VLAN 802, but this should eventually be fixed.

Discussion

Disabling the builtin RAID controller

We tried to disable the built-in RAID controller in order to use software RAID. Hardware RAID is always a headache as it requires proprietary drivers that are hard or impossible to find. By using software RAID, we have the same uniform interface on all servers.

To disable hardware RAID on Cymru hardware (PowerEdge R610 or R620 machines), you have access to the BIOS. This can be done through a Virtual console or serial port, if Serial redirection is first enabled in the BIOS (which requires a virtual console). Then:

1. reboot the server to get into the BIOS dialogs
2. let the BIOS do its thing and wait for the controller to start initializing
3. hit control-r when the controller dialog shows up

This will bring you in the RAID controller interface, which should have a title like:

      PERC H710P Mini BIOS Configuration Utility 4.03-0002

WARNING: the following steps will destroy all the data on the disks!!

In the VD Mgmt tab:

1. press F2 ("operations")
2. select "Clear Config" and confirm

Another way to do this is to:

1. select the "virtual disk"
2. press F2 ("operations")
3. choose "Delete VD" and confirm

press <Control-R>: to enter configuration utility

For good measure, it seems you can also disable the controller completely in the Ctrl Mgmt tab (accessed by pressing control-n twice), by unticking the Enable controller BIOS and Enable BIOS Stop on Error.

To exit the controller, hit Esc ("Escape"). Then you need to send control-alt-delete somehow. This can be done in the Macros menu in the virtual console, or, in the serial console, exiting with control-backslash and then issuing the command:

racadm serveraction powercycle

Unfortunately, when the controller is disabled, the disks just do not show up at all. We haven't been able to bypass the controller so those instructions are left only as future reference.

For systems equipped with the PERC H740P Mini controller (eg. chi-node-14), it's possible to switch it to "Enhanced HBA" mode using the iDrac interface (requires reboot). This mode allows the individual disks to be seen by the operating system, allowing the possibility to use software RAID.

See also raid.

Design

Storage

The iSCSI cluster provides roughly 172TiB of storage in the management network, at least in theory. Debian used this in the past with ganeti, but that involves creating, resizing, and destroying volumes by hand before/after creating, and destroying VMs. While that is not ideal, it is the first step in getting this infrastructure used.

We also use the "normal" DRBD setup with the local SAS disks available on the servers. This is used for the primary disks for Ganeti instances, but provides limited disk space (~350GiB per node) so it should be used sparingly.

Another alternative that was considered is to use CLVM ("The Clustered Logical Volume Manager") which makes it possible to run LVM on top of shared SAN devices like this. This approach was discarded for a few reasons:

1. it's unclear whether CLVM is correctly packaged in Debian
2. we are not familiar with this approach at all, which requires us to get familiar both with iSCSI and CLVM (we already will need to learn the former), and this might be used only for this PoP
3. it's unclear whether CLVM is production ready

We also investigated whether Ceph could use iSCSI backends. It does not: it can provide an iSCSI "target" (a storage server) but it can't be an iSCSI "initiator". We did consider using Ceph instead of DRBD for the SAS disks, but decided against it to save research time in the cluster setup.

multipath configuration

We have tested this multipath.conf from Gabriel Beaver and proxmox on chi-node-01:

# from https://pve.proxmox.com/wiki/ISCSI_Multipath#Dell
defaults {
  polling_interval        2
  path_selector           "round-robin 0"
  path_grouping_policy    multibus
  rr_min_io               100
  failback                immediate
  no_path_retry           queue
}

devices {
  # from https://pve.proxmox.com/wiki/ISCSI_Multipath#Dell
  device {
    vendor                  "DELL"
    product                 "MD32xxi"
    path_grouping_policy    group_by_prio
    prio                    rdac
    path_checker            rdac
    path_selector           "round-robin 0"
    hardware_handler        "1 rdac"
    failback                immediate
    features                "2 pg_init_retries 50"
    no_path_retry           30
    rr_min_io               100
  }
  device {
    vendor                          "SCST_FIO|SCST_BIO"
    product                         "*"
    path_selector                   "round-robin 0"
    path_grouping_policy            multibus
    rr_min_io                       100
  }
}

# from https://gabrielbeaver.me/2013/03/centos-6-x-and-dell-md3000i-setup-guide/
# Gabriel Beaver 03/27/2013
blacklist {
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
  devnode "^sda"
  devnode "^sda[0-9]"
  device {
    vendor DELL
    product "PERC|Universal|Virtual"
  }
}
# END GB

It seems that configuration is actually optional: multipath will still work fine without it, so it's not deployed consistently across nodes at the moment.

Ganeti iSCSI integration

See Ganeti storage reference and Ganeti iSCSI integration.

Private network access considerations

We considered a few ideas to provide access to the management network:

• OpenVPN
• IPsec
• SSH SOCKS proxying
• sshuttle

We somehow expected cymru to provide us with a jump host for this purpose, like they did with peninsulare for the moly server, but this turned out to not happen.

We never really considered OpenVPN seriously: we already use ipsec elsewhere and it seemed like a bad idea to introduce another VPN technology when we were already using another one. This meant more struggles with IPsec, but that, in turn, meant the staff gets more familiar with it. In other words, if IPsec doesn't work, let's get rid of it everywhere, and not make a special case here.

SSH SOCKS proxying (-D) was the idea of using one of the jump hosts as an SSH proxy. It kind of worked: web browsers were able to access the iDRAC web interfaces by manually configured a SOCKS5 proxy in the settings (at least in Firefox). But that, in turn, did not necessarily work across virtual machine boundaries (so that the Java stuff worked), let alone insite the Java JVM itself. So this approach was never seriously considered either, although it worked for the web UI.

sshuttle could have worked as well: it does provide routing somewhat elegantly. Apart from the concerns established in the OpenVPN option above (yet another VPN solution), it added the problem that it needs to run as root on the client side. That makes it difficult to access regular ssh-agent credentials (e.g. using a Yubikey). There are ways to use ssh -A root@localhost to forward agent credentials, but that seemed to hacky.

VLAN allocations

The VLAN allocations described in the network topology above were suggested by Cymru and slightly modified to fit our use case. See issue 40097 for the gory details of that discussion.

There is flexibility upstream on VLAN allocation and possibly bundling network interfaces together. All hosts have 8 interfaces so there's lots of potential there.

It would be possible, for example, to segregate DRBD, iSCSI and Ganeti traffic in three different VLANs. For now, we've adopted the path of simplicity and all those live in the same private VLAN.

Go to the Heztner console and clikety on the web interface to get a new instance. Credentials are in tor-passwords.git in hosts-extra-info under hetzner.

TODO: consider using the hcloud command instead.

Pick the following settings:

1. Location: depends on the project, a monitoring server might be better in a different location than the other VMs
2. Image: Debian 9
3. Type: depends on the project
4. Volume: only if extra space is required
5. Additional features: nothing (no user data or backups)
6. SSH key: enable all configured keys
7. Name: FQDN picked from the doc/naming-scheme
8. Create the server

Then, since we actually want our own Debian install, and since we want the root filesystem to be encrypted, continue with:

1. Continue on Hetzner's web interface, select the server.
2. Reboot into the rescue system ("Rescue, Enable rescue & Power cycle", pick linux64 and your SSH key). this will give you a root password
3. open the console (the icon is near the top right) and login with the root password
4. get the ssh-keygen -l -f /etc/ssh/ssh_host_*.pub output. NOTE: the Hetzner consoles use a different keyboard mapping than "US". Hint: - is on the / key, / is on shift-7 and * is on shift-]
5. login to the new host: ssh root@$IPADDRESS, check the fingerprint matches above
6. start a screen session
7. clone fabric-tasks to the new host: git clone https://gitlab.torproject.org/tpo/tpa/fabric-tasks.git
8. run ./fabric-tasks/installer/tor-install-hetzner (the ipv6 address prefix you find on the web interface. Make it end in ::1) TODO: merge script with the new-machine-hetzner-robot procedure. WARNING: this procedure has been known to leave ping non-functional for regular users, see ticket 31781
9. once done, note down all the info and reboot the VM: reboot
10. ssh -o FingerprintHash=sha1 root@<ipaddr> to unlock the host, (to compare ssh's base64 output to dropbear's b16, you can use perl -MMIME::Base64 -e '$h = unpack("H*", decode_base64(<>)); $h =~ s/(..)(?=.)/\1:/g; print $h, "\n"' to convert base64 to base16.
11. ssh root@<ipaddr> to access it once booted

Then

1. Set the reverse DNS using hetzner's website. It's in the networking section for each virtual server. Set both ipv4 and ipv6 reverse entries.
2. Document the LUKS passphrase and root password in tor-passwords,
3. follow the rest of new-machine.

See new-machine-mandos for setting up the mandos client on this host.

• How to install a new bare metal server at Hetzner
  • Order
  • Automated install procedure
  • Manual install procedure

How to install a new bare metal server at Hetzner

This is for setting up physical metal at Hetzner.

Order

1. get approval for the server, picking the specs from the main website

2. head to the order page and pick the right server. pay close attention to the location, you might want to put it alongside other TPO servers (or not!) depending on redundancy or traffic requirements. Click Add to shopping cart, leaving all other fields as default.

3. in the Server login details page, you should leave Type set to Public key. If you do not recognize your public SSH key in there, head to the server list and click on key management to add your public keys

4. when you're certain of everything, click Checkout in the cart, review the order again and click Order in obligation.

A confirmation email will be sent by Hetzner at the TPA alias when the order is filed. Then you wait for the order to complete before being able to proceed with the install.

Ordering physical servers from Hetzner can be very fast: we've seen 2 minutes turn around times, but it can also take a lot more time in some situations, see their status page for estimates.

Automated install procedure

At this point you should have received an email from Hetzner with a subject like:

Subject: Your ordered SX62 server

It should contain the SSH fingerprint, and IP address of the new host which we'll use below. The machine can be bootstrapped with a basic Debian installer with the Fabric code in the fabric-tasks git repository. Here's an example of a commandline:

./install -H root@88.99.194.57 \
          --fingerprint 0d:4a:c0:85:c4:e1:fe:03:15:e0:99:fe:7d:cc:34:f7 \
          hetzner-robot \
          --fqdn=HOSTNAME.torproject.org \
          --fai-disk-config=installer/disk-config/gnt-fsn-NVMe \
          --package-list=installer/packages \
          --post-scripts-dir=installer/post-scripts/ \
          --mirror=https://mirror.hetzner.de/debian/packages/

Taking that apart:

• -H root@88.99.194.57: the IP address provided by Hetzner in the confirmation email
• --fingerprint: the ed25519 MD5 fingerprint from the same email
• hetzner-robot: the install job type (only robot supported for now)
• --fqdn=HOSTNAME.torproject.org: the Fully Qualified Domain Name to set on the machine, it is used in a few places, but the hostname is correctly set to the HOSTNAME part only
• --fai-disk-config=installer/disk-config/gnt-fsn-NVMe: the disk configuration, in fai-setup-storage(8) format
• --package-list=installer/packages: the base packages to install
• --post-scripts-dir=installer/post-scripts/: post-install scripts, magic glue that does everything

The last two are passed to grml-debootstrap and should rarely be changed (although they could be converted in to Fabric tasks themselves).

Note that the script will show you lines like:

STEP 1: SSH into server with fingerprint ...

Those correspond to the manual install procedure, below. If the procedure stops before the last step (currently STEP 12), there was a problem in the procedure, but the remaining steps can still be performed by hand.

If a problem occurs in the install, you can login to the rescue shell with:

ssh -o FingerprintHash=md5 -o UserKnownHostsFile=~/.ssh/authorized_keys.hetzner-rescue root@88.99.194.57

... and check the fingerprint against the email provided by Hetzner.

Do a reboot before continuing with the install:

reboot

You will need to enter the LUKS passphrase generated by the installer through SSH and the dropbear-initramfs setup. The LUKS password and the SSH keys should be available in the installer backlog. If that fails, then you can either try to recover from the out of band management (KVM, or serial if available), or scrutinize the logs for errors that could hint at a problem, and try a reinstall.

See new-machine for post-install configuration steps, then follow new-machine-mandos for setting up the mandos client on this host.

Manual install procedure

WARNING: this procedure is kept for historical reference, and in case the automatic procedure above fails for some reason. It should not be used.

At this point you should have received an email from Hetzner with a subject like:

Subject: Your ordered SX62 server

It should contain the SSH fingerprint, and IP address of the new host which we'll use below.

1. login to the server using the IP address and host key hash provided above:

   ssh -o FingerprintHash=md5 -o UserKnownHostsFile=~/.ssh/authorized_keys.hetzner-rescue root@159.69.63.226
   

   Note: the FingerprintHash parameter above is to make sure we match the hashing algorithm used by Hetzner in their email, which is, at the time of writing, MD5 (!). Newer versions of SSH will also encode the hash as base64 instead of hexadecimal, so you might want to decode the base64 into the latter using this: The UserKnownHostsFile is to make sure we don't store the (temporary) SSH host key.

   perl -MMIME::Base64 -e '$h = unpack("H*", decode_base64(<>)); $h =~ s/(..)(?=.)/\1:/g; print $h, "\n"'
   

2. Set a hostname (short version, not the FQDN):

   echo -n 'New hostname: ' && read hn && hostname "$hn" && exec bash
   

   TODO: merge this with wrapper script below.

3. Partition disks. This might vary wildly between hosts, but in general, we want:

   • GPT partitioning, with space for a 8MB grub partition and cleartext /boot
   • software RAID (RAID-1 for two drives, RAID-5 for 3, RAID-10 for 4)
   • crypto (LUKS)
   • LVM, with separate volume groups for different medium (SSD vs HDD)

   We are experimenting with FAI's setup-storage to partition disks instead of rolling our own scripts. You first need to checkout the installer's configuration:

       apt install git
       git clone https://gitlab.torproject.org/tpo/tpa/fabric-tasks.git
       cd fabric-tasks/installer
       git show-ref master
   

   Check that the above hashes match a trusted copy.

   Use the following to setup a Ganeti node, for example:

       apt install fai-setup-storage
   
       setup-storage -f "disk-config/gnt-fsn-NVMe" -X
   

   TODO: merge this with wrapper script below.

   TODO: convert the other existing tor-install-format-disks-4HDDs script into a setup-storage configuration.

   And finally mount the filesystems:

       . /tmp/fai/disk_var.sh &&
       mkdir /target &&
       mount "$ROOT_PARTITION" /target &&
       mkdir /target/boot &&
       mount "$BOOT_DEVICE" /target/boot
   

   TODO: test if we can skip that test by passing $ROOT_PARTITION as a --target to grml-debootstrap. Probably not.

   TODO: in any case, this could be all wrapper up in a single wrapper shell script in fabric-tasks instead of this long copy-paste. Possibly merge with tor-install-hetzner from new-machine-hetzner-cloud.

4. Install the system. This can be done with grml-debootstrap which will also configure grub, a root password and so on. This should get you started, assuming the formatted root disk is mounted on /target and that the boot device is defined by $BOOT_DEVICE (populated above by FAI). Note that BOOT_DISK is the disk as opposed to the partition which is $BOOT_DEVICE.

   BOOT_DISK=/dev/nvme0n1 &&
   mkdir -p /target/run && mount -t tmpfs tgt-run /target/run &&
   mkdir /target/run/udev && mount -o bind /run/udev /target/run/udev &&
   apt-get install -y grml-debootstrap && \
   grml-debootstrap \
       --grub "$BOOT_DISK" \
       --target /target \
       --hostname `hostname` \
       --release trixie \
       --mirror https://mirror.hetzner.de/debian/packages/ \
       --packages /root/fabric-tasks/installer/packages \
       --post-scripts /root/fabric-tasks/installer/post-scripts/ \
       --nopassword \
       --remove-configs \
       --defaultinterfaces &&
   umount /target/run/udev /target/run
   

5. setup dropbear-initramfs to unlock the filesystem on boot. this should already have been done by the 50-tor-install-luks-setup hook deployed in the grml-debootstrap stage.

   TODO: in an install following the above procedure, a keyfile was left unprotected in /etc. Make sure we have strong mechanisms to avoid that ever happening again. For example:

   chmod 0 /etc/luks/
   

   TODO: the keyfiles deployed there can be used to bootstrap mandos. Document how to do this better.

6. Review the crypto configuration:

   cat /target/etc/crypttab
   

   If the backing device is NOT an SSD, remove the ,discard option.

   TODO: remove this step, it is probably unnecessary.

7. Review the network configuration, since it will end up in the installed instance:

   cat /target/etc/network/interfaces
   

   An example safe configuration is:

   auto lo
   iface lo inet loopback
   
   allow-hotplug eth0
   iface eth0 inet dhcp
   

   The latter two lines usually need to be added as they are missing from Hetzner rescue shells:

   cat >> /etc/network/interfaces <<EOF
   
   allow-hotplug eth0
   iface eth0 inet dhcp
   EOF
   

   TODO: fix this in a post-install debootstrap hook, or in grml-debootstrap already, see also upstream issue 105 and issue 136.

   Add the hostname, IP address and domain to /etc/hosts and /etc/resolv.conf:

   grep torproject.org /etc/resolv.conf || ( echo 'domain torproject.org'; echo 'nameserver 8.8.8.8' ) >> /etc/resolv.conf
   if ! hostname -f 2>/dev/null || [ "$(hostname)" = "$(hostname -f)" ]; then
       IPADDRESS=$(ip -br -color=never route get to 8.8.8.8 | head -1 | grep -v linkdown | sed 's/.*  *src  *\([^ ]*\)  *.*/\1/')
       echo "$IPADDRESS $(hostname).torproject.org $(hostname)" >> /etc/hosts
   fi
   

   TODO: add the above as a post-hook. possibly merge with tor-puppet/modules/ganeti/files/instance-debootstrap/hooks/gnt-debian-interfaces

   TODO: add IPv6 address configuration. look at how tor-install-generate-ldap guesses as well.

8. If any of those latter things changed, you need to regenerate the initramfs:

   chroot /target update-initramfs -u
   chroot /target update-grub
   

   TODO: remove this step, if the above extra steps are removed.

9. umount things:

   umount /target/run/udev || true &&
   for fs in dev proc run sys  ; do
       umount /target/$fs || true
   done &&
   umount /target/boot &&
   cd / && umount /target
   

   TODO: merge this with wrapper script.

10. close things

    vgchange -a n cryptsetup luksClose crypt_dev_md1 cryptsetup luksClose crypt_dev_md2 mdadm --stop /dev/md*

    TODO: merge this with wrapper script.

11. Document the LUKS passphrase and root password in tor-passwords

12. Cross fingers and reboot:

    reboot

See new-machine for post-install configuration steps, then follow new-machine-mandos for setting up the mandos client on this host.

Mandos is a means to give LUKS keys to machines that want to boot but have an encrypted rootfs.

Here's how you add a new client to our setup:

1. add a new key to the LUKS partition and prepare mandos snippet:

    lsblk --fs &&
    read -p 'encrypted (root/lvm/..) device (e.g. /dev/sda2 or /dev/mb/pv_nvme): ' DEVICE &&
    apt install -y haveged mandos-client &&
    (grep 116.203.128.207 /etc/mandos/plugin-runner.conf || echo '--options-for=mandos-client:--connect=116.203.128.207:16283' | tee -a /etc/mandos/plugin-runner.conf) &&
    umask 077 &&
    t=`tempfile` &&
    dd if=/dev/random bs=1 count=128 of="$t" &&
    cryptsetup luksAddKey $DEVICE "$t" &&
    mandos-keygen --passfile "$t"
   

2. add the roles::fde class to new host in Puppet and run puppet there:

   puppet agent -t
   

   If the class was already applied on a previous Puppet run, ensure the initramfs image is updated at this point:

   update-initramfs -u
   

3. on the mandos server, add the output of mandos-keygen from above to /etc/mandos/clients.conf and restart the service:

   service mandos restart
   

4. on the mandos server, update the firewall after you added the host to ldap:

   puppet agent -t
   

5. on the mandos server, enable the node:

   mandos-ctl --enable $FQDN
   

6. reboot the new host to test unlocking

TODO: Mandos setups should be automatic, see issue 40096.

Billing and ordering

This is kind of hell.

You need to register on their site and pay with a credit card. But you can't be from the US to order in Canada and vice-versa, which makes things pretty complicated if you want to have stuff in one country or the other.

Also, the US side of things can trip over itself and flag you as a compromised account, at which point they will ask you for a driver's license and so on. A workaround is to go on the other site.

Once you have ordered the server, they will send you a confirmation email, then another email when the order is fulfilled, with the username and password to login to the server. Next step is to setup the server.

Preparation

We assume we are creating a new server named test-01.torproject.org. You should have, at this point, received a email with the username and password. Ideally, you'd login through the web interface's console, which they call the "KVM".

1. immediately change the password so the cleartext password sent by email cannot be reused, document in the password manager

2. change the hostname on the server and in the web interface to avoid confusion:

   hostname test-01
   exec bash
   

   In the OVH dashboard, you need to:

   1. navigate to the "product and services" (or "bare metal cloud" then "Virtual private servers")
   2. click on the server name
   3. click on the "..." menu next to the server name
   4. choose "change name"

3. setting up reverse DNS doesn't currently work ("An error has occurred updating the reverse path."), pretend this is not a problem

4. add your SSH key to the root account

5. then follow the normal new-machine procedure, with the understanding reverse DNS is broken and that we do not have full disk encryption

In particular, you will have to:

1. reset the /etc/hosts file (with fabric works)

2. hack at /etc/resolv.conf to change the search domain

3. delete the debian account

See issue tpo/tpa/team#40904 for an example run.

• How to
  • Burn-in
  • Installation
  • Post-install configuration
    • Pre-requisites
    • Main procedure
  • Rescuing a failed install
• Reference
  • Design
  • Issues
• Discussion
  • Overview
  • Goals
    • Must have
    • Nice to have
    • Non-Goals
  • Approvals required
  • Proposed Solution
  • Cost
  • Alternatives considered

How to

Burn-in

Before we even install the machine, we should do some sort of stress-testing or burn-in so that we don't go through the lengthy install process and put into production fautly hardware.

This implies testing the various components to see if they support a moderate to high load. A tool like stressant can be used for that purpose, but a full procedure still needs to be established.

Example stressant run:

apt install stressant
stressant --email torproject-admin@torproject.org --overwrite --writeSize 10% --diskRuntime 120m --logfile $(hostname)-sda.log --diskDevice /dev/sda

This will wipe parts of /dev/sda, so be careful. If instead you want to test inside a directory, use this:

stressant --email torproject-admin@torproject.org  --diskRuntime 120m --logfile fsn-node-05-home-test.log --directory /home/test --writeSize 1024M

Stressant is still in development and currently has serious limitations (e.g. it tests one disk at a time and clunky UI) but should be a good way to get started.

Installation

This document assumes the machine is already installed with a Debian operating system. We preferably install stable or, when close to the release, testing. Here are site-specific installs:

• Hetnzer Cloud
• Hetzner Robot
• Ganeti clusters:
  • new virtual machine: new instance procedure
  • new nodes (which host virtual machines) new node procedure, normally done as a post-install configuration
• Sunet, Linaro and OSUOSL: service/openstack
• Cymru
• OVH cloud
• Quintex

The following sites are not documented yet:

• eclips.is: our account is marked as "suspended" but oddly enough we have 200 credits which would give us (roughly) 32GB of RAM and 8 vCPUs (yearly? monthly? how knows). it is (separately) used by the metrics team for onionperf, that said

The following sites are deprecated:

• KVM/libvirt (really at Hetzner) - replaced by Ganeti
• scaleway - see ticket 32920

Post-install configuration

The post-install configuration mostly takes care of bootstrapping Puppet and everything else follows from there. There are, however, still some unrelated manual steps but those should eventually all be automated (see ticket #31239 for details of that work).

Pre-requisites

The procedure below assumes the following steps have already been taken by the installer:

0. Any new expenses for physical hosting, cloud services and such, need to be approved by accounting and ops before we can move with the creation.

1. a minimal Debian install with security updates has been booted (note that Puppet will deploy unattended-upgrades later, but it's still a good idea to do those updates as soon as possible)

2. partitions have been correctly setup, including some (>=512M) swap file (or swap partition) and a tmpfs in /tmp

   consider expanding the swap file if memory requirements are expected to be higher than usual on this system, such a large database servers, GitLab instances, etc. the steps below will recreate a 1GiB /swapfile volume instead of the default (512MiB):

   swapoff -a &&
   dd if=/dev/zero of=/swapfile bs=1M count=1k status=progress &&
   chmod 0600 /swapfile &&
   mkswap /swapfile &&
   swapon -a
   

3. a hostname has been set, picked from the doc/naming-scheme and the short hostname (e.g. test) resolves to a fully qualified domain name (e.g. test.torproject.org) in the torproject.org domain (i.e. /etc/hosts is correctly configured). this can be fixed with:

   fab -H root@204.8.99.103 host.rewrite-hosts dal-node-03.torproject.org 204.8.99.103
   

   WARNING: The short hostname (e.g. foo in foo.example.com) MUST NOT be longer than 21 characters, as that will crash the backup server because its label will be too long:

   Sep 24 17:14:45 bacula-director-01 bacula-dir[1467]: Config error: name torproject-static-gitlab-shim-source.torproject.org-full.${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}_${Hour:p/2/0/r}:${Minute:p/2/0/r} length 130 too long, max is 127
   

   TODO: this could be replaced by libnss-myhostname if we wish to simplify this, although that could negatively impact things that expect a real IP address from there (e.g. bacula).

4. a public IP address has been set and the host is available over SSH on that IP address. this can be fixed with:

   fab -H root@204.8.99.103 host.rewrite-interfaces 204.8.99.103 24 --ipv4-gateway=204.8.99.254 --ipv6-address=2620:7:6002::3eec:efff:fed5:6ae8 --ipv6-subnet=64 --ipv6-gateway=2620:7:6002::1
   

   If the IPv6 address is not known, it might be guessable from the MAC address. Try this:

   ipv6calc --action prefixmac2ipv6 --in prefix+mac --out ipv6 $SUBNET $MAC
   

   ... where $SUBNET is the (known) subnet from the upstream provider and $MAC is the MAC address as found in ip link show up.

   If the host doesn't have a public IP, reachability has to be sorted out somehow (eg. using a VPN) so Prometheus, our monitoring system, is able to scrape metrics from the host.

5. ensure reverse DNS is set for the machine. this can be done either in the upstream configuration dashboard (e.g. Hetzner) or in our zone files, in the dns/domains.git repository

   Tip: sipcalc -r will show you the PTR record for an IPv6 address. For example:

   $ sipcalc -r 2620:7:6002::466:39ff:fe3d:1e77
   -[ipv6 : 2604:8800:5000:82:baca:3aff:fe5d:8774] - 0
   
   [IPV6 DNS]
   Reverse DNS (ip6.arpa)	-
   4.7.7.8.d.5.e.f.f.f.a.3.a.c.a.b.2.8.0.0.0.0.0.5.0.0.8.8.4.0.6.2.ip6.arpa.
   
   -
   

   dig -x will also show you an SOA record pointing at the authoritative DNS server for the relevant zone, and will even show you the right record to create.

   For example, the IP addresses of chi-node-01 are 38.229.82.104 and 2604:8800:5000:82:baca:3aff:fe5d:8774, so the records to create are:

   $ dig -x 2604:8800:5000:82:baca:3aff:fe5d:8774 38.229.82.104
   [...]
   ;; QUESTION SECTION:
   ;4.7.7.8.d.5.e.f.f.f.a.3.a.c.a.b.2.8.0.0.0.0.0.5.0.0.8.8.4.0.6.2.ip6.arpa. IN PTR
   
   ;; AUTHORITY SECTION:
   2.8.0.0.0.0.0.5.0.0.8.8.4.0.6.2.ip6.arpa. 3552 IN SOA nevii.torproject.org. hostmaster.torproject.org. 2021020201 10800 3600 1814400 3601
   
   [...]
   
   ;; QUESTION SECTION:
   ;104.82.229.38.in-addr.arpa.	IN	PTR
   
   ;; AUTHORITY SECTION:
   82.229.38.in-addr.arpa.	2991	IN	SOA	ns1.cymru.com. noc.cymru.com. 2020110201 21600 3600 604800 7200
   
   [...]
   

   In this case, you should add this record to 82.229.38.in-addr.arpa.:

   104.82.229.38.in-addr.arpa.	IN	PTR chi-node-01.torproject.org.
   

   And this to 2.8.0.0.0.0.0.5.0.0.8.8.4.0.6.2.ip6.arpa.:

   4.7.7.8.d.5.e.f.f.f.a.3.a.c.a.b.2.8.0.0.0.0.0.5.0.0.8.8.4.0.6.2.ip6.arpa. IN PTR chi-node-01.torproject.org.
   

   Inversely, say you need to add an IP address for Hetzner (e.g. 88.198.8.180), they will already have a dummy PTR allocated:

   180.8.198.88.in-addr.arpa. 86400 IN	PTR	static.88-198-8-180.clients.your-server.de.
   

   The your-server.de domain is owned by Hetzner, so you should update that record in their control panel. Hint: try https://robot.hetzner.com/vswitch/index

6. DNS works on the machine (i.e. /etc/resolv.conf is configured to talk to a working resolver, but not necessarily ours, which Puppet will handle)

7. a strong root password has been set in the password manager, this implies resetting the password for Ganeti instance installs the installed password was written to disk (TODO: move to trocla? #33332)

8. grub-pc/install_devices debconf parameter is correctly set, to allow unattended upgrades of grub-pc to function. The command below can be used to bring up an interactive prompt in case it needs to be fixed:

    debconf-show grub-pc | grep -qoP "grub-pc/install_devices: \K.*" || dpkg-reconfigure grub-pc
   

   Warning: this doesn't actually work for EFI deployments.

Main procedure

All commands to be run as root unless otherwise noted.

IMPORTANT: make sure you follow the pre-requisites checklist above! Some installers cover all of those steps, but most do not.

Here's a checklist you can copy in an issue to make sure the following procedure is followed:

• BIOS and OOB setup
• burn-in and basic testing
• OS install and security sources check
• partitions check
• hostname check
• ip address allocation
• reverse DNS
• DNS resolution
• root password set
• grub check
• Nextcloud spreadsheet update
• hosters.yaml update (rare)
• fabric-tasks install
• puppet bootstrap
• dnswl
• /srv filesystem
• upgrade and reboot
• silence alerts
• restart bacula-sd

1. if the machine is not inside a ganeti cluster (which has its own inventory), allocate and document the machine in the Nextcloud spreadsheet, and the services page, if it's a new service

2. add the machine's IP address to hiera/common/hosters.yaml if this is a machine in a new network. This is rare; Puppet will crash its catalog with this error when that's the case:

   Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: \
   Evaluation Error: Error while evaluating a Function Call, \
   IP 195.201.139.202 not found among hosters in hiera data! (file: /etc/puppet/code/environments/production/modules/profile/manifests/facter/hoster.pp, line: 13, column: 5) on node hetzner-nbg1-01.torproject.org
   

   The error was split over multiple lines to outline the IP address more clearly. When this happens, add the IP address and netmask from the main interface to the hosters.yaml file.

   In this case, the sole IP address (195.201.139.202/32) was added to the file.

3. make sure you have the fabric-tasks git repository on your machine, and verify its content. the repos meta-repository should have the necessary trust anchors.

4. bootstrap puppet: on your machine, run the puppet.bootstrap-client task from the fabric-tasks git repository cloned above

5. add the host to LDAP

   The Puppet bootstrap script will show you a snippet to copy-paste to the LDAP server (db.torproject.org). This needs to be done in ldapvi, with:

     ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"
   

   If you lost the blob, it can be generated from the ldap.generate-entry task in Fabric.

   Make sure you review all fields, in particular location (l), physicalHost, description and purpose which do not have good defaults. See the service/ldap page for a description on those, but, generally:

   • physicalHost: where is this machine hosted, either parent host or cluster (e.g. gnt-fsn) or hoster (e.g. hetzner or hetzner-cloud)
   • description: free form description of the host
   • purpose: similar, but can [[link]] to a URL, also added to SSH known hosts, should be added to DNS as well
   • l: physical location,

   See the reboots section for information about the rebootPolicy field. See also the ldapvi manual for more information.

6. ... and if the machine is handling mail, add it to dnswl.org (password in tor-passwords, hosts-extra-info)

7. you will probably want to create a /srv filesystem to hold service files and data unless this is a very minimal system. Typically, installers may create the partition, but will not create the filesystem and configure it in /etc/fstab:

   mkfs -t ext4 -j /dev/sdb &&
   printf 'UUID=%s\t/srv\text4\tdefaults\t1\t2\n' $(blkid --match-tag UUID --output value /dev/sdb) >> /etc/fstab  &&
   mount /srv
   

8. once everything is done, reboot the new machine to make sure that still works. Before that you may want to run package upgrades in order to avoid getting a newer kernel the next day and needing to reboot again:

   apt update && apt upgrade
   reboot
   

9. if the machine was not installed from the Fabric installer (the install.hetzner-robot task), schedule a silence for backup alerts with:

   fab silence.create \
     --comment="machine waiting for first backup" \
     --matchers job=bacula \
     --matchers alias=test-01.torproject.org \
     --ends-at "in 2 days"
   

   TODO: integrate this in other installers.

10. consider running systemctl restart bacula-sd on the backup storage host so that it'll know about the new machine's backup volume

    • On backup-storage-01.torproject.org if the new machine is in Falkenstein
    • On bungei.torproject.org if the new machine is anywhere else then Falkenstein (so for example in Dallas)

At this point, the machine has a basic TPA setup. You will probably need to assign it a "role" in Puppet to get it to do anything.

Rescuing a failed install

If the procedure above fails but in a way that didn't prevent it from completing the setup on disk -- for example if the install goes through to completion but after a reboot you're neither able to login via the BMC console nor able to reach the host via network -- here are some tricks that can help in making the install work correctly:

• on the grub menu, edit the boot entry and remove the kernel parameter quiet to see more meaningful information on screen during boot time.
• in the boot output (without quiet) take a look at what the network interface names are set to and which ones are reachable or not.
• try exchanging the VLANs of the network interfaces to align the interface configured by the installer to where the public network is reachable
• if there's no meaningful output on the BMC console after just a handful of kernel messages, try to remove all console= kernel parameters. this sometimes brings back the output and prompt for crypto from dropbear onto the console screen.
• if you boot into grml via PXE to modify files on disk (see below) and if you want to update the initramfs, make sure that the device name used for the luks device (the name supplied as last argument to cryptsetup open) corresponds to what's set in the file /etc/crypttab inside the installed system.
  • When the device name differs, update-initramfs might fail to really update and only issue a warning about the device name.
  • The device name usually looks like the example commands below, but if you're unsure what name to use, you can unlock crypto, check the contents of /etc/crypttab and then close things up again and reopen with the device name that's present in there.
• if you're unable to figure out which interface name is being used for the public network but if you know which one it is from grml, you can try removing the net.ifnames=0 kernel parameter and also changing the interface name in the ip= kernel parameter, for example by modifying the entry in the grub menu during boot.
  • That might bring dropbear online. Note that you may also need to change the network configuration on disk for the installed system (see below) so that the host stays online after the crypt device was unlocked.

To change things on the installed system, mainly for fixing initramfs, grub config and network configuration, first PXE-boot into grml. Then open and mount the disks:

mdadm --assemble --scan
cryptsetup open /dev/md1 crypt_dev_md1
vgchange -a y
mount /dev/mapper/vg_system-root /mnt
grml-chroot /mnt

After the above, you should be all set for doing changes inside the disk and then running update-initramfs and update-grub if necessary.

Reference

Design

If you want to understand better the different installation procedures there is a install flowchart that was made on Draw.io.

There are also per-site install graphs:

• install-hetzner-cloud.png
• install-hetzner-robot.png
• install-ganeti.png

To edit those graphics, head to the https://draw.io website (or install their Electron desktop app) and load the install.drawio file.

Those diagrams were created as part of the redesign of the install process, to better understand the various steps of the process and see how they could be refactored. They should not be considered an authoritative version of how the process should be followed.

The text representation in this wiki remains the reference copy.

Issues

Issues regarding installation on new machines are far ranging and do not have a specific component.

The install system is manual and not completely documented for all sites. It needs to be automated, which is discussed below and in ticket 31239: automate installs.

A good example of the problems that can come up with variations in the install process is ticket 31781: ping fails as a regular user on new VMs.

Discussion

This section discusses background and implementation details of installation of machines in the project. It shouldn't be necessary for day to day operation.

Overview

The current install procedures work, but have only recently been formalized, mostly because we rarely setup machines. We do expect, however, to setup a significant number of machines in 2019, or at least significant enough to warrant automating the install process better.

Automating installs is also critical according to Tom Limoncelli, the author of the Practice of System and Network Administration. In their Ops report card, question 20 explains:

If OS installation is automated then all machines start out the same. Fighting entropy is difficult enough. If each machine is hand-crafted, it is impossible.

If you install the OS manually, you are wasting your time twice: Once when doing the installation and again every time you debug an issue that would have been prevented by having consistently configured machines.

If two people install OSs manually, half are wrong but you don't know which half. Both may claim they use the same procedure but I assure you they are not. Put each in a different room and have them write down their procedure. Now show each sysadmin the other person's list. There will be a fistfight.

In that context, it's critical to automate a reproducible install process. This gives us a consistent platform that Puppet runs on top of, with no manual configuration.

Goals

The project of automating the install is documented in ticket 31239.

Must have

• unattended installation
• reproducible results
• post-installer configuration (ie. not full installer, see below)
• support for running in our different environments (Hetzner Cloud, Robot, bare metal, Ganeti...)

Nice to have

• packaged in Debian
• full installer support:
  • RAID, LUKS, etc filesystem configuration
  • debootstrap, users, etc

Non-Goals

• full configuration management stack - that's done by service/puppet

Approvals required

TBD.

Proposed Solution

The solution being explored right now is assume the existence of a rescue shell (SSH) of some sort and use fabric to deploy everything on top of it, up to puppet. Then everything should be "puppetized" to remove manual configuration steps. See also ticket 31239 for the discussion of alternatives, which are also detailed below.

Cost

TBD.

Alternatives considered

• Ansible - configuration management that duplicates service/puppet but which we may want to use to bootstrap machines instead of yet another custom thing that operators would need to learn.
• cloud-init - builtin to many cloud images (e.g. Amazon), can do rudimentary filesystem setup (no RAID/LUKS/etc but ext4 and disk partitioning is okay), config can be fetched over HTTPS, assumes it runs on first boot, but could be coerced to run manually (e.g. fgrep -r cloud-init /lib/systemd/ | grep Exec), ganeti-os-interface backend
• cobbler - takes care of PXE and boot, delegates to kickstart the autoinstall, more relevant to RPM-based distros
• curtin - "a "fast path" installer designed to install Ubuntu quickly. It is blunt, brief, snappish, snippety and unceremonious." ubuntu-specific, not in Debian, but has strong partitioning support with ZFS, LVM, LUKS, etc support. part of the larger MAAS project
• FAI - built by a debian developer, used to build live images since buster, might require complex setup (e.g. an NFS server), setup-storage(8) is used inside our fabric-based installer. uses tar archives hosted by FAI, requires a "server" (the fai-server package), control over the boot sequence (e.g. PXE and NFS) or a custom ISO, not directly supported by Ganeti, although there are hacks to make it work and there is a ganeti-os-interface backend now, basically its own Linux distribution
• himblick has some interesting post-install configure bits in Python, along with pyparted bridges
• list of debian setup tools, see also AutomatedInstallation
• livewrapper is also one of those installers, in a way
• vmdb2 - a rewrite of vmdeboostrap, which uses a YAML file to describe a set of "steps" to take to install Debian, should work on VM images but also disks, no RAID support and a significant number of bugs might affect reliability in production
• bdebstrap - yet another one of those tools, built on top of mmdebstrap, YAML
• MAAS - PXE-based, assumes network control which we don't have and has all sorts of features we don't want
• service/puppet - Puppet could bootstrap itself, with puppet apply ran from a clone of the git repo. could be extended as deep as we want.
• terraform - config management for the cloud kind of thing, supports Hetzner Cloud, but not Hetzner Robot or Ganeti (update: there is a Hetzner robot plugin now)
• shoelaces - simple PXE / TFTP server

Unfortunately, I ruled out the official debian-installer because of the complexity of the preseeding system and partman. It also wouldn't work for installs on Hetzner Cloud or Ganeti.

Hi X!

First of all, congratulations and welcome to TPI (Tor Project, Inc.) and the TPA (Admin) team. Exciting times!

We'd like you to join us on your first orientation meeting on TODO Month day, TODO:00 UTC (TODO:00 your local time), in this BBB room:

https://bbb.torproject.net/

TODO: fill in room

Also note that we have our weekly check-in on Monday at 18:00UTC as well.

Make sure you can attend the meeting and pen it down in your calendar. If you cannot make it for some reason, please do let us know as soon as possible so we can reschedule.

Here is the agenda for the meeting:

TODO: copy paste from the OnBoardingAgendaTemplate, and append:

4. Stakeholders for your work:
   • TPA
   • web team
   • consultants
   • the rest of Tor...
5. How the TPA team works:
   • weekly meetings
   • communications: email / IRC / BBB / Signal group, talk about right to disconnect
   • support: "star of the week" shift rotation
   • issues: TPA board, web board
   • wiki: service list, policies, roadmap
6. TPA systems crash course through the new-person wiki page

Note that the "crash course" takes 20 to 30 minutes, so if you ran out of time doing the rest of the page, reschedule, don't rush.

Please have a look at the security policy. Don't worry if you don't comply yet, that will be part of your onboarding.

You will shortly receive the following credentials, in an OpenPGP encrypted email, if you haven't already:

• an LDAP account
• a Nextcloud account
• a GitLab account

If you believe you already have one of those account (GitLab, in particular), do let us know.

You should do the following with these accesses:

1. hook your favorite calendar application with your Nextcloud account
2. configure an SSH key in LDAP
3. login to people.torproject.org (aka perdulce) and download the known hosts, see the jump host documentation on how to partially automate this
4. if you need an IRC bouncer, login to chives.torproject.org and setup a screen/tmux session, or ask @pastly on IRC to get access to the ZNC bouncer
5. provide a merge request on about/people to add your bio and picture, see the documentation on the people page, add yourself to introduction in the wiki

So you also have a lot of reading to do already! The new-person page is a good reference to get started.

But take it slowly! It can be overwhelming to join a new organisation and it will take you some time to get acquainted with everything. Don't hesitate to ask if you have any questions!

See you soon, and welcome aboard!

IMPORTANT NOTE: most Tor servers do not currently use nftables, as we still use the Ferm firewall wrapper, which only uses iptables. Still, we sometimes end up on machines that might have nftables and those instructions will be useful for that brave new future. See tpo/tpa/team#40554 for a followup on that migration.

• Listing rules
• Checking and applying a ruleset
• Inserting a rule to bypass a restriction
• Blocking a host
• Deleting a rule
• Other documentation

Listing rules

nft -a list ruleset

The -a flag shows the handles which is useful to delete a specific rule.

Checking and applying a ruleset

This checks the ruleset of Puppet rule files as created by the puppet/nftables modules before applying it:

nft -c -I /etc/nftables/puppet -f /etc/nftables/puppet.nft

This is done by Puppet before actually applying the ruleset, which is done with:

nft -I /etc/nftables/puppet -f /etc/nftables/puppet.nft

The -I parameter stands for --includepath and tells nft to look for rules in that directory.

You can try to load the ruleset but flush it afterwards in case it crashes your access with:

nft -f /etc/nftables.conf ; sleep 30 ; nft flush ruleset

Inserting a rule to bypass a restriction

Say you have the chain INPUT in the table filter which looks like this:

table inet filter {
	chain INPUT {
		type filter hook input priority filter; policy drop;
		iifname "lo" accept
		ct state established,related accept
		ct state invalid drop
		tcp dport 22 accept
		reject
	}
}

.. and you want to temporarily give access to the web server on port 443. You would do a command like:

nft insert rule inet filter INPUT 'tcp dport 443 accept'

Or if you need to allow a specific IP, you could do:

nft insert rule inet filter INPUT 'ip saddr 192.0.2.0/24 accept'

Blocking a host

Similarly, assuming you have the same INPUT chain in the filter table, you could do this to block a host from accessing the server:

nft insert rule inet filter INPUT 'ip saddr 192.0.2.0/24 reject'

That will generate an ICMP response. If this is a DOS condition, you might rather avoid that and simply drop the packet with:

nft insert rule inet filter INPUT 'ip saddr 192.0.2.0/24 drop'

Deleting a rule

If you added a rule by hand in the above and now want to delete it, you first need to find the handle (with the -a flag to nft list ruleset) and then delete the rule:

nft delete rule inet filter INPUT handle 39

Be VERY CAREFUL with this step as using the wrong handle might lock you out of the server.

Other documentation

• nftables wiki, and especially:
  • quick reference
  • iptables to nftables translation
• RHEL nftables reference
• Debian wiki
• Arch wiki

OpenPGP is an encryption and authentication system which is extensively used at Tor.

• Tutorial
  • OpenPGP with Thunderbird training
• How-to
  • Diffing OpenPGP keys, signatures and encrypted files from Git
  • Generate a Curve25519 key
  • Generating a revocation certificate
  • Revoking a key
  • Rotating keys
  • Backing up an OpenPGP key
  • Secret sharing
    • Dry runs
    • Sample README file
    • Other approaches
  • Pager playbook
  • Disaster recovery
• Reference
  • Installation
  • SLA
  • Design
  • Issues
  • Maintainer, users, and upstream
  • Monitoring and testing
  • Logs and metrics
  • Backups
  • Other documentation
• Discussion
  • Overview
  • Goals
    • Must have
    • Nice to have
    • Non-Goals
  • Approvals required
  • Proposed Solution
  • Cost
  • Alternatives considered
    • Expiration dates
    • Separate certification key
    • Airgapped systems
    • About ECC (elliptic curve cryptography)
    • Why GnuPG
    • Other OpenPGP implementations
      • Sequoia (Rust)
      • RNP (C++)
      • PGPainless (Java)
      • Others
    • OpenPGP backups

Tutorial

This documentation assumes minimal technical knowledge, but it should be noted that OpenPGP is notoriously hard to implement correctly, and that user interfaces have been known to be user-hostile in the past. This documentation tries to alleviate those flaws, but users should be aware that there are challenges in using OpenPGP safely.

If you're looking for documentation on how to use OpenPGP with a YubiKey, that lives in the YubiKey documentation.

OpenPGP with Thunderbird training

Rough notes for the OpenPGP training to be given at the 2023 Tor meeting in Costa Rica.

1. Upgrade Thunderbird to version 78.2.1 or later at https://www.thunderbird.net/ (Mac, Windows, Linux) or through you local package manager (Linux), if you do not have Thunderbird installed, you will need to install it and follow the email setup instructions to setup the Tor mail server
2. Set a Primary Password in Edit -> Settings -> Privacy & Security
   • Check Use a primary password
   • Enter the password and click OK
3. Select the @torproject.org user identity as Default in Edit -> Account Settings -> Manage Identities
4. Generate key with expiration date in Tools -> OpenPGP Key Manager -> Generate -> New Key Pair
   • Make sure you select an expiration date, can be somewhere between one to 3 years, preferably one year
   • Optionally, select ECC (Elliptic Curve) as a Key type in Advanced Settings
   • Click Generate Key and confirm
   • Make a backup: File -> Backup secret key(s) to File
5. Send a signed email to another user, have another user send you such an email as well
6. Send an encrypted mail to a new recipient:
   1. click Encrypt
   2. big yellow warning, click Resolve...
   3. Discover public keys online...
   4. A key is available, but hasn't been accepted yet, click Resolve...
   5. Select the first key
7. Setting up a submission server account, see the email tutorial which involves a LDAP password reset (assuming you already have an LDAP account, otherwise getting TPA to make you one) and sending a signed OpenPGP mail to chpasswd@db.torproject.org with the content Please change my Tor password
8. send your key to TPA:
   1. Tools -> OpenPGP Key Manager
   2. select the key
   3. File -> Export public key(s) to File
   4. in a new ticket, attach the file
9. Verifying incoming mail:
   1. OpenPGP menu: This message claims to contain the sender's OpenPGP public key, click Import...
   2. Click accepted (unverified)
   3. You should now see a little "seal" logo with a triangle "warning sign", click on it and then View signer's key
   4. There you can verify the key
10. Renewing your OpenPGP key:
    1. Edit -> Account Settings -> End-to-End encryption
    2. on the affected key, click Change Expiration Date
    3. send your key to TPA, as detailed above
11. Verifying and trusting keys, a short discussion on "TOFU" and the web of trust, WKD and autocrypt

Notes:

• we do not use key servers and instead rely on WKD and Autocrypt for key discovery
• it seems like Thunderbird and RNP do not support generating revocation certificates, only revoking the key directly
• sequoia-octopus-librnp can provide a drop-in replacement for Thunderbird's RNP library and give access to a normal keyring, but is more for advanced users and not covered here
• openpgp.org is a good entry point, good list of software for example, website source on GitHub
• we must set a master password in thunderbird, it's the password that protects the keyring (to be verified)

Other tutorials:

• How-to geek has a good reference which could be used as a basis, but incorrectly suggests to not have an expiry date, and does not suggest doing a backup
• Tails: uses Kleopatra and Thunderbird, but with the Enigmail stuff, outdated, Linux-specific
• boum's guide: french, but otherwise good reference
• Thunderbird's documentation is a catastrophe. basic, cryptic wiki page that points to a howto and FAQ that is just a pile of questions, utterly useless, other than as a FAQ, their normal guide is still outdated and refers to Enigmail
• the EFF Surveillance Self-Defense guide is also outdated, their Linux, Windows and Mac are marked as "retired"

How-to

Diffing OpenPGP keys, signatures and encrypted files from Git

Say you store OpenPGP keyrings in git. For example, you track package repositories public signing keys or you have a directory of user keys. You need to update those keys but want to make sure the update doesn't add untrusted key material.

This guide will setup your git commands to show a meaningful diff of binary or ascii-armored keyrings.

1. add this to your ~/.gitconfig (or, if you want to restrict it to a single repository, in .git/config:

   # handler to parse keyrings
   [diff "key"]
   textconv = gpg --batch --no-tty --with-sig-list --show-keys <
   
   # handler to verify signatures
   [diff "sig"]
       textconv = gpg --batch --no-tty --verify <
   
   # handler to decrypt files
   [diff "pgp"]
       textconv = gpg --batch --no-tty --decrypt <
   

2. add this to your ~/.config/git/attributes (or, the per repository .gitattributes file), so that those handlers are mapped to file extensions:

   *.key diff=key
   *.sig diff=sig
   *.pgp diff=pgp
   

   .key, .sig, and .pgp are "standard" extensions (as per /etc/mime.types), but people frequently use other extensions, so you might want to have this too:

   *.gpg diff=key
   *.asc diff=key
   

Then, when you change a key, git diff will show you something like this, which is when the GitLab package signing key was renewed:

commit c29047357669cb86cf759ecb8a44e14ca6d5c130
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Wed Mar 2 15:31:36 2022 -0500

    renew gitlab's key which expired yesterday

diff --git a/modules/profile/files/gitlab/gitlab-archive-keyring.gpg b/modules/profile/files/gitlab/gitlab-archive-keyring.gpg
index e38045da..3e57c8e0 100644
--- a/modules/profile/files/gitlab/gitlab-archive-keyring.gpg
+++ b/modules/profile/files/gitlab/gitlab-archive-keyring.gpg
@@ -1,7 +1,7 @@
-pub   rsa4096/3F01618A51312F3F 2020-03-02 [SC] [expired: 2022-03-02]
+pub   rsa4096/3F01618A51312F3F 2020-03-02 [SC] [expires: 2024-03-01]
       F6403F6544A38863DAA0B6E03F01618A51312F3F
 uid                            GitLab B.V. (package repository signing key) <packages@gitlab.com>
-sig 3        3F01618A51312F3F 2020-03-02  GitLab B.V. (package repository signing key) <packages@gitlab.com>
-sub   rsa4096/1193DC8C5FFF7061 2020-03-02 [E] [expired: 2022-03-02]
-sig          3F01618A51312F3F 2020-03-02  GitLab B.V. (package repository signing key) <packages@gitlab.com>
+sig 3        3F01618A51312F3F 2022-03-02  GitLab B.V. (package repository signing key) <packages@gitlab.com>
+sub   rsa4096/1193DC8C5FFF7061 2020-03-02 [E] [expires: 2024-03-01]
+sig          3F01618A51312F3F 2022-03-02  GitLab B.V. (package repository signing key) <packages@gitlab.com>
 
[...] 

The reasoning behind each file extension goes as follows:

• .key - OpenPGP key material. process it with --show-keys < file
• .sig - OpenPGP signature. process it with --verify < file
• .pgp - OpenPGP encrypted material. process it with --decrypt < file
• .gpg - informal. can be anything, but generally assumed to be binary. we treat those as OpenPGP keys, because that's the safest thing to do
• .asc - informal. can be anything, but generally assumed to be ASCII-armored, assumed to be the same as .gpg otherwise.

We also use those options:

• --batch is, well, never sure what --batch is for, but seems reasonable?
• --no-tty is to force GnuPG to not assume a terminal which may make it prompt the user for things, which could break the pager

Note that, you might see the advice to run gpg < file (without any arguments) elsewhere, but we advise against it. In theory, gpg < file can do anything, but it will typically:

1. decrypt encrypted material, or;
2. verify signed material, or;
3. show public key material

From what I can tell in the source code, it will also process private key material and other nasty stuff, so it's unclear if it's actually safe to run at all. See do_proc_packets() that is called with opt.list_packets == 0 in the GnuPG source code.

Also note that, without <, git passes a the payload to gpg through a binary file, and GnuPG then happily decrypts it and puts in publicly readable in /tmp. boom. This behavior was filed in 2017 as a bug upstream (T2945) but was downgraded to a "feature request" by the GnuPG maintainer a few weeks later. No new activity at the time of writing (2022, five years later).

All of this is somewhat brittle: gpg < foo is not supposed to work and may kill your cat. Bugs should be filed to have something that does the right thing, or at least not kill defenseless animals.

Generate a Curve25519 key

Here we're generating a new OpenPGP key as we're transitioning from an old RSA4096 key. DO NOT follow those steps if you wish to keep your old key, of course.

Note that the procedure below generates the key in a temporary, memory-backed, filesystem (/run is assumed to be a tmpfs). The key will be completely lost on next reboot unless it's moved to a security key or to an actual home. See the YubiKey documentation for how to move it to a YubiKey, for example, and see the Airgapped systems for a discussion on that approach.

GnuPG (still) requires --expert mode to generate Curve25519 keys, unfortunately. Note that you could also accomplish this by sending a "batch" file, for example drduh has this example for ed25519 keys, see also GnuPG's guide.

Here's the transcript of a Curve25519 key generation with an encryption and authentication subkey:

export GNUPGHOME=${XDG_RUNTIME_DIR:-/nonexistent}/.gnupg/
anarcat@angela:~[SIGINT]$ gpg --full-gen-key --expert
gpg (GnuPG) 2.2.40; Copyright (C) 2022 g10 Code GmbH
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
   (9) ECC and ECC
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (13) Existing key
  (14) Existing key from card
Your selection? 11

Possible actions for a ECDSA/EdDSA key: Sign Certify Authenticate 
Current allowed actions: Sign Certify 

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? q
Please select which elliptic curve you want:
   (1) Curve 25519
   (3) NIST P-256
   (4) NIST P-384
   (5) NIST P-521
   (6) Brainpool P-256
   (7) Brainpool P-384
   (8) Brainpool P-512
   (9) secp256k1
Your selection? 1
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at mer 29 mai 2024 15:27:14 EDT
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: Antoine Beaupré
Email address: anarcat@anarc.at
Comment: 
You are using the 'utf-8' character set.
You selected this USER-ID:
    "Antoine Beaupré <anarcat@anarc.at>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: directory '/home/anarcat/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/anarcat/.gnupg/openpgp-revocs.d/D0D396D08E761095E2910413DDE8A0D1D4CFEE10.rev'
public and secret key created and signed.

pub   ed25519/DDE8A0D1D4CFEE10 2023-05-30 [SC] [expires: 2024-05-29]
      D0D396D08E761095E2910413DDE8A0D1D4CFEE10
uid                            Antoine Beaupré <anarcat@anarc.at>

anarcat@angela:~$ 

Let's put this fingerprint aside, as we'll be using it over and over again:

FINGERPRINT=D0D396D08E761095E2910413DDE8A0D1D4CFEE10

Let's look at this key:

anarcat@angela:~$ gpg --edit-key $FINGERPRINT
gpg (GnuPG) 2.2.40; Copyright (C) 2022 g10 Code GmbH
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2024-05-29
sec  ed25519/02293A6FA4E53473
     created: 2023-05-30  expires: 2024-05-29  usage: SC  
     trust: ultimate      validity: ultimate
ssb  cv25519/0E1C0B264FC7ADEA
     created: 2023-05-30  expires: 2024-05-29  usage: E   
[ultimate] (1). Antoine Beaupré <anarcat@anarc.at>

gpg>

As we can see, this created two key pairs:

1. "primary key" which is a public/private key with the S (Signing) and C (Certification) purposes. that key can be used to sign messages, certify other keys, new identities, and subkeys (see why we use both in Separate certification key)

2. an E (encryption) "sub-key" pair which is used to encrypt and decrypt messages

Note that the encryption key expires here, which can be annoying. You can delete the key and recreate it this way:

anarcat@angela:~[SIGINT]$ gpg --expert --edit-key $FINGERPRINT 
gpg (GnuPG) 2.2.40; Copyright (C) 2022 g10 Code GmbH
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec  ed25519/02293A6FA4E53473
     created: 2023-05-30  expires: 2024-05-29  usage: SC  
     trust: ultimate      validity: ultimate
ssb  cv25519/0E1C0B264FC7ADEA
     created: 2023-05-30  expires: 2024-05-29  usage: E   
[ultimate] (1). Antoine Beaupré <anarcat@anarc.at>

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
  (14) Existing key from card
Your selection? 12
Please select which elliptic curve you want:
   (1) Curve 25519
   (3) NIST P-256
   (4) NIST P-384
   (5) NIST P-521
   (6) Brainpool P-256
   (7) Brainpool P-384
   (8) Brainpool P-512
   (9) secp256k1
Your selection? 1
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec  ed25519/02293A6FA4E53473
     created: 2023-05-30  expires: 2024-05-29  usage: SC  
     trust: ultimate      validity: ultimate
ssb  cv25519/0E1C0B264FC7ADEA
     created: 2023-05-30  expires: 2024-05-29  usage: E   
ssb  cv25519/9456BA69685EAFFB
     created: 2023-05-30  expires: never       usage: E   
[ultimate] (1). Antoine Beaupré <anarcat@anarc.at>

gpg> key 1

sec  ed25519/02293A6FA4E53473
     created: 2023-05-30  expires: 2024-05-29  usage: SC  
     trust: ultimate      validity: ultimate
ssb* cv25519/0E1C0B264FC7ADEA
     created: 2023-05-30  expires: 2024-05-29  usage: E   
ssb  cv25519/9456BA69685EAFFB
     created: 2023-05-30  expires: never       usage: E   
[ultimate] (1). Antoine Beaupré <anarcat@anarc.at>

gpg> delkey
Do you really want to delete this key? (y/N) y

sec  ed25519/02293A6FA4E53473
     created: 2023-05-30  expires: 2024-05-29  usage: SC  
     trust: ultimate      validity: ultimate
ssb  cv25519/9456BA69685EAFFB
     created: 2023-05-30  expires: never       usage: E   
[ultimate] (1). Antoine Beaupré <anarcat@anarc.at>

See also the Expiration dates discussion.

We'll also add a third key here, which is an A (Authentication) key, which will be used for SSH authentication:

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
  (14) Existing key from card
Your selection? 11

Possible actions for a ECDSA/EdDSA key: Sign Authenticate 
Current allowed actions: Sign 

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? a

Possible actions for a ECDSA/EdDSA key: Sign Authenticate 
Current allowed actions: Sign Authenticate 

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? s

Possible actions for a ECDSA/EdDSA key: Sign Authenticate 
Current allowed actions: Authenticate 

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? q
Please select which elliptic curve you want:
   (1) Curve 25519
   (3) NIST P-256
   (4) NIST P-384
   (5) NIST P-521
   (6) Brainpool P-256
   (7) Brainpool P-384
   (8) Brainpool P-512
   (9) secp256k1
Your selection? 1
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec  ed25519/02293A6FA4E53473
     created: 2023-05-30  expires: 2024-05-29  usage: SC  
     trust: ultimate      validity: ultimate
ssb  cv25519/9456BA69685EAFFB
     created: 2023-05-30  expires: never       usage: E   
ssb  ed25519/9FF21704D101630D
     created: 2023-05-30  expires: never       usage: A   
[ultimate] (1)* Antoine Beaupré <anarcat@anarc.at>

At this point, you should have a functional and valid set of OpenPGP certificates! It's a good idea to check the key with with hokey lint, from hopenpgp-tools:

gpg --export $FINGERPRINT | hokey lint

Following the above guide, I ended up with a key that is all green except for the authentication key having False in embedded cross-cert. According to drduh's guide, that doesn't matter:

hokey may warn (orange text) about cross certification for the authentication key. GPG's Signing Subkey Cross-Certification documentation has more detail on cross certification, and gpg v2.2.1 notes "subkey does not sign and so does not need to be cross-certified".

Also make sure you generate a revocation certificate, see below.

Generating a revocation certificate

If you do not have one already, you should generate a revocation certificate with:

gpg --generate-revocation $FINGERPRINT

This should be stored in a safe place.

The point of a revocation certificate is to provide a last safety measure if you lose control of your key. It allows you to mark your key as unusable to the outside world, which will make it impossible for a compromised key to be used to impersonate you, provided the certificate is distributed properly, of course.

It will not keep an attacker from reading your encrypted material, nor will it allow you to read encrypted material for a key you have lost, however. It will keep people from encrypting new material to you, however.

A good practice is to print this on paper (yes, that old thing) and store it among your other precious papers. The risk to that document is that someone could invalidate your key if they lay their hands on it. But the reverse is that losing it might make you unable to decrypt some messages sent to you if you lost your original key material.

When printing the key, you can optionally add a more "scannable" version by embedding a QR code in the document. One of those tools might be able to help:

• gpg2qr
• paperbackup
• qr-backup

Make sure you can recover from the QR codes before filing them away. Also make sure the printer is plugged in, has toner or ink, no paper jam, and use a fresh ream of paper as used paper tends to jam more. Also send a donation to your local anarchist bookstore, pet your cat, or steal a book to please the printer gods.

Revoking a key

Note: this assumes you generated a revocation certificate when you created the key. If you still have access to the private key material and have not generated a revocation certificate, go ahead and do that right now, see above.

To revoke an OpenPGP key, you first need to find the revocation certificate and, if on paper, digitize it in a text file. Then import the document:

gpg --import < revocation.key

The key can then be published as normal, say:

gpg --send-keys $FINGERPRINT

Rotating keys

First, generate a key as detailed above.

When you are confident the new key can be put in use, sign the the new key with old key:

gpg --default-key $OLDKEY --sign-key $FINGERPRINT

And revoke the old key:

gpg --generate-revocation $OLDKEY

Then you need to publish the new key and retire the old one everywhere. This will vary wildly according to how you have used the old key and intend to use the new one.

In my case, this implied:

• change the default key in GnuPG:

   sed -i "s/default-key.*/default-key $FINGERPRINT/" ~/.gnupg/gpg.conf
  

• changing the PASSWORD_STORE_SIGNING_KEY environment:

   export PASSWORD_STORE_SIGNING_KEY=$FINGERPRINT
   echo PASSWORD_STORE_SIGNING_KEY=$FINGERPRINT >> ~/.config/environment.d/shenv.conf
  

• re-encrypt the whole password manager:

   pass init $FINGERPRINT
  

• change the fingerprint in my WKD setup, which means changing the FINGERPRINT in this Makefile and calling:

   make -C ~/wikis/anarc.at/.well-known/openpgpkey/ hu
  

• upload the new key everywhere which, in my case, means:

   gpg --keyserver keyring.debian.org --send-keys $FINGERPRINT
   gpg --keyserver keys.openpgp.org --send-keys $FINGERPRINT
   gpg --keyserver pool.sks-keyservers.net --send-keys $FINGERPRINT
  

  ... and those sites:

  	* <https://gitlab.torproject.org/-/profile/gpg_keys>
   * <https://gitlab.com/-/profile/gpg_keys>
   * <https://github.com/settings/keys>
  

• change my OpenPGP SSH key in a lot of authorized_keys files, namely:

  • home network (Puppet)
  • work (Puppet)
  • https://gitlab.torproject.org/-/profile/keys
  • https://gitlab.com/-/profile/keys
  • https://github.com/settings/keys

• change your Git signing key:

   git config --global user.signingkey $FINGERPRINT
  

• follow the Debian.org key replacement procedure

• consider publishing a full "key transition statement" (example), signed with both keys:

   gpg --local-user $FINGERPRINT --local-user $OLD_FINGERPRINT --clearsign openpgp-transition-2023.txt
  

You may also want to backup your old encryption key, also removing the password! Otherwise you will likely not remember the password. To do this, first enter the --edit-key mode:

gpg --edit-key $OLD_FINGERPRINT

Then remove the password on the old keyring:

toggle
passwd

Then export the private keys and encrypt them with your key:

gpg --export-secret-keys $OLD_FINGERPRINT | gpg --encrypt -r $FINGERPRINT

Then you can delete the old secret subkeys:

gpg --delete-secret-keys $OLD_FINGERPRINT

Note that the above exports all secret subkeys associated with the $OLD_FINGERPRINT. If you only want to export the encryption subkey, you need to remove the other keys first. You can remove keys by using the "keygrip", which should look something like this:

    $ gpg --with-keygrip --list-secret-keys
    /run/user/1000/ssss/gnupg/pubring.kbx
    -------------------------------------
    sec   ed25519 2023-05-30 [SC] [expires: 2024-05-29]
          BBB6CD4C98D74E1358A752A602293A6FA4E53473
          Keygrip = 23E56A5F9B45CEFE89C20CD244DCB93B0CAFFC73
    uid           [ unknown] Antoine Beaupré <anarcat@anarc.at>
    ssb   cv25519 2023-05-30 [E]
          Keygrip = 74D517AB0466CDF3F27D118A8CD3D9018BA72819

    $ gpg-connect-agent "DELETE_KEY 23E56A5F9B45CEFE89C20CD244DCB93B0CAFFC73" /bye
    $ gpg --list-secret-keys BBB6CD4C98D74E1358A752A602293A6FA4E53473
    sec#  ed25519 2023-05-30 [SC] [expires: 2024-05-29]
          BBB6CD4C98D74E1358A752A602293A6FA4E53473
    uid           [ unknown] Antoine Beaupré <anarcat@anarc.at>
    ssb  cv25519 2023-05-30 [E]

In the above, the first line of the second gpg output shows that the primary ([SC]) key is "unusable" (#).

Backing up an OpenPGP key

OpenPGP keys can typically be backed up normally, unless they are in really active use. For example, an OpenPGP-backed CA that would see a lot of churn in its keyring might have an inconsistent database if a normal backup program is ran while a key is added. This is highly implementation-dependent of course...

You might also want to do a backup for other reasons, for example with a scheme like Shamir's secret sharing to delegate this responsibility to others in case you are somewhat incapacitated.

Therefore, here is a procedure to make a full backup of an OpenPGP key pair stored in a GnuPG keyring, in an in-memory temporary filesystem:

export TMP_BACKUP_DIR=${XDG_RUNTIME_DIR:-/nonexistent}/openpgp-backup-$FINGERPRINT/ &&
(
    umask 0077 &&
    mkdir $TMP_BACKUP_DIR &&
    gpg --export-secret-keys $FINGERPRINT > $TMP_BACKUP_DIR/openpgp-backup-$FINGERPRINT-secret.key &&
    gpg --export $FINGERPRINT > $TMP_BACKUP_DIR/openpgp-backup-public-$FINGERPRINT.key &&
)

The files in $TMP_BACKUP_DIR can now be copied to a safe location. They retain their password encryption, which is fine for short-term backups. If you are doing a backup that you might only use in the far future or want to share with others (see secret sharing below), however, you will probably want to remove the password protection on the secret keys, so that you use some other mechanism to protect the keys, for example with a shared secret or encryption with a security token.

This procedure, therefore, should probably happen in a temporary keyring:

umask 077 &&
TEMP_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/gpg-unsafe/ &&
mkdir $TEMP_DIR &&
export GNUPGHOME=$TEMP_DIR/gnupg &&
cp -Rp ~/.gnupg/ $GNUPGHOME

Then remove the password protection on the keyring:

gpg --edit-key $FINGERPRINT

... then type the passwd command and just hit enter when prompted for the password. Ignore the warnings.

Then export the entire key bundle into a temporary in-memory directory, tar all those files and self-encrypt:

BACKUP_DIR=/mnt/...
export TMP_BACKUP_DIR=${XDG_RUNTIME_DIR:-/nonexistent}/openpgp-backup-$FINGERPRINT/ &&
(
    umask 0077 &&
    mkdir $TMP_BACKUP_DIR &&
    gpg --export-secret-keys $FINGERPRINT > $TMP_BACKUP_DIR/openpgp-backup-$FINGERPRINT-secret.key &&
    gpg --export $FINGERPRINT > $TMP_BACKUP_DIR/openpgp-backup-public-$FINGERPRINT.key &&
    tar -C ${XDG_RUNTIME_DIR:-/nonexistent} -c -f - openpgp-backup-$FINGERPRINT \
        | gpg --encrypt --recipient $FINGERPRINT - \
        > $BACKUP_DIR/openpgp-backup-$FINGERPRINT.tar.pgp &&
    cp $TMP_BACKUP_DIR/openpgp-backup-public-$FINGERPRINT.key $BACKUP_DIR
)

Next, test decryption:

gpg --decrypt $BACKUP_DIR/openpgp-backup-$FINGERPRINT.tar.pgp | file -

Where you store this backup ($BACKUP_DIR above) is up to you. See the OpenPGP backups discussion for details.

Also note how we keep a plain-text copy of the public key. This is an important precaution, especially if you're the paranoid type that doesn't public their key anywhere. You can recover a working setup from a backup secret key only (for example from a YubiKey), but it's much harder if you don't have the public key, so keep that around.

Secret sharing

A backup is nice, but it still assumes you are alive and able to operate your OpenPGP keyring or security key. If you go missing or lose your memory, you're in trouble. To protect you and your relatives from the possibility of total loss of your personal data, you may want to consider a scheme like Shamir's secret sharing.

The basic idea is that you give a symmetrically encrypted file to multiple, trusted people. The decryption key is split among a certain number (N) of tokens, out of which a smaller number (say K) tokens is required to reassemble the secret.

The file contains the private key material and public key. In our specific case, we're only interested in the encryption key: the logic behind this is that this is the important part that cannot be easily recovered from loss. Signing, authentication or certification key can all be revoked and recreated, but the encryption key, if lost, leads to more serious problems as the encrypted data cannot be recovered.

So, in this procedure, we'll take an OpenPGP key, strip out the primary secret key material, export the encryption subkey into an encrypted archive, and split its password into multiple parts. We'll also remove the password on the OpenPGP key so that our participants can use the key without having to learn another secret, the rationale here is that the symmetric encryption is sufficient to protect the key.

1. first, work on a temporary, in-memory copy of your keyring:

   umask 077
   TEMP_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/ssss/
   mkdir $TEMP_DIR
   export GNUPGHOME=$TEMP_DIR/gnupg
   cp -Rp ~/.gnupg/ $GNUPGHOME
   

   This simply copies your GnuPG home into a temporary location, an in-memory filesystem (/run). You could also restore from the backup created in the previous section with:

   umask 077
   TEMP_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/ssss/
   mkdir $TEMP_DIR $TEMP_DIR/gnupg
   gpg --decrypt $BACKUP_DIR/openpgp-backup-$FINGERPRINT.tar.pgp | tar -x -f - --to-stdout | gpg --homedir $TEMP_DIR/gnupg --import
   export GNUPGHOME=$TEMP_DIR/gnupg
   

   At this point, your GNUPGHOME variable should point at /run, make sure it does:

   echo $GNUPGHOME
   gpgconf --list-dir homedir
   

   It's extremely important that GnuPG doesn't start using your normal keyring, as you might delete the key in the wrong keyring. Feel free to move ~/.gnupg out of the way to make sure it doesn't destroy private key material there.

2. remove the passwword on the key with:

   gpg --edit-key $FINGERPRINT
   

   then type the passwd command and just hit enter when prompted for the password. Ignore the warnings.

3. (optional) delete the primary key, for this we need to manipulate the key in a special way, using the "keygrip":

   $ gpg --with-keygrip --list-secret-keys
   /run/user/1000/ssss/gnupg/pubring.kbx
   -------------------------------------
   sec#  ed25519 2023-05-30 [SC] [expires: 2024-05-29]
         BBB6CD4C98D74E1358A752A602293A6FA4E53473
         Keygrip = 23E56A5F9B45CEFE89C20CD244DCB93B0CAFFC73
   uid           [ unknown] Antoine Beaupré <anarcat@anarc.at>
   ssb   cv25519 2023-05-30 [E]
         Keygrip = 74D517AB0466CDF3F27D118A8CD3D9018BA72819
   
   $ gpg-connect-agent "DELETE_KEY 23E56A5F9B45CEFE89C20CD244DCB93B0CAFFC73" /bye
   $ gpg --list-secret-keys BBB6CD4C98D74E1358A752A602293A6FA4E53473
   sec#  ed25519 2023-05-30 [SC] [expires: 2024-05-29]
         BBB6CD4C98D74E1358A752A602293A6FA4E53473
   uid           [ unknown] Antoine Beaupré <anarcat@anarc.at>
   ssb   cv25519 2023-05-30 [E]
   

4. create a password and split it in tokens:

   tr -dc '[:alnum:]' < /dev/urandom | head -c 30 ; echo
   ssss-split -t 3 -n 5
   

   Note: consider using SLIP-0039 instead, see below.

5. export the secrets and create the encrypted archive:

   mkdir openpgp-ssss-backup-$FINGERPRINT
   gpg --export $FINGERPRINT > openpgp-ssss-backup-$FINGERPRINT/openpgp-backup-public-$FINGERPRINT.key
   gpg --export-secret-keys $FINGERPRINT > openpgp-ssss-backup-$FINGERPRINT/openpgp-ssss-backup-$FINGERPRINT-secret.key
   tar -c -f - openpgp-ssss-backup-$FINGERPRINT | gpg --symmetric - > openpgp-ssss-backup-$FINGERPRINT.tar.pgp
   rm -rf openpgp-ssss-backup-$FINGERPRINT
   

   Note that if you expect your peers to access all your data, the above might not be sufficient. It is, for example, typical to store home directories on full disk encryption. The above will therefore not be sufficient to access (say) your OpenPGP-encrypted password manager or emails. So you might want to also include a password for one of the LUKS slots in the directory as well.

6. send a README, the .pgp file and one token for each person

Dry runs

You might want to periodically check in with those people. It's perfectly natural for people to forget or lose things. Ensure they still have control of their part of the secrets and the files, know how to use it and can still contact each other, possibly as a yearly event.

This is a message I send everyone in the group once a year:

Hi!

You're in this group and receiving this message because you
volunteered to be one of my backups. At about this time of the year in
2023, I sent you a secret archive encrypted with a secret spread among
you, that 3 out of 5 people need to share to recover.

Now we're one year later and i'd like to test that this still
works. please try to find the encrypted file, the instructions (which
should be stored in a README along side the encrypted file) and the
sharded secret, and then come back here to confirm that you still have
access to those.

DO NOT share the secret, i am not dead and still fully functional,
this is just a drill.

If anyone fails to reply after 6 weeks, or around mid-august, I'll
start the procedure to reroll the keys to a new group without that
person.

If you want out of the group, now is a good time to say so as well. 

If you don't understand what this is about, it's an excellent time to
ask, don't be shy, it's normal to forget that kind of stuff after a
year, it's why i run those drills!

so TL;DR: confirm that you still have:
1. the secret archive, should be named `openpgp-ssss-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473.tar.gpg`
2. the instructions (optional), should be named `README.md`
3. the shared secret (should be in your password manager)

thanks!

Sample README file

The README file needs to explain how to recover from all of this. Consider that your peers (or yourself!) might not actually remember any of how this works, so it should be detailed more than less, and should be available in clear text.

Here's an example:

# About this file

You are receiving this information because you are deemed trustworthy
to carry out the instructions in this file.

Some of the data you've been given is secret and must be handled with
care. It is important that it is not lost. Your current
operational security and procedures are deemed sufficient to handle
this data. It is expected, for example, that you store those secrets
in your password manager, and that the password manager is backed up.

You can use any name to store the secret token, but I suggest you file
that secret under the name "anarcat-openpgp-ssss-token".

You are among 5 other persons to receive this data. Those people are:

 * [redacted name, email, phone, etc]
 * [redacted name, email, phone, etc]
 * [...]

Three of you are necessary to recover this data. See below for
instructions on how to do so.

It is expected that if you end up in a position to not be able to
recover those secrets, you will notify me or, failing that, the other
participants so that appropriate measures be taken.

It is also expected that, if you completely lose contact with me and
are worried about my disappearance, you will contact next of kin. You
can reach my partner and family at:

 * [redacted name, email, phone, etc]
 * [...]

Those people are the ones responsible for making decisions on
sensitive issues about my life, and should be reached in the event of
my death or incapacity.

Those instructions were written on YYYY-MM-DD and do not constitute a
will.

# Recovery instructions

What follows describes the recovery of anarcat's secrets in case of
emergency, written by myself, anarcat.

## Background

I own and operate a handful of personal servers dispersed around the
globe. Some documentation of those machines is available on the
website:

<https://anarc.at/hardware>

and:

<https://anarc.at/services>

If all goes well, `marcos` is the main server where everything
is. There's a backup server named `tubman` currently hosted at
REDACTED by REDACTED.

Those instructions aim at being able to recover the data on those
servers if I am incapacitated, dead, or somehow loses my memory.

## Recovery

You are one of five people with a copy of those instructions.

Alongside those instructions, you should have received two things:

 * a secret token
 * an encrypted file

The secret token, when assembled with two of the other parties in this
group, should be able to recover the full decryption key for the
OpenPGP-encrypted file. This is done with Shamir's Secret Sharing
Scheme (SSSS):

<https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing>

The encrypted file, in turn, contains two important things:

 1. a password to decrypt the LUKS partition on any of my machines
 2. a password-less copy of my OpenPGP keyring

The latter allows you to access my password manager, typically stored
in `/home/anarcat/.password-store/` on the main server (or my laptop).

So the exact procedure is:

 1. gather three of the five people together
 2. assemble the three tokens with the command `ssss-combine -t 3`
 3. decrypt the file with `gpg --decrypt anarcat-rescue.tar.pgp`
 4. import the OpenPGP secret key material with `gpg --import
    openpgp-BBB6CD4C98D74E1358A752A602293A6FA4E53473-secret.key`
 5. the LUKS decryption key is in the `luks.gpg` file

## Example

Here, three people are there to generate the secret. They call the
magic command and type each their token in turn, it should look
something like this:

    $ ssss-combine -t 3
    Enter 3 shares separated by newlines:
    Share [1/3]: 2-e9b89a7bd56abf0164e57a7e9a0629a268f57e1d1b0475ff5062e101
    Share [2/3]: 5-869c193144bcc58ed864d6648661ab83c7ce5b0751d649d5c54f77a9
    Share [3/3]: 1-039c2941fb73620acf9be7eabb2191160b7474a7cdebc405e612beb0
    Resulting secret: YXtJpJwzCqd1ELh3KQCEuJSvu84d

(Obviously, the above is just an example and not the actual secret.)

Then the "Resulting secret" can be used to decrypt the file:

    $ gpg --decrypt openpgp-ssss-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473.tar.gpg > openpgp-ssss-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473.tar
    gpg: AES256.CFB encrypted data
    gpg: encrypted with 1 passphrase

Then from there, the `tar` archive can be extracted:

    $ tar xfv openpgp-ssss-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473.tar
    openpgp-ssss-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473/
    openpgp-ssss-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473/openpgp-ssss-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473-secret.key
    openpgp-ssss-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473/luks.gpg

The encryption subkey should be importable with:

    gpg --import < anarcat-secrets/openpgp-backup-BBB6CD4C98D74E1358A752A602293A6FA4E53473-secret-subkeys.key

To get access to more resources, you might need to unlock a LUKS (on
the main server, currently `marcos`) or encrypted ZFS (on the
backup server, currently `tubman`) partition. The key should be
readable in the `luks.gpg` file:

    gpg --decrypt luks.gpg

From there you should be able to access either the backup or main
server and, from there, access the password manager in
`.password-store`.

For example, this will show the unlock code for my phone:

    gpg --decrypt < ~/.password-store/phone-lock.gpg

You will need to adapt this to your purposes.

Other approaches

I am considering a more standard secret sharing scheme based on SLIP-039, established in the Bitcoin community, but that is applicable everywhere. The python-shamir-mnemonic implementation, for example, provides human-readable secrets:

anarcat@angela:~> shamir create 3of5
Using master secret: 608e920fc59a6cf2d23bcfe6cb889771
Group 1 of 1 - 3 of 5 shares required:
yield pecan academic acne body teacher elder twin detect vegan solution maiden home switch dryer member purple voice acquire username
yield pecan academic agree ajar cause critical leader admit viral taxi puny curious sled often satoshi lips afraid stadium froth
yield pecan academic amazing blanket decision crystal vexed trial fitness shaped timber helpful beard strategy curious episode sniff object heat
yield pecan academic arcade alcohol vampire employer package tactics extra window sympathy darkness adapt laundry genius laser closet example ruler
yield pecan academic axle aquatic have racism debris spew dive human thumb weapon satoshi curly lobe lecture visitor example alarm

Notice how the first three words are the same in all tokens? That's also useful to identify the secret itself...

Note that if you are comfortable sharing all your secret keys with those peers, a simpler procedure is to re-encrypt your own backup with a symmetric key instead of your Yubikey encryption key. This is much simpler:

gpg --decrypt $BACKUP_DIR/gnupg-backup.tar.pgp | gpg --symmetric - > anarcat-secrets.tar.pgp

Note that a possibly simpler approach to this would be to have an OpenPGP key generated from a passphrase, which itself would then be the shared secret. Software like passphrase2pgp can accomplished this but haven't been reviewed or tested. See also this blog post for background.

Pager playbook

Disaster recovery

Reference

Installation

SLA

Design

OpenPGP is standardized as RFC4880, which defines it as such:

OpenPGP software uses a combination of strong public-key and symmetric cryptography to provide security services for electronic communications and data storage.

The most common OpenPGP implementation is GnuPG, but there are others.

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker.

Maintainer, users, and upstream

Monitoring and testing

Logs and metrics

Backups

Other documentation

Discussion

Overview

Goals

Must have

Nice to have

Non-Goals

Approvals required

Proposed Solution

Cost

Alternatives considered

Expiration dates

Note that we set an expiration date on generated key. This is to protect against total loss of all backups and revocation certificates, not getting the key stolen, as the thief could extend the expiration key on their own.

This does imply that you'll need to renew your key every time the expiration date comes. I set a date in my planner and typically don't miss the renewals.

Separate certification key

Note that some guides favor separating the signing (S) subkey from the certification (C) key. In this guide, we keep the default which is to have both together. This is mostly because we use a YubiKey as storage and it only supports three key slots.

But even if there would be four, the point of having a separate certification key is that it can be stored offline. In my experience, this is risky: the key could be lost and will be less often used so memory of how do use it could be lost. Having an expiration date will help with this in the sense that the user will have to reuse the certification key regularly.

One approach could be to have a separate YubiKey for certification, stored offline and used only for renewals and third-party certifications.

Airgapped systems

In the key generation procedure, we do not explicitly say where the key should be generated, this is left as a precaution to the reader.

Some guides like drduh's guide says this:

To create cryptographic keys, a secure environment that can be reasonably assured to be free of adversarial control is recommended. Here is a general ranking of environments most to least likely to be compromised:

1. Daily-use operating system
2. Virtual machine on daily-use host OS (using virt-manager, VirtualBox, or VMware)
3. Separate hardened Debian or OpenBSD installation which can be dual booted
4. Live image, such as Debian Live or Tails
5. Secure hardware/firmware (Coreboot, Intel ME removed)
6. Dedicated air-gapped system with no networking capabilities

This guide recommends using a bootable "live" Debian Linux image to provide such an environment, however, depending on your threat model, you may want to take fewer or more steps to secure it.

This is good advice, but in our experience adding complexity to guides makes the user more likely to completely fail to follow the instructions altogether, at worst. At best, they will succeed, but could still trip on one tiny step that makes the whole scaffolding fall apart.

A strong focus on key generation also misses the elephant in the room which is that it's basically impossible to establish a trusted cryptographic system on a compromised host. Key generation is only one part in a long chain of operations that must happen on a device for the outputs to be trusted.

The above advice could be applied to your daily computing environment and, indeed, many people use environments like Qubes OS to improve their security.

See also just disconnect the internet for more in-depth critique of the rather broad "airgapped" concept.

About ECC (elliptic curve cryptography)

In the key generation procedures, we're going to generate an Elliptic Curve (ECC) key using Curve25519. It was chosen because the curve has been supported by OpenSSH since 2014 (6.5) and GnuPG since 2021 (2.1) and is the de-facto standard since the revelations surrounding possibly the back-doored NIST curves.

Some guides insist on still using RSA instead of ECC based on this post detailing problems with ECDSA. But that post explicitly says that:

Further, Ed25519, which is EdDSA over Curve25519, is designed to overcome the side-channel attacks that have targeted ECDSA, and it is currently being standardized by NIST.

... and that "ECDSA is fragile, but it is not broken".

ECC is faster than RSA, which is particularly important if cryptographic operations are shifted away from the powerful CPU towards a security key that is inherently slower.

ECC keys are also much smaller, which makes them easier to transfer and copy around. This is especially useful if you need to type down an SSH key on some weird console (which does happen to me surprisingly regularly).

Why GnuPG

A lot of OpenPGP's bad reputation comes from the particularly byzantine implementation that has become the ad-hoc reference implementation, GnuPG.

GnuPG's implementation of the OpenPGP standard is arcane, buggy, and sometimes downright insecure. It has bad defaults, a horrible user interface, the API is a questionable C library running on top of a nightmarish command-line file-descriptors based dialect, and will eat your cat if you don't watch it carefully. (Yes, I know, {{citation needed}}, you'll have to trust me on all of those for now, but I'm pretty sure I can generate a link for each one of those in time.)

Unfortunately, it's the only implementation that can fully support smart cards. So GnuPG it is for now.

Other OpenPGP implementations

Sequoia (Rust)

Sequoia, an alternative OpenPGP implementation written in Rust, has a much better user interface, security, and lots of promises.

It has a GnuPG backwards compatibility layer and a certificate store, but, as of June 2023, it doesn't have private key storage or smart card support.

Sequoia published (in 2022), a comparison with GnuPG that might be of interest and they maintain a comparison in the sq guide as well. They are working on both problems, see the issue 6 and openpgp-card crates.

Update (2024): the OpenPGP card work is progressing steadily. There's now a minimalist, proof-of-concept, ssh-agent implementation. It even supports notifying the user when a touch is required (!). The 0.10 release of the crate also supports signature generation, PIN prompting, and "file-based private key unlocking". Interestingly, this is actually a separate commandline interface from the sq binary in Sequoia, although it does use Sequoia as a library.

RNP (C++)

RNP is the C++ library the Mozilla Thunderbird mail client picked to implement native OpenPGP support. It's not backwards-compatible with GnuPG's key stores.

There's a drop in replacement for RNP by the Sequoia project called octopus which allows one to share the key store with GnuPG.

PGPainless (Java)

The other major OpenPGP library is PGPainless, written in Java, and mainly used on Android implementations.

Others

The OpenPGP.org site maintains a rather good list of OpenPGP implementations.

OpenPGP backups

Some guides propose various solutions for OpenPGP private key backups. drduh's guide, for example, suggests doing a paper backup, as per the Linux Kernel maintainer PGP guide.

Some people might prefer a LUKS-encrypted USB drive hidden under their bed, but I tend to distrust inert storage since it's known to lose data in the long term, especially when unused for a long time.

Full disk encryption is also highly specific to the operating system in use. It assumes a Linux user is around to decrypt a LUKS filesystem (and knows how as well). It also introduces another secret to share or remember.

I find that this is overkill: GnuPG keyring are encrypted with a passphrase, and that should be enough for most purposes.

Another approach is to backup your key on paper. Beware that this approach is time-consuming and exposes your private key to an attacker with physical access. The hand-written approach is also possibly questionable, as you basically need to learn typography for that purpose. The author, here, basically designs their own font, essentially.

• Software RAID
  • Replacing a drive
  • Building a new array
  • Assembling an existing array
• Hardware RAID
  • MegaCLI operation
    • Rebuilding the Debian package
  • References
• SMART monitoring
• Pager playbook
  • Failed disk
  • Disks with "other" errors
• Other documentation

Software RAID

Replacing a drive

If a drive fails in a server, the procedure is essentially to open a ticket, wait for the drive change, partition and re-add it to the RAID array. The following procedure assumes that sda failed and sdb is good in a RAID-1 array, but can vary with other RAID configurations or drive models.

1. file a ticket upstream

   Hetzner Support, for example, has an excellent service which asks you the disk serial number (available in the SMART email notification) and the SMART log (output of smartctl -x /dev/sda). Then they will turn off the machine, replace the disk, and start it up again.

2. wait for the server to return with the new disk

   Hetzner will send an email to the tpa alias when that is done.

3. partition the new drive (sda) to match the old (sdb):

   sfdisk -d /dev/sdb | sfdisk --no-reread /dev/sda --force
   

4. re-add the new disk to the RAID array:

   mdadm /dev/md0 -a /dev/sda
   

Note that Hetzner also has pretty good documentation on how to deal with SMART output.

Building a new array

Assume our new drives are /dev/sdc and /dev/sdd, and the highest array we have is md1, so we're creating a new md2 array:

1. Partition the drive. Easiest is to reuse an existing drive, as above:

   sfdisk -d /dev/sda | sfdisk --no-reread /dev/sdc --force
   sfdisk -d /dev/sda | sfdisk --no-reread /dev/sdd --force
   

   Or, for a fresh new drive in a different configuration, partition the whole drive by hand:

   for disk in /dev/sde /dev/sdd ; do
     parted -s $disk mklabel gpt &&
     parted -s $disk -a optimal mkpart primary 0% 100%
   done
   

2. Create a RAID-1 array:

   mdadm --create --verbose --level=1 --raid-devices=2 \
          /dev/md2 \
          /dev/sde1 /dev/sdd1
   

   Create a RAID-10 array with 6 drives:

    mdadm --create --verbose --level=10 --raid-devices=6 \
          /dev/md2 \
          /dev/sda1 \
          /dev/sdb1 \
          /dev/sdc1 \
          /dev/sdd1 \
          /dev/sde1 \
          /dev/sdf1
   

3. Setup full disk encryption:

    cryptsetup luksFormat /dev/md2 &&
    cryptsetup luksOpen /dev/md2 crypt_dev_md2 &&
    echo crypt_dev_md2 UUID=$(lsblk -n -o UUID /dev/md2 | head -1) none luks,discard | tee -a /etc/crypttab &&
    update-initramfs -u
   

   With an on-disk secret key:

    dd if=/dev/random bs=64 count=128 of=/etc/luks/crypt_dev_md2 &&
    chmod 0 /etc/luks/crypt_dev_md2 &&
    cryptsetup luksFormat --key-file=/etc/luks/crypt_dev_md2 /dev/md2 &&
    cryptsetup luksOpen --key-file=/etc/luks/crypt_dev_md2 /dev/md2 crypt_dev_md2 &&
    echo crypt_dev_md2 UUID=$(lsblk -n -o UUID /dev/md2 | head -1) /etc/luks/crypt_dev_md2 luks,discard | tee -a /etc/crypttab &&
    update-initramfs -u
   

4. Disable dm-crypt work queues (solid state devices only). If you've setup with an on-disk secret key you'll want to add --key-file /etc/luks/crypt_dev_md2 to the options:

    cryptsetup refresh --perf-no_read_workqueue --perf-no_write_workqueue --persistent crypt_dev_md2
   

From here, the array is ready for use in /dev/mapper/crypt_dev_md2. It will be resyncing for a while, you can see the status with:

watch -d cat /proc/mdstat

You can either use it as is with:

mkfs -t ext4 -j /dev/mapper/crypt_dev_md2

... or add it to LVM, see LVM docs. You should at least add it to the /etc/fstab file:

echo UUID=$(lsblk -n -o UUID /dev/mapper/crypt_dev_md2 | head -1) /srv ext4    rw,noatime,errors=remount-ro    0       2 >> /etc/fstab

Then you can test the configuration by unmounting/closing everything:

umount /srv
cryptsetup luksClose crypt_dev_md2

And restarting it again:

systemctl start systemd-cryptsetup@crypt_dev_md2.service srv.mount

Note that this doesn't test the RAID assembly. TODO: show how to disassemble the RAID array and tell systemd to reassemble it to test before reboot.

TODO: consider ditching fstab in favor of whatever systemd is smoking these days.

Assembling an existing array

This typically does the right thing:

mdadm --assemble --scan

Example run that finds two arrays:

# mdadm --assemble --scan
mdadm: /dev/md/0 has been started with 2 drives.
mdadm: /dev/md/2 has been started with 2 drives.

And of course, you can check the status with:

cat /proc/mdstat

Hardware RAID

Note: we do not have hardware RAID servers, nor do we want any in the future.

This documentation is kept only for historical reference, in case we end up with hardware RAID arrays again.

MegaCLI operation

Some TPO machines --particularly at cymru -- have hardware RAID with megaraid controllers. Those are controlled with the MegaCLI command that is ... rather hard to use.

First, alias the megacli command because the package (derived from the upstream RPM by Alien) installs it in a strange location:

alias megacli=/opt/MegaRAID/MegaCli/MegaCli

This will confirm you are using hardware raid:

root@moly:/home/anarcat# lspci | grep -i megaraid
05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)

This will show the RAID levels of each enclosure, for example this is RAID-10:

root@moly:/home/anarcat# megacli -LdPdInfo -aALL | grep "RAID Level"
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0

This is an example of a simple RAID-1 setup:

root@chi-node-04:~# megacli -LdPdInfo -aALL | grep "RAID Level"
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0

This lists a summary of all the disks, for example the first disk has failed here:

root@moly:/home/anarcat# megacli -PDList -aALL | grep -e '^Enclosure' -e '^Slot' -e '^PD' -e '^Firmware' -e '^Raw' -e '^Inquiry'
Enclosure Device ID: 252
Slot Number: 0
Enclosure position: 0
PD Type: SAS
Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Firmware state: Failed
Inquiry Data: SEAGATE ST3600057SS     [REDACTED]
Enclosure Device ID: 252
Slot Number: 1
Enclosure position: 0
PD Type: SAS
Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Firmware state: Online, Spun Up
Inquiry Data: SEAGATE ST3600057SS     [REDACTED]
Enclosure Device ID: 252
Slot Number: 2
Enclosure position: 0
PD Type: SAS
Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Firmware state: Online, Spun Up
Inquiry Data: SEAGATE ST3600057SS     [REDACTED]
Enclosure Device ID: 252
Slot Number: 3
Enclosure position: 0
PD Type: SAS
Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Firmware state: Online, Spun Up
Inquiry Data: SEAGATE ST3600057SS     [REDACTED]

This will make the drive blink (slot number 0 in enclosure 252):

megacli -PdLocate -start -physdrv[252:0] -aALL

Take the disk offline:

megacli -PDOffline -PhysDrv '[252:0]' -a0

Mark the disk as missing:

megacli -PDMarkMissing -PhysDrv '[252:0]' -a0

Prepare the disk for removal:

megacli -PDPrpRmv -PhysDrv '[252:0]' -a0

Reboot the machine, replace the disk, then inspect status again, you may see "Unconfigured(good)" as a status:

root@moly:~# megacli -PDList -aALL | grep -e '^Enclosure Device' -e '^Slot' -e '^Firmware' 
Enclosure Device ID: 252
Slot Number: 0
Firmware state: Unconfigured(good), Spun Up
[...]

Then you need to re-add the disk to the array:

megacli -PdReplaceMissing -PhysDrv[252:0] -Array0 -row0 -a0
megacli -PDRbld -Start -PhysDrv[252:0] -a0

Example output:

root@moly:~# megacli -PdReplaceMissing -PhysDrv[252:0] -Array0 -row0 -a0
                                     
Adapter: 0: Missing PD at Array 0, Row 0 is replaced.

Exit Code: 0x00
root@moly:~# megacli -PDRbld -Start -PhysDrv[252:0] -a0
                                     
Started rebuild progress on device(Encl-252 Slot-0)

Exit Code: 0x00

Then the rebuild should have started:

root@moly:~# megacli -PDList -aALL | grep -e '^Enclosure Device' -e '^Slot' -e '^Firmware' 
Enclosure Device ID: 252
Slot Number: 0
Firmware state: Rebuild
[...]

To follow progress:

watch /opt/MegaRAID/MegaCli/MegaCli64  -PDRbld -ShowProg -PhysDrv[252:0] -a0

Rebuilding the Debian package

The Debian package is based on a binary RPM provided by upstream (LSI corporation). Unfortunately, upstream was acquired by Broadcom in 2014, after which their MegaCLI software development seem to have stopped. Since then the lsi.com domain redirects to broadcom.com and those packages -- that were already hard to find -- are getting even harder to find.

It seems the broadcom search page is the best place to find the megaraid stuff. In that link you should get "search results" and under "Management Software and Tools" there should be a link to some "MegaCLI". The latest is currently (as of 2021) 5.5 P2 (dated 2014-01-19!). Note that this version number differs from the actual version number of the megacli binary (8.07.14). A direct link to the package is currently:

https://docs.broadcom.com/docs-and-downloads/raid-controllers/raid-controllers-common-files/8-07-14_MegaCLI.zip

Obviously, it seems like upstream does not mind breaking those links at any time, so you might have to redo the search to find it. In any case, the package is based on a RPM buried in the ZIP file. So this should get you a package:

unzip 8-07-14_MegaCLI.zip
fakeroot alien Linux/MegaCli-8.07.14-1.noarch.rpm

This gives you a megacli_8.07.14-2_all.deb package which normally gets upload to the proprietary archive on alberti.

An alternative is to use existing packages like the ones from le-vert.net. In particular, megactl is a free software alternative that works on chi-node-13, yet not packaged in Debian so currently not in use:

root@chi-node-13:~# megasasctl
a0       PERC 6/i Integrated      encl:1 ldrv:1  batt:good
a0d0       465GiB RAID 1   1x2  optimal
a0e32s0     465GiB  a0d0  online   errs: media:0  other:819
a0e32s1     465GiB  a0d0  online   errs: media:0  other:819

root@chi-node-13:~# megasasctl
a0       PERC 6/i Integrated      encl:1 ldrv:1  batt:good
a0d0       465GiB RAID 1   1x2  optimal
a0e32s0     465GiB  a0d0  online   errs: media:0  other:819
a0e32s1     465GiB  a0d0  online   errs: media:0  other:819

References

Here are some external documentation links regarding hardware RAID setups:

• https://cs.uwaterloo.ca/twiki/view/CF/MegaRaid
• https://raid.wiki.kernel.org/index.php/Hardware_Raid_Setup_using_MegaCli
• https://sysadmin.compxtreme.ro/how-to-replace-an-lsi-raid-disk-with-megacli/
• https://wikitech.wikimedia.org/wiki/MegaCli

SMART monitoring

Some servers will fail to properly detect disk drives in their SMART configuration. In particular, smartd does not support:

• virtual disks (e.g. /dev/nbd0)
• MMC block devices (e.g. /dev/mmcblk0, commonly found on ARM devices)
• out of the box, CCISS raid devices (e.g. /dev/cciss/c0d0)

The latter can be configured with the following snippet in /etc/smartd.conf:

#DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner
DEFAULT -n standby -m root -M exec /usr/share/smartmontools/smartd-runner
/dev/cciss/c0d0 -d cciss,0
/dev/cciss/c0d0 -d cciss,1
/dev/cciss/c0d0 -d cciss,2
/dev/cciss/c0d0 -d cciss,3
/dev/cciss/c0d0 -d cciss,4
/dev/cciss/c0d0 -d cciss,5

Notice how the DEVICESCAN is commented out to be replaced by the CCISS configuration. One line for each drive should be added (and no, it does not autodetect all drives unfortunately). This hack was deployed on listera which uses that hardware RAID.

Other hardware RAID controllers are better supported. For example, the megaraid controller on moly was correctly detected by smartd which accurately found a broken hard drive.

Pager playbook

Prometheus should be monitoring hardware RAID on servers that support it. This is normally auto-detected by the Prometheus node exporter.

NOTE: those instructions are out of date and need to be rewritten for Prometheus, see tpo/tpa/prometheus-alerts#16.

Failed disk

A normal RAID-1 Nagios check output looks like this:

OK: 0:0:RAID-1:2 drives:465.25GB:Optimal Drives:2

A failed RAID-10 check output looks like this:

CRITICAL: 0:0:RAID-10:4 drives:1.089TB:Degraded Drives:3

It actually has the numbers backwards: in the above situation, there was only one degraded drive, and 3 healthy ones. See above for how to restore a drive in a MegaRAID array.

Disks with "other" errors

The following warning may seem innocuous but actually reports that drives have "errors:

WARNING: 0:0:RAID-1:2 drives:465.25GB:Optimal Drives:2 (1530 Errors: 0 media, 0 predictive, 1530 other) 

The 1530 Errors part is the key here. They are "other" errors. This can be reproduced with the megacli command:

# megacli -PDList -aALL | grep -e '^Enclosure Device' -e '^Slot' -e '^Firmware' -e "Error Count"
Enclosure Device ID: 32
Slot Number: 0
Media Error Count: 0
Other Error Count: 765
Firmware state: Online, Spun Up
Enclosure Device ID: 32
Slot Number: 1
Media Error Count: 0
Other Error Count: 765
Firmware state: Online, Spun Up

The actual error should also be visible in the logs:

megacli -AdpEventLog -GetLatest 100 -f events.log -aALL

... then in events.log, the key part is:

Event Description: Unexpected sense: PD 00(e0x20/s0) Path 1221000000000000, CDB: 4d 00 4d 00 00 00 00 00 20 00, Sense: 5/24/00

The Sense field is Key Code Qualifier ("an error-code returned by a SCSI device") which, for 5/24/00 means "Illegal Request - invalid field in CDB (Command Descriptor Block) ". According to this discussion it seems that newer versions of the megacli binary trigger those errors when older drives are in use. Those errors can be safely ignored.

Other documentation

See also:

• LVM
• RAID wiki (archived)
• md(4) manual page
• mdadm(8) manual page
• md driver kernel documentation

• Reboots
  • Full fleet reboot
    • Testing reboots
    • Rebooting Ganeti nodes
    • Remaining nodes
  • Rebooting a single host
  • Batch rebooting multiple hosts
  • Userland reboots
  • Notifying users

Reboots

Sometimes it is necessary to perform a reboot on the hosts, when the kernel is updated. Prometheus will warn about this with the NeedsReboot alert, which looks like:

Servers running trixie needs to reboot

Sometimes a newer kernel can have been released between the last apt update and apt metrics refresh. So before running reboots, make sure all servers are up to date and have the latest kernel downloaded:

cumin '*' 'apt-get update && unattended-upgrades -v && systemctl start tpa-needrestart-prometheus-metrics.service'

Note that the above triggers an update of metrics for prometheus but you'll need them to get polled before the list of hosts from the fab command below is fully up to date, so wait for 1 minute or two before launching that command to get the full list of hosts.

You can see the list of pending reboots with this Fabric task:

fab fleet.pending-reboots

See below for how to handle specific situations.

Full fleet reboot

This is the most likely scenario, especially when we were able to upgrade all of the servers to the same, stable, release of debian.

In this case, the faster way to run reboots is to reboot ganeti nodes with all of their contained instances in order to clear out reboots for many servers at once, then reboot the hosts that are not in ganeti.

The fleet.reboot-fleet command will tell you whether it's worth it, and might eventually be able to orchestrate the entire reboot on its own. For now, this reboot is only partly automated.

Note that to make the reboots run more smoothly, you can temporarily modify your yubikey touch policy to remove the need to always confirm by touching the key.

So, typically, you'd do a Ganeti fleet reboot, then reboot remaining nodes. See below.

Testing reboots

A good reflex is to test rebooting a single "canary" host as a test:

fab -H idle-fsn-01.torproject.org fleet.reboot-host

Rebooting Ganeti nodes

See the Ganeti reboot procedures for this procedure. Essentially, you run those two batches in parallel, paying close attention to the host list:

• gnt-dal cluster:

   fab -H dal-node-03.torproject.org,dal-node-02.torproject.org,dal-node-01.torproject.org fleet.reboot-host --no-ganeti-migrate
  

• gnt-fsn cluster:

   fab -H fsn-node-08.torproject.org,fsn-node-07.torproject.org,fsn-node-06.torproject.org,fsn-node-05.torproject.org,fsn-node-04.torproject.org,fsn-node-03.torproject.org,fsn-node-02.torproject.org,fsn-node-01.torproject.org fleet.reboot-host --no-ganeti-migrate
  

You want to avoid rebooting mirrors at once. Ideally, the fleet.reboot-fleet script here would handle this for you, but it doesn't right now. This can be done ad-hoc: reboot the host, and pay attention to which instances are rebooted. If too many mirrors are rebooted at once, you can abort the reboot before the timeout (control-c) and cancel the reboot by rerunning the reboot-host command with the --kind cancel flag.

Note that the above assumes only two clusters are present, the host list might have changed since this documentation was written.

Remaining nodes

The Karma alert dashboard will show remaining hosts that might have been missed by the above procedure after a while, but you can already get ahead of that by detecting physical hosts that are not covered by the Ganeti reboots with:

curl -s -G http://localhost:6785/pdb/query/v4         --data-urlencode 'query=inventory[certname]         { facts.virtual = "physical" }'         | jq -r '.[].certname' | grep -v -- -node- | sort

The above assumes you have the local "Cumin hack" to forward port 6785 to PuppetDB's localhost:8080 automatically, use this otherwise:

ssh -n -L 6785:localhost:8080 puppetdb-01.torproject.org &

You can also look for the virtual machines outside of Ganeti clusters:

ssh db.torproject.org \
    "ldapsearch -H ldap://db.torproject.org -x -ZZ -b 'ou=hosts,dc=torproject,dc=org' \
    '(|(physicalHost=hetzner-cloud)(physicalHost=safespring))' hostname \
    | grep ^hostname | sed 's/hostname: //'"

You can list both with this LDAP query:

ssh db.torproject.org 'ldapsearch -H ldap://db.torproject.org -x -ZZ -b "ou=hosts,dc=torproject,dc=org" "(!(physicalHost=gnt-*))" hostname' | sed -n '/hostname/{s/hostname: //;p}' | grep -v ".*-node-[0-9]\+\|^#" | paste -sd ','

This, for example, will reboot all of those hosts in series:

fab -H $(ssh db.torproject.org 'ldapsearch -H ldap://db.torproject.org -x -ZZ -b "ou=hosts,dc=torproject,dc=org" "(!(physicalHost=gnt-*))" hostname' | sed -n '/hostname/{s/hostname: //;p}' | grep -v ".*-node-[0-9]\+\|^#" | paste -sd ',') fleet.reboot-host

We show how to lists those hosts separately because you can also reboot a select number of hosts in parallel with the fleet.reboot-parallel command, and then you need to think more about which hosts to reboot than when you do a normal, serial reboot.

Do not reboot the entire fleet or all hosts blindly with the reboot-parallel method, as it can be pretty confusing, especially with a large number of hosts, as all the output is shown in parallel. It will also possibly reboot multiple components that are redundant mirrors, which we try to avoid.

The reboot-parallel command works a little differently than other reboot commands because the instances are passed as an argument. Here are two examples:

fab fleet.reboot-parallel --instances ci-runner-x86-14.torproject.org,tb-build-03.torproject.org,dal-rescue-01.torproject.org,cdn-backend-sunet-02.torproject.org,hetzner-nbg1-01.torproject.org

Here, the above is safe because there's only a handful (5) of servers and they don't have overlapping tasks (they're not mirrors of each other).

Rebooting a single host

If this is only a virtual machine, and the only one affected, it can be rebooted directly. This can be done with the fabric-tasks task fleet.reboot-host:

fab -H test-01.torproject.org,test-02.torproject.org fleet.reboot-host

By default, the script will wait 2 minutes before hosts: that should be changed to 30 minutes if the hosts are part of a mirror network to give the monitoring systems (mini-nag) time to rotate the hosts in and out of DNS:

fab -H mirror-01.torproject.org,mirror-02.torproject.org fleet.reboot-host --delay-hosts 1800

If the host has an encrypted filesystem and is hooked up with Mandos, it will return automatically. Otherwise it might need a password to be entered at boot time, either through the initramfs (if it has the profile::fde class in Puppet) or manually, after the boot. That is the case for the mandos-01 server itself, for example, as it currently can't unlock itself, naturally.

Note that you can cancel a reboot with --kind=cancel. This also cascades down Ganeti nodes.

Batch rebooting multiple hosts

NOTE: this section has somewhat bit-rotten. It's kept only to document the rebootPolicy but, in general, you should do a fleet-wide reboot or single-host reboots.

IMPORTANT: before following this procedure, make sure that only a subset of the hosts need a restart. If all hosts need a reboot, it's likely going to be faster and easier to reboot the entire clusters at once, see the Ganeti reboot procedures instead.

NOTE: Reboots will tend to stop for user confirmation whenever packages get upgraded just before the reboot. To prevent the process from waiting for your manual input, it is suggested that upgrades are run first, using cumin. See how to run upgrades in the section above.

LDAP hosts have information about how they can be rebooted, in the rebootPolicy field. Here are what the various fields mean:

• justdoit - can be rebooted any time, with a 10 minute delay, possibly in parallel
• rotation - part of a cluster where each machine needs to be rebooted one at a time, with a 30 minute delay for DNS to update
• manual - needs to be done by hand or with a special tool (fabric in case of ganeti, reboot-host in the case of KVM, nothing for windows boxes)

Therefore, it's possible to selectively reboot some of those hosts in batches. Again, this is pretty rare: typically, you would either reboot only a single host or all hosts, in which case a cluster-wide reboot (with Ganeti, below) would be more appropriate.

This routine should be able to reboot all hosts with a rebootPolicy defined to justdoit or rotation:

echo "rebooting 'justdoit' hosts with a 10-minute delay, every 2 minutes...."
fab -H $(ssh db.torproject.org 'ldapsearch -H ldap://db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=justdoit)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R' | paste -sd ',') fleet.reboot-host --delay-shutdown-minutes=10 --delay-hosts-seconds=120

echo "rebooting 'rotation' hosts with a 10-minute delay, every 30 minutes...."
fab -H $(ssh db.torproject.org 'ldapsearch -H ldap://db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=rotation)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R' | paste -sd ',') fleet.reboot-host --delay-shutdown-minutes=10 --delay-hosts-seconds=1800

Another example, this will reboot all hosts running Debian bookworm, in random order:

fab -H $(ssh puppetdb-01.torproject.org "curl -s -G http://localhost:8080/pdb/query/v4 --data-urlencode 'query=inventory[certname] { facts.os.distro.codename = \"bookworm\" }'" | jq -r '.[].certname' | sort -R | paste -sd ',') fleet.reboot-host

And this will reboot all hosts with a pending kernel upgrade (updates only when puppet agent runs), again in random order:

fab -H $(ssh puppetdb-01.torproject.org "curl -s -G http://localhost:8080/pdb/query/v4 --data-urlencode 'query=inventory[certname] { facts.apt_reboot_required = true }'" | jq -r '.[].certname' | sort -R | paste -sd ',') fleet.reboot-host

And this is the list of all physical hosts with a pending upgrade, alphabetically:

fab -H $(ssh puppetdb-01.torproject.org "curl -s -G http://localhost:8080/pdb/query/v4 --data-urlencode 'query=inventory[certname] { facts.apt_reboot_required = true and facts.virtual = \"physical\" }'" | jq -r '.[].certname'  | sort | paste -sd ',') fleet.reboot-host

Userland reboots

systemd 254 (Debian 13 trixie and above) has a special command:

systemctl soft-reboot

That will "shut down and reboot userspace". As the manual page explains:

systemd-soft-reboot.service is a system service that is pulled in by soft-reboot.target and is responsible for performing a userspace-only reboot operation. When invoked, it will send the SIGTERM signal to any processes left running (but does not follow up with SIGKILL, and does not wait for the processes to exit). If the /run/nextroot/ directory exists (which may be a regular directory, a directory mount point or a symlink to either) then it will switch the file system root to it. It then reexecutes the service manager off the (possibly now new) root file system, which will enqueue a new boot transaction as in a normal reboot.

This can therefore be used to fix conditions where systemd itself needs to be restarted, or a lot of processes need to, but not the kernel.

This has not been tested, but could speed up some restart conditions.

Notifying users

Users should be notified when rebooting hosts. Normally, the shutdown(1) command noisily prints warnings on terminals which will give a heads up to connected users, but many services do not rely on interactive terminals. It is therefore important to notify users over our chat rooms (currently IRC).

The reboot script can send notifications when rebooting hosts. For that, credentials must be supplied, either through the HTTP_USER and HTTP_PASSWORD environment, or (preferably) through a ~/.netrc file. The file should look something like this:

machine kgb-bot.torproject.org login TPA password REDACTED

The password (REDACTED in the above line) is available on the bot host (currently chives) in /etc/kgb-bot/kgb.conf.d/client-repo-TPA.conf or in trocla, with the profile::kgb_bot::repo::TPA.

To confirm this works before running reboots, you should run this fabric task directly:

fab kgb.relay "test"

For example:

anarcat@angela:fabric-tasks$ fab kgb.relay "mic check"
INFO: mic check

... should result in:

16:16:26 <KGB-TPA> mic check

When rebooting, the users will see this in the #tor-admin channel:

13:13:56 <KGB-TPA> scheduled reboot on host web-fsn-02.torproject.org in 10 minutes
13:24:56 <KGB-TPA> host web-fsn-02.torproject.org rebooted

A heads up should be (manually) relayed in the #tor-project channel, inviting users to follow that progress in #tor-admin.

Ideally, we would have a map of where each server should send notifications. For example, the tb-build-* servers should notify #tor-browser-dev. This would require a rather more convoluted configuration, as each KGB "account" is bound to a single channel for the moment...

• How to
  • Remove host from Puppet
  • Change the hostname
  • Rename the machine in the infrastructure
    • Ganeti
    • LDAP
    • Mandos
    • DNS
    • DNSWL
    • External hoster platform
  • Re-bootstrap Puppet
  • Schedule backups removal
  • Adjust documentation

How to

This page contains the procedure to rename a host. It hasn't been tested very much, so proceed with caution.

Remove host from Puppet

Start by stopping the puppet-run timer and disabling Puppet on the machine:

systemctl stop puppet.timer && \
puppet agent --disable "renaming in progress"

Then, in tor-puppet, remove references to the host. At the very least the node's classification yaml should be removed from tor-puppet-hiera-enc.git/nodes.

Revoke its certificates from the Puppet server using the retirement script:

fab -H foo.torproject.org retire.revoke-puppet

Change the hostname

On the host being renamed, change the hostname:

hostnamectl set-hostname bar.torproject.org && \
sed -i 's/foo/bar/g' /etc/hosts

Then adjust the SSH host keys. Generating new keys isn't mandatory:

sed -i 's/foo/bar/' /etc/ssh/ssh_host_*.pub

We also need to fix the thishost symlink in ud-ldap data:

ud-replicate
cd /var/lib/misc && ln -sf bar.torproject.org thishost
rm -rf foo.torproject.org

Rename the machine in the infrastructure

Ganeti

ganeti-instance rename foo.torproject.org bar.torproject.org

LDAP

Run a search/replace with the old and new hostname in the host's stanza.

Mandos

We need to let the mandos server know about the new hostname:

sed -i 's/foo/bar/' /etc/mandos/clients.conf && \
systemctl restart mandos.service

DNS

Both forward and reverse DNS should be adjusted to use the new hostname.

DNSWL

External hoster platform

If the host is a machine host at Hetzner or another provider, the name should be changed there as well.

Re-bootstrap Puppet

Now the host is ready to be added back to Puppet. A new certificate will be generated in this step. This should be ran from your computer, in Fabric:

fab -H bar.torproject.org puppet.enable
fab -H bar.torproject.org puppet.bootstrap-client

Schedule backups removal

This will schedule the removal of backups under the old hostname:

fab -H foo.torproject.org retire.remove-backups

Adjust documentation

Adjust documentation that may refer to the old hostname, including the tor-passwords, the wiki and the Tor "VM Hosts" spreadsheet.

• Requirements
• Main procedure
• GitLab
• Nextcloud
• Other

This document explains how to handle requests to rename a user account.

Requirements

• the new LDAP username
• the new "full name"
• a new or updated GPG key with the new email
• a new mail forwarding address, if needed

Main procedure

1. Update account-keyring.git with the new (or updated) GPG key

2. With ldapvi, update the user and group names in the LDAP database (including the DN), along with the new GPG fingerprint if a new key is to be associated with the account and forwarding address if applicable

3. Using cumin, rename home directories on hosts

4. Optionally, add the previous forwarding to profile::mx::aliases in tor-puppet:data/common/mail.yaml

5. Update the information on the main website

GitLab

GitLab users may rename their own accounts with the User Settings panel.

Nextcloud

Changing the login name is not supported at all in Nextcloud, only the display name can be changed.

If a new account is created as part or the renaming process, it's possible to "transfer" files and shares from one account to the other using the files:transfer-ownership command via the CLI. This particular option is however untested, and TPA doesn't have access to the hosted Nextcloud CLI.

Other

It's a good idea to grep the tor-puppet.git repository, this can catch instances of the old username existing in places like /etc/subuid.

Decommissioning a host

Note that this procedure is relevant only to TPA hosts. For Tails hosts, follow the Tails server decommission procedure, which should eventually be merged here.

Retirement checklist to copy-paste in retirement tickets:

• announcement
• retire the host in fabric
• remove from LDAP with ldapvi
• power-grep
• remove from tor-passwords
• remove from DNSwl
• remove from docs
  • wiki pages
  • nextcloud server list if not a VM
  • if an entire service is taken offline with the machine, remove the service page and links to it
• remove from racks
• remove from reverse DNS
• notify accounting if needed

The detailed procedure:

1. long before (weeks or months) the machine is retired, make sure users are aware it will go away and of its replacement services

2. retire the host from its parent, backups and Puppet. Before launching retirement you will need to know:

   • for a ganeti instance, the ganeti parent (primary) host
   • the backup storage server. if the machine in in the fsn cluster, then backup-storage-01.torproject.org. Otherwise, bungei.torproject.org

   for example:

   fab -H $INSTANCE retire.retire-all --parent-host=$PARENT_HOST --backup-host=$BACKUP_HOST
   

   Copy the output of the script in the retirement ticket. Adjust delay for more sensitive hosts with:

   --retirement-delay-vm=30 --retirement-delay-backups=90
   

   Above is 30 days for the destruction of disks, 90 for backups. Default is 7 days for disks, 30 for backups.

   TODO: $PARENT_HOST should be some ganeti node (e.g. fsn-node-01.torproject.org) but could be auto-detected...

   TODO: the backup storage host could be auto-detected

   TODO: cover physical machines

3. remove from LDAP with ldapvi (STEP 6 above), copy-paste it in the ticket

4. do one huge power-grep and find over all our source code, for example with unifolium that was:

   grep -nHr --exclude-dir .git -e 148.251.180.115 -e 2a01:4f8:211:6e8::2 -e kvm2.torproject.org  -e unifolium.torproject.org -e unifolium -e kvm2
   find -iname unifolium\*
   

   TODO: extract those values from LDAP (e.g. purpose) and run the grep in Fabric

5. remove from tor-passwords (TODO: put in fabric). magic command (not great):

   pass rm root/unifolium.torproject.org
   # look for traces of the host elsewhere
   for f in ~/.password-store/*/*; do
       if gpg -d < $f 2>/dev/null | \
           grep -i -e 148.251.180.115 -e 2a01:4f8:211:6e8::2 -e kvm2.torproject.org -e unifolium.torproject.org -e unifolium -e kvm2 
       then
           echo match found in $f
       fi
   done
   

6. remove from DNSwl

7. remove from the machine from this wiki (if present in documentation), the Nextcloud spreadsheet (if it is not in Ganeti), and, if it's an entire service, the services page

8. if it's a physical machine or a virtual host we don't control, schedule removal from racks or hosts with upstream

9. remove from reverse DNS

10. If retiring the machine took out a recurring expense (e.g. physical machines, cloud hosting), contact accounting to tell them about the expected change.

Wiping disks

To wipe disks on servers without a serial console or management interface, you need to be a little more creative. We do this with the nwipe(1) command, which should be installed before anything:

apt install nwipe vmtouch

Run in a screen:

screen

If there's a RAID array, first wipe one of the disks by taking it offline and writing garbage:

mdadm --fail /dev/md0 /dev/sdb1 &&
mdadm --remove /dev/md0 /dev/sdb1 &&
mdadm --fail /dev/md1 /dev/sdb2 &&
mdadm --remove /dev/md1 /dev/sdb2 &&
: etc, for the other RAID elements in /proc/mdstat &&
nwipe --autonuke --method=random --verify=off /dev/sdb

This will take a long time. Note that it will start a GUI which is useful because it will give you timing estimates, which the command-line version does not provide.

WARNING: this procedure doesn't cover the case where the disk is an SSD. See this paper for details on how classic data scrubbing software might not work for SSDs. For now we use this:

nwipe --autonuke --method=random --rounds=2 --verify=off /dev/nvme1n1

TODO: consider hdparm and the "secure erase" procedure for SSDs:

hdparm --user-master u --security-set-pass Eins /dev/sdc
time hdparm --user-master u --security-erase Eins /dev/sdc

See also stressant documentation abnout this.

When you return:

1. start a screen session with a static busybox as your SHELL that will survive disk wiping:

   # make sure /tmp is on a tmpfs first!
   cp -av /root /tmp/root &&
   mount -o bind /tmp/root /root &&
   cp /bin/busybox /tmp/root/sh &&
   export SHELL=/tmp/root/sh &&
   exec screen -s $SHELL
   

2. lock down busybox and screen in memory

   vmtouch -dl /usr/bin/screen /bin/busybox /tmp/root/sh /usr/sbin/nwipe
   

   TODO: the above aims at making busybox survive the destruction, so that it's cached in RAM. It's unclear if that actually works, because typically SSH is also busted and needs a lot more to bootstrap, so we can't log back in if we lose the console. Ideally, we'd run this in a serial console that would have more reliable access... See also vmtouch.

3. kill all processes but the SSH daemon, your SSH connection and shell. this will vary from machine to machine, but a good way is to list all processes with systemctl status and systemctl stop the services one by one. Hint: multiple services can be passed on the same stop command, for example:

   systemctl stop \
       acpid \
       acpid.path \
       acpid.socket \
       apache2 \
       atd \
       bacula-fd \
       bind9 \
       cron \
       dbus \
       dbus.socket \
       fail2ban \
       ganeti \
       haveged \
       irqbalance \
       ipsec \
       iscsid \
       libvirtd \
       lvm2-lvmetad.service \
       lvm2-lvmetad.socket \
       mdmonitor \
       multipathd.service \
       multipathd.socket \
       ntp \
       openvswitch-switch \
       postfix \
       prometheus-bind-exporter \
       prometheus-node-exporter \
       smartd \
       strongswan \
       syslog-ng.service \
       systemd-journald \
       systemd-journald-audit.socket \
       systemd-journald-dev-log.socket \
       systemd-journald.socket \
       systemd-logind.service \
       systemd-udevd \
       systemd-udevd \
       systemd-udevd-control.socket \
       systemd-udevd-control.socket \
       systemd-udevd-kernel.socket \
       systemd-udevd-kernel.socket \
       timers.target \
       ulogd2 \
       unbound \
       virtlogd \
       virtlogd.socket \
   

4. disable swap:

   swapoff -a
   

5. un-mount everything that can be unmounted (except /proc):

   umount -a
   

6. remount everything else read-only:

   mount -o remount,ro /
   

7. sync disks:

   sync
   

8. wipe the remaining disk and shutdown:

   # hit control-a control-g to enable the bell in screen
   wipefs -af /dev/noop3 &&
   wipefs -af /dev/noop && \
   nwipe --autonuke --method=random --rounds=2 --verify=off /dev/noop ; \
   printf "SHUTTING DOWN FOREVER IN ONE MINUTE\a\n" ; \
   sleep 60 ; \
   echo o > /proc/sysrq-trigger ; \
   sleep 60 ; \
   echo b > /proc/sysrq-trigger ; \
   

   Note: as a safety precaution, the above device has been replaced by noop, that should be (say) sda instead.

A few tricks if nothing works in the shell which might work in a case of an emergency:

• cat PATH can be expressed as mapfile -C "printf %s" < PATH in bash
• echo * can be used as a rough approximation of ls

Deprecated manual procedure

Warning: this procedure is difficult to follow and error-prone. A new procedure was established in Fabric, above. It should really just be completely avoided.

1. long before (weeks or months) the machine is retired, make sure users are aware it will go away and of its replacement services

2. if applicable, stop the VM in advance:

   • If the VM is on a KVM host: virsh shutdown $host, or at least stop the primary service on the machine

   • If the machine is on ganeti: gnt-instance stop $host

3. On KVM hosts, undefine the VM: virsh undefine $host

4. wipe host data, possibly with a delay:

   • On some KVM hosts, remove the LVM logical volumes:

     echo 'lvremove -y vgname/lvname' | at now + 7 days
     

     Use lvs will list the logical volumes on the machine.

   • Other KVM hosts use file-backed storage:

     echo 'rm -r /srv/vmstore/gayi.torproject.org/' | at now + 7 days
     

   • On Ganeti hosts, remove the actual instance with a delay, from the Ganeti master:

     echo "gnt-instance remove $host" | at now + 7 days
     

   • for a normal machine or a machine we do not own the parent host for, wipe the disks using the method described below

5. remove it from LDAP: the host entry and any @<host> group memberships there might be as well as any sudo passwords users might have configured for that host

6. if it has any associated records in tor-dns/domains or auto-dns, or upstream's reverse dns thing, remove it from there too. e.g.

   grep -r -e build-x86-07 -e 78.47.38.230 -e 2a01:4f8:211:6e8:0:823:6:1
   

   ... and check upstream reverse DNS.

7. on the puppet server (pauli): read host ; puppet node clean $host.torproject.org && puppet node deactivate $host.torproject.org TODO: That procedure is incomplete, use the retire.revoke-puppet job in fabric instead.

8. grep the tor-puppet repository for the host (and maybe its IP addresses) and clean up; also look for files with hostname in their name

9. clean host from tor-passwords

10. remove any certs and backup keys from letsencrypt-domains.git and letsencrypt-domains/backup-keys.git repositories that are no longer relevant:

    git -C letsencrypt-domains grep -e $host -e storm.torproject.org
    # remove entries found above
    git -C letsencrypt-domains commit
    git -C letsencrypt-domains push
    find letsencrypt-domains/backup-keys -name "$host.torproject.org" -o -name 'storm.torproject.org*' -delete
    git -C letsencrypt-domains/backup-keys commit
    git -C letsencrypt-domains/backup-keys push
    

    Also clean up the relevant files on the letsencrypt master (currently nevii), for example:

    ssh nevii rm -rf /srv/letsencrypt.torproject.org/var/certs/storm.torproject.org
    ssh nevii find /srv/letsencrypt.torproject.org/ -name 'storm.torproject.org.*' -delete
    

11. if the machine is handling mail, remove it from dnswl.org (password in tor-passwords, hosts-extra-info) - consider that it can take a long time (weeks? months?) to be able to "re-add" an IP address in that service, so if that IP can eventually be reused, it might be better to keep it there in the short term

12. schedule a removal of the host's backup, on the backup server (currently bungei):

    cd  /srv/backups/bacula/
    mv $host.torproject.org $host.torproject.org-OLD
    echo rm -rf /srv/backups/bacula/$host.torproject.org.OLD/ | at now + 30 days
    

13. remove from the machine from this wiki (if present in documentation), the Nextcloud spreadsheet (if it is not in ganeti), and, if it's an entire service, the services page

14. if it's a physical machine or a virtual host we don't control, schedule removal from racks or hosts with upstream

15. after 30 days delay, retire from Bacula catalog, on the director (currently bacula-director-01), run bconsole then:

    delete client=$INSTANCE-fd

    for example:

    delete client=archeotrichon.torproject.org-fd

16. after 30 days delay, remove PostgreSQL backups on the storage server (currently /srv/backups/pg on bungi), if relevant

"Retiring" a user can actually mean two things:

• "retired", which disables their access to Tor hosts but keeps email working and then automatically stops after 186 days

• "disabled", which immediately disables everything

At least, that's in theory: in practice, the userdir-ldap code seems to just immediately disable a user when we "lock" it, so that distinction doesn't actually exist and it is unclear where the above actually is coming from.

Note that this documentation is incomplete. Our user management procedures are poorly documented (tpo/tpa/team#40129) and authentication is rather messy as of 2025. TPA-RFC-86 was designed to improve this.

How to retire a user

Typically, the first step in retiring a user is to "lock" their user account, which keeps them from logging in. But the user still lives in the LDAP database, and it might be better to delete it completely.

The user also needs to be checked against all other services that might have their own account database.

Locking an account

So the first step is to lock the account (as in service/ldap):

ssh db.torproject.org ud-lock account

A ticket number can be provided with -r and another state (than "retired") can be specified with -s, for example:

ud-lock -r 'tpo/tpa/team#666' -s inactive account

Note that this only keeps the user from accessing servers, it does not remove the actual account from LDAP nor does it remove it from the passwd database on servers. This is because the user might still own files and we do not want to have files un-owned.

It also does not remove the email alias (the emailForward field in LDAP), for that you need to delete the account altogether.

Deleting an account

You may also want to delete the user and all of its group memberships if it's clear they are unlikely to come back again. For this, the actual LDAP entries for the user must be removed with ldapvi, but only after the files for that user have been destroyed or given to another user.

Note that it's unclear if we should add an email alias in the virtual file when the account expires, see ticket #32558 for details.

Retiring from other services

Then you need to go through the service list and pay close attention to the services that have "authentication" enabled in the list.

In particular, you will want to:

1. Login as admin to GitLab, disable the user account, and remove them from critical groups. another option is to block or ban a user as well.
2. Remove the user from aliases in the virtual alias map (modules/postfix/files/virtual in tor-puppet.git)
3. remove the user from mailing lists, visit https://lists.torproject.org/mailman3/postorius/users.
4. grep for the username in tor-puppet.git, typically you may find a sudo entry
5. remove the key from acccount-keyring.git

There are other manual accounts that are not handled by LDAP. Make sure you check:

• Nextcloud
• Gitolite
• Gitlab
• Github

The service list is the canonical reference for this. The membership-retirements-do-nothing.py from fabric-tasks should be used to go through the list.

How to un-retire a user

To reverse the above, if the user was just "locked", you might be able to re-enable it by doing the following:

• delete the accountStatus, shadowExpire fields
• add the keyFingerprint field matching the (trusted) fingerprint (from account-keyring.git)
• change the user's password to something that is not locked

To set a password, you need to find a way to generate a salted UNIX hashed password, and there are many ways to do that, but if you have a copy of the userdir-ldap source code lying around, this could just do it:

>>> from userdir_ldap.ldap import HashPass, GenPass
>>> print("{crypt}" + HashPass(GenPass()))

If the user was completely deleted from the LDAP database, you need to restore those LDAP fields the way they were before. You can do this by either restoring from the LDAP database (no, that is not fun at all -- be careful to avoid duplicate fields when you re-add them in ldapvi) OR just create a new user.

Time is a complicated concept and it's hard to implement properly in computers.

This page aims at documenting some bits that we have to deal with in TPA.

Daylight saving handling

For some reason, anarcat ended up with the role of "time lord" which consists of sending hilarious reminders describing the chaos that, bi-yearly, we inflict upon ourselves by going through the daylight saving time change routine.

This used to be done as a one-off, but people really like those announcements, so we're trying to make those systematic.

For that purpose, a calendar was created in Nextcloud. First attempts at creating the calendar by hand through the web interface failed. It's unclear why: events would disappear, the end date would shift by a day. I suspect Nextcloud has lots of issues dealing with the uncertain time during which daylight savings occur, particularly when managing events.

So a calendar was crafted, by hand, using a text editor, and stored in time/dst.ics. It was imported in Nextcloud under the Daylight saving times calendar, but perhaps it would be best to add it as a web calendar. Unfortunately, that might not make it visible (does it?) to everyone, so it seems better that way. The calendar was shared with TPI.

Future changes to the calendar will be problematic: perhaps NC will deal with duplicate events and a new ICS can just be imported as is?

The following documentation was consulted to figure things out:

• iCalendar on Wikipedia
• RFC5545 ("Internet Calendaring and Scheduling Core Object Specification (iCalendar)"), which is massive, so I also used the icalendar.org HTML version in particular:
  • Recurrence rules and the rule generator which didn't actually cover the cases we needed ("last Sunday of the month")
  • 3.3.5 Date time
  • 3.8.4.7. Unique Identifier
• the iCalendar.org validator must be used before uploading new files, it's easy to make mistakes

Curses found

Doing this research showed a number of cursed knowledge in the iCal specification:

• if you specify a timezone to an event, you need to ship the entire timezone data inside the .ICS file, including offsets, daylight savings, etc. this effectively makes any calendar file created with a local time eventually outdated if time zone rules change for that zone (and they do), see 3.8.3.1. Time Zone Identifier and 3.6.5. Time Zone Component
• blank lines are not allowed in ICS files, it would make it too readable
• events can have SUMMARY, DESCRIPTION and COMMENT fields, the latter two which look strikingly similar and 3.8.1.4. Comment
• alarms are defined using ISO8601 durations which are actually not defined in a IETF standard, making iCal not fully referenced inside IETF documents
• events MUST have a 3.8.7.2. Date-Time Stamp (DTSTAMP) field that is the "date and time that the instance of the iCalendar object was created" or "last revised" that may differ (or not!) from 3.8.7.1. Date-Time Created (CREATED) and 3.8.7.3. Last Modified (LAST-MODIFIED) depending on the 3.7.2. Method (METHOD), we've elected to use only DTSTAMP since it's mandatory (and the others don't seem to be)

This page documents how upgrades are performed across the fleet in the Tor project. Typically, we're talking about Debian package upgrades, both routine and major upgrades. Service-specific upgrades notes are in their own service, in the "Upgrades" section.

Note that reboot procedures have been moved to a separate page, in the reboot documentation.

• Major upgrades
  • Team-specific upgrade policies
  • All time version graph
• Minor upgrades
  • Unattended upgrades
  • Blocked upgrades
  • Obsolete packages
  • Out of date package lists
  • Manual upgrades with Cumin
• Special cases and manual restarts
  • cron.service
    • ud-replicate special case
  • systemd user manager services
  • Ganeti
  • Open vSwitch
  • Grub
  • user@ services
• Reboots

Major upgrades

Major upgrades are done by hand, with a "cheat sheet" created for each major release. Here are the currently documented ones:

• Debian 13, trixie
• Debian 12, bookworm
• Debian 11, bullseye
• Debian 10, buster

Upgrades have been automated using Fabric, but that could also have been done through Puppet Bolt, Ansible, or be built into Debian, see AutomedUpgrade in the Debian Wiki.

Team-specific upgrade policies

Before we perform a major upgrade, it might be advisable to consult with the team working on the box to see if it will interfere for their work. Some teams might block if they believe the major upgrade will break their service. They are not allowed to indefinitely block the upgrade, however.

Team policies:

• anti-censorship: TBD
• metrics: one or two work-day advance notice (source)
• funding: schedule a maintenance window
• git: TBD
• gitlab: TBD
• translation: TBD

Some teams might be missing from the list.

All time version graph

The above graph currently covers 5 different releases:

We can also count the stretches of time we had to support multiple releases at once: