LDAP is a directory service we use to inventory the users, groups, passwords, (some) email forwards and machines. It distributes some configuration and password files to all machines and can reload services.

Note that this documentation needs work, particularly regarding user management procedures, see issue 40129.

• Tutorial
  • Getting to know LDAP
  • How to change my email forward?
  • Password reset
  • How do I update my OpenPGP key?
• How-to
  • Set a sudo password
  • Operate the mail gateway
    • ping
    • New Password
    • Changes
    • Notes
  • Uploading a SSH user key
  • SSH host keys verification
  • Know when will my change take effect?
  • Locking an account
  • Connecting to LDAP
  • Restoring from backups
    • Partial restore
    • Full rollback
  • User management
    • Listing members of a group
    • Creating users
    • Adding/removing users in a group
    • Adding/removing an admin
  • Searching LDAP
  • Modifying the schema
  • Deploying new userdir-ldap releases
  • Pager playbook
    • LDAP server failure
    • Git server failure
    • Deadlocks in ud-replicate
    • Troubleshooting changes@ failures
    • Dependency loop on new installs
    • Server certificate renewal
  • Disaster recovery
• Reference
  • Installation
  • SLA
  • Design
    • Architecture overview
    • Configuration file distribution
    • Files managed by ud-generate
    • How files get distributed by ud-replicate
      • makedb template files
      • self-configuration: sshdist authorized_keys
      • other special files
    • Authentication mechanisms
    • SSH access controls
    • LDAP user fields
      • sudoPassword field format
      • sshRSAAuthKey field format
    • LDAP host fields
      • rebootPolicy field values
      • purpose field values
      • exportOptions field values
    • Email gateway
    • Web interface
    • Interactions with Puppet
    • DNS zone file management
    • Source file analysis
  • Issues
  • Maintainer, users, and upstream
    • Branching policy
    • usedir-ldap-cgi fork status
    • userdir-ldap fork status
      • 2023 update
      • 2024 update
      • 2025 update
  • Monitoring and testing
  • Logs and metrics
  • Backups
  • Other documentation
• Discussion
  • Overview
  • Goals
    • Must have
    • Nice to have
    • Non-Goals
  • Approvals required
  • Proposed Solution
    • Short term: merge with upstream, port to Python 3 if necessary
    • Mid term: move hosts to Puppet, possibly replace ud-ldap with simpler dashboard
    • Long term: replace LDAP completely, with Puppet, GitLab and Kubernetes, possibly SSO dashboard
  • Cost
  • Alternatives considered
    • Replacement web interfaces
    • Command-line tools
    • Rewrites
    • Single-sign on
      • Keycloak
      • LemonLDAP
    • Others
    • SCIM

Tutorial

Our LDAP configuration is rather exotic. You will typically use the web interface and the OpenPGP-enabled email interface. This documentation aims at getting you familiar with the basics.

Getting to know LDAP

You should have received an email like this when your LDAP account was created:

Subject: New ud-ldap account for <your name here>

That includes information about how to configure email forwarding and SSH keys. You should follow those steps to configure your SSH key to get SSH access to servers (see ssh-jump-host).

How to change my email forward?

If you use Thunderbird and use it to manage your OpenPGP key, compose a new plain text (not HTML) message to changes@db.torproject.org, enter any subject line and write this in the message body:

emailForward: user@example.com

Before sending the email, open the OpenPGP drop-down menu at the top of the compose window and click Digitally Sign.

If you use GnuPG, send an (inline!) signed OpenPGP email to changes@db.torproject.org to change your email forward.

A command like this, in a UNIX shell, would do it:

echo "emailForward: user@example.com" | gpg --armor --sign

Then copy-paste that in your email client, making sure to avoid double-signing the email and sending in clear text (instead of HTML).

The email forward can also be changed in the web interface.

Password reset

If you have lost or forgotten your LDAP password or if you are are newly hired by TPI (congratulations!) and don't know your password yet, you can have it reset by sending a PGP signed message to the mail gateway.

The email should:

• be sent to chpasswd@db.torproject.org
• be composed in plain text (not HTML)
• be PGP signed by your key
• have exactly (and just) this text as the message body: Please change my Tor password

If you use Thunderbird and use it to manage your OpenPGP key, compose a new message in plain text (not HTML). You can configure sending emails in plaintext in your account settings, or if your new messages are usually composed in HTML you can hold the Shift key while clicking on the "+ New Message" button. Enter any subject line and write the message body described above.

Before sending the email, open the OpenPGP drop-down menu at the top of the compose window and click Digitally Sign.

Or, you can use GnuPG directly and then send an (inline!) email with your client of choice. A command like the following, in a UNIX shell, will create the signed text that you can copy-paste in your email. Make sure to avoid double-signing the email and sending it in clear text (instead of HTML):

echo "Please change my Tor password" | gpg --armor --sign

However you sent your signed email, the daemon will then respond with a new randomized password encrypted with your key. You can then use the update form with your new password to change your it to a strong password, in the "Change password" field, that you can remember or (preferably) a stronger password (longer and more random) stored in your password manager. Note: on that "update form" login page the button you should use to login is, unintuitively, labeled "Update my info"

You cannot set a new password via the mail gateway.

Alternatively, you can do without a password and use PGP to manipulate your LDAP information through the mail gateway, which includes instructions on SSH public key authentication, for example.

How do I update my OpenPGP key?

LDAP requires an OpenPGP key fingerprint in its records and uses that trust anchor to review changes like resetting your password or uploading an SSH key.

You can't, unfortunately, update the OpenPGP key yourself. Setting the key should have been done as part of your on-boarding. If it has not been done or you need to perform changes on the key, you should file an issue with TPA, detailing what change you want. Include a copy of the public key certificate.

To check whether your fingerprint is already stored in LDAP, search for your database entry in https://db.torproject.org/search.cgi and check the "PGP/GPG fingerprint" field.

We acknowledge this workflow is far from ideal, see tpo/tpa/team#40129 and tpo/tpa/team#29671 for further discussion and future work.

How-to

Set a sudo password

See the sudo password user configuration.

Operate the mail gateway

The LDAP directory has a PGP secured mail gateway that allows users to safely and conveniently effect changes to their entries. It makes use of PGP signed input messages to positively identify the user and to confirm the validity of the request. Furthermore it implements a replay cache that prevents the gateway from accepting the same message more than once.

There are three functions logically split into 3 separate email addresses that are implemented by the gateway: ping, new password and changes. The function to act on is the first argument to the program.

Error handling is currently done by generating a bounce message and passing descriptive error text to the mailer. This can generate a somewhat hard to read error message, but it does have all the relevant information.

ping

The ping command simply returns the users public record. It is useful for testing the gateway and for the requester to get a basic dump of their record. In future this address might 'freshen' the record to indicate the user is alive. Any PGP signed message will produce a reply.

New Password

If a user loses their password they can request that a new one be generated for them. This is done by sending the phrase "Please change my Tor password" to chpasswd@db.torproject.org. The phrase is required to prevent the daemon from triggering on arbitrary signed email. The best way to invoke this feature is with:

echo "Please change my Tor password" | gpg --armor --sign | mail chpasswd@db.torproject.org

After validating the request the daemon will generate a new random password, set it in the directory and respond with an encrypted message containing the new password. The password can be changed using one of the other interface methods.

Changes

An address (changes@db.torproject.org) is provided for making almost arbitrary changes to the contents of the record. The daemon parses its input line by line and acts on each line in a command oriented manner. Anything, except for passwords, can be changed using this mechanism. Note however that because this is a mail gateway it does stringent checking on its input. The other tools allow fields to be set to virtually anything, the gateway requires specific field formats to be met.

• field: A line of the form field: value will change the contents of the field to value. Some simple checks are performed on value to make sure that it is not set to nonsense. You can't set an empty string as value, use del instead (see below). The values that can be changed are: loginShell, emailForward, ircNick, jabberJID, labledURI, and VoIP

• del field: A line of the form del field will completely remove all occurrences of a field. Useful e.g. to unset your vacation status.

• SSH keys changes, see uploading a SSH user key

• show: If the single word show appears on a line in a PGP signed mail then a PGP encrypted version of the entire record will be attached to the resulting email. For example:

  echo show | gpg --clearsign | mail changes@db.torproject.org
  

Note that the changes alias does not handle PGP/MIME emails.

After processing the requests the daemon will generate a report which contains each input command and the action taken. If there are any parsing errors processing stops immediately, but valid changes up to that point are processed.

Notes

In this document PGP refers to any message or key that GnuPG is able to generate or parse, specifically it includes both PGP2.x and OpenPGP (aka GnuPG) keys.

Due to the replay cache the clock on the computer that generates the signatures has to be accurate to at least one day. If it is off by several months or more then the daemon will outright reject all messages.

Uploading a SSH user key

To upload a key into your authorized_keys file on all servers, simply place the key on a line by itself, sign the message and send it to changes@db.torproject.org. The full SSH key format specification is supported, see sshd(8). Probably the most common way to use this function will be

    gpg --armor --sign < ~/.ssh/id_rsa.pub | mail changes@db.torproject.org 

Which will set your authorized_keys to ~/.ssh/id_rsa.pub on all servers.

Supported key types are RSA (at least 2048 bits) and Ed25519.

Multiple keys per user are supported, but they must all be sent at once. To retrieve the existing SSH keys in order to merge existing keys with new ones, use the show command documented above.

Keys can be exported to a subset of machines by prepending allowed_hosts=$fqdn,$fqdn2 to the specific key. The allowed machines must only be separated by a comma. Example:

allowed_hosts=ravel.debian.org,gluck.debian.org ssh-rsa AAAAB3Nz..mOX/JQ== user@machine
ssh-rsa AAAAB3Nz..uD0khQ== user@machine

SSH host keys verification

The SSH host keys are stored in the LDAP database. The key and its fingerprint will be displayed alongside machine details in the machine list.

Developers that have a secure path to a DNSSEC enabled resolver can verify the existing SSHFP records by adding VerifyHostKeyDNS yes to their ~/.ssh/config file.

On machines in which are updated from the LDAP database, /etc/ssh/ssh_known_hosts contains the keys for all hosts in this domain.

Developers should add StrictHostKeyChecking yes to their ~/.ssh/config file so that they only connect to trusted hosts. Either with the DNSSEC records or the file mentioned above, nearly all hosts in the domain can be trusted automatically.

Developers can also execute ud-host -f or ud-host -f -h host on a server in order to display all host fingerprints or only the fingerprints of a particular host in order to compare it with the output of ssh on an external host.

Know when will my change take effect?

Once a change is saved to LDAP, the actual change will take at least 5 minutes and at most 15 minutes to propagate to the relevant host. See the configuration file distribution section for more details on why it is so.

Locking an account

See the user retirement procedures.

Connecting to LDAP

LDAP is not accessible to the outside world, so you need to get behind the firewall. Most operations are done directly on the LDAP server, by logging in as a regular user on db.torproject.org (currently alberti).

Once that's resolved, you can use ldapvi(1) or ldapsearch(1) to inspect the database. User documentation on that process is in doc/accounts and https://db.torproject.org. See also the rest of this documentation.

Restoring from backups

There's no special backup procedures for the LDAP server: it's backed up like everything else in the backup system.

To restore the OpenLDAP database, you need to head over the Bacula director, and enter the console:

ssh -tt bacula-director-01 bconsole

Then call the restore command and select 6: Select backup for a client before a specified time. Then pick the server (currently alberti.torproject.org) and a date. Then you need to "mark" the right files:

cd /var/lib/ldap
mark *
done

Then confirm the restore. The files will end up in /var/tmp/bacula-restores on the LDAP server.

The next step depends on whether this is a partial or total restore.

Partial restore

If you only need to access a specific field or user or part of the database, you can use slapcat to dump the database from the restored files even if the server is not running. You first need to "configure" a "fake" server in the restore directory. You will need to create two files under /var/tmp/bacula-restores:

• /var/tmp/bacula-restores/etc/ldap/slapd.conf
• /var/tmp/bacula-restores/etc/ldap/userdir-ldap-slapd.conf

They can be copied from /etc, with the following modifications:

diff -ru /etc/ldap/slapd.conf etc/ldap/slapd.conf
--- /etc/ldap/slapd.conf	2011-10-30 15:43:43.000000000 +0000
+++ etc/ldap/slapd.conf	2019-11-25 19:48:57.106055596 +0000
@@ -17,10 +17,10 @@
 
 # Where the pid file is put. The init.d script
 # will not stop the server if you change this.
-pidfile         /var/run/slapd/slapd.pid
+pidfile         /var/tmp/bacula-restores/var/run/slapd/slapd.pid
 
 # List of arguments that were passed to the server
-argsfile        /var/run/slapd/slapd.args
+argsfile        /var/tmp/bacula-restores/var/run/slapd/slapd.args
 
 # Read slapd.conf(5) for possible values
 loglevel        none
@@ -57,4 +57,4 @@
 #backend		<other>
 
 # userdir-ldap
-include /etc/ldap/userdir-ldap-slapd.conf
+include /var/tmp/bacula-restores/etc/ldap/userdir-ldap-slapd.conf
diff -ru /etc/ldap/userdir-ldap-slapd.conf etc/ldap/userdir-ldap-slapd.conf
--- /etc/ldap/userdir-ldap-slapd.conf	2019-11-13 20:55:58.789411014 +0000
+++ etc/ldap/userdir-ldap-slapd.conf	2019-11-25 19:49:45.154197081 +0000
@@ -5,7 +5,7 @@
 suffix          "dc=torproject,dc=org"
 
 # Where the database file are physically stored
-directory       "/var/lib/ldap"
+directory       "/var/tmp/bacula-restores/var/lib/ldap"
 
 moduleload      accesslog
 overlay accesslog
@@ -123,7 +123,7 @@
 
 
 database hdb
-directory       "/var/lib/ldap-log"
+directory       "/var/tmp/bacula-restores/var/lib/ldap-log"
 suffix cn=log
 #
 sizelimit 10000

Then slapcat is able to read those files directly:

slapcat -f /var/tmp/bacula-restores/etc/ldap/slapd.conf -F /var/tmp/bacula-restores/etc/ldap

Copy-paste the stuff you need into ldapvi.

Full rollback

Untested procedure.

If you need to roll back the entire server to this version, you first need to stop the LDAP server:

service slapd stop

Then move the files into place (in /var/lib/ldap):

mv /var/lib/ldap{,.orig}
cp -R /var/tmp/bacula-restores/var/lib/ldap /var/lib/ldap
chown -R openldap:openldap /var/lib/ldap

And start the server again:

service slapd start

User management

Listing members of a group

To tell which users are part of a given group (LDAP or otherwise), you can use the getent(1) command. For example, to see which users are part of the tordnsel group, you would call this command:

$ getent group tordnsel
tordnsel:x:1532:arlo,arma

In the above, arlo and arma are members of the tordnsel group. The fields in the output are in the format of the group(5) file.

Note that the group membership will vary according to the machine on which the command is run, as not all users are present everywhere.

Creating users

Users can be created for either individuals or servers (role account). Refer to the sections Creating a new user and Creating a role of the page about creating a new user for procedures to create users of both types.

Adding/removing users in a group

Using this magical ldapvi command on the LDAP server (db.torproject.org):

ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"

... you get thrown in a text editor showing you the entire dump of the LDAP database. Be careful.

To add or remove a user to/from a group, first locate that user with your editor search function (e.g. in vi, you'd type /uid=ahf to look for the ahf user). You should see a block that looks like this:

351 uid=ahf,ou=users,dc=torproject,dc=org
uid: ahf
objectClass: top
objectClass: inetOrgPerson
objectClass: debianAccount
objectClass: shadowAccount
objectClass: debianDeveloper
uidNumber: 2103
gidNumber: 2103
[...]
supplementaryGid: torproject

To add or remove a group, simply add or remove a supplementaryGid line. For example, in the above, we just added this line:

supplementaryGid: tordnsel

to add ahf to the tordnsel group.

Save the file and exit the editor. ldapvi will prompt you to confirm the changes, you can review with the v key or save with y.

Adding/removing an admin

The LDAP administrator group is a special group that is not defined through the supplementaryGid field, but by adding users into the group itself. With ldapvi (see above), you need to add a member: line, for example:

2 cn=LDAP Administrator,ou=users,dc=torproject,dc=org
objectClass: top
objectClass: groupOfNames
cn: LDAP administrator
member: uid=anarcat,ou=users,dc=torproject,dc=org

To remove the user from the admin group, remove the line.

The group grants the user access to administer LDAP directly, for example making any change through ldapvi.

Typically, admins will also be part of the adm group, with a normal line:

supplementaryGid: adm

Searching LDAP

This will load a text editor with a dump of all the users (useful to modify an existing user or add a new one):

ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"

This dump all known hosts in LDAP:

ldapsearch -ZZ -Lx -H ldap://db.torproject.org -b "ou=hosts,dc=torproject,dc=org"

Note that this will only work on the LDAP host itself or on whitelisted hosts which are few right now. Also note that this uses an "anonymous" connection, which means that some (secret) fields might not show up. For hosts, that's fine, but if you search for users, you will need to use authentication. This, for example, will dump all users with an SSH key:

ldapsearch -ZZ -LxW -H ldap://db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org" -b "ou=users,dc=torproject,dc=org" '(sshRSAAuthKey=*)'

Note how we added a search filter ((sshRSAAuthKey=*)) here. We could also have parsed the output in a script or bash, but this can actually be much simpler. Also note that the previous searches dump the entire objects. Sometimes it might be useful to only list the object handles or certain fields. For example, this will list all hosts rebootPolicy attribute:

ldapsearch -H ldap://db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL '(objectClass=*)' 'rebootPolicy'

This will list all servers with a manual reboot policy:

ldapsearch -H ldap://db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL '(rebootPolicy=manual)' ''

Note here the empty ('') attribute list.

To list hosts that do not have a reboot policy, you need a boolean modifier:

ldapsearch -H ldap://db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL '(!(rebootPolicy=manual))' ''

Such filters can be stacked to do complex searches. For example, this filter lists all active accounts:

ldapsearch -ZZ -vLxW -H ldap://db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org" -b "ou=users,dc=torproject,dc=org" '(&(!(|(objectclass=debianRoleAccount)(objectClass=debianGroup)(objectClass=simpleSecurityObject)(shadowExpire=1)))(objectClass=debianAccount))'

This lists users with access to Gitolite:

((allowedGroups=git-tor)|(exportOptions=GITOLITE))

... inactive users:

(&(shadowExpire=1)(objectClass=debianAccount))

Modifying the schema

If you need to add, change or remove a field in the schema of the LDAP database, it is a different, and complex operation. You will only need to do this if you launch a new service that (say) requires a new password specifically for that service.

The schema is maintained in the userdir-ldap.git repository. It is stored in the userdir-ldap.schema file. Assuming the modified object is a user, you would need to edit the file in three places:

1. as a comment, in the beginning, to allocate a new field, for example:

   @@ -113,6 +113,7 @@
    #   .45 - rebootPolicy
    #   .46 - totpSeed
    #   .47 - sshfpHostname
   +#   .48 - mailPassword
    #
    # .3 - experimental LDAP objectClasses
    #   .1 - debianDeveloper
   

This is purely informative, but it is important as it serves as a central allocation point for that numbering system. Also note that the entire schema lives under a branch of the Debian.org IANA OID allocation. If you reuse the OID space of Debian, it's important to submit the change to Debian sysadmins (dsa@debian.org) so they merge your change and avoid clashes.

2. create the actual attribute, somewhere next to a similar attribute or after the previous OID, in this case we created an attributed called mailPassword right after rtcPassword, since other passwords were also grouped there:

   attributetype ( 1.3.6.1.4.1.9586.100.4.2.48
          NAME 'mailPassword'
          DESC 'mail password for SMTP'
          EQUALITY octetStringMatch
          SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
   

3. finally, the new attribute needs to be added to the objectclass. in our example, the field was added alongside the other password fields in the debianAccount objectclass, which looked like this after the change:

   objectclass ( 1.3.6.1.4.1.9586.100.4.1.1
   	NAME 'debianAccount'
   	DESC 'Abstraction of an account with POSIX attributes and UTF8 support'
   	SUP top AUXILIARY
   	MUST ( cn $ uid $ uidNumber $ gidNumber )
   	MAY ( userPassword $ loginShell $ gecos $ homeDirectory $ description $ mailDisableMessage $ sudoPassword $ webPassword $ rtcPassword $ mailPassword $ totpSeed ) )
   

Once that schema file is propagated to the LDAP server, this should automatically be loaded by slapd when it is restarted (see below). But the ACL for that field should also be modified. In our case, we had to add the mailPassword field to two ACLs:

--- a/userdir-ldap-slapd.conf.in
+++ b/userdir-ldap-slapd.conf.in
@@ -54,7 +54,7 @@ access to attrs=privateSub
        by * break
 
 # allow users write access to an explicit subset of their fields
-access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,birthDate,mailDisableMessage,gender,emailforward,mailCallout,mailGreylisting,mailRBL,mailRHSBL,mailWhitelist,mailContentInspectionAction,mailDefaultOptions,facsimileTelephoneNumber,telephoneNumber,postalAddress,postalCode,loginShell,onVacation,latitude,longitude,VoIP,userPassword,sudoPassword,webPassword,rtcPassword,bATVToken
+access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,birthDate,mailDisableMessage,gender,emailforward,mailCallout,mailGreylisting,mailRBL,mailRHSBL,mailWhitelist,mailContentInspectionAction,mailDefaultOptions,facsimileTelephoneNumber,telephoneNumber,postalAddress,postalCode,loginShell,onVacation,latitude,longitude,VoIP,userPassword,sudoPassword,webPassword,rtcPassword,mailPassword,bATVToken
        by self write
        by * break
 
@@ -64,7 +64,7 @@ access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,bi
 ##
 
 # allow authn/z by anyone
-access to attrs=userPassword,sudoPassword,webPassword,rtcPassword,bATVToken
+access to attrs=userPassword,sudoPassword,webPassword,rtcPassword,mailPassword,bATVToken
        by * compare
 
 # readable only by self

If those are the only required changes, it is acceptable to directly make those changes directly on the LDAP server, as long as the exact same changes are performed in the git repository.

It is preferable, however, to build and upload userdir-ldap as a Debian package instead.

Deploying new userdir-ldap releases

Our userdir-ldap codebase is deployed through Debian packages built by hand on TPA's members computers, from our userdir-ldap repository. Typically, when we make changes to that repository, we should make sure we send the patches upstream, to the DSA userdir-ldap repository. The right way to do that is to send the patch by email, to mailto:dsa@debian.org, since they do not have merge requests enabled on that repository.

If you are lucky, we will have the latest version of the upstream code and your patch will apply cleanly upstream. If unlucky, you'll actually need to merge with upstream first. This process is generally done through those steps:

1. git merge the upstream changes, and resolve the conflicts
2. update the changelog (make sure you have the upstream version with ~tpo1 as a suffix so that upgrades work when if we ever catch up with upstream)
3. build the Debian package: git buildpackage
4. deploy the Debian package

Note that you may want to review our feature branches to see if our changes have been accepted upstream and, if not, update and resend the feature branches. See the branch policy documentation for more ideas.

Note that unless the change is trivial, the Debian package should be deployed very carefully. Because userdir-ldap is such a critical piece of infrastructure, it can easily break stuff like PAM and logins, so it is important to deploy it one machine at a time, and run ud-replicate on the deployed machine (and ud-generate if the machine is the LDAP server).

So "deploy the Debian package" should actually be done by copying, by hand, the package to specific servers over SSH, and only after testing there, uploading it to the Debian archive.

Note that it's probably a good idea to update the userdir-ldap-cgi repository alongside userdir-ldap. The above process should similarly apply.

Pager playbook

An LDAP server failure can trigger lots of emails as ud-ldap fails to synchronize things. But the infrastructure should survive the downtime, because users and passwords are copied over to all hosts. In other words, authentication doesn't rely on the LDAP server being up.

In general, OpenLDAP is very stable and doesn't generally crash, so we haven't had many emergencies scenarios with it yet. If anything happens, make sure the slapd service is running.

The ud-ldap software, on the other hand, is a little more complicated and can be hard to diagnose. It has a large number of moving parts (Python, Perl, Bash, Shell scripts) and talks over a large number of protocols (email, DNS, HTTPS, SSH, finger). The failure modes documented here are far from exhaustive and you should expect exotic failures and error messages.

LDAP server failure

That said, if the LDAP server goes down, password changes will not work, and the server inventory (at https://db.torproject.org/) will be gone. A mitigation is to use Puppet manifests and/or PuppetDB to get a host list and server inventory, see the Puppet documentation for details.

Git server failure

The LDAP server will fail to regenerate (and therefore update) zone files and zone records if the Git server is unavailable. This is described in issue 33766. The fix is to recover the git server. A workaround is to run this command on the primary DNS server (currently nevii):

sudo -u dnsadm /srv/dns.torproject.org/bin/update --force

Deadlocks in ud-replicate

The ud-replicate process keeps a "reader" lock on the LDAP server. If for some reason the network transport fails, that lock might be held on forever. This happened in the past on hosts with flaky network or ipsec problems that null-routed packets between ipsec nodes.

There is a Prometheus metric that will detect stale synchronization.

The fix is to find the offending locked process and kill it. In desperation:

pkill -u sshdist rsync

... but really, you should carefully review the rsync processes before killing them all like that. And obviously, fixing the underlying network issue would be important to avoid such problems in the future.

Also note that the lock file is in /var/cache/userdir-ldap/hosts/ud-generate.lock, and ud-generate tries to get a write lock on the file. This implies that a deadlock will also affect file generation and keep ud-generate from generating fresh config files.

Finally, ud-replicate also holds a lock on /var/lib/misc on the client side, but that rarely causes problems.

Troubleshooting changes@ failures

A common user question is that they are unable to change their SSH key. This can happen if their email client somehow has trouble sending a PGP signature correctly. Most often than not, this is because their email client does a line wrap or somehow corrupts the OpenPGP signature in the email.

A good place to start looking for such problems is the log files on the LDAP server (currently alberti). For example, this has a trace of all the emails received by the changes@ alias:

/srv/db.torproject.org/mail-logs/received.changes

A common problem is people using --clearsign instead of --sign when sending an SSH key. When that happens, many email clients (including Gmail) will word-wrap the SSH key after the comment, breaking the signature. For example, this might happen:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKxqYYEeus8dRXBHhLsp0SjH7ut2X8UM9hdXN=
wJIl89otcJ5qKoXj90K9hq8eBjG2KuAZtp0taGQHqzBOFK+sFm9/gIqvzzQ07Pn0xtkmg10Hunq=
vPKMj4gDFLIqTF0WSPA2E6L/TWaeVJ+IiGuE49j+0Ohd7UFDEquM1H/zno22vIEm/dxWLPWD9gG=
MmwBghvfK/dRyzSEDGlAVeWLzoIvVOG12/ANgic3TlftbhiLKTs52hy8Qhq/aQBqd0McaE4JGxe=
9k71OCg+0WHVS4q7HVdTUqT3VFFfz0kjDzYTYQQcHMqPHvYzZghxMVCmteNdJNwJmGSNPVaUeJG=
MumJ9
anarcat@curie

-----BEGIN PGP SIGNATURE-----
[...]
-----END PGP SIGNATURE-----

Using --sign --armor will work around this problem, as the original message will all be ASCII-armored.

Dependency loop on new installs

Installing a new server requires granting the new server access various machines, including puppet and the LDAP server itself. This is granted ... by Puppet through LDAP!

So a server cannot register itself on the LDAP server and needs an operator to first create a host snippet on the LDAP server, and then run Puppet on the Puppet server. This is documented in the installation notes.

Server certificate renewal

The LDAP server uses a self-signed CA certificate that clients use to verify TLS connections, both on port 389 (via STARTTLS) and port 636.

When the db.torproject.org.pem certificate nears its expiration date, Prometheus will spawn warnings.

To renew this certificate, log on to alberti.torproject.org and create a text file named db.torproject.org.cfg with this content:

ca
signing_key
encryption_key
expiration_days = 730
cn = db.torproject.org

Then the new certificate can be generated using certtool:

certtool --generate-self-signed \
    --load-privkey /etc/ldap/db.torproject.org.key \
    --outfile db.torproject.org.pem \
    --template db.torproject.org.cfg

Copy the contents of the certificate on your machine:

cat db.torproject.org.pem

To bootstrap the new certificate, follow these steps first on alberti:

puppet agent --disable "updating LDAP certificate"
cp db.torproject.org.pem /etc/ssl/certs/db.torproject.org.pem
systemctl restart slapd.service

You can then verify OpenLDAP is working correctly by running:

ldapsearch -n -v -ZZ -x -H ldap://db.torproject.org

If it works, the process can be continued by deploying the certificate manually on pauli (the Puppet server):

puppet agent --disable "updating LDAP certificate"

# replace the old certificate manually
cat > /etc/ssl/certs/db.torproject.org.pem <<EOF
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
EOF

# fully restart Puppet
systemctl stop apache2
systemctl start apache2

At this point, the new certificate can be replaced on the tor-puppet repository, in modules/ldap_client_config/files/db.torproject.org.pem.

Lastly, run puppet agent --enable on alberti and pauli and trigger a Puppet run on all nodes:

cumin -b 5 '*' 'paoc'

Disaster recovery

The LDAP server is mostly built by hand and should therefore be restored from backups in case of a catastrophic failure. Care should be taken to keep the SSH keys of the server intact.

The IP address (and name?) of the LDAP server should not be hard-coded anywhere. When the server was last renumbered (issue 33908), the only changes necessary were on the server itself, in /etc. So in theory, a fresh new server could be deployed (from backups) in a new location (and new address) without having to do much.

Reference

Installation

All ud-ldap components are deployed through Debian packages, compiled from the git repositories. It is assumed that some manual configuration was performed on the main LDAP server to get it bootstrapped, but that procedure was lost in the mists of time.

Only backups keep us from total catastrophe in case of lost. Therefore, this system probably cannot be reinstalled from scratch.

SLA

The LDAP server is designed to be fault-tolerant in the sense that it's database is copied over other hosts. It should otherwise be highly available as it's a key component in managing users authentication and authorization, and machines.

Design

The LDAP setup at Tor is based on the one from Debian.org. It has a long, old and complex history, lost in the mists of time.

Configuration and database files like SSH keys, OpenPGP keyrings, password, group databases, or email forward files are synchronised to various hosts from the LDAP database. Most operations can be performed on the db.torproject.org site or by email.

Architecture overview

This is all implemented by a tool called ud-ldap, inherited from the Debian project. The project is made of a collection of bash, Python and Perl scripts which take care of synchronizing various configuration files to hosts based on the LDAP configuration. Most of this section aims at documenting how this program works.

ud-ldap is made of two Debian packages: userdir-ldap, which ships the various server- and client-side scripts (and is therefore installed everywhere), and userdir-ldap-cgi which ships the web interface (and is therefore installed only on the LDAP server).

Configuration files are generated on the server by the ud-generate command, which goes over the LDAP directory and crafts a tree of configuration files, one directory per host defined in LDAP. Then each host pulls those configuration files with ud-replicate. A common set of files is exported everywhere, while the exportOptions field can override that by disabling some exports or enabling special ones.

An email gateway processes OpenPGP-signed emails which can change a user's fields, passwords or SSH keys, for example.

In general, ud-ldap:

• creates UNIX users and groups on (some or all) machines
• distributes password files for those users or other services
• distributes user SSH public keys
• distributes all SSH host public keys to all hosts
• configures and reload arbitrary services, but particularly handles email, DNS, and git servers
• provides host metadata to Puppet

This diagram covers those inter-dependencies at the time of writing.

Configuration file distribution

An important part of ud-ldap is the ud-generate command, which generates configuration files for each host. Then the ud-replicate command runs on each node to rsync those files. Both commands are ran from cron on regular intervals. ud-replicate is configured by the userdir-ldap package, at every 5 minutes. ud-generate is also configured to run every 5 minutes, starting on the third minute of every hour, in /etc/cron.d/local-ud-generate (so at minute 3, 8, 13, ..., 53, 58).

More specifically, this is what happens:

1. on the LDAP server (currently alberti), ud-generate writes various files (detailed below) in one directory per host

2. on all hosts, ud-replicate rsync's that host's directory from the LDAP server (as the sshdist user)

ud-generate will write files only if the LDAP database or keyring changed since last time, or at most every 24 hours, based on the timestamp (last_update.trace). The --force option can be used to bypass those checks.

Files managed by ud-generate

This is a (hopefully) exhaustive list of files generated by ud-generate as part of userdir-ldap 0.3.97 ("UNRELEASED"). This might have changed since this was documented, on 2020-10-07.

All files are written in the /var/cache/userdir-ldap/hosts/, with one subdirectory per host.

How files get distributed by ud-replicate

The ud-replicate program runs on all hosts every 5 minutes and logs in as the sshdist user on the LDAP server. It rsyncs the files from the /var/cache/userdir-ldap/hosts/$HOST/ directory on the LDAP server to the /var/lib/misc/$HOST directory.

For example, for a host named example.torproject.org, ud-generate will write the files in /var/cache/userdir-ldap/hosts/example.torproject.org/ and ud-replicate will synchronize that directory, on example.torproject.org, in the /var/lib/misc/example.torproject.org/ directory. The /var/lib/misc/thishost symlink will also point to that directory.

Then ud-replicate those special things with some of those files. Otherwise consumers of those files are expected to use them directly in /var/lib/misc/thishost/, as is.

makedb template files

Files labeled with template are inputs for the makedb(1) command. They are like their regular "non-template" counterparts, except they have a prefix that corresponds to:

1. an incremental index, prefixed by zero (e.g. 01, 02, 03, ... 010...)
2. the uid field (the username), prefixed by a dot (e.g. .anarcat)
3. the uidNumber field (the UNIX UID), prefixed by an equal sign (e.g. =1092)

Those are the fields for the passwd file. The shadow file has only prefixes 1 and 2. This file format is used to create the databases in /var/lib/misc/ which are fed into the NSS database with the libnss-db package. The database files get generated by makedb(1) from the templates above. It is what allows the passwd file in /etc/passwd to remain untouched while still allowing ud-ldap to manage extra users.

self-configuration: sshdist authorized_keys

The authorized_keys file gets shipped if AUTHKEYS is set in extraOptions. This is typically set on the LDAP server (currently alberti), so that all servers can login to the server (as the sshdist user) and synchronise their configuration with ud-replicate.

This file gets dropped in /var/lib/misc/authorized_keys by ud-replicate. A symlink in /etc/ssh/userkeys/sshdist ensures those keys are active for the sshdist user.

other special files

More files are handled specially by ud-replicate:

• forward-alias gets modified (@emailappend appended to each line) and replaces /etc/postfix/debian, which gets rehashed by postmap. this is done only if /etc/postfix and forward-alias exist
• the bsmtp config file is deployed in /etc/exim4, if both exist
• if dns-sshfp or dns-zone are changed, the DNS server zone files get regenerated and server reloaded (sudo -u dnsadm /srv/dns.torproject.org/bin/update, see "DNS zone file management" below)
• ssh_known_hosts gets symlinked to /etc/ssh
• the ssh-keys.tar.gz tar archive gets decompressed in /var/lib/misc/userkeys
• the web-passwords file is given to root:www-data and made readable only by the group
• the rtc-passwords file is installed in /var/local/ as:
  • rtc-passwords.freerad if /etc/freeradius exists
  • rtc-passwords.return if /etc/reTurn exists
  • rtc-passwords.prosody if /etc/prosody exists .. and the appropriate service (freeradius, resiprocate-turn-server, prosody, respectively) get reloaded

Authentication mechanisms

ud-ldap deals uses multiple mechanisms to authenticate users and machines.

1. the web interface binds to the LDAP directory anonymously, or as the logged in user, if any. an encrypted copy of the username/password pair is stored on disk, encrypted, and passed around in a URL token
2. the email gateway runs as the sshdist user and binds to the LDAP directory using the sshdist-specific password. the sshdist user has full admin rights to the LDAP database through the slapd configuration. commands are authenticated using OpenPGP signatures, checked against the keyring, maintained outside of LDAP, manually, in the account-keyring.git repository, which needs to be pushed to the LDAP server by hand.
3. ud-generate runs as the sshdist user and binds as that user to LDAP as well
4. ud-replicate runs as root on all servers. it authenticates with the central LDAP server over SSH using the SSH server host private key as a user key, and logs in to the SSH server as the sshdist user. the authorized_keys file for that user on the LDAP server (/etc/ssh/userkeys/sshdist) determines which files the client has access to using a predefined rsync command which restricts to only /var/cache/userdir-ldap/hosts/$HOST/
5. Puppet binds to the LDAP server over LDAPS using the custom CA, anonymously
6. LDAP admins also have access to the LDAP server directly, provided they can get a shell (or a port forward) to access it

This is not related to ud-ldap authentication itself, but ud-ldap obviously distributes authentication systems all over the place:

• PAM and NSS usernames and passwords
• SSH user authentication keys
• SSH server public keys
• webPassword, rtcPassword, mailPassword, and so on
• email forwards and email block list checks
• DNS zone files (which may include things like SSH server public keys, for example)

SSH access controls

A user gets granted access if it is part of a group that has been granted access on the host with the allowedGroups field. An additional group has access to all host, defined as allowedgroupspreload (currently adm) in /etc/userdir-ldap/userdir-ldap.conf on the LDAP server (currently alberti).

Also note the NOPASSWD value for exportOptions: if set, it marks the host as not allowing passwords so the shadow database is not shipped which makes it impossible to login to the host with a password. In practice this has no effect since password-based authentication is disabled at the SSH server level, however.

LDAP user fields

Those are the fields in the user LDAP object as of userdir-ldap 0.3.97 ("UNRELEASED"). This might have changed since this was documented, on 2020-10-07. Some of those fields, but not all, can be modified or deleted by the user through the email interface (ud-mailgate).

sudoPassword field format

The sudoPassword field is special. It has 4 fields separated by spaces:

1. a UUID
2. the status, which is either the string unconfirmed or the string confirmed: followed by a SHA1 (!) HMAC of the string password-is-confirmed, sudo, the UID, the UUID, the host list, and the hashed password, joined by colons (:), primed with a secret key stored in /etc/userdir-ldap/key-hmac-$UID where UID is the numeric identifier of the calling user, generally 33 (probably the web server?) or sshdist? The secret key can also overridden by the UD_HMAC_KEY environment variable
3. the host list, either * (meaning all hosts) or a comma (,) separated list of hosts this password applies to
4. the hashed password, which is restricted to 50 characters: if longer, it is invalid (*)

That password field gets validated by email through ud-mailgate.

The field can, of course, have multiple values.

sshRSAAuthKey field format

The sshRSAAuthKey field can have multiple values. Each one should be a valid authorized_keys(5) file.

Its presence influences whether a user is allowed to login to a host or not. That is, if it is missing, the user will not be added to the shadow database.

The GITOLITE hosts treat the field specially: it looks for allowed_hosts fields and will match only on the right host. If will skip keys that have other options.

LDAP host fields

Those are the fields in the user LDAP object as of userdir-ldap 0.3.97 ("UNRELEASED"). This might have changed since this was documented, on 2020-10-07. Those fields are usually edited by hand by an LDAP admin using ldapvi.

rebootPolicy field values

The rebootPolicy is documented in the reboot procedures.

purpose field values

The purpose field is special in that it supports a crude markup language which can be used to create links in the web interface, but is also used to generate SSH known_hosts files. To quote the ud-generate source code:

In the purpose field, [[host|some other text]] (where some other text is optional) makes a hyperlink on the web [interface]. We now also add these hosts to the ssh known_hosts file. But so that we don't have to add everything we link, we can add an asterisk and say [[*... to ignore it. In order to be able to add stuff to ssh without http linking it we also support [[-hostname]] entries.

Otherwise the description and purpose fields are fairly similar and often contain the same value.

Note that there can be multiple purpose values, in case we need multiple names like that. For example, the prometheus/grafana server has:

purpose: [[-prometheus1.torproject.org]]
purpose: [[prometheus.torproject.org]]
purpose: [[grafana.torproject.org]]

because:

• prometheus1.torproject.org: is an SSH alias but not a web one
• prometheus.torproject.org: because the host also runs Prometheus as a web interface
• grafana.torproject.org: and that is the Grafana web interface

Note that those do not (unfortunately) add a CNAME in DNS. That needs to be done by hand in dns/domains.git.

exportOptions field values

The exportOptions field warrants a more detailed explanation. Its value determines which files are created by ud-generate for a given host. It can either enable or inhibit the creation of certain files.

• AUTHKEYS: ship the authorized_keys file for sshdist, typically on the LDAP server for ud-replicate to connect to it
• BSMTP: ship the bsmtp file
• DNS: ships DNS zone files (dns-sshfp and dns-zone)
• GITOLITE: ship the gitolite-specific SSH authorized_keys file. can also be suffixed, e.g. GITOLITE=OPTIONS where OPTIONS does magic stuff like skip some hosts (?) or change the SSH command restriction
• KEYRING: ship the sync_keyrings GnuPG keyring file (.gpg) defined in userdir-ldap.conf, generated from the admin/account-keyring.git repository (technically: the ssh://db.torproject.org/srv/db.torproject.org/keyrings/keyring.git repository...)
• NOMARKERS: inhibits the creation of the markers file
• NOPASSWD: if present, the passwd database has * in the password field, x otherwise. also inhibits the creation of the shadow file. also marks a host as UNTRUSTED (below)
• PRIVATE: ship the debian-private mailing list registration file
• RTC-PASSWORDS: ship the rtc-passwords file
• MAIL-PASSWORDS: ship the mail-passwords file
• TOTP: ship the users.oath file
• UNTRUSTED: skip sudo passwords for this host unless explicitly set
• WEB-PASSWORDS: ship the web-passwords file

Of those parameters, only AUTHKEYS, DNS and GITOLITE are used at TPO, for, respectively, the LDAP server, DNS servers, and the git server.

Email gateway

The email gateway runs on the LDAP server. There are four aliases, defined in /etc/aliases, which forward to the sshdist user with an extension:

change:           sshdist+changes
changes:          sshdist+changes
chpasswd:         sshdist+chpass
ping:             sshdist+ping

Then three .forward files in the ~sshdist home directory redirect this to the ud-mailgate Python program while also appending a copy of the email into /srv/db.torproject.org/mail-logs/, for example:

# cat ~sshdist/.forward+changes
"| /usr/bin/ud-mailgate change"
/srv/db.torproject.org/mail-logs/received.changes

This is how ud-mailgate processes incoming messages:

1. it parses the email from stdin using Python's email.parser library

2. it tries to find an OpenPGP-signed message and passes it to the GPGCheckSig function to verify the signature against the trusted keyring

3. it does a check against replay attacks by checking:

   • if the OpenPGP signature timestamp is reasonable (less than 3 days in the future, or 4 days in the past)

   • if the signature has already been received in the last 7 days

   The ReplayCache is a dbm database stored in /var/cache/userdir-ldap/mail/replay.

4. it then behaves differently whether it was called with ping, chpass or change as its argument

5. in any case it tries to send a reply to the user by email, encrypted in the case of chpass

The ping routine just responds to the user with their LDAP entry, rendered according to the ping-reply template (in /etc/userdir-ldap/templates).

The chpass routine behaves differently depending on a magic string in the signed message, which can either be:

1. "Please change my Debian password"
2. "Please change my Tor password"
3. "Please change my Kerberos password"
4. "Please change my TOTP seed"

The first two do the same thing. The latter two are not in use at TPO. The main chpass routine basically does this:

1. generate a 15-character random string
2. "hash" it with Python's crypt with a MD5 (!) salt
3. set the hashed password in the user's LDAP object, userPassword field
4. bump the shadowLastChange field in the user's LDAP object
5. render the passwd-changed email template which will include an OpenPGP encrypted copy of the cleartext email

The change routine does one or many of the following, depending on the lines in the signed message:

• on show: send a key: value list of parameters of the user's LDAP object, OpenPGP-encrypted
• change the user's "position marker" (latitude/longitude) with a format like Lat: -10.0 Long: +10.0
• add or replace a dnsZoneEntry if the line looks like host IN {A,AAAA,CNAME,MX,TXT}
• replace LDAP user object fields if the line looks like field: value. only some fields are supported
• add or replace sshRSAAuthKey lines when the line looks like an SSH key (note that this routine sends its error email separately). this gets massaged so that it matches the format expected by ud-generate in LDAP and is validated by piping in ssh-keygen -l -f. the allowed_hosts block is checked against the existing list of servers and it enforces a minimum RSA key size (2048 bits)
• delete an LDAP user field, when provided with a line that looks like del FIELD
• add or replace mailrbl, mailrhsbl and mailwhiltelist fields, except allow a space separator instead of the normal colon separator for arbitrary fields (??)
• if the sudo password is changed, it checks if the HMAC provided matches the expected one from the database and switched from unconfirmed to confirmed

Note that the change routine only operates if the account is not locked (if the userPassword does not contain the string *LK* or starts with the ! string).

Web interface

The web interface is shipped as part of the userdir-ldap-cgi Debian package, built from the userdir-ldap-cgi repository. The web interface is written in Perl, using the builtin CGI module and WML templates. It handles password and settings changes for users, although some settings (like sudo passwords) require an extra confirmation by OpenPGP-signed message through the email gateway. It also lists machines known by LDAP.

The web interface also ships documentation in the form of HTML pages rendered through WML templates.

The web interface binds to the LDAP database as the logged in user (or anonymously, for some listings and searches) and therefore doesn't enjoy any special privilege in itself.

Each "dynamic" page is a standalone CGI script, although it uses some common code from Util.pm to load settings, format some strings, deal with authentication tokens and passwords.

The main page is the search.cgi interface, which allows users to perform a search in the user database, based on a subset of LDAP fields. This script uses the searchform.wml template.

The login form (login.cgi) binds with the LDAP database using the provided user/password. A "hack" is present to "upgrade" the user's passwords to MD5, presumably it was in cleartext before. Authentication persistence is done through an authentication token (authtoken in the URL), which consists of a MD5 "encoded username and a key to decrypt the password stored on disk, the authtoken is protected from modification by an HMAC". In practice, it seems the user's password is stored on disk, encrypted with a Blowfish cipher in CBC mode (from Crypt::CBC), with a 10 bytes (80 bits) key, while the HMAC is based on SHA1 (from Digest::HMAC_SHA1). The tokens are stored in /var/cache/userdir-ldap/web-cookies/ with one file per user, named after a salted MD5 hash of the username. Tokens expire after 10 minutes by the web interface, but it doesn't seem like old tokens get removed unless the user is active on the site.

Although the user/password pair is not stored directly in the user's browser cookies or history, the authentication token effectively acts as a valid user/password to make changes to the LDAP user database. It could be abused to authenticate as an LDAP user and change their password, for example.

The login form uses the login.wml template.

The logout.cgi interface, fortunately, allows users to clear this on-disk data, invalidating possibly leaked tokens.

The update.cgi interface is what processes actual changes requested by users. It will extract the actual LDAP user and password from the on-disk encrypted token and bind with that username and password. It does some processing of the form to massage it into a proper LDAP update, running some password quality checks using a wrapper around cracklib called password-qualify-check which, essentially, looks at a word list, the GECOS fields and the old password. Partial updates are possible: if (say) the rtcPassword fields don't match but the userPassword fields do, the latter will be performed because it is done first. It is here that unconfirmed sudo passwords are set as well. It's the user's responsibility to send the challenge response by signed OpenPGP email afterwards. This script uses the update.wml template.

The machines.cgi script will list servers registered in the LDAP in a table. It binds to the LDAP server anonymously and searches for all hosts. It uses the hostinfo.wml template.

Finally the fetchkey.cgi script will load a public key from the keyrings configuration setting based on the provided fingerprint and dump it in plain text.

Interactions with Puppet

The Puppet server is closely coupled with LDAP, from which it gathers information about servers.

It specifically uses those fields:

The ipHostnumber field is also used to lookup the host in the hoster.yaml database in order to figure out which hosting provider hosts the parent metal. This is, in turn, used in Hiera to change certain parameters, like Debian mirrors.

Note that the above fields are explicitly imported in the allnodeinfo data structure, along with sshRSAHostKey and mXRecord, but those are not used. Furthermore, the nodeinfo data structure imports all of the host's data, so there might be other fields in use that I haven't found.

Puppet connects to the LDAP server directly over LDAPS (port 636) and therefore requires the custom LDAP host CA, although it binds to the server anonymously.

DNS zone file management

One of the configuration files ud-generate generates are, critically, the dns-sshfp and dns-zone files.

The dns-sshfp file holds the following records mapped to LDAP host fields:

The dns-zone file contains user-specific DNS entries. If a user object has a dnsZoneEntry field, that entry is written to the file directly. A TXT record with the user's email address and their PGP key fingerprint is also added for identification. That file is not in use in TPO at the moment, but is (probably?) the mechanism behind the user-editable debian.net zone.

Those files only get distributed to DNS servers (e.g. nevii and falax), which are marked with the DNS flag in the exportOptions field in LDAP.

Here is how zones are propagated from LDAP to the DNS server:

1. ud-replicate will pull the files with rsync, as explained in the previous section

2. if the dns-zone or dns-sshfp files change, ud-replicate will call /srv/dns.torproject.org/bin/update (from dns_helpers.git) as the dnsadm user, which creates the final zonefile in /srv/dns.torproject.org/var/generated/torproject.org

The bin/update script does the following:

1. pulls the auto-dns.git and domains.git git repositories

2. updates the DNSSEC keys (with bin/update-keys)

3. update the GeoIP distribution mechanism (with bin/update-geo)

4. builds the service includes from the auto-dns directory (with auto-dns/build-services), which writes the /srv/dns.torproject.org/var/services-auto/all file

5. for each domain in domains.git, calls write_zonefile (from dns_helpers.git), which in turn:

   1. increments the serial number in the .serial state file
   2. generate a zone header with the new serial number
   3. include the zone from domains.git
   4. compile it with named-compilezone(8), which is the part that expands the various $INCLUDE directives

6. then calls dns-update (from dns_helpers.git) which rewrites the named.conf snippet and reloads bind, if needed

The various $INCLUDE directives in the torproject.org zonefile are currently:

• /var/lib/misc/thishost/dns-sshfp - generated on the LDAP server by ud-generate, contains SSHFP records for each host
• /srv/dns.torproject.org/puppet-extra/include-torproject.org: generated by Puppet modules which call the dnsextras module. This is used, among other things, for TLSA records for HTTPS and SMTP services
• /srv/dns.torproject.org/var/services-auto/all: generated by the build-services script in the auto-dns.git directory
• /srv/letsencrypt.torproject.org/var/hook/snippet: generated by the bin/le-hook in the letsencrypt-domains.git repository, to authenticate against Let's Encrypt and generate TLS certificates.

Note that this procedure fails when the git server is unavailable, see issue 33766 for details.

Source file analysis

Those are the various scripts shipped by userdir-ldap. This table describes which programming language it's written in and a short description of its purpose. The ud? column documents whether the command was considered for implementation in the ud rewrite, and gives us a hint on whether it is important or not.

Note how the ud-guest-upgrade command works. It generates an LDAP snippet like:

delete: allowedHost
-
delete: shadowExpire
-
replace: supplementaryGid
supplementaryGid: $GIDs
-
replace: privateSub
privateSub: $UID@debian.org

where the guest gid is replaced by the "default" defaultgroup set in the userdir-ldap.conf file.

Those are other files in the source distribution which are not directly visible to users but are used as libraries by other files.

Those are the configuration files shipped with the package:

Issues

There is no issue tracker specifically for this project, file or search for issues in the team issue tracker, with the ~LDAP label.

Maintainer, users, and upstream

Our userdir-ldap repository is a fork of the DSA userdir-ldap repository. The codebase is therefore shared with the Debian project, which uses it more heavily than TPO. According to GitLab's analysis, weasel has contributed the most to the repository (since 2007), followed closely by Joey Schulze, which wrote most of the code before that, between 1999 and 2007.

The service is mostly in maintenance mode, both at DSA and in TPO, with small, incremental changes being made to the codebase over all those years. Attempts have been made to rewrite it with a Django frontend (ud, 2013-2014 no change since 2017) or Pylons (userdir-ldap-pylons, 2011, abandoned), all have been abandoned.

Our fork is primarily maintained by anarcat and weasel. It is used by everyone at Tor.

Our fork tries to follow upstream as closely as possible, but the Debian project is hardcoded in a lot of places so we (currently) are forced to keep patches on top of upstream.

Branching policy

In the userdir-ldap and userdir-ldap-cgi repository, we have tried to follow the icebreaker branching strategy used at one of Google's kernel teams. Briefly, the idea is to have patches rebased on top of the latest upstream release, with each feature branch based on top of the tag. Those branches get merged in our "master" branch which contains our latest source code. When a new upstream release is done, a new feature branch is created by merging the previous feature branch and the new release.

See page 24 and page 25 of the talk slides for a view of what that graph looks like. This is what it looks like in userdir-ldap:

$ git log --decorate --oneline --graph  --all
*   97c5660 (master) Merge branch 'tpo-scrub-0.3.104-pre'
|\  
| * 698da3a (tpo-scrub-0.3.104-pre-dd7f9a3) update changelog after rebase
| * b05f7d0 Set emailappend to torproject.org
| * 407775c Use https:// in welcome email
| * fecc816 Re-apply tpo changes to Debian's repo
| * dd7f9a3 (dsa/master) ud-mailgate: fix SPF verification logic to work correctly with "~all"
| * f991671 Actually ship ud-guest-extend

In this case, there is only one feature branch left, and it's now identical to master.

This is what it looks like in userdir-ldap-cgi:

*   25cf477 (master) Merge branch 'tpo-scrub-0.3.43-pre-5091066'
|\  
| * 0982aa0 (tpo-scrub-0.3.43-pre-5091066) remove debian-specific stylesheets, use TPO
| * 5eb5da8 remove email features not enabled on torproject.org
| * 54c03de remove direct access note, disabled in our install
| * fec1282 Removed lines which mention finger (TPO has no finger services)
| * 18f3aeb drop many fields from update form
| * d1dd377 Replace "debian" with "torproject" as much as possible
| * 7dcc1a1 (clean-series-0.3.43-pre-5091066) add keywords in changes mail commands help
| * aecb3c8 use an absolute path in SSH key upload
| * ca110ab remove another needless use of cat
| * 685f36b use relative link for web form, drop SSL
| * b7bd99d don't document SSH key changes in the password lost page (#33134)
| * 05a10e5 explicitly state that we do not support pgp/mime (#33134)
| * f98bba6 clarify that show requires a signature as well (#33134)
| * e41d911 suggest using --sign for the SSH key as well (#33134)
| * 50933fd improve sudo passwords update confirmation string
| * 2907fc2 add spacing in doc-mail
| * 5091066 (dsa/master) Update now broken links to the naming scheme page to use archive.org
| * c08a063 doc-direct: stop referring to access changes from 2003

In this particular case the tpo-scrub branch is based on top of the clean-series patch because there would be too many conflicts otherwise (and we are really, really hoping the patches can be merged). But typically those would both be branched off dsa/master.

This pattern is designed so that it's easier to send patches upstream. Unfortunately, upstream releases are somewhat irregular so this somewhat breaks down because we don't have a solid branch point to base our feature branches off. This is why the branches are named like tpo-scrub-0.3.104-pre-dd7f9a3: the pre-dd7f9a3 is to indicate that we are not branched off a real release.

TODO: consider git's newer --update-refs to see if it may help maintain those branches, see this post

Update: as of 2025-04-17, we have mostly abandoned trying to merge patches upstream after yet again other releases produced upstream that have not merged our patches. See the 2025 update below.

usedir-ldap-cgi fork status

In the last sync, usedir-ldap-cgi was brought from 27 patches down to 16, 10 of which were sent upstream. Our diff there is now:

  22 files changed, 11661 insertions(+), 553 deletions(-)

The large number of inserted lines is because we included the styleguide bootstrap.css which is 11561 lines on its own, so really, this is the diff stat if we ignore that stylesheet:

  21 files changed, 100 insertions(+), 553 deletions(-)

If the patches get merged upstream, our current delta is:

 21 files changed, 23 insertions(+), 527 deletions(-)

Update: none of our recent patches were merged upstream. We still have the following branches:

• auth-status-code-0.3.43: send proper codes on authentication failures, to enable fail2ban parsing
• mailpassword-update-0.3.43: enables mail password edits on the web interface
• clean-series-0.3.43: various cleanups
• tpo-scrub-0.3.43: s/debian.org/torproject.org/, TPO-specific
• feature-pretty-css-0.3.43: CSS cleanups and UI tweaks, TPO-specific

Apart from getting patches merged upstream, the only way forward here is either to make the "Debian" strings "variables" in the WML templates or completely remove the documentation from userdir-ldap-cgi (and move it to the project's respective wikis).

For now, we have changed the navigation to point to our wiki as much as possible. The next step is to remove our patches to the upstream documentation and make sure that documentation is not reachable to avoid confusion.

userdir-ldap fork status

Our diff in userdir-ldap used to be much smaller (in 2021):

 6 files changed, 46 insertions(+), 19 deletions(-)

We had 4 patches there, and a handful were merged upstream. The remaining patches could probably live as configuration files in Puppet, reducing the diff to nil.

2023 update

Update, 2023-05-10: some patches were merged, some weren't, and we had to roll new ones. We have the following diff now:

 debian/changelog           | 22 ++++++++++++++++++++++
 debian/compat              |  2 +-
 debian/control             |  5 ++---
 debian/rules               |  3 +--
 debian/ud-replicate.cron.d |  2 +-
 templates/passwd-changed   |  2 +-
 templates/welcome-message  | 41 ++++++++++++++++++++++++++++-------------
 test/test_pass.py          | 10 ++++++++++
 ud-mailgate                |  5 +++--
 ud-replicate               | 11 +++++++++--
 userdir-ldap.conf          |  2 +-
 userdir_ldap/UDLdap.py     |  5 +++++
 userdir_ldap/generate.py   | 22 +++++++++++++++++++++-
 userdir_ldap/ldap.py       |  2 +-
 14 files changed, 106 insertions(+), 28 deletions(-)

We now have five branches left:

• tpo-scrub-0.3.104:
  • 43c67a3 fix URL in passwd-changed template to torproject.org
  • f9f9a67 Set emailappend to torproject.org
  • c77a70b Use https:// in welcome email
  • 6966895 Re-apply tpo changes to Debian's repo
• mailpassword-generate-0.3.104:
  • 6b09f95 distribute mail-passwords in a location dovecot can read
  • 666c050 expand mail-password file fields
  • 5032f73 add simple getter to Account
• hashpass-test-0.3.104, 7ceb72b add tests for ldap.HashPass
• bookworm-build-0.3.104:
  • 25d89bd fix warning about chown(1) call in bookworm
  • 9c49a4a fix Depends to support python3-only installs
  • 1ece069 bump dh compat to 7
  • 90ef120 make this build without python2
• ssh-sk-0.3.104, a722f6f Add support for security key generated ssh public keys (sk- prefix)

The rebase was done with the following steps.

First we laid down a tag because upstream didn't:

git tag 0.3.104 81d0512e87952d75a249b277e122932382b86ff8

Then we created new branches for each old branch and rebased it on that release:

git checkout -b genpass-fix-0.3.104 origin/genpass-fix-0.3.104-pre-dd7f9a3
git rebase 0.3.104
git branch -m hashpass-test-0.3.104

git checkout -b procmail-0.3.104 procmail-0.3.104-pre-dd7f9a3 
git rebase 0.3.104 
git branch -d procmail-0.3.104

git checkout -b mailpassword-generate-0.3.104 origin/mailpassword-generate-0.3.104-pre-dd7f9a3
git rebase 0.3.104 

git checkout -b tpo-scrub-0.3.104 origin/tpo-scrub-0.3.104-pre-dd7f9a3 
git rebase 0.3.104 

git checkout master 
git merge hashpass-test-0.3.104
git merge mailpassword-generate-0.3.104
git merge tpo-scrub-0.3.104

git checkout -b bookworm-build-0.3.104 0.3.104 
git merge bookworm-build-0.3.104

Verifications of the resulting diffs were made with:

git diff master dsa
git diff master origin/master

Then the package was built and tested on forum-test-01, chives, perdulce and alberti:

dpkg-buildpackage

And finally uploaded to db.tpo and git:

git push origin -u hashpass-test-0.3.104
git push origin -u mailpassword-generate-0.3.104
git push origin -u bookworm-build-0.3.104 0.3.104 
git push origin -u tpo-scrub-0.3.104 
git push

Eventually, we merged with upstream's master branch to be able to use micah's patch (in https://gitlab.torproject.org/tpo/tpa/team/-/issues/41166), so we added an extra branch in there.

2024 update

As of 2024-06-03, the situation has not improved:

anarcat@angela:userdir-ldap$ git diff dsa/master  --stat
 .gitlab-ci.yml               | 18 ------------------
 debian/changelog             | 22 ++++++++++++++++++++++
 debian/rules                 |  2 +-
 debian/ud-replicate.cron.d   |  2 +-
 misc/ud-update-sudopasswords |  4 ++--
 templates/passwd-changed     |  2 +-
 templates/welcome-message    | 41 ++++++++++++++++++++++++++++-------------
 test/test_pass.py            | 10 ++++++++++
 ud-mailgate                  | 14 ++++++++------
 ud-replicate                 |  4 ++--
 userdir-ldap.conf            |  2 +-
 userdir_ldap/generate.py     | 49 ++++++++++++++++++++++++++++++++++++++-----------
 12 files changed, 114 insertions(+), 56 deletions(-)

We seem incapable of getting our changes merged upstream at this point. Numerous patches were sent to DSA only to be either ignored, rewritten, or replaced without attribution. It has become such a problem that we have effectively given up on merging the two code bases.

We should acknowledge that some patches were actually merged, but the patches that weren't were so demotivating that it seems easier to just track this as a non-collaborating upstream, with our code as a friendly fork, than pretending there's real collaboration happening.

Our patch set is currently:

• tpo-scrub-0.3.104 (unchanged, possibly unmergeable):
  • 43c67a3 fix URL in passwd-changed template to torproject.org
  • f9f9a67 Set emailappend to torproject.org
  • c77a70b Use https:// in welcome email
  • 6966895 Re-apply tpo changes to Debian's repo
• mailpassword-generate-0.3.104 (patch rewritten upstream, unclear if still needed)
• hashpass-test-0.3.104 (unchanged)
  • 7ceb72b (add tests for ldap.HashPass, 2021-10-27 15:29:30 -0400)
• fix-crash-without-exim-0.3.104 (new)
  • 51716ed (ud-replicate: fix crash when exim is not installed, 2023-05-11 13:53:33 -0400)
• paramiko-workaround-0.3.104-dff949b (new, not sent upstream considering ssh-openssh-87 was rejected)
  • 6233f8e (workaround SSH host key lookup bug in paramiko, 2023-11-21 14:49:46 -0500)
• sshfp-openssh-87 (new, rejected)
  • 651f280 (disable SSHFP record for initramfs keys, 2023-05-10 14:38:56 -0400)
• py3_allowed_hosts_unicode-0.3.104) (new, rewritten upstream, conflicting)
  • 88bb60d (LDAP now returns bytes, fix another comparison in ud-mailgate, 2023-10-12 10:23:53 -0400)
• thunderbird-sequoia-pgp-0.3.105 (new)
  • 4cb6d49 (extract PGP/MIME multipart mime message content correctly, 2024-06-03)
  • 417f78b (fix Sequoia signature parsing, 2024-06-03)
  • ddc8553 (fix Thunderbird PGP/MIME support, 2024-06-03)

Existing patches were not resent or rebased, but were sent upstream unless otherwise noted.

The following patches were actually merged:

• bookworm-build-0.3.104:
  • d0740a9 (fix implicit int to str cast that broke in bookworm (bullseye?) upgrade, 2023-09-13)
  • 25d89bd fix warning about chown(1) call in bookworm
  • 9c49a4a fix Depends to support python3-only installs
  • 1ece069 bump dh compat to 7
  • 90ef120 make this build without python2
• install-restore-crash-0.3.104:
  • 4ab5d83 (fix crash: LDAP returns a string, cast it to an integer, 2023-09-14 10:28:41 -0400)
• procmail-0.3.104-pre-dd7f9a3:
  • 661875e (drop procmail from userdir-ldap dependencies, 2022-02-28 21:15:41 -0500)

This patch are still in development:

• ssh-sk-0.3.104
  • a722f6f Add support for security key generated ssh public keys (sk- prefix).

It should also be noted that some changes are sitting naked on master, without feature branches and have not been submitted upstream. Those are the known cases but there might be others:

• 91e5b2f (add backtrace to ud-mailgate errors, 2024-06-05)
• 65555da (fix crash in sudo password changes@, 2024-06-05)
• 4315593 (fix changes@ support, 2024-06-05)
• 76a22f0 (note the thunderbird patch merge, 2024-06-04)
• e90f16e (add missing sshpubkeys dependency, 2024-06-04)
• d2cb1d4 (fix passhash test since SHA256 switch, 2024-06-04)
• b566604 (make_hmac expects bytes, convert more callers, 2023-09-28)
• f24a9b5 (remove broken coverage reports, 2023-09-28)

2025 update

We had to do an emergency merge to cover for trixie, which upstream added support for recently. We were disappointed to see the thunderbird-sequoia-pgp-0.3.105 and fix-crash-without-exim-0.3.104 ignored upstream, and another patch rejected.

At this point, we're treating our fork as a downstream and are not trying to contribute back upstream anymore. Concretely, this meant the thunderbird-sequoia-pgp-0.3.105 patch broke and had to be dropped from the tree. Other changes were also committed directly to master and not sent upstream, in particular:

• 9edccfa (fix error on fresh install, 2025-04-17)
• 8c4a9f5 (deal with ud-replicate clients newer than central server, 2025-04-17)

Next step is probably planning for ud-ldap retirement and replacement, see tpo/tpa/team#41839 and TPA-RFC-86.

Monitoring and testing

Prometheus checks the /var/lib/misc/thishost/last_update.trace timestamp and warns if a host is more than an hour out of date.

The web and mail servers are checked as per normal policy.

Logs and metrics

The LDAP directory holds a list of usernames, email addresses, real names, and possibly even physical locations. This information gets destroyed when a user is completely removed but can be kept indefinitely for locked out users.

ud-ldap keeps a full copy of all emails sent to changes@db.torproject.org, ping@db.torproject.org and chpasswd@db.torproject.org in /srv/db.torproject.org/mail-logs/. This includes personally identifiable information (PII) like Received-by headers (which may include user's IP addresses), user's email addresses, SSH public keys, hashed sudo passwords, and junk mail. The mail server should otherwise follow normal mail server logging policies.

The web interface keeps authentication tokens in /var/cache/userdir-ldap/web-cookies, which store encrypted username and password information. Those get removed when a user logs out or after 10 minutes of inactivity, when the user returns. It's unclear what happens when a user forgets to logout and fails to return to the site. Web server logs should otherwise follow the normal TPO policy, see the static mirror network for more information on that.

The OpenLDAP server itself (slapd) keeps no logs.

There are no performance metrics recorded for this service.

Backups

There's no special backup procedures for the LDAP server, it is assumed that the on-disk slapd database can be backed up reliably by Bacula.

Other documentation

• our (TPA) userdir-ldap repository
• our (TPA) userdir-ldap-cgi repository
• the DSA wiki has some ud-ldap documentation, see in particular:
  • how to get sudo passwords
  • how to add a guest account
  • how to add an account
  • how to lock an account
• upstream (DSA) userdir-ldap source code
• upstream (DSA) userdir-ldap-cgi source code
• ud - a partial ud-ldap rewrite in Django from 2013-2014, no change since 2017, the announcement for the rewrite
• userdir-ldap-pylons - a partial ud-ldap rewrite in Pylons from 2011, abandoned
• LDAP.com has extensive documentation, for example on LDAP filters

Discussion

Overview

This section aims at documenting issues with the software and possible alternatives.

ud-ldap is decades old (the ud-generate manpage mentions 1999, but it could be older) and is hard to maintain, debug and extend.

It might have serious security issues. It is a liability, in the long term, in particular for those reasons:

• old cryptographic primitives: SHA-1 is used to hash sudo passwords, MD5 is used to hash user passwords, those hashes are communicated over OpenPGP_encrypted email but stored in LDAP in clear-text. There is a "hack" present in the web interface to enforce MD5 passwords on logins, and the mail interface also has MD5 hard-coded for password resets. Blowfish and HMAC-SHA-1 are also used to store and authenticate (respectively) LDAP passwords in the web interface. MD5 is used to hash usernames.

• rolls its own crypto: ud-ldap ships its own wrapper around GnuPG, implementing the (somewhat arcane) command-line dialect. it has not been determined if that implementation is either accurate or safe.

• email interface hard to use: it has trouble with standard OpenPGP/MIME messages and is hard to use for users

• old web interface: it's made of old Perl CGI scripts that uses a custom template format built on top of WML with custom pattern replacement, without any other framework than Perl's builtin CGI module. it uses in-URL tokens which could be vulnerable to XSS attacks.

• large technical debt

  • ud-ldap is written in (old) Python 2, Perl and shell. it will at least need to be ported to Python 3 in the short term.
  • code reuse is minimal across the project.
  • ud-ldap has no test suite, linting or CI of any form.
  • opening some files (e.g. ud-generate) yield so many style warnings that my editor (Emacs with Elpy) disables checks.
  • it is believed to be impossible or at least impractical to setup a new ud-ldap setup from scratch.

• authentication is overly complex: as detailed in the authentication section, with 6 different authentication methods with the LDAP server.

• replicates configuration management: ud-ldap does configuration management and file distribution, as root (ud-generate/ud-replicate), something which should be reserved to Puppet. this might have been justified when ud-ldap was written, in 1999, since configuration management wasn't very popular back then (Puppet was created in 2005, only cfengine existed back then, which was created in 1993)

• difficult to customize: Tor-specific customizations are made as patches to the git repository and require a package rebuild. they are therefore difficult to merge back upstream and require us to run our own fork.

Our version of ud-ldap has therefore diverged from upstream. The changes are not extensive, but they are still present and require a merge every time we want to upgrade the package. At the time of writing, it is:

anarcat@curie:userdir-ldap(master)$ git diff --stat f1e89a3
 debian/changelog           | 18 ++++++++++++++++++
 debian/rules               |  2 +-
 debian/ud-replicate.cron.d |  2 +-
 templates/welcome-message  | 41 ++++++++++++++++++++++++++++-------------
 ud-generate                |  3 ---
 ud-mailgate                |  2 ++
 ud-replicate               |  2 +-
 userdir-ldap-slapd.conf.in |  4 ++--
 userdir-ldap.conf          |  2 +-
 userdir-ldap.schema        |  9 ++++++++-
 10 files changed, 62 insertions(+), 23 deletions(-)

It seems that upstream doesn't necessarily run released code, and we certainly don't: the above merge point had 47 commits on top of the previous release (0.3.96). The current release, as of October 2020, is 0.3.97, and upstream already has 14 commits on top of it.

The web interface is in a similar conundrum, except worse:

22 files changed, 192 insertions(+), 648 deletions(-)

At least the changes there are only on the HTML templates. The merge task is tracked in issue 40062.

Goals

The goal of the current discussion would be to find a way to fix the problems outlined above, either by rewriting or improving ud-ldap, replacing parts of it, or replacing ud-ldap completely with something else, possibly removing LDAP as a database altogether.

Must have

• framework in use must be supported for the foreseeable future (e.g. not Python 2)
• unit tests or at least upstream support must be active
• system must be simpler to understand and diagnose
• single source of truth: overlap with Puppet must be resolved. either Puppet uses LDAP as a source of truth (e.g. for hosts and users) or LDAP goes away. compromises are possible: Puppet could be the source of truth for hosts, and LDAP for users.

Nice to have

• use one language across the board (e.g. Python 3 everywhere)
• reuse existing project's code, for example an existing LDAP dashboard or authentication system
• ditch LDAP. it's hard to understand and uncommon enough to cause significant confusion for users.

Non-Goals

• we should avoid writing our own control panel, if possible

Approvals required

The proposed solution should be adopted unanimously by TPA. A survey might be necessary to confirm our users would be happy with the change as well.

Proposed Solution

TL;DR: three phase migration away from LDAP

1. stopgap: merge with upstream, port to Python 3 if necessary
2. move hosts to Puppet, replace ud-ldap with another user dashboard
3. move users to Puppet (sysadmins) or Kubernetes / GitLab CI / GitLab Pages (developers), remove LDAP and replace with SSO dashboard

The long version...

Short term: merge with upstream, port to Python 3 if necessary

In the short term, the situation with Python 2 needs to be resolved. Either the Python code needs to be ported to Python 3, or it needs to be replaced by something else. That is "urgent" in the sense that Python 2 is already end of life and will likely not be supported by the next Debian release, around summer 2024. Some work in that direction has been done upstream, but it's currently unclear whether ud-ldap is or will be ported to Python 3 in the short term.

The diff with upstream also makes it hard to collaborate. We should make it possible to use directly the upstream package with a local configuration, without having to ship and maintain our own fork.

Update: there has been progress on both of those fronts. Upstream ported to Python 3 (partially?), but scripts (e.g. ud-generate) still have the python2 header. Preliminary tests seem to show that ud-generate might be capable of running under python3 directly as well (ie. it doesn't error).

The diff with upstream has been reduced, see upstream section for details.

Mid term: move hosts to Puppet, possibly replace ud-ldap with simpler dashboard

In the mid-term, we should remove the duplication of duty between Puppet and LDAP, at least in terms of actual file distribution, which should be delegated to Puppet. In practical terms, this implies replacing ud-generate and ud-replicate with the Puppet server and agents. It could still talk with LDAP for the host directory, but at that point it might be better to simply move all host metadata into Hiera.

It would still be nice to retain a dashboard of sorts to show the different hosts and their configurations. Right now this is accomplished with the machines.cgi web interface, but this could probably be favorably replaced by some static site generator. Gandi implemented hieraviz for this (now deprecated) and still maintain a command-line tool called hieracles that somewhat overlaps with cumin and hieraexplain as well. Finally, a Puppet Dashboard could replace this, see issue tpo/tpa/team#31969 for a discussion on that, which includes the suggestion of moving the host inventory display into Grafana, which has already started.

For users, the situation is less clear: we need some sort of dashboard for users to manage their email forward and, if that project ever sees the light of day, their email (submission, IMAP?) password. It is also needed to manage shell access and SSH keys. So in the mid-term, the LDAP user directory would remain.

At this point, however, it might not be necessary to use ud-ldap at all: another dashboard could be use to manage the LDAP database. The ud-mailgate interface could be retired and the web interface replaced with something simpler, like ldap-user-manager.

So hopefully, in the mid term, it should be possible to completely replace ud-ldap with Puppet for hosts and sysadmins, and an already existing LDAP dashboard for user interaction.

Long term: replace LDAP completely, with Puppet, GitLab and Kubernetes, possibly SSO dashboard

In the long term, the situation is muddier: at this stage, our dependence on ud-ldap is either small (just users) or non-existent (we use a different dashboard). But we still have LDAP, and that might be a database we could get rid of completely.

We could simply stop offering shell access to non-admin users. User access on servers would be managed completely by Puppet: only sudo passwords need to be set for sysadmin anyways and those could live inside Hiera.

Users currently requiring shell access would be encouraged to migrate their service to a container image and workflow. This would be backed by GitLab (for source code), GitLab CI/CD (for deployment) and Kubernetes (for the container backend). Shell access would be limited to sysadmins, which would take on orphan services which would be harder to migrate inside containers.

Because the current shell access provided is very limited, it is believe migration to containers would actually be not only feasible but also beneficial for users, as they would possibly get more privileges than they currently do.

Storage could be provided by Ceph and PostgreSQL clusters.

Those are the current services requiring shell access (as per allowedGroups in the LDAP host directory), and their possible replacements:

Those services were successfully replaced:

Note that this implies the TPA team takes over certain services (e.g. Mailman, RT and Schleuder, in the above list). It might mean expanding the sysadmin team to grant access to service admins.

It also implies switching the email service to another, hopefully simpler, dashboard. Alternatively, this could be migrated back into Puppet as well: we already manage a lot of email forwards by hand in there and we already get support requests for people to change their email forward because they do not understand the ud-ldap interface well enough to do it themselves (e.g. this ticket). We could also completely delegate email hosting to a third-party provider, as was discussed in the submission project.

Those are the applications that would need to be containerized for this approach to be completed:

• BridgeDB
• Check/tordnsel
• Collector
• Consensus health
• CiviCRM
• Doctor
• Exonerator
• Gettor
• Metrics
• OnionOO
• Survey
• Translation
• ZNC

This is obviously a quite large undertaking and would need to be performed progressively. Thankfully, it can be done in parallel without having to convert everything in one go.

Alternatively, a single-sign-on dashboard like FreeIPA or Keycloak could be considered, to unify service authentication and remove the plethora of user/password pairs we use everywhere. This is definitely not being served by the current authentication system (LDAP) which basically offers us a single password for all services (unless we change the schema to add a password for each new service, which is hardly practical).

Cost

This would be part of the running TPA budget.

Alternatives considered

The LDAP landscape in the free world is somewhat of a wasteland, thanks to the "embrace and extend" attitude Microsoft has taken to the standard (replacing LDAP and Kerberos with their proprietary Active Directory standard).

Replacement web interfaces

• eGroupWare: has an LDAP backend, probably not relevant
• LDAP account manager: self-service interface non-free
• ldap-user-manager: "PHP web-based interface for LDAP user account management and self-service password change", seems interesting
• GOsa: "administration frontend for user administration"
• phpLDAPadmin: like phpMyAdmin but for LDAP, for "power users", long history of critical security issues
• web2ldap: web interface, python, still maintained, not exactly intuitive
• Fusion Directory

It might be simpler to rewrite userdir-ldap-cgi with Django, say using the django-auth-ldap authentication plugin.

Command-line tools

• cpu: "Change Password Utility", with an LDAP backend, no release since 2004
• ldapvi: currently in use by sysadmins
• ldap-utils: is part of OpenLDAP, has utilities like ldapadd and ldapmodify that work on LDIF snippets, like ldapvi
• shelldap: similar to ldapvi, but a shell!
• splatd: syncs .forward, SSH keys, home directories, abandoned for 10+ years?

Rewrites

• netauth "can replace LDAP and Kerberos to provide authentication services to a fleet of Linux machines. The Void Linux project uses NetAuth to provide authentication securely over the internet"

Single-sign on

"Single-sign on" (SSO) is "an authentication scheme that allows a user to log in with a single ID to any of several related, yet independent, software systems." -- Wikipedia

In our case, it's something that could allow all our applications that use a single source of truth for usernames and passwords. We could also have a single place to manage the 2FA configurations, so that users wouldn't have to enroll their 2FA setup in each application individually.

Here's a list of the possible applications that could do this that we're aware of:

See also mod_auth_openidc for an Apache module supporting OIDC.

A solution could be to deploy Keycloak or some SSO server on top of the current LDAP server to provide other applications with a single authentication layer. Then the underlying backend could be changed to swap ud-ldap out if we need to, replacing bits of it as we go.

Keycloak

Was briefly considered at Debian.org which ended up using GitLab as an identity provider (!). Concerns raised:

• this post mentions "jboss" and:
  • no self service for group or even OIDC clients
  • no U2F (okay, GitLab also still needs to make the step to webauthn)

See also this discussion and this one. Another HN discussion.

LemonLDAP

https://lemonldap-ng.org/

• has a GPG plugin

Others

• LDAP synchronization connector: "Open source connector to synchronize identities between an LDAP directory and any data source, including any database with a JDBC connector, another LDAP server, flat files, REST API..."
• LDAPjs: pure Javascript LDAP client
• GQLDAP: GTK client, abandoned
• LDAP admin: Desktop interface, written in Lazarus/Pascal (!)
• lldap: rust rewrite, incomplete LDAP implementation, has a control panel
• ldap-git-backup: pull slapcat backups in a git repository, useful for auditing purposes, expiration might be an issue

SCIM

LDAP is a "open, vendor-neutral, industry standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network" (Wikipedia). That's quite a mouthful but concretely, many systems have used LDAP as a single source of truth for authentication, relying on it as an external user database (to simplify).

But that's only one way to do centralized authentication, and some folks are reconsidering that approach altogether. A recent player in there is the SCIM standard: "System for Cross-domain Identity Management (SCIM) is a standard for automating the exchange of user identity information between identity domains, or IT systems" (Wikipedia). Again quoting Wikipedia:

One example might be that as a company onboards new employees and separates from existing employees, they are added and removed from the company's electronic employee directory. SCIM could be used to automatically add/delete (or, provision/de-provision) accounts for those users in external systems such as Google Workspace, Office 365, or Salesforce.com. Then, a new user account would exist in the external systems for each new employee, and the user accounts for former employees might no longer exist in those systems.

In other words, instead of treating the user database as an external database, SCIM synchronizes that database to all systems which still retain their own specific user database. This is great because it removes the authentication system as a single point of failure.

SCIM is standardized as RFC7643 and is built on top of REST with data formatted as JSON or XML.