Debian 10 buster was released on July 6th 2019. Tor started the upgrade to buster during the freeze and hopes to complete the process before the stretch EOL, one year after the stable release, so normally around July 2020.

• Procedure
• Service-specific upgrade procedures
  • PostgreSQL
  • RT
• Notable changes
• Issues
  • Pending
  • Resolved
• References
• Per host progress

Procedure

Before upgrading a box, it might be preferable to coordinate with the service admins to see if the box will survive the upgrade. See howto/upgrades for the list of teams and how they prefer to handle that process.

1. Preparation:

   : reset to the default locale
   export LC_ALL=C.UTF-8 &&
   sudo apt install ttyrec screen debconf-utils apt-show-versions deborphan &&
   sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec
   

2. Backups and checks:

   ( umask 0077 &&
     tar cfz /var/backups/pre-buster-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states $( [ -e /var/lib/aptitude/pkgstates ] && echo /var/lib/aptitude/pkgstates ) /var/cache/debconf &&
     dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-buster.txt &&
     debconf-get-selections > /var/backups/debconf-selections-pre-buster.txt
   ) &&
   apt-mark showhold &&
   dpkg --audit &&
   : look for dkms packages and make sure they are relevant, if not, purge. &&
   ( dpkg -l '*dkms' || true ) &&
   : make sure backups are up to date in Nagios &&
   printf "End of Step 2\a\n"
   

3. Enable module loading (for ferm) and test reboots:

   systemctl disable modules_disabled.timer &&
   puppet agent --disable "running major upgrade" &&
   shutdown -r +1 "rebooting with module loading enabled"
   
   export LC_ALL=C.UTF-8 &&
   sudo ttyrec -a -e screen /var/log/upgrade-buster.ttyrec
   

4. Perform any pending upgrade and clear out old pins:

   : Check for pinned, on hold, packages, and possibly disable &&
   rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
   rm -f /etc/apt/sources.list.d/testing.list &&
   rm -f /etc/apt/sources.list.d/stretch-backports.list &&
   rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
   apt update && apt -y upgrade &&
   : list kernel images and purge unused packages &&
   dpkg -l 'linux-image-*' &&
   : look for packages from backports, other suites or archives &&
   : if possible, switch to official packages by disabling third-party repositories &&
   apt-show-versions | grep -v /stretch | grep -v 'not installed$' &&
   printf "End of Step 4\a\n"
   

5. Check free space (see this guide to free up space), disable auto-upgrades, and download packages:

   systemctl stop apt-daily.timer &&
   sed -i 's/stretch/buster/g' /etc/apt/sources.list.d/* &&
   (apt update && apt -o APT::Get::Trivial-Only=true dist-upgrade || true ) &&
   df -h &&
   apt -y -d upgrade &&
   apt -y -d dist-upgrade &&
   printf "End of Step 5\a\n"
   

6. Actual upgrade run:

   apt install -y dpkg apt &&
   apt install -y ferm &&
   apt dist-upgrade -y &&
   printf "End of Step 6\a\n"
   

7. Post-upgrade procedures:

   apt-get update --allow-releaseinfo-change &&
   apt-mark manual git &&
   apt --purge autoremove &&
   apt purge $(for i in apt-transport-https dh-python emacs24-nox gnupg-agent libbind9-140 libcryptsetup4 libdns-export162 libdns162 libevent-2.0-5 libevtlog0 libgdbm3 libicu57 libisc-export160 libisc160 libisccc140 libisccfg140 liblvm2app2.2 liblvm2cmd2.02 liblwres141 libmpfr4 libncurses5 libperl5.24 libprocps6 libpython3.5 libpython3.5-minimal libpython3.5-stdlib libruby2.3 libssl1.0.2 libunbound2 libunistring0 python3-distutils python3-lib2to3 python3.5 python3.5-minimal ruby-nokogiri ruby-pkg-config ruby-rgen ruby-safe-yaml ruby2.3 sgml-base xml-core git-core gcc-6-base:amd64 nagios-plugins-basic perl-modules-5.24 libsensors4:amd64 grub2 iproute libncursesw5 libustr-1.0-1; do dpkg -l "$i" 2>/dev/null | grep -q '^ii' && echo "$i"; done) &&
   dpkg --purge libsensors4:amd64 syslog-ng-mod-json || true &&
   puppet agent --enable &&
   (puppet agent -t || true) &&
   (puppet agent -t || true) &&
   systemctl start apt-daily.timer &&
   printf "End of Step 7\a\n" &&
   shutdown -r +1 "rebooting to get rid of old kernel image..."
   

8. Post-upgrade checks:

   export LC_ALL=C.UTF-8 &&
   sudo ttyrec -a -e screen /var/log/upgrade-buster.ttyrec
   
   # review and purge old packages, including kernels
   apt --purge autoremove
   dsa-check-packages | tr -d ,
   while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done
   apt --purge autoremove
   dpkg -l '*-dbg' # look for dbg package and possibly replace with -dbgsym
   apt clean
   # review packages that are not in the new distribution
   apt-show-versions | grep -v /buster
   printf "End of Step 8\a\n"
   shutdown -r +1 "testing reboots one final time"
   

9. Change the hostgroup of the host to buster in Nagios (in tor-nagios/config/nagios-master.cfg on git@git-rw.tpo)

Service-specific upgrade procedures

PostgreSQL

PostgreSQL is special and needs to be upgraded manually.

1. make a full backup of the old cluster:

   ssh -tt bungei.torproject.org 'sudo -u torbackup postgres-make-one-base-backup $(grep ^meronense.torproject.org $(which postgres-make-base-backups ))'
   

   The above assumes the host to backup is meronense and the backup server is bungei. See service/postgresql for details of that procedure.

2. Once the backup completes, move the directory out of the way and recreate it:

   ssh bungei.torproject.org "mv /srv/backups/pg/meronense /srv/backups/pg/meronense-9.6 && sudo -u torbackup mkdir /srv/backups/pg/meronense"
   

3. do the actual cluster upgrade, on the database server:

   export LC_ALL=C.UTF-8 &&
   printf "about to drop cluster main on postgresql-11, press enter to continue" &&
   read _ &&
   pg_dropcluster --stop 11 main &&
   pg_upgradecluster -m upgrade -k 9.6 main &&
   for cluster in `ls /etc/postgresql/9.6/`; do
       mv /etc/postgresql/9.6/$cluster/conf.d/* /etc/postgresql/11/$cluster/conf.d/
   done
   

4. make sure the new cluster isn't backed up by bacula:

   touch /var/lib/postgresql/11/.nobackup
   

   TODO: put in Puppet.

5. change the cluster target in the backup system, in tor-puppet, for example:

   --- a/modules/postgres/manifests/backup_source.pp
   +++ b/modules/postgres/manifests/backup_source.pp
   @@ -30,7 +30,7 @@ class postgres::backup_source {
           case $hostname {
                   'gitlab-01': {
                   }
   -               'subnotabile', 'bacula-director-01': {
   +               'meronense', 'subnotabile', 'bacula-director-01': {
                           postgres::backup_cluster { $::hostname:
                                   pg_version => '11',
                           }
   

6. change the postgres version in tor-nagios as well:

   --- a/config/nagios-master.cfg
   +++ b/config/nagios-master.cfg
   @@ -354,7 +354,7 @@ servers:
      meronense:
        address: 94.130.28.195
        parents: kvm4
   -    hostgroups: computers, buster, syslog-ng-hosts, hassrvfs, apache2-hosts, apache-https-host, postgres96-hosts, hassrvfs90
   +    hostgroups: computers, buster, syslog-ng-hosts, hassrvfs, apache2-hosts, apache-https-host, postgres11-hosts, hassrvfs90
      # db.tpo
      alberti:
        address: 94.130.28.196
   

7. once everything works okay, remove the old packages:

   apt purge postgresql-9.6 postgresql-client-9.6
   

8. purge the old backups directory after a week:

   ssh bungei.torproject.org "echo 'rm -r /srv/backups/pg/meronense-9.6/' | at now + 7day"
   

9. make a new full backup of the new cluster:

   ssh -tt bungei.torproject.org 'sudo -u torbackup postgres-make-one-base-backup $(grep ^meronense.torproject.org $(which postgres-make-base-backups ))'
   

RT

RT is not managed by dbconfig, or at least it needs a kick for some upgrades. In the 4.4.1 to 4.4.3 buster upgrade (4.4.2, really), the following had to be ran:

rt-setup-database-4 --action upgrade --upgrade-from 4.4.1 --upgrade-to 4.4.2 --dba rtuser

The password was in /etc/request-tracker4/RT_SiteConfig.d/51-dbconfig-common.pm. See issue 40054 for an example problem that happened when that was forgotten.

Notable changes

Here is a subset of the notable changes in this release, along with our risk analysis and notes:

Many packages were removed from Buster. Anarcat built an exhaustive list on May 16th 2019, but it's probably changed since then. See also the noteworthy obsolete packages list.

Python 2 is unsupported upstream since January 1st 2020. We have a significant number of Python scripts that will need to be upgraded. It is unclear what will happen to Python 2 in Debian in terms of security support for the buster lifetime.

Issues

Pending

• upgrading restarts openvswitch will mean all guests lose network

• At least on kvm5, brpub was having issues. Either ipv4 or ipv6 address was missing, or the v6 route to the guests was missing. Probably because the ipv6 route setting failed since we set a prefsrc and that was only brought up later?

  Rewrote /etc/network/interfaces to set things up more manually. On your host, check if brpub has both ipv4 and ipv6 addresses after boot before launching VMs, and that is has an ipv6 route into brpub with the configured prefsrc address. If not, fiddle likewise.

  See ticket #31083 for followup on possible routing issues.

• On physical hosts witch /etc/sysfs.d/local-io-schedulers.conf, note that deadline no longer existsts. Probably it is also not necessary as Linux might pick the right scheduler anyhow.

• the following config files had conflicts but were managed by Puppet so those changes were ignored for now. eventually they should be upgraded in Puppet as well.

   /etc/bacula/bacula-fd.conf
   /etc/bind/named.conf.options
   /etc/default/stunnel4
   /etc/ferm/ferm.conf
   /etc/init.d/stunnel4
   /etc/nagios/nrpe.cfg
   /etc/ntp.conf
   /etc/syslog-ng/syslog-ng.conf
  

• ferm fails to reload during upgrade, with the following error:

   Couldn't load match `state':No such file or directory
  

• Puppet might try to downgrade the sources.list files to stretch or n/a for some reason, just re-run Puppet after fixing the sources.list files, it will eventually figure it out.

• The official list of known issues

Resolved

• apt-get complains like this after upgrade (bug #929248):

   E: Repository 'https://mirrors.wikimedia.org/debian buster InRelease' changed its 'Suite' value from 'testing' to 'stable'
  

  the following workaround was added to the upgrade instructions, above, but might be necessary on machines where this procedure was followed before the note was added:

   apt-get update --allow-releaseinfo-change
  

• the following config files were updated to buster:

   /etc/logrotate.d/ulogd2
   /etc/ssh/sshd_config
  

• Puppet was warning with the following when running against a master running stretch, harmlessly:

   Warning: Downgrading to PSON for future requests
  

References

Note: the official upgrade guide and release notes not available at the time of writing (2019-04-08) as the documentation is usually written during the freeze and buster is not there yet.

• Official guide
• Release notes
• Koumbit guide
• DSA guide
• Solution proposal to automate this

Per host progress

To followup on the upgrade, search for "buster upgrade" in the GitLab boards, which is fairly reliable.

List of servers to upgrade can be obtained with:

curl -s -G http://localhost:8080/pdb/query/v4 --data-urlencode 'query=nodes { facts { name = "lsbdistcodename" and value = "stretch" }}' | jq .[].certname | sort

Policy established in howto/upgrades.