Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Summary: Tails infra merge roadmap.

Note that the actual future work on this is tracked in milestones:

There, the work is broken down in individual issues and "as-built" plans might change. The page here details the original plan agreed upon at the end of 2024, the authoritative version is made of the various milestones above.

Background

In 2023, Tor and Tails started discussing the possibility of a merge and, in that case, how the future of the two infrastructures would look like. The organizational merge happened in July 2024 with a rough idea of the several components that would have to be taken care of and the clarity that merging infrastructures would be a several-years plan. This document intends to build on the work previously done and describe dependencies, milestones and a detailed timeline containing all services to serve as a basis for future work.

Proposal

Goals

Must have

  • A list of all services with:
    • a description of the service and who are the stakehoders
    • the action to take
    • the complexity
    • a list of dependencies or blocks
    • a time estimation
  • A plan to merge the Puppet codebases and servers
  • A list of milestones with time estimates and and indication of ordering

Non-Goals

  • We don't aim to say exactly who will work on what and when

Scope

This proposal is about:

  • all services that the Tails Sysadmins currently maintain: each of these will either be kept, retired, merged with or migrated to existing TPA services (see the terminology below), depending on several factors such as convenience, functionality, security, etc.
  • some services maintained by TPA that may act as a source or target of a merge, or migration.

Terminology

Actions

  • Keep: Services that will be kept and maintained. They are all impacted by Puppet repo/codebase merge as their building blocks will eventually be replaced (eg. web server, TLS, etc), but they'll nevertheless be kept as fundamental for the work of the Tails Team.
  • Merge: Services that will be kept, are already provided by Tails and TPA using the same software/system, and for which keeping only depends on migration of data and, eventually, configuration.
  • Migrate: Services that are already provided by TPA with a different software/system and need to be migrated.
  • Retire: Services that will be shutdown completely.

Complexity

  • Low: Services that will either be kept as is or for which merging with a Tor service is fairly simple
  • Medium: Services that require either a lot more discussion and analysis or more work than just flipping a switch
  • High: Core services that are already complex on one or both sides but that we still can't manage separately in the long term, so we need to make some hard choices and lots of work to merge

Keep

APT snapshots

BitTorrent

  • Summary: Transmission server used to seed images.
  • Interest-holders: Tails Team
  • Action: Keep
  • Complexity: Low
  • Constraints:
  • References:

HedgeDoc

  • Summary: Collaborative pads with several useful features out of the box.
  • Interest-holders: Tails Team
  • Action: Keep
  • Complexity: Low
  • Constraints:
  • References:
    • https://pad.tails.net

ISO history

  • Summary: Archive of all Tails ISO images, useful for reproducible builds.
  • Interest-holders: Tails Team
  • Action: Keep
  • Complexity: Low
  • Constraints:
  • References:

Schleuder

  • Summary: Tails' and Tor's Schleuder lists.
  • Interest-holders: Tails Team, Community Council
  • Action: Keep
  • Complexity: Low
  • Constraints:
  • References:

Tor Browser archive

  • Summary: Archive of Tor Browser binaries, used for development and release management.
  • Interest-holders: Tails Team
  • Action: Keep
  • Complexity: Low
  • Constraints:
  • References: ∅

Whisperback

  • Summary: Postfix Onion service used to receive bug reports sent directly from the Tails OS.
  • Interest-holders: Tails Team
  • Action: Keep
  • Complexity: Low
  • Constraints:
  • References:

Merge

APT repository

  • Summary: Contains Tails-specific packages, used for development and release management.
  • Interest-holders: Tails Team
  • Action: Merge
  • Complexity: Medium
  • Constraints: ∅
  • References:

Authentication

Colocations

  • Summary:
    • SEACCP: 3 main physical servers (general services and Jenkins CI), USA.
    • Coloclue: 2 small physical servers for backups and some redundancy, Netherlands.
    • PauLLA: dev server, France.
    • Puscii: VM for secondary DNS, Netherlands.
    • Tachanka!: VMs for monitoring and containerized services, USA, somewhere else.
  • Interest-holders: TPA
  • Action: Keep
    • No big changes initially: we'll keep all current PoPs
    • Credentials will be stored in the merged Password Store
    • Documentation and onboarding process will will be consolidated
    • We'll keep a physical machine for development and testing
    • Maybe retire some PoPs if they become empty with retirements/merges
  • Complexity: Low
  • Constraints:
  • References: ∅

Documentation

  • Summary: Public and private Sysadmins' documentation
  • Interest-holders: TPA
  • Action: Merge
    • Get rid of git-remote-gcrypt:
      • Move public info as is to the tpo/tpa/tails/sysadmin wiki
      • Examples of private info that should not be made public: meetings/, planning/, `processes/hiring
      • Archive tpo/tpa/tails/sysadmin-private:
        • What remains there is private history that shouldn't be publicly shared
        • The last people with access to that repo will continue to have access, as long as they still have their private keys
    • Move sysadmin doc from the Tails website to tpo/tpa/tails/sysadmin
    • Rewrite what's left on the fly into Tor's doc as we merge
  • Complexity: Low
  • Constraints: ∅
  • References:

GitLab

  • Summary: Tails has a GitLab instance hosted by a 3rd-party. Some sysadmins' repositories have already been migrated, at this point.
  • Interest-holders: TPA
  • Action: Merge
    • Not before Jan 2025 (due to Tails internal merge timeline)
    • Make sure to somehow archive and not move some obsolete historic projects (eg. accounting, fundraising, summit)
    • Adopt gitlabracadabra to manage Tor's GitLab
  • Complexity: Medium
  • Constraints: ∅
  • References: ∅

LimeSurvey

  • Summary: Mainly used by the UX Team.
  • Interest-holders: UX Team
  • Action: Merge
  • Complexity: Medium
  • Constraints: ∅
  • References:

Mailman

  • Summary: Public mailing listsm, hosted at autistici/inventati.
    • amnesia-news@boum.org
    • tails-dev@boum.org
    • tails-testers@boum.org
    • tails-l10n@boum.org
  • Interest-holders: Tails Team, Community Team
  • Action: Merge
    • Migrate away from the boum.org domain
    • Merge into Tor's Mailman 3
  • Complexity: Medium
  • Constraints: ∅
  • References:
    • https://tails.net/about/contact/index.en.html#public-mailing-lists

MTA

  • Summary: Postfix and Schleuder
  • Interest-holders: TPA
  • Action: Merge
    • Merge Postfix into Tor's MTA
    • Schleuder will be kept
  • Complexity: Medium
  • Constraints: ∅
  • References:

Password Store

  • Summary: Password store containing Sysadmins credentials and secrets.
  • Interest-holders: TPA
  • Action: Merge
  • Complexity: Low
  • Constraints: ∅
  • References: ∅

Puppet Server

  • Summary: Puppet 7, OpenPGP signed commits, published repositories, EYAML for secrets.
  • Interest-holders: TPA
  • Action: Merge
  • Complexity: High
  • Constraints:
    • Blocked by Tor upgrade to Puppet 7
    • Blocks everything we'll "keep", plus Backups, TLS, Monitoring, Firewall, Authentication
  • References:

Registrars

  • Summary: Njal.la
  • Interest-holders: TPA, Finances
  • Action: Keep
    • No big changes initially: we'll keep all current registrars
    • Credentials will be stored in the merged Password Store
    • Documentation needs to be consolidated
  • Complexity: Low
  • Constraints:
  • References: ∅

Shifts

Web servers

  • Summary: Mostly Nginx (voxpupuli module) and some Apache (custom implementation)
  • Interest-holders: TPA
  • Action: Merge
  • Complexity: Medium
  • Constraints:
  • References: ∅

Security Policy

  • Summary: Ongoing adoption by TPA
  • Interest-holders: TPA
  • Action: Merge
  • Complexity: High
  • Constraints: ∅
  • References: tpo/tpa/team#41727

Weblate

  • Summary: Translations are currently made by volunteers and the process is tightly coupled with automatic updating of PO files in the Tails repository (done by IkiWiki and custom code).
  • Interest-holders: Tails Team, Community Team
  • Action: Merge
    • May help mitigate certain risks (eg. Tails Issue 20455, Tails Issue 20456)
    • Tor already has community and translation management processes in place
    • Pending decision:
      • Option 1: Move Tor's Weblate to Tails' self-hosted instance (need to check with Tor's community/translation team for potential blockers for self-hosting)
      • Option 2: Move Tails Weblate to Tor's hosted instance (needs a plan to change the current Translation platform design, as it depends on Weblate being self-hosted)
      • Whether to move the staging website build to GitLab CI and use the same mechanism as the main website build.
  • Complexity: High
  • Constraints: ∅
  • References:

Website

  • Summary: Lives in the main Tails repository and is built and deployed by the GitLab CI using a patched IkiWiki.
  • Interest-holders: Tails Team
  • Action: Merge
    • Change deployment to the Tor's CDN
    • Retire the mirror VMs in Tails infra.
    • Postpone retirement of IkiWiki to a future discussion (see reference below)
    • Consider splitting the website from the main Tails repository
  • Complexity: Medium
  • Constraints:
    • Blocks migration of DNS
    • Requires po4a from Bullseye
    • Requires ikiwiki from https://deb.tails.boum.org (relates to the merge of the APT repository)
  • References:
    • https://gitlab.tails.boum.org/tails/tails/-/issues/18721
    • https://gitlab.tails.boum.org/sysadmin-team/container-images/-/blob/main/ikiwiki/Containerfile

Migrate

Backups

  • Summary: Borg backup into an append-only Masterless Puppet client.
  • Interest-holders: TPA
  • Action: Migrate one side to either Borg or Bacula
    • Experiment with Borg in Tor
    • Choose either Borg or Bacula and migrate everything to one of them
    • Create a plan for compromised servers scenario
  • Complexity: Medium
  • Constraints:
  • References:

Calendar

  • Summary: Only the Sysadmins calendar is left to retire.
  • Interest-holders: TPA, Tails Team
  • Action: Migrate to Nextcloud
  • Complexity: Low
  • Constraints: ∅
  • References:
    • tpo/tpa/team#41836

DNS

  • Summary: PowerDNS:
    • Primary: dns.lizard
    • Secondary: teels.tails.net (at Puscii)
    • MySQL replication
    • LUA records to only serve working mirrors
  • Interest-holders: TPA
  • Action: Migrate
    • Migrate into a simpler design
    • Migrate to either tor's configuration or, if impractical, use tails' PowerDNS as primary
    • Blocked by the merge of Puppet Server.
  • Complexity: High
  • Constraints:
  • References:

EYAML

  • Summary: Secrets are stored encrypted in EYAML files in the Tails Puppet codebase.
  • Interest-holders: TPA
  • Action: Keep for now, then decide whether to Migrate
    • We want to have experience with both before deciding what to do
  • Complexity: Medium
  • Constraints:
  • References: ∅

Firewall

git-annex

  • Summary: Currently used as data backend for https://torbrowser-archive.tails.net and https://iso-history.tails.net, blocker for Gitolite retirement.
  • Interest-holders: Tails Team
  • Action: Migrate to GitLab's Git LFS
  • Complexity: Low
  • Constraints:
  • References:

Gitolite

  • Summary: Provides repositories used by the Tails Team for development and release management, as well as data sources for the website.
  • Interest-holders: TPA, Tails Team
  • Action: Migrate to GitLab
    • etcher-binary: Obsolete (already migrated to GitLab)
    • gitlab-migration-private: Migrate to GitLab and archive
    • gitolite-admin: Obsolete (after migration of other repos)
    • isos: Migrate to GitLab and Git LFS
    • jenkins-jobs: Migrate to GitLab (note: has hooks)
    • jenkins-lizard-config: Obsolete
    • mirror-pool-dispatcher: Obsolete
    • myprivatekeyispublic/testing: Obsolete
    • promotion-material: Obsolete (already migrated to GitLab)
    • tails: Migrate to GitLab (note: has hooks)
    • test: Obsolete
    • torbrowser-archive: Migrate to GitLab and Git LFS
    • weblate-gatekeeper: Migrate to GitLab (note: has hooks)
  • Complexity: Medium
  • Constraints:
  • References:
    • tpo/tpa/team#41837

Jenkins

  • Summary: One Jenkins Controller and 12 Jenkins Agents.
  • Interest-holders: Tails Team
  • Action: Migrate to GitLab CI
  • Complexity: High
  • Constraints:
    • Blocks the retirement of VPN
  • References:

Mirror pool

  • Summary: Tails currently distributes images and updates via volunteer mirrors that pull from an Rsync server. Selection of the closest mirror is done using Mirrorbits.
  • Interest-holders: TPA
  • Action: Migrate to Tor's CDN:
    • Advantages:
      • Can help mitigate certain risks
      • Improves the release management process if devs can push to the mirrors (as opposed to wait for 3rd-party mirrors to sync)
    • Disadvantages:
      • Bandwidth costs
      • Less global coverage
      • Less volunteer participation
  • Complexity: Medium
  • Constraints: ∅
  • References:
    • https://tails.net/contribute/design/mirrors/
    • https://gitlab.torproject.org/tpo/tpa/tails/sysadmin/-/issues/18117
    • Tor's CDN
    • Other options discussed while dealing with router overload caused by Tails mirrors

Monitoring

  • Summary: Icinga2 and Icingaweb2.
  • Interest-holders: TPA
  • Action: Migrate to Prometheus
  • Complexity: High
  • Constraints:
  • References:

TLS

XMPP bot

  • Summary: It's only feature is to paste URLs and titles on issue mentions.
  • Interest-holders: Tails Team
  • Action: Migrate to the same bot used by TPA
  • Complexity: Low
  • Constraints:
    • Blocked by the migration of XMPP
  • References:

XMPP

Virtualization

  • Summary: Libvirt config is managed by Puppet, VM definitions not, custom deploy script.
  • Interest-holders: TPA
  • Action: Keep, as legacy
  • Complexity: Low
    • Treat Tails' VMs as legacy and do not create new ones.
    • New hosts and VMs will be created in Ganeti.
    • If/when hosts become empty, consider whether to retire them or make them part of Ganeti clusters
  • Constraints:
  • References: ∅

Retire

Bitcoin

  • Summary: Tails' Bitcoin wallet.
  • Interest-holders: Finances
  • Action: Retire, hand-over to Tor accounting
  • Complexity: Low
  • Constraints: ∅
  • References:

Tor Bridge

  • Summary: Not used for dev, but rather to "give back to the community".
  • Interest-holders: Tor Users
  • Action: Retire
  • Complexity: Low
  • Constraints: ∅
  • References:

VPN

  • Summary: Tinc connecting VMs hosted by 3rd-parties and physical servers.
  • Interest-holders: TPA
  • Action: Retire
    • Depending on timeline, could be replaced by Wireguard mesh (if Tor decides to implement it)
  • Complexity: High
  • Constraints:
    • Blocked by the migration of Jenkins
  • References:

Dependency graph

flowchart TD
    classDef keep fill:#9f9,stroke:#090,color:black;
    classDef merge fill:#adf,stroke:#00f,color:black;
    classDef migrate fill:#f99,stroke:#f00,color:black;
    classDef white fill:#fff,stroke:#000;

    subgraph Captions [Captions]
      Keep; class Keep keep
      Merge; class Merge merge
      Migrate; class Migrate migrate
      Retire; class Retire retire

      Low([Low complexity])
      Medium>Medium complexity]
      High{{High complexity}}
    end

    subgraph Independent [Independent of Puppet]
        Calendar([Calendar]) ~~~
        Documentation([Documentation]) ~~~
        PasswordStore([Password Store]) --> Colocations([Colocations]) & Registrars([Registrars]) ~~~
        Mailman>Mailman lists] ~~~
        GitLab>GitLab] ~~~
        Shifts>Shifts] ~~~
        SecurityPolicy{{Security Policy}}
    end

    subgraph Parallelizable
        AptRepository>APT repository] ~~~
        LimeSurvey>LimeSurvey] ~~~
        Weblate{{Weblate}} ~~~
        git-annex([git-annex]) -->
        Gitolite([Gitolite]) ~~~
        Jenkins{{Jenkins}} -->
        VPN{{VPN}}
        MTA>MTA] ~~~
        Website>Website] ~~~
        MirrorPool{{Mirror pool}} ~~~
        XMPP>XMPP] -->
        XmppBot([XMPP bot]) ~~~
        Bitcoin([Bitcoin]) ~~~
        TorBridge([Tor Bridge])
    end

    subgraph Puppet [Puppet repo and server]
    direction TB
        TorPuppet7>Upgrade Tor's Puppet Server to Puppet 7] --> PuppetModules & CommitSigning & Eyaml
        PuppetModules>Puppet modules] --> HybridPuppet
        Eyaml([EYAML]) --> HybridPuppet
        CommitSigning>Commit signing] --> HybridPuppet
        HybridPuppet{{Puppet Server}}
    end

    subgraph Basic [Basic system functionality]
        WebServer>Web servers] ~~~
        Authentication{{Authentication}} ~~~
        Backups([Backups]) --> Monitoring{{Monitoring}}
        TLS([TLS]) --> Monitoring ~~~
        DNS{{DNS}} ~~~
        Firewall{{Firewall}}
        Authentication ~~~ TLS
    end

    subgraph ToKeep [Services to keep]
        direction TB;
        HedgeDoc([HedgeDoc]) ~~~
        IsoHistory([ISO history]) ~~~
        TbArchive([Tor Browser archive]) ~~~
        BitTorrent([BitTorrent]) ~~~
        WhisperBack([WhisperBack]) ~~~
        Schleuder([Schleuder]) ~~~
        AptSnapshots{{APT snapshots}}
    end

    subgraph Deferred
        EyamlTrocla>EYAML or Trocla]
    end

    Captions ~~~ Puppet & Independent & Parallelizable
    Independent ~~~~~ PuppetCodebase
    Puppet --> ToKeep & Basic --> Deferred
    Deferred --> PuppetCodebase{{Consolidated Puppet codebase}}
    Parallelizable ----> PuppetCodebase
    PuppetCodebase --> Virtualization([Virtualization])

    class AptRepository merge
    class AptSnapshots keep
    class Authentication merge
    class Backups migrate
    class BitTorrent keep
    class Bitcoin retire
    class Calendar migrate
    class Colocations keep
    class CommitSigning keep
    class DNS migrate
    class DNS migrate
    class Documentation merge
    class Eyaml keep
    class EyamlTrocla migrate
    class Firewall migrate
    class GitLab merge
    class Gitolite migrate
    class HedgeDoc keep
    class HybridPuppet merge
    class IsoHistory keep
    class Jenkins migrate
    class LimeSurvey merge
    class MTA merge
    class Mailman merge
    class MirrorPool migrate
    class Monitoring migrate
    class PasswordStore merge
    class PuppetCodebase merge
    class PuppetModules merge
    class Registrars keep
    class Schleuder keep
    class SecurityPolicy merge
    class Shifts merge
    class TLS migrate
    class TbArchive keep
    class TorBridge retire
    class TorPuppet7 keep
    class VPN retire
    class Virtualization keep
    class WebServer merge
    class Weblate merge
    class Website merge
    class WhisperBack keep
    class XMPP migrate
    class XmppBot migrate
    class git-annex migrate

Timeline

2024

Milestone: %"TPA-RFC-73: Tails merge (2024)"

2025

Milestone: %"TPA-RFC-73: Tails merge (2025)"

2026

2027

2028

2029

Alternatives considered

Converge both codebases before merging repositories and Puppet Servers

This approach would have the following disadvantages:

  • keeping two different Puppet codebase repositories in sync is more prone to errors and regressions,
  • no possibility of using exported resources would make some migrations more difficult (eg. Backups, Monitoring, TLS, etc)

References

See the TPA/Tails sysadmins overview document that was used to inform the decision about the merger.