Move all of Jenkins to Hetzner and retire dragon.tails.net
Context
The Jenkins service used by the Tails Team for development has been suffering from hardware issues since 2021, when the newly acquired hardware needed to have its capacity tweaked down to prevent it from overheating.
In mid-2025, one of the physical machines that used to host parts of the service started failing with crashes and disks disappearing from the OS. The first hypothesis of disk failure was ruled out because issues persisted after disk replacement. Another hypothesis related to overheating was also ruled out because the issues also happened with a cool, freshly booted machine. That scenario led TPA to consider that machine broken and decide to retire it.
Meanwhile, TPA made a budget request for replacement of three physical servers that host Tails services: the one described above is broken (dragon.tails.net), another one is already 13 years-old (lizard.tails.net), and the 3rd one has been operating with disabled cores since it was deployed 3.5 years ago, as a workaround to avoid overheating (iguana.tails.net). The requested budget was unfeasible at the time, and the alternative solution was to rent 2 physical servers at Hetzner as a stopgap solution.
At the moment, the Jenkins service is split in two different points of presence (Riseup in the US and Hetzner in Germany), but now the CI jobs suffer from a network transfer speed bottleneck, most probably due to multiple causes such as routing issues and VPN overhead.
Decision
TPA will move all of Jenkins to the two rented Hetzner physical machines. This will remove the need for data transfer between different points of presence, and allow for retirement of the broken machine (dragon.tails.net) as well as for repurposing one physical machine (iguana.tails.net).
More concretely, TPA will:
- move the Jenkins controller and its supporting mail service to Hetzner
- retire dragon.tails.net and its virtual machines
- retire the Jenkins agents in iguana.tails.net
- request that iguana.tails.net is moved to a better case with more fans, and check whether that allows for more CPU cores to be enabled
- repurpose iguana.tails.net and use it to run GitLab Runners for Tails
Consequences
The retired machine will be removed from the datacenter, which will save hosting cost of 75 USD/month.
More Information
The work will be done during Jan/Feb 2026 and expected to be completed on time for the release of Tails 7.5 on Feb 26th.
The retired machine will be offered to the Tails Team in case they want to experiment with it somehow else.
Metadata
- status: approved
- decision-date: 2026-02-02
- decision-makers: TPA team lead
- consulted: tpa-team@lists.torproject.org, foundations@tails.net, taggart@riseup.net
- informed: tor-project@lists.torproject.org
- forum-url: https://gitlab.torproject.org/tpo/tpa/tails-sysadmin/-/issues/18238