Automated ISO/IMG builds and tests on Jenkins

:warning: This service will change during policy/tpa-rfc-73-tails-infra-merge-roadmap and this page should be updated when that happens.

• Access
  • jenkins.lizard
  • isoworker2.dragon
  • isoworker3.dragon
  • isoworker4.dragon
  • isoworker5.dragon
  • isoworker6.iguana
  • isoworker7.iguana
  • isoworker8.iguana
  • isoworker9.fsn-libvirt-01
  • isoworker10.fsn-libvirt-01
  • isoworker11.fsn-libvirt-01
  • isoworker12.fsn-libvirt-01
  • isoworker13.fsn-libvirt-02
  • isoworker14.fsn-libvirt-02
  • isoworker15.fsn-libvirt-02
  • isoworker16.fsn-libvirt-02
• Configuration
  • Controller
  • Manual controller reboots
  • Agents
  • Web server
• Upgrades
  • Upgrade policy
  • Upgrade procedure
• Agent to controller connections
• Generating jobs
• Passing parameters through jobs
• Builds
• Tests

Access

jenkins.lizard

• SSH onion service: fw4ntog3yqvtsjielgqhzqiync5p4gjz4t7hg2wp6glym23jwc6vedqd.onion
• SSH fingerprints:
  • SHA256:EtL9m3hZGBPvu/iqrtwa4P/J86nE1at9LwymH66d1JI (ECDSA)
  • SHA256:HEvr0mTfY4TU781SOb0xAqGa52lHPl00tI0mxH5okyE (ED25519)
  • SHA256:sgH1SYzajDrChpu26h24W1C8l+IKYV2PsAxzSGxemGk (RSA)

isoworker2.dragon

• SSH onion service: xdgjizxtbx2nnpxvlv63wedp7ruiycxxd2onivt4u4xyfuhwgz33ikyd.onion
• SSH fingerprints:
  • SHA256:/YUC5h2NM9mMIv8CDgvQff4F1lCcrJEH3eKzSFOMwDA (ECDSA)
  • SHA256:dqHiHIPpgIkraIW8FjNsRxwH8is++/UOA8d8rGcwJd0 (ED25519)
  • SHA256:FOOQTcVtParu2Tr9LT6i9pkXAOPZMLjO/HMmD7G3cQw (RSA)

isoworker3.dragon

• SSH onion service: 3iyrk76brjx3syp2d7etwgsnd7geeikoaowhrfdwn2gk3zax4xavigqd.onion
• SSH fingerprints:
  • SHA256:IStnb3Nmi8Bg8KVjCFFdt1MHopbddEZmo5jxeEuYZf8 (ECDSA)
  • SHA256:wZ3WQOc75f0WuCnoLtXREKuNLheLkAKRnQM2k5xd+X4 (ED25519)
  • SHA256:yJuhwnDwq3pzyxJVQP1U3eXlSUb4xceORopDQhs3vMU (RSA)

isoworker4.dragon

• SSH onion service: 5wpwcpsoeunziylis45ax4zvr7dnwtini6y2id4ixktmlfbjdz4izkyd.onion
• SSH fingerprints:
  • SHA256:IQ2cd5t7D4PigIlbDb50B33OvoWwavKOEmGbnz7gQZ0 (ECDSA)
  • SHA256:bxqMdon5kYpu1/vsw8kYpSIXdsYh6rnDzPmP3j25+W4 (ED25519)
  • SHA256:UYwTZVSqYOU1dpruXfZTs/AO7I7jYPFROd20Z5PeXWc (RSA)

isoworker5.dragon

• SSH onion service: dimld3wfazcimopetopeikrngqc6gzgxn5ozladbr5bwsoqtfj7fzaqd.onion
• SSH fingerprints:
  • SHA256:7tF5FYunYVoFWwzDcShOPKrYqzSbKo56BWQjR++xXrw (ECDSA)
  • SHA256:mT5q/FLyvm24FmRKCGafwaoEaJORYCjZu/3N0Q10X+o (ED25519)
  • SHA256:7CPhM2zZZhTerlCyYLyDEnTltPv9nK7rAmVCRbg64Qg (RSA)

isoworker6.iguana

• SSH onion service: katl2wa6urkwtpkhr4y6z5vit2c3in6hhmvz4lsx3qw22p5px7ae4syd.onion
• SSH fingerprints:
  • SHA256:KTzB+DufG+NISjYL35BjY4cF3ArPMl2iIm/+9pBO0LE (ECDSA)
  • SHA256:t0OboSv/JFmViKKmv8oiFbpURMZdilNK3/LQ99pAQaM (ED25519)
  • SHA256:UHdM8EZ8ZTAxbutXfZYQQLNrxItpmNAKEChreC/bl+o (RSA)

isoworker7.iguana

• SSH onion service: ooqzyfecxzbfram766hkcaf6u4pe4weaonbstz355phsiej4flus7myd.onion
• SSH fingerprints:
  • SHA256:7HmibtchW6iu9+LS7f5bumONrzMIj1toSaYaPRq3FwU root@isoworker7 (ECDSA)
  • SHA256:VSvGkrpw49ywmHrHtEOgHnpFVkUvlfBoxFswY3JeMpk root@isoworker7 (ED25519)
  • SHA256:k+TeoXoEeF3yIFLHorHOKp9MlJWAjkojS3spbToW5/U root@isoworker7 (RSA)

isoworker8.iguana

• SSH onion service: j6slikp4fck5drnkkshjhtwvxbywuf5bruivteojt3b52conum6dskqd.onion
• SSH fingerprints:
  • SHA256:cSLhbY3CSi6h5kQyseuAR64d0EPn0JE3o6rwfIXJqgQ root@isoworker8 (ECDSA)
  • SHA256:iZT9WstFjoX93yphLNS062Vll5KjIQF6Y2FQbc1/prw root@isoworker8 (ED25519)
  • SHA256:dzaWqWYO/4HERtFx2xBhv9S1Jnzv1GjGfHegusEK4X0 root@isoworker8 (RSA)

isoworker9.fsn-libvirt-01

• SSH onion service: c4d57ibn4qejn6lm7pl3l74fjuotg254qy5kr6acw7oor5kpcawswaad.onion
• SSH fingerprints:
  • SHA256:7JedU9WOpshC8/zTikSOOM4QReXwIKbXQttDgWOc8dI root@isoworker9 (ECDSA)
  • SHA256:3DZN3lI2DRU/FgEW5mdEq3azCAgxdQ9gIjzSA0NAbNY root@isoworker9 (ED25519)
  • SHA256:s+M5MjzdkbO0UIt6epp55rJj2ZupR9FgMprEnCrkoi8 root@isoworker9 (RSA)

isoworker10.fsn-libvirt-01

• SSH onion service: ll7pqydikkegnierd43ai7qsca2ot3mmcwximcnjdsgnut64rlabjqad.onion
• SSH fingerprints:
  • SHA256:YHvdxT/1dW00fZxlsH2n39uHFrlSpB1yH/vlhrD/6ys root@isoworker10 (ECDSA)
  • SHA256:DavYFCzp1j3/U006hqN7MIPz7TWiF/XGfHxsdGCEyk8 root@isoworker10 (ED25519)
  • SHA256:/kaC2sHHWnLlY8wYgx6b2lDabD6FLZhc4Y3tBP0mOK0 root@isoworker10 (RSA)

isoworker11.fsn-libvirt-01

• SSH onion service: omqd3fort2zqk7cvl5noabrsu6rraxayuckvyn676elfn7lkm6oow5id.onion
• SSH fingerprints:
  • SHA256:vhQZ+USa+imx2u9qm9W9Ew+u36Iq945nfFtK+4Sr5gU root@isoworker11 (ED25519)
  • SHA256:PJG7CxUH6dP/0V1JhhqJ2DdDKB+NyYcW3EcRRbzk3h8 root@isoworker11 (ECDSA)
  • SHA256:bwk2X4U0GARV1uBE8KQtijvAaKKPOgacS6eRIByDlvM root@isoworker11 (RSA)

isoworker12.fsn-libvirt-01

• SSH onion service: aydh4cz4pljudiidzcayijesf2jbjewmevojud7qgmnjln74vzvaxyid.onion
• SSH fingerprints:
  • SHA256:VKV6J2Yplw5nYmAUFICzbmLp12dg35uw/6MjOoHf68g root@isoworker12 (ECDSA)
  • SHA256:GT7ycAz2TQCHX/hK17RhCfgcyMwAfXkAHz4RO24Jtus root@isoworker12 (ED25519)
  • SHA256:++O4M4Ulu5Lfuk1RDR6GMb+lttxssMEqkhnezfhjFYU root@isoworker12 (RSA)

isoworker13.fsn-libvirt-02

• SSH onion service: 562c5bs5jnehnlc36ymocpd3nu7gdz43usmwz5c4w5qxbwt6oti46uyd.onion
• SSH fingerprints:
  • SHA256:/KFzBNTLeIpJ2jLVGHpKzGnNXa/NpCPfxVLEfzFBq5Q root@isoworker13 (ECDSA)
  • SHA256:enBTQXpDzQ7PNYFK+P+6ylEI9wDpMMNFiKkEdaOUC7Q root@isoworker13 (ED25519)
  • SHA256:CHZGOHrXOGCECWIAvXTyMSTdyo19+SYAVJ2WNfb57dQ root@isoworker13 (RSA)

isoworker14.fsn-libvirt-02

• SSH onion service: 4yvipyvtrb7nmdsrnomrlrwasugcwabzpfyqucm5j2j3mr4y5xrkrdad.onion
• SSH fingerprints:
  • SHA256:yPyC+3ho1DjrUBKXnQPah7VmXOi1xvh+ecNEZq0lT14 root@isoworker14 (ECDSA)
  • SHA256:r/puMoK4v8riXkCnVcquHA9mCpNdduPYG5R6sGj2JRg root@isoworker14 (ED25519)
  • SHA256:KfADgGhIHhHUCX5RX7jDwBNXKBbqIf9Bkkjw3mOrmoA root@isoworker14 (RSA)

isoworker15.fsn-libvirt-02

• SSH onion service: 5wspkgfoakkfv37tag243w6d52hzkzmr5uc74xzw2ydjvucykuwqgxid.onion
• SSH fingerprints:
  • SHA256:dkl7h3S7SeBrYmoLjTo6US5KbqOMCDizpwzhaG3Jja8 root@isoworker15 (ECDSA)
  • SHA256:gtrLSIO4Tv39SJ6DMGg8xVaumY9o7NnoSfGt0Wr5vko root@isoworker15 (ED25519)
  • SHA256:M2wjro+aBFRBJoPc94G5e/pV0JIyuAfTu2e1RqtA4R0 root@isoworker15 (RSA)

isoworker16.fsn-libvirt-02

• SSH onion service: 2mnqjpzqaxw44ikdowpmw5oem3nwta2ydptoehecd44zyozklinjknqd.onion
• SSH fingerprints:
  • SHA256:dGbyptYvItqEpQ1iO6nb+70lgMpbd+S0T4WeVQpDSJQ root@isoworker16 (ECDSA)
  • SHA256:HWIoVUuw2ghzAzo/uxV6ehrpnoxnhXugsckyvAhi/P0 root@isoworker16 (ED25519)
  • SHA256:I7Gvbgk66DFX5p4c4ELIcQ/7vx9PW4VXzjYR5f+c+fA root@isoworker16 (RSA)

Configuration

Controller

• Puppet code.
• YAML jobs configuration lives in a dedicated Git repository; Jenkins Job Builder uses it to configure Jenkins
• Manual configuration (not handled by Puppet):
  • In the Jenkins web interface:
    • Security → Agents → TCP port for inbound agents → Fixed: 42585
    • System → # of executors: 8 (actually, set to the same number of configured agents)
    • System → Git plugin → Global Config user.name Value: jenkins
    • System → Git plugin → Global Config user.email Value: sysadmins@tails.net
    • System → Priority Sorter → Only Admins can edit job priorities: checked
    • Job Priorities → Add 2 job groups:
      • Description: Top priority
        • Jobs to include: Jobs marked for inclusion
        • Job Group Name: 1
        • Priority: 1
      • Description: Test suite
        • Jobs to include: Jobs marked for inclusion
        • Job Name: 2
        • Priority: 2
    • Create one node for each agent, in Nodes → New node:
      • Node name: use the hostname of the agent (eg. "isoworker6.iguana")
      • Number of executors: 1
      • Remote root directory: /var/lib/jenkins
      • Usage: Use this node as much as possible
      • Launch method: Launch agent by connecting it to the controller
      • Disable WorkDir: checked
      • Internal data directory: remoting
      • Availability: Keep this agent online as much as possible
      • Preference of Node: choose a preference depending on the node specs
  • In the Jenkins VM:
    • For backups: Make sure there exists an SSH key for root and it's public part is configured in profile::tails::backupserver::backupagents for stone.tails.net (or the current backup server).
    • Document the onion server address and SSH fingerprints for the VM.
  • The configuration for the build_IUKs job is only stored in /var/lib/jenkins and nowhere else.
  • Create 4 different "Views":
    • RM:
      • Use a regular expression to include jobs into the view
        • Regular expression: ^(build_IUKs|(reproducibly_)?(test|build)_Tails_ISO_(devel|stable|testing|feature-trixie|experimental|feature-tor-nightly-master)(-force-all-tests)?)
    • Tails Build:
      • Use a regular expression to include jobs into the view
        • Regular expression: build_Tails_ISO_.*
    • Tails Build Reproducibility:
      • Use a regular expression to include jobs into the view
        • Regular expression: reproducibly_build_.*
    • Tails Test Suite:
      • Use a regular expression to include jobs into the view
        • Regular expression: test_Tails_ISO_.*

Manual controller reboots

Some times, the Jenkins controller needs to be manually rebooted (example), so we have sudo config in place that allows the jenkins user in the Jenkins controller VM to do that.

When logged in to the controller as the jenkins user, this should work:

jenkins@jenkins:~$ sudo reboot

Agents

• Puppet code.

Web server

• Puppet code.

Upgrades

Upgrade policy

Here are some guidelines to triage security vulnerabilities in Jenkins and the plugins we have installed:

1. Protecting our infra from folks who have access to Jenkins

   → Upgrading quarterly is sufficient.

2. Protecting our infra from attacks against folks who have access to Jenkins

   For example, XSS that could lead a legitimate user to perform unintended actions with Jenkins credentials (i.e. root in practice).

   → We should stay on top of security advisories and react more quickly than "in less than 3 months".

3. Protecting our infra from other 3rd-parties that affect Jenkins' security

   For example, say some Jenkins plugin, that connects to remote services, has a TLS certificate checking bug. This could potentially allow a MitM to run arbitrary code with Jenkins controller or workers permissions, i.e. root.

   → We should stay on top of security advisories and react more quickly than "in less than 3 months".

Upgrade procedure

• Preparation:

  • Go through the changelog, paying attention to changes on how agents connect to controller, config changes that may need update, important changes in plugins, etc.

• Deployment:

  • Take note of currently running builds before starting the upgrades.

  • Deploy Jenkins upgrade to latest version available using Puppet.

  • Generate a list of up-to-date plugins by running this Groovy script in the Jenkins Script Console. Make sure to update the initial list containing actively used plugins if there were changes.

  • Generate updated Puppet code for tails::jenkins::master using this Python3 script and the output of the above script.

  • Deploy plugin upgrades using the code generated above.

  • Restart all agents.

  • Manually run the Update jobs script (may be needed so XML is valid with current Jenkins):

    sudo -u jenkins /usr/local/sbin/deploy_jenkins_jobs update
    

• Wrap up:

  • Go through warnings in Jenkins interface.
  • Manually remove unneeded plugins from /var/lib/jenkins/plugins.
  • Restart builds that were interrupted by Jenkins restart.
  • Update the Jenkins upgrade steps documentation in case there were changes.
  • Schedule next update.

Agent to controller connections

These are the steps a Jenkins agent does when connecting to the controller:

1. Fetch connection info from http://jenkins.dragon:8080 (see the tails::jenkins::slave Puppet class).
2. Receive the connection URL https://jenkins.tails.net ("Jenkins URL", manually configured in Configure System).
3. Resolve jenkins.tails.net to 192.168.122.1 (because of libvirt config).
4. Connect using HTTPS to jenkins.tails.net:443.
5. Learn about port 42585 (fixed "TCP port for inbound agents", manually configured in Configure Global Security).
6. Finally, connect using HTTP to jenkins.tails.net:42585.

Generating jobs

We generate automatically a set of Jenkins jobs for branches that are active in the Tails main Git repository.

The first brick extracts the list of active branches and output the needed information:

• config/chroot_local-includes/usr/lib/python3/dist-packages/tailslib/git.py
• config/chroot_local-includes/usr/lib/python3/dist-packages/tailslib/jenkins.py

This list is parsed by the generate_tails_iso_jobs script run by a cronjob and deployed by our puppet-tails tails::jenkins::iso_jobs_generator manifest.

This script output YAML files compatible with jenkins-job-builder. It creates one project for each active branch, which in turn uses several JJB job templates to create jobs for each branch:

• build_Tails_ISO_*
• reproducibly_build_Tails_ISO_*
• test_Tails_ISO_*

This changes are pushed to our jenkins-jobs git repo by the cronjob, and thanks to their automatic deployment in our tails::jenkins::master and tails::gitolite::hooks::jenkins_jobs manifests in our puppet-tails repo, these new changes are applied to our Jenkins instance.

Passing parameters through jobs

We pass information from build job to follow-up jobs (reproducibility testing, test suite) via two means:

• the Parameterized Trigger plugin, whenever it's sufficient
• the EnvInject plugin, for more complex cases:
  • In the build job, a script collects the needed information and writes it to a file that's saved as a build artifact.
  • This file is used by the build job itself, to setup the variables it needs (currently only $NOTIFY_TO).
  • Follow-up jobs imported this file in the workspace along with the build artifacts, then use an EnvInject pre-build step to load it and set up variables accordingly.

Builds

See jenkins/automated-builds-in-jenkins.

Tests

See jenkins/automated-tests-in-jenkins.