Automated ISO/IMG tests on Jenkins

• For developers
• For sysadmins
  • Old ISO used in the test suite in Jenkins
  • Restarting slave VMs between test suite jobs
  • Executors on the Jenkins controller

For developers

See: Automated test suite - Introduction - Jenkins.

For sysadmins

Old ISO used in the test suite in Jenkins

Some tests like upgrading Tails are done against a Tails installation made from the previously released ISO and USB images. Those images are retrieved using wget from https://iso-history.tails.net.

In some cases (e.g when the Tails Installer interface has changed), we need to temporarily change this behaviour to make tests work. To have Jenkins use the ISO being tested instead of the last released one:

1. Set USE_LAST_RELEASE_AS_OLD_ISO=no in the macros/test_Tails_ISO.yaml file in the jenkins-jobs Git repository (gitolite@git.tails.net:jenkins-jobs).

   See for example commit 371be73.

   Treat the repositories on GitLab as read-only mirrors: any change pushed there does not affect our infrastructure and will be overwritten.

   Under the hood, once this change is applied Jenkins will pass the ISO being tested (instead of the last released one) to run_test_suite's --old-iso argument.

2. File an issue to ensure this temporarily change gets reverted in due time.

Restarting slave VMs between test suite jobs

For background, see #9486, #11295, and #10601.

Our test suite doesn't always clean after itself properly (e.g. when tests simply hang and timeout), so we have to reboot isotesterN.lizard between ISO test jobs. We have ideas to solve this problem, but that's where we're at.

We can't reboot these VMs as part of a test job itself: this would fail the test job even when the test suite has succeeded.

Therefore, each "build" of a test_Tail_ISO_* job runs the test suite, and then:

1. Triggers a high priority "build" of the keep_node_busy_during_cleanup job, on the same node. That job will ensure the isotester is kept busy until it has rebooted and is ready for another test suite run.

2. Gives Jenkins some time to add that keep_node_busy_during_cleanup build to the queue.

3. Gives the Jenkins Priority Sorter plugin some time to assign its intended priority to the keep_node_busy_during_cleanup build.

4. Does everything else it should do, such as cleaning up and moving artifacts around.

5. Finally, triggers a "build" of the reboot_node job on the Jenkins controller, which will put the isotester offline, and reboot it.

6. After the isotester has rebooted, when jenkins-slave.service starts, it puts the node back online.

For more details, see the heavily commented implementation in jenkins-jobs:

• macros/test_Tails_ISO.yaml
• macros/keep_node_busy_during_cleanup.yaml
• macros/reboot_node.yaml

Executors on the Jenkins controller

We need to ensure the Jenkins controller has enough executors configured so it can run as many reboot_job concurrent builds as necessary.

This job can't run in parallel for a given test_Tails_ISO_* build, so what we strictly need is: as many executors on the controller as we have nodes allowed to run test_Tails_ISO_*. This currently means: as many executors on the controller as we have isotesters.