Updating Mirrors

Analyzing failures
JSON pool
DNS pool
- Adding a mirror
- Removing a mirror

Analyzing failures

When the cron job returns an error:

If the error is about an URL or a DNS host, read on to disable it temporarily from the JSON pool. For example:
```
[https://tails.dustri.org/tails/] No version available.
```
Else, if the error is about an IP address, refer instead to the section "Updating → DNS pool → Removing a mirror" below. For example:
```
[198.145.21.9] No route to host
```

Test the mirror:
```
ruby check-mirrors.rb --ip $IP --debug --fast --url-prefix=$URL
```
If the mirror is working now, it might have been a transient error, either on the mirror or on [lizard]. Depending on the error, it might make sense to still notify the admin.

If the mirror is still broken, continue.
Has the mirror failed already in the last 6 months?

If this is the second time this mirror failed in the last 6 months, and there's no indication the root cause of the problem will go away once for all, go to the "Removing a mirror completely" section below.
Is the problem a red flag?

If the problem is one of these:
- The mirror regularly delivers data slowly enough for our cronjob to report about it.
- We're faster than the mirror operator to notice breakage on their side.
- The mirror uses an expired TLS certificate.
- The web server does not run under a supervisor that would restart it if it crashes.
- Maintenance operations that take the server down are not announced in advance.
Then it is a red flag, that suggests the mirror is operated in a way that will cause recurring trouble. These red flags warrant removing the mirror permanently, using your judgment on a case-by-case basis. In which case, go to the "Removing a mirror completely" section below.

For context, see https://gitlab.tails.boum.org/tails/blueprints/-/wikis/HTTP_mirror_pool#improve-ux-and-lower-maintenance-cost-2021
Else, go to "Disabling a mirror temporarily".

JSON pool

Adding a mirror

Test the mirror:

ruby check-mirrors.rb --ip $IP --debug --fast --url-prefix=$URL

Add the mirror to mirrors.json in mirror-pool.git.

Add a note in the "notes" field about when the mirror was added.
Commit and push to mirror-pool.git.

If you get an error on the size of the file, while committing:
```
mirrors.json too big (9041 >= 8192 B). Aborting...
```
Then you need to remove some bits of the file before committing.

For example, consider removing some of the mirrors with the most notes about failures. See "Removing a mirror completely" below.

Answer to the mirror administrator. For example:

Hi,

Your mirror seems to be configured correctly so we added it to
our pool of mirrors. You should start serving downloads right
away.

Thanks for setting this up and contributing to Tails!

Disabling a mirror temporarily

Update mirrors.json in mirror-pool.git:
- Change the weight of the mirror to 0 to disable it temporarily.
- Add a note to document the failure, for example:
```
2020-05-21: No route to host
```
  The "notes" fields has no strict format. I find it easier to document the latest failure first in the string.
Commit and push to mirror-pool.git.

Notify the admin. For example:

Hi,

Today, your mirror is $ERROR_DESCRIPTION:

    https://$MIRROR/tails/

Could you have a look?

Thanks for operating this mirror!

Keep track of the notification and ping the admin after a few weeks.

It's easy to miss one email, but let's not bother chasing those who don't answer twice.

Updating weights

To decrease the impact of unreliable mirrors in the pool, we give different weights to mirrors depending on their last failure:

We give a weight of:
- 10 to a few mirrors that haven't failed in the last 12 months and have a huge capacity.
- 5 to mirrors that haven't failed in the past 12 months.
- 2 to mirrors that haven't failed in the past 6 months.
- 2 to new mirrors.
- 1 to mirrors that have failed in the past 6 months.
We only keep notes of failures that happened less than 12 months ago.

We don't have a strict schedule to update these weights or remove notes on failures older than 12 months.

Removing a mirror completely

We remove mirrors and clean the JSON file either:

Sometimes proactively, from time to time, though we don't have a fixed schedule for that.
Mostly reactively, when the JSON file gets so big that we cannot commit changes to it.

The mirrors that can be completely removed are either:

Mirrors that expose red flags documented above.
Mirrors that had problems at least twice in the last 6 months.
Mirrors that have been disabled, notified, and pinged once.

Mirrors that have seen the most failures in the past year or so.

Template message for mirrors that are repeatedly broken:

Hi,

Today your mirror is XXX:

	https://tails.XXX/

So I removed it from the pool for the time being.

I also wanted to let you know that your mirror has been the most
unreliable of the pool in the past year or so:

  - YYYY-MM-DD: XXX
  - etc.

We have a lot of mirrors already and right now we are more worried about
reliability and performance than about the raw number of mirrors.
We have some ideas to make our setup more
resilient to broken mirrors but we're not there yet. So right now, a
broken mirror means a broken download or a broken upgrade for users.

So unless, you think that the recent instability has a very good reason
to go away once and for all, maybe you can also consider retiring your
mirror, until our mirror pool management software can accommodate less
reliable mirrors. How would you feel about that?

It has been anyway very kind of you to host this mirror until now and we
are very grateful to your contribution to Tails!

Cheers,

DNS pool

Adding a mirror

On dns.lizard:

pdnsutil edit-zone amnesia.boum.org

Then add an A entry mapping dl.amnesia.boum.org to their IP.

Removing a mirror

On dns.lizard:

pdnsutil edit-zone amnesia.boum.org

Then remove the A entry mapping dl.amnesia.boum.org to their IP.

You probably want to compensate this loss by adding another mirror to the DNS pool, if the pool has four members or less after this removal.

Keyboard shortcuts