This procedure documents various benchmarking procedures in use inside TPA.

• HTTP load testing
  • Common procedure
  • Siege
  • apachebench
  • bombardier
  • wrk
  • Other tools

HTTP load testing

Those procedures were quickly established to compare various caching software as part of the cache service setup.

Common procedure

1. punch a hole in the firewall to allow the test server to access tested server, in case it is not public yet

   iptables -I INPUT -s 78.47.61.104 -j ACCEPT
   ip6tables -I INPUT -s 2a01:4f8:c010:25ff::1 -j ACCEPT
   

2. point the test site (e.g. blog.torproject.org) to the tested server on the test server, in /etc/hosts:

   116.202.120.172	blog.torproject.org
   2a01:4f8:fff0:4f:266:37ff:fe26:d6e1 blog.torproject.org
   

3. disable Puppet on the test server:

   puppet agent --disable 'benchmarking requires /etc/hosts override'
   

4. launch the benchmark on the test server

Siege

Siege configuration sample:

verbose = false
fullurl = true
concurrent = 100
time = 2M
url = http://www.example.com/
delay = 1
internet = false
benchmark = true

Might require this, which might work only with varnish:

proxy-host = 209.44.112.101
proxy-port = 80

Alternative is to hack /etc/hosts.

apachebench

Classic commandline:

ab2 -n 1000 -c 100 -X cache01.torproject.org https://example.com/

-X also doesn't work with ATS, modify /etc/hosts instead.

bombardier

We tested bombardier as an alternative to go-wrk in previous benchmarks. The goal of using go-wrk was that it supported HTTP/2 (while wrk didn't), but go-wrk had performance issues, so we went with the next best (and similar) thing.

Unfortunately, the bombardier package in Debian is not the HTTP benchmarking tool but a commandline game. It's still possible to install it in Debian with:

export GOPATH=$HOME/go
apt install golang
go get -v github.com/codesenberg/bombardier

Then running the benchmark is as simple as:

./go/bin/bombardier --duration=2m --latencies https://blog.torproject.org/

wrk

Note that wrk works similarly to bombardier, sampled above, and has the advantage of being already packaged in Debian. Simple cheat sheet:

sudo apt install wrk
echo "10.0.0.0 target.example.com" >> /etc/hosts
wrk --latency -c 100 --duration 2m https://target.example.com/

The main disadvantage is that it doesn't (seem to) support HTTP/2 or similarly advanced protocols.

Other tools

Siege has trouble going above ~100 concurrent clients because of its design (and ulimit) limitations. Its interactive features are also limited, here's a set of interesting alternatives:

Note that the Proto(col) and Features columns are not exhaustive: a tool might support (say) HTTPS, HTTP/2, or HTTP/3 even if it doesn't explicitly mention it, although it's unlikely.

It should be noted that very few (if any) benchmarking tools seem to support HTTP/3 (or even QUIC) at this point. Even HTTP/2 support is spotty: for example, while bombardier supports HTTP/2, it only does so with the slower net/http library at the time of writing (2021). It's unclear how many (if any) other projects to support HTTP/2 as well.

More tools, unreviewed:

• https://github.com/ddosify/ddosify