Tor-weather is a web service that alerts relay operators about issues with their relays. It runs on the host weather-01.
- Tutorial
- How-to
- Reference
- Discussion
Tutorial
How-to
Pager playbook
Disaster recovery
Reference
Installation
the profile::weather class in the tor-puppet repository configures a systemd service to run the tor_weather app with gunicorn, as well as an apache site config to proxy requests to http://localhost:8000. tor-weather handles its own database schema creation, but database and database user creation are still manual.
add the profile::weather class to a node in puppet, then follow the instructions below to configure and deploy the application.
Creating the postgres database
First, follow the postgresql installation howto.
Run sudo -u postgres psql and enter the following commands. Make sure you generate a secure password for the torweather user. The password must be url safe (ASCII alphanumeric characters, -, _, ~) since we'll be using the password in a URI later.
CREATE DATABASE torweather;
CREATE USER torweather;
alter user torweather password '<password>';
GRANT ALL ON DATABASE torweather TO torweather;
Preparing a release
because tor-weather is managed using poetry, there are a few steps necessary to prepare a release before deploying:
- clone the tor-weather repo locally
- export the dependencies using poetry:
poetry export --output requirements.txt - note which dependencies are installable by apt using this list of packages in debian
- check out the latest release, and build a wheel:
poetry build --format=wheel scpthe wheel and requirements files to the server:scp requirements.txt dist/tor_weather-*.whl weather-01.torproject.org:/home/weather/
Installing on the server
- deploy the
role::weatherPuppet class to the server - create a virtual environment:
python3 -m venv tor-weather-venvand source it:. tor-weather-venv/bin/activate - install the remaining dependencies from requirements.txt:
pip install -r requirements.txt - enable and start the systemd user service units:
tor-weather.service tor-weather-celery.service tor-weather-celerybeat.service - the /home/weather/.tor-weather.env file configures the tor-weather application through environment variables. This file is managed by Puppet.
SMTP_HOST=localhost
SMTP_PORT=25
SMTP_USERNAME=weather@torproject.org
SMTP_PASSWORD=''
SQLALCHEMY_DATABASE_URI='postgresql+psycopg2://torweather:<database password>@localhost:5432/torweather'
BROKER_URL='amqp://torweather:<broker password>@localhost:5672'
API_URL='https://onionoo.torproject.org'
BASE_URL='https://weather.torproject.org'
ONIONOO_JOB_INTERVAL=15
# XXX: change this
# EMAIL_ENCRYPT_PASS is a 32 byte string that has been base64-encoded
EMAIL_ENCRYPT_PASS='Q0hBTkdFTUVDSEFOR0VNRUNIQU5HRU1FQ0hBTkdFTUU='
# XXX: change this
SECRET_KEY='secret'
SQLALCHEMY_TRACK_MODIFICATIONS=
CELERY_BIN=/home/weather/tor-weather-venv/bin/celery
CELERY_APP=tor_weather.celery.celery
CELERYD_NODES=worker1
CELERYD_LOG_FILE=logs/celery/%n%I.log
CELERYD_LOG_LEVEL=info
CELERYD_OPTS=
CELERYBEAT_LOG_FILE=logs/celery/beat.log
CELERYBEAT_LOG_LEVEL=info
Upgrades
- activate the tor-weather virtualenv
- install the latest tor-weather:
pip install tor-weather --index-url https://gitlab.torproject.org/api/v4/projects/1550/packages/pypi/simple --upgrade - restart the service:
sudo -u weather env XDG_RUNTIME_DIR=/run/user/$(id -u weather) systemctl --user restart tor-weather.service - restart the celery service:
sudo -u weather env XDG_RUNTIME_DIR=/run/user/$(id -u weather) systemctl --user restart tor-weather-celery.service - restart the celery beat service:
sudo -u weather env XDG_RUNTIME_DIR=/run/user/$(id -u weather) systemctl --user restart tor-weather-celerybeat.service
Migrating the database schema
After an upgrade or an initial deployment, you'll need to create or migrate the database schema. This script will activate the tor-weather virtual environment, export the tor-weather envvar settings, and then create/migrate the database schema. Note: the flask command might need to get updated dependent on the Python version running.
sudo -u weather bash
cd /home/weather
source tor-weather-venv/bin/activate
set -a
source .tor-weather.env
set +a
flask --app tor_weather.app db upgrade --directory /home/weather/tor-weather-venv/lib/python3.11/site-packages/tor_weather/migrations
exit
SLA
Design and architecture
Services
The tor-weather deployment consists of three main services:
- apache: configured in puppet. proxies requests to
http://localhost:8000 - gunicorn: started by a systemd service file configured in puppet. runs with 5 workers (recommended by gunicorn docs: (2 * nproc) + 1), listens on localhost port 8000
- postgres: a base postgres installation with a
torweatheruser and database
Additionally, there are three services related to task scheduling:
- rabbitmq: configured in puppet, a message broker (listening on
localhost:5672) - celery: task queue, started by a systemd service file configured in puppet
- celery beat: scheduler, started by a systemd service file configured in puppet
Storage
tor-weather is backed by a postgres database. the postgres database is configured in the /home/weather/.tor-weather.env file, using a sqlalchemy connection URI.
Queues
Onionoo Update Job
The tor-weather-celerybeat.service file triggers a job every 15 minutes to update tor-weather's onionoo metrics information.
Interfaces
Authentication
tor-weather handles its own user creation and authentication via the web interface.
Implementation
Related services
Issues
Issues can be filed on the tor-weather issue tracker.
Maintainer
tor-weather is maintained by the network-health team.
Users
Monitoring and metrics
Tests
Logs
Logs are kept in <working directory>/logs. In the current deployment this is /home/weather/tor-weather/logs.