Summary: Create a new GarageHQ-based object storage cluster, then move all
objects to it and have the new cluster replace the minio-based one. After a
while and if we're satisfied, decommission the minio VMs minio-01.torproject.org
and minio-fsn-02.torproject.org.
Background
We've been using minio for about two years now and it's working fine in daily usage.
One thing that we've however recently discovered was that managing expansions to the cluster was more involved than we were hoping it to be. But that in itself was not enough to make us move away from it.
MinIO, the company, has abandoned their free software option and are instead promoting their new closed-source product named AIStore. See tpo/tpa/team#42352 for more details about this.
Before really abandoning the software, the MinIO company made some decisions which prompted us to write this RFC since they were all pointing towards the conclusion that we see now, that the free software's development was completely stopped. In September 2025 they decided to unexpectedly remove the management web UI leaving our users out of ways to manage their buckets independently.
Before abandoning the software, upstream has suddenly stopped publishing docker images for minio without communicating this clearly with the community. This means that we're currently running a version that's affected by at least one CVE and surely more will come with time. This forces us to maintain our own docker image for this service.
Because of those events, we've decided to migrate away to a different alternative to avoid being stuck with an abandonware.
Also, on their side the GrageHQ project has started scheduling regular major releases since their 2.0 release in order to acknowledge that it might be necessary for them to create API-breaking changes once in a while.
Garage is still lacking some of the features we had originally wanted like bucket versioning, bucket replication and bucket encryption. However, since the needs of the network health team have changed, we believe that we can deprioritize those features for now.
Proposal
Migrate from minio to GarageHQ for the object-storage service.
This RFC is mainly aimed at replacing the choice of software that was made in TPA-RFC-56 and also referenced in TPA-RFC-84
Goals
Must have
- Completely replace the minio cluster with a new garage cluster
- Documentation about this new software for some basic operations we already need to perform
Nice to have
- Documentation about advanced cluster management like scaling out the storage space
Non-Goals
- We are not aiming here to enroll any new application or team into the object-storage service. That can happen once the migration in the proposal has been completed fully
Tasks
- Create a new object storage cluster based on GarageHQ
- Document and test how maintenance tasks should be done with that cluster
- Transfer all buckets with all of their objects to this new cluster. Also create necessary policies to mimic the ones in place in the minio cluster.
- Point all applications to the new cluster (currently only gitlab, but the network health team should be updated on the situation of this service)
- After a grace period of 3 months, decommission the VMs of the minio cluster.
Scope
Affected users
Currently only the gitlab service is affected.
The network team also used to have a bucket that was planned to host files for the team, but this has been abandoned for now after Tor received the donation of a new server. The network team may still want to use the object service in the future, for example to host backups, but currently they are not affected by this change.
Timeline
Costs estimates
Hardware
0$ in hardware is needed: we will create the new cluster in VMs on our ganeti clusters.
Staff
Alternatives considered
See TPA-RFC-56 for software alternatives that were considered.
References
See TPA-RFC-56 and TPA-RFC-84