Hi,
Bartłomiej pointed out that we somehow stopped using the mailing list
for discussion so here we go.
Currently we only create backups on vostok using borg. A possibly quite
big problem with this is that if an attacker gains access to a server,
they also have sufficient access to remove all backups of that server.
We could restrict that in borg, but then borg wouldn't be able to clean
up old backups regularly and our repos would grow forever. A better
solution is to create backups of the backups in a way that the front end
servers can not delete any of these secondary backups.
Possible solutions include:
- Creation of a second layer of backups (backups of the backups) on
vostok. This roughly doubles our space requirement and we are
currently at 44% usage so this won't work well/for long. Unless we
can use file system level snapshots or similar for this to reduce the
required space, it's out.
- Put the secondary backups on a different, possibly new, machine using
borg. The backup would be created on vostok from the existing backup
data.
- Put them on AWS glacier. Roughly 4€/month and TB; suggested by Tyler
from Arch Linux 32
Using glacier would require that we export tarballs (supported by borg)
and then upload them. However, since the backups are encrypted and
vostok is not supposed to be able to read them, the tarballs need to be
created on and uploaded from the servers themselves. This may become a
cpu/bandwidth/traffic concern if done often. Tyler is currently
investigating this for Arch Linux 32's backups AFAIK.
Does anyone have other ideas what we could do here to ensure that we
have backups of the backups? The most important requirements are that no
matter which server an attacker manages to get access to, they cannot
read any user data from other servers and that they cannot remove all
backups from that server alone.
Florian