Le 11/01/2018 à 17:54, Florian Pritz via arch-devops a écrit :

Hi,

Bartłomiej pointed out that we somehow stopped using the mailing list for discussion so here we go.

Currently we only create backups on vostok using borg. A possibly quite big problem with this is that if an attacker gains access to a server, they also have sufficient access to remove all backups of that server. We could restrict that in borg, but then borg wouldn't be able to clean up old backups regularly and our repos would grow forever.

I haven’t digged deep into borg doc, but I suppose there is no mode where only the client has the encryption/decryption key, but only the server can remove data? If e.g. backup date metadata is available to the server this could be done, but I don’t know how it works exactly so…

A better solution is to create backups of the backups in a way that the front end servers can not delete any of these secondary backups.

Possible solutions include: - Creation of a second layer of backups (backups of the backups) on vostok. This roughly doubles our space requirement and we are currently at 44% usage so this won't work well/for long. Unless we can use file system level snapshots or similar for this to reduce the required space, it's out. - Put the secondary backups on a different, possibly new, machine using borg. The backup would be created on vostok from the existing backup data. - Put them on AWS glacier. Roughly 4€/month and TB; suggested by Tyler from Arch Linux 32

Using glacier would require that we export tarballs (supported by borg) and then upload them. However, since the backups are encrypted and vostok is not supposed to be able to read them, the tarballs need to be created on and uploaded from the servers themselves. This may become a cpu/bandwidth/traffic concern if done often. Tyler is currently investigating this for Arch Linux 32's backups AFAIK.

Does anyone have other ideas what we could do here to ensure that we have backups of the backups? The most important requirements are that no matter which server an attacker manages to get access to, they cannot read any user data from other servers and that they cannot remove all backups from that server alone.

Florian

I’m not sure how that works technically speaking, but I suppose that for AWS Glacier you intend to use borg append-only mode to avoid an attacker deleting the backups? How would the cleaning work in this case? We would just not care about it because Glacier? In your ideas #1 and #2, it seems to me that an attacker gaining access to vostok is able to remove all backups from all servers, right? So “the most import requirement that no matter which server an attacker manages to get access to they cannot remove all backups [from that server alone]” seems flawed from the start. Or you implies that if an attacker only get access to vostok, that’s OK because all servers are still running so that there is no loss of data? Bruno

Hey, On 11.01.18 - 17:54, Florian Pritz via arch-devops wrote:

Does anyone have other ideas what we could do here to ensure that we have backups of the backups? The most important requirements are that no matter which server an attacker manages to get access to, they cannot read any user data from other servers and that they cannot remove all backups from that server alone.

Do we have the same contraints/requirements for the secondary (backup) backups, that we have for the current primary ones? That is, do we need to have the same intervals for the secondary as for the primary? (I'm going to explore all ideas even if already determined "not optimal") What I first had in mind sadly does not work properly in the long run after thinking it through: Rent a second server (let's call it "B"), that is being push-rsync'ed from vostok. Here we could use rrsync to deny deletion of files on "B", so that even if vostok is compromised, the attacker could not do something like: rsync --delete -a /empty B:/the/precious/backups/ The problem with is: we lose the ability of borg prune cleanups and the backups on "B" would continue to grow and grow and grow.... (It is not possible to use find -mtime +90 or similar due to the nature of borg) Second idea: Use rsnapshot or similar pull-based backups on "B" to pull the data from vostok. The access permissions of "B" would need to be limitied to "read only" on vostok, so that an attacker on "B" is not able to delete antyhing on vostok. It *would* be possible to do something like rsync --delete-after now, so that the local copy on "B" would also be cleaned up, but this has another caveat: If vostok gets compromised and the attacker deletes all backups there, a subsequent rsync from "B" would propagate the deletion to "B" and all backups would be lost. So we are back at the problem of missing cleanup abilities again. Third idea: Use a different backup interval on the individual servers and use borg to push the backups to "B". Doing all backups twice with the same interval would create a considerable (at least double) amount of backup-data. Ideally we would deny prune/delete through borg on "B", so that even if a server gets compromised, the attacker would not be able do delete the backups of that server on "B". This in turn would again result in continuesly growing backup/archive space. Another thing I would like to mention: It would be best to use a completely different backup-software AND filesystem for the secondary backups. So that even if there happens to be a real bad in borg or the the filesystem where our current borg backups reside on (I suppose ext4?), it would ensure that our secondary backups remain unaffected by that. Unfortunately I have no complete solution for the concers raised by Bartłomiej and Florian (yet). I hope that someone might be able to pick up where I left off with my ideas and can put them into proper concepts. Cheers, Thore --