On 11.01.2018 20:17, Thore Bödecker wrote:
Do we have the same contraints/requirements for the secondary (backup) backups, that we have for the current primary ones? That is, do we need to have the same intervals for the secondary as for the primary?
Intervals don't matter too much I'd say, but something like every one or two days would be good I think. Once a week may be a little bit much loss if it should ever come to us needing those backups.
Here we could use rrsync to deny deletion of files on "B", so that even if vostok is compromised, the attacker could not do something like: rsync --delete -a /empty B:/the/precious/backups/
There is actually a note in the rrsync source that it is not as secure as you might imagine. It only checks the rsync command that is executed, but apparently the rsync protocol itself may be vulnerable to attack even with the correct options set. Sure, it may be unlikely that someone tries this, but we're exploring all options here.
Second idea:
Use rsnapshot or similar pull-based backups on "B" to pull the data from vostok. The access permissions of "B" would need to be limitied to "read only" on vostok, so that an attacker on "B" is not able to delete antyhing on vostok. It *would* be possible to do something like rsync --delete-after now, so that the local copy on "B" would also be cleaned up, but this has another caveat: If vostok gets compromised and the attacker deletes all backups there, a subsequent rsync from "B" would propagate the deletion to "B" and all backups would be lost.
They wouldn't because we use rsnapshot and thus have old snapshots still available. Sure, an attacker could delete the backups and after X days B would have finally rotated away all snapshots with data, but the point is that we have sufficient time to notice the problem and deal with it. One potential problem here is that rsnapshot doesn't have verification support so if we are unlucky, the files could get eaten by bit rot and borg is somewhat susceptible to this thanks to deduplication and encryption. We could probably work around this by creating checksum files in rsnapshot (it can execute commands too) and then writing a custom verification script. I like this idea. Does anyone see any more potential issues with it? Feel free to take it apart as good as you can.
Third idea:
Use a different backup interval on the individual servers and use borg to push the backups to "B". Doing all backups twice with the same interval would create a considerable (at least double) amount of backup-data. Ideally we would deny prune/delete through borg on "B", so that even if a server gets compromised, the attacker would not be able do delete the backups of that server on "B". This in turn would again result in continuesly growing backup/archive space.
I know that running borg twice is what upstream recommends, but I think we can do better without having additional load on the servers. Also the append-only problem you mentioned is still there...
It would be best to use a completely different backup-software AND filesystem for the secondary backups. So that even if there happens to be a real bad in borg or the the filesystem where our current borg backups reside on (I suppose ext4?), it would ensure that our secondary backups remain unaffected by that.
The file system is a good point. We currently don't have automatic verification, so that should be improved. We might not notice bit rot or broken file systems otherwise. Using a different software might be a nice idea, but that certainly puts additional load on the source servers and I'm not sure if it's really worth it. Borg is well tested and old versions are available. It might be worth thinking about this more though. Florian