[arch-devops] Secondary backup ideas

[Florian Pritz]

On 11.01.2018 20:17, Thore Bödecker wrote:
> Do we have the same contraints/requirements for the secondary (backup)
> backups, that we have for the current primary ones?
> That is, do we need to have the same intervals for the secondary as
> for the primary?

Intervals don't matter too much I'd say, but something like every one or
two days would be good I think. Once a week may be a little bit much
loss if it should ever come to us needing those backups.

> Here we could use rrsync to deny deletion of files on "B", so that
> even if vostok is compromised, the attacker could not do something
> like:
> rsync --delete -a /empty B:/the/precious/backups/

There is actually a note in the rrsync source that it is not as secure
as you might imagine. It only checks the rsync command that is executed,
but apparently the rsync protocol itself may be vulnerable to attack
even with the correct options set. Sure, it may be unlikely that someone
tries this, but we're exploring all options here.

> Second idea:
> 
> Use rsnapshot or similar pull-based backups on "B" to pull the data
> from vostok.
> The access permissions of "B" would need to be limitied to "read only"
> on vostok, so that an attacker on "B" is not able to delete antyhing
> on vostok.
> It *would* be possible to do something like rsync --delete-after now,
> so that the local copy on "B" would also be cleaned up, but this has
> another caveat:
> If vostok gets compromised and the attacker deletes all backups there,
> a subsequent rsync from "B" would propagate the deletion to "B" and
> all backups would be lost.

They wouldn't because we use rsnapshot and thus have old snapshots still
available. Sure, an attacker could delete the backups and after X days B
would have finally rotated away all snapshots with data, but the point
is that we have sufficient time to notice the problem and deal with it.

One potential problem here is that rsnapshot doesn't have verification
support so if we are unlucky, the files could get eaten by bit rot and
borg is somewhat susceptible to this thanks to deduplication and
encryption. We could probably work around this by creating checksum
files in rsnapshot (it can execute commands too) and then writing a
custom verification script.

I like this idea. Does anyone see any more potential issues with it?
Feel free to take it apart as good as you can.

> Third idea:
> 
> Use a different backup interval on the individual servers and use borg
> to push the backups to "B".
> Doing all backups twice with the same interval would create a
> considerable (at least double) amount of backup-data.
> Ideally we would deny prune/delete through borg on "B", so that even
> if a server gets compromised, the attacker would not be able do delete
> the backups of that server on "B".
> This in turn would again result in continuesly growing backup/archive
> space.

I know that running borg twice is what upstream recommends, but I think
we can do better without having additional load on the servers. Also the
append-only problem you mentioned is still there...

> It would be best to use a completely different backup-software AND
> filesystem for the secondary backups.
> So that even if there happens to be a real bad in borg or the the
> filesystem where our current borg backups reside on (I suppose ext4?),
> it would ensure that our secondary backups remain unaffected by that.

The file system is a good point. We currently don't have automatic
verification, so that should be improved. We might not notice bit rot or
broken file systems otherwise.

Using a different software might be a nice idea, but that certainly puts
additional load on the source servers and I'm not sure if it's really
worth it. Borg is well tested and old versions are available. It might
be worth thinking about this more though.

Florian

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 858 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/arch-devops/attachments/20180111/4706fea7/attachment.asc>