[arch-devops] Secondary backup ideas (Task #50)

[Florian Pritz]

On 05.03.2018 09:02, Phillip Smith via arch-devops wrote:
> (Sorry, I can't reply to the original thread as I wasn't a member of the
> list at the time)

In that case you could go to the list archive, open the post you want to
reply to and then click the email address of the sender which will set
the correct subject and In-Reply-To headers.

> Specifically, I'm thinking the multiple key feature would allow us to
> have the client push backups to the backup server over a read-only
> connection (SSH, S3, rsync, whatever), then be able to run restic
> server-side to handle the cleanups.

The problem here is that restic doesn't work with glacier according to
this[1]. So we'd need to use s3 which is more expensive. How much mostly
depends on how long we want to keep the data and how well restic
compresses/deduplicates it. An alternative would be wasabi[2] which also
supports the s3 protocol, but I have no idea how well their service works.

[1] https://github.com/restic/restic/issues/541
[2] https://wasabi.com

I like the idea of using a different tool with (hopefully) good
deduplication/compression though. This is certainly better than sending
many gigabytes of tarballs around for each backup.

As for the cleanups, I understand that the server and the client would
both have keys to access the backup data, correct? That means that the
server can read all of the data which makes it a good target for an
attacker. Currently we avoid this by only storing client-side encrypted
data on the server. I'd like to keep it this way.

I also like the idea of having a WORM s3/glacier bucket. However, I'm
not sure how this can be combined sanely with anything other than
tarballs. From looking at the restic documentation it seems that they
also use an object store so even old objects might still be used in
recent backups. Is there another way to achieve cleanup with restic that
doesn't require a server with access to the backup keys?

Also, how badly do outside changes impact the performance? Let's say we
have the keys on the admin machines (which we need for restores anyway)
and perform the cleanup there. How long would it take to run, how much
data would it need to transfer (few megabytes, few hundred megabytes,
gigabytes, ...?) and do the clients then need to regenerate their caches
or can they run at full performance just like before?

Florian

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 858 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/arch-devops/attachments/20180305/cb4e9f13/attachment.asc>