<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 5 March 2018 at 21:06, Florian Pritz via arch-devops <span dir="ltr"><<a href="mailto:arch-devops@lists.archlinux.org" target="_blank">arch-devops@lists.archlinux.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">
</span>In that case you could go to the list archive, open the post you want to<br>
reply to and then click the email address of the sender which will set<br>
the correct subject and In-Reply-To headers.<br></blockquote><div><br><div style="font-family:verdana,sans-serif;display:inline" class="gmail_default">TIL. Thank-you :)<br></div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-"><br>
</span>The problem here is that restic doesn't work with glacier according to<br>
this[1]. So we'd need to use s3 which is more expensive. How much mostly<br>
depends on how long we want to keep the data and how well restic<br>
compresses/deduplicates it.<br></blockquote><div><br><div style="font-family:verdana,sans-serif" class="gmail_default">restic is yet to implement compression [0]. The deduplication seems quite<br>functional, especially when you can share a restic repository between<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">multiple clients (if desired), so common data across clients is deduped in<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">the repo.<br></div><div style="font-family:verdana,sans-serif" class="gmail_default"><br></div><div style="font-family:verdana,sans-serif" class="gmail_default">Have we committed to the idea of paying for Amazon or similar service<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">for this project?<br></div><div style="font-family:verdana,sans-serif" class="gmail_default"><br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I like the idea of using a different tool with (hopefully) good<br>
deduplication/compression though. This is certainly better than sending<br>
many gigabytes of tarballs around for each backup.<br></blockquote><div><br><div style="font-family:verdana,sans-serif;display:inline" class="gmail_default">Definitely! :)<br></div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
As for the cleanups, I understand that the server and the client would<br>
both have keys to access the backup data, correct? That means that the<br>
server can read all of the data which makes it a good target for an<br>
attacker. Currently we avoid this by only storing client-side encrypted<br>
data on the server. I'd like to keep it this way.<br></blockquote><div><br><div style="font-family:verdana,sans-serif" class="gmail_default">I don't see any way to allow the client to manage cleanups without<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">having write access (and therefore the ability to delete) to the 2nd<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">backup.<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">Perhaps we could consider having the 2nd backup on a snapshotting<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">file-system (ie, ZFS) with something like <a href="http://rsync.net">rsync.net</a> [1]. Then it would<br>just be a dumb rsync from primary backup to secondary backup and<br>the 2nd host retains snapshots to protect against malicious 'cleanups'<br>from the 1st backup host.<br></div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I also like the idea of having a WORM s3/glacier bucket. However, I'm<br>
not sure how this can be combined sanely with anything other than<br>
tarballs. From looking at the restic documentation it seems that they<br>
also use an object store so even old objects might still be used in<br>
recent backups. Is there another way to achieve cleanup with restic that<br>
doesn't require a server with access to the backup keys?<br></blockquote><div><br><div style="font-family:verdana,sans-serif" class="gmail_default">Indexes etc would have to be updated I'm sure, so I don't think there<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">is any tricky ways to do this. I did read somewhere that the repo<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">format is 'read-only' to ensure consistency (ie, files only ever get <br></div><div style="font-family:verdana,sans-serif" class="gmail_default">added to the repo on disk). I can't find the reference to that right<br></div><div style="font-family:verdana,sans-serif" class="gmail_default">now though sorry.<br></div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Also, how badly do outside changes impact the performance? Let's say we<br>
have the keys on the admin machines (which we need for restores anyway)<br>
and perform the cleanup there. How long would it take to run, how much<br>
data would it need to transfer (few megabytes, few hundred megabytes,<br>
gigabytes, ...?) and do the clients then need to regenerate their caches<br>
or can they run at full performance just like before?<span class="gmail-HOEnZb"><font color="#888888"><br></font></span></blockquote><div><br><div style="font-family:verdana,sans-serif" class="gmail_default">I'll do some testing to get some idea of the answers for this.<br></div> </div></div><br><div style="font-family:verdana,sans-serif" class="gmail_default">[0] <a href="https://github.com/restic/restic/issues/21">https://github.com/restic/restic/issues/21</a></div><div style="font-family:verdana,sans-serif" class="gmail_default">[1] <a href="http://www.rsync.net/resources/howto/snapshots.html">http://www.rsync.net/resources/howto/snapshots.html</a></div><br></div></div>