On Sun, 10 Feb 2019 at 12:35, Florian Pritz via arch-devops < arch-devops@lists.archlinux.org> wrote:

On Sun, Feb 10, 2019 at 12:14:52PM +0100, Jelle van der Waa < jelle@vdwaa.nl> wrote:

...

From what I've heard is that the forum ansible role is mostly done, can we keep this migration moving? What's currently holding it back?

Also to which server do we move the forums?

I believe that's the question that's mostly blocking this (and other stuff on luna). I think I'd be in favour of hetzner's cloud VMs, but then again I have no idea how large of a VM we'd need for the bbs. I'm also fine with getting a second server like apollo to replace luna with, but then we should get the migration done in a reasonable time frame so that we don't keep luna around forever and pay two boxes. Is anyone interested in working on either of those two solutions?

Florian

I have some good experience with the Hetzner Cloud stuff and I generally think we should use those more going forward as it eases provisioning and general maintenance (as we don't have to write support to switch disks and other faulty hardware). I'd like to work on that with Jelle. We'll figure out how much space is required (and we can always upgrade). I suggest we use Terraform for the actual server acquisition.

On Mon, Feb 11, 2019 at 05:30:47PM +0100, Bartłomiej Piotrowski via arch-devops <arch-devops@lists.archlinux.org> wrote:

We also need to migrate Synapse, Kanboard and Quassel away from soyuz.

Kanboard is on apollo, not soyuz. Apart from that, yeah we should migrate those away so that soyuz is really only a build server. Florian

On Tue, Feb 12, 2019 at 11:33:42AM +0100, Jelle van der Waa <jelle@vdwaa.nl> wrote:

So today, we discussed it again on IRC and I'm not sure what we want to do. Move all small services to a VPS? If we do that, do we want to move to a separate DB hosted on a dedicated server (and what about HA).

I'd say that we should keep the databases on the same systems as the services for now. That way, if a machine has problems or goes down, it only affects the services of that machine. It also limits potential load issues to fewer services. Note that I'm not opposed to having multiple services on a single machine in cases where it makes sense.

That would sum up: (2 or 4x EX42?)

Please only look at machines with ECC memory. I don't want another machine with bad memory like alderaan (I think?). We had a few weird database issues with that box before we figured our what it was. Not a pleasant experience.

VPS plan: CX11 (~3-4?): - phrik - security.archlinux.org - grafana - quassel - zabbix - kanboard - archweb

+ patchwork

separate CX21: - forum - bugtracker - wiki - matrix?

Not sure if we can fit matrix on 40gb disk space, but apart from that, the list looks fine. Missing from the list are git and mailman. Git can survive with a CX11. The mailman directory is around 18GB. Of that 12GB are the arch-commits archive. Do we need that or can we drop it? Nearly everything from that list is in the svn log anyways. Without the arch-commits archive, a CX11 would be enough. Otherwise we need another CX21. So the total tally would be (with some margin for error and rounded costs): 5 CX11 (your list + git), 2.50€ each, total 12.5€ 4 CX21 (your list + mailman), 5€ each, total 20€ 2 CX31 (matrix and AUR), 9€ each, total 18€ 1 CX41 (see below), 16€ grand total = 66.5€ (rounded) Did we forget anything? If not, that's a lot cheaper than what we have right now while still distributing services across multiple machines. Sounds good to me.

Dedicated server: (AX60?) - aur

Looking at resource usage on luna, that's a total overkill. I'd say a CX41 should be sufficient. That said, I do wonder why mysql uses so much CPU. Pretty much everything of luna's mysql cpu usage comes from the AUR. bbs is the next big thing with about 0.7-2%. The AUR has between 60 and 140. It might be possible to optimize that a bit more. Once we have a roughly complete list, I'd suggest that we talk to Aaron about the funds and then we start by migrating the services that are currently on luna/soyuz to their respective new homes. Afterwards we deal with apollo based services. PS: I just noticed that hetzner also offers dedicated block storage that can be attached to VMs. This may or may not be useful for mailman and matrix. Regarding matrix I'm not sure if the performance is sufficient, but it's likely perfectly fine for mailman. Florian

Hi list! On 11/02/2019 20:35, Jelle van der Waa wrote:

For security@archlinux.org the Security Team wants to setup a way for reporters to securely mail encrypted issues to our email address. Not actually questioning the motives but, is there really a *need* for this? From now on, I'll assume "yes" here.

* Cheapest Hetzner server 34 euro / month and 40 euro setup fees. * Hetzner auction server ~ 25 / month and no setup fees. * Different dedicated server hoster which allows custom usb devices.I would go with an auction server. They're reliable and cheap (had 2 personally already).

* Nitrokey is out of our control, but we trust Hetzner already (ie. they could easily hook up a malicious USB/BMC device already and gain root privileges). We can't be *that* paranoid (we actually can, but given the circumstances, I really don't see the need to)

* Server dies, the Nitrokey has to be moved to the new server. That's a bummer. Let's not forget downtime to this. How long does it take to Hetzner to move the key? *Who* is allowed to request it?

Questions: * Do we backup the key? Let someone have a separate nitrokey? I would vote for "yes" here. Let someone have a backup key that can be used it case the production one get's lost/broken/<insert_your_catastrophic_event_here>