[arch-dev-public] New build server in the US?

[Baptiste Jonglez]

Hi,

My proposal was basically « here is a possibly cheap opportunity for a
nice new build server ».  If it turns out that it requires more work to
support and that we can afford renting a commercial server, or that we
actually don't need a new build server, it's fine.

On 19-04-18, Florian Pritz via arch-dev-public wrote:
> We already have a second build server (sgp.pkgbuild.com) which is hardly
> used.

I always thought that sgp would be much slower because of I/O (soyuz
builds everything in a ramfs, while sgp.pkgbuild.com does not).

But after testing, it turns out that the difference is not that big: for
community/coq, it takes around 7 minutes to build on soyuz and around 8
minutes on sgp.pkgbuild.com.

> Souyz is really used quite a bit, but in general it has quite some
> resources left to spare. I guess it depends on what you want and when.
> Do you want to get builds done quickly (like on a local machine) or are
> you happy with waiting until the machine is free? Maybe someone wants to
> work on a system that allows to pause builds if people don't care when
> they finish so that those that want them done quickly get priority? That
> way we could possibly use our available resources better. Maybe it's
> even as simple as queuing builds, but I don't know how long the builds
> that people run on soyuz take. If a single build takes an hour, queuing
> won't really work.

Running several builds simultaneously is often not an issue, because many
build steps are sequential (installing dependencies, final compression...).
So simultaneous builds of small packages won't slow each other down.

It becomes an issue when one of the package being built is huge and well
parallelised (chromium, llvm, libreoffice...).  In that case, a second
package build is really slowed down.

The new machine with 32 threads would be nice on that perspective, because
we could build each package with e.g. -j20 and still get good performance
with simultaneous builds.

> Some feedback on how people use soyuz would probably help a lot here.
> What are your build times, how quickly do you want the result, do you
> need to see live output, does the latency to the machine matter
> (interactive usage?), ...?

I use soyuz mostly interactively: I have a script that copies the sources
to soyuz, builds through SSH, then copies back the built packages.  I do
often look at the output, and I expect a build to take a couple of minutes
at most.

Maybe we should just nice our builds when they are big and we don't care
about the extra runtime:

    nice -n10 extra-x86_64-build

> Regarding the OSUOSL idea: I'm slightly against getting machines from
> yet another organisation. While it doesn't matter much during normal
> operation, fixing problems is usually more difficult because things like
> getting a live system booted, getting serial console access or even just
> getting support are often not easily possible when servers are hosted at
> companies that don't see hosting as a core goal. We've had this happen
> before and it's not great. I don't know what the situation is for this
> offer, but let's not rush into creating even more work for us.

Good point, support is often a pain, especially if you're not in the same
timezone.

In this case, hosting is a core goal of OSUOSL, and they generally provide
good support (they pay students to provide support, and in my experience
they are really helpful).  But of course it's always possible to have bad luck.

> > I honestly have no idea about Arch's financial situation, or how soyuz is
> > paid for in practice.
> 
> Soyuz is a rented server at hetzner.de (like all our other important
> servers) and payed for with donation funds.
> 
> Looking at a recent treasury report from SPI[1] we have around $47k
> right now and looking back a few months, it seems to be growing.
> 
> [1] http://lists.spi-inc.org/pipermail/spi-general/2018-April/003845.html

Ok, so there is no financial issue then.

Baptiste
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.archlinux.org/pipermail/arch-dev-public/attachments/20180420/a6094037/attachment.asc>