[arch-dev-public] New build server in the US?
Hi, OSUOSL [1] in the US has several "new" refurb servers with dual Xeon E5-2660. Each machine has a total of 16 cores / 32 threads and 140 GB RAM, so that would make nice build machines. Would this be useful to the dev community? Soyuz is a bit overloaded at times, and people in the US/Canada might appreciate the lower latency. The hosting conditions are described here: http://osuosl.org/services/hosting/ Regarding money, OSUOSL works through donation: if we ask for a server and we have money, it would be nice to support them financially. I honestly have no idea about Arch's financial situation, or how soyuz is paid for in practice. Let's discuss this! Baptiste [1] http://osuosl.org
On 18.04.2018 13:27, Baptiste Jonglez wrote:
OSUOSL [1] in the US has several "new" refurb servers with dual Xeon E5-2660. Each machine has a total of 16 cores / 32 threads and 140 GB RAM, so that would make nice build machines.
Would this be useful to the dev community? Soyuz is a bit overloaded at times, and people in the US/Canada might appreciate the lower latency.
We already have a second build server (sgp.pkgbuild.com) which is hardly used. Souyz is really used quite a bit, but in general it has quite some resources left to spare. I guess it depends on what you want and when. Do you want to get builds done quickly (like on a local machine) or are you happy with waiting until the machine is free? Maybe someone wants to work on a system that allows to pause builds if people don't care when they finish so that those that want them done quickly get priority? That way we could possibly use our available resources better. Maybe it's even as simple as queuing builds, but I don't know how long the builds that people run on soyuz take. If a single build takes an hour, queuing won't really work. Some feedback on how people use soyuz would probably help a lot here. What are your build times, how quickly do you want the result, do you need to see live output, does the latency to the machine matter (interactive usage?), ...? Is anyone interested in taking on this project and maybe also setting up/maintaining some build service if that turns out to be a good idea? Regarding the OSUOSL idea: I'm slightly against getting machines from yet another organisation. While it doesn't matter much during normal operation, fixing problems is usually more difficult because things like getting a live system booted, getting serial console access or even just getting support are often not easily possible when servers are hosted at companies that don't see hosting as a core goal. We've had this happen before and it's not great. I don't know what the situation is for this offer, but let's not rush into creating even more work for us.
I honestly have no idea about Arch's financial situation, or how soyuz is paid for in practice.
Soyuz is a rented server at hetzner.de (like all our other important servers) and payed for with donation funds. Looking at a recent treasury report from SPI[1] we have around $47k right now and looking back a few months, it seems to be growing. [1] http://lists.spi-inc.org/pipermail/spi-general/2018-April/003845.html Florian
[2018-04-19 21:19:43 +0200] Florian Pritz via arch-dev-public:
Some feedback on how people use soyuz would probably help a lot here. What are your build times, how quickly do you want the result, do you need to see live output, does the latency to the machine matter (interactive usage?), ...?
For my own limited use, short build times are what matters most: if a build is slowed down or delayed I'll likely have forgotten about it by the time it's finished. I'm glad you mentioned the Singapore server was underused. I've switched my build scripts to using it since I'm the same ping time away from it and soyuz. Cheers. -- Gaetan
On 04/19/2018 03:19 PM, Florian Pritz via arch-dev-public wrote:
We already have a second build server (sgp.pkgbuild.com) which is hardly used. Souyz is really used quite a bit, but in general it has quite some resources left to spare. I guess it depends on what you want and when. Do you want to get builds done quickly (like on a local machine) or are you happy with waiting until the machine is free? Maybe someone wants to work on a system that allows to pause builds if people don't care when they finish so that those that want them done quickly get priority? That way we could possibly use our available resources better. Maybe it's even as simple as queuing builds, but I don't know how long the builds that people run on soyuz take. If a single build takes an hour, queuing won't really work.
Some feedback on how people use soyuz would probably help a lot here. What are your build times, how quickly do you want the result, do you need to see live output, does the latency to the machine matter (interactive usage?), ...?
I usually only use soyuz for annoyingly long builds like my custom repo with glibc-git, thunderbird-gtk2, etc. which can take as long as it needs to (assuming I remember to run it in tmux so it doesn't die when I lose internet).
Is anyone interested in taking on this project and maybe also setting up/maintaining some build service if that turns out to be a good idea?
We could probably do something dead simple with a wrapper script that checks pgrep makechrootpkg || sleep 1m in a loop, before running a build. Anyone with a build they just want to run as soon as no one else is doing one, could use that. Just run it in tmux in case of latency/disconnection.
Regarding the OSUOSL idea: I'm slightly against getting machines from yet another organisation. While it doesn't matter much during normal operation, fixing problems is usually more difficult because things like getting a live system booted, getting serial console access or even just getting support are often not easily possible when servers are hosted at companies that don't see hosting as a core goal. We've had this happen before and it's not great. I don't know what the situation is for this offer, but let's not rush into creating even more work for us.
I honestly have no idea about Arch's financial situation, or how soyuz is paid for in practice.
Soyuz is a rented server at hetzner.de (like all our other important servers) and payed for with donation funds.
Looking at a recent treasury report from SPI[1] we have around $47k right now and looking back a few months, it seems to be growing.
[1] http://lists.spi-inc.org/pipermail/spi-general/2018-April/003845.html
I guess we could afford to rent another server if we really needed one! But I guess it is probably more used by people with large packages, in which case faster builds was the reason they decided to use soyuz in the first place. We could just upgrade soyuz. I think that might double what we're currently paying, but OTOH so would buying a second server. -- Eli Schwartz Bug Wrangler and Trusted User
Hi, My proposal was basically « here is a possibly cheap opportunity for a nice new build server ». If it turns out that it requires more work to support and that we can afford renting a commercial server, or that we actually don't need a new build server, it's fine. On 19-04-18, Florian Pritz via arch-dev-public wrote:
We already have a second build server (sgp.pkgbuild.com) which is hardly used.
I always thought that sgp would be much slower because of I/O (soyuz builds everything in a ramfs, while sgp.pkgbuild.com does not). But after testing, it turns out that the difference is not that big: for community/coq, it takes around 7 minutes to build on soyuz and around 8 minutes on sgp.pkgbuild.com.
Souyz is really used quite a bit, but in general it has quite some resources left to spare. I guess it depends on what you want and when. Do you want to get builds done quickly (like on a local machine) or are you happy with waiting until the machine is free? Maybe someone wants to work on a system that allows to pause builds if people don't care when they finish so that those that want them done quickly get priority? That way we could possibly use our available resources better. Maybe it's even as simple as queuing builds, but I don't know how long the builds that people run on soyuz take. If a single build takes an hour, queuing won't really work.
Running several builds simultaneously is often not an issue, because many build steps are sequential (installing dependencies, final compression...). So simultaneous builds of small packages won't slow each other down. It becomes an issue when one of the package being built is huge and well parallelised (chromium, llvm, libreoffice...). In that case, a second package build is really slowed down. The new machine with 32 threads would be nice on that perspective, because we could build each package with e.g. -j20 and still get good performance with simultaneous builds.
Some feedback on how people use soyuz would probably help a lot here. What are your build times, how quickly do you want the result, do you need to see live output, does the latency to the machine matter (interactive usage?), ...?
I use soyuz mostly interactively: I have a script that copies the sources to soyuz, builds through SSH, then copies back the built packages. I do often look at the output, and I expect a build to take a couple of minutes at most. Maybe we should just nice our builds when they are big and we don't care about the extra runtime: nice -n10 extra-x86_64-build
Regarding the OSUOSL idea: I'm slightly against getting machines from yet another organisation. While it doesn't matter much during normal operation, fixing problems is usually more difficult because things like getting a live system booted, getting serial console access or even just getting support are often not easily possible when servers are hosted at companies that don't see hosting as a core goal. We've had this happen before and it's not great. I don't know what the situation is for this offer, but let's not rush into creating even more work for us.
Good point, support is often a pain, especially if you're not in the same timezone. In this case, hosting is a core goal of OSUOSL, and they generally provide good support (they pay students to provide support, and in my experience they are really helpful). But of course it's always possible to have bad luck.
I honestly have no idea about Arch's financial situation, or how soyuz is paid for in practice.
Soyuz is a rented server at hetzner.de (like all our other important servers) and payed for with donation funds.
Looking at a recent treasury report from SPI[1] we have around $47k right now and looking back a few months, it seems to be growing.
[1] http://lists.spi-inc.org/pipermail/spi-general/2018-April/003845.html
Ok, so there is no financial issue then. Baptiste
On 20 April 2018 at 01:19, Florian Pritz via arch-dev-public
We already have a second build server (sgp.pkgbuild.com) which is hardly used.
That's mostly my fault. I had requested that server down in SEA but did not resume my activities. Otherwise, I would've been building stuff all the time with the sub-100ms latencies, which is real-time now compared to the 2-300ms I get on Soyuz. But I'm pretty sure I'm not the only one who benefits.
Some feedback on how people use soyuz would probably help a lot here. What are your build times, how quickly do you want the result, do you need to see live output, does the latency to the machine matter (interactive usage?), ...?
I also use(d) the build server non-interactively with scripts for pushing and queuing packages for builds, pulling them, signing and releasing. I intend(ed) to do the same on sgp, although I was more excited to stay logged in. -- GPG/PGP ID: C0711BF1
participants (5)
-
Baptiste Jonglez
-
Eli Schwartz
-
Florian Pritz
-
Gaetan Bisson
-
Rashif Ray Rahman