Hi all,
Tomorrow we will have another devops meeting at one hour earlier then
our usual time. The topics should be:
* Orion replacement/migration
* Gitlab/keycloak
Greetings,
Jelle
Hey y'all,
I just finished up setting up gemini.archlinux.org which will serve as the
new repos and archive host as orion.archlinux.org is getting way too small
for this. gemini should easily serve us for the next decade or two given
our current rate of growth.
It now needs the archive and repo data migrated to it and then we need to
do the actual switchover. For users (packagers, that is), not much should
actually change except for an SSH warning. We'll inform you about new host
keys once this is switched over.
After the initial data sync, we need to set up the actual tools that allow
for repo operation (mainly dbscripts and I believe some parts of archweb)
and then switch the DNS. We'll likely disallow package updating for a short
critical switch over period to make sure we are perfectly synced but this
is yet to be coordinated.
The machine is an Intel E3-1270 v3 with 32GiB of ECC DDR3 and 4*10TiB in
BTRFS RAID6 for an effective 20TiB of usable space. I know, this seems
daring given BTRFS's history but we looked at the recent stability
improvements of BTRFS RAID56 and it now seems stable enough for production
usage. At any rate, we have backups.
Cheers,
Sven
With the refactoring of the hetzner cloud inventory script python-typer
is now a dependency, don't forgot to install it before running Ansible :)
Greetings,
Jelle
Hey all,
Just writing real quick to inform you that I just set up
secure-runner1.archlinux.org which is a new hardware box that'll serve as a
GitLab runner that we can trust. The idea is to never allow it to run
unreviewed or untrusted code by limiting it to protected branches of
specific projects (and only those projects, not their forks). Effectively,
that'll be the master branch of selected Arch projects which need to
produce trustable artifacts (like archiso for automatic ISO creation or VM
images, for instance). The current idea is not to use it for any automatic
package building.
This is currently an experiment and will still need to be security audited
(and we're not using it for producing any real artifacts until we feel
comfortable with it). It might turn out we need one such secure runner per
project, who knows.
We'll control this runner rather tightly for obvious reasons and projects
need to be hand-selected one-by-one.
In case anyone's wondering, the box is a Ryzen 5 3600 with 64 GiB ECC DDR4
and 512 GiB BTRFS RAID1.
Cheers,
Sven
Hello all,
After updates on orion and a reboot we had a boot failure with the
following error message. It is unknown what caused this, but grub was
also updated:
[2020-05-18T19:18:52+0000] [ALPM] upgraded grub (2:2.04-5 -> 2:2.04-7)
The grub error:
symbol `grub_file_filters` not found
When trying to grub-install with the latest grub package it fails and
results into the following bug FS#63656
We downgraded grub to grub-2:2.02-8-x86_64.pkg.tar.xz from archive.org,
to get a package listing you can use the handy url. [2]
This raised the question of grazzolini whether we should continue using
a partionless btrfs setup and as we are about to setup a new
orion.archlinux.org this might be relevant :)
I wonder if the new box also has UEFI already and if we should switch
away from grub? (although I think all our vps'es use grub)
[1] https://bugs.archlinux.org/task/63656
[2] https://archive.org/download/archlinux_pkg_grub
Greetings,
Jelle
Hi all,
As we had an boot failure on orion, Foxboron pointed out it might be the
recent intel-ucode update which caused issues on Ubuntu. [1]
I've downgraded it on orion, but grub was the curlpit. I'll update a vps
tommorow and see how that goes, but for our dedicated boxes be aware
that the update might cause issues.
[1] https://usn.ubuntu.com/4385-2/
Hi,
I've been having issues trying to login to repos.archlinux.org. First,
it went offline suddenly, now it reports a different SSH key. I tried
logging in and it no longer recognizes my key.
I'm only asking cos I was in the middle of moving a package from
community-testing to community that introduces breaking changes and now
I'm not sure how to finish the process.
--
Regards,
Konstantin
Hi,
I've been having issues trying to login to repos.archlinux.org. First,
it went offline suddenly, now it reports a different SSH key. I tried
logging in and it no longer recognizes my key.
I'm only asking cos I was in the middle of moving a package from
community-testing to community that introduces breaking changes and now
I'm not sure how to finish the process.
--
Regards,
Konstantin
Hi All,
We are running awfully low on diskspace on orion and need to start
migrating the archive away ASAP as we can't keep hosting it on orion.
The biggest offender is the archive which seems to be requiring around ~
1TB per year, this might be due to the nature of how we archive as
heftig pointed out we could use btrfs snapshots however no one has
stepped up and I don't expect anyone to.
I would propose to get us a new server, an AX51 which has 2 x 8TB disks
which allows us to grow the archive. Currently we pay 35 / month for
orion which has 2 x 3 TB disks.
The hard question is if we tackle multiple problems at once and make
this server only for the archive which might be tricky as it's now part
of dbscripts to archive packages with "db-archive" which means we have
to migrate our repos, svn server and sourceballs as that all sadly
expects to be present on the machine.
This would leave orion left with mail handling and hefur (for the arch
iso torrent webseeds) which should be moved to a separate vps both
shouldn't require a too big of a vps.
https://www.hetzner.com/dedicated-rootserver/ex52-nvme
Please respond ASAP, as we don't have too much time and Hetzner only
deploys from Monday -Friday.
Greetings,
Jelle van der Waa
Hi all,
Tomorrow we will have another devops meeting at the usual time, there
are no topics in the agenda but I expect us to discuss Gitlab/Keyclaok
progress.
Greetings,
Jelle