[arch-dev-public] GitLab switchover and SSO migration

Sun May 17 18:39:25 UTC 2020

Hi everyone,

As some of you know, we've been toying with two ideas for a while:
Arch-wide centralized user management as well as using GitLab to
consolidate some of our current services. The overall goal is manifold.
In no particular order, the goals are to
- make Arch more contributor friendly
- provide more modern tools for ourselves
- enable more automation
- make Arch services more secure
- make team management activities less error-prone and more streamlined

These two topics (SSO + Gitlab) are a bit intertwined because we wanted
to have SSO on GitLab before starting off with that so we'd have a
properly validated user base to work with going forward. Also, GitLab
seemed like a good first service for SSO due to it having good support
for that.

After looking at various solutions, we eventually settled on Keycloak
since it seemed like a modern, well-maintained, and secure piece of
software. It allows us to enable logins for services via OpenID Connect
and SAML which is likely the best coverage we could hope for. It also
allows us to connect other social login providers such as github.com and
gitlab.com and it supports Recaptcha, 2FA, and WebAuthn out of the box.
The idea is to eventually transition all our online service as well as
SSH keys to Keycloak to ease on-/offboarding and make it less error-prone.

As for GitLab, a few months back, we applied for a GitLab Ultimate
license in their open-source program [0] and we received one [1]. It's
an official program that many other open-source projects benefit from as
well and we think it's safe to assume that it'll continue being a thing
for the foreseeable future. We have to renew our license yearly. The
current license we have has support for 1000 seats but we can likely get
more seats if we need them.

Our general path is:
1) Transition as many staff-only services to Keycloak as possible. We
looked at our current services and put up a table on the wiki that shows
support per service [2].
Some of the services that we operate are deprecated or have
functionality that is also provided by GitLab. In our current
understanding, this concerns Flyspray, Kanboard, and Patchwork. Of
those, Kanboard and some of the Flyspray projects will be our first
targets to transition to GitLab. We'll continue using Flyspray for the
time being for package bugs but will discontinue its use for all
non-packaging bugs. The reason for this is that how we manage our bugs
for packages is somewhat intertwined with the svn2git migration which is
yet to be done and might dictate a different repo structure than what we
would come up with currently and we don't want to block on this. This
was also discussed in a recent DevOps meeting [3].
2) We'd like to get rid of our own cgit instance at git.archlinux.org
and transition our git hosting into GitLab. AUR git access will stay as
it is due to its special shell magic.
3) Eventually, after an internal testing phase of at least a few weeks,
we'll want to open Keycloak and GitLab up for outside contributors. We
know of the abuse potential and the potential moderation problems and
have to make sure to set proper limits and set up monitoring before
opening this up.
4) Connect remaining services like BBS and wiki to SSO. In 1) we only
mentioned staff-only services because those are less problematic.
However, in the future, we'd also like to enable our remaining services
to connect to Keycloak.

We tried hard to come up with a good source of users to import into
Keycloak so that we could seed that database with a solid user base but
sadly it appears that there is no trusted source of users that we can
rely on. Potential candidates were the wiki, BBS, AUR but we ruled them
all out in their current state as none of them have always had email
verification and so we can't trust those emails to be the sole source of
truth. In order to still allow users to keep their old contributions in
cases where they can prove their identity via email, we'll build a new
small web application that allows them to connect their new Keycloak
identity to their other identities. For now, we seeded the Keycloak
database with the only known-good source of trusted emails: Staff from
Archweb.

We'd like to make heavy use of GitLab CI for running automatic tests and
release automation. We're aware that the implication of eventually
allowing non-staff users to come in will result in untrusted code being
run on our CI. This is fine by itself but security-wise would prevent us
from creating trusted releases on the same CI runners. We currently have
two sponsored bare-metal GitLab CI runners that we plan on using for
running untrusted code. We'll get a new bare-metal box from Hetzner for
trusted releases that will only run on hand-picked pipelines that only a
select few of us can push to. Bare metal runners also allow us to test
and build VM images and such which isn't usually possible on most VPS.

On more goal we had is automatic github.com mirroring in some fashion.
We looked at creating a two-way github.com <-> GitLab mirror but that
setup can break easily in the case of force pushes and race conditions
and also would have us looking at both places for pull requests. It
seems simpler to us for the time being to have one-way mirroring from
our GitLab to github.com only and then allow github.com users to easily
collaborate on our GitLab via github.com social login. It's a little bit
more hassle for the users than collaborating directly on github.com but
it's a lot less hassle for us so it's perhaps the best compromise.

These changes will affect all of you in some way but we'll try to make
it as painless as possible. As we progress, we'll send concise emails
with instructions on what to do. I'm sure that we missed a few details
here and there but I'm still convinced that this is a change for the
better across the board, even if we don't immediately get everything right.

Sorry for the long mail. All of this has been a long time in the making
and has been the subject of at least four hackathons with many hours
spent in-between. It was a ton of work but I'm happy that we're finally
at a stage where we can present something tangible with a plan of attack.

Cheers,
Sven and the DevOps team

[0] https://gitlab.com/gitlab-com/marketing/community-relations/gitlab-oss
[1]
https://gitlab.com/gitlab-com/marketing/community-relations/gitlab-oss/-/commit/9c40905b9ceef41628d9a3ebbf06060060bc1384
[2] https://wiki.archlinux.org/index.php/DeveloperWiki:SSOMigration
[3]
https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-05-06

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/arch-dev-public/attachments/20200517/35c0ee45/attachment.sig>