Hi everyone, As some of you know, we've been toying with two ideas for a while: Arch-wide centralized user management as well as using GitLab to consolidate some of our current services. The overall goal is manifold. In no particular order, the goals are to - make Arch more contributor friendly - provide more modern tools for ourselves - enable more automation - make Arch services more secure - make team management activities less error-prone and more streamlined These two topics (SSO + Gitlab) are a bit intertwined because we wanted to have SSO on GitLab before starting off with that so we'd have a properly validated user base to work with going forward. Also, GitLab seemed like a good first service for SSO due to it having good support for that. After looking at various solutions, we eventually settled on Keycloak since it seemed like a modern, well-maintained, and secure piece of software. It allows us to enable logins for services via OpenID Connect and SAML which is likely the best coverage we could hope for. It also allows us to connect other social login providers such as github.com and gitlab.com and it supports Recaptcha, 2FA, and WebAuthn out of the box. The idea is to eventually transition all our online service as well as SSH keys to Keycloak to ease on-/offboarding and make it less error-prone. As for GitLab, a few months back, we applied for a GitLab Ultimate license in their open-source program [0] and we received one [1]. It's an official program that many other open-source projects benefit from as well and we think it's safe to assume that it'll continue being a thing for the foreseeable future. We have to renew our license yearly. The current license we have has support for 1000 seats but we can likely get more seats if we need them. Our general path is: 1) Transition as many staff-only services to Keycloak as possible. We looked at our current services and put up a table on the wiki that shows support per service [2]. Some of the services that we operate are deprecated or have functionality that is also provided by GitLab. In our current understanding, this concerns Flyspray, Kanboard, and Patchwork. Of those, Kanboard and some of the Flyspray projects will be our first targets to transition to GitLab. We'll continue using Flyspray for the time being for package bugs but will discontinue its use for all non-packaging bugs. The reason for this is that how we manage our bugs for packages is somewhat intertwined with the svn2git migration which is yet to be done and might dictate a different repo structure than what we would come up with currently and we don't want to block on this. This was also discussed in a recent DevOps meeting [3]. 2) We'd like to get rid of our own cgit instance at git.archlinux.org and transition our git hosting into GitLab. AUR git access will stay as it is due to its special shell magic. 3) Eventually, after an internal testing phase of at least a few weeks, we'll want to open Keycloak and GitLab up for outside contributors. We know of the abuse potential and the potential moderation problems and have to make sure to set proper limits and set up monitoring before opening this up. 4) Connect remaining services like BBS and wiki to SSO. In 1) we only mentioned staff-only services because those are less problematic. However, in the future, we'd also like to enable our remaining services to connect to Keycloak. We tried hard to come up with a good source of users to import into Keycloak so that we could seed that database with a solid user base but sadly it appears that there is no trusted source of users that we can rely on. Potential candidates were the wiki, BBS, AUR but we ruled them all out in their current state as none of them have always had email verification and so we can't trust those emails to be the sole source of truth. In order to still allow users to keep their old contributions in cases where they can prove their identity via email, we'll build a new small web application that allows them to connect their new Keycloak identity to their other identities. For now, we seeded the Keycloak database with the only known-good source of trusted emails: Staff from Archweb. We'd like to make heavy use of GitLab CI for running automatic tests and release automation. We're aware that the implication of eventually allowing non-staff users to come in will result in untrusted code being run on our CI. This is fine by itself but security-wise would prevent us from creating trusted releases on the same CI runners. We currently have two sponsored bare-metal GitLab CI runners that we plan on using for running untrusted code. We'll get a new bare-metal box from Hetzner for trusted releases that will only run on hand-picked pipelines that only a select few of us can push to. Bare metal runners also allow us to test and build VM images and such which isn't usually possible on most VPS. On more goal we had is automatic github.com mirroring in some fashion. We looked at creating a two-way github.com <-> GitLab mirror but that setup can break easily in the case of force pushes and race conditions and also would have us looking at both places for pull requests. It seems simpler to us for the time being to have one-way mirroring from our GitLab to github.com only and then allow github.com users to easily collaborate on our GitLab via github.com social login. It's a little bit more hassle for the users than collaborating directly on github.com but it's a lot less hassle for us so it's perhaps the best compromise. These changes will affect all of you in some way but we'll try to make it as painless as possible. As we progress, we'll send concise emails with instructions on what to do. I'm sure that we missed a few details here and there but I'm still convinced that this is a change for the better across the board, even if we don't immediately get everything right. Sorry for the long mail. All of this has been a long time in the making and has been the subject of at least four hackathons with many hours spent in-between. It was a ton of work but I'm happy that we're finally at a stage where we can present something tangible with a plan of attack. Cheers, Sven and the DevOps team [0] https://gitlab.com/gitlab-com/marketing/community-relations/gitlab-oss [1] https://gitlab.com/gitlab-com/marketing/community-relations/gitlab-oss/-/com... [2] https://wiki.archlinux.org/index.php/DeveloperWiki:SSOMigration [3] https://wiki.archlinux.org/index.php/DeveloperWiki:DevopsMeetings/2020-05-06