[arch-devops] Moving projects.archlinux.org to luna.archlinux.org
Hello,
Florian suggested to move cgit and archweb to nymeria or luna. In this mail, I will only focus on cgit, better known as projects.archlinux.org.
Currently: - the web interface is a cgit on gudrun. - http server is apache and run cgit via cgi-bin - cgit scan /srv/git for repositories - /srv/git is a NFS mountpoint from gerolde - so, the git repositories are on gerolde - users push to repos via ssh directly on gerolde. They will have to update their remotes url. - the total space of git repositories is about 730MiB
Currently, my preference would be to move the storage and the web interface on the same host. I see more benefits than moving web interface on luna and git repo on nymeria. We don't eat the space twice and we have all the repo accessible under the projects.al.org dns.
Nonetheless, if we move the storage to nymeria, we save the configuration of ssh access to all devs + externals to luna. Not sure if this is relevant if we move to accounts to ldap.
Thoughts?
Here is a short planning if we go through this option: 1) setup a new cgit on luna (with nginx and uwsgi) 2) copy repos for testing purpose 3) create gitshell access to users 4) plan official migration date 5) reduce dns ttl of projects.al.org 6) wait for it 7) cut write access to gerolde repo 8) final repo sync 9) update dns records and restore ttl 10) take a beer
I see there is also dev.archlinux.org on gudrun that I could also move to luna too.
Cheers,
On 10.01.2016 00:57, Sébastien Luttringer wrote:
- users push to repos via ssh directly on gerolde. They will have to update
their remotes url.
Let's make sure we use some generic subdomain this time (something like git.archlinux.org or projects-push.archlinux.org).
Nonetheless, if we move the storage to nymeria, we save the configuration of ssh access to all devs + externals to luna. Not sure if this is relevant if we move to accounts to ldap.
If we have the time to set up ldap (including replication and either ssl or a vpn), then that's fine by me. I'd like to clean up gudrun/gerolde somewhat soon so we can get rid of xen and potentially the ancient hardware that is running it. I'd prefer to delay this cleanup as little as (sanely) possible.
While saving space is always nice, nymeria has plenty of it. It currently has >500gb free disk so 1gb repos won't be a problem. luna has to store them anyway, so no gain there.
I intended to look into ldap at some point myself, but if we want to use it for git that would delay the move too much. Anyone here has experience and an idea how long it would take to set up?
We could still move to ldap later without affecting users (dedicated subdomain).
Here is a short planning if we go through this option:
The plan looks good (if we take the ldap route).
I see there is also dev.archlinux.org on gudrun that I could also move to luna too.
I'm not sure what dev.archlinux.org is used for. Isn't that some kind of personal web space for devs? Can we move it that easily?
On 10/01, Florian Pritz wrote:
I'm not sure what dev.archlinux.org is used for. Isn't that some kind of personal web space for devs? Can we move it that easily?
I think it might be meant to be used as the domain for running dev versions of archweb, as aur-dev is for the AUR.
On 2016-01-10 10:23, Florian Pritz wrote:
I'm not sure what dev.archlinux.org is used for. Isn't that some kind of personal web space for devs? Can we move it that easily?
I think that's the only use if this subdomain at the moment. Does anyone use it actively nowadays? We could probably drop it entirely.
Bartłomiej
On dim., 2016-01-10 at 10:23 +0100, Florian Pritz wrote:
On 10.01.2016 00:57, Sébastien Luttringer wrote: Let's make sure we use some generic subdomain this time (something like git.archlinux.org or projects-push.archlinux.org).
I agree. Let's go with git.archlinux.org.
If we have the time to set up ldap (including replication and either ssl or a vpn), then that's fine by me. I'd like to clean up gudrun/gerolde somewhat soon so we can get rid of xen and potentially the ancient hardware that is running it. I'd prefer to delay this cleanup as little as (sanely) possible.
My idea was to locally create the needed accounts to move projects to luna and then deal with ldap on all our hosts. So we can get ride of the Xen hardware without deps on ldap.
While saving space is always nice, nymeria has plenty of it. It currently has >500gb free disk so 1gb repos won't be a problem. luna has to store them anyway, so no gain there.
Correct.
Another *small* pro of having repo on luna is that we have git push immediately available, no sync delay.
I intended to look into ldap at some point myself, but if we want to use it for git that would delay the move too much. Anyone here has experience and an idea how long it would take to set up?
If nobody start before, I will have free time in the beginning of February. So at this moment I can manage to setup an ldap. Currently I can't.
I see there is also dev.archlinux.org on gudrun that I could also move to luna too.
I'm not sure what dev.archlinux.org is used for. Isn't that some kind of personal web space for devs? Can we move it that easily?
Yep, it's personal web space for devs. The only difficulties I see is moving user accounts to celestia (like for projects to luna) until we have ldap. I think we should move it to celestia and merge it with the existing personal webspace of pkgbuild.com (making one url redirect to the other)
Cheers,
On dim., 2016-01-10 at 00:57 +0100, Sébastien Luttringer wrote:
Here is a short planning if we go through this option:
- setup a new cgit on luna (with nginx and uwsgi)
- copy repos for testing purpose
- create gitshell access to users
- plan official migration date
- reduce dns ttl of projects.al.org
- wait for it
- cut write access to gerolde repo
- final repo sync
- update dns records and restore ttl
- take a beer
Re,
To give you a status of where I am. - A working cgit is setup on luna (accessible via a vhost git.archlinux.org - no dns atm) - The cgit is based on our package in community (not our custom version) - I managed users deploy via an ansible playbook (still few missing)
I need to refine some details of the cgit config. I also need to setup the ssh and git-daemon access. Then DNS stuff.
I'm planning to finish the config and annonce the migration on a-d-p@al.org before the end of the week.
Cheers,
On 2016-02-23 01:18, Sébastien Luttringer wrote:
- The cgit is based on our package in community (not our custom version)
Please talk to Lukas Fleischer about it, I'm not sure what exactly was customized but surely Lukas did it.
BP
On mar., 2016-02-23 at 01:57 +0100, Bartłomiej Piotrowski wrote:
On 2016-02-23 01:18, Sébastien Luttringer wrote:
- The cgit is based on our package in community (not our custom version)
Please talk to Lukas Fleischer about it, I'm not sure what exactly was customized but surely Lukas did it.
He's reading us. If this is a bad idea, he will answer and explain why here.
Cheers,
On mar., 2016-02-23 at 01:18 +0100, Sébastien Luttringer wrote:
On dim., 2016-01-10 at 00:57 +0100, Sébastien Luttringer wrote:
Here is a short planning if we go through this option:
- setup a new cgit on luna (with nginx and uwsgi)
- copy repos for testing purpose
- create gitshell access to users
- plan official migration date
- reduce dns ttl of projects.al.org
- wait for it
- cut write access to gerolde repo
- final repo sync
- update dns records and restore ttl
- take a beer
.. I'm planning to finish the config and annonce the migration on a-d-p@al.org before the end of the week.
Cheers,
Ok, so I'm late, as usual...
Where we are now: - git.al.org dns is public. - the new download url (https, git, ssh) are working. - cgit config looks good. - a crontab is syncing git repos every hour (from gerolde to luna only). - users with git repo have been created on luna (via ansible). - I moved the remaining fellows devs git repos under the "Obsolete or Dead Projects".
As url will be redirected (except for pushing) I would plan a short migration delay. Something like this Saturday night CET.
If no objection, I will start discutions tomorrow about migration in a-d-p@al.o rg.
Cheers,
On 29.02.2016 05:35, Sébastien Luttringer wrote:
Where we are now:
The performance is nice, but I've noticed that you disable gzip compression. I prefer the performance boost it provides and it can really save tons of traffic. Why have you disabled it for cgit?
Florian
n lun., 2016-02-29 at 05:35 +0100, Sébastien Luttringer wrote:
On mar., 2016-02-23 at 01:18 +0100, Sébastien Luttringer wrote:
On dim., 2016-01-10 at 00:57 +0100, Sébastien Luttringer wrote:
Here is a short planning if we go through this option:
- setup a new cgit on luna (with nginx and uwsgi)
- copy repos for testing purpose
- create gitshell access to users
- plan official migration date
- reduce dns ttl of projects.al.org
- wait for it
- cut write access to gerolde repo
- final repo sync
- update dns records and restore ttl
- take a beer
.. I'm planning to finish the config and annonce the migration on a-d-p@al.org before the end of the week.
Cheers,
Ok, so I'm late, as usual...
Where we are now:
- git.al.org dns is public.
- the new download url (https, git, ssh) are working.
- cgit config looks good.
- a crontab is syncing git repos every hour (from gerolde to luna only).
- users with git repo have been created on luna (via ansible).
- I moved the remaining fellows devs git repos under the "Obsolete or Dead
Projects".
As url will be redirected (except for pushing) I would plan a short migration delay. Something like this Saturday night CET.
If no objection, I will start discutions tomorrow about migration in a-d-p@al .o
Lukas objected to the move to the community version of cgit because we have a patch to remove the branch switch for svntogit tree in order to save few Kibibytes. This patch was rejected by upstream and he thinks it's not a valuable solution to add it to the community package.
So, I will move back to the version we maintain in order to move forward to the migration.
Cheers,
On Wed, 23 Mar 2016 at 11:27:05, Sébastien Luttringer wrote:
[...] Lukas objected to the move to the community version of cgit because we have a patch to remove the branch switch for svntogit tree in order to save few Kibibytes. This patch was rejected by upstream and he thinks it's not a valuable solution to add it to the community package.
Just to clarify: "few Kibibytes" means bandwidth savings of ~95% for the svntogit cgit pages. And I do think that the patch is valuable but I do not think that patching an official package to add a feature that was rejected by upstream is in line with the Arch Linux packaging guidelines.
So, I will move back to the version we maintain in order to move forward to the migration. [...]
Not sure if you read my earlier email on this but there is [1] which can be used to automate all our cgit customizations (including updating the copyright information in the footer etc.) We might want to rename that repository to git.archlinux.org.git if that is the new name.
Regards, Lukas
[1] https://projects.archlinux.org/vhosts/projects.archlinux.org.git/
On mer., 2016-03-23 at 12:11 +0100, Lukas Fleischer wrote:
On Wed, 23 Mar 2016 at 11:27:05, Sébastien Luttringer wrote:
Not sure if you read my earlier email on this but there is [1] which can be used to automate all our cgit customizations (including updating the copyright information in the footer etc.) We might want to rename that repository to git.archlinux.org.git if that is the new name.
Yes, I read it and it's perfectly fine to me to go this way. I was trying to make a public summary of what I will do next and why there is a little change in what was announced before.
Cheers,
On mer., 2016-03-23 at 13:31 +0100, Sébastien Luttringer wrote:
On mer., 2016-03-23 at 12:11 +0100, Lukas Fleischer wrote:
On Wed, 23 Mar 2016 at 11:27:05, Sébastien Luttringer wrote:
Not sure if you read my earlier email on this but there is [1] which can be used to automate all our cgit customizations (including updating the copyright information in the footer etc.) We might want to rename that repository to git.archlinux.org.git if that is the new name.
Yes, I read it and it's perfectly fine to me to go this way. I was trying to make a public summary of what I will do next and why there is a little change in what was announced before.
Cheers,
Hello,
git.archlinux.org is now running with projects.al.org git version.
Users are deployed on luna based on ansible scripts[1]. -git groups stay "local".
I'd appreciate some testing. I'll announce the move tonight on a-d-p.
Cheers,
[1] https://git.archlinux.org/users/seblu/ansible.git/
participants (6)
-
Bartłomiej Piotrowski
-
Florian Pritz
-
Johannes Löthberg
-
Lukas Fleischer
-
Sébastien Luttringer
-
Sébastien Luttringer