[arch-dev-public] Getting rid of the ABS <-> CVS link
Hey guys, I wanted to bring this up here to see if anyone had any input. For the longest time I have hated that ABS is dependent on our version control system. Not only that but we have this whole csup/cvsup thing which is annoying, and _now_ we hit yet another snag: csup doesn't have cvsupd, which means we can't get rid of the cvsup package on gerolde. So yeah, CVS == headache. I wanted to try to get rid of this, and Dan had a brilliant idea: * Some cron job (or even cvs hook) that does a simple export of the repo when it needs to. * rsyncd supplying that exports that dir * Modify 'abs' to use rsync * Boom, done Does anyone have any issues with something like that? Or any better ideas?
On Fri, 2007-11-16 at 15:03 -0600, Aaron Griffin wrote:
Hey guys, I wanted to bring this up here to see if anyone had any input.
For the longest time I have hated that ABS is dependent on our version control system. Not only that but we have this whole csup/cvsup thing which is annoying, and _now_ we hit yet another snag: csup doesn't have cvsupd, which means we can't get rid of the cvsup package on gerolde.
So yeah, CVS == headache.
I wanted to try to get rid of this, and Dan had a brilliant idea: * Some cron job (or even cvs hook) that does a simple export of the repo when it needs to. * rsyncd supplying that exports that dir * Modify 'abs' to use rsync * Boom, done
Does anyone have any issues with something like that? Or any better ideas?
What about installing an export in place when we run the db-* scripts. Shouldn't be that hard, as we export CVS to generate the DB anyways.
On 11/16/07, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Hey guys, I wanted to bring this up here to see if anyone had any input.
For the longest time I have hated that ABS is dependent on our version control system. Not only that but we have this whole csup/cvsup thing which is annoying, and _now_ we hit yet another snag: csup doesn't have cvsupd, which means we can't get rid of the cvsup package on gerolde.
So yeah, CVS == headache.
I wanted to try to get rid of this, and Dan had a brilliant idea: * Some cron job (or even cvs hook) that does a simple export of the repo when it needs to. * rsyncd supplying that exports that dir * Modify 'abs' to use rsync * Boom, done
Does anyone have any issues with something like that? Or any better ideas?
Sounds full of win to me. 1. We already leverage rsync. It will just be some setup on our end. 2. rsync is scm agnostic. If we change scm in the future, we just change the export dir generation sequence. To the end user, it is seamless. 3. It should migrate well. End users will just get the new abs package, and sync. There should be no need to nuke and refetch abs..just update with rsync. 4. One less service running on gerolde. Ever run top and watch cvsupd? lulz 5. We can throttle the speed with rsyncd configuration options if we want. This might be a nice way to make sure people don't go crazy with it or something. 6. Standard from the end user perspective. Lots of similar things use rsync. I think bsd ports can be rsync fetched, for instance.
On Fri, 2007-11-16 at 13:29 -0800, eliott wrote:
On 11/16/07, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Hey guys, I wanted to bring this up here to see if anyone had any input.
For the longest time I have hated that ABS is dependent on our version control system. Not only that but we have this whole csup/cvsup thing which is annoying, and _now_ we hit yet another snag: csup doesn't have cvsupd, which means we can't get rid of the cvsup package on gerolde.
So yeah, CVS == headache.
I wanted to try to get rid of this, and Dan had a brilliant idea: * Some cron job (or even cvs hook) that does a simple export of the repo when it needs to. * rsyncd supplying that exports that dir * Modify 'abs' to use rsync * Boom, done
Does anyone have any issues with something like that? Or any better ideas?
Sounds full of win to me. 1. We already leverage rsync. It will just be some setup on our end.
We should set it up on a different hostname and port. rsync://abs.archlinux.org:874 comes to mind. This will allow us to run a separate rsync service where we can force compression,etc on or off as we see fit.
2. rsync is scm agnostic. If we change scm in the future, we just change the export dir generation sequence. To the end user, it is seamless.
Agreed.
3. It should migrate well. End users will just get the new abs package, and sync. There should be no need to nuke and refetch abs..just update with rsync.
Right. But do we use --delete in the abs client?
4. One less service running on gerolde. Ever run top and watch cvsupd? lulz
rsync has spent its fair share of time at the top of top as well.
5. We can throttle the speed with rsyncd configuration options if we want. This might be a nice way to make sure people don't go crazy with it or something.
Yep. See my response to #1.
6. Standard from the end user perspective. Lots of similar things use rsync. I think bsd ports can be rsync fetched, for instance.
Good to know. Can you confirm this? Dale
1. We already leverage rsync. It will just be some setup on our end.
We should set it up on a different hostname and port. rsync://abs.archlinux.org:874 comes to mind. This will allow us to run a separate rsync service where we can force compression,etc on or off as we see fit.
I believe we can do so on a per stanza basis, so if we set a target for abs, then we can set options inside that stanza. It will take a bit of reworking of the existing config file maybe, but I don't think it will be overly problematic. I would like to avoid using alternate ports for this kind of stuff. For services that are for public consumption, it is good to use standard ports..so that it is more conformant to firewalling, etc.
3. It should migrate well. End users will just get the new abs package, and sync. There should be no need to nuke and refetch abs..just update with rsync.
Right. But do we use --delete in the abs client?
I would think so. We would just want to make sure to set a client side exclude option for /var/abs/local/
6. Standard from the end user perspective. Lots of similar things use rsync. I think bsd ports can be rsync fetched, for instance.
Good to know. Can you confirm this?
I can't seem to find a reference for my anecdotal statement. I didn't search very hard though. I thought I ran across it at one point, but who knows which BSD it was... maybe netbsd... *head scratching*
On Nov 16, 2007 5:59 PM, eliott <eliott@cactuswax.net> wrote:
1. We already leverage rsync. It will just be some setup on our end.
We should set it up on a different hostname and port. rsync://abs.archlinux.org:874 comes to mind. This will allow us to run a separate rsync service where we can force compression,etc on or off as we see fit.
I believe we can do so on a per stanza basis, so if we set a target for abs, then we can set options inside that stanza. It will take a bit of reworking of the existing config file maybe, but I don't think it will be overly problematic.
I would like to avoid using alternate ports for this kind of stuff. For services that are for public consumption, it is good to use standard ports..so that it is more conformant to firewalling, etc.
3. It should migrate well. End users will just get the new abs package, and sync. There should be no need to nuke and refetch abs..just update with rsync.
Right. But do we use --delete in the abs client?
I would think so. We would just want to make sure to set a client side exclude option for /var/abs/local/
6. Standard from the end user perspective. Lots of similar things use rsync. I think bsd ports can be rsync fetched, for instance.
Good to know. Can you confirm this?
I can't seem to find a reference for my anecdotal statement. I didn't search very hard though. I thought I ran across it at one point, but who knows which BSD it was... maybe netbsd... *head scratching*
Poke. Eliott- would you be willing to work on the server side of this? Obviously this is no rush deal, but I feel like it would be easier to write a client tool if I knew the layout and specifics of the server side. I think we would stick with our current abs layout: <basedir> |--core |--extra |--community |--unstable |--testing | |--core | |--extra | |--unstable | |--community (ok, not yet, but in the future) |--PKGBUILD.proto |--proto.install |--local (empty directory) I don't know if it would need to be laid out like this on the server (it isn't now, but that is csup), but it would probably be in our best interest if we limited rsync connections to one per abs fetch. -Dan
Poke. Eliott- would you be willing to work on the server side of this? Obviously this is no rush deal, but I feel like it would be easier to write a client tool if I knew the layout and specifics of the server side.
I'm down chief.
I think we would stick with our current abs layout: <basedir> |--core |--extra |--community |--unstable |--testing | |--core | |--extra | |--unstable | |--community (ok, not yet, but in the future) |--PKGBUILD.proto |--proto.install |--local (empty directory)
I don't know if it would need to be laid out like this on the server (it isn't now, but that is csup), but it would probably be in our best interest if we limited rsync connections to one per abs fetch.
It would be optimal (read: easiest) if we could just rsync for the whole tree..one config stanza. That might require a bit more work on the checkout client thing (that will populate the server side rsync dir). Not sure..
On Nov 18, 2007 9:28 PM, eliott <eliott@cactuswax.net> wrote:
That might require a bit more work on the checkout client thing (that will populate the server side rsync dir). Not sure..
Hopefully it is something this easy: cd /srv/abs/i686 cvs export -d /home/cvs-core -r CURRENT core cvs export -d /home/cvs-extra -r CURRENT extra cvs export -d /home/cvs-community -r CURRENT community cd ./testing cvs export -d /home/cvs-core -r TESTING core cvs export -d /home/cvs-extra -r TESTING extra <repeat for x86_64> See ~dan/tempexport/ for the above. We could cronjob this on the hour or something. -Dan
participants (5)
-
Aaron Griffin
-
Dale Blount
-
Dan McGee
-
eliott
-
Jan de Groot