[aur-general] community back-end rewrite
Hi all, In the TU meetings there was some discussion about doing a rewrite/update/fix-up of the [community] back-end. I am not sure who all is looking at this and what the plans are - hence this email. Can people who are looking at this or interested in looking at this give a ping back to this message with whatever plans you have or work you have already done. Things I want to flag: - we should get a git repo for a community-backend project on the arch server and separate this from the aur site code. - we should switch SCM tools from CVS. I am strongly in favour of using the same system as the main repos which would allow much code reuse and easily get us a testing repo for community packages. (No more big rebuild issues!). Allan
On Sun, Nov 30, 2008 at 20:35, Allan McRae <allan@archlinux.org> wrote:
Hi all,
In the TU meetings there was some discussion about doing a rewrite/update/fix-up of the [community] back-end. I am not sure who all is looking at this and what the plans are - hence this email.
Can people who are looking at this or interested in looking at this give a ping back to this message with whatever plans you have or work you have already done.
You're probably referring to me here. I have been considering writing a tupkg-like tool that will upload and repo-add things, and be able to have any sort of post-upload hooks that you want. I haven't really written anything yet. One of the main attractions is that you wouldn't really need any sort of cron job running to add things, which is one sucky thing about the current community backend
Heh. This happens when you don't read all of your e-mails before answering. So, here's a bit of copy 'n paste from my last message, and some new thoughts. On Mon, Dec 1, 2008 at 2:35 AM, Allan McRae <allan@archlinux.org> wrote:
In the TU meetings there was some discussion about doing a rewrite/update/fix-up of the [community] back-end. I am not sure who all is looking at this and what the plans are - hence this email.
I think the only work done is in AUR2, but I don't know if they have touched the backend. Could any of the involved people update us on its status?
Can people who are looking at this or interested in looking at this give a ping back to this message with whatever plans you have or work you have already done.
From my lst mail: «This is a very important point, IMHO, which we need to discuss with the AUR2 guys. In fact I think we could, as we say in Italian, "catch two pigeons with one broad bean" :) and plan the backend changes to include the move to SVN. Personally, I'd like to see an independent RCS layer so that we won't see this situation again when we need to move to $ZOMGCOOLRCS.»
Things I want to flag: - we should get a git repo for a community-backend project on the arch server and separate this from the aur site code. - we should switch SCM tools from CVS. I am strongly in favour of using the same system as the main repos which would allow much code reuse and easily get us a testing repo for community packages. (No more big rebuild issues!).
Here I agree with Loui: it doesn't make sense to separate the AUR and the [community] backend, they are tightly coupled and so they should be. The risk is to break things if two groups move independently. And I'm not really sure we could reuse some code, but writing an SCM abstraction layer that makes us independent from CVS and SVN could be userful for the main repos, too. I don't think it's a stupid idea because we just need the basic operations that every SCM provides... Corrado
On Mon, Dec 1, 2008 at 5:23 PM, bardo <ilbardo@gmail.com> wrote:
I think the only work done is in AUR2, but I don't know if they have touched the backend. Could any of the involved people update us on its status?
What AUR2?
Here I agree with Loui: it doesn't make sense to separate the AUR and the [community] backend, they are tightly coupled and so they should be. The risk is to break things if two groups move independently. And I'm not really sure we could reuse some code, but writing an SCM abstraction layer that makes us independent from CVS and SVN could be userful for the main repos, too. I don't think it's a stupid idea because we just need the basic operations that every SCM provides...
Also if we're talking about AUR2 I think the plan is definitely to separate them, but when that happens there won't be two sets of queries but rather abstracted. -- Callan Barrett
On Sun, Nov 30, 2008 at 7:35 PM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
In the TU meetings there was some discussion about doing a rewrite/update/fix-up of the [community] back-end. I am not sure who all is looking at this and what the plans are - hence this email.
Can people who are looking at this or interested in looking at this give a ping back to this message with whatever plans you have or work you have already done.
Things I want to flag: - we should get a git repo for a community-backend project on the arch server and separate this from the aur site code. - we should switch SCM tools from CVS. I am strongly in favour of using the same system as the main repos which would allow much code reuse and easily get us a testing repo for community packages. (No more big rebuild issues!).
To whoever is doing this: Please remain in touch with me for this. I would *love* it if the community backend just ran the official db scripts under the 'aur' user. This would not only simplify the code, but allow us to unify lots of changes. Additionally, the database itself should be updated the same way the official database is done - parse the db.tar.gz and insert/update/delete accordingly. In short: the community backend should really just accept packages via upload (authenticated, or course), dump them in a common location (/home/aur/staging) and then setup a cronjob to run db-update periodically. This cronjob should also find packages in the db.tar.gz and not in CVS and run db-remove on them. If we can collaborate enough on this, we could even use the same functionality for extra and core.
On Mon, Dec 1, 2008 at 9:55 AM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Sun, Nov 30, 2008 at 7:35 PM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
In the TU meetings there was some discussion about doing a rewrite/update/fix-up of the [community] back-end. I am not sure who all is looking at this and what the plans are - hence this email.
Can people who are looking at this or interested in looking at this give a ping back to this message with whatever plans you have or work you have already done.
Things I want to flag: - we should get a git repo for a community-backend project on the arch server and separate this from the aur site code. - we should switch SCM tools from CVS. I am strongly in favour of using the same system as the main repos which would allow much code reuse and easily get us a testing repo for community packages. (No more big rebuild issues!).
To whoever is doing this: Please remain in touch with me for this. I would *love* it if the community backend just ran the official db scripts under the 'aur' user. This would not only simplify the code, but allow us to unify lots of changes.
Additionally, the database itself should be updated the same way the official database is done - parse the db.tar.gz and insert/update/delete accordingly.
In short: the community backend should really just accept packages via upload (authenticated, or course), dump them in a common location (/home/aur/staging) and then setup a cronjob to run db-update periodically. This cronjob should also find packages in the db.tar.gz and not in CVS and run db-remove on them.
If we can collaborate enough on this, we could even use the same functionality for extra and core.
Note that the above regarding db-update requires a move to svn at the same time. I do not plan to add special code to deal with CVS
In short: the community backend should really just accept packages via upload (authenticated, or course), dump them in a common location (/home/aur/staging) and then setup a cronjob to run db-update periodically. This cronjob should also find packages in the db.tar.gz and not in CVS and run db-remove on them.
If we can collaborate enough on this, we could even use the same functionality for extra and core.
Note that the above regarding db-update requires a move to svn at the same time. I do not plan to add special code to deal with CVS
Regarding db-remove: I think is is an _absolute_ requirement that packages be easily moved back to unsupported. Currently the only way is to delete it entirely and re-upload as a new package (losing all related metadata). It's really horrible, and I think contributes to the glut of useless stuff in community at the moment.
On Mon, Dec 1, 2008 at 1:12 PM, Daenyth Blank <daenyth+arch@gmail.com> wrote:
In short: the community backend should really just accept packages via upload (authenticated, or course), dump them in a common location (/home/aur/staging) and then setup a cronjob to run db-update periodically. This cronjob should also find packages in the db.tar.gz and not in CVS and run db-remove on them.
If we can collaborate enough on this, we could even use the same functionality for extra and core.
Note that the above regarding db-update requires a move to svn at the same time. I do not plan to add special code to deal with CVS
Regarding db-remove: I think is is an _absolute_ requirement that packages be easily moved back to unsupported. Currently the only way is to delete it entirely and re-upload as a new package (losing all related metadata). It's really horrible, and I think contributes to the glut of useless stuff in community at the moment.
Sure, but that'd be more up to the AUR front-end that controls unsupported. It'd just have to copy the entries from CVS and put them in the unsupported dir. Should be simple. As for removing them, perhaps the front-end could even remove them from CVS, but that seems a little gross.
On Mon, Dec 1, 2008 at 1:33 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Mon, Dec 1, 2008 at 1:12 PM, Daenyth Blank <daenyth+arch@gmail.com> wrote:
In short: the community backend should really just accept packages via upload (authenticated, or course), dump them in a common location (/home/aur/staging) and then setup a cronjob to run db-update periodically. This cronjob should also find packages in the db.tar.gz and not in CVS and run db-remove on them.
If we can collaborate enough on this, we could even use the same functionality for extra and core.
Note that the above regarding db-update requires a move to svn at the same time. I do not plan to add special code to deal with CVS
Regarding db-remove: I think is is an _absolute_ requirement that packages be easily moved back to unsupported. Currently the only way is to delete it entirely and re-upload as a new package (losing all related metadata). It's really horrible, and I think contributes to the glut of useless stuff in community at the moment.
Sure, but that'd be more up to the AUR front-end that controls unsupported. It'd just have to copy the entries from CVS and put them in the unsupported dir. Should be simple. As for removing them, perhaps the front-end could even remove them from CVS, but that seems a little gross.
Hmm, perhaps we could just set a rule such that any removal at all is copied to the unsupported dir, and then people could simply delete the package from the front-end.
On Mon, Dec 1, 2008 at 14:46, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Hmm, perhaps we could just set a rule such that any removal at all is copied to the unsupported dir, and then people could simply delete the package from the front-end.
That makes a lot of good sense.
On Mon, Dec 1, 2008 at 9:22 PM, Daenyth Blank <daenyth+arch@gmail.com> wrote:
On Mon, Dec 1, 2008 at 14:46, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Hmm, perhaps we could just set a rule such that any removal at all is copied to the unsupported dir, and then people could simply delete the package from the front-end.
That makes a lot of good sense.
Seconded. Unless we want repos to be more hierarchical, were a removal from core goes to extra, from extra to community, from community to unsupported. Somewhat impractical, imho. An alternative could be the implementation of a "move" abstraction. Then moving packages from any repo to any other (unsupported included) could be really simple and straightforward. Corrado
On Tue, Dec 2, 2008 at 5:38 AM, bardo <ilbardo@gmail.com> wrote:
On Mon, Dec 1, 2008 at 9:22 PM, Daenyth Blank <daenyth+arch@gmail.com> wrote:
On Mon, Dec 1, 2008 at 14:46, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Hmm, perhaps we could just set a rule such that any removal at all is copied to the unsupported dir, and then people could simply delete the package from the front-end.
That makes a lot of good sense.
Seconded. Unless we want repos to be more hierarchical, were a removal from core goes to extra, from extra to community, from community to unsupported. Somewhat impractical, imho. An alternative could be the implementation of a "move" abstraction. Then moving packages from any repo to any other (unsupported included) could be really simple and straightforward.
The official repos are handled much differently and DO have move syntax. It's how we move things from testing to core. This is easy because they are in the same svn repo. But I don't thing we need to go overboard here. A system to move to and from extra and community would be good. but it's not needed at the moment.
participants (5)
-
Aaron Griffin
-
Allan McRae
-
bardo
-
Callan Barrett
-
Daenyth Blank