[aur-dev] [AUR2] Need some feedback
I'll cover a few things in this mail since I didn't want to spam the ML with 3+ threads. First off I'd like some feedback on the RPC interface. I know that AUR started work on using JSON, and in my mind it's the best contender. Django has inbuilt support for serialising to XML, JSON and YAML, so I could easily use either of those. I'm mainly interested in the user end of this - the AUR frontend (yaourt, etc.) developers. I haven't found anything to parse JSON or XML easily from bash, so I don't think it would be easy to implement either way in yaourt and other frontends. There's a Perl CPAN module[1] which may be easily used to quickly parse the data and allow the rest of the work to be done via bash, which, I guess, would be a decent compromise. In my opinion XML is just way too heavy and has too many unnecessary features for what we're going to use it for. YAML seems to be about the same, but JSON is a nice data oriented RPC that can be quickly parsed. If all goes well and we choose JSON, the interace will be compatible with AUR's, though I haven't looked int othe code much yet. On the wiki[1] someone asked to have versioned dependencies. Currently I've only implemented listing of dependencies and don't even track the version requirements. I'm not certain that this functionality is necessary at all. It has very little use, and the only use cases I can think of are validation the package, so that it's compatible with the dependencies (ie. disallowing to upload a package that depends on an older version of a package in AUR), and simply being notified that the versions don't match. Both of these, in my opinion, are pointless. As with versioned dependencies, someone asked for the ability to search for outdated packages. There are two use cases that I can think of for this. One is for a TU to find any outdated packages in Community in order to update them. I'm not exactly sure of the TU etiquette, as I'm not a TU, so I don't know if TUs update other TUs packages if they are out of date and thus this use case may not be valid. Another use case is to simply find all the outdated packages that you maintain. This is a bit pointless and probably not suitable for the search page. Any maintainer will most likely be able to quickly see any outdated packages in their Account page or by searching for packages maintained by them (since they are marked). Otherwise it could be implemented in the Account page, something like a checkbox which would toggle whether to show only out dated packages. I'm not really enthusiastic about this feature, since it is quite pointless. Another request was for regexp searching. This can probably be easily implemented since Django has this functionality built in [3]. I'm not certain that this feature would be useful. Personally I can usually (99% of the time) find the packages I want by using simple fulltext searches. I know that AUR frontends do regexp searches to be "compatible", in a way, with pacman's searching abilities. This is the only real use case I can think of. Most users of the web interface probably won't need it, and therefor it could be implemented in the RPC interface instead. Now the biggest issue I have is integrating an SCM. This should probably go in a separate thread, but I may as well discuss it here. I don't know how AUR or the main site handle it, I'd have to look into that, but I would really like to integrate an SCM backend for handling files. Initially I had the idea for the interaction to go both ways; a user could simply "git commit -a -m 'bumped version'; git push" and the everything would be updated in the database, but also for the ability to do it via the web interface like it works now. Dan pointed out that it would be hard to restrict access for the maintainers to the various repositories/modules (or however the repository would be split). The whole idea could generally raise many problems, so it may be better to go for an easier solution and have it only work one way, ie. via the web interface. It appears the TUs use CVS directly to upload packages, so they may be a but frustrated with that move. There's also the issue of which SCM to use. It seems the main site is leaning towards converting to SVN (and rsync) from CVS, but git is also a contender. Optimally only portions of the repositories could be checked out, so instead of checking out the whole repository, as it currently is with abs, you'd be able to just have the packages that you maintain. I don't know of any SCM that implements this well though. I would really love some feedback on any ways to implement this, which SCM to use, etc. But I don't want discussions to go on for ever like they are with the main site conversion. Thanks for any feedback. [1] http://search.cpan.org/~makamaka/JSON-2.02/lib/JSON.pm [2] http://wiki.archlinux.org/index.php/AUR_2#Ideas [3] http://www.djangoproject.com/documentation/db-api/#regex
First off I'd like some feedback on the RPC interface. I know that AUR started work on using JSON, and in my mind it's the best contender. Django has inbuilt support for serialising to XML, JSON and YAML, so I could easily use either of those. I'm mainly interested in the user end of this - the AUR frontend (yaourt, etc.) developers. I haven't found anything to parse JSON or XML easily from bash, so I don't think it would be easy to implement either way in yaourt and other frontends. There's a Perl CPAN module[1] which may be easily used to quickly parse the data and allow the rest of the work to be done via bash, which, I guess, would be a decent compromise. In my opinion XML is just way too heavy and has too many unnecessary features for what we're going to use it for. YAML seems to be about the same, but JSON is a nice data oriented RPC that can be quickly parsed. If all goes well and we choose JSON, the interace will be compatible with AUR's, though I haven't looked int othe code much yet.
I myself am *very* fond of json data exchange. Simple http get requests returning json formatted data is lightweight, fairly restful, and damn easy to scale (even cache/proxy). I am not a huge fan of xml for data exchange, and consider it more of a ... display language. *shrug* json is a subset of yaml. I like yaml just fine, but it doesn't seem as well supported by most things web oriented as json (raw javascript, javascript libraries, other language libraries, etc). Other than that, I do like yaml. As far as bash processing, since many languages have decent json libraries (like python), it would be fairly simple to have a python helper script (or some simple C binary) that spews out the data in some well defined format that people can sed/awk/grep to joy. SOME_DATA=`wget http://masterful.aur.site/json?q=my+awesome+app+query -O - | /usr/sbin/return-joy_aur-helper`
On the wiki[1] someone asked to have versioned dependencies. Currently I've only implemented listing of dependencies and don't even track the version requirements. I'm not certain that this functionality is necessary at all. It has very little use, and the only use cases I can think of are validation the package, so that it's compatible with the dependencies (ie. disallowing to upload a package that depends on an older version of a package in AUR), and simply being notified that the versions don't match. Both of these, in my opinion, are pointless.
I agree. Largely pointless for the vast majority. Edge cases shouldn't be the focus of development.
As with versioned dependencies, someone asked for the ability to search for outdated packages.
Outdated packages in the aur? Right now there is only a comment system for people knowing if a package is out of date. This doesn't yield useful searchable criteria.
Another request was for regexp searching.
Regex searching seems a bit... excessive. mysql FULLTEXT style searching seems 'good enough' for the aur. I just can't think of a scenario where I would need to bust out with regex to find the name of a package I was looking for. One of the main strengths of regex is the capture ability, and that just isn't a usecase for raw searching. Like I said, for the aur it seems like an over engineering style feature request just for coolness sake.
Now the biggest issue I have is integrating an SCM. This should probably go in a separate thread, but I may as well discuss it here. I don't know how AUR or the main site handle it, I'd have to look into that, but I would really like to integrate an SCM backend for handling files. Initially I had the idea for the interaction to go both ways; a user could simply "git commit -a -m 'bumped version'; git push" and the everything would be updated in the database, but also for the ability to do it via the web interface like it works now. Dan pointed out that it would be hard to restrict access for the maintainers to the various repositories/modules (or however the repository would be split). The whole idea could generally raise many problems, so it may be better to go for an easier solution and have it only work one way, ie. via the web interface. It appears the TUs use CVS directly to upload packages, so they may be a but frustrated with that move. There's also the issue of which SCM to use. It seems the main site is leaning towards converting to SVN (and rsync) from CVS, but git is also a contender. Optimally only portions of the repositories could be checked out, so instead of checking out the whole repository, as it currently is with abs, you'd be able to just have the packages that you maintain. I don't know of any SCM that implements this well though. I would really love some feedback on any ways to implement this, which SCM to use, etc. But I don't want discussions to go on for ever like they are with the main site conversion.
The arch dev team will be using subversion. Aaron decided enough discussion was enough, and we can always change later if need be. No sense sitting in mud because you don't know if you want to wear pants or shorts once you get clean. I don't think general scm submission access makes any sense at all for the aur. Now, for backend storage, *maybe*, but that is debatable. I think the AUR needs to get 'back to roots', and focus on doing the job it needs to do, and doing it well. I think extraneous features that people want to add to AUR2 development should be punted, or forgotten altogether. Focus on core use and mainstream features. ** Massive Sidebar ** As a sidenote, I think the TU devs (community) should have a more custom arch-dev like interface (cli tools, flag out of date interface, etc), and not really be included in the aur at all. Alot of the TU functionality was somewhat 'bolted on', and wasn't ever really a good fit for the real core use cases of the AUR.
I don't think general scm submission access makes any sense at all for the aur. Now, for backend storage, *maybe*, but that is debatable.
I think an SCM backend would be useful. Having a history of changes would help if someone wanted to downgrade a package, or possibly even if someone found a bug in it. Without the two-way interaction with the SCM I guess there aren't any other benefits, and a history could be maintained via comments like it is now (or a special History relation). I'd have to look further into the advantages and disadvantages of implementing it this way.
I think the AUR needs to get 'back to roots', and focus on doing the job it needs to do, and doing it well. I think extraneous features that people want to add to AUR2 development should be punted, or forgotten altogether. Focus on core use and mainstream features.
Do you have any suggestions? I can't think of any other useful features at the moment. I haven't actually implemented any of the community stuff, or user management, so there's still more to dig into in terms of replicating features.
** Massive Sidebar ** As a sidenote, I think the TU devs (community) should have a more custom arch-dev like interface (cli tools, flag out of date interface, etc), and not really be included in the aur at all. Alot of the TU functionality was somewhat 'bolted on', and wasn't ever really a good fit for the real core use cases of the AUR.
Suggestions would be very welcome. I'm not a TU or dev and therefore am not familiar with the flow. I guess I'd have to setup AUR and possibly the main site locally and see what's available. I still won't get any insight into what's required or what would be better though.
participants (2)
-
eliott
-
Sebastian