[aur-general] AUR package metadata dump

[Dmitry Marakasov]

Hi!

I'm maintainer of Repology.org, a service which monitors, aggregates
and compares package vesion accross 200+ package repositories with
a purpose of simplifying package maintainers work by discovering
new versions faster, improving collaboration between maintainers
and giving software authhors a complete overview of how well their
projects are packaged.

Repology does obviously support AUR, however there were some problems
with retrieving information on AUR packages and I think this could
be improved.

The way Repology currently fetches AUR package data is as follows:
- fetch https://aur.archlinux.org/packages.gz
- split packages into 100 item packs
- fetch JSON data for packages in each pack from https://aur.archlinux.org/rpc/?v=5&type=info&arg[]=<packages>

While fetching data from API, Repology does a 1 second pause between
requests to not create excess load on the server, but there are still
frequent 429 errors. I've tried 2 second delays, but the 429s are still
there, and fetch time increases dramatically as we have to do more than
500 requests. Probably API is loaded by other clients as well.

I suggest to implement a regularly updated JSON dump of information
on all packages and make it available for the site, like packages.gz is.
The content should be similar to what https://aur.archlinux.org/rpc/?v=5&type=info
would return for all packages at once.

This will eliminate the need to access the API and generate load
on it, simplify and speed up fetching dramatically for both Repology
and possible other clients.

Additionally, I'd like to suggest to add information on distfiles to the
dump (and probably an API as well for consistency). For instance,
Repology checks availability for all (homepage and download) links
it retreives from package repositories and reports broken ones so
the packages could be fixed.

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amdmi3 at amdmi3.ru  ..:              https://github.com/AMDmi3