On Thu, Dec 20, 2018 at 9:45 PM Jelle van der Waa wrote:

On 12/21/18 at 03:40am, Macca wrote:

...

On 21 Dec. 2018, 3:26 am, at 3:26 am, Drew DeVault via aur-general <

aur-general@archlinux.org> wrote:

...

...

Following up from IRC. Hello!

I run a public build service which has Arch Linux support and uses yay to install AUR packages when requested. I recently had some users running up against the AUR rate limit.

The current rate limit window (1 day) seems a bit strict. Would it be possible to apply the rate limit hourly? Today it's 4000 requests per day, but ~166 requests per hour would be a similar load on the server but a bit more forgiving, users would only have to wait an hour before attempting their builds again.

The issue here seems to be that yay queries for provides in the AUR. Which causes excsessive requests to be made. [1]

[1] https://builds.sr.ht/~delthas/job/17157

That should probably be fixed as well, but I agree with making the rate limit window 1 hour, at most. A 24 hour API restriction on the AUR API is really nasty imo.

On 21/12/18 5:21 am, Florian Pritz via aur-general wrote:

...

That should probably be fixed as well, but I agree with making the rate limit window 1 hour, at most. A 24 hour API restriction on the AUR API is really nasty imo. So is running cower -u in conky every 5 seconds and not even knowing

On Thu, Dec 20, 2018 at 11:02:31PM +0200, Jerome Leclanche wrote: that this may send 50 requests per seconds on average because you have so many packages installed. The limit is there to show people that there is something wrong with their API usage and if the window is too small they'll never notice.

We have resolved this particular issue for now though via a temporary reset of the limit. For the future I think the way to go is to implement configurable limits for certain subnets so that we can raise them to a suitable value. If someone wants to write a patch for this that would be very welcome.

My initial idea is that a simple db table with the limits for each subnet should be enough I think. We don't (yet) need a webui for maintaining it. The tricky part is probably figuring out which limit shall apply to which IP. Since the AUR supports mysql and sqlite, any code has to work with both.

If anyone wants to implement this, feel free to talk to me via mail or on IRC.

Florian

Is there anything wrong with creating a pertner file to the packages file that has the pkgname, provides, description and git address in it. Then encourage it to be downloaded - say at an interval of 12 hours, then use this file to search and encourage the use of the git address rather than a direct download of the pkgbuild. They can then cache these pkgbuilds directories in a build tree. The aur helper packages should then be encouraged to use this method and thus take the load off the system. Macca

On 23/12/18 2:40 am, Florian Pritz via aur-general wrote:

On Fri, Dec 21, 2018 at 04:41:28PM +0800, hagar wrote:

...

Is there anything wrong with creating a pertner file to the packages file that has the pkgname, provides, description and git address in it. Not sure what 'pertner' is supposed to mean, but I'd say sure, we could offer a database dump. It's just that someone would have to check which information people need and write a patch for that.

It's probably fine to dump all the info that you'd normally get from and RPC search request into a file. A really quick calculation tells me that this file would be around 20-30MiB (RPC search for 'perl', ~900KiB response with nearly 2k packages, 900*25 = 23MiB).

If someone wants to work on a patch for that, please feel free. If you want, I can also provide some feedback on the patch in addition to what you get on aur-dev. Also please talk to Lukas to check if he is ok with the general idea of this before you spend too much time on it.

Florian

Typo - I meant partner file :} It should only have limited info. pkgname, provides, description. and would it help if it was in the git repository, so you don't download the full thing? This was just an idea that I put out for discussion. If someone wanted more info would it hurt them to download the pkgbuild git? Thanks Macca

On 23/12/18 7:13 am, Eli Schwartz via aur-general wrote:

On 12/22/18 5:40 PM, hagar wrote:

...

On 23/12/18 2:40 am, Florian Pritz via aur-general wrote:

...

On Fri, Dec 21, 2018 at 04:41:28PM +0800, hagar wrote:

...

Is there anything wrong with creating a pertner file to the packages file that has the pkgname, provides, description and git address in it. Not sure what 'pertner' is supposed to mean, but I'd say sure, we could offer a database dump. It's just that someone would have to check which information people need and write a patch for that.

It's probably fine to dump all the info that you'd normally get from and RPC search request into a file. A really quick calculation tells me that this file would be around 20-30MiB (RPC search for 'perl', ~900KiB response with nearly 2k packages, 900*25 = 23MiB).

If someone wants to work on a patch for that, please feel free. If you want, I can also provide some feedback on the patch in addition to what you get on aur-dev. Also please talk to Lukas to check if he is ok with the general idea of this before you spend too much time on it.

Florian Typo - I meant partner file :}

It should only have limited info.

pkgname, provides, description. And a lot of people will probably want to get the dependency info too, at a minimum. :)

...

and would it help if it was in the git repository, so you don't download the full thing? What git repository? The per-package git repository? This is already available as the .SRCINFO file.

...

This was just an idea that I put out for discussion.

If someone wanted more info would it hurt them to download the pkgbuild git? Only if they wanted to download 50K git repositories in order to search all of them for packages matching certain heuristics. And most new AUR helpers these days have been attempting to pretend that .SRCINFO does not exist, then trying to exclusively use the RPC instead, because apparently .SRCINFO is "slow".

I meant create its own ie - https://aur.archlinux.org/packagedesc.git I personally think that some "helpers" are trying too hard. I only have a need to search for "provides" and "description", anything else is just icing.