[aur-general] AUR rate limiting
Following up from IRC. Hello! I run a public build service which has Arch Linux support and uses yay to install AUR packages when requested. I recently had some users running up against the AUR rate limit. The current rate limit window (1 day) seems a bit strict. Would it be possible to apply the rate limit hourly? Today it's 4000 requests per day, but ~166 requests per hour would be a similar load on the server but a bit more forgiving, users would only have to wait an hour before attempting their builds again. The long term solution is probably to mirror the AUR or adjust the approach to try and minimize API requests (perhaps by cloning the repos and sorting out dependencies by sourcing the PKGBUILDs.... would be a bit involved). Any thoughts or advice? -- Drew DeVault
I have personally run afoul of the rate limit. I have gotten around it by downloading the package list file. And then by using the got versions of the pkgbuilds. Once you have the pkgbuild tree you only have to update it once and build m any times. You shouldn't have to download it. Some of these air tools realy need to be rewritten. Thanks Macca Sent from BlueMail On 21 Dec. 2018, 3:26 am, at 3:26 am, Drew DeVault via aur-general <aur-general@archlinux.org> wrote:
Following up from IRC. Hello!
I run a public build service which has Arch Linux support and uses yay to install AUR packages when requested. I recently had some users running up against the AUR rate limit.
The current rate limit window (1 day) seems a bit strict. Would it be possible to apply the rate limit hourly? Today it's 4000 requests per day, but ~166 requests per hour would be a similar load on the server but a bit more forgiving, users would only have to wait an hour before attempting their builds again.
The long term solution is probably to mirror the AUR or adjust the approach to try and minimize API requests (perhaps by cloning the repos and sorting out dependencies by sourcing the PKGBUILDs.... would be a bit involved).
Any thoughts or advice?
-- Drew DeVault
On 12/21/18 at 03:40am, Macca wrote:
On 21 Dec. 2018, 3:26 am, at 3:26 am, Drew DeVault via aur-general <aur-general@archlinux.org> wrote:
Following up from IRC. Hello!
I run a public build service which has Arch Linux support and uses yay to install AUR packages when requested. I recently had some users running up against the AUR rate limit.
The current rate limit window (1 day) seems a bit strict. Would it be possible to apply the rate limit hourly? Today it's 4000 requests per day, but ~166 requests per hour would be a similar load on the server but a bit more forgiving, users would only have to wait an hour before attempting their builds again.
The issue here seems to be that yay queries for provides in the AUR. Which causes excsessive requests to be made. [1] [1] https://builds.sr.ht/~delthas/job/17157
On Thu, Dec 20, 2018 at 9:45 PM Jelle van der Waa <jelle@vdwaa.nl> wrote:
On 12/21/18 at 03:40am, Macca wrote:
On 21 Dec. 2018, 3:26 am, at 3:26 am, Drew DeVault via aur-general <
aur-general@archlinux.org> wrote:
Following up from IRC. Hello!
I run a public build service which has Arch Linux support and uses yay to install AUR packages when requested. I recently had some users running up against the AUR rate limit.
The current rate limit window (1 day) seems a bit strict. Would it be possible to apply the rate limit hourly? Today it's 4000 requests per day, but ~166 requests per hour would be a similar load on the server but a bit more forgiving, users would only have to wait an hour before attempting their builds again.
The issue here seems to be that yay queries for provides in the AUR. Which causes excsessive requests to be made. [1]
That should probably be fixed as well, but I agree with making the rate limit window 1 hour, at most. A 24 hour API restriction on the AUR API is really nasty imo.
On Thu, Dec 20, 2018 at 11:02:31PM +0200, Jerome Leclanche <jerome@leclan.ch> wrote:
That should probably be fixed as well, but I agree with making the rate limit window 1 hour, at most. A 24 hour API restriction on the AUR API is really nasty imo.
So is running cower -u in conky every 5 seconds and not even knowing that this may send 50 requests per seconds on average because you have so many packages installed. The limit is there to show people that there is something wrong with their API usage and if the window is too small they'll never notice. We have resolved this particular issue for now though via a temporary reset of the limit. For the future I think the way to go is to implement configurable limits for certain subnets so that we can raise them to a suitable value. If someone wants to write a patch for this that would be very welcome. My initial idea is that a simple db table with the limits for each subnet should be enough I think. We don't (yet) need a webui for maintaining it. The tricky part is probably figuring out which limit shall apply to which IP. Since the AUR supports mysql and sqlite, any code has to work with both. If anyone wants to implement this, feel free to talk to me via mail or on IRC. Florian
On 21/12/18 5:21 am, Florian Pritz via aur-general wrote:
On Thu, Dec 20, 2018 at 11:02:31PM +0200, Jerome Leclanche <jerome@leclan.ch> wrote:
That should probably be fixed as well, but I agree with making the rate limit window 1 hour, at most. A 24 hour API restriction on the AUR API is really nasty imo. So is running cower -u in conky every 5 seconds and not even knowing that this may send 50 requests per seconds on average because you have so many packages installed. The limit is there to show people that there is something wrong with their API usage and if the window is too small they'll never notice.
We have resolved this particular issue for now though via a temporary reset of the limit. For the future I think the way to go is to implement configurable limits for certain subnets so that we can raise them to a suitable value. If someone wants to write a patch for this that would be very welcome.
My initial idea is that a simple db table with the limits for each subnet should be enough I think. We don't (yet) need a webui for maintaining it. The tricky part is probably figuring out which limit shall apply to which IP. Since the AUR supports mysql and sqlite, any code has to work with both.
If anyone wants to implement this, feel free to talk to me via mail or on IRC.
Florian
Is there anything wrong with creating a pertner file to the packages file that has the pkgname, provides, description and git address in it. Then encourage it to be downloaded - say at an interval of 12 hours, then use this file to search and encourage the use of the git address rather than a direct download of the pkgbuild. They can then cache these pkgbuilds directories in a build tree. The aur helper packages should then be encouraged to use this method and thus take the load off the system. Macca
On Fri, Dec 21, 2018 at 04:41:28PM +0800, hagar <hagar@iinet.net.au> wrote:
Is there anything wrong with creating a pertner file to the packages file that has the pkgname, provides, description and git address in it.
Not sure what 'pertner' is supposed to mean, but I'd say sure, we could offer a database dump. It's just that someone would have to check which information people need and write a patch for that. It's probably fine to dump all the info that you'd normally get from and RPC search request into a file. A really quick calculation tells me that this file would be around 20-30MiB (RPC search for 'perl', ~900KiB response with nearly 2k packages, 900*25 = 23MiB). If someone wants to work on a patch for that, please feel free. If you want, I can also provide some feedback on the patch in addition to what you get on aur-dev. Also please talk to Lukas to check if he is ok with the general idea of this before you spend too much time on it. Florian
On 23/12/18 2:40 am, Florian Pritz via aur-general wrote:
On Fri, Dec 21, 2018 at 04:41:28PM +0800, hagar <hagar@iinet.net.au> wrote:
Is there anything wrong with creating a pertner file to the packages file that has the pkgname, provides, description and git address in it. Not sure what 'pertner' is supposed to mean, but I'd say sure, we could offer a database dump. It's just that someone would have to check which information people need and write a patch for that.
It's probably fine to dump all the info that you'd normally get from and RPC search request into a file. A really quick calculation tells me that this file would be around 20-30MiB (RPC search for 'perl', ~900KiB response with nearly 2k packages, 900*25 = 23MiB).
If someone wants to work on a patch for that, please feel free. If you want, I can also provide some feedback on the patch in addition to what you get on aur-dev. Also please talk to Lukas to check if he is ok with the general idea of this before you spend too much time on it.
Florian
Typo - I meant partner file :} It should only have limited info. pkgname, provides, description. and would it help if it was in the git repository, so you don't download the full thing? This was just an idea that I put out for discussion. If someone wanted more info would it hurt them to download the pkgbuild git? Thanks Macca
On 12/22/18 5:40 PM, hagar wrote:
On 23/12/18 2:40 am, Florian Pritz via aur-general wrote:
On Fri, Dec 21, 2018 at 04:41:28PM +0800, hagar <hagar@iinet.net.au> wrote:
Is there anything wrong with creating a pertner file to the packages file that has the pkgname, provides, description and git address in it. Not sure what 'pertner' is supposed to mean, but I'd say sure, we could offer a database dump. It's just that someone would have to check which information people need and write a patch for that.
It's probably fine to dump all the info that you'd normally get from and RPC search request into a file. A really quick calculation tells me that this file would be around 20-30MiB (RPC search for 'perl', ~900KiB response with nearly 2k packages, 900*25 = 23MiB).
If someone wants to work on a patch for that, please feel free. If you want, I can also provide some feedback on the patch in addition to what you get on aur-dev. Also please talk to Lukas to check if he is ok with the general idea of this before you spend too much time on it.
Florian
Typo - I meant partner file :}
It should only have limited info.
pkgname, provides, description.
And a lot of people will probably want to get the dependency info too, at a minimum. :)
and would it help if it was in the git repository, so you don't download the full thing?
What git repository? The per-package git repository? This is already available as the .SRCINFO file.
This was just an idea that I put out for discussion.
If someone wanted more info would it hurt them to download the pkgbuild git?
Only if they wanted to download 50K git repositories in order to search all of them for packages matching certain heuristics. And most new AUR helpers these days have been attempting to pretend that .SRCINFO does not exist, then trying to exclusively use the RPC instead, because apparently .SRCINFO is "slow". -- Eli Schwartz Bug Wrangler and Trusted User
On 23/12/18 7:13 am, Eli Schwartz via aur-general wrote:
On 12/22/18 5:40 PM, hagar wrote:
On 23/12/18 2:40 am, Florian Pritz via aur-general wrote:
On Fri, Dec 21, 2018 at 04:41:28PM +0800, hagar <hagar@iinet.net.au> wrote:
Is there anything wrong with creating a pertner file to the packages file that has the pkgname, provides, description and git address in it. Not sure what 'pertner' is supposed to mean, but I'd say sure, we could offer a database dump. It's just that someone would have to check which information people need and write a patch for that.
It's probably fine to dump all the info that you'd normally get from and RPC search request into a file. A really quick calculation tells me that this file would be around 20-30MiB (RPC search for 'perl', ~900KiB response with nearly 2k packages, 900*25 = 23MiB).
If someone wants to work on a patch for that, please feel free. If you want, I can also provide some feedback on the patch in addition to what you get on aur-dev. Also please talk to Lukas to check if he is ok with the general idea of this before you spend too much time on it.
Florian Typo - I meant partner file :}
It should only have limited info.
pkgname, provides, description. And a lot of people will probably want to get the dependency info too, at a minimum. :)
and would it help if it was in the git repository, so you don't download the full thing? What git repository? The per-package git repository? This is already available as the .SRCINFO file.
This was just an idea that I put out for discussion.
If someone wanted more info would it hurt them to download the pkgbuild git? Only if they wanted to download 50K git repositories in order to search all of them for packages matching certain heuristics. And most new AUR helpers these days have been attempting to pretend that .SRCINFO does not exist, then trying to exclusively use the RPC instead, because apparently .SRCINFO is "slow".
I meant create its own ie - https://aur.archlinux.org/packagedesc.git I personally think that some "helpers" are trying too hard. I only have a need to search for "provides" and "description", anything else is just icing.
participants (7)
-
Drew DeVault
-
Eli Schwartz
-
Florian Pritz
-
hagar
-
Jelle van der Waa
-
Jerome Leclanche
-
Macca