[aur-general] AUR package metadata dump

brent s. bts at square-r00t.net
Thu Nov 15 19:50:48 UTC 2018

On 11/15/18 14:26, Eli Schwartz via aur-general wrote:
> On 11/15/18 1:25 PM, Dmitry Marakasov wrote:
>> Hi!
>> While fetching data from API, Repology does a 1 second pause between
>> requests to not create excess load on the server, but there are still
>> frequent 429 errors. I've tried 2 second delays, but the 429s are still
>> there, and fetch time increases dramatically as we have to do more than
>> 500 requests. Probably API is loaded by other clients as well.
> Our rate limit is 4000 per 24 hours. One-second pauses aren't taken into
> account, and our initial motivation to add rate limiting was to ban
> users who were using 5-second delays...

don't forget about the URI max length, too. staggering into requests of
100 pkgs would work fine, but worth noting the max length is 4443 bytes


brent saner
GPG info: https://square-r00t.net/gpg-info

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 899 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/aur-general/attachments/20181115/67a9088f/attachment.asc>

More information about the aur-general mailing list