[aur-general] AUR package metadata dump

[Thore Bödecker]

On 16.11.18 - 18:27, Florian Pritz via aur-general wrote:
> My idea is to either generate the results on demand or cache them in the
> code. If cached in the code, there would be no database load. It would
> just pass through the code so we can perform rate limiting. Granted, if
> we can implemented the rate limit in nginx (see below) that would be
> essentially the same and fine too. Then we/you could indeed just dump it
> to a file and serve that.

I've just been discussing an idea with Florian, which might provide a
reasonable way for both sides:

The clients could send a timestamp to the API, that implies some sort
of "give me all updates since *that*".
The update timestamps for the packages are already tracked in the
database anyway, putting an index on that column would make requesting
various ranges quite efficient.

When there were no changes since the client-supplied timestamp, the
API could respond with HTTP 304 "Not Modified" (possibly without a
body even) which would provide a suitable meaning and a very tiny
response.
We could actually think about not logging those 304 responses then,
dunno what the general opinion on that is.

If a new client wants to get started and build his own archive, it
could supply the timestamp "since=0" (e.g. talking about unix
timestamps here), which would simply result in a response with all
packages.
To prevent abuse of such (very large) deltas, we could implement some
sort of shared rate-limit, like Florian mentioned.
A first idea would be to use a shared rate-limit for all requests with
a timestamp older than 48 hours for example. We could allow something
like 200 of such requests per hour and if that limit was exceeded, the
API would reply with maybe HTTP 400 "Bad Request" or HTTP 412
"Precondition Failed", along with a "Retry-After: [0-9]+" header to
tell the client when to try again.


Anywho, I just wanted to put this out there and gather some thoughts, feedback
and opinions on this.


Cheers,
Thore

-- 
Thore Bödecker

GPG ID: 0xD622431AF8DB80F3
GPG FP: 0F96 559D 3556 24FC 2226  A864 D622 431A F8DB 80F3
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.archlinux.org/pipermail/aur-general/attachments/20181116/3f17cdb1/attachment-0001.asc>