On 16.11.18 - 18:27, Florian Pritz via aur-general wrote:
My idea is to either generate the results on demand or cache them in the code. If cached in the code, there would be no database load. It would just pass through the code so we can perform rate limiting. Granted, if we can implemented the rate limit in nginx (see below) that would be essentially the same and fine too. Then we/you could indeed just dump it to a file and serve that.
I've just been discussing an idea with Florian, which might provide a reasonable way for both sides: The clients could send a timestamp to the API, that implies some sort of "give me all updates since *that*". The update timestamps for the packages are already tracked in the database anyway, putting an index on that column would make requesting various ranges quite efficient. When there were no changes since the client-supplied timestamp, the API could respond with HTTP 304 "Not Modified" (possibly without a body even) which would provide a suitable meaning and a very tiny response. We could actually think about not logging those 304 responses then, dunno what the general opinion on that is. If a new client wants to get started and build his own archive, it could supply the timestamp "since=0" (e.g. talking about unix timestamps here), which would simply result in a response with all packages. To prevent abuse of such (very large) deltas, we could implement some sort of shared rate-limit, like Florian mentioned. A first idea would be to use a shared rate-limit for all requests with a timestamp older than 48 hours for example. We could allow something like 200 of such requests per hour and if that limit was exceeded, the API would reply with maybe HTTP 400 "Bad Request" or HTTP 412 "Precondition Failed", along with a "Retry-After: [0-9]+" header to tell the client when to try again. Anywho, I just wanted to put this out there and gather some thoughts, feedback and opinions on this. Cheers, Thore -- Thore Bödecker GPG ID: 0xD622431AF8DB80F3 GPG FP: 0F96 559D 3556 24FC 2226 A864 D622 431A F8DB 80F3