[aur-dev] Making the AUR package list more useful

[Alad Wenter]

On 29.04.2016 13:29, Johannes Löthberg wrote:

> Hey,
>
> On 29/04, Lukas Fleischer wrote:
>> 1. The list is outdated. Right now, it is updated every two hours. I
>>   do not think there is a good reason for those long intervals.
>>   Reducing it to, say, ten minutes should be totally fine. Or maybe
>>   even trigger list generation whenever a package is created or deleted
>>   (which is clearly a lot more work, though). Thoughts?
>>
>
> Generating it more often sounds good to me, and triggering it on 
> create/delete does sound like a good thing to implement eventually, 
> but would probably want to have a task queue thing set up for that, 
> and have workers do the actual generation.
>
>
>> 2. Transferring the whole package name list is inefficient. Even if we
>>   use gzip compression here, the whole list is several hundreds of
>>   kilobytes large. We need to retransfer the full list, even if only a
>>   single package is added. Maybe we can do better than pacman here. My
>>   idea is to add zsync support to the lists such that only relevant
>>   parts are downloaded (for those who do not know: zsync is like rsync
>>   but it works via HTTP as well and does not require any special
>>   software on the server side). I did not experiment with how much
>>   bandwidth we can actually save using this yet. Maybe the block size
>>   needs to be adjusted. Are there any opinions or other suggestions on
>>   this topic?
>>
>
> Did some testing locally with zsync, and curling the file locally took 
> between 0.005 and 0.010 seconds.  A first zsync download takes between 
> 0.011 and 0.016 seconds.  A zsync with no changes takes between 0.004 
> and 0.012 seconds.
>
> It's a bit tricky to reliably test a zsync with a changed packages 
> file since I don't have multiple different ones saved down, and 
> modifying them myself will give different results from what the AUR 
> creates, so it'd be hard to get representative results.
>
> When I get a few versions of the file generated by the AUR I'll try 
> doing it from my server, and with multiple zsyncs, but I'm not sure if 
> it'll really matter much, since it takes just ~0.4 seconds to download 
> the file from the AUR in the first place.
>

As a note, differences between downloading a complete version of the 
package archive, and just the differences, should be mainly noticeable 
on poor internet connections (dial-up, 3G, etc.)