[pacman-dev] download / database problems

[Xavier]

On Thu, Sep 24, 2009 at 8:45 AM, Xavier <shiningxc at gmail.com> wrote:
> On Thu, Sep 24, 2009 at 1:15 AM, Dan McGee <dpmcgee at gmail.com> wrote:
>> On Wed, Sep 23, 2009 at 6:04 PM, Aaron Griffin <aaronmgriffin at gmail.com> wrote:
>>> On Wed, Sep 23, 2009 at 5:25 PM, Xavier <shiningxc at gmail.com> wrote:
>>>> Today I got very strange results downloading packages with pacman.
>>>> For example :
>>>>  gnome-common-2.28.0...     8,9K  112,2K/s 00:00:00
>>>> [#####################] 21167%
>>>>
>>>>> ls -lh /var/cache/pacman/pkg/gnome-common-2.28.0-1-any.pkg.tar.gz
>>>> -rw-r--r-- 1 root root 8,9K sept. 24 00:17
>>>> /var/cache/pacman/pkg/gnome-common-2.28.0-1-any.pkg.tar.gz
>>>>
>>>> /var/lib/pacman/sync/gnome-unstable/gnome-common-2.28.0-1/desc:%CSIZE%
>>>> /var/lib/pacman/sync/gnome-unstable/gnome-common-2.28.0-1/desc-43
>>>>
>>>> 43 bytes instead of 8900 ? wtf ?
>>>
>>> Hmmm repo-add problem with symlinks? The any packages are symlinks...
>>> maybe it needs fixing for how it determines filesize
>>
>> Ding! We have a winner, the symlink size itself is 43:
>>
>> $ ll /srv/ftp/gnome-unstable/os/i686/gnome-common-2.28.0-1-any.pkg.tar.gz
>> lrwxrwxrwx 1 jgc ftp-extra 43 2009-09-21 18:35
>> /srv/ftp/gnome-unstable/os/i686/gnome-common-2.28.0-1-any.pkg.tar.gz
>> -> ../any/gnome-common-2.28.0-1-any.pkg.tar.gz
>>
>> Too bad you point this out the day after we release, hah. Looks like
>> we need to add the -L option to the stat call in sizecmd, but let's
>> talk about that on the pacman ML.
>>
>> For now I added -L to the repo-add script on gerolde; that will be
>> blown away the next time we update though.
>>
>> -Dan
>>
>>
>
> Well, I was wondering why I only found this problem yesterday.
> It probably has nothing to do with that csize/any problem after all.
> These database errors are probably around for a while, and never
> affected us, because we use libfetch url_stat.size for download
> progress.
> So yesterday, it was more likely libfetch/network/mirrors problem I
> was having, and for some reasons, it could not get the total size or
> something... I don't know.
>

Damn it, I have been wrong again ! I am so stupid !

So in the normal/default case, we don't use CSIZE, so this does not
affect download progress bar.

But when we use TotalDownload, we do use CSIZE, because we need to
know all the file size in advance !
So using for example gnome-unstable repos with several any packages,
this causes the following funny output :
http://bbs.archlinux.org/viewtopic.php?pid=625617#p625617
That's actually how I found the problem too...

Maybe we should attempt to make the code safer. For instance, catching
the case where current size > total size.
And maybe in this case, we could fallback to TotalDownload disabled.