[pacman-dev] download / database problems

[Dan McGee]

On Fri, Sep 25, 2009 at 9:39 AM, Xavier <shiningxc at gmail.com> wrote:
> On Thu, Sep 24, 2009 at 8:45 AM, Xavier <shiningxc at gmail.com> wrote:
>> On Thu, Sep 24, 2009 at 1:15 AM, Dan McGee <dpmcgee at gmail.com> wrote:
>>> On Wed, Sep 23, 2009 at 6:04 PM, Aaron Griffin <aaronmgriffin at gmail.com> wrote:
>>>> On Wed, Sep 23, 2009 at 5:25 PM, Xavier <shiningxc at gmail.com> wrote:
>>>>> Today I got very strange results downloading packages with pacman.
>>>>> For example :
>>>>>  gnome-common-2.28.0...     8,9K  112,2K/s 00:00:00
>>>>> [#####################] 21167%
>>>>>
>>>>>> ls -lh /var/cache/pacman/pkg/gnome-common-2.28.0-1-any.pkg.tar.gz
>>>>> -rw-r--r-- 1 root root 8,9K sept. 24 00:17
>>>>> /var/cache/pacman/pkg/gnome-common-2.28.0-1-any.pkg.tar.gz
>>>>>
>>>>> /var/lib/pacman/sync/gnome-unstable/gnome-common-2.28.0-1/desc:%CSIZE%
>>>>> /var/lib/pacman/sync/gnome-unstable/gnome-common-2.28.0-1/desc-43
>>>>>
>>>>> 43 bytes instead of 8900 ? wtf ?
>>>>
>>>> Hmmm repo-add problem with symlinks? The any packages are symlinks...
>>>> maybe it needs fixing for how it determines filesize
>>>
>>> Ding! We have a winner, the symlink size itself is 43:
>>>
>>> $ ll /srv/ftp/gnome-unstable/os/i686/gnome-common-2.28.0-1-any.pkg.tar.gz
>>> lrwxrwxrwx 1 jgc ftp-extra 43 2009-09-21 18:35
>>> /srv/ftp/gnome-unstable/os/i686/gnome-common-2.28.0-1-any.pkg.tar.gz
>>> -> ../any/gnome-common-2.28.0-1-any.pkg.tar.gz
>>>
>>> Too bad you point this out the day after we release, hah. Looks like
>>> we need to add the -L option to the stat call in sizecmd, but let's
>>> talk about that on the pacman ML.
>>>
>>> For now I added -L to the repo-add script on gerolde; that will be
>>> blown away the next time we update though.
>>>
>>> -Dan
>>>
>>>
>>
>> Well, I was wondering why I only found this problem yesterday.
>> It probably has nothing to do with that csize/any problem after all.
>> These database errors are probably around for a while, and never
>> affected us, because we use libfetch url_stat.size for download
>> progress.
>> So yesterday, it was more likely libfetch/network/mirrors problem I
>> was having, and for some reasons, it could not get the total size or
>> something... I don't know.
>>
>
> Damn it, I have been wrong again ! I am so stupid !
>
> So in the normal/default case, we don't use CSIZE, so this does not
> affect download progress bar.
>
> But when we use TotalDownload, we do use CSIZE, because we need to
> know all the file size in advance !
> So using for example gnome-unstable repos with several any packages,
> this causes the following funny output :
> http://bbs.archlinux.org/viewtopic.php?pid=625617#p625617
> That's actually how I found the problem too...
>
> Maybe we should attempt to make the code safer. For instance, catching
> the case where current size > total size.
> And maybe in this case, we could fallback to TotalDownload disabled.

So "safer" meaning doesn't display crazy percentages? Other than that,
there is no corruption issue or anything, right?

-Dan