[pacman-dev] pacman cold caches performance, too much stat()ing

[Dan McGee]

On Sat, Dec 12, 2009 at 5:46 AM, Allan McRae <allan at archlinux.org> wrote:
> Dimitrios Apostolou wrote:
>>
>> On Fri, 11 Dec 2009, Nagy Gabor wrote:
>>>
>>> depends files are read in order to ensure that the upgraded package
>>> won't break any "old" dependencies.
>>>
>>> Example: local foo requires bar=2.0 (which is installed)
>>> Then "pacman -S bar" is not allowed (if bar in sync has different
>>> version).
>>
>> I see now, thanks! So if we somehow had an %RDEPENDS% field
>> (reverse-dependencies) for every package in local that would not be
>> necessary. I will see if this is doable during every install.
>
> pacman used to do such a thing, but from my understanding it caused more
> issues than it solved so it was removed.

This was the former %REQUIREDBY% field
(http://code.toofishes.net/cgit/dan/pacman.git/commit/?id=7219326dd4d01d7e49b8a40746f5495c1c329c9c).
It did end up causing more problems than it solved, as they never
seemed to be stored quite right. Instead, we switched to computing
them on the fly. This is the reason for the delay on -Qi in the cold
cache case, for instance.

I think most of this thread is the wrong approach to the problem.
Rather than try to meld the DB to fit pacman, we should just swap out
DBs so it doesn't have these bad worst-case conditions.

$ du -sh --apparent-size local/ sync/*
15M	local/
8.4M	sync/community
31K	sync/community-testing
812K	sync/core
11M	sync/extra
302K	sync/testing

With those numbers in mind, we're talking about ~30 to ~35 MB of raw
text data here. That is not a lot of data; most hard drives have at
least ~30 MB/sec performance so this whole mess could be read in under
a second if it was stored in a single file. So that would be one way
of thinking about the issue differently.

Other ways of course have come up many times on this list. BDB,
Sqlite, reading the compressed DB directly, etc.

-Dan