[pacman-dev] pacman -Qs first-run performance

[Dan McGee]

On Thu, Sep 1, 2011 at 12:33 PM, Philipp <hollunder at lavabit.com> wrote:
> Hi there,
> pacman works very well for me, with one exception: searching for
> installed packages. For me a -Ss takes about 5 seconds, a -Qs 25
> seconds. I also noticed that this seems to be true for the first search
> run only, the following ones are less than a second, no matter whether
> it's a repo or local search. But if you want to just quickly find out a
> thing, 25 seconds or so is a long time.

Of course subsequent runs are faster- the page cache comes into play.

25 seconds is beyond slow. I have some results from my slowest machine
(HDDs are running using PIO4 right now, not even DMA), and I can't top
5 seconds on -Qs or 10 seconds on -Ss *with* caches dropped [1]. The
first timing in each command is uncached; the second is then the
cached time.

> Anyway, I had a very brief look at the code and am far from
> understanding it, but I think libalpm/db.c handles the search, the
> package names, descriptions, etc. are stored in a linked list of
> structs, the whole thing is cached in memory only and regex wizardry
> is used for the search. If that's true, the bottleneck I experience is
> the caching.
>
> 5 seconds for -Ss is acceptable for me, but I wonder whether there's a
> reasonably easy way to improve the 25 seconds of -Qs.
No, the bottleneck is your slow hard drive, your lack of RAM, and/or
your slow CPU. Notice that on a cached search, my times don't exceed
0.3 seconds for any operation.

> My knowledge of pacman internals is non-existant and my C skills are
> minimal, so I don't think I can be a lot of help with this one, but
> maybe there's something else I can be of help with instead.
If you could tell us more about your setup, that would be very
helpful. CPU info from /proc/cpuinfo, memory info from /proc/meminfo
and `free -m` output, `hdparm -Tt` numbers, etc.

We've massively sped up pacman database reading for both hot and cold
cases over what it was a year or so ago, so most of us consider this a
"solved" problem by now. None of us see anywhere near the times you
are seeing for operations, so anything you can provide us would be
helpful.

The output from these two commands might also be useful:

echo 3 > /proc/sys/vm/drop_caches; /usr/bin/time pacman -Qs foobarbaz;
/usr/bin/time pacman -Qs foobarbaz
echo 3 > /proc/sys/vm/drop_caches; /usr/bin/time pacman -Ss foobarbaz;
/usr/bin/time pacman -Ss foobarbaz

-Dan

[1]
[root at dublin ~]# echo 3 > /proc/sys/vm/drop_caches; time pacman -Ss
foobarbaz; time pacman -Ss foobarbaz

real    0m5.533s
user    0m0.360s
sys     0m0.553s

real    0m0.349s
user    0m0.343s
sys     0m0.007s

[root at dublin ~]# echo 3 > /proc/sys/vm/drop_caches; time pacman -Qs
foobarbaz; time pacman -Qs foobarbaz

real    0m10.030s
user    0m0.050s
sys     0m0.290s

real    0m0.063s
user    0m0.040s
sys     0m0.020s