2006/10/26, Aaron Griffin :

What I'm saying is that you have the overhead of a full-scale database with little gain. Indexing by package names, yes that's great and all, but that doesn't help with the slowest-of-the-slow -Ss operation. -Ss searches package names AND descriptions, allowing for regex matching. Sqlite (and most DBs) do not support regex matching. Indexing the search by package name means little, because a -Ss "foo" still SHOULD match a package named "barfoo" and a package with "foo" in the description. This means a sequential search. Not only that, but because the DB will not support regex, that means that one must iterate over each and every entry, get the values at the C level, apply a regex pattern, and note if it is a match or not. The only speed you gain would be in the initial opening of the files. To me, this does not mean a DB backend is the solution. It may be better than the files backend, yes, but not the best, and outperforming the files backend is not hard... I was able to improve performance approximately 6 times by simply using the db.tar.gz files in place of disparate text files.

Yes, the main problem with files backend is that huge amount of files in /var/lib/pacman that leads to slow performance on many filesystems. gdbm/sqlite/whatever has a big "+" that everithing can be in one file. Plus, as I said, with SQLite you have the ability to easily use in-memory database + you have easy ACID transactions support. This will be faster than seeking through the /var/lib/pacman/ anyway. And there will be no need for pacman-optimize-like scripts for database backends. The overhead is not big. (A quote from sqlite.org: "Small code footprint: less than 250KiB fully configured or less than 150KiB with optional features omitted") Anyway nobody imposes use of some database backend as default and the only one. But having alternative is good. Users can test performance by themself and choose the solution that fits their needs in the best way. Otherwise, what's the point of the ALPM's ability to have different backends? ;-) About regexps - yes, SQLite doesn't support them, but adding this on top of it will be not slower than in files backend anyway. I don't see a problem here. I'm not "advertising" SQLite or other databases because "everybody use databases for everything". I just think that this will offer some new abilities for libalpm and will simplify (and speedup, yes - speedup) some things. That's IMHO, of course, but I think it worth a try gdbm and sqlite as alternatives to files backend. -- Roman Kyrylych (Роман Кирилич)

2006/10/27, Jürgen Hötzel :

...

Yes, the main problem with files backend is that huge amount of files in /var/lib/pacman that leads to slow performance on many filesystems. gdbm/sqlite/whatever has a big "+" that everithing can be in one file.

This is a big "-", because unix lovers prefer multiple text files. This makes it easy to use powerful text/file-processing tools like "sed", "awk", "shell" and "find" to build package-management related scripts.

There're nothing that don't allow to create some "API" of shell functions for doing most common operations that can be used by other shell scripts. And this way will be more backward compatible in case there will be some changes to storage format.

...

Plus, as I said, with SQLite you have the ability to easily use in-memory database + you have easy ACID transactions support. This will be faster than seeking through the /var/lib/pacman/ anyway. And there will be no need for pacman-optimize-like scripts for database backends. The overhead is not big. (A quote from sqlite.org: "Small code footprint: less than 250KiB fully configured or less than 150KiB with optional features omitted")

You will gain little performance (some seconds on a slow system?) at the expense of complexity.

Why not make a benchmark? ;-) Really, there are some users that complain about pacman slowness, and it's not about really slow operations like -Ss, but about really trivial operations, that shouldn't be so slow. That's because files backend is highly dependent on the type of underlying filesystem. Database backend may make these users happy,because it's not dependent on filesystem type. Personally, I don't think there's a problem _at_all_! Anyway files backend will be default in pacman3, and anyway I doubt it will be replaced soon (if ever). Why not just have an _alternative_? Keep in mind that ALPM can and will be used by other distros. Except Arch Linux and Frugalware there're Underground Linux (mostly Arch Linux Install CD + KDE and other desktop software) and Rubix Linux (formally, simply Rubix, anyway it's dead now), but I'm sure there will be some more in future (though I don't think more than 300 distros is good, even DistroWatch author thinks the same). Other distros may choose another backend. Of course, they can implement them by themselves, but anyway there was a reason why multiple backend fearure was included into ALPM. So why not to use it? Especially when it does not impose anything. Having alternatives is good, IMHO. -- Roman Kyrylych (Роман Кирилич)

2006/10/27, Aaron Griffin :

No no no.... you're missing the point. Because reading individual files is slow, we need a new backend. This does not imply a database backend is a good idea. It implies a single-file solution is ideal, yes. What I've been trying to show was that, considering the slow-down is from reading hundreds of files, that does not imply a database. ANY single-file solution would work.

Yes, I know this. You just get words out of my mouth! :) But single file will need some structure anyway. So, either we'll have our own format, or use something like gdbm or even simpler, or something more complex like sqlite.

That is why I was on about string matching and things of that nature - because if we take "single file solution A" and compare it to "DB solution A", there are negligable differences, and the DB one adds much more complexity and security flaws (i.e. embedded SQL statement strings).

OK, so, what do you propose? Use a simple tar.gz and process it in memory, or what? Still, I think sqlite or gdbm can be tried, but that's will be not in the near future anyway. There're more important things now anyway. :-) -- Roman Kyrylych (Роман Кирилич)

On Thu, 26 Oct 2006 17:11:05 -0500 "Aaron Griffin" wrote:

No no no.... you're missing the point. Because reading individual files is slow, we need a new backend. This does not imply a database backend is a good idea. It implies a single-file solution is ideal, yes. What I've been trying to show was that, considering the slow-down is from reading hundreds of files, that does not imply a database. ANY single-file solution would work.

Heh, well almost any. See pacman1 for an implementation that did not work. Well, it worked, but it was painfully slow -- sequential searching through a massive text file to find the package in question. - J