[pacman-dev] pacman cold caches performance, too much stat()ing

[Dimitrios Apostolou]

On Fri, 11 Dec 2009, Nagy Gabor wrote:
> -Su is even worse: We have to read all desc files in sync (to get
> %REPLACES%)

I changed pacman DB using the following script, and applied the attached 
script to pacman so that "pacman -Su" is possible only by reading 
"depends" files, not "desc".

for d in $PACDB/sync/*/* $PACDB/local/*
do
 	sed -ne '/%REPLACES%/,/^$/p' $d/desc >> $d/depends
 	sed -i -e '/%REPLACES%/,/^$/d' $d/desc
done


The timings I got for the command "pacman -Sup -b $PACDB" are the 
following.

Before:
2m15s (cold caches)
0.98s (hot caches)

After:
2m11s (cold caches)
0.78s (hot caches)

Unfortunately it didn't provide such a big improvement as I expected. I 
can now justify this because desc files in sync/ were being read *instead* 
of depends, not in addition to them. So I _replaced_ the read("*/desc") 
with read("*/depends"), but disk seeks are still happening, even for these 
smaller files.

I believe big differences will show when people have many upgrades 
pending, in which case the already cached depends will come handy. 
Unfortunately I only had 6 upgrades pending.

But I think it's vital that not all descriptions and stuff are read during 
a common operation like -Su, and with the provided script (run only for 
local, since sync should be upgraded on the server) the transition should 
be smooth for end-users. What's your opinion?


Dimitris
-------------- next part --------------
diff --git a/lib/libalpm/be_files.c b/lib/libalpm/be_files.c
index d09d72a..a6087bd 100644
--- a/lib/libalpm/be_files.c
+++ b/lib/libalpm/be_files.c
@@ -460,12 +460,6 @@ int _alpm_db_read(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq)
 					goto error;
 				}
 				STRDUP(info->md5sum, _alpm_strtrim(line), goto error);
-			} else if(strcmp(line, "%REPLACES%") == 0) {
-				while(fgets(line, sline, fp) && strlen(_alpm_strtrim(line))) {
-					char *linedup;
-					STRDUP(linedup, _alpm_strtrim(line), goto error);
-					info->replaces = alpm_list_add(info->replaces, linedup);
-				}
 			} else if(strcmp(line, "%FORCE%") == 0) {
 				info->force = 1;
 			}
@@ -528,6 +522,12 @@ int _alpm_db_read(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq)
 					STRDUP(linedup, _alpm_strtrim(line), goto error);
 					info->conflicts = alpm_list_add(info->conflicts, linedup);
 				}
+			} else if(strcmp(line, "%REPLACES%") == 0) {
+				while(fgets(line, sline, fp) && strlen(_alpm_strtrim(line))) {
+					char *linedup;
+					STRDUP(linedup, _alpm_strtrim(line), goto error);
+					info->replaces = alpm_list_add(info->replaces, linedup);
+				}
 			} else if(strcmp(line, "%PROVIDES%") == 0) {
 				while(fgets(line, sline, fp) && strlen(_alpm_strtrim(line))) {
 					char *linedup;
diff --git a/lib/libalpm/package.c b/lib/libalpm/package.c
index 83a2fb8..387cb1b 100644
--- a/lib/libalpm/package.c
+++ b/lib/libalpm/package.c
@@ -387,8 +387,8 @@ alpm_list_t SYMEXPORT *alpm_pkg_get_replaces(pmpkg_t *pkg)
 	ASSERT(handle != NULL, return(NULL));
 	ASSERT(pkg != NULL, return(NULL));
 
-	if(pkg->origin == PKG_FROM_CACHE && !(pkg->infolevel & INFRQ_DESC)) {
-		_alpm_db_read(pkg->origin_data.db, pkg, INFRQ_DESC);
+	if(pkg->origin == PKG_FROM_CACHE && !(pkg->infolevel & INFRQ_DEPENDS)) {
+		_alpm_db_read(pkg->origin_data.db, pkg, INFRQ_DEPENDS);
 	}
 	return pkg->replaces;
 }