[pacman-dev] Weird bug in sync/upgrade behavior
dmcgee@dublin ~ $ pacSyu :: Synchronizing package databases... pacman-git 0.5K 3.1M/s 00:00:00 [---------------------] 100% testing 15.1K 130.1K/s 00:00:00 [---------------------] 100% core is up to date extra 312.3K 214.5K/s 00:00:01 [---------------------] 100% community 335.9K 148.4K/s 00:00:02 [---------------------] 100% unstable 4.7K 92.6K/s 00:00:00 [---------------------] 100% :: Starting full system upgrade... warning: bzip2: local (1.0.5-2) is newer than core (1.0.4-3) warning: kernel26: local (2.6.25-1) is newer than core (2.6.24.4-1) warning: libldap: local (2.3.40-1) is newer than core (2.3.39-2) warning: libtool: local (2.2.4-1) is newer than core (2.2-2) warning: licenses: local (2.4-1) is newer than core (2.3-1) warning: links: local (2.1pre35-1) is newer than core (2.1pre33-1) warning: ntfs-3g: local (1.2412-1) is newer than core (1.2310-1) warning: openssh: local (5.0p1-1) is newer than core (4.7p1-6) warning: pcre: local (7.7-1) is newer than core (7.6-3) warning: sudo: local (1.6.9p15-1) is newer than core (1.6.9p12-1) local database is up to date dmcgee@dublin ~ $ pacSyu :: Synchronizing package databases... pacman-git is up to date testing is up to date core is up to date extra is up to date community is up to date unstable is up to date :: Starting full system upgrade... warning: pacman-git: local (20080511-1) is newer than pacman-git (20080427-1) local database is up to date Not sure when this got fuggered up (although It was probably me), but as you can see, we have a problem above. For some reason, when all databases have been updated except for core, it prefers packages in core over those in testing? This could be a super-old bug, thinking about it- We have an alpm_list_t of databases that are stored in conf file order, and since all the other ones got updated (which means removed from the list and readded?), core ends up getting bumped to the top and testing ends up below it, meaning the core packages are preferred. Note that when I first ran this, a libtool upgrade was available, and it did not prompt me for that. However, the second run did prompt me. Can anyone else try to reproduce this? Try deleting all .lastupdate files except the one for core, if you have testing enabled, and seeing what happens on the first and second runs of -Syu. This could be a prime case for git-bisect if we need to track this down. I'm currently running a pacman-git I built yesterday (I think). -Dan
Dan McGee wrote:
dmcgee@dublin ~ $ pacSyu :: Synchronizing package databases... pacman-git 0.5K 3.1M/s 00:00:00 [---------------------] 100% testing 15.1K 130.1K/s 00:00:00 [---------------------] 100% core is up to date extra 312.3K 214.5K/s 00:00:01 [---------------------] 100% community 335.9K 148.4K/s 00:00:02 [---------------------] 100% unstable 4.7K 92.6K/s 00:00:00 [---------------------] 100% :: Starting full system upgrade... warning: bzip2: local (1.0.5-2) is newer than core (1.0.4-3) warning: kernel26: local (2.6.25-1) is newer than core (2.6.24.4-1) warning: libldap: local (2.3.40-1) is newer than core (2.3.39-2) warning: libtool: local (2.2.4-1) is newer than core (2.2-2) warning: licenses: local (2.4-1) is newer than core (2.3-1) warning: links: local (2.1pre35-1) is newer than core (2.1pre33-1) warning: ntfs-3g: local (1.2412-1) is newer than core (1.2310-1) warning: openssh: local (5.0p1-1) is newer than core (4.7p1-6) warning: pcre: local (7.7-1) is newer than core (7.6-3) warning: sudo: local (1.6.9p15-1) is newer than core (1.6.9p12-1) local database is up to date
dmcgee@dublin ~ $ pacSyu :: Synchronizing package databases... pacman-git is up to date testing is up to date core is up to date extra is up to date community is up to date unstable is up to date :: Starting full system upgrade... warning: pacman-git: local (20080511-1) is newer than pacman-git (20080427-1) local database is up to date
Not sure when this got fuggered up (although It was probably me), but as you can see, we have a problem above. For some reason, when all databases have been updated except for core, it prefers packages in core over those in testing? This could be a super-old bug, thinking about it- We have an alpm_list_t of databases that are stored in conf file order, and since all the other ones got updated (which means removed from the list and readded?), core ends up getting bumped to the top and testing ends up below it, meaning the core packages are preferred. Note that when I first ran this, a libtool upgrade was available, and it did not prompt me for that. However, the second run did prompt me.
Can anyone else try to reproduce this? Try deleting all .lastupdate files except the one for core, if you have testing enabled, and seeing what happens on the first and second runs of -Syu.
This could be a prime case for git-bisect if we need to track this down. I'm currently running a pacman-git I built yesterday (I think).
-Dan
I can not replicate this at all... pacman-3.1.4 :: Synchronizing package databases... testing 15.1K 58.9K/s 00:00:00 [#####################] 100% core is up to date extra 312.3K 58.7K/s 00:00:05 [#####################] 100% community 335.9K 58.7K/s 00:00:06 [#####################] 100% :: Starting full system upgrade... local database is up to date pacman built from git (in last 10 minutes) :: Synchronizing package databases... testing 15.1K 58.0K/s 00:00:00 [#####################] 100% core is up to date extra 312.3K 58.7K/s 00:00:05 [#####################] 100% community 335.9K 58.8K/s 00:00:06 [#####################] 100% :: Starting full system upgrade... local database is up to date Allan
Allan McRae wrote:
Dan McGee wrote:
Can anyone else try to reproduce this? Try deleting all .lastupdate files except the one for core, if you have testing enabled, and seeing what happens on the first and second runs of -Syu.
I can not replicate this at all...
I have even tried adding the extra repos you have enabled with no success in replicating. Allan
On Mon, May 12, 2008 at 7:38 AM, Allan McRae <mcrae_allan@hotmail.com> wrote:
Allan McRae wrote:
Dan McGee wrote:
Can anyone else try to reproduce this? Try deleting all .lastupdate files except the one for core, if you have testing enabled, and seeing what happens on the first and second runs of -Syu.
I can not replicate this at all...
I have even tried adding the extra repos you have enabled with no success in replicating.
I was able to bisect it at least- and it was me. I haven't looked into why this broke it yet, but if anyone else wants to poke at it, feel free. 046003844739416ff6d168dd2dec76490adb0727 is first bad commit commit 046003844739416ff6d168dd2dec76490adb0727 Author: Dan McGee <dan@archlinux.org> Date: Sun May 11 19:29:23 2008 -0500 Remove some useless abstraction and start db cleanup We have some useless abstractions like an alpm_db_rewind function. I've read somewhere that readdir() was the worst filesystem function call invented, and what do we do? Add a wrapper around it. Kill this abstraction and move some other things into be_files that should be there anyway because they are so tied to how a files backend works. Signed-off-by: Dan McGee <dan@archlinux.org> :040000 040000 534812c4e1341825ff0ef0446bba8a7e865f6d43 24fd6a8b87d064809de09a295e7b8fd4ede198b9 M lib bisect run success $ git bisect log # bad: [663408532ae852e7123da6b9658df1cacc0c642d] Merge branch 'maint' # good: [e3d35b3274f13cc9ed8d79fa873d6348e424259e] Add detailed description to alpm_pkg_load git-bisect start 'master' 'maint' # good: [35135c0a0cbac592e72296c0ca64e9def0308942] Add -Rss option git-bisect good 35135c0a0cbac592e72296c0ca64e9def0308942 # good: [db4258c1fdf27708968baf9c4e730a014af40fd8] Some comments for _alpm_unpack. git-bisect good db4258c1fdf27708968baf9c4e730a014af40fd8 # good: [4e6361642e632b8631e3d12ee33b1cf5293edb83] Rework extract_single_file() temp file creation git-bisect good 4e6361642e632b8631e3d12ee33b1cf5293edb83 # good: [502645c0e304a8ee062d8da7f162ff4c195e6be8] makepkg: Unify start and end messages git-bisect good 502645c0e304a8ee062d8da7f162ff4c195e6be8 # good: [0bfc8adf377e7c0d4870fd79999b359a00bc96e2] contrib/paclist: list packages installed from given repo. git-bisect good 0bfc8adf377e7c0d4870fd79999b359a00bc96e2 # good: [3c3cb001a441656c2afba62f0361b83d4987339c] Make all error messages use pm_fprintf git-bisect good 3c3cb001a441656c2afba62f0361b83d4987339c # bad: [13f24a5bdabf9c2c7bfa07aff7176495bb775a0d] Refactor pkg_load/parse_descfile into a new backend file git-bisect bad 13f24a5bdabf9c2c7bfa07aff7176495bb775a0d # bad: [046003844739416ff6d168dd2dec76490adb0727] Remove some useless abstraction and start db cleanup git-bisect bad 046003844739416ff6d168dd2dec76490adb0727
On Mon, May 12, 2008 at 7:54 AM, Dan McGee <dpmcgee@gmail.com> wrote:
On Mon, May 12, 2008 at 7:38 AM, Allan McRae <mcrae_allan@hotmail.com> wrote:
Allan McRae wrote:
Dan McGee wrote:
Can anyone else try to reproduce this? Try deleting all .lastupdate files except the one for core, if you have testing enabled, and seeing what happens on the first and second runs of -Syu.
I can not replicate this at all...
I have even tried adding the extra repos you have enabled with no success in replicating.
I was able to bisect it at least- and it was me. I haven't looked into why this broke it yet, but if anyone else wants to poke at it, feel free.
046003844739416ff6d168dd2dec76490adb0727 is first bad commit commit 046003844739416ff6d168dd2dec76490adb0727 Author: Dan McGee <dan@archlinux.org> Date: Sun May 11 19:29:23 2008 -0500
Remove some useless abstraction and start db cleanup
We have some useless abstractions like an alpm_db_rewind function. I've read somewhere that readdir() was the worst filesystem function call invented, and what do we do? Add a wrapper around it. Kill this abstraction and move some other things into be_files that should be there anyway because they are so tied to how a files backend works.
Signed-off-by: Dan McGee <dan@archlinux.org>
:040000 040000 534812c4e1341825ff0ef0446bba8a7e865f6d43 24fd6a8b87d064809de09a295e7b8fd4ede198b9 M lib bisect run success
$ git bisect log # bad: [663408532ae852e7123da6b9658df1cacc0c642d] Merge branch 'maint' # good: [e3d35b3274f13cc9ed8d79fa873d6348e424259e] Add detailed description to alpm_pkg_load git-bisect start 'master' 'maint' # good: [35135c0a0cbac592e72296c0ca64e9def0308942] Add -Rss option git-bisect good 35135c0a0cbac592e72296c0ca64e9def0308942 # good: [db4258c1fdf27708968baf9c4e730a014af40fd8] Some comments for _alpm_unpack. git-bisect good db4258c1fdf27708968baf9c4e730a014af40fd8 # good: [4e6361642e632b8631e3d12ee33b1cf5293edb83] Rework extract_single_file() temp file creation git-bisect good 4e6361642e632b8631e3d12ee33b1cf5293edb83 # good: [502645c0e304a8ee062d8da7f162ff4c195e6be8] makepkg: Unify start and end messages git-bisect good 502645c0e304a8ee062d8da7f162ff4c195e6be8 # good: [0bfc8adf377e7c0d4870fd79999b359a00bc96e2] contrib/paclist: list packages installed from given repo. git-bisect good 0bfc8adf377e7c0d4870fd79999b359a00bc96e2 # good: [3c3cb001a441656c2afba62f0361b83d4987339c] Make all error messages use pm_fprintf git-bisect good 3c3cb001a441656c2afba62f0361b83d4987339c # bad: [13f24a5bdabf9c2c7bfa07aff7176495bb775a0d] Refactor pkg_load/parse_descfile into a new backend file git-bisect bad 13f24a5bdabf9c2c7bfa07aff7176495bb775a0d # bad: [046003844739416ff6d168dd2dec76490adb0727] Remove some useless abstraction and start db cleanup git-bisect bad 046003844739416ff6d168dd2dec76490adb0727
diff --git a/lib/libalpm/cache.c b/lib/libalpm/cache.c index fcd555e..bfbf995 100644 --- a/lib/libalpm/cache.c +++ b/lib/libalpm/cache.c @@ -24,6 +24,7 @@ #include <stdlib.h> #include <errno.h> #include <string.h> +#include <dirent.h> /* libalpm */ #include "cache.h" @@ -54,6 +55,7 @@ int _alpm_db_load_pkgcache(pmdb_t *db) _alpm_log(PM_LOG_DEBUG, "loading package cache for repository '%s'\n", db->treename); + rewinddir(db->handle); while((info = _alpm_db_scan(db, NULL)) != NULL) { _alpm_log(PM_LOG_FUNCTION, "adding '%s' to package cache for db '%s'\n", alpm_pkg_get_name(info), db->treename);
On Mon, May 12, 2008 at 2:58 PM, Dan McGee <dpmcgee@gmail.com> wrote:
diff --git a/lib/libalpm/cache.c b/lib/libalpm/cache.c index fcd555e..bfbf995 100644 --- a/lib/libalpm/cache.c +++ b/lib/libalpm/cache.c @@ -24,6 +24,7 @@ #include <stdlib.h> #include <errno.h> #include <string.h> +#include <dirent.h>
/* libalpm */ #include "cache.h" @@ -54,6 +55,7 @@ int _alpm_db_load_pkgcache(pmdb_t *db) _alpm_log(PM_LOG_DEBUG, "loading package cache for repository '%s'\n", db->treename);
+ rewinddir(db->handle); while((info = _alpm_db_scan(db, NULL)) != NULL) { _alpm_log(PM_LOG_FUNCTION, "adding '%s' to package cache for db '%s'\n",
alpm_pkg_get_name(info), db->treename);
Oh no shit, I noticed this and wanted to ask you about it, why you removed that rewinddir call... But then I looked at db_scan code, and it seems like it did a rewinddir itself, so I thought it was fine ... And when you showed your weird bug, I thought about that rewinddir stuff again.. But I didn't say anything.. I suck. I was still building your last git branch to try reproducing it first.
On Mon, May 12, 2008 at 8:04 AM, Xavier <shiningxc@gmail.com> wrote:
On Mon, May 12, 2008 at 2:58 PM, Dan McGee <dpmcgee@gmail.com> wrote:
diff --git a/lib/libalpm/cache.c b/lib/libalpm/cache.c index fcd555e..bfbf995 100644 --- a/lib/libalpm/cache.c +++ b/lib/libalpm/cache.c @@ -24,6 +24,7 @@ #include <stdlib.h> #include <errno.h> #include <string.h> +#include <dirent.h>
/* libalpm */ #include "cache.h" @@ -54,6 +55,7 @@ int _alpm_db_load_pkgcache(pmdb_t *db) _alpm_log(PM_LOG_DEBUG, "loading package cache for repository '%s'\n", db->treename);
+ rewinddir(db->handle); while((info = _alpm_db_scan(db, NULL)) != NULL) { _alpm_log(PM_LOG_FUNCTION, "adding '%s' to package cache for db '%s'\n",
alpm_pkg_get_name(info), db->treename);
Oh no shit, I noticed this and wanted to ask you about it, why you removed that rewinddir call... But then I looked at db_scan code, and it seems like it did a rewinddir itself, so I thought it was fine ... And when you showed your weird bug, I thought about that rewinddir stuff again.. But I didn't say anything.. I suck. I was still building your last git branch to try reproducing it first.
I thought the same thing- I saw a rewinddir call in db_scan, so I removed what I thought was an extra one here. Either way, I don't plan on applying the above patch, as we should just do the rewinddir() call inside db_scan() or do something more sane than the above hackjob. I just wanted to point out this was the issue. Allan- I don't know what filesystem you are using, but I wonder if it has slightly different *dir() semantics so the bug did not manifest itself there... -Dan
On Mon, May 12, 2008 at 4:00 PM, Dan McGee <dpmcgee@gmail.com> wrote:
I thought the same thing- I saw a rewinddir call in db_scan, so I removed what I thought was an extra one here. Either way, I don't plan on applying the above patch, as we should just do the rewinddir() call inside db_scan() or do something more sane than the above hackjob. I just wanted to point out this was the issue.
Ok, I just understood how that worked now... I first thought we could do rewinddir in db_scan but it's not easy. When we do a full scan, we want do init the scan first with db_rewind, then scan each entry one by one using one db_scan call each time. I preferred the old way rather than the above hack, even if they do exactly the same thing. It was a bit less obscure and more consistent: _alpm_db_rewind(db); while((info = _alpm_db_scan(db, NULL)) != NULL) { Unless you find a better way of course :)
Allan- I don't know what filesystem you are using, but I wonder if it has slightly different *dir() semantics so the bug did not manifest itself there...
I was able to reproduce it without problems, with latest git and ext3 filesystem.
Xavier wrote:
On Mon, May 12, 2008 at 4:00 PM, Dan McGee <dpmcgee@gmail.com> wrote:
Allan- I don't know what filesystem you are using, but I wonder if it has slightly different *dir() semantics so the bug did not manifest itself there...
I was able to reproduce it without problems, with latest git and ext3 filesystem.
I am using latest git and ext3 too... I must be immune to all bugs!
Commit 046003844739416ff6d168dd2dec76490adb0727 caused a regression when rereading the pkgcache after updating the on-disk databases. A rewinddir call was errantly removed. This patch looks a bit more significant than simply replacing the rewinddir command, but that is what it does in addition to moving the db pkgcache population to a new _alpm_db_populate method and removing what I'm 99% positive are extra calls setting pkg->origin data. Signed-off-by: Dan McGee <dan@archlinux.org> --- lib/libalpm/be_files.c | 17 +++++++++++++++++ lib/libalpm/cache.c | 13 ++----------- lib/libalpm/db.h | 1 + 3 files changed, 20 insertions(+), 11 deletions(-) diff --git a/lib/libalpm/be_files.c b/lib/libalpm/be_files.c index 1e59055..7261e06 100644 --- a/lib/libalpm/be_files.c +++ b/lib/libalpm/be_files.c @@ -373,6 +373,23 @@ pmpkg_t *_alpm_db_scan(pmdb_t *db, const char *target) return(pkg); } +int _alpm_db_populate(pmdb_t *db) +{ + pmpkg_t *info; + int count = 0; + + rewinddir(db->handle); + while((info = _alpm_db_scan(db, NULL)) != NULL) { + _alpm_log(PM_LOG_FUNCTION, "adding '%s' to package cache for db '%s'\n", + alpm_pkg_get_name(info), db->treename); + /* add to the collection */ + db->pkgcache = alpm_list_add(db->pkgcache, info); + count++; + } + + return(count); +} + int _alpm_db_read(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq) { FILE *fp = NULL; diff --git a/lib/libalpm/cache.c b/lib/libalpm/cache.c index fcd555e..57784e8 100644 --- a/lib/libalpm/cache.c +++ b/lib/libalpm/cache.c @@ -40,8 +40,7 @@ */ int _alpm_db_load_pkgcache(pmdb_t *db) { - pmpkg_t *info; - int count = 0; + int count; ALPM_LOG_FUNC; @@ -54,15 +53,7 @@ int _alpm_db_load_pkgcache(pmdb_t *db) _alpm_log(PM_LOG_DEBUG, "loading package cache for repository '%s'\n", db->treename); - while((info = _alpm_db_scan(db, NULL)) != NULL) { - _alpm_log(PM_LOG_FUNCTION, "adding '%s' to package cache for db '%s'\n", - alpm_pkg_get_name(info), db->treename); - info->origin = PKG_FROM_CACHE; - info->origin_data.db = db; - /* add to the collection */ - db->pkgcache = alpm_list_add(db->pkgcache, info); - count++; - } + count = _alpm_db_populate(db); db->pkgcache = alpm_list_msort(db->pkgcache, count, _alpm_pkg_cmp); return(0); diff --git a/lib/libalpm/db.h b/lib/libalpm/db.h index f6e0c3c..17cf317 100644 --- a/lib/libalpm/db.h +++ b/lib/libalpm/db.h @@ -63,6 +63,7 @@ alpm_list_t *_alpm_db_whatprovides(pmdb_t *db, const char *package); int _alpm_db_open(pmdb_t *db); void _alpm_db_close(pmdb_t *db); pmpkg_t *_alpm_db_scan(pmdb_t *db, const char *target); +int _alpm_db_populate(pmdb_t *db); int _alpm_db_read(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq); int _alpm_db_write(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq); int _alpm_db_remove(pmdb_t *db, pmpkg_t *info); -- 1.5.5.1
Idézés Xavier <shiningxc@gmail.com>:
On Mon, May 12, 2008 at 4:00 PM, Dan McGee <dpmcgee@gmail.com> wrote:
I thought the same thing- I saw a rewinddir call in db_scan, so I removed what I thought was an extra one here. Either way, I don't plan on applying the above patch, as we should just do the rewinddir() call inside db_scan() or do something more sane than the above hackjob. I just wanted to point out this was the issue.
Ok, I just understood how that worked now... I first thought we could do rewinddir in db_scan but it's not easy. When we do a full scan, we want do init the scan first with db_rewind, then scan each entry one by one using one db_scan call each time.
I preferred the old way rather than the above hack, even if they do exactly the same thing. It was a bit less obscure and more consistent: _alpm_db_rewind(db); while((info = _alpm_db_scan(db, NULL)) != NULL) {
As I see, we always call _alpm_db_scan with NULL param. I think we can remove this param and modify alpm_db_scan to return with alpm_list_t of packages, which can be thought as TOC, and the packages can be accessed via alpm_db_read. Thus the rewinddir stuff can be moved there. On the other side we would lose some flexibility (however, we can read whole db, and do an ~alpm_pkg_find + alpm_db_read; and if this is needed more than once it worth loading pkgcache). Bye ---------------------------------------------------- SZTE Egyetemi Könyvtár - http://www.bibl.u-szeged.hu This mail sent through IMP: http://horde.org/imp/
participants (5)
-
Allan McRae
-
Dan McGee
-
Dan McGee
-
Nagy Gabor
-
Xavier