[pacman-dev] Removing directory symlink support
The bugs in 4.1 are mostly addressed and I will make a 4.1.1 release sometime soon, so now is time to think about pacman-4.2. I wanted to discuss the removal of directory symlink support again because there was not much response on the list last time. Currently, if there is a symlink to a directory on the filesystem, we can treat it like a directory. e.g. Arch Linux has /lib -> usr/lib. So if a package installs a file in /lib, pacman silently puts it in /usr/lib. I'd like to remove that support for the following reasons: 1) It requires we resolve the true path of every directory involved in an upgrade for conflict checking. e.g. /lib/foo.so and /usr/lib/foo.so are conflicts. This has substantial overhead, both in terms of computing and in code. Removing this would mean all symlinks would be treated like a file. 2) We do not handle it very well. e.g. if a package contains a /foo -> some/dir symlink, it needs to be installed before any package with a file in /foo in the same transaction. We "implement" this through flagging a conflict. When the package containing the directory symlink is removed, we need to scan all packages to ensure it is not being "used" by any other package in the local database. This is covered by the _failing_ pactests sync700-702. 3) If packages really want to install files in a directory where there is a symlink, it is easily dealt with in the PKGBUILD. e.g. package() { ln -s usr/lib $pkgdir/lib make DESTDIR=$pkgdir rm $pkgdir/lib } 4) There is no use case for it (at least on linux). I have only ever seen this used as a poor man's bind mount. 5) It believe it would actually make replacing /bin and /sbin able to occur in a single transaction (no --ignore filesystem) as we can replace a completely removed directory by a file. Any comments here? I see there is a patchset already available that does most (all?) of the removal. What we would need is to have something like a pacman-db-upgrade that changed filelists to avoid symlink paths. E.g a package with /lib/foo haves its filelist adjusted to /usr/lib/foo. The other option is to have testdb check for these and advise people to run that before updating to pacman-4.2? Any comments? Allan
Allan McRae
5) It believe it would actually make replacing /bin and /sbin able to occur in a single transaction (no --ignore filesystem) as we can replace a completely removed directory by a file.
So completing the /usr move is delayed until pacman 4.2 is released? -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
On 04/12/13 at 03:59pm, Allan McRae wrote:
I wanted to discuss the removal of directory symlink support again because there was not much response on the list last time. <snip> 5) It believe it would actually make replacing /bin and /sbin able to occur in a single transaction (no --ignore filesystem) as we can replace a completely removed directory by a file.
Sadly, this is not the case. In order for that to work, the directory to file transition logic in conflict.c would have to be able to take the packages being upgraded into account, which would require more complex transaction ordering logic than what we have now. Adding packages being removed as conflicts to the dir->file logic, however, was pretty easy once symlinks were removed, so I've added that, which fixes FS#32770. To better illustrate the effect removing symlink support has, here are the tests whose results changed from my no-symlink branch. --- fileconflict013.py | 4 ++-- fileconflict023.py | 6 +++--- fileconflict025.py | 6 +++--- symlink001.py | 9 ++++----- sync700.py | 9 ++++----- sync701.py | 9 ++++----- sync702.py | 9 ++++----- 10 files changed, 42 insertions(+), 29 deletions(-) diff --git a/test/pacman/tests/fileconflict013.py b/test/pacman/tests/fileconflict013.py index a83923c..94c169d 100644 --- a/test/pacman/tests/fileconflict013.py +++ b/test/pacman/tests/fileconflict013.py @@ -1,20 +1,20 @@ self.description = "file->file path change with same effective path (/lib as symlink)" lp1 = pmpkg("filesystem", "1.0-1") lp1.files = ["usr/", "usr/lib/", "lib -> usr/lib/"] self.addpkg2db("local", lp1) lp2 = pmpkg("pkg1", "1.0-1") lp2.files = ["lib/libfoo.so"] self.addpkg2db("local", lp2) sp1 = pmpkg("pkg1", "1.0-2") sp1.files = ["usr/lib/libfoo.so"] self.addpkg2db("sync", sp1) self.args = "-Su" -self.addrule("PACMAN_RETCODE=0") -self.addrule("PKG_VERSION=pkg1|1.0-2") +self.addrule("PACMAN_RETCODE=1") +self.addrule("PKG_VERSION=pkg1|1.0-1") diff --git a/test/pacman/tests/fileconflict023.py b/test/pacman/tests/fileconflict023.py index ce72087..1367080 100644 --- a/test/pacman/tests/fileconflict023.py +++ b/test/pacman/tests/fileconflict023.py @@ -1,18 +1,18 @@ self.description = "File conflict between package with symlink and package with real path resolved by removal" self.filesystem = ["usr/", "usr/lib/", "lib -> usr/lib/"] lp1 = pmpkg("foo") lp1.files = ["lib/", "lib/file"] self.addpkg2db("local", lp1) sp1 = pmpkg("bar") sp1.conflicts = ["foo"] sp1.files = ["usr/", "usr/lib/", "usr/lib/file"] self.addpkg2db("sync", sp1) self.args = "-S %s --ask=4" % sp1.name -self.addrule("PACMAN_RETCODE=0") -self.addrule("!PKG_EXIST=foo") -self.addrule("PKG_EXIST=bar") +self.addrule("PACMAN_RETCODE=1") +self.addrule("PKG_EXIST=foo") +self.addrule("!PKG_EXIST=bar") diff --git a/test/pacman/tests/fileconflict025.py b/test/pacman/tests/fileconflict025.py index 3c6f063..013c223 100644 --- a/test/pacman/tests/fileconflict025.py +++ b/test/pacman/tests/fileconflict025.py @@ -1,18 +1,18 @@ self.description = "File conflict between package with symlink and package with real path resolved by removal (reversed)" self.filesystem = ["usr/lib/", "lib -> usr/lib/"] lp1 = pmpkg("foo") lp1.files = ["usr/", "usr/lib/", "usr/lib/file"] self.addpkg2db("local", lp1) sp1 = pmpkg("bar") sp1.conflicts = ["foo"] sp1.files = ["lib/", "lib/file"] self.addpkg2db("sync", sp1) self.args = "-S %s --ask=4" % sp1.name -self.addrule("PACMAN_RETCODE=0") -self.addrule("!PKG_EXIST=foo") -self.addrule("PKG_EXIST=bar") +self.addrule("PACMAN_RETCODE=1") +self.addrule("PKG_EXIST=foo") +self.addrule("!PKG_EXIST=bar") diff --git a/test/pacman/tests/symlink001.py b/test/pacman/tests/symlink001.py index cbf71cc..9136551 100644 --- a/test/pacman/tests/symlink001.py +++ b/test/pacman/tests/symlink001.py @@ -1,20 +1,19 @@ self.description = "Dir symlinks overwritten on install (the perl/git bug)" lp = pmpkg("dummy") lp.files = ["dir/realdir/", "dir/symdir -> realdir"] self.addpkg2db("local", lp) p = pmpkg("pkg1") p.files = ["dir/symdir/tmp"] self.addpkg(p) self.args = "-U %s" % p.filename() -self.addrule("PACMAN_RETCODE=0") -self.addrule("PKG_EXIST=pkg1") -self.addrule("FILE_EXIST=dir/symdir/tmp") -self.addrule("FILE_EXIST=dir/realdir/tmp") -self.addrule("FILE_TYPE=dir/symdir/tmp|file") +self.addrule("PACMAN_RETCODE=1") +self.addrule("!PKG_EXIST=pkg1") +self.addrule("!FILE_EXIST=dir/symdir/tmp") +self.addrule("!FILE_EXIST=dir/realdir/tmp") self.addrule("FILE_TYPE=dir/symdir|link") self.addrule("FILE_TYPE=dir/realdir|dir") diff --git a/test/pacman/tests/sync700.py b/test/pacman/tests/sync700.py index 9748c81..8e908e0 100644 --- a/test/pacman/tests/sync700.py +++ b/test/pacman/tests/sync700.py @@ -1,22 +1,21 @@ self.description = "do not remove directory symlink if another package has file in its path" lp1 = pmpkg("pkg1") lp1.files = ["usr/lib/foo", "lib -> usr/lib"] self.addpkg2db("local", lp1) lp2 = pmpkg("pkg2") lp2.files = ["lib/bar"] self.addpkg2db("local", lp2) p = pmpkg("pkg1", "1.0-2") p.files = ["usr/lib/foo"] self.addpkg2db("sync", p) self.args = "-S pkg1" -self.addrule("PACMAN_RETCODE=1") -self.addrule("PKG_VERSION=pkg1|1.0-1") -self.addrule("FILE_EXIST=lib/bar") - -self.expectfailure = True +self.addrule("PACMAN_RETCODE=0") +self.addrule("PKG_VERSION=pkg1|1.0-2") +self.addrule("FILE_EXIST=usr/lib/bar") +self.addrule("!FILE_EXIST=lib/bar") diff --git a/test/pacman/tests/sync701.py b/test/pacman/tests/sync701.py index 201f602..977b1a0 100644 --- a/test/pacman/tests/sync701.py +++ b/test/pacman/tests/sync701.py @@ -1,22 +1,21 @@ self.description = "do not remove directory symlink if incoming package has file in its path (order 1)" lp = pmpkg("pkg1") lp.files = ["usr/lib/foo", "lib -> usr/lib"] self.addpkg2db("local", lp) p1 = pmpkg("pkg1", "1.0-2") p1.files = ["usr/lib/foo"] self.addpkg2db("sync", p1) p2 = pmpkg("pkg2") p2.files = ["lib/bar"] self.addpkg2db("sync", p2) self.args = "-S pkg1 pkg2" -self.addrule("PACMAN_RETCODE=1") -self.addrule("PKG_VERSION=pkg1|1.0-1") -self.addrule("!PKG_EXIST=pkg2") - -self.expectfailure = True +self.addrule("PACMAN_RETCODE=0") +self.addrule("PKG_VERSION=pkg1|1.0-2") +self.addrule("PKG_EXIST=pkg2") +self.addrule("FILE_TYPE=lib|dir") diff --git a/test/pacman/tests/sync702.py b/test/pacman/tests/sync702.py index ee4eef9..a7904c3 100644 --- a/test/pacman/tests/sync702.py +++ b/test/pacman/tests/sync702.py @@ -1,22 +1,21 @@ self.description = "do not remove directory symlink if incoming package has file in its path (order 2)" lp = pmpkg("pkg2") lp.files = ["usr/lib/foo", "lib -> usr/lib"] self.addpkg2db("local", lp) p1 = pmpkg("pkg1") p1.files = ["lib/bar"] self.addpkg2db("sync", p1) p2 = pmpkg("pkg2", "1.0-2") p2.files = ["usr/lib/foo"] self.addpkg2db("sync", p2) self.args = "-S pkg1 pkg2" -self.addrule("PACMAN_RETCODE=1") -self.addrule("PKG_VERSION=pkg2|1.0-1") -self.addrule("!PKG_EXIST=pkg1") - -self.expectfailure = True +self.addrule("PACMAN_RETCODE=0") +self.addrule("PKG_VERSION=pkg2|1.0-2") +self.addrule("PKG_EXIST=pkg1") +self.addrule("FILE_TYPE=lib|dir")
Two weeks with no objections, so here is the full patchset. Several of the tests should be redundant now, but I left them in with updated rules anyway, just in case. Also available on my no-symlinks branch for those interested. Andrew Gregory (8): update tests for symlink support removal alpm_filelist: remove resolved_path conflict.c: do not ignore symlink<->dir conflicts conflict.c: use real path for filesystem checks extract_single_file: consolidate symlink cases unlink_file: treat symlinks like normal files query_fileowner: remove symlink support improve dir->file transition conflict resolution lib/libalpm/add.c | 33 ++--- lib/libalpm/alpm.h | 3 +- lib/libalpm/conflict.c | 192 +++++++++++++----------------- lib/libalpm/filelist.c | 225 ++--------------------------------- lib/libalpm/filelist.h | 4 - lib/libalpm/package.c | 22 +--- lib/libalpm/remove.c | 11 +- src/common/util-common.c | 43 +++++++ src/common/util-common.h | 1 + src/pacman/query.c | 90 +++----------- test/pacman/tests/fileconflict007.py | 1 + test/pacman/tests/fileconflict013.py | 5 +- test/pacman/tests/fileconflict022.py | 1 - test/pacman/tests/fileconflict023.py | 7 +- test/pacman/tests/fileconflict025.py | 8 +- test/pacman/tests/fileconflict030.py | 17 +++ test/pacman/tests/symlink001.py | 9 +- test/pacman/tests/sync700.py | 12 +- test/pacman/tests/sync701.py | 11 +- test/pacman/tests/sync702.py | 11 +- 20 files changed, 217 insertions(+), 489 deletions(-) create mode 100644 test/pacman/tests/fileconflict030.py -- 1.8.2.1
Signed-off-by: Andrew Gregory
Signed-off-by: Andrew Gregory
Signed-off-by: Andrew Gregory
In the event of a symlink<->dir transition, the paths for the incoming
package need to be resolved to match the filelist for installed
packages.
Note: this only enables the symlink<->dir transition itself. Any
installed packages with invalid file lists containing symlinks are not
detected.
Signed-off-by: Andrew Gregory
Signed-off-by: Andrew Gregory
We always want to work with the package file itself, not its target if
it's a symlink.
Signed-off-by: Andrew Gregory
Signed-off-by: Andrew Gregory
Packages removed due to conflicts are always removed at the beginning of
the transaction and as such can be included in the check for whether all
owners of a directory will be removed in a transaction. Installed
versions of packages being upgraded, other than the one with the
conflict, cannot be used because our transaction ordering is not
intelligent enough to ensure that they are removed prior to the
installation of the conflicted package.
Also, return false from dir_belongsto_pkgs on errors. Previously, we
simply continued which could return true even if we were unable to
actually establish that the package owned the entire tree.
Signed-off-by: Andrew Gregory
On 27/04/13 10:00, Andrew Gregory wrote:
Signed-off-by: Andrew Gregory
--- lib/libalpm/add.c | 33 ++++++++------------------------- 1 file changed, 8 insertions(+), 25 deletions(-) diff --git a/lib/libalpm/add.c b/lib/libalpm/add.c index 1d9db60..0b8aedf 100644 --- a/lib/libalpm/add.c +++ b/lib/libalpm/add.c @@ -204,19 +204,17 @@ static int extract_single_file(alpm_handle_t *handle, struct archive *archive, * D | 10 | 11 | 12 * * 1,2,3- extract, no magic necessary. lstat (_alpm_lstat) will fail here. - * 4,5,6,7,8- conflict checks should have caught this. either overwrite + * 4,5,6,7,8,9- conflict checks should have caught this. either overwrite * or backup the file. - * 9- follow the symlink, hopefully it is a directory, check it. - * 10- file replacing directory- don't allow it. - * 11- don't extract symlink- a dir exists here. we don't want links to - * links, etc. + * 10,11- file replacing directory- don't allow it. * 12- skip extraction, dir already exists. */
- /* do both a lstat and a stat, so we can see what symlinks point to */ struct stat lsbuf, sbuf; - if(_alpm_lstat(filename, &lsbuf) != 0 || stat(filename, &sbuf) != 0) { - /* cases 1,2,3: couldn't stat an existing file, skip all backup checks */ + if(stat(filename, &sbuf) != 0 || _alpm_lstat(filename, &lsbuf) != 0) { + /* cases 1,2,3: file doesn't exist or is a broken symlink, + * skip all backup checks */ + /* TODO: should we really treat broken symlinks as if they don't exist? */
NO! We definitely should not... In fact, I'd like to see this patch result in this: * (F=file, N=node, S=symlink, D=dir) * | F/N/S | D * non-existent | 1 | 2 * F/N/S | 3 | 4 * D | 5 | 6 because now there is no difference between a file and a symlink. Which probably means getting rid of the lstat call (although I did not look at the code to confirm...)
} else { if(S_ISDIR(lsbuf.st_mode)) { if(S_ISDIR(entrymode)) { @@ -243,23 +241,8 @@ static int extract_single_file(alpm_handle_t *handle, struct archive *archive, archive_read_data_skip(archive); return 1; } - } else if(S_ISLNK(lsbuf.st_mode) && S_ISDIR(entrymode)) { - /* case 9: existing symlink, dir in package */ - if(S_ISDIR(sbuf.st_mode)) { - /* the symlink on FS is to a directory, so we'll use it */ - _alpm_log(handle, ALPM_LOG_DEBUG, "extract: skipping symlink overwrite of %s\n", - filename); - archive_read_data_skip(archive); - return 0; - } else { - /* this is BAD. symlink was not to a directory */ - _alpm_log(handle, ALPM_LOG_ERROR, _("extract: symlink %s does not point to dir\n"), - filename); - archive_read_data_skip(archive); - return 1; - } - } else if(S_ISREG(lsbuf.st_mode) && S_ISDIR(entrymode)) { - /* case 6: trying to overwrite file with dir */ + } else if(!S_ISDIR(lsbuf.st_mode) && S_ISDIR(entrymode)) { + /* case 6,9: trying to overwrite file with dir */ _alpm_log(handle, ALPM_LOG_DEBUG, "extract: overwriting file with dir %s\n", filename); } else if(S_ISREG(entrymode)) {
On 27/04/13 10:00, Andrew Gregory wrote:
In the event of a symlink<->dir transition, the paths for the incoming package need to be resolved to match the filelist for installed packages.
Can you explain this to me? Every time I think I understand, I don't... e.g. outdated local package: /usr/foo -> /usr/bar/ /usr/bar/baz update sync package: /usr/foo/baz So /usr/foo/baz is a new file so it is tested for presence in the filesystem. But /usr/foo/baz "exists" on the filesystem as /usr/bar/baz.
From what I understand, you resolve the package file "/usr/foo/baz" to "/usr/bar/baz" and note that it is in the old package so not a conflict. Which feels wrong to me...
Can we do something like testing whether the directory of the "conflicting" file and the resolved path of that directory are different? If not, then we must be dealing with a case of a filesystem symlink being replaced with a package directory (otherwise the package directory would have conflicted with the filesystem symlink earlier).
Note: this only enables the symlink<->dir transition itself. Any installed packages with invalid file lists containing symlinks are not detected.
Signed-off-by: Andrew Gregory
--- lib/libalpm/conflict.c | 42 ++++++++++++++++------------------- src/common/util-common.c | 43 ++++++++++++++++++++++++++++++++++++ src/common/util-common.h | 1 + test/pacman/tests/fileconflict007.py | 2 -- test/pacman/tests/sync701.py | 2 -- test/pacman/tests/sync702.py | 2 -- 6 files changed, 63 insertions(+), 29 deletions(-) diff --git a/lib/libalpm/conflict.c b/lib/libalpm/conflict.c index 518a2c6..08c1cbe 100644 --- a/lib/libalpm/conflict.c +++ b/lib/libalpm/conflict.c @@ -501,7 +501,7 @@ alpm_list_t *_alpm_db_find_fileconflicts(alpm_handle_t *handle, /* have we acted on this conflict? */ int resolved_conflict = 0; struct stat lsbuf; - char path[PATH_MAX]; + char path[PATH_MAX], resolved_path[PATH_MAX]; size_t pathlen;
pathlen = snprintf(path, PATH_MAX, "%s%s", handle->root, filestr); @@ -513,7 +513,7 @@ alpm_list_t *_alpm_db_find_fileconflicts(alpm_handle_t *handle,
_alpm_log(handle, ALPM_LOG_DEBUG, "checking possible conflict: %s\n", path);
- if(filestr[strlen(filestr) - 1] == '/') { + if(path[pathlen - 1] == '/') { if(S_ISDIR(lsbuf.st_mode)) { _alpm_log(handle, ALPM_LOG_DEBUG, "file is a directory, not a conflict\n"); continue; @@ -524,7 +524,19 @@ alpm_list_t *_alpm_db_find_fileconflicts(alpm_handle_t *handle, path[pathlen - 1] = '\0'; }
- relative_path = path + rootlen; + if(lrealpath(path, resolved_path) + && strncmp(resolved_path, handle->root, rootlen) == 0) { + relative_path = resolved_path + rootlen; + } else { + relative_path = path + rootlen; + } + + if(!resolved_conflict && dbpkg + && alpm_filelist_contains(alpm_pkg_get_files(dbpkg), relative_path)) { + _alpm_log(handle, ALPM_LOG_DEBUG, + "package contained the resolved realpath\n"); + resolved_conflict = 1; + }
/* Check remove list (will we remove the conflicting local file?) */ for(k = rem; k && !resolved_conflict; k = k->next) { @@ -546,12 +558,12 @@ alpm_list_t *_alpm_db_find_fileconflicts(alpm_handle_t *handle, alpm_pkg_t *localp2 = _alpm_db_get_pkgfromcache(handle->db_local, p2->name);
/* localp2->files will be removed (target conflicts are handled by CHECK 1) */ - if(localp2 && alpm_filelist_contains(alpm_pkg_get_files(localp2), filestr)) { + if(localp2 && alpm_filelist_contains(alpm_pkg_get_files(localp2), relative_path)) { /* skip removal of file, but not add. this will prevent a second * package from removing the file when it was already installed * by its new owner (whether the file is in backup array or not */ handle->trans->skip_remove = - alpm_list_add(handle->trans->skip_remove, strdup(filestr)); + alpm_list_add(handle->trans->skip_remove, strdup(relative_path)); _alpm_log(handle, ALPM_LOG_DEBUG, "file changed packages, adding to remove skiplist\n"); resolved_conflict = 1; @@ -560,8 +572,8 @@ alpm_list_t *_alpm_db_find_fileconflicts(alpm_handle_t *handle,
/* check if all files of the dir belong to the installed pkg */ if(!resolved_conflict && S_ISDIR(lsbuf.st_mode) && dbpkg) { - char *dir = malloc(strlen(filestr) + 2); - sprintf(dir, "%s/", filestr); + char *dir = malloc(strlen(relative_path) + 2); + sprintf(dir, "%s/", relative_path); if(alpm_filelist_contains(alpm_pkg_get_files(dbpkg), dir)) { _alpm_log(handle, ALPM_LOG_DEBUG, "checking if all files in %s belong to %s\n", @@ -571,22 +583,6 @@ alpm_list_t *_alpm_db_find_fileconflicts(alpm_handle_t *handle, free(dir); }
- /* check if a component of the filepath was a link. canonicalize the path - * and look for it in the old package. note that the actual file under - * consideration cannot itself be a link, as it might be unowned- path - * components can be safely checked as all directories are "unowned". */ - if(!resolved_conflict && dbpkg && !S_ISLNK(lsbuf.st_mode)) { - char rpath[PATH_MAX]; - if(realpath(path, rpath)) { - const char *relative_rpath = rpath + rootlen; - if(alpm_filelist_contains(alpm_pkg_get_files(dbpkg), relative_rpath)) { - _alpm_log(handle, ALPM_LOG_DEBUG, - "package contained the resolved realpath\n"); - resolved_conflict = 1; - } - } - } - /* is the file unowned and in the backup list of the new package? */ if(!resolved_conflict && _alpm_needbackup(filestr, p1)) { alpm_list_t *local_pkgs = _alpm_db_get_pkgcache(handle->db_local); diff --git a/src/common/util-common.c b/src/common/util-common.c index 7145e49..c2422e0 100644 --- a/src/common/util-common.c +++ b/src/common/util-common.c @@ -73,6 +73,49 @@ char *mdirname(const char *path) return strdup("."); }
+/** Resolve the canonicalized absolute path of a symlink. + * @param path path to resolve + * @param resolved_path destination for the resolved path, will be malloc'd if + * NULL + * @return the resolved path + */ +char *lrealpath(const char *path, char *resolved_path) +{ + const char *bname = mbasename(path); + char *rpath = NULL, *dname = NULL; + int success = 0; + + if(strcmp(path, ".") == 0 || strcmp(path, "..") == 0) { + /* the entire path needs to be resolved */ + return realpath(path, resolved_path); + } + + if(!(dname = mdirname(path))) { + goto cleanup; + } + if(!(rpath = realpath(dname, NULL))) { + goto cleanup; + } + if(!resolved_path) { + if(!(resolved_path = malloc(strlen(rpath) + strlen(bname) + 2))) { + goto cleanup; + } + } + + strcpy(resolved_path, rpath); + if(resolved_path[strlen(resolved_path) - 1] != '/') { + strcat(resolved_path, "/"); + } + strcat(resolved_path, bname); + success = 1; + +cleanup: + free(dname); + free(rpath); + + return (success ? resolved_path : NULL); +} + #ifndef HAVE_STRNDUP /* A quick and dirty implementation derived from glibc */ /** Determines the length of a fixed-size string. diff --git a/src/common/util-common.h b/src/common/util-common.h index 04d4e9d..b3a4df1 100644 --- a/src/common/util-common.h +++ b/src/common/util-common.h @@ -22,6 +22,7 @@
const char *mbasename(const char *path); char *mdirname(const char *path); +char *lrealpath(const char *path, char *resolved_path);
#ifndef HAVE_STRNDUP char *strndup(const char *s, size_t n); diff --git a/test/pacman/tests/fileconflict007.py b/test/pacman/tests/fileconflict007.py index b61ddb4..7fe65ed 100644 --- a/test/pacman/tests/fileconflict007.py +++ b/test/pacman/tests/fileconflict007.py @@ -16,5 +16,3 @@ self.addrule("PKG_EXIST=pkg") self.addrule("PKG_VERSION=pkg|1.0-2") self.addrule("FILE_TYPE=dir/symdir/|dir") - -self.expectfailure = True diff --git a/test/pacman/tests/sync701.py b/test/pacman/tests/sync701.py index 912c794..590845f 100644 --- a/test/pacman/tests/sync701.py +++ b/test/pacman/tests/sync701.py @@ -19,5 +19,3 @@ self.addrule("PKG_VERSION=pkg1|1.0-2") self.addrule("PKG_EXIST=pkg2") self.addrule("FILE_TYPE=lib|dir") - -self.expectfailure = True diff --git a/test/pacman/tests/sync702.py b/test/pacman/tests/sync702.py index 8f4c0ad..c3e2320 100644 --- a/test/pacman/tests/sync702.py +++ b/test/pacman/tests/sync702.py @@ -19,5 +19,3 @@ self.addrule("PKG_VERSION=pkg2|1.0-2") self.addrule("PKG_EXIST=pkg1") self.addrule("FILE_TYPE=lib|dir") - -self.expectfailure = True
On 27/04/13 10:00, Andrew Gregory wrote:
Two weeks with no objections, so here is the full patchset. Several of the tests should be redundant now, but I left them in with updated rules anyway, just in case.
Thanks - patches look good to me. I made a couple of comments on them. What we also need to figure out is the upgrade path here. There are packages in the Arch repos with files in /lib/... relying on the symlink at the moment. They will create conflicts after these patches (which is fine), but we need to make sure they will upgrade fine to the fixed versions. (They might already - but needs tested.) @Dan, @Dave (or anyone else...): Do you intend to comment on this proposal? Even an "ack" would be appreciated here. For reference: https://mailman.archlinux.org/pipermail/pacman-dev/2013-April/017001.html
Also available on my no-symlinks branch for those interested.
Andrew Gregory (8): update tests for symlink support removal alpm_filelist: remove resolved_path conflict.c: do not ignore symlink<->dir conflicts conflict.c: use real path for filesystem checks extract_single_file: consolidate symlink cases unlink_file: treat symlinks like normal files query_fileowner: remove symlink support improve dir->file transition conflict resolution
lib/libalpm/add.c | 33 ++--- lib/libalpm/alpm.h | 3 +- lib/libalpm/conflict.c | 192 +++++++++++++----------------- lib/libalpm/filelist.c | 225 ++--------------------------------- lib/libalpm/filelist.h | 4 - lib/libalpm/package.c | 22 +--- lib/libalpm/remove.c | 11 +- src/common/util-common.c | 43 +++++++ src/common/util-common.h | 1 + src/pacman/query.c | 90 +++----------- test/pacman/tests/fileconflict007.py | 1 + test/pacman/tests/fileconflict013.py | 5 +- test/pacman/tests/fileconflict022.py | 1 - test/pacman/tests/fileconflict023.py | 7 +- test/pacman/tests/fileconflict025.py | 8 +- test/pacman/tests/fileconflict030.py | 17 +++ test/pacman/tests/symlink001.py | 9 +- test/pacman/tests/sync700.py | 12 +- test/pacman/tests/sync701.py | 11 +- test/pacman/tests/sync702.py | 11 +- 20 files changed, 217 insertions(+), 489 deletions(-) create mode 100644 test/pacman/tests/fileconflict030.py
On 04/29/13 at 12:33am, Allan McRae wrote:
On 27/04/13 10:00, Andrew Gregory wrote:
In the event of a symlink<->dir transition, the paths for the incoming package need to be resolved to match the filelist for installed packages.
Can you explain this to me? Every time I think I understand, I don't...
e.g.
outdated local package: /usr/foo -> /usr/bar/ /usr/bar/baz
update sync package: /usr/foo/baz
So /usr/foo/baz is a new file so it is tested for presence in the filesystem. But /usr/foo/baz "exists" on the filesystem as /usr/bar/baz.
From what I understand, you resolve the package file "/usr/foo/baz" to "/usr/bar/baz" and note that it is in the old package so not a conflict. Which feels wrong to me...
Why would that be wrong? /usr/foo itself isn't a conflict and both /usr/foo and /usr/bar/baz will both be removed/overwritten in the upgrade process so /usr/foo/baz should be fine, should it not? In fact, we already do what you outlined (see the code below), this patch just does the resolution upfront so the real path can be used for the rest of the checks as well (ie "changing hands" from one package to another).
- /* check if a component of the filepath was a link. canonicalize the path - * and look for it in the old package. note that the actual file under - * consideration cannot itself be a link, as it might be unowned- path - * components can be safely checked as all directories are "unowned". */ - if(!resolved_conflict && dbpkg && !S_ISLNK(lsbuf.st_mode)) { - char rpath[PATH_MAX]; - if(realpath(path, rpath)) { - const char *relative_rpath = rpath + rootlen; - if(alpm_filelist_contains(alpm_pkg_get_files(dbpkg), relative_rpath)) { - _alpm_log(handle, ALPM_LOG_DEBUG, - "package contained the resolved realpath\n"); - resolved_conflict = 1; - } - } - } -
On Mon, Apr 29, 2013 at 12:41:50AM +1000, Allan McRae wrote:
On 27/04/13 10:00, Andrew Gregory wrote:
Two weeks with no objections, so here is the full patchset. Several of the tests should be redundant now, but I left them in with updated rules anyway, just in case.
Thanks - patches look good to me. I made a couple of comments on them.
What we also need to figure out is the upgrade path here. There are packages in the Arch repos with files in /lib/... relying on the symlink at the moment. They will create conflicts after these patches (which is fine), but we need to make sure they will upgrade fine to the fixed versions. (They might already - but needs tested.)
@Dan, @Dave (or anyone else...): Do you intend to comment on this proposal? Even an "ack" would be appreciated here.
As I mentioned to you on IRC yesterday, I'm still concerned that this is going to break my install pretty hard with /bin and /sbin symlinked to usr/bin. I've had this running for the past year or so without too many problems.
For reference: https://mailman.archlinux.org/pipermail/pacman-dev/2013-April/017001.html
Also available on my no-symlinks branch for those interested.
Andrew Gregory (8): update tests for symlink support removal alpm_filelist: remove resolved_path conflict.c: do not ignore symlink<->dir conflicts conflict.c: use real path for filesystem checks extract_single_file: consolidate symlink cases unlink_file: treat symlinks like normal files query_fileowner: remove symlink support improve dir->file transition conflict resolution
lib/libalpm/add.c | 33 ++--- lib/libalpm/alpm.h | 3 +- lib/libalpm/conflict.c | 192 +++++++++++++----------------- lib/libalpm/filelist.c | 225 ++--------------------------------- lib/libalpm/filelist.h | 4 - lib/libalpm/package.c | 22 +--- lib/libalpm/remove.c | 11 +- src/common/util-common.c | 43 +++++++ src/common/util-common.h | 1 + src/pacman/query.c | 90 +++----------- test/pacman/tests/fileconflict007.py | 1 + test/pacman/tests/fileconflict013.py | 5 +- test/pacman/tests/fileconflict022.py | 1 - test/pacman/tests/fileconflict023.py | 7 +- test/pacman/tests/fileconflict025.py | 8 +- test/pacman/tests/fileconflict030.py | 17 +++ test/pacman/tests/symlink001.py | 9 +- test/pacman/tests/sync700.py | 12 +- test/pacman/tests/sync701.py | 11 +- test/pacman/tests/sync702.py | 11 +- 20 files changed, 217 insertions(+), 489 deletions(-) create mode 100644 test/pacman/tests/fileconflict030.py
On 29/04/13 02:33, Dave Reisner wrote:
On Mon, Apr 29, 2013 at 12:41:50AM +1000, Allan McRae wrote:
On 27/04/13 10:00, Andrew Gregory wrote:
Two weeks with no objections, so here is the full patchset. Several of the tests should be redundant now, but I left them in with updated rules anyway, just in case.
Thanks - patches look good to me. I made a couple of comments on them.
What we also need to figure out is the upgrade path here. There are packages in the Arch repos with files in /lib/... relying on the symlink at the moment. They will create conflicts after these patches (which is fine), but we need to make sure they will upgrade fine to the fixed versions. (They might already - but needs tested.)
@Dan, @Dave (or anyone else...): Do you intend to comment on this proposal? Even an "ack" would be appreciated here.
As I mentioned to you on IRC yesterday, I'm still concerned that this is going to break my install pretty hard with /bin and /sbin symlinked to usr/bin. I've had this running for the past year or so without too many problems.
Is that an objection? Every time there is a package in Arch with a file in /usr/bin and a symlink in /bin your system gets "broken". I'll also point out that you could only do this for that period of time because you were using the git version of pacman. Anyway, I'm guessing there are two problems here: 1) Your local database does not match what is on your filesystem. As our conflict checking assumes the local db to be right, this will probably miss conflicts. So we need to correct the local db for anyone who is currently using this "feature". This falls into the category of figuring out an upgrade path. 2) When the local db is right, programs with files in /bin will cause conflicts until Arch officially does that set up. That I do not consider a pacman problem and you will just need to add them to your rebuild list for this setup. I'd say this is not a pacman issue. Allan
On 29/04/13 00:59, Andrew Gregory wrote:
On 04/29/13 at 12:33am, Allan McRae wrote:
On 27/04/13 10:00, Andrew Gregory wrote:
In the event of a symlink<->dir transition, the paths for the incoming package need to be resolved to match the filelist for installed packages.
Can you explain this to me? Every time I think I understand, I don't...
e.g.
outdated local package: /usr/foo -> /usr/bar/ /usr/bar/baz
update sync package: /usr/foo/baz
So /usr/foo/baz is a new file so it is tested for presence in the filesystem. But /usr/foo/baz "exists" on the filesystem as /usr/bar/baz.
From what I understand, you resolve the package file "/usr/foo/baz" to "/usr/bar/baz" and note that it is in the old package so not a conflict. Which feels wrong to me...
Why would that be wrong? /usr/foo itself isn't a conflict and both /usr/foo and /usr/bar/baz will both be removed/overwritten in the upgrade process so /usr/foo/baz should be fine, should it not?
In fact, we already do what you outlined (see the code below), this patch just does the resolution upfront so the real path can be used for the rest of the checks as well (ie "changing hands" from one package to another).
My point is, why do we need to do that at all? Using my example again: outdated local package: /usr/foo -> /usr/bar/ /usr/bar/baz update sync package: /usr/foo/baz Initially /usr/foo/baz is flagged as a potential conflict. But because the realpath of /usr/foo/ is not /usr/foo, we know there must be a symlink to a directory somewhere in that path. So, we know the symlink is being replaced with a directory and either: 1) it can not be and has already been flagged as a conflict or 2) all files in this new directory are new and can not be conflicts. Allan
On Mon, Apr 29, 2013 at 10:32:49AM +1000, Allan McRae wrote:
On 29/04/13 02:33, Dave Reisner wrote:
On Mon, Apr 29, 2013 at 12:41:50AM +1000, Allan McRae wrote:
On 27/04/13 10:00, Andrew Gregory wrote:
Two weeks with no objections, so here is the full patchset. Several of the tests should be redundant now, but I left them in with updated rules anyway, just in case.
Thanks - patches look good to me. I made a couple of comments on them.
What we also need to figure out is the upgrade path here. There are packages in the Arch repos with files in /lib/... relying on the symlink at the moment. They will create conflicts after these patches (which is fine), but we need to make sure they will upgrade fine to the fixed versions. (They might already - but needs tested.)
@Dan, @Dave (or anyone else...): Do you intend to comment on this proposal? Even an "ack" would be appreciated here.
As I mentioned to you on IRC yesterday, I'm still concerned that this is going to break my install pretty hard with /bin and /sbin symlinked to usr/bin. I've had this running for the past year or so without too many problems.
Is that an objection? Every time there is a package in Arch with a file in /usr/bin and a symlink in /bin your system gets "broken". I'll also point out that you could only do this for that period of time because you were using the git version of pacman.
Ok, that's fair. I suppose it's not really an objection -- I realize that I'm the unique snowflake here and simplifying the code like this is preferrable.
Anyway, I'm guessing there are two problems here:
1) Your local database does not match what is on your filesystem. As our conflict checking assumes the local db to be right, this will probably miss conflicts. So we need to correct the local db for anyone who is currently using this "feature". This falls into the category of figuring out an upgrade path.
2) When the local db is right, programs with files in /bin will cause conflicts until Arch officially does that set up. That I do not consider a pacman problem and you will just need to add them to your rebuild list for this setup. I'd say this is not a pacman issue.
Yep, I agree with this. So, +1 from me overall. d
Signed-off-by: Andrew Gregory
After the initial checks, we either use the path as a directory and have
to append the trailing slash anyway or use it as a file in which case
the trailing slash should be excluded.
Signed-off-by: Andrew Gregory
Signed-off-by: Andrew Gregory
Signed-off-by: Andrew Gregory
On 11/05/13 12:21, Andrew Gregory wrote:
Signed-off-by: Andrew Gregory
--- Note the newly modified test at the bottom.
lib/libalpm/add.c | 56 +++++++++++++---------------------------- test/pacman/tests/upgrade045.py | 2 +- 2 files changed, 18 insertions(+), 40 deletions(-)
diff --git a/lib/libalpm/add.c b/lib/libalpm/add.c index 3ef81e3..49f36ce 100644 --- a/lib/libalpm/add.c +++ b/lib/libalpm/add.c @@ -197,30 +197,25 @@ static int extract_single_file(alpm_handle_t *handle, struct archive *archive, * to get 'right'. Here are the possibilities, with the filesystem * on the left and the package on the top: * (F=file, N=node, S=symlink, D=dir) - * | F/N | S | D - * non-existent | 1 | 2 | 3 - * F/N | 4 | 5 | 6 - * S | 7 | 8 | 9 - * D | 10 | 11 | 12 + * | F/N | D + * non-existent | 1 | 2 + * F/N | 3 | 4 + * D | 5 | 6 * - * 1,2,3- extract, no magic necessary. lstat (_alpm_lstat) will fail here. - * 4,5,6,7,8- conflict checks should have caught this. either overwrite + * 1,2- extract, no magic necessary. lstat (_alpm_lstat) will fail here. + * 3,4- conflict checks should have caught this. either overwrite * or backup the file. - * 9- follow the symlink, hopefully it is a directory, check it. - * 10- file replacing directory- don't allow it. - * 11- don't extract symlink- a dir exists here. we don't want links to - * links, etc. - * 12- skip extraction, dir already exists. + * 5- file replacing directory- don't allow it. + * 6- skip extraction, dir already exists. */
- /* do both a lstat and a stat, so we can see what symlinks point to */ - struct stat lsbuf, sbuf; - if(_alpm_lstat(filename, &lsbuf) != 0 || stat(filename, &sbuf) != 0) { - /* cases 1,2,3: couldn't stat an existing file, skip all backup checks */ + struct stat lsbuf; + if(_alpm_lstat(filename, &lsbuf) != 0) { + /* cases 1,2: file doesn't exist, skip all backup checks */ } else { if(S_ISDIR(lsbuf.st_mode)) { if(S_ISDIR(entrymode)) { - /* case 12: existing dir, ignore it */ + /* case 6: existing dir, ignore it */ if(lsbuf.st_mode != entrymode) { /* if filesystem perms are different than pkg perms, warn user */ mode_t mask = 07777; @@ -237,33 +232,18 @@ static int extract_single_file(alpm_handle_t *handle, struct archive *archive, archive_read_data_skip(archive); return 0; } else { - /* case 10/11: trying to overwrite dir with file/symlink, don't allow it */ + /* case 5: trying to overwrite dir with file, don't allow it */ _alpm_log(handle, ALPM_LOG_ERROR, _("extract: not overwriting dir with file %s\n"), filename); archive_read_data_skip(archive); return 1; } - } else if(S_ISLNK(lsbuf.st_mode) && S_ISDIR(entrymode)) { - /* case 9: existing symlink, dir in package */ - if(S_ISDIR(sbuf.st_mode)) { - /* the symlink on FS is to a directory, so we'll use it */ - _alpm_log(handle, ALPM_LOG_DEBUG, "extract: skipping symlink overwrite of %s\n", - filename); - archive_read_data_skip(archive); - return 0; - } else { - /* this is BAD. symlink was not to a directory */ - _alpm_log(handle, ALPM_LOG_ERROR, _("extract: symlink %s does not point to dir\n"), - filename); - archive_read_data_skip(archive); - return 1; - } - } else if(S_ISREG(lsbuf.st_mode) && S_ISDIR(entrymode)) { - /* case 6: trying to overwrite file with dir */ + } else if(S_ISDIR(entrymode)) { + /* case 4: trying to overwrite file with dir */ _alpm_log(handle, ALPM_LOG_DEBUG, "extract: overwriting file with dir %s\n", filename); - } else if(S_ISREG(entrymode)) { - /* case 4,7: */ + } else { + /* case 3: */ /* if file is in NoUpgrade, don't touch it */ if(alpm_list_find(handle->noupgrade, entryname, _alpm_fnmatch)) { notouch = 1; @@ -288,8 +268,6 @@ static int extract_single_file(alpm_handle_t *handle, struct archive *archive, } } } - /* else if(S_ISLNK(entrymode)) */ - /* case 5,8: don't need to do anything special */ }
/* we need access to the original entryname later after calls to diff --git a/test/pacman/tests/upgrade045.py b/test/pacman/tests/upgrade045.py index 3b7f476..e165ba3 100644 --- a/test/pacman/tests/upgrade045.py +++ b/test/pacman/tests/upgrade045.py @@ -14,4 +14,4 @@
self.addrule("PACMAN_RETCODE=0") self.addrule("PKG_VERSION=foo|1.0-2") -self.addrule("FILE_EXIST=etc/foo.cfg") +self.addrule("LINK_EXIST=etc/foo.cfg")
Looks good. For that pactest, add a FILE_EXISTS=/etc/foo.cfg.pacnew too. Also, in that pactest: p1.files = ["etc/foo.cfg*"] what is the "*" for there?
On 27/04/13 10:00, Andrew Gregory wrote:
Two weeks with no objections, so here is the full patchset. Several of the tests should be redundant now, but I left them in with updated rules anyway, just in case.
Also available on my no-symlinks branch for those interested.
Andrew Gregory (8): update tests for symlink support removal alpm_filelist: remove resolved_path conflict.c: do not ignore symlink<->dir conflicts conflict.c: use real path for filesystem checks extract_single_file: consolidate symlink cases unlink_file: treat symlinks like normal files query_fileowner: remove symlink support improve dir->file transition conflict resolution
Finally got through the last of these I had in my review queue. As it stands, you symlinks branch is good to go. However, there are two things I am considering before merging this: 1) Arch will finish the /usr merge in the near future (weeks, not months). As a reasonable proportion of people running pacman-git have done silly things like symlink /bin -> usr/bin already and thus need the old behaviour, I think we should merge after the transition is done. (One person is a reasonable portion!). 2) We need to figure out how to update peoples dbs where they have paths that assume directories which are really symlinks. Attached is a really bad script to list packages with issues. I'm sure this can be improved... The question remains, do we want to encourage a "pacman-db-upgrade" run here, or something else? Unlike the last db-upgrade, we can not detect whether this needs to be run or not - so I don't think that is the way to go. Allan
participants (4)
-
Allan McRae
-
Andrew Gregory
-
Christian Hesse
-
Dave Reisner