[pacman-dev] [PATCH] repo-add; add option to remove existing package files from disk
From: Phillip Smith <fukawi2@gmail.com> when maintaining a custom repo, often it is undesirable to retain older versions of packages. this patch adds the --remove option to remove the current package file from disk before adding the new one to the database. Signed-off-by: Phillip Smith <fukawi2@gmail.com> --- scripts/repo-add.sh.in | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 4470dd0..2fdb165 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -31,6 +31,7 @@ declare -r confdir='@sysconfdir@' QUIET=0 DELTA=0 ONLYADDNEW=0 +RMEXISTING=0 WITHFILES=0 SIGN=0 VERIFY=0 @@ -58,6 +59,7 @@ Multiple packages to add can be specified on the command line.\n")" printf -- "$(gettext "Options:\n")" printf -- "$(gettext " -d, --delta generate and add delta for package update\n")" printf -- "$(gettext " -n, --new only add packages that are not already in the database\n")" + printf -- "$(gettext " -R, --remove remove any existing packages with the same name (eg, an older version)\n")" printf -- "$(gettext " -f, --files update database's file list\n")" elif [[ $cmd == "repo-remove" ]] ; then printf -- "$(gettext "Usage: repo-remove [options] <path-to-db> <packagename|delta> ...\n")" @@ -304,6 +306,15 @@ db_write_entry() { local oldfilename=$(grep -A1 FILENAME "$pkgentry/desc" | tail -n1) local oldfile="$(dirname "$1")/$oldfilename" fi + elif (( RMEXISTING )); then + # only remove existing package if we're not doing deltas + pkgentry=$(find_pkgentry "$pkgname") + if [[ -n $pkgentry ]]; then + local oldfilename=$(grep -A1 FILENAME "$pkgentry/desc" | tail -n1) + local oldfile="$(dirname "$1")/$oldfilename" + warning "$(gettext "Removed existing file '%s'")" "$oldfilename" + rm -f $oldfile + fi fi fi @@ -611,6 +622,7 @@ while (( $# )); do -q|--quiet) QUIET=1;; -d|--delta) DELTA=1;; -n|--new) ONLYADDNEW=1;; + -R|--remove) RMEXISTING=1;; -f|--files) WITHFILES=1;; --nocolor) USE_COLOR='n';; -s|--sign) -- 1.8.2.3
On Thu, May 30, 2013 at 10:11:49AM +1000, Phillip Smith wrote:
From: Phillip Smith <fukawi2@gmail.com>
when maintaining a custom repo, often it is undesirable to retain older versions of packages. this patch adds the --remove option to remove the current package file from disk before adding the new one to the database.
You're probably going to get a lot more reliable cleaning using paccache.
Signed-off-by: Phillip Smith <fukawi2@gmail.com>
-- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF
On 30 May 2013 11:06, William Giokas <1007380@gmail.com> wrote:
You're probably going to get a lot more reliable cleaning using paccache.
If I'm not mistaken, paccache cleans pacman's local cache (ie, /var/cache/pacman/pkg). This patch it for the repository on the server. paccache certainly doesn't appear to achieve the same goal. In the example below, I would expect the pkgrel -1, -3, -4 of alsa-utils and libassuan-2.0.3-1-x86_64.pkg.tar.xz to be removed. ict-phil-pc /tmp/testrmexisting # ls -1 alsa-utils-1.0.26-1-x86_64.pkg.tar.xz alsa-utils-1.0.27-3-x86_64.pkg.tar.xz alsa-utils-1.0.27-4-x86_64.pkg.tar.xz alsa-utils-1.0.27-5-x86_64.pkg.tar.xz libassuan-2.0.3-1-x86_64.pkg.tar.xz libassuan-2.1.0-1-x86_64.pkg.tar.xz test.db test.db.tar.gz test.db.tar.gz.old ict-phil-pc /tmp/testrmexisting # paccache -d -k1 ./ ==> no candidate packages found for pruning
On 30 May 2013 11:18, Phillip Smith <lists@fukawi2.nl> wrote:
paccache certainly doesn't appear to achieve the same goal. In the example below, I would expect the pkgrel -1, -3, -4 of alsa-utils and libassuan-2.0.3-1-x86_64.pkg.tar.xz to be removed.
Apologies, I was misusing it. I needed the -c option. Still, this patch enables it to be done in one action when adding the new package(s), and paccache could be run on a schedule to cleanup any mishaps.
On Thu, May 30, 2013 at 11:28:18AM +1000, Phillip Smith wrote:
On 30 May 2013 11:18, Phillip Smith <lists@fukawi2.nl> wrote:
paccache certainly doesn't appear to achieve the same goal. In the example below, I would expect the pkgrel -1, -3, -4 of alsa-utils and libassuan-2.0.3-1-x86_64.pkg.tar.xz to be removed.
Apologies, I was misusing it. I needed the -c option.
Still, this patch enables it to be done in one action when adding the new package(s), and paccache could be run on a schedule to cleanup any mishaps.
Honestly, this is when I would say to call paccache in your repo-add function. As there is no separation between pacman and pacman-contrib anymore, it doesn't need any extra deps. Then things could still be done in one step while not duplicating code and introducing possible bugs. Thanks, -- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF
On 30 May 2013 11:32, William Giokas <1007380@gmail.com> wrote:
Honestly, this is when I would say to call paccache in your repo-add function. As there is no separation between pacman and pacman-contrib anymore, it doesn't need any extra deps. Then things could still be done in one step while not duplicating code and introducing possible bugs.
Ah, I understand. I thought you meant "just use paccache and forget the patch". Using paccache instead does sound better; I will amend and resubmit. Thanks.
On 30/05/13 11:32, William Giokas wrote:
On Thu, May 30, 2013 at 11:28:18AM +1000, Phillip Smith wrote:
On 30 May 2013 11:18, Phillip Smith <lists@fukawi2.nl> wrote:
paccache certainly doesn't appear to achieve the same goal. In the example below, I would expect the pkgrel -1, -3, -4 of alsa-utils and libassuan-2.0.3-1-x86_64.pkg.tar.xz to be removed.
Apologies, I was misusing it. I needed the -c option.
Still, this patch enables it to be done in one action when adding the new package(s), and paccache could be run on a schedule to cleanup any mishaps.
Honestly, this is when I would say to call paccache in your repo-add function. As there is no separation between pacman and pacman-contrib anymore, it doesn't need any extra deps. Then things could still be done in one step while not duplicating code and introducing possible bugs.
I'm actually leaning towards liking this patch (but have a fix that is needed...). Currently repo management is a very tedious process. This alleviates some of the strain. If you look at it, there is no code duplication. All it does is delete the current version from the repo directory before it updates to the new package. It won't clean up older packages than that. paccache is a very much more generic cache cleaner. This is purely repo management and I think belongs in repo-add. Does that make sense? Allan
On Thu, May 30, 2013 at 11:46:16AM +1000, Allan McRae wrote:
On 30/05/13 11:32, William Giokas wrote:
Honestly, this is when I would say to call paccache in your repo-add function. As there is no separation between pacman and pacman-contrib anymore, it doesn't need any extra deps. Then things could still be done in one step while not duplicating code and introducing possible bugs.
I'm actually leaning towards liking this patch (but have a fix that is needed...). Currently repo management is a very tedious process. This alleviates some of the strain.
Makes sense. I have no problems with it, I just think that if we already have a tool to do this and more, why not use that, but something simple works.
If you look at it, there is no code duplication. All it does is delete the current version from the repo directory before it updates to the new package. It won't clean up older packages than that.
Yes, but if this is going to expand, then it may become more and more similar to paccache.
paccache is a very much more generic cache cleaner. This is purely repo management and I think belongs in repo-add.
Does that make sense?
Makes complete sense. Just think that it's good to look at alternatives that we already have.
Allan
One more thing. I don't want repo-add depending on stuff in contrib. repo-add gets installed with "make install" but contrib does not.
Okay. I understand.
Allan
-- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF
On 30/05/13 11:32, William Giokas wrote:
On Thu, May 30, 2013 at 11:28:18AM +1000, Phillip Smith wrote:
On 30 May 2013 11:18, Phillip Smith <lists@fukawi2.nl> wrote:
paccache certainly doesn't appear to achieve the same goal. In the example below, I would expect the pkgrel -1, -3, -4 of alsa-utils and libassuan-2.0.3-1-x86_64.pkg.tar.xz to be removed.
Apologies, I was misusing it. I needed the -c option.
Still, this patch enables it to be done in one action when adding the new package(s), and paccache could be run on a schedule to cleanup any mishaps.
Honestly, this is when I would say to call paccache in your repo-add function. As there is no separation between pacman and pacman-contrib anymore, it doesn't need any extra deps. Then things could still be done in one step while not duplicating code and introducing possible bugs.
One more thing. I don't want repo-add depending on stuff in contrib. repo-add gets installed with "make install" but contrib does not. Allan
On 30 May 2013 11:55, Allan McRae <allan@archlinux.org> wrote:
One more thing. I don't want repo-add depending on stuff in contrib. repo-add gets installed with "make install" but contrib does not.
And one more I thought of while thinking about this more. paccache only keeps the _newest_ package it finds. Do we really want to assume that the user is adding the very newest package in the directory and have paccache blat the package they are trying to add? It's probably reasonable in 99.9% of cases, but there's that corner case where unexpected behaviour will result.
On Thu, May 30, 2013 at 12:02:56PM +1000, Phillip Smith wrote:
On 30 May 2013 11:55, Allan McRae <allan@archlinux.org> wrote:
One more thing. I don't want repo-add depending on stuff in contrib. repo-add gets installed with "make install" but contrib does not.
And one more I thought of while thinking about this more. paccache only keeps the _newest_ package it finds. Do we really want to assume that the user is adding the very newest package in the directory and have paccache blat the package they are trying to add?
It's probably reasonable in 99.9% of cases, but there's that corner case where unexpected behaviour will result.
Honestly, something I thought of, maybe have it move the package, instead of deleting it. If users want things to be deleted they can point the location to /dev/null or something. That way if something is changed and you want to go back, you can just move around some packages instead of having to rebuild. Thanks, -- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF
On Thu, May 30, 2013 at 10:11:49AM +1000, Phillip Smith wrote:
From: Phillip Smith <fukawi2@gmail.com>
when maintaining a custom repo, often it is undesirable to retain older versions of packages. this patch adds the --remove option to remove the current package file from disk before adding the new one to the database.
Signed-off-by: Phillip Smith <fukawi2@gmail.com> --- scripts/repo-add.sh.in | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 4470dd0..2fdb165 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -31,6 +31,7 @@ declare -r confdir='@sysconfdir@' QUIET=0 DELTA=0 ONLYADDNEW=0 +RMEXISTING=0 WITHFILES=0 SIGN=0 VERIFY=0 @@ -58,6 +59,7 @@ Multiple packages to add can be specified on the command line.\n")" printf -- "$(gettext "Options:\n")" printf -- "$(gettext " -d, --delta generate and add delta for package update\n")" printf -- "$(gettext " -n, --new only add packages that are not already in the database\n")" + printf -- "$(gettext " -R, --remove remove any existing packages with the same name (eg, an older version)\n")" printf -- "$(gettext " -f, --files update database's file list\n")" elif [[ $cmd == "repo-remove" ]] ; then printf -- "$(gettext "Usage: repo-remove [options] <path-to-db> <packagename|delta> ...\n")" @@ -304,6 +306,15 @@ db_write_entry() { local oldfilename=$(grep -A1 FILENAME "$pkgentry/desc" | tail -n1) local oldfile="$(dirname "$1")/$oldfilename" fi + elif (( RMEXISTING )); then + # only remove existing package if we're not doing deltas + pkgentry=$(find_pkgentry "$pkgname") + if [[ -n $pkgentry ]]; then + local oldfilename=$(grep -A1 FILENAME "$pkgentry/desc" | tail -n1)
Just one thing. Replace this with: local oldfilename="$(awk '/%FILENAME%/ {getline; print}')" Yeah, the quotes aren't that important, but this way you save spawning a tail process when you could just do it with this. (I just had to do this to get pkgbase entries from db's in one of my pet scripts, so yeah)
+ local oldfile="$(dirname "$1")/$oldfilename" + warning "$(gettext "Removed existing file '%s'")" "$oldfilename" + rm -f $oldfile + fi fi fi
@@ -611,6 +622,7 @@ while (( $# )); do -q|--quiet) QUIET=1;; -d|--delta) DELTA=1;; -n|--new) ONLYADDNEW=1;; + -R|--remove) RMEXISTING=1;; -f|--files) WITHFILES=1;; --nocolor) USE_COLOR='n';; -s|--sign) -- 1.8.2.3
-- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF
On 30/05/13 12:13, William Giokas wrote:
On Thu, May 30, 2013 at 10:11:49AM +1000, Phillip Smith wrote:
From: Phillip Smith <fukawi2@gmail.com>
when maintaining a custom repo, often it is undesirable to retain older versions of packages. this patch adds the --remove option to remove the current package file from disk before adding the new one to the database.
Signed-off-by: Phillip Smith <fukawi2@gmail.com> --- scripts/repo-add.sh.in | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 4470dd0..2fdb165 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -31,6 +31,7 @@ declare -r confdir='@sysconfdir@' QUIET=0 DELTA=0 ONLYADDNEW=0 +RMEXISTING=0 WITHFILES=0 SIGN=0 VERIFY=0 @@ -58,6 +59,7 @@ Multiple packages to add can be specified on the command line.\n")" printf -- "$(gettext "Options:\n")" printf -- "$(gettext " -d, --delta generate and add delta for package update\n")" printf -- "$(gettext " -n, --new only add packages that are not already in the database\n")" + printf -- "$(gettext " -R, --remove remove any existing packages with the same name (eg, an older version)\n")" printf -- "$(gettext " -f, --files update database's file list\n")" elif [[ $cmd == "repo-remove" ]] ; then printf -- "$(gettext "Usage: repo-remove [options] <path-to-db> <packagename|delta> ...\n")" @@ -304,6 +306,15 @@ db_write_entry() { local oldfilename=$(grep -A1 FILENAME "$pkgentry/desc" | tail -n1) local oldfile="$(dirname "$1")/$oldfilename" fi + elif (( RMEXISTING )); then + # only remove existing package if we're not doing deltas + pkgentry=$(find_pkgentry "$pkgname") + if [[ -n $pkgentry ]]; then + local oldfilename=$(grep -A1 FILENAME "$pkgentry/desc" | tail -n1)
Just one thing. Replace this with:
local oldfilename="$(awk '/%FILENAME%/ {getline; print}')"
Yeah, the quotes aren't that important, but this way you save spawning a tail process when you could just do it with this. (I just had to do this to get pkgbase entries from db's in one of my pet scripts, so yeah)
Note that line is cpoy-paste from just above. My bet is a bunch of such improvements could be made.
+ local oldfile="$(dirname "$1")/$oldfilename" + warning "$(gettext "Removed existing file '%s'")" "$oldfilename" + rm -f $oldfile
Remember the .sig file too!
+ fi fi fi
@@ -611,6 +622,7 @@ while (( $# )); do -q|--quiet) QUIET=1;; -d|--delta) DELTA=1;; -n|--new) ONLYADDNEW=1;; + -R|--remove) RMEXISTING=1;; -f|--files) WITHFILES=1;; --nocolor) USE_COLOR='n';; -s|--sign) -- 1.8.2.3
On Thu, May 30, 2013 at 12:23:30PM +1000, Allan McRae wrote:
On 30/05/13 12:13, William Giokas wrote:
Just one thing. Replace this with:
local oldfilename="$(awk '/%FILENAME%/ {getline; print}')"
Yeah, the quotes aren't that important, but this way you save spawning a tail process when you could just do it with this. (I just had to do this to get pkgbase entries from db's in one of my pet scripts, so yeah)
Note that line is cpoy-paste from just above. My bet is a bunch of such improvements could be made.
I know how I'm going to spend the next few days, then. -- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF
On Wed, May 29, 2013 at 09:13:26PM -0500, William Giokas wrote:
On Thu, May 30, 2013 at 10:11:49AM +1000, Phillip Smith wrote:
From: Phillip Smith <fukawi2@gmail.com>
when maintaining a custom repo, often it is undesirable to retain older versions of packages. this patch adds the --remove option to remove the current package file from disk before adding the new one to the database.
Signed-off-by: Phillip Smith <fukawi2@gmail.com> --- scripts/repo-add.sh.in | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 4470dd0..2fdb165 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -31,6 +31,7 @@ declare -r confdir='@sysconfdir@' QUIET=0 DELTA=0 ONLYADDNEW=0 +RMEXISTING=0 WITHFILES=0 SIGN=0 VERIFY=0 @@ -58,6 +59,7 @@ Multiple packages to add can be specified on the command line.\n")" printf -- "$(gettext "Options:\n")" printf -- "$(gettext " -d, --delta generate and add delta for package update\n")" printf -- "$(gettext " -n, --new only add packages that are not already in the database\n")" + printf -- "$(gettext " -R, --remove remove any existing packages with the same name (eg, an older version)\n")" printf -- "$(gettext " -f, --files update database's file list\n")" elif [[ $cmd == "repo-remove" ]] ; then printf -- "$(gettext "Usage: repo-remove [options] <path-to-db> <packagename|delta> ...\n")" @@ -304,6 +306,15 @@ db_write_entry() { local oldfilename=$(grep -A1 FILENAME "$pkgentry/desc" | tail -n1) local oldfile="$(dirname "$1")/$oldfilename" fi + elif (( RMEXISTING )); then + # only remove existing package if we're not doing deltas + pkgentry=$(find_pkgentry "$pkgname") + if [[ -n $pkgentry ]]; then + local oldfilename=$(grep -A1 FILENAME "$pkgentry/desc" | tail -n1)
Just one thing. Replace this with:
local oldfilename="$(awk '/%FILENAME%/ {getline; print}')"
If you're going to go through the effort of "optimizing" this, then there's 2 things wrong here: - you're reading the entire output, when you only care a single line - the execution time for sed is faster than awk in 99% of cases so... oldfilename=$(sed -n '/^%FILENAME%$/ {n;p;q;}' "$pkgentry/desc") I suppose if you wanted to stick with awk... oldfilename=$(awk '$0 == "%FILENAME%" { getline; print; exit; }' ...)
Yeah, the quotes aren't that important, but this way you save spawning a tail process when you could just do it with this. (I just had to do this to get pkgbase entries from db's in one of my pet scripts, so yeah)
+ local oldfile="$(dirname "$1")/$oldfilename" + warning "$(gettext "Removed existing file '%s'")" "$oldfilename" + rm -f $oldfile + fi fi fi
@@ -611,6 +622,7 @@ while (( $# )); do -q|--quiet) QUIET=1;; -d|--delta) DELTA=1;; -n|--new) ONLYADDNEW=1;; + -R|--remove) RMEXISTING=1;; -f|--files) WITHFILES=1;; --nocolor) USE_COLOR='n';; -s|--sign) -- 1.8.2.3
-- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF
On Thu, May 30, 2013 at 10:11:49AM +1000, Phillip Smith wrote:
From: Phillip Smith <fukawi2@gmail.com>
when maintaining a custom repo, often it is undesirable to retain older versions of packages. this patch adds the --remove option to remove the current package file from disk before adding the new one to the database.
Sorry if this isn't the most appropriate place to share this, but its somewhat relevant to the problem. I've actually been developing a 3rd-party tool to replace repo-add for database management: https://github.com/vodik/repose It works different from repo-add - it tries to keep a directory and a database in sync with each other. Filesystem changes propagate to the database instead of having to be manually specified. For example: repose -Uscf /srv/http/packages/vodik.db Repose reads the existing database into memory, automatically scan /srv/http/packages to determine which packages are newest, add them into the database. It has various abilities to clean up after itself and delete old packages. If a package has been removed from the filesystem the update operation will also drop it from the database, no need to manually call repo-remove. With this tool all I need to do is dump all the packages into one directory and when I'm done run repose once. The problem I had with repo-add is I had to remember the packages which got updated to properly update the database. Often I just ended up doing `repo-add vodik.db.tar.gz *.pkg.tar.xz`, which is expensive and sometimes does the wrong thing (replaces a new package with an older version). Repose is slightly faster then repo-add when generating a new database and substantially faster at updating and existing one. Most importantly it let me remove _a lot_ of logic from my package rebuild scripts.
On 02/06/13 07:49, Simon Gomizelj wrote:
he problem I had with repo-add is I had to remember the packages which got updated to properly update the database.
FYI: There is a bug report and draft patch adding a flag to makepkg specifying a repo that the newly built package should be added to.
On Sun, Jun 02, 2013 at 11:23:00AM +1000, Allan McRae wrote:
On 02/06/13 07:49, Simon Gomizelj wrote:
he problem I had with repo-add is I had to remember the packages which got updated to properly update the database.
FYI: There is a bug report and draft patch adding a flag to makepkg specifying a repo that the newly built package should be added to.
Yeah you pointed that out to me before on IRC (It was actually me, a _long_ time ago, when I was starting out. I forgot I did that: http://www.mail-archive.com/pacman-dev@archlinux.org/msg07369.html) That said, dropping a bunch of packages from a repo is also kinda tedious. I have delete the files and run repo-remove while repose will just do the right thing itself. I picked this project up as an excuse to learn how pacman's internals work, but stuck with it because of the performance and scriptablity improvements.
participants (5)
-
Allan McRae
-
Dave Reisner
-
Phillip Smith
-
Simon Gomizelj
-
William Giokas