[pacman-dev] [PATCH 0/5][RFC] Die delta, die!
Deltas are broken. So much so that I would strongly recommend never using a delta from a repo that you did not generate yourself. In short, we call "system(command)", with a command that includes the name of a delta file, and the name of the package file before and after applying the delta. The name of the delta and the package files is controlled by the information in the repo, and could contain a malicious command to be run as root. We could possibly work around this, but it is a very risky piece of code and I believe it would be very hard to fully secure. Instead, I propose to remove delta support completely. Allan McRae (5): Remove delta support from repo-add Remove pkgdelta Remove cleanupdelta Make pacman forget deltas exist Remove support for deltas from libalpm doc/Makefile.am | 2 - doc/index.asciidoc | 1 - doc/meson.build | 1 - doc/pacman.conf.5.asciidoc | 8 - doc/pkgdelta.8.asciidoc | 53 ------ doc/repo-add.8.asciidoc | 22 +-- lib/libalpm/Makefile.am | 1 - lib/libalpm/alpm.h | 53 ------ lib/libalpm/be_sync.c | 21 +-- lib/libalpm/delta.c | 361 ------------------------------------- lib/libalpm/delta.h | 33 ---- lib/libalpm/error.c | 5 - lib/libalpm/handle.c | 20 -- lib/libalpm/handle.h | 5 - lib/libalpm/meson.build | 1 - lib/libalpm/package.c | 14 -- lib/libalpm/package.h | 2 - lib/libalpm/po/POTFILES.in | 1 - lib/libalpm/sync.c | 207 +-------------------- meson.build | 8 - scripts/.gitignore | 1 - scripts/Makefile.am | 3 - scripts/meson.build | 1 - scripts/pkgdelta.sh.in | 234 ------------------------ scripts/po/POTFILES.in | 1 - scripts/repo-add.sh.in | 168 +---------------- src/pacman/callback.c | 19 -- src/pacman/conf.c | 25 --- src/pacman/conf.h | 1 - src/pacman/pacman-conf.c | 13 -- src/pacman/po/POTFILES.in | 1 - src/pacman/sync.c | 5 +- src/util/.gitignore | 2 - src/util/Makefile.am | 5 +- src/util/cleanupdelta.c | 135 -------------- src/util/meson.build | 1 - 36 files changed, 27 insertions(+), 1407 deletions(-) delete mode 100644 doc/pkgdelta.8.asciidoc delete mode 100644 lib/libalpm/delta.c delete mode 100644 lib/libalpm/delta.h delete mode 100644 scripts/pkgdelta.sh.in delete mode 100644 src/util/cleanupdelta.c -- 2.20.1
Signed-off-by: Allan McRae <allan@archlinux.org> --- doc/repo-add.8.asciidoc | 22 ++---- scripts/repo-add.sh.in | 168 ++-------------------------------------- 2 files changed, 15 insertions(+), 175 deletions(-) diff --git a/doc/repo-add.8.asciidoc b/doc/repo-add.8.asciidoc index 945fff75..8de4485b 100644 --- a/doc/repo-add.8.asciidoc +++ b/doc/repo-add.8.asciidoc @@ -8,27 +8,25 @@ repo-add - package database maintenance utility Synopsis -------- -'repo-add' [options] <path-to-db> <package|delta> [<package|delta> ...] +'repo-add' [options] <path-to-db> <package> [<package> ...] -'repo-remove' [options] <path-to-db> <packagename|delta> [<packagename|delta> ...] +'repo-remove' [options] <path-to-db> <packagename> [<packagename> ...] Description ----------- 'repo-add' and 'repo-remove' are two scripts to help build a package database for packages built with linkman:makepkg[8] and installed with linkman:pacman[8]. -They also handle package deltas produced by linkman:pkgdelta[8]. -'repo-add' will update a package database by reading a built package or package -delta file. Multiple packages and/or deltas to add can be specified on the -command line. +'repo-add' will update a package database by reading a built package file. +Multiple packages to add can be specified on the command line. If a matching ``.sig'' file is found alongside a package file, the signature will automatically be embedded into the database. -'repo-remove' will update a package database by removing the package name or -delta specified on the command line. Multiple packages and/or delta to remove -can be specified on the command line. +'repo-remove' will update a package database by removing the package name +specified on the command line. Multiple packages to remove can be specified +on the command line. A package database is a tar file, optionally compressed. Valid extensions are ``.db'' followed by an archive extension of ``.tar'', ``.tar.gz'', ``.tar.bz2'', @@ -64,10 +62,6 @@ Common Options repo-add Options ---------------- -*-d, \--delta*:: - Automatically generate and add a delta file between the old entry and the - new one, if the old package file is found next to the new one. - *-n, \--new*:: Only add packages that are not already in the database. Warnings will be printed upon detection of existing packages, but they will not be re-added. @@ -89,6 +83,6 @@ db.tar* extension), there is currently no additional benefit for the larger down See Also -------- -linkman:makepkg[8], linkman:pacman[8], linkman:pkgdelta[8] +linkman:makepkg[8], linkman:pacman[8] include::footer.asciidoc[] diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 21a5ea8e..57413df5 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -31,7 +31,6 @@ declare -r confdir='@sysconfdir@' LIBRARY=${LIBRARY:-'@libmakepkgdir@'} QUIET=0 -DELTA=0 ONLYADDNEW=0 RMEXISTING=0 SIGN=0 @@ -55,18 +54,17 @@ usage() { cmd=${0##*/} printf -- "%s (pacman) %s\n\n" "$cmd" "$myver" if [[ $cmd == "repo-add" ]] ; then - printf -- "$(gettext "Usage: repo-add [options] <path-to-db> <package|delta> ...\n")" + printf -- "$(gettext "Usage: repo-add [options] <path-to-db> <package> ...\n")" printf -- "\n" printf -- "$(gettext "\ repo-add will update a package database by reading a package file.\n\ Multiple packages to add can be specified on the command line.\n")" printf -- "\n" printf -- "$(gettext "Options:\n")" - printf -- "$(gettext " -d, --delta generate and add delta for package update\n")" printf -- "$(gettext " -n, --new only add packages that are not already in the database\n")" printf -- "$(gettext " -R, --remove remove old package file from disk after updating database\n")" elif [[ $cmd == "repo-remove" ]] ; then - printf -- "$(gettext "Usage: repo-remove [options] <path-to-db> <packagename|delta> ...\n")" + printf -- "$(gettext "Usage: repo-remove [options] <path-to-db> <packagename> ...\n")" printf -- "\n" printf -- "$(gettext "\ repo-remove will update a package database by removing the package name\n\ @@ -131,90 +129,6 @@ find_pkgentry() { return 1 } -# Get the package name from the delta filename -get_delta_pkgname() { - local tmp - - tmp=${1##*/} - echo ${tmp%-*-*_to*} -} - -# write a delta entry -# arg1 - path to delta file -db_write_delta() { - deltafile=$1 - pkgname=$(get_delta_pkgname "$deltafile") - - pkgentry=$(find_pkgentry "$pkgname") - if [[ -z $pkgentry ]]; then - error "$(gettext "No database entry for package '%s'.")" "$pkgname" - return 1 - fi - deltas=$pkgentry/deltas - if [[ ! -f $deltas ]]; then - echo -e "%DELTAS%" >"$deltas" - fi - # get md5sum and compressed size of package - md5sum=$(md5sum "$deltafile") - md5sum=${md5sum%% *} - csize=$(wc -c "$deltafile" | cut -d' ' -f1) - - oldfile=$(xdelta3 printhdr "$deltafile" | sed -n 's/XDELTA filename (source):\s\+\(\.*\)/\1/p') - newfile=$(xdelta3 printhdr "$deltafile" | sed -n 's/XDELTA filename (output):\s\+\(\.*\)/\1/p') - - if grep -q "$oldfile.*$newfile" "$deltas"; then - sed -i.backup "/$oldfile.*$newfile/d" "$deltas" && rm -f "$deltas.backup" - fi - msg2 "$(gettext "Adding 'deltas' entry : %s -> %s")" "$oldfile" "$newfile" - echo "${deltafile##*/} $md5sum $csize $oldfile $newfile" >> "$deltas" - - # copy updated deltas entry into "files" database - local filesentry=$(echo "$pkgentry" | sed 's/\(.*\)\/db\//\1\/files\//') - mkdir -p "$filesentry" - cp $deltas "$filesentry" - - return 0 -} # end db_write_delta - -# remove a delta entry -# arg1 - path to delta file -db_remove_delta() { - deltafile=$1 - filename=${deltafile##*/} - pkgname=$(get_delta_pkgname "$deltafile") - - pkgentry=$(find_pkgentry "$pkgname") - if [[ -z $pkgentry ]]; then - return 1 - fi - deltas=$pkgentry/deltas - if [[ ! -f $deltas ]]; then - return 1 - fi - if grep -q "$filename" "$deltas"; then - sed -i.backup "/$filename/d" "$deltas" && rm -f "$deltas.backup" - msg2 "$(gettext "Removing existing entry '%s'...")" "$filename" - # empty deltas file contains only "%DELTAS%" - if (( $(wc -l < "$deltas") == 1 )); then - msg2 "$(gettext "Removing empty deltas file...")" - rm "$deltas" - fi - - # copy updated deltas entry into "files" database - local filesentry=$(echo "$pkgentry" | sed 's/\(.*\)\/db\//\1\/files\//') - if [[ -f $deltas ]]; then - mkdir -p "$filesentry" - cp $deltas "$filesentry" - else - rm -f "$filesentry/deltas" - fi - - return 0 - fi - - return 1 -} # end db_remove_delta - check_gpg() { if ! type -p gpg >/dev/null; then error "$(gettext "Cannot find the gpg binary! Is GnuPG installed?")" @@ -233,30 +147,6 @@ check_gpg() { fi } -check_xdelta() { - local need_xdelta=0 - - if (( DELTA )); then - need_xdelta=1 - else - if [[ $cmd == "repo-add" ]]; then - for f in ${args[@]:1}; do - case $f in - *.delta) need_xdelta=1 ;; - *) ;; - esac - done - fi - fi - - if (( need_xdelta )); then - if ! type xdelta3 &>/dev/null; then - error "$(gettext "Cannot find the xdelta3 binary! Is xdelta3 installed?")" - exit 1 - fi - fi -} - # sign the package database once repackaged create_signature() { (( ! SIGN )) && return @@ -358,7 +248,7 @@ db_write_entry() { return 0 fi else - if (( DELTA || RMEXISTING )); then + if (( RMEXISTING )); then pkgentry=$(find_pkgentry "$pkgname") if [[ -n $pkgentry ]]; then local oldfilename="$(sed -n '/^%FILENAME%$/ {n;p;q;}' "$pkgentry/desc")" @@ -399,9 +289,6 @@ db_write_entry() { mkdir "$pkgname-$pkgver" pushd "$pkgname-$pkgver" >/dev/null - # restore an eventual deltas file - [[ -f ../$pkgname.deltas ]] && mv "../$pkgname.deltas" deltas - # create desc entry msg2 "$(gettext "Creating '%s' db entry...")" 'desc' { @@ -439,20 +326,6 @@ db_write_entry() { popd >/dev/null popd >/dev/null - # create a delta file - if (( DELTA )); then - if [[ -n $oldfilename ]]; then - if [[ -f $oldfile ]]; then - delta=$(pkgdelta -q "$oldfile" "$1") - if [[ -f $delta ]]; then - db_write_delta "$delta" - fi - else - warning "$(gettext "Old package file not found: %s")" "$oldfilename" - fi - fi - fi - # copy updated package entry into "files" database cp -a "$tmpdir/db/$pkgname-$pkgver" "$tmpdir/files/$pkgname-$pkgver" @@ -478,9 +351,7 @@ db_remove_entry() { local pkgentry=$(find_pkgentry "$pkgname") while [[ -n $pkgentry ]]; do notfound=0 - if [[ -f $pkgentry/deltas ]]; then - mv "$pkgentry/deltas" "$tmpdir/db/$pkgname.deltas" - fi + msg2 "$(gettext "Removing existing entry '%s'...")" \ "${pkgentry##*/}" rm -rf "$pkgentry" @@ -574,16 +445,6 @@ add() { return 1 fi - if [[ $1 = *-*-*_to_*-*-*.delta ]]; then - deltafile=$1 - msg "$(gettext "Adding delta '%s'")" "$deltafile" - if db_write_delta "$deltafile"; then - return 0 - else - return 1 - fi - fi - pkgfile=$1 if ! bsdtar -tqf "$pkgfile" .PKGINFO >/dev/null 2>&1; then error "$(gettext "'%s' is not a package file, skipping")" "$pkgfile" @@ -596,27 +457,15 @@ add() { } remove() { - if [[ $1 = *-*-*_to_*-*-*.delta ]]; then - deltafile=$1 - msg "$(gettext "Searching for delta '%s'...")" "$deltafile" - if db_remove_delta "$deltafile"; then - return 0 - else - error "$(gettext "Delta matching '%s' not found.")" "$deltafile" - return 1 - fi - fi - pkgname=$1 msg "$(gettext "Searching for package '%s'...")" "$pkgname" - if db_remove_entry "$pkgname"; then - rm -f "$tmpdir/db/$pkgname.deltas" - return 0 - else + if ! db_remove_entry "$pkgname"; then error "$(gettext "Package matching '%s' not found.")" "$pkgname" return 1 fi + + return 0 } rotate_db() { @@ -755,7 +604,6 @@ declare -a args while (( $# )); do case $1 in -q|--quiet) QUIET=1;; - -d|--delta) DELTA=1;; -n|--new) ONLYADDNEW=1;; -R|--remove) RMEXISTING=1;; --nocolor) USE_COLOR='n';; @@ -806,8 +654,6 @@ if (( SIGN || VERIFY )); then check_gpg fi -check_xdelta - if (( VERIFY && ${#args[@]} == 1 )); then for repo in "db" "files"; do dbfile=${repodir}/$REPO_DB_PREFIX.$repo.$REPO_DB_SUFFIX -- 2.20.1
Signed-off-by: Allan McRae <allan@archlinux.org> --- doc/Makefile.am | 2 - doc/index.asciidoc | 1 - doc/meson.build | 1 - doc/pkgdelta.8.asciidoc | 53 --------- scripts/.gitignore | 1 - scripts/Makefile.am | 3 - scripts/meson.build | 1 - scripts/pkgdelta.sh.in | 234 ---------------------------------------- scripts/po/POTFILES.in | 1 - 9 files changed, 297 deletions(-) delete mode 100644 doc/pkgdelta.8.asciidoc delete mode 100644 scripts/pkgdelta.sh.in diff --git a/doc/Makefile.am b/doc/Makefile.am index de87c409..f4b0b943 100644 --- a/doc/Makefile.am +++ b/doc/Makefile.am @@ -10,7 +10,6 @@ MANPAGES = \ makepkg-template.1 \ repo-add.8 \ vercmp.8 \ - pkgdelta.8 \ pacman-key.8 \ PKGBUILD.5 \ makepkg.conf.5 \ @@ -41,7 +40,6 @@ EXTRA_DIST = \ makepkg-template.1.asciidoc \ repo-add.8.asciidoc \ vercmp.8.asciidoc \ - pkgdelta.8.asciidoc \ pacman-key.8.asciidoc \ PKGBUILD.5.asciidoc \ PKGBUILD-example.txt \ diff --git a/doc/index.asciidoc b/doc/index.asciidoc index 35de0b90..9f9ab3a2 100644 --- a/doc/index.asciidoc +++ b/doc/index.asciidoc @@ -52,7 +52,6 @@ configuration files dealing with pacman. * linkman:pacman[8] * linkman:pacman-key[8] * linkman:pacman.conf[5] -* linkman:pkgdelta[8] * linkman:repo-add[8] * linkman:vercmp[8] diff --git a/doc/meson.build b/doc/meson.build index b7514ef7..f29aaf5f 100644 --- a/doc/meson.build +++ b/doc/meson.build @@ -5,7 +5,6 @@ manpages = [ { 'name': 'makepkg-template.1' }, { 'name': 'repo-add.8' }, { 'name': 'vercmp.8' }, - { 'name': 'pkgdelta.8' }, { 'name': 'pacman-key.8' }, { 'name': 'PKGBUILD.5', 'extra_depends' : [ 'PKGBUILD-example.txt' ] }, { 'name': 'makepkg.conf.5' }, diff --git a/doc/pkgdelta.8.asciidoc b/doc/pkgdelta.8.asciidoc deleted file mode 100644 index 412e607e..00000000 --- a/doc/pkgdelta.8.asciidoc +++ /dev/null @@ -1,53 +0,0 @@ -pkgdelta(8) -========= - -Name ----- -pkgdelta - package delta generation utility - - -Synopsis --------- -'pkgdelta' [options] <package1> <package2> - - -Description ------------ -'pkgdelta' is used to create package delta files between two versions of the -same package. These files are essentially binary patches. linkman:pacman[8] can -download deltas instead of full package upgrades, and use them with the -previous versions of packages (in the package cache) to synthesize the upgraded -version of the packages. This likely reduces download sizes for upgrades -significantly. - -'pkgdelta' requires linkman:xdelta3[1] to do its job. - - -Options -------- -*--max-delta-size <ratio>*:: - Only create delta files if the delta is smaller than ratio * package_size. - Possible values: 0.0 to 2.0. - Recommended values: 0.2 to 0.9. - Default value: 0.7 - -*--min-pkg-size <size>*:: - Minimal size of the package file in bytes to be considered for delta creation. - Default value: 1048576 bytes = 1 MiB. This may be any absolute size in bytes or - a human-readable value such as `4MiB` or `3.5MB`. - -*-q, \--quiet*:: - Be quiet. Do not output anything but warnings and errors. - - -Examples --------- - - $ pkgdelta libreoffice-3.3.2-1-x86_64.pkg.tar.xz libreoffice-3.3.2-2-x86_64.pkg.tar.xz - - -See Also --------- -linkman:pacman[8], linkman:xdelta3[1] - -include::footer.asciidoc[] diff --git a/scripts/.gitignore b/scripts/.gitignore index 3a5ac8a7..b8bb1f83 100644 --- a/scripts/.gitignore +++ b/scripts/.gitignore @@ -2,7 +2,6 @@ makepkg makepkg-template pacman-db-upgrade pacman-key -pkgdelta repo-add repo-elephant repo-remove diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 7fffd93b..6628e8cf 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -16,7 +16,6 @@ OURSCRIPTS = \ makepkg \ pacman-db-upgrade \ pacman-key \ - pkgdelta \ repo-add EXTRA_DIST = \ @@ -24,7 +23,6 @@ EXTRA_DIST = \ makepkg-template.pl.in \ pacman-db-upgrade.sh.in \ pacman-key.sh.in \ - pkgdelta.sh.in \ repo-add.sh.in \ wrapper.sh.in \ $(COMPLETION_DIST) \ @@ -137,7 +135,6 @@ WRAPPER = \ makepkg-wrapper \ pacman-db-upgrade-wrapper \ pacman-key-wrapper \ - pkgdelta-wrapper \ repo-add-wrapper COMPLETION_IN = \ diff --git a/scripts/meson.build b/scripts/meson.build index 344333be..db442973 100644 --- a/scripts/meson.build +++ b/scripts/meson.build @@ -2,7 +2,6 @@ wrapped_scripts = [ 'makepkg.sh.in', 'pacman-db-upgrade.sh.in', 'pacman-key.sh.in', - 'pkgdelta.sh.in', 'repo-add.sh.in' ] diff --git a/scripts/pkgdelta.sh.in b/scripts/pkgdelta.sh.in deleted file mode 100644 index b0b0b21f..00000000 --- a/scripts/pkgdelta.sh.in +++ /dev/null @@ -1,234 +0,0 @@ -#!/bin/bash -# -# pkgdelta - create delta files for use with pacman and repo-add -# @configure_input@ -# -# Copyright (c) 2009 Xavier Chantry <shiningxc@gmail.com> -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. -# - -# bash options -set -o errexit - -# gettext initialization -export TEXTDOMAIN='pacman-scripts' -export TEXTDOMAINDIR='@localedir@' - -declare -r myver='@PACKAGE_VERSION@' - -LIBRARY=${LIBRARY:-'@libmakepkgdir@'} - -# Import libmakepkg -source "$LIBRARY"/util/message.sh -source "$LIBRARY"/util/parseopts.sh - -# Options -QUIET=0 -USE_COLOR='y' - -# minimal of package before deltas are generated (bytes) -min_pkg_size=$((1024*1024)) - -# percent of new package above which the delta will be discarded -max_delta_size=70 - - -# ensure we have a sane umask set -umask 0022 - -# print usage instructions -usage() { - printf "pkgdelta (pacman) %s\n" "${myver}" - echo - printf -- "$(gettext "Usage: pkgdelta [options] <package1> <package2>\n")" - echo - printf -- "$(gettext "\ -pkgdelta will create a delta file between two packages.\n\ -This delta file can then be added to a database using repo-add.\n")" - echo - printf -- "$(gettext "Example: pkgdelta pacman-3.0.0.pkg.tar.gz pacman-3.0.1.pkg.tar.gz")\n" - echo - printf -- "$(gettext "Options:\n")" - printf -- "$(gettext " -q, --quiet minimize output\n")" - printf -- "$(gettext " --nocolor remove color from output\n")" - printf -- "$(gettext " --min-pkg-size minimum package size before deltas are generated\n")" - printf -- "$(gettext " --max-delta-size percent of new package above which the delta will be discarded\n")" -} - -version() { - printf "pkgdelta (pacman) %s\n" "$myver" - printf -- "Copyright (c) 2009 Xavier Chantry <shiningxc@gmail.com>.\n" - printf '\n' - printf -- "$(gettext "\ -This is free software; see the source for copying conditions.\n\ -There is NO WARRANTY, to the extent permitted by law.\n")" -} - -m4_include(library/human_to_size.sh) - -isnumeric() { - [[ $1 != *[!0-9]* ]] -} - -read_pkginfo() { - unset pkgver pkgname arch - while IFS='=' read -r field value; do - # skip comments and invalid lines - [[ $field = '#'* || -z $value ]] && continue - - # skip lines which aren't fields we care about - [[ $field != @(pkgver|pkgname|arch) ]] || continue - - declare -g "${field% }=${value# }" - - [[ $pkgname && $pkgver && $arch ]] && return 0 - done < <(bsdtar -xOqf "$1" .PKGINFO 2>/dev/null) - - error "$(gettext "Invalid package file '%s'.")" "$1" - return 1 -} - -# $oldfile $oldmd5 $newfile $newmd5 $deltafile $deltamd5 $deltasize -create_xdelta() -{ - local oldfile=$1 - local newfile=$2 - local \ - oldname oldver oldarch \ - newname newver newarch \ - deltafile - - read_pkginfo "$oldfile" || return 1 - oldname="$pkgname" - oldver="$pkgver" - oldarch="$arch" - read_pkginfo "$newfile" || return 1 - newname="$pkgname" - newver="$pkgver" - newarch="$arch" - - pkgsize="$(wc -c "$newfile" | cut -d' ' -f1)" - - if ((pkgsize < min_pkg_size)); then - msg "$(gettext "Skipping delta creation for small package: %s - size %s")" "$newname" "$pkgsize" - return 0 - fi - - if [[ $oldname != "$newname" ]]; then - error "$(gettext "The package names don't match : '%s' and '%s'")" "$oldname" "$newname" - return 1 - fi - - if [[ $oldarch != "$newarch" ]]; then - error "$(gettext "The package architectures don't match : '%s' and '%s'")" "$oldarch" "$newarch" - return 1 - fi - - if [[ $oldver == "$newver" ]]; then - error "$(gettext "Both packages have the same version : '%s'")" "$newver" - return 1 - fi - - msg "$(gettext "Generating delta from version %s to version %s")" "$oldver" "$newver" - deltafile=$(dirname "$newfile")/$pkgname-${oldver}_to_${newver}-$arch.delta - local ret=0 - - xdelta3 -q -f -9 -S lzma -s "$oldfile" "$newfile" "$deltafile" || ret=$? - if (( ret )); then - error "$(gettext "Delta could not be created.")" - return 1 - fi - - deltasize="$(wc -c "$deltafile" | cut -d' ' -f1)" - - if ((max_delta_size * pkgsize / 100 < deltasize)); then - msg "$(gettext "Delta package larger than maximum size. Removing.")" - rm -f "$deltafile" - return 0 - fi - - msg "$(gettext "Generated delta : '%s'")" "$deltafile" - (( QUIET )) && echo "$deltafile" - - return 0 -} - -OPT_SHORT='hqV' -OPT_LONG=('help' 'quiet' 'max-delta-size:' 'min-pkg-size:' 'nocolor' 'version') -if ! parseopts "$OPT_SHORT" "${OPT_LONG[@]}" -- "$@"; then - exit 1 -fi -set -- "${OPTRET[@]}" -unset OPT_SHORT OPT_LONG OPTRET - -# parse arguments -while :; do - case $1 in - -h|--help) - usage - exit 0 ;; - -V|--version) - version - exit 0 ;; - -q|--quiet) - QUIET=1;; - --nocolor) - USE_COLOR='n';; - --min-pkg-size) - if ! min_pkg_size=$(human_to_size "$2"); then - echo "invalid argument '$2' for option -- '$1'" - exit 1 - fi - shift ;; - --max-delta-size) - arg=$(awk -v val="$2" 'BEGIN { print val * 100 }') - if ! isnumeric "$arg" || (( arg > 200 )); then - echo "invalid argument '$2' for option -- '$1'" - exit 1 - fi - max_delta_size=$arg - shift ;; - --) - shift - break ;; - esac - shift -done - -# check if messages are to be printed using color -if [[ -t 2 && $USE_COLOR != "n" ]]; then - colorize -else - unset ALL_OFF BOLD BLUE GREEN RED YELLOW -fi - -if (( $# != 2 )); then - usage - exit 1 -fi - -for i in "$@"; do - if [[ ! -f $i ]]; then - error "$(gettext "File '%s' does not exist")" "$i" - exit 1 - fi -done - -if ! type xdelta3 &>/dev/null; then - error "$(gettext "Cannot find the xdelta3 binary! Is xdelta3 installed?")" - exit 1 -fi - -create_xdelta "$@" diff --git a/scripts/po/POTFILES.in b/scripts/po/POTFILES.in index 25eb9d3f..5f02393d 100644 --- a/scripts/po/POTFILES.in +++ b/scripts/po/POTFILES.in @@ -5,7 +5,6 @@ scripts/makepkg.sh.in scripts/makepkg-template.pl.in scripts/pacman-db-upgrade.sh.in scripts/pacman-key.sh.in -scripts/pkgdelta.sh.in scripts/repo-add.sh.in scripts/libmakepkg/integrity.sh.in scripts/libmakepkg/integrity/generate_checksum.sh.in -- 2.20.1
Signed-off-by: Allan McRae <allan@archlinux.org> --- meson.build | 8 --- src/pacman/po/POTFILES.in | 1 - src/util/.gitignore | 2 - src/util/Makefile.am | 5 +- src/util/cleanupdelta.c | 135 -------------------------------------- src/util/meson.build | 1 - 6 files changed, 1 insertion(+), 151 deletions(-) delete mode 100644 src/util/cleanupdelta.c diff --git a/meson.build b/meson.build index 0f706c64..b810e2fb 100644 --- a/meson.build +++ b/meson.build @@ -403,14 +403,6 @@ executable( install : true, ) -executable( - 'cleanupdelta', - cleanupdelta_sources, - include_directories : includes, - link_with : [libalpm], - install : true, -) - executable( 'testpkg', testpkg_sources, diff --git a/src/pacman/po/POTFILES.in b/src/pacman/po/POTFILES.in index f233e9bd..a408d32b 100644 --- a/src/pacman/po/POTFILES.in +++ b/src/pacman/po/POTFILES.in @@ -22,6 +22,5 @@ src/pacman/util.c # utililty files src/pacman/pacman-conf.c -src/util/cleanupdelta.c src/util/testpkg.c src/util/vercmp.c \ No newline at end of file diff --git a/src/util/.gitignore b/src/util/.gitignore index 4cb3103e..8fc636aa 100644 --- a/src/util/.gitignore +++ b/src/util/.gitignore @@ -1,7 +1,5 @@ .deps .libs -cleanupdelta -cleanupdelta.exe testpkg testpkg.exe vercmp diff --git a/src/util/Makefile.am b/src/util/Makefile.am index 470c79f5..8c6a5771 100644 --- a/src/util/Makefile.am +++ b/src/util/Makefile.am @@ -4,7 +4,7 @@ dbpath = ${localstatedir}/lib/pacman/ gpgdir = ${sysconfdir}/pacman.d/gnupg/ cachedir = ${localstatedir}/cache/pacman/pkg/ -bin_PROGRAMS = vercmp testpkg cleanupdelta +bin_PROGRAMS = vercmp testpkg AM_CPPFLAGS = \ -imacros $(top_builddir)/config.h \ @@ -18,9 +18,6 @@ AM_CPPFLAGS = \ AM_CFLAGS = -pedantic -D_GNU_SOURCE $(WARNING_CFLAGS) \ $(LIBARCHIVE_CFLAGS) -cleanupdelta_SOURCES = cleanupdelta.c -cleanupdelta_LDADD = $(top_builddir)/lib/libalpm/.libs/libalpm.la - testpkg_SOURCES = testpkg.c testpkg_LDADD = $(top_builddir)/lib/libalpm/.libs/libalpm.la diff --git a/src/util/cleanupdelta.c b/src/util/cleanupdelta.c deleted file mode 100644 index 34e09c52..00000000 --- a/src/util/cleanupdelta.c +++ /dev/null @@ -1,135 +0,0 @@ -/* - * cleanupdelta.c : return list of unused delta in a given sync database - * - * Copyright (c) 2011-2018 Pacman Development Team <pacman-dev@archlinux.org> - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program. If not, see <http://www.gnu.org/licenses/>. - */ - -#include <stdio.h> -#include <stdlib.h> -#include <string.h> - -#include <alpm.h> -#include <alpm_list.h> - -alpm_handle_t *handle = NULL; - -static void cleanup(int signum) -{ - if(handle && alpm_release(handle) == -1) { - fprintf(stderr, "error releasing alpm\n"); - } - - exit(signum); -} - -__attribute__((format(printf, 2, 0))) -static void output_cb(alpm_loglevel_t level, const char *fmt, va_list args) -{ - if(strlen(fmt)) { - switch(level) { - case ALPM_LOG_ERROR: printf("error: "); break; - case ALPM_LOG_WARNING: printf("warning: "); break; - /* case ALPM_LOG_DEBUG: printf("debug: "); break; */ - default: return; - } - vprintf(fmt, args); - } -} - - -static void checkpkgs(alpm_list_t *pkglist) -{ - alpm_list_t *i, *j; - for(i = pkglist; i; i = alpm_list_next(i)) { - alpm_pkg_t *pkg = i->data; - alpm_list_t *unused = alpm_pkg_unused_deltas(pkg); - for(j = unused; j; j = alpm_list_next(j)) { - const char *delta = j->data; - printf("%s\n", delta); - } - alpm_list_free(unused); - } -} - -static void checkdbs(alpm_list_t *dbnames) -{ - alpm_db_t *db = NULL; - alpm_list_t *i; - const int siglevel = ALPM_SIG_DATABASE | ALPM_SIG_DATABASE_OPTIONAL; - - for(i = dbnames; i; i = alpm_list_next(i)) { - const char *dbname = i->data; - db = alpm_register_syncdb(handle, dbname, siglevel); - if(db == NULL) { - fprintf(stderr, "error: could not register sync database '%s' (%s)\n", - dbname, alpm_strerror(alpm_errno(handle))); - continue; - } - checkpkgs(alpm_db_get_pkgcache(db)); - } - -} - -static void usage(void) -{ - fprintf(stderr, "cleanupdelta (pacman) v" PACKAGE_VERSION "\n\n" - "Returns a list of unused delta in a given sync database.\n\n" - "Usage: cleanupdelta [options]\n\n" - " -b <pacman db> core extra ... : check the listed sync databases\n"); - exit(1); -} - -int main(int argc, char *argv[]) -{ - const char *dbpath = DBPATH; - alpm_errno_t err; - int a = 1; - alpm_list_t *dbnames = NULL; - - while(a < argc) { - if(strcmp(argv[a], "-b") == 0) { - if(++a < argc) { - dbpath = argv[a]; - } else { - usage(); - } - } else if(strcmp(argv[a], "-h") == 0 || - strcmp(argv[a], "--help") == 0 ) { - usage(); - } else { - dbnames = alpm_list_add(dbnames, argv[a]); - } - a++; - } - - if(!dbnames) { - usage(); - } - - handle = alpm_initialize(ROOTDIR, dbpath, &err); - if(!handle) { - fprintf(stderr, "cannot initialize alpm: %s\n", alpm_strerror(err)); - return 1; - } - - /* let us get log messages from libalpm */ - alpm_option_set_logcb(handle, output_cb); - - checkdbs(dbnames); - alpm_list_free(dbnames); - - cleanup(0); -} diff --git a/src/util/meson.build b/src/util/meson.build index cc219670..c0063f02 100644 --- a/src/util/meson.build +++ b/src/util/meson.build @@ -1,3 +1,2 @@ -cleanupdelta_sources = files('cleanupdelta.c') testpkg_sources = files('testpkg.c') vercmp_sources = files('vercmp.c') -- 2.20.1
Dummy callbacks are still present to prevent compiler warnings until libalpm is delta free. Also remove Delta parsing from pacman.conf. Signed-off-by: Allan McRae <allan@archlinux.org> --- doc/pacman.conf.5.asciidoc | 8 -------- src/pacman/callback.c | 27 ++++++++------------------- src/pacman/conf.c | 25 ------------------------- src/pacman/conf.h | 1 - src/pacman/pacman-conf.c | 13 ------------- src/pacman/sync.c | 5 +---- 6 files changed, 9 insertions(+), 70 deletions(-) diff --git a/doc/pacman.conf.5.asciidoc b/doc/pacman.conf.5.asciidoc index 9810fc7f..b297e332 100644 --- a/doc/pacman.conf.5.asciidoc +++ b/doc/pacman.conf.5.asciidoc @@ -186,14 +186,6 @@ Options *Color*:: Automatically enable colors only when pacman's output is on a tty. -*UseDelta* [= ratio]:: - Download delta files instead of complete packages if possible. Requires - the `xdelta3` program to be installed. If a ratio is specified (e.g., - `0.5`), then it is used as a cutoff for determining whether to use deltas. - Allowed values are between `0.0` and `2.0`; sensible values are between - `0.2` and `0.9`. Using a value above `1.0` is not recommended. The - default is `0.7` if left unspecified. - *TotalDownload*:: When downloading, display the amount downloaded, download rate, ETA, and completed percentage of the entire download list rather diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 40e7832c..b8e3dd73 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -267,23 +267,6 @@ void cb_event(alpm_event_t *event) printf(_("loading package files...\n")); } break; - case ALPM_EVENT_DELTA_INTEGRITY_START: - printf(_("checking delta integrity...\n")); - break; - case ALPM_EVENT_DELTA_PATCHES_START: - printf(_("applying deltas...\n")); - break; - case ALPM_EVENT_DELTA_PATCH_START: - printf(_("generating %s with %s... "), - event->delta_patch.delta->to, - event->delta_patch.delta->delta); - break; - case ALPM_EVENT_DELTA_PATCH_DONE: - printf(_("success!\n")); - break; - case ALPM_EVENT_DELTA_PATCH_FAILED: - printf(_("failed.\n")); - break; case ALPM_EVENT_SCRIPTLET_INFO: fputs(event->scriptlet_info.line, stdout); break; @@ -355,8 +338,6 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_KEYRING_DONE: case ALPM_EVENT_KEY_DOWNLOAD_DONE: case ALPM_EVENT_LOAD_DONE: - case ALPM_EVENT_DELTA_INTEGRITY_DONE: - case ALPM_EVENT_DELTA_PATCHES_DONE: case ALPM_EVENT_DISKSPACE_DONE: case ALPM_EVENT_RETRIEVE_DONE: case ALPM_EVENT_RETRIEVE_FAILED: @@ -366,6 +347,14 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_PKGDOWNLOAD_START: case ALPM_EVENT_PKGDOWNLOAD_DONE: case ALPM_EVENT_PKGDOWNLOAD_FAILED: + /* temporary until removed from libalpm */ + case ALPM_EVENT_DELTA_INTEGRITY_START: + case ALPM_EVENT_DELTA_INTEGRITY_DONE: + case ALPM_EVENT_DELTA_PATCHES_START: + case ALPM_EVENT_DELTA_PATCHES_DONE: + case ALPM_EVENT_DELTA_PATCH_START: + case ALPM_EVENT_DELTA_PATCH_DONE: + case ALPM_EVENT_DELTA_PATCH_FAILED: /* nothing */ break; } diff --git a/src/pacman/conf.c b/src/pacman/conf.c index 29f69052..febd35e3 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -103,7 +103,6 @@ config_t *config_new(void) newconfig->op = PM_OP_MAIN; newconfig->logmask = ALPM_LOG_ERROR | ALPM_LOG_WARNING; newconfig->configfile = strdup(CONFFILE); - newconfig->deltaratio = 0.0; if(alpm_capabilities() & ALPM_CAPABILITY_SIGNATURES) { newconfig->siglevel = ALPM_SIG_PACKAGE | ALPM_SIG_PACKAGE_OPTIONAL | ALPM_SIG_DATABASE | ALPM_SIG_DATABASE_OPTIONAL; @@ -485,9 +484,6 @@ static int _parse_options(const char *key, char *value, } else if(strcmp(key, "VerbosePkgLists") == 0) { config->verbosepkglists = 1; pm_printf(ALPM_LOG_DEBUG, "config: verbosepkglists\n"); - } else if(strcmp(key, "UseDelta") == 0) { - config->deltaratio = 0.7; - pm_printf(ALPM_LOG_DEBUG, "config: usedelta (default 0.7)\n"); } else if(strcmp(key, "TotalDownload") == 0) { config->totaldownload = 1; pm_printf(ALPM_LOG_DEBUG, "config: totaldownload\n"); @@ -525,26 +521,6 @@ static int _parse_options(const char *key, char *value, if(!config->arch) { config_set_arch(value); } - } else if(strcmp(key, "UseDelta") == 0) { - double ratio; - char *endptr; - const char *oldlocale; - - /* set the locale to 'C' for consistent decimal parsing (0.7 and never - * 0,7) from config files, then restore old setting when we are done */ - oldlocale = setlocale(LC_NUMERIC, NULL); - setlocale(LC_NUMERIC, "C"); - ratio = strtod(value, &endptr); - setlocale(LC_NUMERIC, oldlocale); - - if(*endptr != '\0' || ratio < 0.0 || ratio > 2.0) { - pm_printf(ALPM_LOG_ERROR, - _("config file %s, line %d: invalid value for '%s' : '%s'\n"), - file, linenum, "UseDelta", value); - return 1; - } - config->deltaratio = ratio; - pm_printf(ALPM_LOG_DEBUG, "config: usedelta = %f\n", ratio); } else if(strcmp(key, "DBPath") == 0) { /* don't overwrite a path specified on the command line */ if(!config->dbpath) { @@ -766,7 +742,6 @@ static int setup_libalpm(void) alpm_option_set_arch(handle, config->arch); alpm_option_set_checkspace(handle, config->checkspace); alpm_option_set_usesyslog(handle, config->usesyslog); - alpm_option_set_deltaratio(handle, config->deltaratio); alpm_option_set_ignorepkgs(handle, config->ignorepkg); alpm_option_set_ignoregroups(handle, config->ignoregrp); diff --git a/src/pacman/conf.h b/src/pacman/conf.h index ababf2e0..f45ed436 100644 --- a/src/pacman/conf.h +++ b/src/pacman/conf.h @@ -56,7 +56,6 @@ typedef struct __config_t { unsigned short usesyslog; unsigned short color; unsigned short disable_dl_timeout; - double deltaratio; char *arch; char *print_format; /* unfortunately, we have to keep track of paths both here and in the library diff --git a/src/pacman/pacman-conf.c b/src/pacman/pacman-conf.c index 56196f79..df874029 100644 --- a/src/pacman/pacman-conf.c +++ b/src/pacman/pacman-conf.c @@ -125,14 +125,6 @@ static void list_repos(void) } } -static void show_float(const char *directive, float val) -{ - if(verbose) { - printf("%s = ", directive); - } - printf("%f%c", val, sep); -} - static void show_bool(const char *directive, short unsigned int val) { if(val) { @@ -269,8 +261,6 @@ static void dump_config(void) show_bool("DisableDownloadTimeout", config->disable_dl_timeout); show_bool("ILoveCandy", config->chomp); - show_float("UseDelta", config->deltaratio); - show_cleanmethod("CleanMethod", config->cleanmethod); show_siglevel("SigLevel", config->siglevel, 0); @@ -380,9 +370,6 @@ static int list_directives(void) } else if(strcasecmp(i->data, "DisableDownloadTimeout") == 0) { show_bool("DisableDownloadTimeout", config->disable_dl_timeout); - } else if(strcasecmp(i->data, "UseDelta") == 0) { - show_float("UseDelta", config->deltaratio); - } else if(strcasecmp(i->data, "CleanMethod") == 0) { show_cleanmethod("CleanMethod", config->cleanmethod); diff --git a/src/pacman/sync.c b/src/pacman/sync.c index 2406fed5..7dc69079 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -222,9 +222,7 @@ static int sync_cleancache(int level) /* skip package databases within the cache directory */ "*.db*", "*.files*", /* skip source packages within the cache directory */ - "*.src.tar.*", - /* skip package deltas, we aren't smart enough to clean these yet */ - "*.delta" + "*.src.tar.*" }; size_t j; @@ -847,7 +845,6 @@ int sync_prepare_execute(void) case ALPM_ERR_PKG_INVALID: case ALPM_ERR_PKG_INVALID_CHECKSUM: case ALPM_ERR_PKG_INVALID_SIG: - case ALPM_ERR_DLT_INVALID: for(i = data; i; i = alpm_list_next(i)) { char *filename = i->data; printf(_("%s is invalid or corrupted\n"), filename); -- 2.20.1
Signed-off-by: Allan McRae <allan@archlinux.org> --- lib/libalpm/Makefile.am | 1 - lib/libalpm/alpm.h | 53 ------ lib/libalpm/be_sync.c | 21 +-- lib/libalpm/delta.c | 361 ------------------------------------- lib/libalpm/delta.h | 33 ---- lib/libalpm/error.c | 5 - lib/libalpm/handle.c | 20 -- lib/libalpm/handle.h | 5 - lib/libalpm/meson.build | 1 - lib/libalpm/package.c | 14 -- lib/libalpm/package.h | 2 - lib/libalpm/po/POTFILES.in | 1 - lib/libalpm/sync.c | 207 +-------------------- src/pacman/callback.c | 8 - 14 files changed, 10 insertions(+), 722 deletions(-) delete mode 100644 lib/libalpm/delta.c delete mode 100644 lib/libalpm/delta.h diff --git a/lib/libalpm/Makefile.am b/lib/libalpm/Makefile.am index 8201dcbd..1e01feeb 100644 --- a/lib/libalpm/Makefile.am +++ b/lib/libalpm/Makefile.am @@ -34,7 +34,6 @@ libalpm_la_SOURCES = \ be_sync.c \ conflict.h conflict.c \ db.h db.c \ - delta.h delta.c \ deps.h deps.c \ diskspace.h diskspace.c \ dload.h dload.c \ diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 337b30d2..ffb2ad96 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -104,9 +104,6 @@ typedef enum _alpm_errno_t { /* Signatures */ ALPM_ERR_SIG_MISSING, ALPM_ERR_SIG_INVALID, - /* Deltas */ - ALPM_ERR_DLT_INVALID, - ALPM_ERR_DLT_PATCHFAILED, /* Dependencies */ ALPM_ERR_UNSATISFIED_DEPS, ALPM_ERR_CONFLICTING_DEPS, @@ -273,22 +270,6 @@ typedef struct _alpm_group_t { alpm_list_t *packages; } alpm_group_t; -/** Package upgrade delta */ -typedef struct _alpm_delta_t { - /** filename of the delta patch */ - char *delta; - /** md5sum of the delta file */ - char *delta_md5; - /** filename of the 'before' file */ - char *from; - /** filename of the 'after' file */ - char *to; - /** filesize of the delta file */ - off_t delta_size; - /** download filesize of the delta file */ - off_t download_size; -} alpm_delta_t; - /** File in a package */ typedef struct _alpm_file_t { char *name; @@ -405,21 +386,6 @@ typedef enum _alpm_event_type_t { ALPM_EVENT_LOAD_START, /** Target package is finished loading. */ ALPM_EVENT_LOAD_DONE, - /** Target delta's integrity will be checked. */ - ALPM_EVENT_DELTA_INTEGRITY_START, - /** Target delta's integrity was checked. */ - ALPM_EVENT_DELTA_INTEGRITY_DONE, - /** Deltas will be applied to packages. */ - ALPM_EVENT_DELTA_PATCHES_START, - /** Deltas were applied to packages. */ - ALPM_EVENT_DELTA_PATCHES_DONE, - /** Delta patch will be applied to target package; See - * alpm_event_delta_patch_t for arguments.. */ - ALPM_EVENT_DELTA_PATCH_START, - /** Delta patch was applied to target package. */ - ALPM_EVENT_DELTA_PATCH_DONE, - /** Delta patch failed to apply to target package. */ - ALPM_EVENT_DELTA_PATCH_FAILED, /** Scriptlet has printed information; See alpm_event_scriptlet_info_t for * arguments. */ ALPM_EVENT_SCRIPTLET_INFO, @@ -509,13 +475,6 @@ typedef struct _alpm_event_optdep_removal_t { alpm_depend_t *optdep; } alpm_event_optdep_removal_t; -typedef struct _alpm_event_delta_patch_t { - /** Type of event. */ - alpm_event_type_t type; - /** Delta info */ - alpm_delta_t *delta; -} alpm_event_delta_patch_t; - typedef struct _alpm_event_scriptlet_info_t { /** Type of event. */ alpm_event_type_t type; @@ -589,7 +548,6 @@ typedef union _alpm_event_t { alpm_event_any_t any; alpm_event_package_operation_t package_operation; alpm_event_optdep_removal_t optdep_removal; - alpm_event_delta_patch_t delta_patch; alpm_event_scriptlet_info_t scriptlet_info; alpm_event_database_missing_t database_missing; alpm_event_pkgdownload_t pkgdownload; @@ -915,9 +873,6 @@ const char *alpm_option_get_arch(alpm_handle_t *handle); /** Sets the targeted architecture. */ int alpm_option_set_arch(alpm_handle_t *handle, const char *arch); -double alpm_option_get_deltaratio(alpm_handle_t *handle); -int alpm_option_set_deltaratio(alpm_handle_t *handle, double ratio); - int alpm_option_get_checkspace(alpm_handle_t *handle); int alpm_option_set_checkspace(alpm_handle_t *handle, int checkspace); @@ -1294,12 +1249,6 @@ alpm_list_t *alpm_pkg_get_conflicts(alpm_pkg_t *pkg); */ alpm_list_t *alpm_pkg_get_provides(alpm_pkg_t *pkg); -/** Returns the list of available deltas for pkg. - * @param pkg a pointer to package - * @return a reference to an internal list of strings. - */ -alpm_list_t *alpm_pkg_get_deltas(alpm_pkg_t *pkg); - /** Returns the list of packages to be replaced by pkg. * @param pkg a pointer to package * @return a reference to an internal list of alpm_depend_t structures. @@ -1397,8 +1346,6 @@ int alpm_pkg_has_scriptlet(alpm_pkg_t *pkg); */ off_t alpm_pkg_download_size(alpm_pkg_t *newpkg); -alpm_list_t *alpm_pkg_unused_deltas(alpm_pkg_t *pkg); - /** Set install reason for a package in the local database. * The provided package object must be from the local database or this method * will fail. The write to the local database is performed immediately. diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index deaec27c..9db271b9 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -37,7 +37,6 @@ #include "alpm_list.h" #include "package.h" #include "handle.h" -#include "delta.h" #include "deps.h" #include "dload.h" #include "filelist.h" @@ -427,9 +426,8 @@ static alpm_pkg_t *load_pkg_for_entry(alpm_db_t *db, const char *entryname, /* This function doesn't work as well as one might think, as size of database * entries varies considerably. Adding signatures nearly doubles the size of a - * single entry; deltas also can make for large variations in size. These - * current values are heavily influenced by Arch Linux; databases with no - * deltas and a single signature per package. */ + * single entry. These current values are heavily influenced by Arch Linux; + * databases with a single signature per package. */ static size_t estimate_package_count(struct stat *st, struct archive *archive) { int per_package; @@ -511,7 +509,7 @@ static int sync_db_populate(alpm_db_t *db) while((archive_ret = archive_read_next_header(archive, &entry)) == ARCHIVE_OK) { mode_t mode = archive_entry_mode(entry); if(!S_ISDIR(mode)) { - /* we have desc, depends or deltas - parse it */ + /* we have desc or depends - parse it */ if(sync_db_read(db, archive, entry, &pkg) != 0) { _alpm_log(db->handle, ALPM_LOG_ERROR, _("could not parse package description file '%s' from db '%s'\n"), @@ -637,8 +635,7 @@ static int sync_db_read(alpm_db_t *db, struct archive *archive, } if(strcmp(filename, "desc") == 0 || strcmp(filename, "depends") == 0 - || strcmp(filename, "files") == 0 - || (strcmp(filename, "deltas") == 0 && db->handle->deltaratio > 0.0) ) { + || strcmp(filename, "files") == 0) { int ret; while((ret = _alpm_archive_fgets(archive, &buf)) == ARCHIVE_OK) { char *line = buf.line; @@ -707,14 +704,6 @@ static int sync_db_read(alpm_db_t *db, struct archive *archive, READ_AND_SPLITDEP(pkg->conflicts); } else if(strcmp(line, "%PROVIDES%") == 0) { READ_AND_SPLITDEP(pkg->provides); - } else if(strcmp(line, "%DELTAS%") == 0) { - /* Different than the rest because of the _alpm_delta_parse call. */ - while(1) { - READ_NEXT(); - if(strlen(line) == 0) break; - pkg->deltas = alpm_list_add(pkg->deltas, - _alpm_delta_parse(db->handle, line)); - } } else if(strcmp(line, "%FILES%") == 0) { /* TODO: this could lazy load if there is future demand */ size_t files_count = 0, files_size = 0; @@ -751,8 +740,6 @@ static int sync_db_read(alpm_db_t *db, struct archive *archive, goto error; } *likely_pkg = pkg; - } else if(strcmp(filename, "deltas") == 0) { - /* skip reading delta files if UseDelta is unset */ } else { /* unknown database file */ _alpm_log(db->handle, ALPM_LOG_DEBUG, "unknown database file: %s\n", filename); diff --git a/lib/libalpm/delta.c b/lib/libalpm/delta.c deleted file mode 100644 index 33da4a3f..00000000 --- a/lib/libalpm/delta.c +++ /dev/null @@ -1,361 +0,0 @@ -/* - * delta.c - * - * Copyright (c) 2006-2018 Pacman Development Team <pacman-dev@archlinux.org> - * Copyright (c) 2007-2006 by Judd Vinet <jvinet@zeroflux.org> - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program. If not, see <http://www.gnu.org/licenses/>. - */ - -#include <stdlib.h> -#include <string.h> -#include <stdint.h> /* intmax_t */ -#include <limits.h> -#include <sys/types.h> -#include <regex.h> - -/* libalpm */ -#include "delta.h" -#include "alpm_list.h" -#include "util.h" -#include "log.h" -#include "graph.h" - -static alpm_list_t *graph_init(alpm_list_t *deltas, int reverse) -{ - alpm_list_t *i, *j; - alpm_list_t *vertices = NULL; - /* create the vertices */ - for(i = deltas; i; i = i->next) { - alpm_graph_t *v = _alpm_graph_new(); - if(!v) { - alpm_list_free(vertices); - return NULL; - } - alpm_delta_t *vdelta = i->data; - vdelta->download_size = vdelta->delta_size; - v->weight = LONG_MAX; - v->data = vdelta; - vertices = alpm_list_add(vertices, v); - } - - /* compute the edges */ - for(i = vertices; i; i = i->next) { - alpm_graph_t *v_i = i->data; - alpm_delta_t *d_i = v_i->data; - /* loop a second time so we make all possible comparisons */ - for(j = vertices; j; j = j->next) { - alpm_graph_t *v_j = j->data; - alpm_delta_t *d_j = v_j->data; - /* We want to create a delta tree like the following: - * 1_to_2 - * | - * 1_to_3 2_to_3 - * \ / - * 3_to_4 - * If J 'from' is equal to I 'to', then J is a child of I. - * */ - if((!reverse && strcmp(d_j->from, d_i->to) == 0) || - (reverse && strcmp(d_j->to, d_i->from) == 0)) { - v_i->children = alpm_list_add(v_i->children, v_j); - } - } - v_i->iterator = v_i->children; - } - return vertices; -} - -static void graph_init_size(alpm_handle_t *handle, alpm_list_t *vertices) -{ - alpm_list_t *i; - - for(i = vertices; i; i = i->next) { - char *fpath, *md5sum; - alpm_graph_t *v = i->data; - alpm_delta_t *vdelta = v->data; - - /* determine whether the delta file already exists */ - fpath = _alpm_filecache_find(handle, vdelta->delta); - if(fpath) { - md5sum = alpm_compute_md5sum(fpath); - if(md5sum && strcmp(md5sum, vdelta->delta_md5) == 0) { - vdelta->download_size = 0; - } - FREE(md5sum); - FREE(fpath); - } else { - char *fnamepart; - CALLOC(fnamepart, strlen(vdelta->delta) + 6, sizeof(char), return); - sprintf(fnamepart, "%s.part", vdelta->delta); - fpath = _alpm_filecache_find(handle, fnamepart); - if(fpath) { - struct stat st; - if(stat(fpath, &st) == 0) { - vdelta->download_size = vdelta->delta_size - st.st_size; - vdelta->download_size = vdelta->download_size < 0 ? 0 : vdelta->download_size; - } - FREE(fpath); - } - FREE(fnamepart); - } - - /* determine whether a base 'from' file exists */ - fpath = _alpm_filecache_find(handle, vdelta->from); - if(fpath) { - v->weight = vdelta->download_size; - } - FREE(fpath); - } -} - - -static void dijkstra(alpm_list_t *vertices) -{ - alpm_list_t *i; - alpm_graph_t *v; - while(1) { - v = NULL; - /* find the smallest vertice not visited yet */ - for(i = vertices; i; i = i->next) { - alpm_graph_t *v_i = i->data; - - if(v_i->state == ALPM_GRAPH_STATE_PROCESSING) { - continue; - } - - if(v == NULL || v_i->weight < v->weight) { - v = v_i; - } - } - if(v == NULL || v->weight == LONG_MAX) { - break; - } - - v->state = ALPM_GRAPH_STATE_PROCESSING; - - v->iterator = v->children; - while(v->iterator) { - alpm_graph_t *v_c = v->iterator->data; - alpm_delta_t *d_c = v_c->data; - if(v_c->weight > v->weight + d_c->download_size) { - v_c->weight = v->weight + d_c->download_size; - v_c->parent = v; - } - - v->iterator = (v->iterator)->next; - - } - } -} - -static off_t shortest_path(alpm_list_t *vertices, const char *to, alpm_list_t **path) -{ - alpm_list_t *i; - alpm_graph_t *v = NULL; - off_t bestsize = 0; - alpm_list_t *rpath = NULL; - - for(i = vertices; i; i = i->next) { - alpm_graph_t *v_i = i->data; - alpm_delta_t *d_i = v_i->data; - - if(strcmp(d_i->to, to) == 0) { - if(v == NULL || v_i->weight < v->weight) { - v = v_i; - bestsize = v->weight; - } - } - } - - while(v != NULL) { - alpm_delta_t *vdelta = v->data; - rpath = alpm_list_add(rpath, vdelta); - v = v->parent; - } - *path = alpm_list_reverse(rpath); - alpm_list_free(rpath); - - return bestsize; -} - -/** Calculates the shortest path from one version to another. - * The shortest path is defined as the path with the smallest combined - * size, not the length of the path. - * @param handle the context handle - * @param deltas the list of alpm_delta_t * objects that a file has - * @param to the file to start the search at - * @param path the pointer to a list location where alpm_delta_t * objects that - * have the smallest size are placed. NULL is set if there is no path - * possible with the files available. - * @return the size of the path stored, or LONG_MAX if path is unfindable - */ -off_t _alpm_shortest_delta_path(alpm_handle_t *handle, alpm_list_t *deltas, - const char *to, alpm_list_t **path) -{ - alpm_list_t *bestpath = NULL; - alpm_list_t *vertices; - off_t bestsize = LONG_MAX; - - if(deltas == NULL) { - *path = NULL; - return bestsize; - } - - _alpm_log(handle, ALPM_LOG_DEBUG, "started delta shortest-path search for '%s'\n", to); - - vertices = graph_init(deltas, 0); - graph_init_size(handle, vertices); - dijkstra(vertices); - bestsize = shortest_path(vertices, to, &bestpath); - - _alpm_log(handle, ALPM_LOG_DEBUG, "delta shortest-path search complete : '%jd'\n", (intmax_t)bestsize); - - alpm_list_free_inner(vertices, _alpm_graph_free); - alpm_list_free(vertices); - - *path = bestpath; - return bestsize; -} - -static alpm_list_t *find_unused(alpm_list_t *deltas, const char *to, off_t quota) -{ - alpm_list_t *unused = NULL; - alpm_list_t *vertices; - alpm_list_t *i; - vertices = graph_init(deltas, 1); - - for(i = vertices; i; i = i->next) { - alpm_graph_t *v = i->data; - alpm_delta_t *vdelta = v->data; - if(strcmp(vdelta->to, to) == 0) { - v->weight = vdelta->download_size; - } - } - dijkstra(vertices); - for(i = vertices; i; i = i->next) { - alpm_graph_t *v = i->data; - alpm_delta_t *vdelta = v->data; - if(v->weight > quota) { - unused = alpm_list_add(unused, vdelta->delta); - } - } - alpm_list_free_inner(vertices, _alpm_graph_free); - alpm_list_free(vertices); - return unused; -} - -/** \addtogroup alpm_deltas Delta Functions - * @brief Functions to manipulate libalpm deltas - * @{ - */ - -alpm_list_t SYMEXPORT *alpm_pkg_unused_deltas(alpm_pkg_t *pkg) -{ - ASSERT(pkg != NULL, return NULL); - return find_unused(pkg->deltas, pkg->filename, - pkg->size * pkg->handle->deltaratio); -} - -/** @} */ - -#define NUM_MATCHES 6 - -/** Parses the string representation of a alpm_delta_t object. - * This function assumes that the string is in the correct format. - * This format is as follows: - * $deltafile $deltamd5 $deltasize $oldfile $newfile - * @param handle the context handle - * @param line the string to parse - * @return A pointer to the new alpm_delta_t object - */ -alpm_delta_t *_alpm_delta_parse(alpm_handle_t *handle, const char *line) -{ - alpm_delta_t *delta; - size_t len; - regmatch_t pmatch[NUM_MATCHES]; - char filesize[32]; - - /* this is so we only have to compile the pattern once */ - if(!handle->delta_regex_compiled) { - /* $deltafile $deltamd5 $deltasize $oldfile $newfile*/ - regcomp(&handle->delta_regex, - "^([^[:space:]]+) ([[:xdigit:]]{32}) ([[:digit:]]+)" - " ([^[:space:]]+) ([^[:space:]]+)$", - REG_EXTENDED | REG_NEWLINE); - handle->delta_regex_compiled = 1; - } - - if(regexec(&handle->delta_regex, line, NUM_MATCHES, pmatch, 0) != 0) { - /* delta line is invalid, return NULL */ - return NULL; - } - - CALLOC(delta, 1, sizeof(alpm_delta_t), return NULL); - - /* start at index 1 -- match 0 is the entire match */ - len = pmatch[1].rm_eo - pmatch[1].rm_so; - STRNDUP(delta->delta, &line[pmatch[1].rm_so], len, goto error); - - len = pmatch[2].rm_eo - pmatch[2].rm_so; - STRNDUP(delta->delta_md5, &line[pmatch[2].rm_so], len, goto error); - - len = pmatch[3].rm_eo - pmatch[3].rm_so; - if(len < sizeof(filesize)) { - strncpy(filesize, &line[pmatch[3].rm_so], len); - filesize[len] = '\0'; - delta->delta_size = _alpm_strtoofft(filesize); - } - - len = pmatch[4].rm_eo - pmatch[4].rm_so; - STRNDUP(delta->from, &line[pmatch[4].rm_so], len, goto error); - - len = pmatch[5].rm_eo - pmatch[5].rm_so; - STRNDUP(delta->to, &line[pmatch[5].rm_so], len, goto error); - - return delta; - -error: - _alpm_delta_free(delta); - return NULL; -} - -#undef NUM_MATCHES - -void _alpm_delta_free(alpm_delta_t *delta) -{ - ASSERT(delta != NULL, return); - FREE(delta->delta); - FREE(delta->delta_md5); - FREE(delta->from); - FREE(delta->to); - FREE(delta); -} - -alpm_delta_t *_alpm_delta_dup(const alpm_delta_t *delta) -{ - alpm_delta_t *newdelta; - CALLOC(newdelta, 1, sizeof(alpm_delta_t), return NULL); - STRDUP(newdelta->delta, delta->delta, goto error); - STRDUP(newdelta->delta_md5, delta->delta_md5, goto error); - STRDUP(newdelta->from, delta->from, goto error); - STRDUP(newdelta->to, delta->to, goto error); - newdelta->delta_size = delta->delta_size; - newdelta->download_size = delta->download_size; - - return newdelta; - -error: - _alpm_delta_free(newdelta); - return NULL; -} diff --git a/lib/libalpm/delta.h b/lib/libalpm/delta.h deleted file mode 100644 index d80c0745..00000000 --- a/lib/libalpm/delta.h +++ /dev/null @@ -1,33 +0,0 @@ -/* - * delta.h - * - * Copyright (c) 2006-2018 Pacman Development Team <pacman-dev@archlinux.org> - * Copyright (c) 2007-2006 by Judd Vinet <jvinet@zeroflux.org> - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program. If not, see <http://www.gnu.org/licenses/>. - */ -#ifndef ALPM_DELTA_H -#define ALPM_DELTA_H - -#include <sys/types.h> /* off_t */ - -#include "alpm.h" - -alpm_delta_t *_alpm_delta_parse(alpm_handle_t *handle, const char *line); -void _alpm_delta_free(alpm_delta_t *delta); -alpm_delta_t *_alpm_delta_dup(const alpm_delta_t *delta); -off_t _alpm_shortest_delta_path(alpm_handle_t *handle, alpm_list_t *deltas, - const char *to, alpm_list_t **path); - -#endif /* ALPM_DELTA_H */ diff --git a/lib/libalpm/error.c b/lib/libalpm/error.c index 95be9d7b..a5c6f4fe 100644 --- a/lib/libalpm/error.c +++ b/lib/libalpm/error.c @@ -130,11 +130,6 @@ const char SYMEXPORT *alpm_strerror(alpm_errno_t err) return _("missing PGP signature"); case ALPM_ERR_SIG_INVALID: return _("invalid PGP signature"); - /* Deltas */ - case ALPM_ERR_DLT_INVALID: - return _("invalid or corrupted delta"); - case ALPM_ERR_DLT_PATCHFAILED: - return _("delta patch failed"); /* Dependencies */ case ALPM_ERR_UNSATISFIED_DEPS: return _("could not satisfy dependencies"); diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index c491d87c..476779c4 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -34,7 +34,6 @@ #include "alpm_list.h" #include "util.h" #include "log.h" -#include "delta.h" #include "trans.h" #include "alpm.h" #include "deps.h" @@ -44,7 +43,6 @@ alpm_handle_t *_alpm_handle_new(void) alpm_handle_t *handle; CALLOC(handle, 1, sizeof(alpm_handle_t), return NULL); - handle->deltaratio = 0.0; handle->lockfd = -1; return handle; @@ -75,8 +73,6 @@ void _alpm_handle_free(alpm_handle_t *handle) FREELIST(handle->known_keys); #endif - regfree(&handle->delta_regex); - /* free memory */ _alpm_trans_free(handle->trans); FREE(handle->root); @@ -303,12 +299,6 @@ const char SYMEXPORT *alpm_option_get_arch(alpm_handle_t *handle) return handle->arch; } -double SYMEXPORT alpm_option_get_deltaratio(alpm_handle_t *handle) -{ - CHECK_HANDLE(handle, return -1); - return handle->deltaratio; -} - int SYMEXPORT alpm_option_get_checkspace(alpm_handle_t *handle) { CHECK_HANDLE(handle, return -1); @@ -755,16 +745,6 @@ int SYMEXPORT alpm_option_set_arch(alpm_handle_t *handle, const char *arch) return 0; } -int SYMEXPORT alpm_option_set_deltaratio(alpm_handle_t *handle, double ratio) -{ - CHECK_HANDLE(handle, return -1); - if(ratio < 0.0 || ratio > 2.0) { - RET_ERR(handle, ALPM_ERR_WRONG_ARGS, -1); - } - handle->deltaratio = ratio; - return 0; -} - alpm_db_t SYMEXPORT *alpm_get_localdb(alpm_handle_t *handle) { CHECK_HANDLE(handle, return NULL); diff --git a/lib/libalpm/handle.h b/lib/libalpm/handle.h index 44c4904b..14d20bbe 100644 --- a/lib/libalpm/handle.h +++ b/lib/libalpm/handle.h @@ -95,7 +95,6 @@ struct __alpm_handle_t { /* options */ char *arch; /* Architecture of packages we should allow */ - double deltaratio; /* Download deltas if possible; a ratio value */ int usesyslog; /* Use syslog instead of logfile? */ /* TODO move to frontend */ int checkspace; /* Check disk space before installing */ char *dbext; /* Sync DB extension */ @@ -110,10 +109,6 @@ struct __alpm_handle_t { /* lock file descriptor */ int lockfd; - - /* for delta parsing efficiency */ - int delta_regex_compiled; - regex_t delta_regex; }; alpm_handle_t *_alpm_handle_new(void); diff --git a/lib/libalpm/meson.build b/lib/libalpm/meson.build index 84c3dde3..607e91a3 100644 --- a/lib/libalpm/meson.build +++ b/lib/libalpm/meson.build @@ -9,7 +9,6 @@ libalpm_sources = files(''' be_sync.c conflict.h conflict.c db.h db.c - delta.h delta.c deps.h deps.c diskspace.h diskspace.c dload.h dload.c diff --git a/lib/libalpm/package.c b/lib/libalpm/package.c index 97a78781..94716666 100644 --- a/lib/libalpm/package.c +++ b/lib/libalpm/package.c @@ -31,7 +31,6 @@ #include "log.h" #include "util.h" #include "db.h" -#include "delta.h" #include "handle.h" #include "deps.h" @@ -374,13 +373,6 @@ alpm_list_t SYMEXPORT *alpm_pkg_get_replaces(alpm_pkg_t *pkg) return pkg->ops->get_replaces(pkg); } -alpm_list_t SYMEXPORT *alpm_pkg_get_deltas(alpm_pkg_t *pkg) -{ - ASSERT(pkg != NULL, return NULL); - pkg->handle->pm_errno = ALPM_ERR_OK; - return pkg->deltas; -} - alpm_filelist_t SYMEXPORT *alpm_pkg_get_files(alpm_pkg_t *pkg) { ASSERT(pkg != NULL, return NULL); @@ -620,9 +612,6 @@ int _alpm_pkg_dup(alpm_pkg_t *pkg, alpm_pkg_t **new_ptr) newpkg->optdepends = list_depdup(pkg->optdepends); newpkg->conflicts = list_depdup(pkg->conflicts); newpkg->provides = list_depdup(pkg->provides); - for(i = pkg->deltas; i; i = i->next) { - newpkg->deltas = alpm_list_add(newpkg->deltas, _alpm_delta_dup(i->data)); - } if(pkg->files.count) { size_t filenum; @@ -696,9 +685,6 @@ void _alpm_pkg_free(alpm_pkg_t *pkg) free_deplist(pkg->optdepends); free_deplist(pkg->conflicts); free_deplist(pkg->provides); - alpm_list_free_inner(pkg->deltas, (alpm_list_fn_free)_alpm_delta_free); - alpm_list_free(pkg->deltas); - alpm_list_free(pkg->delta_path); alpm_list_free(pkg->removes); _alpm_pkg_free(pkg->oldpkg); diff --git a/lib/libalpm/package.h b/lib/libalpm/package.h index fbcde266..bc66a569 100644 --- a/lib/libalpm/package.h +++ b/lib/libalpm/package.h @@ -118,8 +118,6 @@ struct __alpm_pkg_t { alpm_list_t *makedepends; alpm_list_t *conflicts; alpm_list_t *provides; - alpm_list_t *deltas; - alpm_list_t *delta_path; alpm_list_t *removes; /* in transaction targets only */ alpm_pkg_t *oldpkg; /* in transaction targets only */ diff --git a/lib/libalpm/po/POTFILES.in b/lib/libalpm/po/POTFILES.in index b6108629..0fa444ed 100644 --- a/lib/libalpm/po/POTFILES.in +++ b/lib/libalpm/po/POTFILES.in @@ -12,7 +12,6 @@ lib/libalpm/be_package.c lib/libalpm/be_sync.c lib/libalpm/conflict.c lib/libalpm/db.c -lib/libalpm/delta.c lib/libalpm/deps.c lib/libalpm/diskspace.c lib/libalpm/dload.c diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index de579eac..cbd072e6 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -43,7 +43,6 @@ #include "handle.h" #include "alpm.h" #include "dload.h" -#include "delta.h" #include "remove.h" #include "diskspace.h" #include "signing.h" @@ -351,21 +350,6 @@ static int compute_download_size(alpm_pkg_t *newpkg) /* tell the caller that we have a partial */ ret = 1; - } else if(handle->deltaratio > 0.0) { - off_t dltsize; - - dltsize = _alpm_shortest_delta_path(handle, newpkg->deltas, - newpkg->filename, &newpkg->delta_path); - - if(newpkg->delta_path && (dltsize < newpkg->size * handle->deltaratio)) { - _alpm_log(handle, ALPM_LOG_DEBUG, "using delta size\n"); - size = dltsize; - } else { - _alpm_log(handle, ALPM_LOG_DEBUG, "using package size\n"); - size = newpkg->size; - alpm_list_free(newpkg->delta_path); - newpkg->delta_path = NULL; - } } else { size = newpkg->size; } @@ -696,118 +680,6 @@ off_t SYMEXPORT alpm_pkg_download_size(alpm_pkg_t *newpkg) return newpkg->download_size; } -static int endswith(const char *filename, const char *extension) -{ - const char *s = filename + strlen(filename) - strlen(extension); - return strcmp(s, extension) == 0; -} - -/** Applies delta files to create an upgraded package file. - * - * All intermediate files are deleted, leaving only the starting and - * ending package files. - * - * @param handle the context handle - * - * @return 0 if all delta files were able to be applied, 1 otherwise. - */ -static int apply_deltas(alpm_handle_t *handle) -{ - alpm_list_t *i; - size_t deltas_found = 0; - int ret = 0; - const char *cachedir = _alpm_filecache_setup(handle); - alpm_trans_t *trans = handle->trans; - alpm_event_delta_patch_t event; - - for(i = trans->add; i; i = i->next) { - alpm_pkg_t *spkg = i->data; - alpm_list_t *delta_path = spkg->delta_path; - alpm_list_t *dlts = NULL; - - if(!delta_path) { - continue; - } - - if(!deltas_found) { - /* only show this if we actually have deltas to apply, and it is before - * the very first one */ - event.type = ALPM_EVENT_DELTA_PATCHES_START; - EVENT(handle, &event); - deltas_found = 1; - } - - for(dlts = delta_path; dlts; dlts = dlts->next) { - alpm_delta_t *d = dlts->data; - char *delta, *from, *to; - char command[PATH_MAX]; - size_t len = 0; - - delta = _alpm_filecache_find(handle, d->delta); - /* the initial package might be in a different cachedir */ - if(dlts == delta_path) { - from = _alpm_filecache_find(handle, d->from); - } else { - /* len = cachedir len + from len + '/' + null */ - len = strlen(cachedir) + strlen(d->from) + 2; - MALLOC(from, len, free(delta); RET_ERR(handle, ALPM_ERR_MEMORY, 1)); - snprintf(from, len, "%s/%s", cachedir, d->from); - } - len = strlen(cachedir) + strlen(d->to) + 2; - MALLOC(to, len, free(delta); free(from); RET_ERR(handle, ALPM_ERR_MEMORY, 1)); - snprintf(to, len, "%s/%s", cachedir, d->to); - - /* build the patch command */ - if(endswith(to, ".gz")) { - /* special handling for gzip : we disable timestamp with -n option */ - snprintf(command, PATH_MAX, "xdelta3 -d -q -R -c -s %s %s | gzip -n > %s", from, delta, to); - } else { - snprintf(command, PATH_MAX, "xdelta3 -d -q -s %s %s %s", from, delta, to); - } - - _alpm_log(handle, ALPM_LOG_DEBUG, "command: %s\n", command); - - event.type = ALPM_EVENT_DELTA_PATCH_START; - event.delta = d; - EVENT(handle, &event); - - int retval = system(command); - if(retval == 0) { - event.type = ALPM_EVENT_DELTA_PATCH_DONE; - EVENT(handle, &event); - - /* delete the delta file */ - unlink(delta); - - /* Delete the 'from' package but only if it is an intermediate - * package. The starting 'from' package should be kept, just - * as if deltas were not used. */ - if(dlts != delta_path) { - unlink(from); - } - } - FREE(from); - FREE(to); - FREE(delta); - - if(retval != 0) { - /* one delta failed for this package, cancel the remaining ones */ - event.type = ALPM_EVENT_DELTA_PATCH_FAILED; - EVENT(handle, &event); - handle->pm_errno = ALPM_ERR_DLT_PATCHFAILED; - ret = 1; - break; - } - } - } - if(deltas_found) { - event.type = ALPM_EVENT_DELTA_PATCHES_DONE; - EVENT(handle, &event); - } - - return ret; -} - /** * Prompts to delete the file now that we know it is invalid. * @param handle the context handle @@ -832,44 +704,6 @@ static int prompt_to_delete(alpm_handle_t *handle, const char *filepath, return question.remove; } -static int validate_deltas(alpm_handle_t *handle, alpm_list_t *deltas) -{ - alpm_list_t *i, *errors = NULL; - alpm_event_t event; - - if(!deltas) { - return 0; - } - - /* Check integrity of deltas */ - event.type = ALPM_EVENT_DELTA_INTEGRITY_START; - EVENT(handle, &event); - for(i = deltas; i; i = i->next) { - alpm_delta_t *d = i->data; - char *filepath = _alpm_filecache_find(handle, d->delta); - - if(_alpm_test_checksum(filepath, d->delta_md5, ALPM_PKG_VALIDATION_MD5SUM)) { - errors = alpm_list_add(errors, filepath); - } else { - FREE(filepath); - } - } - event.type = ALPM_EVENT_DELTA_INTEGRITY_DONE; - EVENT(handle, &event); - - if(errors) { - for(i = errors; i; i = i->next) { - char *filepath = i->data; - prompt_to_delete(handle, filepath, ALPM_ERR_DLT_INVALID); - FREE(filepath); - } - alpm_list_free(errors); - handle->pm_errno = ALPM_ERR_DLT_INVALID; - return -1; - } - return 0; -} - static struct dload_payload *build_payload(alpm_handle_t *handle, const char *filename, size_t size, alpm_list_t *servers) { @@ -882,7 +716,7 @@ static struct dload_payload *build_payload(alpm_handle_t *handle, return payload; } -static int find_dl_candidates(alpm_db_t *repo, alpm_list_t **files, alpm_list_t **deltas) +static int find_dl_candidates(alpm_db_t *repo, alpm_list_t **files) { alpm_list_t *i; alpm_handle_t *handle = repo->handle; @@ -891,8 +725,6 @@ static int find_dl_candidates(alpm_db_t *repo, alpm_list_t **files, alpm_list_t alpm_pkg_t *spkg = i->data; if(spkg->origin != ALPM_PKG_FROM_FILE && repo == spkg->origin_data.db) { - alpm_list_t *delta_path = spkg->delta_path; - if(!repo->servers) { handle->pm_errno = ALPM_ERR_SERVER_NONE; _alpm_log(handle, ALPM_LOG_ERROR, "%s: %s\n", @@ -900,22 +732,7 @@ static int find_dl_candidates(alpm_db_t *repo, alpm_list_t **files, alpm_list_t return 1; } - if(delta_path) { - /* using deltas */ - alpm_list_t *dlts; - for(dlts = delta_path; dlts; dlts = dlts->next) { - alpm_delta_t *delta = dlts->data; - if(delta->download_size != 0) { - struct dload_payload *payload = build_payload( - handle, delta->delta, delta->delta_size, repo->servers); - ASSERT(payload, return -1); - *files = alpm_list_add(*files, payload); - } - /* keep a list of all the delta files for md5sums */ - *deltas = alpm_list_add(*deltas, delta); - } - - } else if(spkg->download_size != 0) { + if(spkg->download_size != 0) { struct dload_payload *payload; ASSERT(spkg->filename != NULL, RET_ERR(handle, ALPM_ERR_PKG_INVALID_NAME, -1)); payload = build_payload(handle, spkg->filename, spkg->size, repo->servers); @@ -963,7 +780,7 @@ static int download_single_file(alpm_handle_t *handle, struct dload_payload *pay return -1; } -static int download_files(alpm_handle_t *handle, alpm_list_t **deltas) +static int download_files(alpm_handle_t *handle) { const char *cachedir; alpm_list_t *i, *files = NULL; @@ -987,7 +804,7 @@ static int download_files(alpm_handle_t *handle, alpm_list_t **deltas) } for(i = handle->dbs_sync; i; i = i->next) { - errors += find_dl_candidates(i->data, &files, deltas); + errors += find_dl_candidates(i->data, &files); } if(files) { @@ -1309,24 +1126,12 @@ static int load_packages(alpm_handle_t *handle, alpm_list_t **data, int _alpm_sync_load(alpm_handle_t *handle, alpm_list_t **data) { - alpm_list_t *i, *deltas = NULL; + alpm_list_t *i; size_t total = 0; uint64_t total_bytes = 0; alpm_trans_t *trans = handle->trans; - if(download_files(handle, &deltas)) { - alpm_list_free(deltas); - return -1; - } - - if(validate_deltas(handle, deltas)) { - alpm_list_free(deltas); - return -1; - } - alpm_list_free(deltas); - - /* Use the deltas to generate the packages */ - if(apply_deltas(handle)) { + if(download_files(handle)) { return -1; } diff --git a/src/pacman/callback.c b/src/pacman/callback.c index b8e3dd73..22865614 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -347,14 +347,6 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_PKGDOWNLOAD_START: case ALPM_EVENT_PKGDOWNLOAD_DONE: case ALPM_EVENT_PKGDOWNLOAD_FAILED: - /* temporary until removed from libalpm */ - case ALPM_EVENT_DELTA_INTEGRITY_START: - case ALPM_EVENT_DELTA_INTEGRITY_DONE: - case ALPM_EVENT_DELTA_PATCHES_START: - case ALPM_EVENT_DELTA_PATCHES_DONE: - case ALPM_EVENT_DELTA_PATCH_START: - case ALPM_EVENT_DELTA_PATCH_DONE: - case ALPM_EVENT_DELTA_PATCH_FAILED: /* nothing */ break; } -- 2.20.1
On 2/3/19 8:19 pm, Allan McRae wrote:
Deltas are broken. So much so that I would strongly recommend never using a delta from a repo that you did not generate yourself. In short, we call "system(command)", with a command that includes the name of a delta file, and the name of the package file before and after applying the delta. The name of the delta and the package files is controlled by the information in the repo, and could contain a malicious command to be run as root.
We could possibly work around this, but it is a very risky piece of code and I believe it would be very hard to fully secure. Instead, I propose to remove delta support completely.
FYI, I'll retract my statement that it would be hard to fully secure. It is entirely possible to avoid spiking in shell code into the file names. But I'd still be happy removing deltas. A
On 03/04/19 at 11:23am, Allan McRae wrote:
On 2/3/19 8:19 pm, Allan McRae wrote:
Deltas are broken. So much so that I would strongly recommend never using a delta from a repo that you did not generate yourself. In short, we call "system(command)", with a command that includes the name of a delta file, and the name of the package file before and after applying the delta. The name of the delta and the package files is controlled by the information in the repo, and could contain a malicious command to be run as root.
We could possibly work around this, but it is a very risky piece of code and I believe it would be very hard to fully secure. Instead, I propose to remove delta support completely.
FYI, I'll retract my statement that it would be hard to fully secure. It is entirely possible to avoid spiking in shell code into the file names. But I'd still be happy removing deltas.
I've wanted to remove deltas for some time just due to how poorly tested and maintained they are, so this still gets a +1 from me. You missed references to deltaratio in etc/pacman.conf and README.
On 4/3/19 4:07 pm, Andrew Gregory wrote:
You missed references to deltaratio in etc/pacman.conf and README.
Thanks. Fixed. A
On Sat, Mar 02, 2019 at 08:19:11PM +1000, Allan McRae wrote:
Deltas are broken. So much so that I would strongly recommend never using a delta from a repo that you did not generate yourself. In short, we call "system(command)", with a command that includes the name of a delta file, and the name of the package file before and after applying the delta. The name of the delta and the package files is controlled by the information in the repo, and could contain a malicious command to be run as root.
We could possibly work around this, but it is a very risky piece of code and I believe it would be very hard to fully secure. Instead, I propose to remove delta support completely.
This issue was assigned CVE-2019-18183. https://security.archlinux.org/CVE-2019-18183 -- Morten Linderud PGP: 9C02FF419FECBE16
participants (3)
-
Allan McRae
-
Andrew Gregory
-
Morten Linderud