- Pacman-dev - lists.archlinux.org

[pacman-dev] Dan's pacman tree build&test
by Gerhard Brauer 25 Dec '08

25 Dec '08

Hi Dan, git clone http://code.toofishes.net/gitprojects/pacman.git (branch is master) 1. if have to install asciidox (for a2x), that's current not a (build)depend of pacman make stops with an error make[2]: Entering directory `/home/gerhard/al-de/pac-git/pacman/doc' a2x -d manpage -f manpage --xsltproc-opts='-param man.endnotes.list.enabled 0' --xsltproc-opts='-param man.en dnotes.are.numbered 0' --asciidoc-opts="-f asciidoc.conf -a pacman_version="3.2.1" -a pacman_date="`date +%Y- %m-%d`" -a sysconfdir=/etc" PKGBUILD.5.txt ./PKGBUILD.5.xml:148: element literal: validity error : Element emphasis is not declared in literal list of p ossible children e. The syntax is: <literal>source=(<emphasis>filename::url</emphasis>)</literal> ^ ./PKGBUILD.5.xml:780: element programlisting: validity error : No declaration for attribute language of eleme nt programlisting <programlisting language="sh" linenumbering="unnumbered"># Maintainer: Joe User ^ ./PKGBUILD.5.xml:780: parser error : error parsing attribute name isting language="sh" linenumbering="unnumbered"># Maintainer: Joe User <joe.user ^ ./PKGBUILD.5.xml:780: parser error : attributes construct error isting language="sh" linenumbering="unnumbered"># Maintainer: Joe User <joe.user ^ ./PKGBUILD.5.xml:780: parser error : Couldn't find end of Start Tag joe.user line 780 isting language="sh" linenumbering="unnumbered"># Maintainer: Joe User <joe.user ^ a2x: failed: xmllint --nonet --noout --valid "./PKGBUILD.5.xml" make[2]: *** [PKGBUILD.5] Fehler 1 make[2]: Leaving directory `/home/gerhard/al-de/pac-git/pacman/doc' make[1]: *** [all-recursive] Fehler 1 make[1]: Leaving directory `/home/gerhard/al-de/pac-git/pacman' make: *** [all] Fehler 2 Seems any error in XML-Input doc file. Is it ok to post to this ML? Or better a private mail address? Gerhard

5 7

Re: [pacman-dev] makepkg: package splitting
by Marc - A. Dahlhaus [ Administration | Westermann GmbH ] 23 Dec '08

23 Dec '08

Am Montag, den 15.12.2008, 21:43 +1000 schrieb Allan McRae: > Allan McRae wrote: > > Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote: > >> i think i found another bug. Every package that is a split-build and not > >> in the first position in the pkgname build-order shows an empty > >> "Optional dependencies" string during installation. I think there is > >> something wrong in restore_package_variables during variable resetting > >> to initial values or in backup_package_variables during the backup > >> declaration. It looks like there are to much quotes around the > >> declaration so that it declares an empty string. Maybe a "-z" check > >> before the actual variable declaration could fix that particular > >> problem. > >> > > > > Fix now on my working branch: > > http://dev.archlinux.org/~allan/gitweb/gitweb.cgi?p=pacman.git;a=commitdiff… > > > > > > And when I say my working branch, I mean my splitpkg branch... Nice, i'll test and report back... Is there any schedule on when it will be merged to mainline? Marc

3 3

[pacman-dev] [PATCH] makepkg: Add docmunetation for separate pacakge function
by Allan McRae 22 Dec '08

22 Dec '08

Signed-off-by: Allan McRae <allan(a)archlinux.org> --- doc/PKGBUILD.5.txt | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/doc/PKGBUILD.5.txt b/doc/PKGBUILD.5.txt index e42a8b8..5171798 100644 --- a/doc/PKGBUILD.5.txt +++ b/doc/PKGBUILD.5.txt @@ -254,6 +254,15 @@ If you create any variables of your own in the build function, it is recommended to use the bash `local` keyword to scope the variable to inside the build function. +package() Function +------------------ +An optional package() function can be specified in addtion to the build() function. +This function is directly sourced by makepkg and run immediately after the build() +function. When specified in combination with the fakeroot BUILDENV option in +linkman:makepkg.conf[5], fakeroot usage will be limited to running the package() +function and the subsequent packing of the package. The build() function will be +run as the user calling makepkg. + Install/Upgrade/Remove Scripting -------------------------------- Pacman has the ability to store and execute a package-specific script when it -- 1.6.0.5

3 6

[pacman-dev] [PATCH] makepkg: Replace getopt with internal function
by Allan McRae 22 Dec '08

22 Dec '08

This will allow makepkg to work on systems like Mac OS X where the default getopt is too old to properly handle long options. The new parse_options function should replicate getopt's behaviour completely. Original work: Yun Zheng Hu <yunzheng.hu(a)gmail.com> [Allan: Rewrite and bug fixes] Signed-off-by: Allan McRae <allan(a)archlinux.org> --- scripts/makepkg.sh.in | 91 +++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 89 insertions(+), 2 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index ea034ad..8cf301c 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -1186,6 +1186,93 @@ devel_update() { fi } +# getopt like parser +parse_options() { + local short_options=$1; shift; + local long_options=$1; shift; + local ret=0; + local unused_options="" + + while [ -n "$1" ]; do + if [ ${1:0:2} = '--' ]; then + if [ -n "${1:2}" ]; then + local match="" + for i in ${long_options//,/ }; do + if [ ${1:2} = ${i//:} ]; then + match=$i + break + fi + done + if [ -n "$match" ]; then + if [ ${1:2} = $match ]; then + printf ' %s' "$1" + else + if [ -n "$2" ]; then + printf ' %s' "$1" + shift + printf " '%s'" "$1" + else + echo "makepkg: option '$1' $(gettext "requires an argument")" >&2 + ret=1 + fi + fi + else + echo "makepkg: $(gettext "unrecognized option") '$1'" >&2 + ret=1 + fi + else + shift + break + fi + elif [ ${1:0:1} = '-' ]; then + for ((i=1; i<${#1}; i++)); do + if [[ "$short_options" =~ "${1:i:1}" ]]; then + if [[ "$short_options" =~ "${1:i:1}:" ]]; then + if [ -n "${1:$i+1}" ]; then + printf ' -%s' "${1:i:1}" + printf " '%s'" "${1:$i+1}" + else + if [ -n "$2" ]; then + printf ' -%s' "${1:i:1}" + shift + printf " '%s'" "${1}" + else + echo "makepkg: option $(gettext "requires an argument") -- '${1:i:1}'" >&2 + ret=1 + fi + fi + break + else + printf ' -%s' "${1:i:1}" + fi + else + echo "makepkg: $(gettext "invalid option") -- '${1:i:1}'" >&2 + ret=1 + fi + done + else + unused_options="${unused_options} '$1'" + fi + shift + done + + printf " --" + if [ -n "$unused_options" ]; then + for i in ${unused_options[@]}; do + printf ' %s' "$i" + done + fi + if [ -n "$1" ]; then + while [ -n "$1" ]; do + printf " '%s'" "${1}" + shift + done + fi + printf "\n" + + return $ret +} + usage() { printf "makepkg (pacman) %s\n" "$myver" echo @@ -1251,8 +1338,8 @@ OPT_LONG="$OPT_LONG,install,log,nocolor,nobuild,rmdeps,repackage,source" OPT_LONG="$OPT_LONG,syncdeps,version,config:" # Pacman Options OPT_LONG="$OPT_LONG,noconfirm,noprogressbar" -OPT_TEMP="$(getopt -o "$OPT_SHORT" -l "$OPT_LONG" -n "$(basename "$0")" -- "$@" || echo 'GETOPT GO BANG!')" -if echo "$OPT_TEMP" | grep -q 'GETOPT GO BANG!'; then +OPT_TEMP="$(parse_options $OPT_SHORT $OPT_LONG "$@" || echo 'PARSE_OPTIONS FAILED')" +if echo "$OPT_TEMP" | grep -q 'PARSE_OPTIONS FAILED'; then # This is a small hack to stop the script bailing with 'set -e' echo; usage; exit 1 # E_INVALID_OPTION; fi -- 1.6.0.5

1 0

[pacman-dev] [PATCH] Added the function parse_options to replace getopt
by Yun Zheng Hu 22 Dec '08

22 Dec '08

This will make makepkg work properly on systems like Mac OS X, where the default installed getopt does not handle long options. The new parse_options function tries to behave like the original getopt command as much as possible. see also: http://www.archlinux.org/pipermail/pacman-dev/2008-May/011830.html Signed-off-by: Yun Zheng Hu <yunzheng.hu(a)gmail.com> --- scripts/makepkg.sh.in | 81 +++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 79 insertions(+), 2 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 6dc7db4..8e3ad25 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -1117,6 +1117,83 @@ devel_update() { fi } +## +# usage : match_long_opt( $needle, $options ) +# return : matched option +## +match_long_opt() { + local needle=$1; shift + local matches item + for item in "$@"; do + [[ "$item" =~ ^"$needle" ]] && matches="$matches $item" + done + matches=( $matches ) + [[ ${#matches[@]} == 1 ]] && echo ${matches[0]} +} + +# getopt like parser, returns 1 on error +parse_options() { + local short_options=$1; shift; + local long_options=$1; shift; + + local needs_args=0 consume_args=0 retcode=0 + local opt_args="" last_opt="" + until [ -z "$1" ]; do + local handle_opt=$((!$needs_args && !$consume_args)) + + if [[ ${1:0:2} = '--' && $handle_opt == 1 ]]; then + # handle long option + opt=${1:2} + if [ ! -z "$opt" ]; then + longopt_match=$(match_long_opt $opt ${long_options//,/ }) + longopt_real="${longopt_match/:/}" # strip possible ':' + [[ "${longopt_match}" == "${longopt_real}:" ]] && needs_args=1 + [[ ! -z "$longopt_match" ]] && printf ' --%s' "$longopt_real" || { + echo "makepkg: $(gettext "unrecognized option") '$1'" >&2 + retcode=1 + } + elif [ $consume_args == 0 ]; then + # option '--', stop parsing options + consume_args=1 + fi + elif [[ ${1:0:1} = '-' && $handle_opt == 1 ]]; then + # handle short option(s) + opts=${1:1} + local i=0 args="" + while [ $i -lt ${#opts} ]; do + opt=${opts:$((i++)):1} + [[ "$needs_args" == 1 ]] && args="${param}${opt}" && continue + [[ "$short_options" =~ "${opt}:" ]] && needs_args=1 + [[ "$short_options" =~ "${opt}" ]] && printf ' -%s' "${opt}" || { + echo "makepkg: $(gettext "invalid option") -- '$opt'" >&2 + retcode=1 + } + done + # check if the user supplied the argument directly after the option + [ ! -z "$args" ] && printf " '$param'" && needs_args=0 + else + # handle non option + [ "$needs_args" == 1 ] && printf " '$1'" && needs_args=0 + [ "$consume_args" == 1 ] && opt_args="$opt_args '$1'" + fi + last_opt="$1" + shift + done + + # check if the user forgot to supply a required argument + if [[ "$needs_args" == 1 ]]; then + if [[ ${last_opt:0:2} = '--' ]]; then + echo "makepkg: option '$last_opt' $(gettext "requires an argument")" >&2 + else + echo "makepkg: option $(gettext "requires an argument") -- '${last_opt:1}'" >&2 + fi + retcode=1 + fi + + printf " --$opt_args\n" + return $retcode +} + usage() { printf "makepkg (pacman) %s\n" "$myver" echo @@ -1182,8 +1259,8 @@ OPT_LONG="$OPT_LONG,install,log,nocolor,nobuild,rmdeps,repackage,source" OPT_LONG="$OPT_LONG,syncdeps,version,config:" # Pacman Options OPT_LONG="$OPT_LONG,noconfirm,noprogressbar" -OPT_TEMP="$(getopt -o "$OPT_SHORT" -l "$OPT_LONG" -n "$(basename "$0")" -- "$@" || echo 'GETOPT GO BANG!')" -if echo "$OPT_TEMP" | grep -q 'GETOPT GO BANG!'; then +OPT_TEMP="$(parse_options $OPT_SHORT $OPT_LONG "$@" || echo 'PARSE_OPTIONS FAILED')" +if echo "$OPT_TEMP" | grep -q 'PARSE_OPTIONS FAILED'; then # This is a small hack to stop the script bailing with 'set -e' echo; usage; exit 1 # E_INVALID_OPTION; fi -- 1.5.6.5

3 6

[pacman-dev] PKGBUILD.proto - Take3
by Jud 19 Dec '08

19 Dec '08

Hi, Edit: Take3, suggested changes - thanks everyone! Dan suggested I send this to the pacman-dev list. After completing some research and asking alot of questions I present some minor changes to PKGBUILD.proto supplied as a .diff to be merged after your approval. I believe it helps the intended audience create a better PKGBUILD in less time according to the latest Arch Packaging Standards. Cheers Jud Inline: --- PKGBUILD.proto 2008-12-05 23:32:33.000005000 +1000 +++ PKGBUILD.proto.new 2008-12-08 17:17:15.536302100 +1000 @@ -4,12 +4,13 @@ # then please put 'unknown'. # Contributor: Your Name <youremail(a)domain.com> + pkgname=NAME -pkgver=VERSION +pkgver=VERSION # Note: Cannot contain hyphens pkgrel=1 pkgdesc="" +url="http://ADDRESS/" arch=() -url="" license=('GPL') groups=() depends=() @@ -20,17 +21,14 @@ replaces=() backup=() options=() -install= -source=($pkgname-$pkgver.tar.gz) +install=$pkgname.install +source=(http://ADDRESS/TO/FILE/$pkgname-$pkgver.tar.gz) noextract=() -md5sums=() #generate with 'makepkg -g' +md5sums=() # Generate with 'makepkg -g' build() { cd "$srcdir/$pkgname-$pkgver" - ./configure --prefix=/usr make || return 1 - make DESTDIR="$pkgdir/" install + make DESTDIR="$pkgdir/" install || return 1 } - -# vim:set ts=2 sw=2 et:

4 3

[pacman-dev] [PATCH] (newgpg) Let pacman specify GnuPG's home directory.
by Chris Brannon 18 Dec '08

18 Dec '08

GnuPG looks for configuration files and keyrings in its home directory. For a user, that is typically ~/.gnupg. This patch causes pacman to use /etc/pacman.d/gnupg/ as the default GnuPG home. One may override the default using --gpgdir on the command-line or GPGDir in pacman's configuration file. Signed-off-by: Chris Brannon <cmbrannon(a)cox.net> --- doc/pacman.8.txt | 7 +++++++ doc/pacman.conf.5.txt | 6 ++++++ src/pacman/Makefile.am | 2 ++ src/pacman/conf.h | 1 + src/pacman/pacman.c | 25 +++++++++++++++++++++++++ 5 files changed, 41 insertions(+), 0 deletions(-) diff --git a/doc/pacman.8.txt b/doc/pacman.8.txt index 6f071ba..a780627 100644 --- a/doc/pacman.8.txt +++ b/doc/pacman.8.txt @@ -136,6 +136,13 @@ Options *\--config* <'file'>:: Specify an alternate configuration file. +*\--gpgdir* <'dir':: + Specify a directory of files used by GnuPG to verify package + signatures. This directory should contain two files: + ``pubring.gpg'' and ``trustdb.gpg''. ``pubring.gpg'' holds the public + keys of all packagers. ``trustdb.gpg'' contains a so-called + trust database, which specifies that the keys are authentic and trusted. + *\--logfile* <'file'>:: Specify an alternate log file. This is an absolute path, regardless of the installation root setting. diff --git a/doc/pacman.conf.5.txt b/doc/pacman.conf.5.txt index 8ef11ec..fa69bfa 100644 --- a/doc/pacman.conf.5.txt +++ b/doc/pacman.conf.5.txt @@ -69,6 +69,12 @@ Options path, the root path is not automatically prepended. +*GPGDir =* path/to/gpg/dir:: + Overrides the default location of the directory containing + configuration files for GnuPG. + A typical default is ``/etc/pacman.d/gnupg''. + This is an absolute path, and the root directory is not prepended. + *LogFile =* '/path/to/file':: Overrides the default location of the pacman log file. A typical default is ``/var/log/pacman.log''. This is an absolute path and the root directory diff --git a/src/pacman/Makefile.am b/src/pacman/Makefile.am index 220ee9c..4da6ef3 100644 --- a/src/pacman/Makefile.am +++ b/src/pacman/Makefile.am @@ -1,6 +1,7 @@ # paths set at make time conffile = ${sysconfdir}/pacman.conf dbpath = ${localstatedir}/lib/pacman/ +gpgdir = ${sysconfdir}/pacman.d/gnupg/ cachedir = ${localstatedir}/cache/pacman/pkg/ logfile = ${localstatedir}/log/pacman.log @@ -10,6 +11,7 @@ DEFS = -DLOCALEDIR=\"@localedir@\" \ -DCONFFILE=\"$(conffile)\" \ -DROOTDIR=\"$(ROOTDIR)\" \ -DDBPATH=\"$(dbpath)\" \ + -DGPGDIR=\"$(gpgdir)\" \ -DCACHEDIR=\"$(cachedir)\" \ -DLOGFILE=\"$(logfile)\" \ @DEFS@ diff --git a/src/pacman/conf.h b/src/pacman/conf.h index 8ea6662..f491057 100644 --- a/src/pacman/conf.h +++ b/src/pacman/conf.h @@ -37,6 +37,7 @@ typedef struct __config_t { char *rootdir; char *dbpath; char *logfile; + char *gpgdir; /* TODO how to handle cachedirs? */ unsigned short op_q_isfile; diff --git a/src/pacman/pacman.c b/src/pacman/pacman.c index 3255cdf..18fd3a8 100644 --- a/src/pacman/pacman.c +++ b/src/pacman/pacman.c @@ -138,6 +138,7 @@ static void usage(int op, const char * const myname) printf(_(" -q, --quiet show less information for query and search\n")); } printf(_(" --config <path> set an alternate configuration file\n")); + printf(_(" --gpgdir <path> set an alternate home directory for GnuPG\n")); printf(_(" --logfile <path> set an alternate log file\n")); printf(_(" --noconfirm do not ask for any confirmation\n")); printf(_(" --noprogressbar do not show a progress bar when downloading files\n")); @@ -306,6 +307,20 @@ static void setlibpaths(void) } } + /* + * Set GnuPG's home directory. This is not relative to + * rootdir, even if rootdir is defined. + * Reasoning: gpgdir contains configuration data. +*/ + if(config->gpgdir) { + ret = alpm_option_set_signaturedir(config->gpgdir); + if(ret != 0) { + pm_printf(PM_LOG_ERROR, _("problem setting gpgdir '%s' (%s)\n"), + config->gpgdir, alpm_strerrorlast()); + cleanup(ret); + } + } + /* add a default cachedir if one wasn't specified */ if(alpm_option_get_cachedirs() == NULL) { alpm_option_add_cachedir(CACHEDIR); @@ -366,6 +381,7 @@ static int parseargs(int argc, char *argv[]) {"debug", optional_argument, 0, 1003}, {"noprogressbar", no_argument, 0, 1004}, {"noscriptlet", no_argument, 0, 1005}, + {"gpgdir", required_argument, 0, 1006}, {"cachedir", required_argument, 0, 1007}, {"asdeps", no_argument, 0, 1008}, {"logfile", required_argument, 0, 1009}, @@ -446,6 +462,9 @@ static int parseargs(int argc, char *argv[]) case 1012: config->flags |= PM_TRANS_FLAG_ALLEXPLICIT; break; + case 1006: + config->gpgdir = strdup(optarg); + break; case 'Q': config->op = (config->op != PM_OP_MAIN ? 0 : PM_OP_QUERY); break; case 'R': config->op = (config->op != PM_OP_MAIN ? 0 : PM_OP_REMOVE); break; case 'S': config->op = (config->op != PM_OP_MAIN ? 0 : PM_OP_SYNC); break; @@ -725,6 +744,11 @@ static int _parseconfig(const char *file, const char *givensection, config->rootdir = strdup(ptr); pm_printf(PM_LOG_DEBUG, "config: rootdir: %s\n", ptr); } + } else if (strcmp(key, "GPGDir") == 0) { + if(!config->gpgdir) { + config->gpgdir = strdup(ptr); + pm_printf(PM_LOG_DEBUG, "config: gpgdir: %s\n", ptr); + } } else if (strcmp(key, "LogFile") == 0) { if(!config->logfile) { config->logfile = strdup(ptr); @@ -864,6 +888,7 @@ int main(int argc, char *argv[]) /* define paths to reasonable defaults */ alpm_option_set_root(ROOTDIR); alpm_option_set_dbpath(DBPATH); + alpm_option_set_signaturedir(GPGDIR); alpm_option_set_logfile(LOGFILE); /* Priority of options: -- 1.6.0.5

6 16

[pacman-dev] [PATCH 1/2] Added a new database backend
by Sivert Berg 18 Dec '08

18 Dec '08

This is an attempt to add a one-file-per-database backend. The main reason I wrote this was because load times of the database on my laptop when it's not yet cached is around 40-50 seconds. When you just want to do a quick "pacman -Ss" that can be a little annoying. I know you have discussed the current database vs. one-file earlier, and I think someone said another backend could be supported too. This first patch contains the main database code and a program to convert .db.tar.gz and file databases into the new format. Signed-off-by: Sivert Berg <sivertb(a)stud.ntnu.no> --- lib/libalpm/be_packed.h | 757 +++++++++++++++++++++++++++++++++++++++++++++++ src/util/Makefile.am | 4 +- src/util/convdb.c | 236 +++++++++++++++ 3 files changed, 996 insertions(+), 1 deletions(-) create mode 100644 lib/libalpm/be_packed.h create mode 100644 src/util/convdb.c diff --git a/lib/libalpm/be_packed.h b/lib/libalpm/be_packed.h new file mode 100644 index 0000000..e118a3a --- /dev/null +++ b/lib/libalpm/be_packed.h @@ -0,0 +1,757 @@ +/* + * Copyright (c) 2008 Sivert Berg + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +/* _HOW IT WORKS_ + * + * If you've ever looked at the ext2 filesystem you will find alot + * of similarities (among others the inode that I didn't have the + * imagination to find a new name for). + * + * _GROUP_ + * Basicly the database is divided into groups. Each group got the + * same number of blocks and inodes. + * + * _INODE_ + * Each file is assigned an unique inode. The inode got a list of + * blocks where the actual data for the file is stored. Worth noting + * is that the last entry in the block list is indirect. That means + * it points to a block that points to blocks. + */ + +#include <stdint.h> +#include <sys/mman.h> +#include <stdlib.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <stdio.h> +#include <string.h> +#include <unistd.h> + + +#ifndef BE_PACKED_H +#define BE_PACKED_H + +#define BLOCK_SIZE 256 +#define GROUP_SIZE (BLOCK_SIZE * 4096) +#define BLOCKS_PER_GROUP 3936 +#define INODES_PER_GROUP 1024 + +#define MIN(a, b) (((a) < (b))?(a):(b)) + +/* _pack_open_file flags */ +#define PO_READ 1 +#define PO_WRITE 2 + + +typedef struct __inode_t { + int32_t size; + int32_t block[7]; +} inode_t; + +typedef struct __group_desc_t { + uint32_t inodes_free; + uint32_t blocks_free; + uint32_t checksum; + inode_t i_table[INODES_PER_GROUP]; + uint32_t bitmap[BLOCKS_PER_GROUP / 32]; + char data[]; +} group_desc_t; + +typedef struct __package_t { + int32_t inode; + uint16_t rec_len; + uint16_t name_len; + char name[]; +} package_t; + +typedef struct __pack_db_t { + int fd; + int groups; + int read_only; + group_desc_t *map; +} pack_db_t; + +typedef struct __pack_file_t { + int inode; + size_t pos; +} pack_file_t; + +typedef struct __pack_dir_t { + size_t pos; +} pack_dir_t; + + +/** + ** GROUP FUNCTIONS + */ +/* Initilizes a new group */ +void _pack_init_group(group_desc_t *group) +{ + int i, j; + + for (i = 0; i < INODES_PER_GROUP; i++) { + group->i_table[i].size = -1; + for (j = 0; j < 7; j++) + group->i_table[i].block[j] = -1; + } + + for (i = 0; i < BLOCKS_PER_GROUP / 32; i++) { + group->bitmap[i] = 0; + } + + group->inodes_free = INODES_PER_GROUP; + group->blocks_free = BLOCKS_PER_GROUP; + group->checksum = 0; +} + + +/* Adds a new group to the file */ +void _pack_new_group(pack_db_t *db) +{ + db->groups++; + ftruncate(db->fd, db->groups * GROUP_SIZE); + db->map = mremap(db->map, (db->groups - 1) * GROUP_SIZE, + db->groups * GROUP_SIZE, MREMAP_MAYMOVE); + + _pack_init_group((group_desc_t*)((char*)db->map + + GROUP_SIZE * (db->groups - 1))); +} + + +/** + ** POINTER FUNCTIONS (return pointers to various structures) + */ +group_desc_t *_pack_get_group(pack_db_t *db, int group) +{ + return (group_desc_t*)((char*)db->map + group*GROUP_SIZE); +} + + +inode_t *_pack_get_inode(pack_db_t *db, int inode) +{ + int group = inode / INODES_PER_GROUP; + group_desc_t *groupd = _pack_get_group(db, group); + inode %= INODES_PER_GROUP; + + return groupd->i_table + inode; +} + +void *_pack_get_block(pack_db_t *db, int block) +{ + if (block == -1) + return NULL; + + group_desc_t *group = _pack_get_group(db, block / BLOCKS_PER_GROUP); + return group->data + (block % BLOCKS_PER_GROUP) * BLOCK_SIZE; +} + + +/* Searches for a free block in the bitmap and marks the bit + * it finds as used. + */ +int _pack_check_bitmap(group_desc_t *group) +{ + int i = 0, j = 0, t = 0; + + for (i = 0; i < BLOCKS_PER_GROUP / 32; i++) { + if (group->bitmap[i] != 0xffffffff) { + t = group->bitmap[i]; + for (j = 0; j < 32 && (t & 1); j++, t >>= 1); + break; + } + } + + group->bitmap[i] |= 1 << j; + return i * 32 + j; +} + + +/* Clears the n'th bit */ +int _pack_bitmap_free(group_desc_t *group, int n) +{ + int i = n / 32; + int j = n % 32; + + group->bitmap[i] &= ~(i << j); + return 0; +} + + +/* Returns the number of a new unused block + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +int _pack_get_free_block(pack_db_t *db) +{ + int i; + group_desc_t *group; + int ret; + + for (i = 0; i < db->groups; i++) { + group = _pack_get_group(db, i); + if (group->blocks_free) { + group->blocks_free--; + ret = _pack_check_bitmap(group); + ret += i * BLOCKS_PER_GROUP; + return ret;; + } + } + + _pack_new_group(db); + group = _pack_get_group(db, i); + group->blocks_free--; + ret = _pack_check_bitmap(group); + ret += i * BLOCKS_PER_GROUP; + + return ret; +} + + +/* Frees the block given */ +int _pack_block_free(pack_db_t *db, int block) +{ + group_desc_t *group; + int g, b; + + g = block / BLOCKS_PER_GROUP; + b = block & BLOCKS_PER_GROUP; + + group = _pack_get_group(db, g); + group->blocks_free++; + + return _pack_bitmap_free(group, b); +} + + +/* Returns the number of a new free inode. + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +int _pack_get_free_inode(pack_db_t *db) +{ + int i = 0, j = 0; + group_desc_t *group; + + for (i = 0; i < db->groups; i++) { + group = _pack_get_group(db, i); + if (group->inodes_free) { + group->inodes_free--; + for (j = 0; j < INODES_PER_GROUP; j++) { + if (group->i_table[j].size == -1) { + group->i_table[j].size = 0; + return i * INODES_PER_GROUP + j; + } + } + } + } + + _pack_new_group(db); + group = _pack_get_group(db, i); + group->inodes_free--; + + return i * INODES_PER_GROUP; +} + + +/* Frees an inode and all of the blocks it might be using */ +int _pack_inode_free(pack_db_t *db, int in) +{ + inode_t *inode = _pack_get_inode(db, in); + group_desc_t *group = _pack_get_group(db, in / INODES_PER_GROUP); + int i; + int block; + int32_t *blocks; + int size = BLOCK_SIZE / sizeof(uint32_t) - 1; + + inode->size = -1; + + for (i = 0; i < 6; i++) { + if (inode->block[i] != -1) { + _pack_block_free(db, inode->block[i]); + inode->block[i] = -1; + } + } + + block = inode->block[6]; + blocks = _pack_get_block(db, block); + + while(blocks != NULL) { + for (i = 0; i < size; i++) { + if (blocks[i] != -1) + _pack_block_free(db, blocks[i]); + } + + _pack_block_free(db, block); + block = blocks[size]; + blocks = _pack_get_block(db, block); + } + + group->inodes_free++; + + return 0; +} + + +/* Returns the number of the block associated with the inode */ +int _pack_inode_get_block(pack_db_t *db, int i, int block) +{ + inode_t *inode = _pack_get_inode(db, i); + if (block < 6) + return inode->block[block]; + + block -= 6; + int size = BLOCK_SIZE / sizeof(uint32_t) - 1; + uint32_t *blocks = _pack_get_block(db, inode->block[6]); + + for (;blocks != NULL && block >= size; block -= size) + blocks = _pack_get_block(db, blocks[size]); + + if (blocks == NULL) + return -1; + + return blocks[block]; +} + + +/* Returns the number of the block block associated with the inode. + * If there is no such block, it gets allocated. + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +int _pack_inode_add_block(pack_db_t *db, int in, int block) +{ + inode_t *inode = _pack_get_inode(db, in); + int tmp; + + if (block < 6) { + if (inode->block[block] != -1) + return inode->block[block]; + + tmp = _pack_get_free_block(db); + inode = _pack_get_inode(db, in); + return inode->block[block] = tmp; + } + + block -= 6; + int size = BLOCK_SIZE / sizeof(int32_t) - 1; + int i; + int old_block; + int32_t *blocks; + + if (inode->block[6] == -1) { + tmp = _pack_get_free_block(db); + inode = _pack_get_inode(db, in); + inode->block[6] = tmp; + blocks = _pack_get_block(db, inode->block[6]); + for (i = 0; i < size + 1; i++) + blocks[i] = -1; + } + else + blocks = _pack_get_block(db, inode->block[6]); + + old_block = inode->block[6]; + + for (;block >= size; block -= size) { + if (blocks[size] == -1) { + tmp = _pack_get_free_block(db); + blocks = _pack_get_block(db, old_block); + blocks[size] = tmp; + old_block = tmp; + blocks = _pack_get_block(db, tmp); + for (i = 0; i < size + 1; i++) + blocks[i] = -1; + } + else { + old_block = blocks[size]; + blocks = _pack_get_block(db, blocks[size]); + } + } + + if (blocks[block] == -1) { + tmp = _pack_get_free_block(db); + blocks = _pack_get_block(db, old_block); + blocks[block] = tmp; + } + + return blocks[block]; +} + +/* Returns a pointer to the package with the name pkgname. + * Returns NULL if it's not found + */ +package_t *_pack_find_package(pack_db_t *db, const char *pkgname) +{ + inode_t *inode = _pack_get_inode(db, 0); + int s_len = strlen(pkgname); + package_t *pkg; + int block = 0; + int b_pos = 0; + + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, block)); + + while (block < (inode->size / BLOCK_SIZE)) { + if (pkg->inode != 0 && + pkg->name_len == s_len && + strncmp(pkgname, pkg->name, s_len) == 0) + break; + + b_pos += pkg->rec_len; + pkg = (package_t*)((char*)pkg + pkg->rec_len); + if (b_pos >= BLOCK_SIZE) { + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, ++block)); + b_pos = 0; + } + } + + return pkg; +} + +/* Creates a new package with the name pkgname. + * Returns a pointer to the new package. + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +package_t *_pack_add_package(pack_db_t *db, const char *pkgname) +{ + inode_t *inode = _pack_get_inode(db, 0); + int s_len = strlen(pkgname); + int len = s_len + sizeof(package_t); + package_t *pkg; + int block = 0; + int b_pos = 0; + + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, block)); + + while (pkg && block < (inode->size / BLOCK_SIZE)) { + if (pkg->rec_len >= len && pkg->inode == 0) + break; + + b_pos += pkg->rec_len; + pkg = (package_t*)((char*)pkg + pkg->rec_len); + if (b_pos >= BLOCK_SIZE) { + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, ++block)); + b_pos = 0; + } + } + + if (pkg == NULL) { + pkg = _pack_get_block(db, _pack_inode_add_block(db, 0, block)); + pkg->rec_len = BLOCK_SIZE; + pkg->inode = 0; + inode = _pack_get_inode(db, 0); + inode->size += BLOCK_SIZE; + } + + strncpy(pkg->name, pkgname, s_len); + if (pkg->rec_len > (len + sizeof(package_t) + 10)) { + package_t *foo = (package_t*)((char*)pkg + len); + foo->rec_len = pkg->rec_len - len; + foo->inode = 0; + pkg->rec_len = len; + } + + pkg->inode = -1; + pkg->name_len = s_len; + + return pkg; +} + + +/* Reads data from the file until buf is full or we hit a newline */ +int _pack_file_gets(pack_db_t *db, pack_file_t *file, char *buf, size_t size) +{ + inode_t *inode = _pack_get_inode(db, file->inode); + size_t i; + size_t b_pos = file->pos % BLOCK_SIZE; + const char *src = _pack_get_block(db, + _pack_inode_get_block(db, file->inode, file->pos / BLOCK_SIZE)); + int newline = 0; + + for (i = 0 + ;i < size && file->pos < inode->size && !newline + ;i++, b_pos++, file->pos++) { + if (b_pos >= BLOCK_SIZE) { + src = _pack_get_block(db, + _pack_inode_get_block(db, file->inode, file->pos / BLOCK_SIZE)); + b_pos = 0; + } + + buf[i] = src[b_pos]; + + if (src[b_pos] == '\n') + newline = 1; + } + + if (i < size - 1) + buf[i] = '\0'; + + return i; +} + + +/* Write the data in buf into the file + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +int _pack_file_puts(pack_db_t *db, pack_file_t *file, + const char *buf, size_t size) +{ + size_t i; + size_t b_pos = file->pos % BLOCK_SIZE; + char *dest = _pack_get_block(db, + _pack_inode_add_block(db, file->inode, file->pos / BLOCK_SIZE)); + int block; + + for (i = 0; i < size; i++, file->pos++, b_pos++) + { + if (b_pos >= BLOCK_SIZE) { + block = _pack_inode_add_block(db, file->inode, file->pos / BLOCK_SIZE); + dest = _pack_get_block(db, block); + b_pos = 0; + } + dest[b_pos] = buf[i]; + } + + inode_t *inode = _pack_get_inode(db, file->inode); + inode->size = file->pos; + + return i; +} + + +int _pack_file_eof(pack_db_t *db, pack_file_t *file) +{ + inode_t *inode = _pack_get_inode(db, file->inode); + return file->pos >= inode->size; +} + + +/* Opens a file in the database + * flags: if you pass PO_READ, it will return -1 + * if the file does not exist, if you pass PO_WRITE + * the file will be created if it does not exist + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +pack_file_t *_pack_open_file(pack_db_t *db, const char *pkgname, + const char *filename, int flags) +{ + char name[512]; + + snprintf(name, 512, "%s/%s", pkgname, filename); + + package_t *pkg = _pack_find_package(db, name); + int w = flags & PO_WRITE; + int tmp; + + if (w && db->read_only) + return NULL; + + if (pkg == NULL && w) + pkg = _pack_add_package(db, name); + else if (pkg == NULL) + return NULL; + + pack_file_t *ret = malloc(sizeof(pack_file_t)); + ret->pos = 0; + + if (pkg->inode == -1 && w) { + tmp = _pack_get_free_inode(db); + pkg = _pack_find_package(db, name); + ret->inode = pkg->inode = tmp; + } + else + ret->inode = pkg->inode; + + return ret; +} + + +/* Closes the file given */ +void _pack_close_file(pack_file_t *file) +{ + free(file); +} + + +/* Removes a package along with all it's files */ +int _pack_package_remove(pack_db_t *db, const char *pkgname) +{ + inode_t *inode = _pack_get_inode(db, 0); + int s_len = strlen(pkgname); + package_t *prev, *pkg; + int block = 0; + int b_pos = 0; + + if (db->read_only) + return -1; + + prev = NULL; + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, block)); + + while (block < (inode->size / BLOCK_SIZE)) { + if (pkg->inode != 0 && + pkg->name_len > s_len && + strncmp(pkgname, pkg->name, s_len) == 0 && + pkg->name[s_len] == '/') { + _pack_inode_free(db, pkg->inode); + + if (prev != NULL && prev->inode == 0) { + prev->rec_len += pkg->rec_len; + pkg = prev; + } + else + pkg->inode = 0; + } + + prev = pkg; + b_pos += pkg->rec_len; + pkg = (package_t*)((char*)pkg + pkg->rec_len); + if (b_pos >= BLOCK_SIZE) { + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, ++block)); + b_pos = 0; + prev = NULL; + } + } + + return 0; +} + + +/* Gets the next entry in the directory */ +int _pack_dir_next(pack_db_t *db, pack_dir_t *dir, char *name, size_t len) +{ + inode_t *inode = _pack_get_inode(db, 0); + + if (dir->pos >= inode->size) + return -1; + + int block, b_pos; + package_t *pkg; + + do { + block = dir->pos / BLOCK_SIZE; + b_pos = dir->pos % BLOCK_SIZE; + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, block)); + if (!pkg) + return -1; + pkg = (package_t*)((char*)pkg + b_pos); + dir->pos += pkg->rec_len; + } while (pkg->inode == 0); + + strncpy(name, pkg->name, MIN(len, pkg->name_len)); + if (len > pkg->name_len) + name[pkg->name_len] = '\0'; + + return 0; +} + + +/* Opens the directory for reading */ +pack_dir_t *_pack_open_dir(pack_db_t *db) +{ + pack_dir_t *dir = malloc(sizeof(pack_dir_t)); + dir->pos = 0; + + return dir; +} + + +/* Closes the given directory */ +void _pack_close_dir(pack_dir_t *dir) +{ + free(dir); +} + + +/* Opens a packed database */ +pack_db_t *_pack_db_open(const char *filename) +{ + struct stat st; + int is_new = 0; + + pack_db_t *db = malloc(sizeof(pack_db_t)); + db->fd = open(filename, O_RDWR, 0); + db->read_only = 0; + + if (db->fd == -1) { + db->fd = open(filename, O_RDONLY, 0); + if (db->fd == -1) { + db->fd = open(filename, O_RDWR | O_CREAT, 0644); + + if (db->fd == -1) { + free(db); + printf("Could not open %s\n", filename); + return NULL; + } + + ftruncate(db->fd, GROUP_SIZE); + is_new = 1; + } + else + db->read_only = 1; + } + + fstat(db->fd, &st); + db->groups = st.st_size / GROUP_SIZE; + + int flags = PROT_READ; + if (!db->read_only) + flags |= PROT_WRITE; + + db->map = mmap(NULL, GROUP_SIZE * db->groups, flags, MAP_SHARED, + db->fd, 0); + + if (db->map == MAP_FAILED) { + perror("mmap"); + free(db); + exit(0); + } + + if (is_new) { + _pack_init_group(db->map); + db->map->i_table[0].size = 0; + } + + return db; +} + + +/* Closes a packed database */ +void _pack_db_close(pack_db_t *db) +{ + munmap(db->map, db->groups * GROUP_SIZE); + close(db->fd); + free(db); +} + +#endif /* BE_PACKED_H */ diff --git a/src/util/Makefile.am b/src/util/Makefile.am index 97a0ffa..64e18ad 100644 --- a/src/util/Makefile.am +++ b/src/util/Makefile.am @@ -3,7 +3,7 @@ conffile = ${sysconfdir}/pacman.conf dbpath = ${localstatedir}/lib/pacman/ cachedir = ${localstatedir}/cache/pacman/pkg/ -bin_PROGRAMS = vercmp testpkg testdb +bin_PROGRAMS = vercmp testpkg testdb convdb DEFS = -DLOCALEDIR=\"@localedir@\" \ -DCONFFILE=\"$(conffile)\" \ @@ -18,6 +18,8 @@ AM_CFLAGS = -pedantic -D_GNU_SOURCE vercmp_SOURCES = vercmp.c vercmp_LDADD = $(top_builddir)/lib/libalpm/.libs/libalpm.la +convdb_SOURCES = convdb.c + testpkg_SOURCES = testpkg.c testpkg_LDADD = $(top_builddir)/lib/libalpm/.libs/libalpm.la diff --git a/src/util/convdb.c b/src/util/convdb.c new file mode 100644 index 0000000..f1f39bd --- /dev/null +++ b/src/util/convdb.c @@ -0,0 +1,236 @@ +/* + * Copyright (c) 2008 Sivert Berg + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + + +/* WARNING: This file is pretty much just a quick hack thrown together + * in a hurry. It seems to work, but don't take my word for it. + */ + +#include <be_packed.h> +#include <dirent.h> +#include <archive.h> +#include <archive_entry.h> + + +/* Convert an archieved database into a packed database + * Expects a gziped database ending in .db.tar.gz + */ +void convert_archive(const char *filename) +{ + struct archive_entry *aent = NULL; + struct archive *arch = archive_read_new(); + archive_read_support_compression_gzip(arch); + archive_read_support_format_tar(arch); + if (archive_read_open_filename(arch, filename, 1) != ARCHIVE_OK) { + printf("Could not open archive %s\n", archive_error_string(arch)); + return; + } + + char *db_name = strndup(filename, strlen(filename) - strlen(".tar.gz")); + pack_db_t *db = _pack_db_open(db_name); + free(db_name); + char buffer[4096]; + size_t size; + pack_file_t *pf; + + while (!archive_read_next_header(arch, &aent)) { + const char *type = NULL; + if (strstr(archive_entry_pathname(aent), "/desc")) + type = "desc"; + else if (strstr(archive_entry_pathname(aent), "/depends")) + type = "depends"; + else if (strstr(archive_entry_pathname(aent), "/install")) + type = "install"; + else if (strstr(archive_entry_pathname(aent), "/files")) + type = "files"; + + if (type == NULL) + continue; + + char *name = strdup(archive_entry_pathname(aent)); + int i; + for (i = 0; name[i] != '/' && name[i]; i++); + name[i] = '\0'; + + pf = _pack_open_file(db, name, type, PO_WRITE); + + free(name); + + do { + size = archive_read_data(arch, buffer, 4096); + _pack_file_puts(db, pf, buffer, size); + } while (size == 4096); + + _pack_close_file(pf); + + } + + _pack_db_close(db); + archive_read_close(arch); + archive_read_finish(arch); +} + + + +const char *dir_name(const char *full_path) +{ + const char *slash = full_path; + while (*full_path) + if (*full_path++ == '/') + slash = full_path; + return slash; +} + + +/* Convert a directory database into a packed database */ +void convert_directory(const char *pathname) +{ + char db_name[1024]; + pack_db_t *db; + DIR *dir = opendir(pathname); + struct dirent *dire; + char path[1024]; + char buffer[8196]; + size_t size; + FILE *f; + pack_file_t *pf; + + strcpy(db_name, dir_name(pathname)); + strcat(db_name, ".db"); + db = _pack_db_open(db_name); + + if (dir == NULL) { + perror("opendir"); + return; + } + + dire = readdir(dir); + while (dire != NULL) { + if (strcmp(dire->d_name, ".") && strcmp(dire->d_name, "..")) { + + + size_t foo = 0; + sprintf(path, "%s/%s/desc", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto depends; + pf = _pack_open_file(db, dire->d_name, "desc", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +depends: + sprintf(path, "%s/%s/depends", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto files; + pf = _pack_open_file(db, dire->d_name, "depends", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +files: + sprintf(path, "%s/%s/files", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto install; + pf = _pack_open_file(db, dire->d_name, "files", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +install: + sprintf(path, "%s/%s/install", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto changelog; + pf = _pack_open_file(db, dire->d_name, "install", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +changelog: + sprintf(path, "%s/%s/changelog", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto deltas; + pf = _pack_open_file(db, dire->d_name, "changelog", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +deltas: + sprintf(path, "%s/%s/deltas", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto done; + pf = _pack_open_file(db, dire->d_name, "deltas", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +done: + pf = NULL; + } + + dire = readdir(dir); + } + + closedir(dir); + _pack_db_close(db); +} + +int main(int argc, char *argv[]) +{ + if (argc < 2) { + printf("Usage: %s <dir/file>\n", argv[0]); + exit(0); + } + + if (strstr(argv[1], ".tar.gz") != NULL) + convert_archive(argv[1]); + else + convert_directory(argv[1]); + + return 0; +} -- 1.6.0.5

3 7

[pacman-dev] [patch]GPG verification
by Jatheendra 17 Dec '08

17 Dec '08

These patches will add VerifySig option to pacman.conf. VerifySig takes options Always, Optional or Never [repo-name] Server = ServerName VerifySig = Always Include = IncludePath >From 77be2c5cbfa3c7a750fe46d115c23096d2cf51e5 Mon Sep 17 00:00:00 2001 From: shankar <jatheendra(a)gmail.com> Date: Wed, 17 Dec 2008 20:52:21 +0530 Subject: [PATCH] changed gpg verification logic Signed-off-by: shankar <jatheendra(a)gmail.com> --- lib/libalpm/signing.c | 3 +++ lib/libalpm/sync.c | 26 ++++++++++++++++++++++---- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/lib/libalpm/signing.c b/lib/libalpm/signing.c index ddb89bc..0835b5e 100644 --- a/lib/libalpm/signing.c +++ b/lib/libalpm/signing.c @@ -166,6 +166,9 @@ pgpcheck_t _alpm_gpgme_checksig(const char *pkgpath, const pmpgpsig_t *sig) if(gpgsig->summary & GPGME_SIGSUM_VALID) { /* good signature, continue */ + ret = PM_PGP_SIG_VALID; + _alpm_log(PM_LOG_DEBUG, _("Package %s has a valid signature.\n"), + pkgpath); } else if(gpgsig->summary & GPGME_SIGSUM_GREEN) { /* 'green' signature, not sure what to do here */ _alpm_log(PM_LOG_WARNING, _("Package %s has a green signature.\n"), diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 24f2b98..f658ae2 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -901,12 +901,30 @@ int _alpm_sync_commit(pmtrans_t *trans, pmdb_t *db_local, alpm_list_t **data) *data = alpm_list_add(*data, strdup(filename)); } /* check PGP signature next */ - if(_alpm_gpgme_checksig(filepath, pgpsig) == PM_PGP_SIG_INVALID) { - errors++; - *data = alpm_list_add(*data, strdup(filename)); + pmdb_t *sdb = alpm_pkg_get_db(spkg); + + if(sdb->verify_gpg == PM_GPG_VERIFY_ALWAYS) { + if(_alpm_gpgme_checksig(filepath, pgpsig) != PM_PGP_SIG_VALID) { + errors++; + *data = alpm_list_add(*data, strdup(filename)); + _alpm_log(PM_LOG_ERROR, _("Invalid GPG signature on package: %s\n"),alpm_pkg_get_name(spkg)); + } + FREE(filepath); + } else if (sdb->verify_gpg == PM_GPG_VERIFY_OPTIONAL) { + pgpcheck_t ret1 = _alpm_gpgme_checksig(filepath, pgpsig); + + if(ret1 == PM_PGP_SIG_MISSING) { + /*no problems here*/ + } else if (ret1 != PM_PGP_SIG_VALID) { + errors++; + *data = alpm_list_add(*data, strdup(filename)); + _alpm_log(PM_LOG_ERROR, _("Invalid GPG signature on package: %s\n"),alpm_pkg_get_name(spkg)); + } + FREE(filepath); } - FREE(filepath); } + + if(errors) { pm_errno = PM_ERR_PKG_INVALID; goto error; -- 1.6.0.4

1 0

[pacman-dev] [PATCH] Added a new database backend
by Sivert Berg 16 Dec '08

16 Dec '08

This is an attempt to add a one-file-per-database backend. This leads to significatly faster loading of the database when the database is not cached. As you can see from the patch minimal changes to the rest of ALPM was needed. The only change was to make reading of the changelog files happen through the backend. The database is basically a scaled down ext2-like filesystem. To make corruption detection possible in an early phase I have planned to add a checksum, but have not come that far yet. I have been using it on my laptop for a week or so now, and everything seems to be working as expected. I'm sorry it became such a large patch, but about half of it is mostly old code (be_packed.c is essentially be_files.c with fopen fprintf etc changed with new functions). Signed-off-by: Sivert Berg <sivertb(a)stud.ntnu.no> --- configure.ac | 7 + lib/libalpm/Makefile.am | 2 +- lib/libalpm/be_files.c | 21 ++ lib/libalpm/be_packed.c | 845 +++++++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/be_packed.h | 757 ++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/db.h | 3 + lib/libalpm/package.c | 12 +- src/util/Makefile.am | 4 +- src/util/convdb.c | 236 +++++++++++++ 9 files changed, 1876 insertions(+), 11 deletions(-) create mode 100644 lib/libalpm/be_packed.c create mode 100644 lib/libalpm/be_packed.h create mode 100644 src/util/convdb.c diff --git a/configure.ac b/configure.ac index c7841b8..6e28ac9 100644 --- a/configure.ac +++ b/configure.ac @@ -73,6 +73,12 @@ AC_ARG_WITH(root-dir, AS_HELP_STRING([--with-root-dir=path], [set the location of pacman's root operating directory]), [ROOTDIR=$withval], [ROOTDIR=/]) +# Help line for database backend +AC_ARG_WITH(backend, + AS_HELP_STRING([--with-backend=backend], [choose which backend to use. You can choose between 'files' and 'packed', it defaults to files]), + [BACKEND=$withval], [BACKEND=files]) +AC_SUBST(BACKEND) + # Help line for package extension AC_ARG_WITH(pkg-ext, AS_HELP_STRING([--with-pkg-ext=ext], [set the file extension used by packages]), @@ -362,6 +368,7 @@ ${PACKAGE_NAME}: package extension : ${PKGEXT} source pkg extension : ${SRCEXT} database extension : ${DBEXT} + database backend : ${BACKEND} Compilation options: Run make in doc/ dir : ${wantdoc} diff --git a/lib/libalpm/Makefile.am b/lib/libalpm/Makefile.am index 871855e..79b9d26 100644 --- a/lib/libalpm/Makefile.am +++ b/lib/libalpm/Makefile.am @@ -25,7 +25,7 @@ libalpm_la_SOURCES = \ alpm.h alpm.c \ alpm_list.h alpm_list.c \ backup.h backup.c \ - be_files.c \ + be_@BACKEND@.c \ be_package.c \ cache.h cache.c \ conflict.h conflict.c \ diff --git a/lib/libalpm/be_files.c b/lib/libalpm/be_files.c index b9ff646..81d7a17 100644 --- a/lib/libalpm/be_files.c +++ b/lib/libalpm/be_files.c @@ -874,4 +874,25 @@ int _alpm_db_remove(pmdb_t *db, pmpkg_t *info) return(ret); } +void *_alpm_db_changelog_open(pmdb_t *db, pmpkg_t *pkg) +{ + char clfile[PATH_MAX]; + snprintf(clfile, PATH_MAX, "%s/%s/%s-%s/changelog", + alpm_option_get_dbpath(), + alpm_db_get_name(handle->db_local), + alpm_pkg_get_name(pkg), + alpm_pkg_get_version(pkg)); + return fopen(clfile, "r"); +} + +int _alpm_db_changelog_close(pmdb_t *db, void *stream) +{ + return fclose(stream); +} + +size_t _alpm_db_changelog_read(pmdb_t *db, const void *pf, void *ptr, size_t size) +{ + return fread(ptr, 1, size, pf); +} + /* vim: set ts=2 sw=2 noet: */ diff --git a/lib/libalpm/be_packed.c b/lib/libalpm/be_packed.c new file mode 100644 index 0000000..3e4ea68 --- /dev/null +++ b/lib/libalpm/be_packed.c @@ -0,0 +1,845 @@ +/* + * be_packed.c + * + * Copyright (c) 2006 by Christian Hamar <krics(a)linuxforum.hu> + * Copyright (c) 2006 by Miklos Vajna <vmiklos(a)frugalware.org> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include "config.h" + +#include <unistd.h> +#include <stdio.h> +#include <stdlib.h> +#include <errno.h> +#include <string.h> +#include <stdint.h> /* uintmax_t, intmax_t */ +#include <sys/stat.h> +#include <dirent.h> +#include <ctype.h> +#include <time.h> +#include <utime.h> +#include <limits.h> /* PATH_MAX */ +#include <locale.h> /* setlocale */ + +/* libalpm */ +#include "db.h" +#include "alpm_list.h" +#include "cache.h" +#include "log.h" +#include "util.h" +#include "alpm.h" +#include "handle.h" +#include "package.h" +#include "delta.h" +#include "deps.h" +#include "dload.h" + +#include "be_packed.h" + +/** Update a package database + * @param force if true, then forces the update, otherwise update only in case + * the database isn't up to date + * @param db pointer to the package database to update + * @return 0 on success, > 0 on error (pm_errno is set accordingly), < 0 if up + * to date + */ +int SYMEXPORT alpm_db_update(int force, pmdb_t *db) +{ + char *dbfile, *dbfilepath; + time_t newmtime = 0; + char dbpath[PATH_MAX]; + char dbname[PATH_MAX]; + char buffer[1024]; + size_t len; + struct stat st; + struct utimbuf utb; + + int ret; + + ALPM_LOG_FUNC; + + /* Sanity checks */ + ASSERT(handle != NULL, RET_ERR(PM_ERR_HANDLE_NULL, -1)); + ASSERT(db != NULL && db != handle->db_local, RET_ERR(PM_ERR_WRONG_ARGS, -1)); + /* Verify we are in a transaction. This is done _mainly_ because we need a DB + * lock - if we update without a db lock, we may kludge some other pacman + * process that _has_ a lock. + */ + ASSERT(handle->trans != NULL, RET_ERR(PM_ERR_TRANS_NULL, -1)); + ASSERT(handle->trans->state == STATE_INITIALIZED, RET_ERR(PM_ERR_TRANS_NOT_INITIALIZED, -1)); + ASSERT(handle->trans->type == PM_TRANS_TYPE_SYNC, RET_ERR(PM_ERR_TRANS_TYPE, -1)); + + if(!alpm_list_find_ptr(handle->dbs_sync, db)) { + RET_ERR(PM_ERR_DB_NOT_FOUND, -1); + } + + strncpy(dbname, db->path, PATH_MAX - 3); + /* Strip of the trailing / */ + dbname[strlen(dbname) - 1] = '\0'; + strcat(dbname, ".db"); + stat(dbname, &st); + + if(!force) { + /* get the lastupdate time */ + if(st.st_mtime == 0) { + _alpm_log(PM_LOG_DEBUG, "failed to get last write time for %s\n", + db->treename); + } + } + else + st.st_mtime = 0; + + len = strlen(db->treename) + strlen(DBEXT) + 1; + MALLOC(dbfile, len, RET_ERR(PM_ERR_MEMORY, -1)); + sprintf(dbfile, "%s" DBEXT, db->treename); + + strncpy(dbpath, alpm_option_get_dbpath(), PATH_MAX); + strcpy(dbpath + strlen(dbpath), "sync/"); + + ret = _alpm_download_single_file(dbfile, db->servers, dbpath, + st.st_mtime, &newmtime); + free(dbfile); + + if(ret == 1) { + /* mtimes match, do nothing */ + pm_errno = 0; + return(1); + } else if(ret == -1) { + /* pm_errno was set by the download code */ + _alpm_log(PM_LOG_DEBUG, "failed to sync db: %s\n", alpm_strerrorlast()); + return(-1); + } else { + /* remove the old db */ + _alpm_log(PM_LOG_DEBUG, "removing database %s\n", db->path); + unlink(dbname); + + /* cache needs to be rebuilt */ + _alpm_db_free_pkgcache(db); + + /* form the path to the db location */ + len = strlen(dbpath) + strlen(db->treename) + strlen(DBEXT) + 1; + MALLOC(dbfilepath, len, RET_ERR(PM_ERR_MEMORY, -1)); + sprintf(dbfilepath, "%s%s" DBEXT, dbpath, db->treename); + + /* convert the database file */ + snprintf(buffer, 1024, "convdb %s", dbfilepath); + system(buffer); + + /* set the correct time on the new database */ + utb.actime = newmtime; + utb.modtime = newmtime; + utime(dbname, &utb); + + /* Reopen the database */ + _pack_db_close(db->handle); + db->handle = _pack_db_open(dbname); + + unlink(dbfilepath); + free(dbfilepath); + } + + return(0); +} + +int _alpm_db_open(pmdb_t *db) +{ + ALPM_LOG_FUNC; + char path[PATH_MAX]; + + if(db == NULL) { + RET_ERR(PM_ERR_DB_NULL, -1); + } + + strncpy(path, db->path, PATH_MAX - 3); + /* Strip of the trailing / */ + path[strlen(path) - 1] = '\0'; + strcat(path, ".db"); + + _alpm_log(PM_LOG_DEBUG, "opening database from path '%s'\n", db->path); + db->handle = _pack_db_open(path); + + if(db->handle == NULL) { + RET_ERR(PM_ERR_DB_OPEN, -1); + } + + return(0); +} + +void _alpm_db_close(pmdb_t *db) +{ + ALPM_LOG_FUNC; + + if(db == NULL) { + return; + } + + if(db->handle) { + _pack_db_close(db->handle); + db->handle = NULL; + } +} + +static int splitname(const char *target, pmpkg_t *pkg) +{ + /* the format of a db entry is as follows: + * package-version-rel/ + * package name can contain hyphens, so parse from the back- go back + * two hyphens and we have split the version from the name. + */ + char *tmp, *p, *q; + + if(target == NULL || pkg == NULL) { + return(-1); + } + STRDUP(tmp, target, RET_ERR(PM_ERR_MEMORY, -1)); + p = tmp + strlen(tmp); + + /* do the magic parsing- find the beginning of the version string + * by doing two iterations of same loop to lop off two hyphens */ + for(q = --p; *q && *q != '-'; q--); + for(p = --q; *p && *p != '-'; p--); + if(*p != '-' || p == tmp) { + return(-1); + } + + /* copy into fields and return */ + if(pkg->version) { + FREE(pkg->version); + } + STRDUP(pkg->version, p+1, RET_ERR(PM_ERR_MEMORY, -1)); + /* insert a terminator at the end of the name (on hyphen)- then copy it */ + *p = '\0'; + if(pkg->name) { + FREE(pkg->name); + } + STRDUP(pkg->name, tmp, RET_ERR(PM_ERR_MEMORY, -1)); + + free(tmp); + return(0); +} + +int _alpm_db_populate(pmdb_t *db) +{ + int count = 0; + int i; + char name[PATH_MAX]; + pack_dir_t *dir; + + ALPM_LOG_FUNC; + + ASSERT(db != NULL, RET_ERR(PM_ERR_DB_NULL, -1)); + + dir = _pack_open_dir(db->handle); + + while(_pack_dir_next(db->handle, dir, name, PATH_MAX) != -1) { + pmpkg_t *pkg; + + pkg = _alpm_pkg_new(); + if(pkg == NULL) { + return(-1); + } + + /* Deal only with desc entries, that way we're sure we do + * not add the same package twice + */ + if(strstr(name, "/desc") == NULL) + continue; + + for (i = 0; name[i] && name[i] != '/'; i++); + name[i] = '\0'; + + /* split the db entry name */ + if(splitname(name, pkg) != 0) { + _alpm_log(PM_LOG_ERROR, _("invalid name for database entry '%s'\n"), + name); + _alpm_pkg_free(pkg); + continue; + } + + /* explicitly read with only 'BASE' data, accessors will handle the rest */ + if(_alpm_db_read(db, pkg, INFRQ_BASE) == -1) { + _alpm_log(PM_LOG_ERROR, _("corrupted database entry '%s'\n"), name); + _alpm_pkg_free(pkg); + continue; + } + pkg->origin = PKG_FROM_CACHE; + pkg->origin_data.db = db; + /* add to the collection */ + _alpm_log(PM_LOG_FUNCTION, "adding '%s' to package cache for db '%s'\n", + pkg->name, db->treename); + db->pkgcache = alpm_list_add(db->pkgcache, pkg); + count++; + } + + db->pkgcache = alpm_list_msort(db->pkgcache, count, _alpm_pkg_cmp); + return(count); +} + +int _alpm_db_read(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq) +{ + char line[513]; + char pkgname[PATH_MAX]; + pack_file_t *pf; + ALPM_LOG_FUNC; + + if(db == NULL) { + RET_ERR(PM_ERR_DB_NULL, -1); + } + + if(info == NULL || info->name == NULL || info->version == NULL) { + _alpm_log(PM_LOG_DEBUG, "invalid package entry provided to _alpm_db_read, skipping\n"); + return(-1); + } + + if(info->origin == PKG_FROM_FILE) { + _alpm_log(PM_LOG_DEBUG, "request to read database info for a file-based package '%s', skipping...\n", info->name); + return(-1); + } + + /* bitmask logic here: + * infolevel: 00001111 + * inforeq: 00010100 + * & result: 00000100 + * == to inforeq? nope, we need to load more info. */ + if((info->infolevel & inforeq) == inforeq) { + /* already loaded this info, do nothing */ + return(0); + } + _alpm_log(PM_LOG_FUNCTION, "loading package data for %s : level=0x%x\n", + info->name, inforeq); + + /* clear out 'line', to be certain - and to make valgrind happy */ + memset(line, 0, 513); + snprintf(pkgname, PATH_MAX, "%s-%s", info->name, info->version); + + /* DESC */ + if(inforeq & INFRQ_DESC) { + if((pf = _pack_open_file(db->handle, pkgname, "desc", PO_READ)) == NULL) { + _alpm_log(PM_LOG_ERROR, _("could not open package %s\n"), pkgname); + goto error; + } + while(!_pack_file_eof(db->handle, pf)) { + if(_pack_file_gets(db->handle, pf, line, 256) == 0) { + break; + } + _alpm_strtrim(line); + if(strcmp(line, "%NAME%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + if(strcmp(_alpm_strtrim(line), info->name) != 0) { + _alpm_log(PM_LOG_ERROR, _("%s database is inconsistent: name " + "mismatch on package %s %s\n"), db->treename, info->name, line); + } + } else if(strcmp(line, "%VERSION%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + if(strcmp(_alpm_strtrim(line), info->version) != 0) { + _alpm_log(PM_LOG_ERROR, _("%s database is inconsistent: version " + "mismatch on package %s\n"), db->treename, info->name); + } + } else if(strcmp(line, "%FILENAME%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + STRDUP(info->filename, _alpm_strtrim(line), goto error); + } else if(strcmp(line, "%DESC%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + STRDUP(info->desc, _alpm_strtrim(line), goto error); + } else if(strcmp(line, "%GROUPS%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + char *linedup; + STRDUP(linedup, _alpm_strtrim(line), goto error); + info->groups = alpm_list_add(info->groups, linedup); + } + } else if(strcmp(line, "%URL%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + STRDUP(info->url, _alpm_strtrim(line), goto error); + } else if(strcmp(line, "%LICENSE%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + char *linedup; + STRDUP(linedup, _alpm_strtrim(line), goto error); + info->licenses = alpm_list_add(info->licenses, linedup); + } + } else if(strcmp(line, "%ARCH%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + STRDUP(info->arch, _alpm_strtrim(line), goto error); + } else if(strcmp(line, "%BUILDDATE%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + _alpm_strtrim(line); + + char first = tolower(line[0]); + if(first > 'a' && first < 'z') { + struct tm tmp_tm = {0}; //initialize to null incase of failure + setlocale(LC_TIME, "C"); + strptime(line, "%a %b %e %H:%M:%S %Y", &tmp_tm); + info->builddate = mktime(&tmp_tm); + setlocale(LC_TIME, ""); + } else { + info->builddate = atol(line); + } + } else if(strcmp(line, "%INSTALLDATE%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + _alpm_strtrim(line); + + char first = tolower(line[0]); + if(first > 'a' && first < 'z') { + struct tm tmp_tm = {0}; //initialize to null incase of failure + setlocale(LC_TIME, "C"); + strptime(line, "%a %b %e %H:%M:%S %Y", &tmp_tm); + info->installdate = mktime(&tmp_tm); + setlocale(LC_TIME, ""); + } else { + info->installdate = atol(line); + } + } else if(strcmp(line, "%PACKAGER%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + STRDUP(info->packager, _alpm_strtrim(line), goto error); + } else if(strcmp(line, "%REASON%") == 0) { + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + info->reason = atol(_alpm_strtrim(line)); + } else if(strcmp(line, "%SIZE%") == 0 || strcmp(line, "%CSIZE%") == 0) { + /* NOTE: the CSIZE and SIZE fields both share the "size" field + * in the pkginfo_t struct. This can be done b/c CSIZE + * is currently only used in sync databases, and SIZE is + * only used in local databases. + */ + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + info->size = atol(_alpm_strtrim(line)); + /* also store this value to isize if isize is unset */ + if(info->isize == 0) { + info->isize = info->size; + } + } else if(strcmp(line, "%ISIZE%") == 0) { + /* ISIZE (installed size) tag only appears in sync repositories, + * not the local one. */ + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + info->isize = atol(_alpm_strtrim(line)); + } else if(strcmp(line, "%MD5SUM%") == 0) { + /* MD5SUM tag only appears in sync repositories, + * not the local one. */ + if(_pack_file_gets(db->handle, pf, line, 512) == 0) { + goto error; + } + STRDUP(info->md5sum, _alpm_strtrim(line), goto error); + } else if(strcmp(line, "%REPLACES%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + char *linedup; + STRDUP(linedup, _alpm_strtrim(line), goto error); + info->replaces = alpm_list_add(info->replaces, linedup); + } + } else if(strcmp(line, "%FORCE%") == 0) { + info->force = 1; + } + } + _pack_close_file(pf); + pf = NULL; + } + + /* FILES */ + if(inforeq & INFRQ_FILES) { + if((pf = _pack_open_file(db->handle, pkgname, "files", PO_READ)) == NULL) { + _alpm_log(PM_LOG_ERROR, _("could not open package %s\n"), pkgname); + goto error; + } + while(_pack_file_gets(db->handle, pf, line, 256) && + strlen(_alpm_strtrim(line))) { + _alpm_strtrim(line); + if(strcmp(line, "%FILES%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + char *linedup; + STRDUP(linedup, _alpm_strtrim(line), goto error); + info->files = alpm_list_add(info->files, linedup); + } + } else if(strcmp(line, "%BACKUP%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + char *linedup; + STRDUP(linedup, _alpm_strtrim(line), goto error); + info->backup = alpm_list_add(info->backup, linedup); + } + } + } + _pack_close_file(pf); + pf = NULL; + } + + /* DEPENDS */ + if(inforeq & INFRQ_DEPENDS) { + if((pf = _pack_open_file(db->handle, pkgname, "depends", PO_READ)) == NULL) { + _alpm_log(PM_LOG_ERROR, _("could not open package %s\n"), pkgname); + goto error; + } + while(!_pack_file_eof(db->handle, pf)) { + _pack_file_gets(db->handle, pf, line, 255); + _alpm_strtrim(line); + if(strcmp(line, "%DEPENDS%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + pmdepend_t *dep = _alpm_splitdep(_alpm_strtrim(line)); + info->depends = alpm_list_add(info->depends, dep); + } + } else if(strcmp(line, "%OPTDEPENDS%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + char *linedup; + STRDUP(linedup, _alpm_strtrim(line), goto error); + info->optdepends = alpm_list_add(info->optdepends, linedup); + } + } else if(strcmp(line, "%CONFLICTS%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + char *linedup; + STRDUP(linedup, _alpm_strtrim(line), goto error); + info->conflicts = alpm_list_add(info->conflicts, linedup); + } + } else if(strcmp(line, "%PROVIDES%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + char *linedup; + STRDUP(linedup, _alpm_strtrim(line), goto error); + info->provides = alpm_list_add(info->provides, linedup); + } + } + } + _pack_close_file(pf); + pf = NULL; + } + + /* DELTAS */ + if(inforeq & INFRQ_DELTAS) { + if((pf = _pack_open_file(db->handle, pkgname, "deltas", PO_READ)) != NULL) { + while(!_pack_file_eof(db->handle, pf)) { + _pack_file_gets(db->handle, pf, line, 255); + _alpm_strtrim(line); + if(strcmp(line, "%DELTAS%") == 0) { + while(_pack_file_gets(db->handle, pf, line, 512) && + strlen(_alpm_strtrim(line))) { + pmdelta_t *delta = _alpm_delta_parse(line); + if(delta) { + info->deltas = alpm_list_add(info->deltas, delta); + } + } + } + } + _pack_close_file(pf); + pf = NULL; + } + } + /* INSTALL */ + if(inforeq & INFRQ_SCRIPTLET) { + if((pf = _pack_open_file(db->handle, pkgname, "depends", PO_READ)) != NULL) { + info->scriptlet = 1; + _pack_close_file(pf); + } + } + + /* internal */ + info->infolevel |= inforeq; + + return(0); + +error: + if(pf) { + _pack_close_file(pf); + } + return(-1); +} + +int _alpm_db_write(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq) +{ + char pkgname[PATH_MAX]; + alpm_list_t *lp = NULL; + int retval = 0; + int local = 0; + pack_file_t *pf; + char buffer[1024]; + size_t len; + + ALPM_LOG_FUNC; + + if(db == NULL || info == NULL) { + return(-1); + } + + if(strcmp(db->treename, "local") == 0) { + local = 1; + } + + snprintf(pkgname, PATH_MAX, "%s-%s", info->name, info->version); + + /* DESC */ + if(inforeq & INFRQ_DESC) { + _alpm_log(PM_LOG_DEBUG, "writing %s-%s DESC information back to db\n", + info->name, info->version); + if((pf = _pack_open_file(db->handle, pkgname, "desc", PO_WRITE)) == NULL) { + _alpm_log(PM_LOG_ERROR, _("could not open package %s\n"), pkgname); + retval = -1; + goto cleanup; + } + + len = snprintf(buffer, 1024, "%%NAME%%\n%s\n\n" + "%%VERSION%%\n%s\n\n", info->name, info->version); + _pack_file_puts(db->handle, pf, buffer, len); + if(info->desc) { + len = snprintf(buffer, 1024, "%%DESC%%\n" + "%s\n\n", info->desc); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->groups) { + _pack_file_puts(db->handle, pf, "%GROUPS%\n", 9); + for(lp = info->groups; lp; lp = lp->next) { + len = snprintf(buffer, 1024, "%s\n", (char *)lp->data); + _pack_file_puts(db->handle, pf, buffer, len); + } + _pack_file_puts(db->handle, pf, "\n", 1); + } + if(info->replaces) { + _pack_file_puts(db->handle, pf, "%REPLACES%\n", 11); + for(lp = info->replaces; lp; lp = lp->next) { + len = snprintf(buffer, 1024, "%s\n", (char *)lp->data); + _pack_file_puts(db->handle, pf, buffer, len); + } + len = snprintf(buffer, 1024, "\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->force) { + len = snprintf(buffer, 1024, "%%FORCE%%\n\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(local) { + if(info->url) { + len = snprintf(buffer, 1024, "%%URL%%\n" + "%s\n\n", info->url); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->licenses) { + _pack_file_puts(db->handle, pf, "%LICENSE%\n", 10); + for(lp = info->licenses; lp; lp = lp->next) { + len = snprintf(buffer, 1024, "%s\n", (char *)lp->data); + _pack_file_puts(db->handle, pf, buffer, len); + } + len = snprintf(buffer, 1024, "\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->arch) { + len = snprintf(buffer, 1024, "%%ARCH%%\n" + "%s\n\n", info->arch); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->builddate) { + len = snprintf(buffer, 1024, "%%BUILDDATE%%\n" + "%ju\n\n", (uintmax_t)info->builddate); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->installdate) { + len = snprintf(buffer, 1024, "%%INSTALLDATE%%\n" + "%ju\n\n", (uintmax_t)info->installdate); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->packager) { + len = snprintf(buffer, 1024, "%%PACKAGER%%\n" + "%s\n\n", info->packager); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->isize) { + /* only write installed size, csize is irrelevant once installed */ + len = snprintf(buffer, 1024, "%%SIZE%%\n" + "%ju\n\n", (intmax_t)info->isize); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->reason) { + len = snprintf(buffer, 1024, "%%REASON%%\n" + "%u\n\n", info->reason); + _pack_file_puts(db->handle, pf, buffer, len); + } + } else { + if(info->size) { + len = snprintf(buffer, 1024, "%%CSIZE%%\n" + "%ju\n\n", (intmax_t)info->size); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->isize) { + len = snprintf(buffer, 1024, "%%ISIZE%%\n" + "%ju\n\n", (intmax_t)info->isize); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->md5sum) { + len = snprintf(buffer, 1024, "%%MD5SUM%%\n" + "%s\n\n", info->md5sum); + _pack_file_puts(db->handle, pf, buffer, len); + } + } + _pack_close_file(pf); + pf = NULL; + } + + /* FILES */ + if(local && (inforeq & INFRQ_FILES)) { + _alpm_log(PM_LOG_DEBUG, "writing %s-%s FILES information back to db\n", + info->name, info->version); + if((pf = _pack_open_file(db->handle, pkgname, "files", PO_WRITE)) == NULL) { + _alpm_log(PM_LOG_ERROR, _("could not open package %s\n"), pkgname); + retval = -1; + goto cleanup; + } + + if(info->files) { + len = snprintf(buffer, 1024, "%%FILES%%\n"); + _pack_file_puts(db->handle, pf, buffer, len); + for(lp = info->files; lp; lp = lp->next) { + len = snprintf(buffer, 1024, "%s\n", (char *)lp->data); + _pack_file_puts(db->handle, pf, buffer, len); + } + len = snprintf(buffer, 1024, "\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->backup) { + len = snprintf(buffer, 1024, "%%BACKUP%%\n"); + _pack_file_puts(db->handle, pf, buffer, len); + for(lp = info->backup; lp; lp = lp->next) { + len = snprintf(buffer, 1024, "%s\n", (char *)lp->data); + _pack_file_puts(db->handle, pf, buffer, len); + } + len = snprintf(buffer, 1024, "\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + _pack_close_file(pf); + pf = NULL; + } + + /* DEPENDS */ + if(inforeq & INFRQ_DEPENDS) { + _alpm_log(PM_LOG_DEBUG, "writing %s-%s DEPENDS information back to db\n", + info->name, info->version); + if((pf = _pack_open_file(db->handle, pkgname, "depends", PO_WRITE)) == NULL) { + _alpm_log(PM_LOG_ERROR, _("could not open package %s\n"), pkgname); + retval = -1; + goto cleanup; + } + + if(info->depends) { + _pack_file_puts(db->handle, pf, "%DEPENDS%\n", 10); + for(lp = info->depends; lp; lp = lp->next) { + char *depstring = alpm_dep_get_string(lp->data); + len = snprintf(buffer, 1024, "%s\n", depstring); + _pack_file_puts(db->handle, pf, buffer, len); + free(depstring); + } + len = snprintf(buffer, 1024, "\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->optdepends) { + _pack_file_puts(db->handle, pf, "%OPTDEPENDS%\n", 13); + for(lp = info->optdepends; lp; lp = lp->next) { + len = snprintf(buffer, 1024, "%s\n", (char *)lp->data); + _pack_file_puts(db->handle, pf, buffer, len); + } + len = snprintf(buffer, 1024, "\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->conflicts) { + _pack_file_puts(db->handle, pf, "%CONFLICTS%\n", 12); + for(lp = info->conflicts; lp; lp = lp->next) { + len = snprintf(buffer, 1024, "%s\n", (char *)lp->data); + _pack_file_puts(db->handle, pf, buffer, len); + } + len = snprintf(buffer, 1024, "\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + if(info->provides) { + _pack_file_puts(db->handle, pf, "%PROVIDES%\n", 11); + for(lp = info->provides; lp; lp = lp->next) { + len = snprintf(buffer, 1024, "%s\n", (char *)lp->data); + _pack_file_puts(db->handle, pf, buffer, len); + } + len = snprintf(buffer, 1024, "\n"); + _pack_file_puts(db->handle, pf, buffer, len); + } + _pack_close_file(pf); + pf = NULL; + } + + /* INSTALL */ + /* nothing needed here (script is automatically extracted) */ + +cleanup: + if(pf) { + _pack_close_file(pf); + } + + return(retval); +} + +int _alpm_db_remove(pmdb_t *db, pmpkg_t *info) +{ + int ret = 0; + char pkgname[512]; + + ALPM_LOG_FUNC; + + snprintf(pkgname, 512, "%s-%s", info->name, info->version); + _pack_package_remove(db->handle, pkgname); + + if(db == NULL || info == NULL) { + RET_ERR(PM_ERR_DB_NULL, -1); + } + + return(ret); +} + +void *_alpm_db_changelog_open(pmdb_t *db, pmpkg_t *pkg) +{ + char name[512]; + snprintf(name, 512, "%s-%s", pkg->name, pkg->version); + return _pack_open_file(db->handle, name, "changelog", PO_READ); +} + +int _alpm_db_changelog_close(pmdb_t *db, void *stream) +{ + _pack_close_file(stream); + return 0; +} + +size_t _alpm_db_changelog_read(pmdb_t *db, const void *pf, void *ptr, size_t size) +{ + return _pack_file_gets(db->handle, pf, ptr, size); +} + +/* vim: set ts=2 sw=2 noet: */ diff --git a/lib/libalpm/be_packed.h b/lib/libalpm/be_packed.h new file mode 100644 index 0000000..e118a3a --- /dev/null +++ b/lib/libalpm/be_packed.h @@ -0,0 +1,757 @@ +/* + * Copyright (c) 2008 Sivert Berg + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +/* _HOW IT WORKS_ + * + * If you've ever looked at the ext2 filesystem you will find alot + * of similarities (among others the inode that I didn't have the + * imagination to find a new name for). + * + * _GROUP_ + * Basicly the database is divided into groups. Each group got the + * same number of blocks and inodes. + * + * _INODE_ + * Each file is assigned an unique inode. The inode got a list of + * blocks where the actual data for the file is stored. Worth noting + * is that the last entry in the block list is indirect. That means + * it points to a block that points to blocks. + */ + +#include <stdint.h> +#include <sys/mman.h> +#include <stdlib.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <stdio.h> +#include <string.h> +#include <unistd.h> + + +#ifndef BE_PACKED_H +#define BE_PACKED_H + +#define BLOCK_SIZE 256 +#define GROUP_SIZE (BLOCK_SIZE * 4096) +#define BLOCKS_PER_GROUP 3936 +#define INODES_PER_GROUP 1024 + +#define MIN(a, b) (((a) < (b))?(a):(b)) + +/* _pack_open_file flags */ +#define PO_READ 1 +#define PO_WRITE 2 + + +typedef struct __inode_t { + int32_t size; + int32_t block[7]; +} inode_t; + +typedef struct __group_desc_t { + uint32_t inodes_free; + uint32_t blocks_free; + uint32_t checksum; + inode_t i_table[INODES_PER_GROUP]; + uint32_t bitmap[BLOCKS_PER_GROUP / 32]; + char data[]; +} group_desc_t; + +typedef struct __package_t { + int32_t inode; + uint16_t rec_len; + uint16_t name_len; + char name[]; +} package_t; + +typedef struct __pack_db_t { + int fd; + int groups; + int read_only; + group_desc_t *map; +} pack_db_t; + +typedef struct __pack_file_t { + int inode; + size_t pos; +} pack_file_t; + +typedef struct __pack_dir_t { + size_t pos; +} pack_dir_t; + + +/** + ** GROUP FUNCTIONS + */ +/* Initilizes a new group */ +void _pack_init_group(group_desc_t *group) +{ + int i, j; + + for (i = 0; i < INODES_PER_GROUP; i++) { + group->i_table[i].size = -1; + for (j = 0; j < 7; j++) + group->i_table[i].block[j] = -1; + } + + for (i = 0; i < BLOCKS_PER_GROUP / 32; i++) { + group->bitmap[i] = 0; + } + + group->inodes_free = INODES_PER_GROUP; + group->blocks_free = BLOCKS_PER_GROUP; + group->checksum = 0; +} + + +/* Adds a new group to the file */ +void _pack_new_group(pack_db_t *db) +{ + db->groups++; + ftruncate(db->fd, db->groups * GROUP_SIZE); + db->map = mremap(db->map, (db->groups - 1) * GROUP_SIZE, + db->groups * GROUP_SIZE, MREMAP_MAYMOVE); + + _pack_init_group((group_desc_t*)((char*)db->map + + GROUP_SIZE * (db->groups - 1))); +} + + +/** + ** POINTER FUNCTIONS (return pointers to various structures) + */ +group_desc_t *_pack_get_group(pack_db_t *db, int group) +{ + return (group_desc_t*)((char*)db->map + group*GROUP_SIZE); +} + + +inode_t *_pack_get_inode(pack_db_t *db, int inode) +{ + int group = inode / INODES_PER_GROUP; + group_desc_t *groupd = _pack_get_group(db, group); + inode %= INODES_PER_GROUP; + + return groupd->i_table + inode; +} + +void *_pack_get_block(pack_db_t *db, int block) +{ + if (block == -1) + return NULL; + + group_desc_t *group = _pack_get_group(db, block / BLOCKS_PER_GROUP); + return group->data + (block % BLOCKS_PER_GROUP) * BLOCK_SIZE; +} + + +/* Searches for a free block in the bitmap and marks the bit + * it finds as used. + */ +int _pack_check_bitmap(group_desc_t *group) +{ + int i = 0, j = 0, t = 0; + + for (i = 0; i < BLOCKS_PER_GROUP / 32; i++) { + if (group->bitmap[i] != 0xffffffff) { + t = group->bitmap[i]; + for (j = 0; j < 32 && (t & 1); j++, t >>= 1); + break; + } + } + + group->bitmap[i] |= 1 << j; + return i * 32 + j; +} + + +/* Clears the n'th bit */ +int _pack_bitmap_free(group_desc_t *group, int n) +{ + int i = n / 32; + int j = n % 32; + + group->bitmap[i] &= ~(i << j); + return 0; +} + + +/* Returns the number of a new unused block + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +int _pack_get_free_block(pack_db_t *db) +{ + int i; + group_desc_t *group; + int ret; + + for (i = 0; i < db->groups; i++) { + group = _pack_get_group(db, i); + if (group->blocks_free) { + group->blocks_free--; + ret = _pack_check_bitmap(group); + ret += i * BLOCKS_PER_GROUP; + return ret;; + } + } + + _pack_new_group(db); + group = _pack_get_group(db, i); + group->blocks_free--; + ret = _pack_check_bitmap(group); + ret += i * BLOCKS_PER_GROUP; + + return ret; +} + + +/* Frees the block given */ +int _pack_block_free(pack_db_t *db, int block) +{ + group_desc_t *group; + int g, b; + + g = block / BLOCKS_PER_GROUP; + b = block & BLOCKS_PER_GROUP; + + group = _pack_get_group(db, g); + group->blocks_free++; + + return _pack_bitmap_free(group, b); +} + + +/* Returns the number of a new free inode. + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +int _pack_get_free_inode(pack_db_t *db) +{ + int i = 0, j = 0; + group_desc_t *group; + + for (i = 0; i < db->groups; i++) { + group = _pack_get_group(db, i); + if (group->inodes_free) { + group->inodes_free--; + for (j = 0; j < INODES_PER_GROUP; j++) { + if (group->i_table[j].size == -1) { + group->i_table[j].size = 0; + return i * INODES_PER_GROUP + j; + } + } + } + } + + _pack_new_group(db); + group = _pack_get_group(db, i); + group->inodes_free--; + + return i * INODES_PER_GROUP; +} + + +/* Frees an inode and all of the blocks it might be using */ +int _pack_inode_free(pack_db_t *db, int in) +{ + inode_t *inode = _pack_get_inode(db, in); + group_desc_t *group = _pack_get_group(db, in / INODES_PER_GROUP); + int i; + int block; + int32_t *blocks; + int size = BLOCK_SIZE / sizeof(uint32_t) - 1; + + inode->size = -1; + + for (i = 0; i < 6; i++) { + if (inode->block[i] != -1) { + _pack_block_free(db, inode->block[i]); + inode->block[i] = -1; + } + } + + block = inode->block[6]; + blocks = _pack_get_block(db, block); + + while(blocks != NULL) { + for (i = 0; i < size; i++) { + if (blocks[i] != -1) + _pack_block_free(db, blocks[i]); + } + + _pack_block_free(db, block); + block = blocks[size]; + blocks = _pack_get_block(db, block); + } + + group->inodes_free++; + + return 0; +} + + +/* Returns the number of the block associated with the inode */ +int _pack_inode_get_block(pack_db_t *db, int i, int block) +{ + inode_t *inode = _pack_get_inode(db, i); + if (block < 6) + return inode->block[block]; + + block -= 6; + int size = BLOCK_SIZE / sizeof(uint32_t) - 1; + uint32_t *blocks = _pack_get_block(db, inode->block[6]); + + for (;blocks != NULL && block >= size; block -= size) + blocks = _pack_get_block(db, blocks[size]); + + if (blocks == NULL) + return -1; + + return blocks[block]; +} + + +/* Returns the number of the block block associated with the inode. + * If there is no such block, it gets allocated. + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +int _pack_inode_add_block(pack_db_t *db, int in, int block) +{ + inode_t *inode = _pack_get_inode(db, in); + int tmp; + + if (block < 6) { + if (inode->block[block] != -1) + return inode->block[block]; + + tmp = _pack_get_free_block(db); + inode = _pack_get_inode(db, in); + return inode->block[block] = tmp; + } + + block -= 6; + int size = BLOCK_SIZE / sizeof(int32_t) - 1; + int i; + int old_block; + int32_t *blocks; + + if (inode->block[6] == -1) { + tmp = _pack_get_free_block(db); + inode = _pack_get_inode(db, in); + inode->block[6] = tmp; + blocks = _pack_get_block(db, inode->block[6]); + for (i = 0; i < size + 1; i++) + blocks[i] = -1; + } + else + blocks = _pack_get_block(db, inode->block[6]); + + old_block = inode->block[6]; + + for (;block >= size; block -= size) { + if (blocks[size] == -1) { + tmp = _pack_get_free_block(db); + blocks = _pack_get_block(db, old_block); + blocks[size] = tmp; + old_block = tmp; + blocks = _pack_get_block(db, tmp); + for (i = 0; i < size + 1; i++) + blocks[i] = -1; + } + else { + old_block = blocks[size]; + blocks = _pack_get_block(db, blocks[size]); + } + } + + if (blocks[block] == -1) { + tmp = _pack_get_free_block(db); + blocks = _pack_get_block(db, old_block); + blocks[block] = tmp; + } + + return blocks[block]; +} + +/* Returns a pointer to the package with the name pkgname. + * Returns NULL if it's not found + */ +package_t *_pack_find_package(pack_db_t *db, const char *pkgname) +{ + inode_t *inode = _pack_get_inode(db, 0); + int s_len = strlen(pkgname); + package_t *pkg; + int block = 0; + int b_pos = 0; + + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, block)); + + while (block < (inode->size / BLOCK_SIZE)) { + if (pkg->inode != 0 && + pkg->name_len == s_len && + strncmp(pkgname, pkg->name, s_len) == 0) + break; + + b_pos += pkg->rec_len; + pkg = (package_t*)((char*)pkg + pkg->rec_len); + if (b_pos >= BLOCK_SIZE) { + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, ++block)); + b_pos = 0; + } + } + + return pkg; +} + +/* Creates a new package with the name pkgname. + * Returns a pointer to the new package. + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +package_t *_pack_add_package(pack_db_t *db, const char *pkgname) +{ + inode_t *inode = _pack_get_inode(db, 0); + int s_len = strlen(pkgname); + int len = s_len + sizeof(package_t); + package_t *pkg; + int block = 0; + int b_pos = 0; + + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, block)); + + while (pkg && block < (inode->size / BLOCK_SIZE)) { + if (pkg->rec_len >= len && pkg->inode == 0) + break; + + b_pos += pkg->rec_len; + pkg = (package_t*)((char*)pkg + pkg->rec_len); + if (b_pos >= BLOCK_SIZE) { + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, ++block)); + b_pos = 0; + } + } + + if (pkg == NULL) { + pkg = _pack_get_block(db, _pack_inode_add_block(db, 0, block)); + pkg->rec_len = BLOCK_SIZE; + pkg->inode = 0; + inode = _pack_get_inode(db, 0); + inode->size += BLOCK_SIZE; + } + + strncpy(pkg->name, pkgname, s_len); + if (pkg->rec_len > (len + sizeof(package_t) + 10)) { + package_t *foo = (package_t*)((char*)pkg + len); + foo->rec_len = pkg->rec_len - len; + foo->inode = 0; + pkg->rec_len = len; + } + + pkg->inode = -1; + pkg->name_len = s_len; + + return pkg; +} + + +/* Reads data from the file until buf is full or we hit a newline */ +int _pack_file_gets(pack_db_t *db, pack_file_t *file, char *buf, size_t size) +{ + inode_t *inode = _pack_get_inode(db, file->inode); + size_t i; + size_t b_pos = file->pos % BLOCK_SIZE; + const char *src = _pack_get_block(db, + _pack_inode_get_block(db, file->inode, file->pos / BLOCK_SIZE)); + int newline = 0; + + for (i = 0 + ;i < size && file->pos < inode->size && !newline + ;i++, b_pos++, file->pos++) { + if (b_pos >= BLOCK_SIZE) { + src = _pack_get_block(db, + _pack_inode_get_block(db, file->inode, file->pos / BLOCK_SIZE)); + b_pos = 0; + } + + buf[i] = src[b_pos]; + + if (src[b_pos] == '\n') + newline = 1; + } + + if (i < size - 1) + buf[i] = '\0'; + + return i; +} + + +/* Write the data in buf into the file + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +int _pack_file_puts(pack_db_t *db, pack_file_t *file, + const char *buf, size_t size) +{ + size_t i; + size_t b_pos = file->pos % BLOCK_SIZE; + char *dest = _pack_get_block(db, + _pack_inode_add_block(db, file->inode, file->pos / BLOCK_SIZE)); + int block; + + for (i = 0; i < size; i++, file->pos++, b_pos++) + { + if (b_pos >= BLOCK_SIZE) { + block = _pack_inode_add_block(db, file->inode, file->pos / BLOCK_SIZE); + dest = _pack_get_block(db, block); + b_pos = 0; + } + dest[b_pos] = buf[i]; + } + + inode_t *inode = _pack_get_inode(db, file->inode); + inode->size = file->pos; + + return i; +} + + +int _pack_file_eof(pack_db_t *db, pack_file_t *file) +{ + inode_t *inode = _pack_get_inode(db, file->inode); + return file->pos >= inode->size; +} + + +/* Opens a file in the database + * flags: if you pass PO_READ, it will return -1 + * if the file does not exist, if you pass PO_WRITE + * the file will be created if it does not exist + * CAUTION: THIS FUNCTION MAY RESULT IN A MMREMAP! + * that means ANY pointers into the database may be invalid + * after you've called this function. + */ +pack_file_t *_pack_open_file(pack_db_t *db, const char *pkgname, + const char *filename, int flags) +{ + char name[512]; + + snprintf(name, 512, "%s/%s", pkgname, filename); + + package_t *pkg = _pack_find_package(db, name); + int w = flags & PO_WRITE; + int tmp; + + if (w && db->read_only) + return NULL; + + if (pkg == NULL && w) + pkg = _pack_add_package(db, name); + else if (pkg == NULL) + return NULL; + + pack_file_t *ret = malloc(sizeof(pack_file_t)); + ret->pos = 0; + + if (pkg->inode == -1 && w) { + tmp = _pack_get_free_inode(db); + pkg = _pack_find_package(db, name); + ret->inode = pkg->inode = tmp; + } + else + ret->inode = pkg->inode; + + return ret; +} + + +/* Closes the file given */ +void _pack_close_file(pack_file_t *file) +{ + free(file); +} + + +/* Removes a package along with all it's files */ +int _pack_package_remove(pack_db_t *db, const char *pkgname) +{ + inode_t *inode = _pack_get_inode(db, 0); + int s_len = strlen(pkgname); + package_t *prev, *pkg; + int block = 0; + int b_pos = 0; + + if (db->read_only) + return -1; + + prev = NULL; + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, block)); + + while (block < (inode->size / BLOCK_SIZE)) { + if (pkg->inode != 0 && + pkg->name_len > s_len && + strncmp(pkgname, pkg->name, s_len) == 0 && + pkg->name[s_len] == '/') { + _pack_inode_free(db, pkg->inode); + + if (prev != NULL && prev->inode == 0) { + prev->rec_len += pkg->rec_len; + pkg = prev; + } + else + pkg->inode = 0; + } + + prev = pkg; + b_pos += pkg->rec_len; + pkg = (package_t*)((char*)pkg + pkg->rec_len); + if (b_pos >= BLOCK_SIZE) { + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, ++block)); + b_pos = 0; + prev = NULL; + } + } + + return 0; +} + + +/* Gets the next entry in the directory */ +int _pack_dir_next(pack_db_t *db, pack_dir_t *dir, char *name, size_t len) +{ + inode_t *inode = _pack_get_inode(db, 0); + + if (dir->pos >= inode->size) + return -1; + + int block, b_pos; + package_t *pkg; + + do { + block = dir->pos / BLOCK_SIZE; + b_pos = dir->pos % BLOCK_SIZE; + pkg = _pack_get_block(db, _pack_inode_get_block(db, 0, block)); + if (!pkg) + return -1; + pkg = (package_t*)((char*)pkg + b_pos); + dir->pos += pkg->rec_len; + } while (pkg->inode == 0); + + strncpy(name, pkg->name, MIN(len, pkg->name_len)); + if (len > pkg->name_len) + name[pkg->name_len] = '\0'; + + return 0; +} + + +/* Opens the directory for reading */ +pack_dir_t *_pack_open_dir(pack_db_t *db) +{ + pack_dir_t *dir = malloc(sizeof(pack_dir_t)); + dir->pos = 0; + + return dir; +} + + +/* Closes the given directory */ +void _pack_close_dir(pack_dir_t *dir) +{ + free(dir); +} + + +/* Opens a packed database */ +pack_db_t *_pack_db_open(const char *filename) +{ + struct stat st; + int is_new = 0; + + pack_db_t *db = malloc(sizeof(pack_db_t)); + db->fd = open(filename, O_RDWR, 0); + db->read_only = 0; + + if (db->fd == -1) { + db->fd = open(filename, O_RDONLY, 0); + if (db->fd == -1) { + db->fd = open(filename, O_RDWR | O_CREAT, 0644); + + if (db->fd == -1) { + free(db); + printf("Could not open %s\n", filename); + return NULL; + } + + ftruncate(db->fd, GROUP_SIZE); + is_new = 1; + } + else + db->read_only = 1; + } + + fstat(db->fd, &st); + db->groups = st.st_size / GROUP_SIZE; + + int flags = PROT_READ; + if (!db->read_only) + flags |= PROT_WRITE; + + db->map = mmap(NULL, GROUP_SIZE * db->groups, flags, MAP_SHARED, + db->fd, 0); + + if (db->map == MAP_FAILED) { + perror("mmap"); + free(db); + exit(0); + } + + if (is_new) { + _pack_init_group(db->map); + db->map->i_table[0].size = 0; + } + + return db; +} + + +/* Closes a packed database */ +void _pack_db_close(pack_db_t *db) +{ + munmap(db->map, db->groups * GROUP_SIZE); + close(db->fd); + free(db); +} + +#endif /* BE_PACKED_H */ diff --git a/lib/libalpm/db.h b/lib/libalpm/db.h index 96fac0d..96291fc 100644 --- a/lib/libalpm/db.h +++ b/lib/libalpm/db.h @@ -62,6 +62,9 @@ int _alpm_db_populate(pmdb_t *db); int _alpm_db_read(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq); int _alpm_db_write(pmdb_t *db, pmpkg_t *info, pmdbinfrq_t inforeq); int _alpm_db_remove(pmdb_t *db, pmpkg_t *info); +void *_alpm_db_changelog_open(pmdb_t *db, pmpkg_t *pkg); +int _alpm_db_changelog_close(pmdb_t *db, void *stream); +size_t _alpm_db_changelog_read(pmdb_t *db, const void *pf, void *ptr, size_t size); #endif /* _ALPM_DB_H */ diff --git a/lib/libalpm/package.c b/lib/libalpm/package.c index ee0ff6f..7ab9710 100644 --- a/lib/libalpm/package.c +++ b/lib/libalpm/package.c @@ -447,13 +447,7 @@ void SYMEXPORT *alpm_pkg_changelog_open(pmpkg_t *pkg) ASSERT(pkg != NULL, return(NULL)); if(pkg->origin == PKG_FROM_CACHE) { - char clfile[PATH_MAX]; - snprintf(clfile, PATH_MAX, "%s/%s/%s-%s/changelog", - alpm_option_get_dbpath(), - alpm_db_get_name(handle->db_local), - alpm_pkg_get_name(pkg), - alpm_pkg_get_version(pkg)); - return fopen(clfile, "r"); + return(_alpm_db_changelog_open(handle->db_local, pkg)); } else if(pkg->origin == PKG_FROM_FILE) { struct archive *archive = NULL; struct archive_entry *entry; @@ -500,7 +494,7 @@ size_t SYMEXPORT alpm_pkg_changelog_read(void *ptr, size_t size, { size_t ret = 0; if(pkg->origin == PKG_FROM_CACHE) { - ret = fread(ptr, 1, size, (FILE*)fp); + ret = _alpm_db_changelog_read(handle->db_local, fp, ptr, size); } else if(pkg->origin == PKG_FROM_FILE) { ret = archive_read_data((struct archive*)fp, ptr, size); } @@ -533,7 +527,7 @@ int SYMEXPORT alpm_pkg_changelog_close(const pmpkg_t *pkg, void *fp) { int ret = 0; if(pkg->origin == PKG_FROM_CACHE) { - ret = fclose((FILE*)fp); + ret = _alpm_db_changelog_close(handle->db_local, fp); } else if(pkg->origin == PKG_FROM_FILE) { ret = archive_read_finish((struct archive *)fp); } diff --git a/src/util/Makefile.am b/src/util/Makefile.am index 97a0ffa..64e18ad 100644 --- a/src/util/Makefile.am +++ b/src/util/Makefile.am @@ -3,7 +3,7 @@ conffile = ${sysconfdir}/pacman.conf dbpath = ${localstatedir}/lib/pacman/ cachedir = ${localstatedir}/cache/pacman/pkg/ -bin_PROGRAMS = vercmp testpkg testdb +bin_PROGRAMS = vercmp testpkg testdb convdb DEFS = -DLOCALEDIR=\"@localedir@\" \ -DCONFFILE=\"$(conffile)\" \ @@ -18,6 +18,8 @@ AM_CFLAGS = -pedantic -D_GNU_SOURCE vercmp_SOURCES = vercmp.c vercmp_LDADD = $(top_builddir)/lib/libalpm/.libs/libalpm.la +convdb_SOURCES = convdb.c + testpkg_SOURCES = testpkg.c testpkg_LDADD = $(top_builddir)/lib/libalpm/.libs/libalpm.la diff --git a/src/util/convdb.c b/src/util/convdb.c new file mode 100644 index 0000000..f1f39bd --- /dev/null +++ b/src/util/convdb.c @@ -0,0 +1,236 @@ +/* + * Copyright (c) 2008 Sivert Berg + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + + +/* WARNING: This file is pretty much just a quick hack thrown together + * in a hurry. It seems to work, but don't take my word for it. + */ + +#include <be_packed.h> +#include <dirent.h> +#include <archive.h> +#include <archive_entry.h> + + +/* Convert an archieved database into a packed database + * Expects a gziped database ending in .db.tar.gz + */ +void convert_archive(const char *filename) +{ + struct archive_entry *aent = NULL; + struct archive *arch = archive_read_new(); + archive_read_support_compression_gzip(arch); + archive_read_support_format_tar(arch); + if (archive_read_open_filename(arch, filename, 1) != ARCHIVE_OK) { + printf("Could not open archive %s\n", archive_error_string(arch)); + return; + } + + char *db_name = strndup(filename, strlen(filename) - strlen(".tar.gz")); + pack_db_t *db = _pack_db_open(db_name); + free(db_name); + char buffer[4096]; + size_t size; + pack_file_t *pf; + + while (!archive_read_next_header(arch, &aent)) { + const char *type = NULL; + if (strstr(archive_entry_pathname(aent), "/desc")) + type = "desc"; + else if (strstr(archive_entry_pathname(aent), "/depends")) + type = "depends"; + else if (strstr(archive_entry_pathname(aent), "/install")) + type = "install"; + else if (strstr(archive_entry_pathname(aent), "/files")) + type = "files"; + + if (type == NULL) + continue; + + char *name = strdup(archive_entry_pathname(aent)); + int i; + for (i = 0; name[i] != '/' && name[i]; i++); + name[i] = '\0'; + + pf = _pack_open_file(db, name, type, PO_WRITE); + + free(name); + + do { + size = archive_read_data(arch, buffer, 4096); + _pack_file_puts(db, pf, buffer, size); + } while (size == 4096); + + _pack_close_file(pf); + + } + + _pack_db_close(db); + archive_read_close(arch); + archive_read_finish(arch); +} + + + +const char *dir_name(const char *full_path) +{ + const char *slash = full_path; + while (*full_path) + if (*full_path++ == '/') + slash = full_path; + return slash; +} + + +/* Convert a directory database into a packed database */ +void convert_directory(const char *pathname) +{ + char db_name[1024]; + pack_db_t *db; + DIR *dir = opendir(pathname); + struct dirent *dire; + char path[1024]; + char buffer[8196]; + size_t size; + FILE *f; + pack_file_t *pf; + + strcpy(db_name, dir_name(pathname)); + strcat(db_name, ".db"); + db = _pack_db_open(db_name); + + if (dir == NULL) { + perror("opendir"); + return; + } + + dire = readdir(dir); + while (dire != NULL) { + if (strcmp(dire->d_name, ".") && strcmp(dire->d_name, "..")) { + + + size_t foo = 0; + sprintf(path, "%s/%s/desc", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto depends; + pf = _pack_open_file(db, dire->d_name, "desc", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +depends: + sprintf(path, "%s/%s/depends", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto files; + pf = _pack_open_file(db, dire->d_name, "depends", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +files: + sprintf(path, "%s/%s/files", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto install; + pf = _pack_open_file(db, dire->d_name, "files", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +install: + sprintf(path, "%s/%s/install", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto changelog; + pf = _pack_open_file(db, dire->d_name, "install", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +changelog: + sprintf(path, "%s/%s/changelog", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto deltas; + pf = _pack_open_file(db, dire->d_name, "changelog", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +deltas: + sprintf(path, "%s/%s/deltas", pathname, dire->d_name); + f = fopen(path, "r"); + if (f == NULL) + goto done; + pf = _pack_open_file(db, dire->d_name, "deltas", PO_WRITE); + do { + size = fread(buffer, 1, 8196, f); + _pack_file_puts(db, pf, buffer, size); + foo += size; + } while (size == 8196); + _pack_close_file(pf); + fclose(f); +done: + pf = NULL; + } + + dire = readdir(dir); + } + + closedir(dir); + _pack_db_close(db); +} + +int main(int argc, char *argv[]) +{ + if (argc < 2) { + printf("Usage: %s <dir/file>\n", argv[0]); + exit(0); + } + + if (strstr(argv[1], ".tar.gz") != NULL) + convert_archive(argv[1]); + else + convert_directory(argv[1]); + + return 0; +} -- 1.6.0.5

3 4

[pacman-dev] [PATCH] makepkg: ensure PKGBUILD does not contain CRLF characters
by Dan McGee 16 Dec '08

16 Dec '08

Do a simple check before sourcing the file to ensure we are a valid bash script. Signed-off-by: Dan McGee <dan(a)archlinux.org> --- scripts/makepkg.sh.in | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index b889625..179746d 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -1354,6 +1354,12 @@ if [ ! -f "$BUILDSCRIPT" ]; then # PKGBUILD passed through a pipe BUILDSCRIPT=/dev/stdin fi +else + crlftest=$(file $BUILDSCRIPT | grep -F 'CRLF' || true) + if [ "$crlftest" != "" ]; then + error "$(gettext "%s contains CRLF characters and cannot be sourced.")" "$BUILDSCRIPT" + exit 1 + fi fi source "$BUILDSCRIPT" -- 1.6.0.4

4 3

[pacman-dev] GPG work
by Dan McGee 15 Dec '08

15 Dec '08

I did quite a bit more work with GPG today. I wrapped my head around GPGME, which presents a nice C interface to the GPG stuff so we are now a lot closer to a working implementation: http://code.toofishes.net/gitweb.cgi?p=pacman.git;a=shortlog;h=refs/heads/n… >From the script side of things, I didn't change much. The libalpm code has changed considerably, and there is still a lot of room for improvement. Let me know if you guys have questions. -Dan

8 13

Re: [pacman-dev] makepkg: package splitting
by Marc - A. Dahlhaus [ Administration | Westermann GmbH ] 15 Dec '08

15 Dec '08

Am Montag, den 15.12.2008, 22:24 +1000 schrieb Allan McRae: > Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote: > > Am Montag, den 15.12.2008, 21:43 +1000 schrieb Allan McRae: > > > > Nice, i'll test and report back... > > Is there any schedule on when it will be merged to mainline? > > > > > > The current schedule is when I tell Dan it ready... I want to add > documentation and clean up some of the output with the package splitting > but otherwise I am happy with it now. So, the answer is "soonish". > > Allan If you need any help, be it implementation of missing functionality or testing of something, let me know. I'll also try to come up with a option to build only a subset of the split-packages (if it makes sense) but haven't looked deeper into it yet. Marc

1 0

Re: [pacman-dev] makepkg: package splitting
by Marc - A. Dahlhaus [ Administration | Westermann GmbH ] 15 Dec '08

15 Dec '08

Am Dienstag, den 09.12.2008, 22:32 +1000 schrieb Allan McRae: > Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote: > > > > I'll tested that again and i forgot to change two of the DESTDIR > > locations to the new home of the pkgdirs... So the missing cleanup was > > actually a stupid user error... > > > > You should only be doing something like "make DESTDIR=$pkgdir install". > Makepkg points $pkgdir in the right place for each split package. (I > will get around to documenting all this at some stage) > > Allan Hello Allan, i think i found another bug. Every package that is a split-build and not in the first position in the pkgname build-order shows an empty "Optional dependencies" string during installation. I think there is something wrong in restore_package_variables during variable resetting to initial values or in backup_package_variables during the backup declaration. It looks like there are to much quotes around the declaration so that it declares an empty string. Maybe a "-z" check before the actual variable declaration could fix that particular problem. Marc

2 2

[pacman-dev] Order of option parsing and sourcing makepkg.conf
by Allan McRae 14 Dec '08

14 Dec '08

Hi, With the commit which enables us to specify the makepkg config file on the command-line (http://projects.archlinux.org/?p=pacman.git;a=commitdiff;h=4b183bf9) the sourcing of the config file gets moved after the parsing of the options. This creates problems with the --help flag as it needs to know what BUILDSCRIPT is defined as. It also causes problems when specifying a different BUILDSCRIPT with -p as this gets set during option parsing then gets overwritten with the makepkg config file gets sourced. I don't see a nice fix for this. Sourcing the conf file before option parsing and then again after if it is changed seems not good to me. So can we back that patch out at least temporarily. Note that this does not effect maint so the 3.2.2 release will be bug free as usual. Allan

3 6

Re: [pacman-dev] makepkg: package splitting
by Miklos Vajna 10 Dec '08

10 Dec '08

On Mon, Dec 08, 2008 at 12:35:41PM +1000, Allan McRae <allan(a)archlinux.org> wrote: > pkgname=('abs-core' 'abs-readme') > pkgver=2.3 > pkgrel=1 > arch=('i686' 'x86_64') > url="http://projects.archlinux.org/git/?p=abs.git" > license=('GPL') > source=(ftp://ftp.archlinux.org/other/abs/abs-${pkgver}.tar.gz) > md5sums=('d6fd791aa487ba8bb5ff48c3ace20592') > > build() { > cd ${srcdir}/abs > make CONFDIR=/etc/ || return 1 > > # change ABS tags for x86_64 to correct values > if [ "$CARCH" = "x86_64" ]; then > sed -i "s|i686|x86_64|g" ${pkgdir}/etc/abs.conf > fi > } > > package_abs-core() { > pkgdesc="Utilities to download and work with the Arch Build System (ABS)" > depends=('bash' 'rsync') > backup=(etc/abs.conf) > > cd ${srcdir}/abs > make CONFDIR=/etc/ DESTDIR=${pkgdir} install || return 1 > } > > package_abs-readme() { > pkgdesc="ABS base directory and README file" > depends=('abs') > install=abs.install > > # Add readme file, and make base /var/abs path > install -dm0755 ${pkgdir}/var/abs/local/ > install -Dm0644 ${srcdir}/abs/README ${pkgdir}/var/abs/README > } How does this allow one to define package-dependent dependencies? Typically foo-python will have to depend on python, but foo won't. Thanks.

1 1

[pacman-dev] pacman summary!
by smurfd 10 Dec '08

10 Dec '08

Hi, i have a suggestion, like the suggestion i gave to the portage developers for the gentoo "package" manager. It would be really nice, if after a "pacman -Suy" or just a simple "pacman -S <packagelist>" that you got a summary of all the messages that has been spit out on the screen while doing the upgrade of packages. If you upgrade several packages, say 100 of them or so, it could be hard to read all the notices that every package spits out when beeing installed. i mean stuff like : *>>> Updating module dependencies. Please wait ... >>> MKINITCPIO SETUP >>> ---------------- >>> If you use LVM2, Encrypted root or software RAID, >>> Ensure you enable support in /etc/mkinitcpio.conf . >>> More information about mkinitcpio setup can be found here: >>> http://wiki.archlinux.org/index.php/Mkinitcpio >>> Generating initial ramdisk, using mkinitcpio. Please wait... * Hope you like the idea, and i dont think it should be so hard to apply. cut all of the >>> things to a temp file and after doing the last upgrade you cat whatever is in that temp file.. or something like that. Best regards /smurfd

2 2

[pacman-dev] makepkg: package splitting
by Allan McRae 09 Dec '08

09 Dec '08

I have finished implementing the basics of package splitting in makepkg. Everything works as expected when just making packages and capturing logs but there is some small output issues that need tidying up and checking that other functionality work (e.g. repackaging is an issue....) Anyway you can see the work on my "splitpkg" branch: http://dev.archlinux.org/~allan/gitweb/gitweb.cgi?p=pacman.git;a=shortlog;h… For your interest, I have attached the same package written in three different ways to show you what makepkg can now do: PKGBUILD.1 - Traditional build PKGBUILD.2 - Splitting the packaging step from build step. Limits fakeroot usage PKGBUILD.3 - Package splitting # $Id: PKGBUILD 366 2008-04-19 13:01:23Z travis $ # Maintainer: Allan McRae <allan(a)archlinux.org> pkgname=abs pkgver=2.3 pkgrel=1 pkgdesc="Utilities to download and work with the Arch Build System (ABS)" arch=('i686' 'x86_64') url="http://projects.archlinux.org/git/?p=abs.git" license=('GPL') depends=('bash' 'rsync') backup=(etc/abs.conf) install=abs.install source=(ftp://ftp.archlinux.org/other/abs/${pkgname}-${pkgver}.tar.gz) md5sums=('d6fd791aa487ba8bb5ff48c3ace20592') build() { cd ${srcdir}/${pkgname} make CONFDIR=/etc/ || return 1 make CONFDIR=/etc/ DESTDIR=${pkgdir} install || return 1 # Add readme file, and make base /var/abs path install -dm0755 ${pkgdir}/var/abs/local/ install -Dm0644 ${srcdir}/abs/README ${pkgdir}/var/abs/README # change ABS tags for x86_64 to correct values if [ "$CARCH" = "x86_64" ]; then sed -i "s|i686|x86_64|g" ${pkgdir}/etc/abs.conf fi } # vim: set ts=2 sw=2 noet: # $Id: PKGBUILD 366 2008-04-19 13:01:23Z travis $ # Maintainer: Allan McRae <allan(a)archlinux.org> pkgname=abs pkgver=2.3 pkgrel=1 pkgdesc="Utilities to download and work with the Arch Build System (ABS)" arch=('i686' 'x86_64') url="http://projects.archlinux.org/git/?p=abs.git" license=('GPL') depends=('bash' 'rsync') backup=(etc/abs.conf) install=abs.install source=(ftp://ftp.archlinux.org/other/abs/${pkgname}-${pkgver}.tar.gz) md5sums=('d6fd791aa487ba8bb5ff48c3ace20592') build() { cd ${srcdir}/${pkgname} make CONFDIR=/etc/ || return 1 # change ABS tags for x86_64 to correct values if [ "$CARCH" = "x86_64" ]; then sed -i "s|i686|x86_64|g" ${pkgdir}/etc/abs.conf fi } package() { cd ${srcdir}/${pkgname} make CONFDIR=/etc/ DESTDIR=${pkgdir} install || return 1 # Add readme file, and make base /var/abs path install -dm0755 ${pkgdir}/var/abs/local/ install -Dm0644 ${srcdir}/abs/README ${pkgdir}/var/abs/README } # vim: set ts=2 sw=2 noet: # $Id: PKGBUILD 366 2008-04-19 13:01:23Z travis $ # Maintainer: Allan McRae <allan(a)archlinux.org> pkgname=('abs-core' 'abs-readme') pkgver=2.3 pkgrel=1 arch=('i686' 'x86_64') url="http://projects.archlinux.org/git/?p=abs.git" license=('GPL') source=(ftp://ftp.archlinux.org/other/abs/abs-${pkgver}.tar.gz) md5sums=('d6fd791aa487ba8bb5ff48c3ace20592') build() { cd ${srcdir}/abs make CONFDIR=/etc/ || return 1 # change ABS tags for x86_64 to correct values if [ "$CARCH" = "x86_64" ]; then sed -i "s|i686|x86_64|g" ${pkgdir}/etc/abs.conf fi } package_abs-core() { pkgdesc="Utilities to download and work with the Arch Build System (ABS)" depends=('bash' 'rsync') backup=(etc/abs.conf) cd ${srcdir}/abs make CONFDIR=/etc/ DESTDIR=${pkgdir} install || return 1 } package_abs-readme() { pkgdesc="ABS base directory and README file" depends=('abs') install=abs.install # Add readme file, and make base /var/abs path install -dm0755 ${pkgdir}/var/abs/local/ install -Dm0644 ${srcdir}/abs/README ${pkgdir}/var/abs/README } # vim: set ts=2 sw=2 noet:

2 3

Re: [pacman-dev] makepkg: package splitting
by Marc - A. Dahlhaus [ Administration | Westermann GmbH ] 09 Dec '08

09 Dec '08

Am Dienstag, den 09.12.2008, 21:57 +1000 schrieb Allan McRae: > Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote: > > Am Montag, den 08.12.2008, 12:35 +1000 schrieb Allan McRae: > > > >> I have finished implementing the basics of package splitting in > >> makepkg. Everything works as expected when just making packages and > >> capturing logs but there is some small output issues that need tidying > >> up and checking that other functionality work (e.g. repackaging is an > >> issue....) > >> > > > > Hello Allan, > > > > > > i've tested the script and found some bugs: > > > > The sourcing of makepkg.conf is in a wrong location (far to late). > > If you display the usage via --help you get no PKGBUILD mentioned in > > usages output. If you specify another buildscript via -p you get that > > mentioned in usage but it gets overwritten by the contents of > > makepkg.conf when it gets sourced in later before the actual build is > > done. > > > > Good catch.... I have no idea what caused that! Should not be herd to > track down though. > > > If you give -c to makepkg the pkgdirs of the splitpackages don't get > > erased. > > > > Someone is either running makepkg as root or without fakeroot... > Anyway, thats an easy fix. I'll tested that again and i forgot to change two of the DESTDIR locations to the new home of the pkgdirs... So the missing cleanup was actually a stupid user error... > > But the actual build and logging went fine, thank you :) > > > > Good to see the bits I tested actually working! Thanks for your help. > > Allan

2 1

Re: [pacman-dev] makepkg: package splitting
by Marc - A. Dahlhaus [ Administration | Westermann GmbH ] 09 Dec '08

09 Dec '08

Am Montag, den 08.12.2008, 12:35 +1000 schrieb Allan McRae: > I have finished implementing the basics of package splitting in > makepkg. Everything works as expected when just making packages and > capturing logs but there is some small output issues that need tidying > up and checking that other functionality work (e.g. repackaging is an > issue....) Hello Allan, i've tested the script and found some bugs: The sourcing of makepkg.conf is in a wrong location (far to late). If you display the usage via --help you get no PKGBUILD mentioned in usages output. If you specify another buildscript via -p you get that mentioned in usage but it gets overwritten by the contents of makepkg.conf when it gets sourced in later before the actual build is done. If you give -c to makepkg the pkgdirs of the splitpackages don't get erased. But the actual build and logging went fine, thank you :) Hope that helps, Marc

2 2

[pacman-dev] pacman 3.2.2 release
by Dan McGee 08 Dec '08

08 Dec '08

I'd like to do a 3.2.2 release sometime soon (this week, more than likely). Let me know if there are any outstanding patches for this release that aren't already in the maint branch as of tonight. I will put a call for updated translations out in the coming few days. -Dan

2 1

[pacman-dev] [translation] small fix for Russian translation
by serg 08 Dec '08

08 Dec '08

pacman/po/ru.po 972c972 < msgstr "ОШИбКА:" --- > msgstr "ОШИБКА:"

1 0

[pacman-dev] [PATCH] makepkg: Introduce restrict option
by Allan McRae 08 Dec '08

08 Dec '08

The restrict option, combined with the RESTRICTED_FILES variable, allows makepkg to automatically remove common confliting files (e.g. /usr/share/info/dir). Signed-off-by: Allan McRae <allan(a)archlinux.org> --- doc/makepkg.conf.5.txt | 8 ++++++++ etc/makepkg.conf.in | 7 +++++-- scripts/makepkg.sh.in | 8 ++++++-- 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt index 18dbf35..84e8192 100644 --- a/doc/makepkg.conf.5.txt +++ b/doc/makepkg.conf.5.txt @@ -125,6 +125,9 @@ Options *zipman*;; Compress manual (man and info) pages with gzip. + *restrict*;; + Remove restricted files from the package. + **INTEGRITY_CHECK=(**check1 ...**)**:: File integrity checks to use. Multiple checks may be specified; this affects both generation and checking. The current valid options are: @@ -150,6 +153,11 @@ Options to this array. *NOTE:* Do not add the leading slash to the directory name. +**RESTRICTED_FILES=(**usr/{,share}/info/dir ...**)**:: + If "restrict" is specified in the OPTIONS array, this variable will + instruct makepkg which files to remove from the package. This is + useful for index files that are added to by multiple packages. + **PKGDEST=**"/path/to/folder":: If this value is not set, packages will by default be placed in the current directory (location of the linkman:PKGBUILD[5]). Many people diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index 82722be..545a215 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -58,7 +58,7 @@ BUILDENV=(fakeroot !distcc color !ccache !xdelta) # These are default values for the options=() settings ######################################################################### # -# Default: OPTIONS=(strip !docs libtool emptydirs zipman) +# Default: OPTIONS=(strip !docs libtool emptydirs zipman restrict) # A negated option will do the opposite of the comments below. # #-- strip: Strip symbols from binaries/libraries @@ -66,8 +66,9 @@ BUILDENV=(fakeroot !distcc color !ccache !xdelta) #-- libtool: Leave libtool (.la) files in packages #-- emptydirs: Leave empty directories in packages #-- zipman: Compress manual (man and info) pages with gzip +#-- restrict: Remove files sepecified below from package # -OPTIONS=(strip !docs libtool emptydirs zipman) +OPTIONS=(strip !docs libtool emptydirs zipman restrict) #-- File integrity checks to use. Valid: md5, sha1, sha256, sha384, sha512 INTEGRITY_CHECK=(md5) @@ -77,6 +78,8 @@ MAN_DIRS=({usr{,/local}{,/share},opt/*}/{man,info}) DOC_DIRS=(usr/{,share/}{doc,gtk-doc} opt/*/{doc,gtk-doc}) #-- Directories to be searched for the strip option (if option set correctly above) STRIP_DIRS=(bin lib sbin usr/{bin,lib,sbin,local/{bin,lib,sbin}} opt/*/{bin,lib,sbin}) +#-- Files to be removed from all packages +RESTRICTED_FILES=(usr/{,share}/info/dir) ######################################################################### # PACKAGE OUTPUT diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index bf6524d..ea27883 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -41,7 +41,7 @@ confdir='@sysconfdir@' startdir="$PWD" srcdir="$startdir/src" pkgdir="$startdir/pkg" -packaging_options=('strip' 'docs' 'libtool' 'emptydirs' 'zipman') +packaging_options=('strip' 'docs' 'libtool' 'emptydirs' 'zipman' 'restrict') other_options=('ccache' 'distcc' 'makeflags' 'force') readonly -a packaging_options other_options @@ -792,7 +792,6 @@ tidy_install() { done fi - if [ "$(check_option strip)" = "y" ]; then msg2 "$(gettext "Stripping debugging symbols from binaries and libraries...")" local binary @@ -818,6 +817,11 @@ tidy_install() { find . ! -type d -name "*.la" -exec rm -f -- '{}' \; fi + if [ "$(check_option restrict)" = "y" -a -n "RESTRICTED_FILES" ]; then + msg2 "$(gettext "Removing restricted files...")" + rm -f ${RESTRICTED_FILES[@]} + fi + if [ "$(check_option emptydirs)" = "n" ]; then msg2 "$(gettext "Removing empty directories...")" find . -depth -type d -empty -delete -- 1.6.0.4

2 2

[pacman-dev] [GIT] The official pacman repository branch, master, updated. v3.2.1-59-gbd62827
by dan＠archlinux.org 08 Dec '08

08 Dec '08

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, master has been updated via bd628274cc8db81704860e69894fcd217c2068d0 (commit) via 9ae7eb1292a42eb568fd982f32d3dca5b9794a2d (commit) via 818fae320f22ea758765a55b3a3a98e2f0a66a45 (commit) via 69be73f68cc770ddcbe388af96e32f8bba7886f7 (commit) via 78cf32e194a1a58c6a7ee3d1c10623e668be71d6 (commit) via 59776ef306935399b2bf1e9cbdd7a7bfa6df1b49 (commit) via b373b1d16b6235bef2e34a9a21e043418222a813 (commit) from 61c6552862345cb155903cd1566f1cef5c527a94 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit bd628274cc8db81704860e69894fcd217c2068d0 Merge: 9ae7eb1292a42eb568fd982f32d3dca5b9794a2d 78cf32e194a1a58c6a7ee3d1c10623e668be71d6 Author: Dan McGee <dan(a)archlinux.org> Date: Sun Dec 7 22:12:17 2008 -0600 Merge branch 'maint' commit 9ae7eb1292a42eb568fd982f32d3dca5b9794a2d Author: Giovanni Scafora <linuxmania(a)gmail.com> Date: Thu Dec 4 21:03:31 2008 -0600 Update Italian translation Signed-off-by: Dan McGee <dan(a)archlinux.org> commit 818fae320f22ea758765a55b3a3a98e2f0a66a45 Author: Dan McGee <dan(a)archlinux.org> Date: Sat Dec 6 23:32:34 2008 -0600 makepkg: ensure PKGBUILD does not contain CRLF characters Do a simple check before sourcing the file to ensure we are a valid bash script. Signed-off-by: Dan McGee <dan(a)archlinux.org> commit 69be73f68cc770ddcbe388af96e32f8bba7886f7 Author: Allan McRae <allan(a)archlinux.org> Date: Fri Nov 14 03:08:52 2008 +1000 makepkg: several small bits of tidying 1. Do not warn people about missing arch if they are using --ignorearch. 2. Remove unneed reference to bug report about using fakeroot as little as possible. We want to do that, bug report of not. 3. Removes superfluous warning given when building as root. The user has already used the "--asroot" flag. 4. Move comment about skipping warning message to above where it occurs 5. Do not warn about skipping source retreval, integrety checks and extraction when using --repackage 6. Do not warn about skipping build when using --repackage 7. Move comment about fakeroot usage to above test condition Signed-off-by: Allan McRae <allan(a)archlinux.org> Signed-off-by: Dan McGee <dan(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: contrib/pactree | 3 +- doc/PKGBUILD.5.txt | 2 +- doc/repo-add.8.txt | 4 +- po/it.po | 141 +++++++++++++++++++++++-------------------------- scripts/makepkg.sh.in | 41 +++++++-------- 5 files changed, 90 insertions(+), 101 deletions(-) hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [GIT] The official pacman repository branch, maint, updated. v3.2.1-25-g78cf32e
by dan＠archlinux.org 08 Dec '08

08 Dec '08

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, maint has been updated via 78cf32e194a1a58c6a7ee3d1c10623e668be71d6 (commit) via 59776ef306935399b2bf1e9cbdd7a7bfa6df1b49 (commit) via b373b1d16b6235bef2e34a9a21e043418222a813 (commit) from a1f7c83dbf3bce492163362d2896e3a4176be616 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- ----------------------------------------------------------------------- Summary of changes: contrib/pactree | 3 +-- doc/PKGBUILD.5.txt | 2 +- doc/repo-add.8.txt | 4 ++-- scripts/makepkg.sh.in | 4 ++++ 4 files changed, 8 insertions(+), 5 deletions(-) hooks/post-receive -- The official pacman repository

1 0