- Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH v2] Introduce alpm_dbs_update() function for parallel db updates
by Anatol Pomozov 14 Apr '20

14 Apr '20

This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/alpm.h | 29 ++++++++++ lib/libalpm/be_sync.c | 128 ++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/dload.c | 12 ++++ lib/libalpm/dload.h | 5 ++ 4 files changed, 174 insertions(+) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 5d559db1..c985429d 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -1049,6 +1049,35 @@ int alpm_db_remove_server(alpm_db_t *db, const char *url); */ int alpm_db_update(int force, alpm_db_t *db); +/** Update package databases + * + * An update of the package databases \a dbs will be attempted. Unless + * \a force is true, the update will only be performed if the remote + * databases were modified since the last update. + * + * This operation requires a database lock, and will return an applicable error + * if the lock could not be obtained. + * + * Example: + * @code + * alpm_list_t *dbs = alpm_get_syncdbs(); + * ret = alpm_dbs_update(config->handle, dbs, force); + * if(ret < 0) { + * pm_printf(ALPM_LOG_ERROR, _("failed to synchronize all databases (%s)\n"), + * alpm_strerror(alpm_errno(config->handle))); + * } + * @endcode + * + * @note After a successful update, the \link alpm_db_get_pkgcache() + * package cache \endlink will be invalidated + * @param handle the context handle + * @param dbs list of package databases to update + * @param force if true, then forces the update, otherwise update only in case + * the databases aren't up to date + * @return 0 on success, -1 on error (pm_errno is set accordingly) + */ +int alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force); + /** Get a package entry from a package database. * @param db pointer to the package database to get the package from * @param name of the package diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index aafed15d..497e1054 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -301,6 +301,134 @@ int SYMEXPORT alpm_db_update(int force, alpm_db_t *db) return ret; } +int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force) { + char *syncpath; + const char *dbext = handle->dbext; + alpm_list_t *i; + int ret = -1; + mode_t oldmask; + alpm_list_t *payloads = NULL; + + /* Sanity checks */ + ASSERT(dbs != NULL, return -1); + handle->pm_errno = ALPM_ERR_OK; + + syncpath = get_sync_dir(handle); + ASSERT(syncpath != NULL, return -1); + + /* make sure we have a sane umask */ + oldmask = umask(0022); + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + int dbforce = force; + struct dload_payload *payload = NULL; + size_t len; + int siglevel; + + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + ASSERT(db != handle->db_local, RET_ERR(handle, ALPM_ERR_WRONG_ARGS, -1)); + ASSERT(db->servers != NULL, RET_ERR(handle, ALPM_ERR_SERVER_NONE, -1)); + + /* force update of invalid databases to fix potential mismatched database/signature */ + if(db->status & DB_STATUS_INVALID) { + dbforce = 1; + } + + CALLOC(payload, 1, sizeof(*payload), RET_ERR(handle, ALPM_ERR_MEMORY, -1)); + + /* set hard upper limit of 128MiB */ + payload->max_size = 128 * 1024 * 1024; + ASSERT(db->servers != NULL, RET_ERR(handle, ALPM_ERR_SERVER_NONE, -1)); + payload->servers = db->servers; + + /* print server + filename into a buffer */ + len = strlen(db->treename) + strlen(dbext) + 1; + MALLOC(payload->filepath, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(payload->filepath, len, "%s%s", db->treename, dbext); + payload->handle = handle; + payload->force = dbforce; + payload->unlink_on_fail = 1; + + payloads = alpm_list_add(payloads, payload); + + siglevel = alpm_db_get_siglevel(db); + if(siglevel & ALPM_SIG_DATABASE) { + struct dload_payload *sig_payload; + CALLOC(sig_payload, 1, sizeof(*sig_payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + + /* print filename into a buffer (leave space for separator and .sig) */ + len = strlen(db->treename) + strlen(dbext) + 5; + MALLOC(sig_payload->filepath, len, goto cleanup); + snprintf(sig_payload->filepath, len, "%s%s.sig", db->treename, dbext); + + sig_payload->handle = handle; + sig_payload->force = dbforce; + sig_payload->errors_ok = (siglevel & ALPM_SIG_DATABASE_OPTIONAL); + + /* set hard upper limit of 16KiB */ + sig_payload->max_size = 16 * 1024; + sig_payload->servers = db->servers; + + payloads = alpm_list_add(payloads, sig_payload); + } + } + + /* attempt to grab a lock */ + if(_alpm_handle_lock(handle)) { + GOTO_ERR(handle, ALPM_ERR_HANDLE_LOCK, cleanup); + } + + ret = _alpm_multi_download(handle, payloads, syncpath); + if(ret < 0) { + goto cleanup; + } + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + /* Cache needs to be rebuilt */ + _alpm_db_free_pkgcache(db); + + /* clear all status flags regarding validity/existence */ + db->status &= ~DB_STATUS_VALID; + db->status &= ~DB_STATUS_INVALID; + db->status &= ~DB_STATUS_EXISTS; + db->status &= ~DB_STATUS_MISSING; + + /* if the download failed skip validation to preserve the download error */ + if(sync_db_validate(db) != 0) { + /* pm_errno should be set */ + ret = -1; + } + } + +cleanup: + _alpm_handle_unlock(handle); + + if(ret == -1) { + /* pm_errno was set by the download code */ + _alpm_log(handle, ALPM_LOG_DEBUG, "failed to sync dbs: %s\n", + alpm_strerror(handle->pm_errno)); + } else { + handle->pm_errno = ALPM_ERR_OK; + } + + if(payloads) { + alpm_list_free_inner(payloads, (alpm_list_fn_free)_alpm_dload_payload_reset); + FREELIST(payloads); + } + free(syncpath); + umask(oldmask); + return ret; +} + /* Forward decl so I don't reorganize the whole file right now */ static int sync_db_read(alpm_db_t *db, struct archive *archive, struct archive_entry *entry, alpm_pkg_t **likely_pkg); diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 670da03d..7cd3e3a4 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -636,6 +636,16 @@ int _alpm_download(struct dload_payload *payload, const char *localpath, } } +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath) +{ + (void)handle; + (void)payloads; + (void)localpath; + return 0; +} + static char *filecache_find_url(alpm_handle_t *handle, const char *url) { const char *filebase = strrchr(url, '/'); @@ -738,6 +748,7 @@ void _alpm_dload_payload_reset(struct dload_payload *payload) FREE(payload->destfile_name); FREE(payload->content_disp_name); FREE(payload->fileurl); + FREE(payload->filepath); *payload = (struct dload_payload){0}; } @@ -746,6 +757,7 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) ASSERT(payload, return); FREE(payload->fileurl); + FREE(payload->filepath); payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 1e8f75f3..3eb7fbe1 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -31,6 +31,7 @@ struct dload_payload { char *destfile_name; char *content_disp_name; char *fileurl; + char *filepath; /* download URL path */ alpm_list_t *servers; long respcode; off_t initial_size; @@ -53,4 +54,8 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload); int _alpm_download(struct dload_payload *payload, const char *localpath, char **final_file, const char **final_url); +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath); + #endif /* ALPM_DLOAD_H */ -- 2.25.1

2 6

[pacman-dev] [PATCH] Add NoProgressbar to pacman.conf options
by escondida＠iff.ink 13 Apr '20

13 Apr '20

From: Ivy Foster <escondida(a)iff.ink> Signed-off-by: Ivy Foster <escondida(a)iff.ink> --- doc/pacman.conf.5.asciidoc | 3 +++ etc/pacman.conf.in | 1 + src/pacman/conf.c | 2 ++ src/pacman/pacman-conf.c | 3 +++ 4 files changed, 9 insertions(+) diff --git a/doc/pacman.conf.5.asciidoc b/doc/pacman.conf.5.asciidoc index b297e332..73f57eeb 100644 --- a/doc/pacman.conf.5.asciidoc +++ b/doc/pacman.conf.5.asciidoc @@ -186,6 +186,9 @@ Options *Color*:: Automatically enable colors only when pacman's output is on a tty. +*NoProgressbar*:: + Disables progressbars. + *TotalDownload*:: When downloading, display the amount downloaded, download rate, ETA, and completed percentage of the entire download list rather diff --git a/etc/pacman.conf.in b/etc/pacman.conf.in index 7446944f..fb473fe9 100644 --- a/etc/pacman.conf.in +++ b/etc/pacman.conf.in @@ -31,6 +31,7 @@ Architecture = auto # Misc options #UseSyslog #Color +#NoProgressbar #TotalDownload CheckSpace #VerbosePkgLists diff --git a/src/pacman/conf.c b/src/pacman/conf.c index f9de386f..92614860 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -579,6 +579,8 @@ static int _parse_options(const char *key, char *value, config->color = isatty(fileno(stdout)) ? PM_COLOR_ON : PM_COLOR_OFF; enable_colors(config->color); } + } else if(strcmp(key, "NoProgressbar") == 0) { + config->noprogressbar = 1; } else if(strcmp(key, "DisableDownloadTimeout") == 0) { config->disable_dl_timeout = 1; } else { diff --git a/src/pacman/pacman-conf.c b/src/pacman/pacman-conf.c index 6e25d243..48152697 100644 --- a/src/pacman/pacman-conf.c +++ b/src/pacman/pacman-conf.c @@ -259,6 +259,7 @@ static void dump_config(void) show_bool("VerbosePkgLists", config->verbosepkglists); show_bool("DisableDownloadTimeout", config->disable_dl_timeout); show_bool("ILoveCandy", config->chomp); + show_bool("NoProgressbar", config->noprogressbar); show_cleanmethod("CleanMethod", config->cleanmethod); @@ -368,6 +369,8 @@ static int list_directives(void) show_bool("VerbosePkgLists", config->verbosepkglists); } else if(strcasecmp(i->data, "DisableDownloadTimeout") == 0) { show_bool("DisableDownloadTimeout", config->disable_dl_timeout); + } else if(strcasecmp(i->data, "NoProgressbar") == 0) { + show_bool("NoProgressbar", config->noprogressbar); } else if(strcasecmp(i->data, "CleanMethod") == 0) { show_cleanmethod("CleanMethod", config->cleanmethod); -- 2.26.0

2 2

[pacman-dev] [PATCH 1/2] Use STRDUP for error checking in more places
by Allan McRae 13 Apr '20

13 Apr '20

Use STRDUP() over strdup() to catch memory allocation errors. There are still some instances of strdup left, but these are in functions that currently have no error path and would require a larger rework. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- lib/libalpm/be_package.c | 10 +++++++--- lib/libalpm/handle.c | 2 +- lib/libalpm/util.c | 3 ++- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/lib/libalpm/be_package.c b/lib/libalpm/be_package.c index 4832966b..38ba365d 100644 --- a/lib/libalpm/be_package.c +++ b/lib/libalpm/be_package.c @@ -201,11 +201,15 @@ static int parse_descfile(alpm_handle_t *handle, struct archive *a, alpm_pkg_t * } else if(strcmp(key, "pkgdesc") == 0) { STRDUP(newpkg->desc, ptr, return -1); } else if(strcmp(key, "group") == 0) { - newpkg->groups = alpm_list_add(newpkg->groups, strdup(ptr)); + char *tmp = NULL; + STRDUP(tmp, ptr, return -1); + newpkg->groups = alpm_list_add(newpkg->groups, tmp); } else if(strcmp(key, "url") == 0) { STRDUP(newpkg->url, ptr, return -1); } else if(strcmp(key, "license") == 0) { - newpkg->licenses = alpm_list_add(newpkg->licenses, strdup(ptr)); + char *tmp = NULL; + STRDUP(tmp, ptr, return -1); + newpkg->licenses = alpm_list_add(newpkg->licenses, tmp); } else if(strcmp(key, "builddate") == 0) { newpkg->builddate = _alpm_parsedate(ptr); } else if(strcmp(key, "packager") == 0) { @@ -660,7 +664,7 @@ alpm_pkg_t *_alpm_pkg_load_internal(alpm_handle_t *handle, /* internal fields for package struct */ newpkg->origin = ALPM_PKG_FROM_FILE; - newpkg->origin_data.file = strdup(pkgfile); + STRDUP(newpkg->origin_data.file, pkgfile, goto error); newpkg->ops = get_file_pkg_ops(); newpkg->handle = handle; newpkg->infolevel = INFRQ_BASE | INFRQ_DESC | INFRQ_SCRIPTLET; diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index 8ae08d11..e3b2c911 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -105,7 +105,7 @@ int _alpm_handle_lock(alpm_handle_t *handle) ASSERT(handle->lockfd < 0, return 0); /* create the dir of the lockfile first */ - dir = strdup(handle->lockfile); + STRDUP(dir, handle->lockfile, return -1); ptr = strrchr(dir, '/'); if(ptr) { *ptr = '\0'; diff --git a/lib/libalpm/util.c b/lib/libalpm/util.c index cebc87ec..cb838e43 100644 --- a/lib/libalpm/util.c +++ b/lib/libalpm/util.c @@ -350,7 +350,8 @@ int _alpm_unpack(alpm_handle_t *handle, const char *path, const char *prefix, /* If specific files were requested, skip entries that don't match. */ if(list) { - char *entry_prefix = strdup(entryname); + char *entry_prefix = NULL; + STRDUP(entry_prefix, entryname, ret = 1; goto cleanup); char *p = strstr(entry_prefix,"/"); if(p) { *(p + 1) = '\0'; -- 2.25.1

1 1

[pacman-dev] [PATCH] Touch up proto files
by Daniel M. Capella 13 Apr '20

13 Apr '20

- split: Fix erroneous $pkgname calls to use $pkgbase - VCS: - Add missing fields - Remove unnecessary $srcdir's - Move ./autogen.sh to prepare() - 80 char line length for comments and make license note stand out - Add ./autogen.sh to the other proto's Signed-off-by: Daniel M. Capella <polyzen(a)archlinux.org> --- proto/PKGBUILD-split.proto | 18 ++++++++++-------- proto/PKGBUILD-vcs.proto | 21 +++++++++++++-------- proto/PKGBUILD.proto | 1 + 3 files changed, 24 insertions(+), 16 deletions(-) diff --git a/proto/PKGBUILD-split.proto b/proto/PKGBUILD-split.proto index 9898ef81..6fc91a56 100644 --- a/proto/PKGBUILD-split.proto +++ b/proto/PKGBUILD-split.proto @@ -1,7 +1,8 @@ -# This is an example of a PKGBUILD for splitting packages. Use this as a -# start to creating your own, and remove these comments. For more information, -# see 'man PKGBUILD'. NOTE: Please fill out the license field for your package! -# If it is unknown, then please put 'unknown'. +# This is an example of a PKGBUILD for splitting packages. Use this as a start +# to creating your own, and remove these comments. For more information, see +# 'man PKGBUILD'. +# NOTE: Please fill out the license field for your package! If it is unknown, +# then please put 'unknown'. # Maintainer: Your Name <youremail(a)domain.com> pkgname=('pkg1' 'pkg2') @@ -26,14 +27,15 @@ options=() install= changelog= source=("$pkgbase-$pkgver.tar.gz" - "$pkgname-$pkgver.patch") + "$pkgbase-$pkgver.patch") noextract=() md5sums=() validpgpkeys=() prepare() { - cd "$pkgname-$pkgver" - patch -p1 -i "$srcdir/$pkgname-$pkgver.patch" + cd "$pkgbase-$pkgver" + patch -p1 -i "$srcdir/$pkgbase-$pkgver.patch" + ./autogen.sh } build() { @@ -43,7 +45,7 @@ build() { } check() { - cd "$pkgname-$pkgver" + cd "$pkgbase-$pkgver" make -k check } diff --git a/proto/PKGBUILD-vcs.proto b/proto/PKGBUILD-vcs.proto index ae9956a9..01eddf98 100644 --- a/proto/PKGBUILD-vcs.proto +++ b/proto/PKGBUILD-vcs.proto @@ -1,5 +1,6 @@ -# This is an example PKGBUILD file. Use this as a start to creating your own, -# and remove these comments. For more information, see 'man PKGBUILD'. +# This is an example of a PKGBUILD for VCS packages. Use this as a start to +# creating your own, and remove these comments. For more information, see 'man +# PKGBUILD'. # NOTE: Please fill out the license field for your package! If it is unknown, # then please put 'unknown'. @@ -10,6 +11,7 @@ pkgname=NAME-VCS # '-bzr', '-git', '-hg' or '-svn' pkgver=VERSION pkgrel=1 +epoch= pkgdesc="" arch=() url="" @@ -17,21 +19,24 @@ license=('GPL') groups=() depends=() makedepends=('VCS_PACKAGE') # 'bzr', 'git', 'mercurial' or 'subversion' +checkdepends=() provides=("${pkgname%-VCS}") conflicts=("${pkgname%-VCS}") replaces=() backup=() options=() install= +changelog= source=('FOLDER::VCS+URL#FRAGMENT') noextract=() md5sums=('SKIP') +validpgpkeys=() # Please refer to the 'USING VCS SOURCES' section of the PKGBUILD man page for # a description of each element in the source array. pkgver() { - cd "$srcdir/${pkgname%-VCS}" + cd "${pkgname%-VCS}" # The examples below are not absolute and need to be adapted to each repo. The # primary goal is to generate version numbers that will increase according to @@ -56,23 +61,23 @@ pkgver() { } prepare() { - cd "$srcdir/${pkgname%-VCS}" + cd "${pkgname%-VCS}" patch -p1 -i "$srcdir/${pkgname%-VCS}.patch" + ./autogen.sh } build() { - cd "$srcdir/${pkgname%-VCS}" - ./autogen.sh + cd "${pkgname%-VCS}" ./configure --prefix=/usr make } check() { - cd "$srcdir/${pkgname%-VCS}" + cd "${pkgname%-VCS}" make -k check } package() { - cd "$srcdir/${pkgname%-VCS}" + cd "${pkgname%-VCS}" make DESTDIR="$pkgdir/" install } diff --git a/proto/PKGBUILD.proto b/proto/PKGBUILD.proto index a2c600d5..e72b1aa8 100644 --- a/proto/PKGBUILD.proto +++ b/proto/PKGBUILD.proto @@ -33,6 +33,7 @@ validpgpkeys=() prepare() { cd "$pkgname-$pkgver" patch -p1 -i "$srcdir/$pkgname-$pkgver.patch" + ./autogen.sh } build() { -- 2.26.0

2 1

[pacman-dev] [PATCH] Use noextract with pacman-conf NoExtract
by Eli Schwartz 13 Apr '20

13 Apr '20

From: Earnestly <zibeon(a)googlemail.com> Current code accidently uses noupgrade for the NoExtract directive. https://medium.com/@Code_Analysis/the-last-line-effect-7b1cb7f0d2a1 Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- Previously fixed in https://github.com/andrewgregory/pacutils/commit/64c7d0aca1a3aa3be5b7789f10… but the unfixed version got merged here. :/ src/pacman/pacman-conf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pacman/pacman-conf.c b/src/pacman/pacman-conf.c index 939cbdbd..f67ec21f 100644 --- a/src/pacman/pacman-conf.c +++ b/src/pacman/pacman-conf.c @@ -348,7 +348,7 @@ static int list_directives(void) } else if(strcasecmp(i->data, "NoUpgrade") == 0) { show_list_str("NoUpgrade", config->noupgrade); } else if(strcasecmp(i->data, "NoExtract") == 0) { - show_list_str("NoExtract", config->noupgrade); + show_list_str("NoExtract", config->noextract); } else if(strcasecmp(i->data, "Architecture") == 0) { -- 2.26.0

3 3

[pacman-dev] [PATCH] Add REALLOC macro to simplify realloc error handling
by Rikard Falkeborn 13 Apr '20

13 Apr '20

realloc can fail just like the other memory allocation functions. Add a macro to simplify handling of realloc failures, similar to the already existing MALLOC, CALLOC, etc. Replace the existing realloc uses with the new macro, allowing us to move tedious error handling to the macro. Also, in be_package and be_sync, this fixes hypothetical memory leaks (and thereafter null pointer dereferences) in case realloc fails to shrink the allocated memory. Signed-off-by: Rikard Falkeborn <rikard.falkeborn(a)gmail.com> --- lib/libalpm/be_local.c | 7 +------ lib/libalpm/be_package.c | 3 +-- lib/libalpm/be_sync.c | 2 +- lib/libalpm/util.c | 13 +++---------- lib/libalpm/util.h | 1 + 5 files changed, 7 insertions(+), 19 deletions(-) diff --git a/lib/libalpm/be_local.c b/lib/libalpm/be_local.c index 9ebdfa40..689f0102 100644 --- a/lib/libalpm/be_local.c +++ b/lib/libalpm/be_local.c @@ -843,12 +843,7 @@ static int local_db_read(alpm_pkg_t *info, int inforeq) } /* attempt to hand back any memory we don't need */ if(files_count > 0) { - alpm_file_t *newfiles; - - newfiles = realloc(files, sizeof(alpm_file_t) * files_count); - if(newfiles != NULL) { - files = newfiles; - } + REALLOC(files, sizeof(alpm_file_t) * files_count, (void)0); } else { FREE(files); } diff --git a/lib/libalpm/be_package.c b/lib/libalpm/be_package.c index 7a118d2a..4832966b 100644 --- a/lib/libalpm/be_package.c +++ b/lib/libalpm/be_package.c @@ -669,8 +669,7 @@ alpm_pkg_t *_alpm_pkg_load_internal(alpm_handle_t *handle, if(full) { if(newpkg->files.files) { /* attempt to hand back any memory we don't need */ - newpkg->files.files = realloc(newpkg->files.files, - sizeof(alpm_file_t) * newpkg->files.count); + REALLOC(newpkg->files.files, sizeof(alpm_file_t) * newpkg->files.count, (void)0); /* "checking for conflicts" requires a sorted list, ensure that here */ _alpm_log(handle, ALPM_LOG_DEBUG, "sorting package filelist for %s\n", pkgfile); diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index aafed15d..5f457122 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -689,7 +689,7 @@ static int sync_db_read(alpm_db_t *db, struct archive *archive, } /* attempt to hand back any memory we don't need */ if(files_count > 0) { - files = realloc(files, sizeof(alpm_file_t) * files_count); + REALLOC(files, sizeof(alpm_file_t) * files_count, (void)0); } else { FREE(files); } diff --git a/lib/libalpm/util.c b/lib/libalpm/util.c index c46b1397..cebc87ec 100644 --- a/lib/libalpm/util.c +++ b/lib/libalpm/util.c @@ -1427,22 +1427,15 @@ int _alpm_fnmatch(const void *pattern, const void *string) */ void *_alpm_realloc(void **data, size_t *current, const size_t required) { - char *newdata; - - newdata = realloc(*data, required); - if(!newdata) { - _alpm_alloc_fail(required); - return NULL; - } + REALLOC(*data, required, return NULL); if (*current < required) { /* ensure all new memory is zeroed out, in both the initial * allocation and later reallocs */ - memset(newdata + *current, 0, required - *current); + memset((char*)*data + *current, 0, required - *current); } *current = required; - *data = newdata; - return newdata; + return *data; } /** This automatically grows data based on current/required. diff --git a/lib/libalpm/util.h b/lib/libalpm/util.h index 675eedec..3306a022 100644 --- a/lib/libalpm/util.h +++ b/lib/libalpm/util.h @@ -53,6 +53,7 @@ void _alpm_alloc_fail(size_t size); #define MALLOC(p, s, action) do { p = malloc(s); if(p == NULL) { _alpm_alloc_fail(s); action; } } while(0) #define CALLOC(p, l, s, action) do { p = calloc(l, s); if(p == NULL) { _alpm_alloc_fail(l * s); action; } } while(0) +#define REALLOC(p, s, action) do { void* np = realloc(p, s); if(np == NULL) { _alpm_alloc_fail(s); action; } else { p = np; } } while(0) /* This strdup macro is NULL safe- copying NULL will yield NULL */ #define STRDUP(r, s, action) do { if(s != NULL) { r = strdup(s); if(r == NULL) { _alpm_alloc_fail(strlen(s)); action; } } else { r = NULL; } } while(0) #define STRNDUP(r, s, l, action) do { if(s != NULL) { r = strndup(s, l); if(r == NULL) { _alpm_alloc_fail(l); action; } } else { r = NULL; } } while(0) -- 2.26.0

2 1

[pacman-dev] [PATCH v3] Introduce alpm_dbs_update() function for parallel db updates
by Anatol Pomozov 13 Apr '20

13 Apr '20

This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/alpm.h | 29 ++++++++++ lib/libalpm/be_sync.c | 128 ++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/dload.c | 12 ++++ lib/libalpm/dload.h | 5 ++ 4 files changed, 174 insertions(+) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 5d559db1..2cf20343 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -1049,6 +1049,35 @@ int alpm_db_remove_server(alpm_db_t *db, const char *url); */ int alpm_db_update(int force, alpm_db_t *db); +/** Update package databases + * + * An update of the package databases in the list \a dbs will be attempted. + * Unless \a force is true, the update will only be performed if the remote + * databases were modified since the last update. + * + * This operation requires a database lock, and will return an applicable error + * if the lock could not be obtained. + * + * Example: + * @code + * alpm_list_t *dbs = alpm_get_syncdbs(); + * ret = alpm_dbs_update(config->handle, dbs, force); + * if(ret < 0) { + * pm_printf(ALPM_LOG_ERROR, _("failed to synchronize all databases (%s)\n"), + * alpm_strerror(alpm_errno(config->handle))); + * } + * @endcode + * + * @note After a successful update, the \link alpm_db_get_pkgcache() + * package cache \endlink will be invalidated + * @param handle the context handle + * @param dbs list of package databases to update + * @param force if true, then forces the update, otherwise update only in case + * the databases aren't up to date + * @return 0 on success, -1 on error (pm_errno is set accordingly) + */ +int alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force); + /** Get a package entry from a package database. * @param db pointer to the package database to get the package from * @param name of the package diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index aafed15d..7ab52301 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -301,6 +301,134 @@ int SYMEXPORT alpm_db_update(int force, alpm_db_t *db) return ret; } +int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force) { + char *syncpath; + const char *dbext = handle->dbext; + alpm_list_t *i; + int ret = -1; + mode_t oldmask; + alpm_list_t *payloads = NULL; + + /* Sanity checks */ + CHECK_HANDLE(handle, return -1); + ASSERT(dbs != NULL, return -1); + handle->pm_errno = ALPM_ERR_OK; + + syncpath = get_sync_dir(handle); + ASSERT(syncpath != NULL, return -1); + + /* make sure we have a sane umask */ + oldmask = umask(0022); + + /* attempt to grab a lock */ + if(_alpm_handle_lock(handle)) { + GOTO_ERR(handle, ALPM_ERR_HANDLE_LOCK, cleanup); + } + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + int dbforce = force; + struct dload_payload *payload = NULL; + size_t len; + int siglevel; + + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + ASSERT(db != handle->db_local, GOTO_ERR(handle, ALPM_ERR_WRONG_ARGS, cleanup)); + ASSERT(db->servers != NULL, GOTO_ERR(handle, ALPM_ERR_SERVER_NONE, cleanup)); + + /* force update of invalid databases to fix potential mismatched database/signature */ + if(db->status & DB_STATUS_INVALID) { + dbforce = 1; + } + + CALLOC(payload, 1, sizeof(*payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + + /* set hard upper limit of 128 MiB */ + payload->max_size = 128 * 1024 * 1024; + payload->servers = db->servers; + + /* print server + filename into a buffer */ + len = strlen(db->treename) + strlen(dbext) + 1; + MALLOC(payload->filepath, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(payload->filepath, len, "%s%s", db->treename, dbext); + payload->handle = handle; + payload->force = dbforce; + payload->unlink_on_fail = 1; + + payloads = alpm_list_add(payloads, payload); + + siglevel = alpm_db_get_siglevel(db); + if(siglevel & ALPM_SIG_DATABASE) { + struct dload_payload *sig_payload; + CALLOC(sig_payload, 1, sizeof(*sig_payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + + /* print filename into a buffer (leave space for separator and .sig) */ + len = strlen(db->treename) + strlen(dbext) + 5; + MALLOC(sig_payload->filepath, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(sig_payload->filepath, len, "%s%s.sig", db->treename, dbext); + + sig_payload->handle = handle; + sig_payload->force = 1; + sig_payload->errors_ok = (siglevel & ALPM_SIG_DATABASE_OPTIONAL); + + /* set hard upper limit of 16 KiB */ + sig_payload->max_size = 16 * 1024; + sig_payload->servers = db->servers; + + payloads = alpm_list_add(payloads, sig_payload); + } + } + + ret = _alpm_multi_download(handle, payloads, syncpath); + if(ret < 0) { + goto cleanup; + } + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + /* Cache needs to be rebuilt */ + _alpm_db_free_pkgcache(db); + + /* clear all status flags regarding validity/existence */ + db->status &= ~DB_STATUS_VALID; + db->status &= ~DB_STATUS_INVALID; + db->status &= ~DB_STATUS_EXISTS; + db->status &= ~DB_STATUS_MISSING; + + /* if the download failed skip validation to preserve the download error */ + if(sync_db_validate(db) != 0) { + /* pm_errno should be set */ + ret = -1; + } + } + +cleanup: + _alpm_handle_unlock(handle); + + if(ret == -1) { + /* pm_errno was set by the download code */ + _alpm_log(handle, ALPM_LOG_DEBUG, "failed to sync dbs: %s\n", + alpm_strerror(handle->pm_errno)); + } else { + handle->pm_errno = ALPM_ERR_OK; + } + + if(payloads) { + alpm_list_free_inner(payloads, (alpm_list_fn_free)_alpm_dload_payload_reset); + FREELIST(payloads); + } + free(syncpath); + umask(oldmask); + return ret; +} + /* Forward decl so I don't reorganize the whole file right now */ static int sync_db_read(alpm_db_t *db, struct archive *archive, struct archive_entry *entry, alpm_pkg_t **likely_pkg); diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 670da03d..7cd3e3a4 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -636,6 +636,16 @@ int _alpm_download(struct dload_payload *payload, const char *localpath, } } +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath) +{ + (void)handle; + (void)payloads; + (void)localpath; + return 0; +} + static char *filecache_find_url(alpm_handle_t *handle, const char *url) { const char *filebase = strrchr(url, '/'); @@ -738,6 +748,7 @@ void _alpm_dload_payload_reset(struct dload_payload *payload) FREE(payload->destfile_name); FREE(payload->content_disp_name); FREE(payload->fileurl); + FREE(payload->filepath); *payload = (struct dload_payload){0}; } @@ -746,6 +757,7 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) ASSERT(payload, return); FREE(payload->fileurl); + FREE(payload->filepath); payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 1e8f75f3..3eb7fbe1 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -31,6 +31,7 @@ struct dload_payload { char *destfile_name; char *content_disp_name; char *fileurl; + char *filepath; /* download URL path */ alpm_list_t *servers; long respcode; off_t initial_size; @@ -53,4 +54,8 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload); int _alpm_download(struct dload_payload *payload, const char *localpath, char **final_file, const char **final_url); +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath); + #endif /* ALPM_DLOAD_H */ -- 2.26.0

2 2

[pacman-dev] [PATCH] Use C18 language standard
by Anatol Pomozov 05 Apr '20

05 Apr '20

C18 is the latest released version of the language spec. The toolchains support it starting with GCC 8.1.0 and Clang 7. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- configure.ac | 2 +- meson.build | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/configure.ac b/configure.ac index 9db39232..84869ce5 100644 --- a/configure.ac +++ b/configure.ac @@ -183,7 +183,7 @@ AC_SUBST(LFS_CFLAGS) # Checks for programs. AC_PROG_AWK -AC_PROG_CC_C99 +AC_PROG_CC_C18 AC_PROG_INSTALL AC_CHECK_PROGS([PYTHON], [python3 python], [false]) AC_PATH_PROGS([BASH_SHELL], [bash bash4], [false]) diff --git a/meson.build b/meson.build index fc81fa27..2f4e7cc4 100644 --- a/meson.build +++ b/meson.build @@ -3,7 +3,7 @@ project('pacman', version : '5.2.1', license : 'GPLv2+', default_options : [ - 'c_std=gnu99', + 'c_std=c18', 'prefix=/usr', 'sysconfdir=/etc', 'localstatedir=/var', -- 2.26.0

3 6

[pacman-dev] [PATCH v2] meson: use 'pedantic' compiler warning level by default
by Anatol Pomozov 27 Mar '20

27 Mar '20

This sets parity with the autotools version of the build that has -pedantic flag enbled unconditionally. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- meson.build | 1 + 1 file changed, 1 insertion(+) diff --git a/meson.build b/meson.build index 572526b2..fc81fa27 100644 --- a/meson.build +++ b/meson.build @@ -7,6 +7,7 @@ project('pacman', 'prefix=/usr', 'sysconfdir=/etc', 'localstatedir=/var', + 'warning_level=3', ], meson_version : '>= 0.51') -- 2.26.0

2 1

[pacman-dev] [PATCH] meson: use 'pedantic' compiler warning level
by Anatol Pomozov 27 Mar '20

27 Mar '20

Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- meson.build | 1 + 1 file changed, 1 insertion(+) diff --git a/meson.build b/meson.build index 572526b2..fc81fa27 100644 --- a/meson.build +++ b/meson.build @@ -7,6 +7,7 @@ project('pacman', 'prefix=/usr', 'sysconfdir=/etc', 'localstatedir=/var', + 'warning_level=3', ], meson_version : '>= 0.51') -- 2.25.1

3 13