May 2020 - Pacman-dev - lists.archlinux.org

[pacman-dev] [GIT] The official pacman repository branch, master, updated. v5.2.1-83-g8ce142a2
by Allan McRae 11 May '20

11 May '20

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, master has been updated via 8ce142a2552418f64a74e773f659d92b065d6209 (commit) via 8e769ddb8a59a9fbacf4614283d2fb519f022386 (commit) via 7423b166047ff168ef337818958bf86ed1d42a0a (commit) via 559590256c48fa5f995944de3802911e5a56ba7f (commit) from 22a58f5420438f35effb991696d37529d7a31969 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 8ce142a2552418f64a74e773f659d92b065d6209 Author: Levente Polyak <anthraxx(a)archlinux.org> Date: Mon May 11 00:45:49 2020 +0200 makepkg: deterministic PKGINFO libprovides for multiple library versions While iterating over the provides array, the find call for locating a shared library may result in listing multiple entries which by itself does not produce a stable deterministic order and may vary depending on the underlying filesystem. To provide a stable listing and a reproducible .PKGINFO file the result of find is piped to sort with a static LC_ALL=C localisation. Signed-off-by: Levente Polyak <anthraxx(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 8e769ddb8a59a9fbacf4614283d2fb519f022386 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Sun May 10 00:32:39 2020 -0400 Log invalid conf settings as an error This is not a warning, _parse_options() returns failure without even parsing further lines and the attempted pacman/pacman-conf program execution immediately aborts. Warnings are for when e.g. later on if we don't recognize a setting at all, we skip over it and have enough confidence in this to continue executing the program. The current implementation results in pacman-conf aborting with: warning: config file /etc/pacman.conf, line 60: invalid value for 'ParallelDownloads' : '2.5' error parsing '/etc/pacman.conf' or pacman -Syu aborting with the entirely more cryptic: warning: config file /etc/pacman.conf, line 59: invalid value for 'ParallelDownloads' : '2.5' and this isn't just a problem for the newly added ParallelDownloads setting, either, you could get the same problem if you specified a broken XferCommand, but that's harder as it's more accepting of input and you probably don't hit this except with unbalanced quotes. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 7423b166047ff168ef337818958bf86ed1d42a0a Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Sun May 10 00:32:40 2020 -0400 pacman-conf: fix incomplete support for ILoveCandy This was only partially implemented in the original implementation. `pacman-conf | grep ILoveCandy` would tell you if it was set, but querying directly by name would not. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 559590256c48fa5f995944de3802911e5a56ba7f Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Sun May 10 00:32:38 2020 -0400 pacman-conf: add support for new ParallelDownloads config option This was forgotten in the initial implementation, so it was impossible to figure out the value from a script, or correctly roundtrip the config file. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: scripts/makepkg.sh.in | 2 +- src/pacman/conf.c | 8 ++++---- src/pacman/pacman-conf.c | 15 +++++++++++++++ 3 files changed, 20 insertions(+), 5 deletions(-) hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [PATCH] makepkg: deterministic PKGINFO libprovides for multiple library versions
by anthraxx＠archlinux.org 11 May '20

11 May '20

From: Levente Polyak <anthraxx(a)archlinux.org> While iterating over the provides array, the find call for locating a shared library may result in listing multiple entries which by itself does not produce a stable deterministic order and may vary depending on the underlying filesystem. To provide a stable listing and a reproducible .PKGINFO file the result of find is piped to sort with a static LC_ALL=C localisation. Signed-off-by: Levente Polyak <anthraxx(a)archlinux.org> --- scripts/makepkg.sh.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index d1416d15..b95a03af 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -521,7 +521,7 @@ find_libprovides() { missing=0 case "$p" in *.so) - mapfile -t filename < <(find "$pkgdir" -type f -name $p\*) + mapfile -t filename < <(find "$pkgdir" -type f -name $p\* | LC_ALL=C sort) if [[ $filename ]]; then # packages may provide multiple versions of the same library for fn in "${filename[@]}"; do -- 2.26.2

3 2

[pacman-dev] [PATCH 1/3] pacman-conf: add support for new ParallelDownloads config option
by Eli Schwartz 10 May '20

10 May '20

This was forgotten in the initial implementation, so it was impossible to figure out the value from a script, or correctly roundtrip the config file. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- src/pacman/pacman-conf.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/pacman/pacman-conf.c b/src/pacman/pacman-conf.c index 60739bf6..d76c0985 100644 --- a/src/pacman/pacman-conf.c +++ b/src/pacman/pacman-conf.c @@ -142,6 +142,14 @@ static void show_str(const char *directive, const char *val) printf("%s%c", val, sep); } +static void show_int(const char *directive, unsigned int val) +{ + if (verbose) { + printf("%s = ", directive); + } + printf("%u%c", val, sep); +} + static void show_list_str(const char *directive, alpm_list_t *list) { alpm_list_t *i; @@ -261,6 +269,8 @@ static void dump_config(void) show_bool("ILoveCandy", config->chomp); show_bool("NoProgressBar", config->noprogressbar); + show_int("ParallelDownloads", config->parallel_downloads); + show_cleanmethod("CleanMethod", config->cleanmethod); show_siglevel("SigLevel", config->siglevel, 0); @@ -372,6 +382,9 @@ static int list_directives(void) } else if(strcasecmp(i->data, "NoProgressBar") == 0) { show_bool("NoProgressBar", config->noprogressbar); + } else if(strcasecmp(i->data, "ParallelDownloads") == 0) { + show_int("ParallelDownloads", config->parallel_downloads); + } else if(strcasecmp(i->data, "CleanMethod") == 0) { show_cleanmethod("CleanMethod", config->cleanmethod); -- 2.26.2

2 8

[pacman-dev] [GIT] The official pacman repository branch, master, updated. v5.2.1-79-g22a58f54
by Allan McRae 09 May '20

09 May '20

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, master has been updated via 22a58f5420438f35effb991696d37529d7a31969 (commit) via 557845bc971ff272c53da773baea277a2d2d47b8 (commit) via 0346e0eef224ab8ba22b659026ffdf2bfe95f3ae (commit) via b96e0df4dceaa2677baa1a3563211950708d3e63 (commit) via c78eb48d915dc22146073162dda08ddf73c4a508 (commit) via 64c4669f579dc5ad8d05329abffbd752ad0ed8f2 (commit) via 6a331af27fe6dc7842725d067fd2fb4a1c60c139 (commit) via 1d42a8f954eed3205e84cfb946b67ba889b04135 (commit) via fa68c33fa821cd7ddc5ae32baf62af2a238e44dc (commit) via dc98d0ea09f3632cd28a12099f3f09d466dcad1d (commit) via a8a1a1bb3ec98a8471cb5cd13d096f39a267f789 (commit) via fe8e13341bdeae4a59c0270a632c29e71ae9deda (commit) via cffda331adca0aedd7c1fc17d739c27fc8041a20 (commit) from 411b12d09ddb47722acf6d354c17ee9ee5228964 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 22a58f5420438f35effb991696d37529d7a31969 Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Sat Apr 18 19:15:44 2020 -0700 Swap alpm_db_update() implementation to multiplexed version Now when all callers of the old alpm_db_update() function are gone we can remove this implementation. And then rename alpm_dbs_update() function to alpm_db_update(). Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 557845bc971ff272c53da773baea277a2d2d47b8 Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Sat Apr 18 19:05:13 2020 -0700 Convert downloading databases to the new multiplexed API Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 0346e0eef224ab8ba22b659026ffdf2bfe95f3ae Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Sat Apr 18 19:00:49 2020 -0700 Convert download packages logic to multiplexed API Create a list of dload_payloads and pass it to the new _alpm_multi_* interface. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit b96e0df4dceaa2677baa1a3563211950708d3e63 Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Fri Apr 17 10:36:38 2020 -0700 Implement multibar UI Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down and then redraw the required progress bar. `pacman_multibar_ui.active_downloads` field represents the list of active downloads that correspond to progress bars. `struct pacman_progress_bar` is a data structure for a progress bar. In some cases (e.g. database downloads) we want to keep progress bars in order. In some other cases (package downloads) we want to move completed items to the top of the screen. Function `multibar_move_completed_up` allows to configure such behavior. Per discussion in the maillist we do not want to show download progress for signature files. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit c78eb48d915dc22146073162dda08ddf73c4a508 Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Tue May 5 19:50:51 2020 -0700 Extend download callback interface with start/complete events With the previous download interface the callback uses the first progress event as 'download has started' signal. Unfortunately it does not work with up-to-date files that never receive 'download progress' events. Up-to-date database messages are currently handled in sync_syncdbs() after the sequential download is completed and a result from ALPM is received. But this is not going to work with multiplexed download interface that returns the result only after all files are completed. Another problem with 'first progress event is the beginning of the download' is that such events time are unpredictable. Thus the UI progress bar order might differ from what has been passed by client to alpm_dbs_update() function. We actually want to keep the dbs progress bars in a strict order. To help to solve the given problems extend the download callback to allow 2 more events - download started and completed. 'Download started' events appear in the same order as in the list given by a client. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 64c4669f579dc5ad8d05329abffbd752ad0ed8f2 Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Tue May 5 18:28:49 2020 -0700 Introduce event types for start/end database list download Multiplexed database/files downloads will use multiple progress bars. The UI logic is quite complicated and printing error messages while handling multiple progress bars is going to be challenging. Instead we are going to save all ALPM error messages to a list and flush it at the end of the download process. Use on_progress variable that blocks error messages printing. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 6a331af27fe6dc7842725d067fd2fb4a1c60c139 Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Mon Apr 13 21:48:55 2020 -0700 Implement multiplexed download using mCURL curl_multi_download_internal() is the main loop that creates up to 'ParallelDownloads' easy curl handles, adds them to mcurl and then performs curl execution. This is when the paralled downloads happens. Once any of the downloads complete the function checks its result. In case if the download fails it initiates retry with the next server from payload->servers list. At the download completion all the payload resources are cleaned up. curl_multi_check_finished_download() is essentially refactored version of curl_download_internal() adopted for multi_curl. Once mcurl porting is complete curl_download_internal() will be removed. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 1d42a8f954eed3205e84cfb946b67ba889b04135 Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Mon Apr 13 21:40:54 2020 -0700 Implement _alpm_multi_download It is an equivalent of _alpm_download but accepts a list of payloads. curl_multi_download_internal() is a stub at this moment and will be implemented in the later commits of this patch series. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit fa68c33fa821cd7ddc5ae32baf62af2a238e44dc Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Mon Mar 9 15:36:48 2020 -0700 Inline dload_payload->curlerr field into a local variable dload_payload->curlerr is a field that is used inside curl_download_internal() function only. It can be converted to a local variable. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit dc98d0ea09f3632cd28a12099f3f09d466dcad1d Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Mon Mar 9 15:23:12 2020 -0700 Add multi_curl handle to ALPM global context To be able to run multiple download in parallel efficiently we need to use curl_multi interface [1]. It introduces a set of APIs over new type of handler 'CURLM'. Create CURLM object at the application start and set it to global ALPM context. The 'single-download' CURL handle moves to payload struct. A new CURL handle is created for each payload with intention to be processed by CURLM. Note that curl_download_internal() is not ported to CURLM interface due to the fact that the function will go away soon. [1] https://curl.haxx.se/libcurl/c/libcurl-multi.html Signed-off-by: Allan McRae <allan(a)archlinux.org> commit a8a1a1bb3ec98a8471cb5cd13d096f39a267f789 Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Thu Mar 26 13:19:59 2020 -0700 Introduce alpm_dbs_update() function for parallel db updates This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit fe8e13341bdeae4a59c0270a632c29e71ae9deda Author: Anatol Pomozov <anatol.pomozov(a)gmail.com> Date: Sun Mar 8 13:33:32 2020 -0700 Add config option to specify amount of parallel download streams It includes pacman.conf new 'ParallelDownloads' option that specifies how many concurrent downloads cURL starts in parallel. Add alpm_option_set_parallel_downloads() ALPM function that allows to set this config option programmatically. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit cffda331adca0aedd7c1fc17d739c27fc8041a20 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Tue May 5 22:09:42 2020 -0400 doc: remove vim modelines from BUILDINFO(5) We (thought we) removed all modelines from the project in commit 860e4c4943ad062bd0eff99f28e7d64804b3c08e, but apparently this one sneaked in by virtue of this manpage being added to the project after the "remove all the modelines" patch was submitted, but before it was applied. I must have failed to update the patch to remove it from this file also. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: README | 14 ++ doc/BUILDINFO.5.asciidoc | 3 - doc/pacman.conf.5.asciidoc | 5 + etc/pacman.conf.in | 1 + lib/libalpm/alpm.c | 12 +- lib/libalpm/alpm.h | 106 +++++---- lib/libalpm/be_sync.c | 195 ++++++++--------- lib/libalpm/dload.c | 521 +++++++++++++++++++++++++++++++++++++++++---- lib/libalpm/dload.h | 11 +- lib/libalpm/handle.c | 20 +- lib/libalpm/handle.h | 3 +- lib/libalpm/sync.c | 75 +++---- src/pacman/callback.c | 408 +++++++++++++++++++++++------------ src/pacman/callback.h | 7 +- src/pacman/conf.c | 58 ++++- src/pacman/conf.h | 2 + src/pacman/sync.c | 2 + src/pacman/util.c | 45 ++-- src/pacman/util.h | 4 + 19 files changed, 1086 insertions(+), 406 deletions(-) hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [PATCH v3] Convert download packages logic to multiplexed API
by Anatol Pomozov 08 May '20

08 May '20

Create a list of dload_payloads and pass it to the new _alpm_multi_* interface. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- README | 11 +++++++ lib/libalpm/alpm.h | 9 ------ lib/libalpm/dload.c | 1 - lib/libalpm/sync.c | 73 +++++++++++++++---------------------------- src/pacman/callback.c | 4 --- 5 files changed, 36 insertions(+), 62 deletions(-) diff --git a/README b/README index 6818ef70..f5bbaf02 100644 --- a/README +++ b/README @@ -655,3 +655,14 @@ API CHANGES BETWEEN 5.1 AND 5.2 - alpm_errno_t - added member ALPM_ERR_MISSING_CAPABILITY_SIGNATURES - alpm_sync_newversion() replaced with alpm_sync_get_new_version() which does not filter on any ALPM_DB_USAGE_*. + + +API CHANGES BETWEEN 5.2 AND 6.0 +=============================== + +[REMOVED] +- ALPM_EVENT_PKGDOWNLOAD_START, ALPM_EVENT_PKGDOWNLOAD_DONE, ALPM_EVENT_PKGDOWNLOAD_FAILED + +[CHANGED] + +[ADDED] diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 534a8189..903e2fbc 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -414,15 +414,6 @@ typedef enum _alpm_event_type_t { ALPM_EVENT_PKG_RETRIEVE_DONE, /** Not all package files were successfully downloaded from a repository. */ ALPM_EVENT_PKG_RETRIEVE_FAILED, - /** A file will be downloaded from a repository; See alpm_event_pkgdownload_t - * for arguments */ - ALPM_EVENT_PKGDOWNLOAD_START, - /** A file was downloaded from a repository; See alpm_event_pkgdownload_t - * for arguments */ - ALPM_EVENT_PKGDOWNLOAD_DONE, - /** A file failed to be downloaded from a repository; See - * alpm_event_pkgdownload_t for arguments */ - ALPM_EVENT_PKGDOWNLOAD_FAILED, /** Disk space usage will be computed for a package. */ ALPM_EVENT_DISKSPACE_START, /** Disk space usage was computed for a package. */ diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 9ab0e5c4..13aa4086 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -952,7 +952,6 @@ static int curl_multi_download_internal(alpm_handle_t *handle, } payloads = payloads->next; - // TODO: report that download has started } else { // the payload failed to start, do not start any new downloads just wait until // active one complete. diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 855ca69c..f329de10 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -726,47 +726,13 @@ static int find_dl_candidates(alpm_handle_t *handle, alpm_list_t **files) return 0; } -static int download_single_file(alpm_handle_t *handle, struct dload_payload *payload, - const char *cachedir) -{ - alpm_event_pkgdownload_t event = { - .type = ALPM_EVENT_PKGDOWNLOAD_START, - .file = payload->remote_name - }; - const alpm_list_t *server; - - payload->handle = handle; - payload->allow_resume = 1; - - EVENT(handle, &event); - for(server = payload->servers; server; server = server->next) { - const char *server_url = server->data; - size_t len; - - /* print server + filename into a buffer */ - len = strlen(server_url) + strlen(payload->remote_name) + 2; - MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); - snprintf(payload->fileurl, len, "%s/%s", server_url, payload->remote_name); - - if(_alpm_download(payload, cachedir, NULL, NULL) != -1) { - event.type = ALPM_EVENT_PKGDOWNLOAD_DONE; - EVENT(handle, &event); - return 0; - } - _alpm_dload_payload_reset_for_retry(payload); - } - - event.type = ALPM_EVENT_PKGDOWNLOAD_FAILED; - EVENT(handle, &event); - return -1; -} - static int download_files(alpm_handle_t *handle) { const char *cachedir; alpm_list_t *i, *files = NULL; int errors = 0; alpm_event_t event; + alpm_list_t *payloads = NULL; cachedir = _alpm_filecache_setup(handle); handle->trans->state = STATE_DOWNLOADING; @@ -814,26 +780,37 @@ static int download_files(alpm_handle_t *handle) event.type = ALPM_EVENT_PKG_RETRIEVE_START; EVENT(handle, &event); - event.type = ALPM_EVENT_PKG_RETRIEVE_DONE; for(i = files; i; i = i->next) { const alpm_pkg_t *pkg = i->data; - struct dload_payload payload = {0}; - - STRDUP(payload.remote_name, pkg->filename, GOTO_ERR(handle, ALPM_ERR_MEMORY, finish)); - payload.servers = pkg->origin_data.db->servers; - payload.max_size = pkg->size; - - if(download_single_file(handle, &payload, cachedir) == -1) { - errors++; - event.type = ALPM_EVENT_PKG_RETRIEVE_FAILED; - _alpm_log(handle, ALPM_LOG_WARNING, _("failed to retrieve some files\n")); - } - _alpm_dload_payload_reset(&payload); + struct dload_payload *payload = NULL; + + CALLOC(payload, 1, sizeof(*payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, finish)); + STRDUP(payload->remote_name, pkg->filename, FREE(payload); GOTO_ERR(handle, ALPM_ERR_MEMORY, finish)); + STRDUP(payload->filepath, pkg->filename, + FREE(payload->remote_name); FREE(payload); + GOTO_ERR(handle, ALPM_ERR_MEMORY, finish)); + payload->max_size = pkg->size; + payload->servers = pkg->origin_data.db->servers; + payload->handle = handle; + payload->allow_resume = 1; + + payloads = alpm_list_add(payloads, payload); + } + event.type = ALPM_EVENT_PKG_RETRIEVE_DONE; + if(_alpm_multi_download(handle, payloads, cachedir) == -1) { + errors++; + event.type = ALPM_EVENT_PKG_RETRIEVE_FAILED; + _alpm_log(handle, ALPM_LOG_WARNING, _("failed to retrieve some files\n")); } EVENT(handle, &event); } finish: + if(payloads) { + alpm_list_free_inner(payloads, (alpm_list_fn_free)_alpm_dload_payload_reset); + FREELIST(payloads); + } + if(files) { alpm_list_free(files); } diff --git a/src/pacman/callback.c b/src/pacman/callback.c index c2e516ec..25909e02 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -405,10 +405,6 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_DISKSPACE_DONE: case ALPM_EVENT_HOOK_DONE: case ALPM_EVENT_HOOK_RUN_DONE: - /* we can safely ignore those as well */ - case ALPM_EVENT_PKGDOWNLOAD_START: - case ALPM_EVENT_PKGDOWNLOAD_DONE: - case ALPM_EVENT_PKGDOWNLOAD_FAILED: /* nothing */ break; } -- 2.26.2

1 0

[pacman-dev] [PATCH v3] Implement multibar UI
by Anatol Pomozov 08 May '20

08 May '20

Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down and then redraw the required progress bar. `pacman_multibar_ui.active_downloads` field represents the list of active downloads that correspond to progress bars. `struct pacman_progress_bar` is a data structure for a progress bar. In some cases (e.g. database downloads) we want to keep progress bars in order. In some other cases (package downloads) we want to move completed items to the top of the screen. Function `multibar_move_completed_up` allows to configure such behavior. Per discussion in the maillist we do not want to show download progress for signature files. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/be_sync.c | 1 + lib/libalpm/dload.c | 61 +++---- lib/libalpm/dload.h | 2 +- src/pacman/callback.c | 387 ++++++++++++++++++++++++++++-------------- src/pacman/callback.h | 4 + src/pacman/sync.c | 2 + src/pacman/util.c | 18 ++ src/pacman/util.h | 4 + 8 files changed, 320 insertions(+), 159 deletions(-) diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index add1a576..86015ab6 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -366,6 +366,7 @@ int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force if(siglevel & ALPM_SIG_DATABASE) { struct dload_payload *sig_payload; CALLOC(sig_payload, 1, sizeof(*sig_payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + sig_payload->signature = 1; /* print filename into a buffer (leave space for separator and .sig) */ len = strlen(db->treename) + strlen(dbext) + 5; diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 2f360ab8..9ab0e5c4 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -90,6 +90,11 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, off_t current_size, total_size; alpm_download_event_progress_t cb_data = {0}; + /* do not print signature files progress bar */ + if(payload->signature) { + return 0; + } + /* avoid displaying progress bar for redirects with a body */ if(payload->respcode >= 300) { return 0; @@ -100,6 +105,11 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, return 1; } + if(dlnow < 0 || dltotal <= 0 || dlnow > dltotal) { + /* bogus values : stop here */ + return 0; + } + current_size = payload->initial_size + dlnow; /* is our filesize still under any set limit? */ @@ -115,34 +125,15 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, total_size = payload->initial_size + dltotal; - if(dltotal == 0 || payload->prevprogress == total_size) { + if(payload->prevprogress == total_size) { return 0; } - /* initialize the progress bar here to avoid displaying it when - * a repo is up to date and nothing gets downloaded. - * payload->handle->dlcb will receive the remote_name - * and the following arguments: - * 0, -1: download initialized - * 0, 0: non-download event - * x {x>0}, x: download complete - * x {x>0, x<y}, y {y > 0}: download progress, expected total is known */ - if(!payload->cb_initialized) { - cb_data.downloaded = 0; - cb_data.total = -1; - payload->cb_initialized = 1; - } - if(payload->prevprogress == current_size) { - cb_data.downloaded = 0; - cb_data.total = 0; - } else { /* do NOT include initial_size since it wasn't part of the package's * download_size (nor included in the total download size callback) */ - cb_data.downloaded = dlnow; - cb_data.total = dltotal; - } + cb_data.total = dltotal; + cb_data.downloaded = dlnow; payload->handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_PROGRESS, &cb_data); - payload->prevprogress = current_size; return 0; @@ -375,6 +366,8 @@ static int curl_download_internal(struct dload_payload *payload, double remote_size, bytes_dl; struct sigaction orig_sig_pipe, orig_sig_int; CURLcode curlerr; + alpm_download_event_init_t init_cb_data = {0}; + alpm_download_event_completed_t completed_cb_data = {0}; /* shortcut to our handle within the payload */ alpm_handle_t *handle = payload->handle; CURL *curl = curl_easy_init(); @@ -444,6 +437,8 @@ static int curl_download_internal(struct dload_payload *payload, dload_interrupted = 0; mask_signal(SIGINT, &inthandler, &orig_sig_int); + handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_INIT, &init_cb_data); + /* perform transfer */ curlerr = curl_easy_perform(curl); _alpm_log(handle, ALPM_LOG_DEBUG, "curl returned error %d from transfer\n", @@ -604,6 +599,10 @@ cleanup: raise(SIGINT); } + completed_cb_data.total = bytes_dl; + completed_cb_data.result = ret; + handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_COMPLETED, &completed_cb_data); + return ret; } @@ -665,7 +664,6 @@ static int curl_multi_check_finished_download(CURLM *curlm, CURLMsg *msg, long remote_time = -1; struct stat st; char hostname[HOSTNAME_SIZE]; - alpm_download_event_completed_t cb_data = {0}; int ret = -1; curlerr = curl_easy_getinfo(curl, CURLINFO_PRIVATE, &payload); @@ -825,9 +823,12 @@ cleanup: unlink(payload->tempfile_name); } - cb_data.total = bytes_dl; - cb_data.result = ret; - handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_COMPLETED, &cb_data); + if(!payload->signature) { + alpm_download_event_completed_t cb_data = {0}; + cb_data.total = bytes_dl; + cb_data.result = ret; + handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_COMPLETED, &cb_data); + } curl_multi_remove_handle(curlm, curl); curl_easy_cleanup(curl); @@ -945,9 +946,11 @@ static int curl_multi_download_internal(alpm_handle_t *handle, struct dload_payload *payload = payloads->data; if(curl_multi_add_payload(handle, curlm, payload, localpath) == 0) { - alpm_download_event_init_t cb_data = {.optional = payload->errors_ok}; + if(!payload->signature) { + alpm_download_event_init_t cb_data = {.optional = payload->errors_ok}; + handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_INIT, &cb_data); + } - handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_INIT, &cb_data); payloads = payloads->next; // TODO: report that download has started } else { @@ -1129,6 +1132,7 @@ char SYMEXPORT *alpm_fetch_pkgurl(alpm_handle_t *handle, const char *url) sig_filepath = filecache_find_url(handle, payload.fileurl); if(sig_filepath == NULL) { + payload.signature = 1; payload.handle = handle; payload.trust_remote_name = 1; payload.force = 1; @@ -1184,5 +1188,4 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; - payload->cb_initialized = 0; } diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 3f2fb9ea..8b52cdae 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -42,7 +42,7 @@ struct dload_payload { int errors_ok; int unlink_on_fail; int trust_remote_name; - int cb_initialized; + int signature; /* specifies if the payload is a signature file */ #ifdef HAVE_LIBCURL CURL *curl; char error_buffer[CURL_ERROR_SIZE]; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 613d59d4..c2e516ec 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -18,6 +18,8 @@ * along with this program. If not, see <http://www.gnu.org/licenses/>. */ +#include <assert.h> +#include <stdbool.h> #include <stdio.h> #include <stdlib.h> #include <string.h> @@ -52,6 +54,50 @@ static alpm_list_t *output = NULL; #define CLOCK_MONOTONIC_COARSE CLOCK_MONOTONIC #endif +struct pacman_progress_bar { + const char *filename; + off_t xfered; + off_t total_size; + uint64_t init_time; /* Time when this download started doing any progress */ + uint64_t sync_time; /* Last time we updated the bar info */ + double rate; + unsigned int eta; /* ETA in seconds */ + bool completed; /* transfer is completed */ +}; + +/* This datastruct represents the state of multiline progressbar UI */ +struct pacman_multibar_ui { + /* List of active downloads handled by multibar UI. + * Once the first download in the list is completed it is removed + * from this list and we never redraw it anymore. + * If the download is in this list, then the UI can redraw the progress bar or change + * the order of the bars (e.g. moving completed bars to the top of the list) + */ + alpm_list_t *active_downloads; /* List of type 'struct pacman_progress_bar' */ + + /* Number of active download bars that multibar UI handles. */ + size_t active_downloads_num; + + /* Specifies whether a completed progress bar need to be reordered and moved + * to the top of the list. + */ + bool move_completed_up; + + /* Cursor position relative to the first active progress bar, + * e.g. 0 means the first active progress bar, active_downloads_num-1 means the last bar, + * active_downloads_num - is the line below all progress bars. + */ + int cursor_lineno; +}; + +struct pacman_multibar_ui multibar_ui = {0}; + +static void cursor_goto_end(void); + +void multibar_move_completed_up(bool value) { + multibar_ui.move_completed_up = value; +} + static int64_t get_time_ms(void) { #if defined(_POSIX_TIMERS) && (_POSIX_TIMERS > 0) && defined(CLOCK_MONOTONIC_COARSE) @@ -152,11 +198,7 @@ static void fill_progress(const int bar_percent, const int disp_percent, printf(" %3d%%", disp_percent); } - if(bar_percent == 100) { - putchar('\n'); - } else { - putchar('\r'); - } + putchar('\r'); fflush(stdout); } @@ -346,6 +388,7 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_DB_RETRIEVE_FAILED: case ALPM_EVENT_PKG_RETRIEVE_DONE: case ALPM_EVENT_PKG_RETRIEVE_FAILED: + cursor_goto_end(); flush_output_list(); on_progress = 0; break; @@ -629,6 +672,7 @@ void cb_progress(alpm_progress_t event, const char *pkgname, int percent, fill_progress(percent, percent, cols - infolen); if(percent == 100) { + putchar('\n'); flush_output_list(); on_progress = 0; } else { @@ -647,149 +691,87 @@ void cb_dl_total(off_t total) } } -/* callback to handle display of download progress */ -static void dload_progress_event(const char *filename, off_t file_xfered, off_t file_total) +static int dload_progressbar_enabled(void) +{ + return !config->noprogressbar && (getcols() != 0); +} + +/* Goto the line that corresponds to num-th active download */ +static void cursor_goto_bar(int num) +{ + if(num > multibar_ui.cursor_lineno) { + console_cursor_move_down(num - multibar_ui.cursor_lineno); + } else if(num < multibar_ui.cursor_lineno) { + console_cursor_move_up(multibar_ui.cursor_lineno - num); + } + multibar_ui.cursor_lineno = num; +} + +/* Goto the line *after* the last active progress bar */ +static void cursor_goto_end(void) +{ + cursor_goto_bar(multibar_ui.active_downloads_num); +} + +/* Returns true if element with the specified name is found, false otherwise */ +static bool find_bar_for_filename(const char *filename, int *index, struct pacman_progress_bar **bar) +{ + int i = 0; + alpm_list_t *listitem = multibar_ui.active_downloads; + for(; listitem; listitem = listitem->next, i++) { + struct pacman_progress_bar *b = listitem->data; + if (strcmp(b->filename, filename) == 0) { + /* we found a progress bar with the given name */ + *index = i; + *bar = b; + return true; + } + } + + return false; +} + +static void draw_pacman_progress_bar(struct pacman_progress_bar *bar) { - static double rate_last; - static off_t xfered_last; - static int64_t initial_time = 0; int infolen; int filenamelen; char *fname, *p; /* used for wide character width determination and printing */ int len, wclen, wcwid, padwid; wchar_t *wcfname; - - int totaldownload = 0; - off_t xfered, total; - double rate = 0.0; - unsigned int eta_h = 0, eta_m = 0, eta_s = 0; + unsigned int eta_h = 0, eta_m = 0, eta_s = bar->eta; double rate_human, xfered_human; const char *rate_label, *xfered_label; - int file_percent = 0, total_percent = 0; + int file_percent = 0; const unsigned short cols = getcols(); - /* Nothing has changed since last callback; stop here */ - if(file_xfered == 0 && file_total == 0) { - return; - } - - if(config->noprogressbar || cols == 0) { - if(file_xfered == 0 && file_total == -1) { - printf(_("downloading %s...\n"), filename); - fflush(stdout); - } - return; - } - - infolen = cols * 6 / 10; - if(infolen < 50) { - infolen = 50; - } - /* only use TotalDownload if enabled and we have a callback value */ - if(config->totaldownload && list_total) { - /* sanity check */ - if(list_xfered + file_total <= list_total) { - totaldownload = 1; - } else { - /* bogus values : don't enable totaldownload and reset */ - list_xfered = 0; - list_total = 0; - } - } - - if(totaldownload) { - xfered = list_xfered + file_xfered; - total = list_total; - } else { - xfered = file_xfered; - total = file_total; - } - - /* this is basically a switch on xfered: 0, total, and - * anything else */ - if(file_xfered == 0 && file_total == -1) { - /* set default starting values, ensure we only call this once - * if TotalDownload is enabled */ - if(!totaldownload || (totaldownload && list_xfered == 0)) { - initial_time = get_time_ms(); - xfered_last = (off_t)0; - rate_last = 0.0; - get_update_timediff(1); - } - } else if(xfered > total || xfered < 0) { - /* bogus values : stop here */ - return; - } else if(file_xfered == file_total) { - /* compute final values */ - int64_t timediff = get_time_ms() - initial_time; - if(timediff > 0) { - rate = (double)xfered / (timediff / 1000.0); - /* round elapsed time (in ms) to the nearest second */ - eta_s = (unsigned int)(timediff + 500) / 1000; - } else { - eta_s = 0; - } - } else { - /* compute current average values */ - int64_t timediff = get_update_timediff(0); - - if(timediff < UPDATE_SPEED_MS) { - /* return if the calling interval was too short */ - return; - } - rate = (double)(xfered - xfered_last) / (timediff / 1000.0); - /* average rate to reduce jumpiness */ - rate = (rate + 2 * rate_last) / 3; - if(rate > 0.0) { - eta_s = (total - xfered) / rate; - } else { - eta_s = UINT_MAX; - } - rate_last = rate; - xfered_last = xfered; - } - - if(file_total) { - file_percent = (file_xfered * 100) / file_total; + if(bar->total_size) { + file_percent = (bar->xfered * 100) / bar->total_size; } else { file_percent = 100; } - if(totaldownload) { - total_percent = ((list_xfered + file_xfered) * 100) / - list_total; - - /* if we are at the end, add the completed file to list_xfered */ - if(file_xfered == file_total) { - list_xfered += file_total; - } - } - /* fix up time for display */ eta_h = eta_s / 3600; eta_s -= eta_h * 3600; eta_m = eta_s / 60; eta_s -= eta_m * 60; - len = strlen(filename); + len = strlen(bar->filename); fname = malloc(len + 1); - memcpy(fname, filename, len + 1); + memcpy(fname, bar->filename, len + 1); /* strip package or DB extension for cleaner look */ if((p = strstr(fname, ".pkg")) || (p = strstr(fname, ".db")) || (p = strstr(fname, ".files"))) { - /* tack on a .sig suffix for signatures */ - if(memcmp(&filename[len - 4], ".sig", 4) == 0) { - memcpy(p, ".sig", 4); - - /* adjust length for later calculations */ - len = p - fname + 4; - } else { - len = p - fname; - } + len = p - fname; fname[len] = '\0'; } + infolen = cols * 6 / 10; + if(infolen < 50) { + infolen = 50; + } + /* 1 space + filenamelen + 1 space + 6 for size + 1 space + 3 for label + * + 2 spaces + 4 for rate + 1 space + 3 for label + 2 for /s + 1 space + * 8 for eta, gives us the magic 33 */ @@ -824,8 +806,8 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t } - rate_human = humanize_size((off_t)rate, '\0', -1, &rate_label); - xfered_human = humanize_size(xfered, '\0', -1, &xfered_label); + rate_human = humanize_size((off_t)bar->rate, '\0', -1, &rate_label); + xfered_human = humanize_size(bar->xfered, '\0', -1, &xfered_label); printf(" %ls%-*s ", wcfname, padwid, ""); /* We will show 1.62 MiB/s, 11.6 MiB/s, but 116 KiB/s and 1116 KiB/s */ @@ -850,19 +832,166 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t free(fname); free(wcfname); - if(totaldownload) { - fill_progress(file_percent, total_percent, cols - infolen); + fill_progress(file_percent, file_percent, cols - infolen); + return; +} + +static void dload_init_event(const char *filename, alpm_download_event_init_t *data) +{ + (void)data; + + if(!dload_progressbar_enabled()) { + printf(_(" %s downloading...\n"), filename); + return; + } + + struct pacman_progress_bar *bar = calloc(1, sizeof(struct pacman_progress_bar)); + assert(bar); + bar->filename = filename; + bar->init_time = get_time_ms(); + bar->rate = 0.0; + multibar_ui.active_downloads = alpm_list_add(multibar_ui.active_downloads, bar); + + cursor_goto_end(); + printf(_(" %s downloading...\n"), filename); + multibar_ui.cursor_lineno++; + multibar_ui.active_downloads_num++; +} + +/* Draws download progress */ +static void dload_progress_event(const char *filename, alpm_download_event_progress_t *data) +{ + int index; + struct pacman_progress_bar *bar; + int64_t curr_time = get_time_ms(); + double last_chunk_rate; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + + /* compute current average values */ + timediff = curr_time - bar->sync_time; + + if(timediff < UPDATE_SPEED_MS) { + /* return if the calling interval was too short */ + return; + } + bar->sync_time = curr_time; + + last_chunk_rate = (double)(data->downloaded - bar->xfered) / (timediff / 1000.0); + /* average rate to reduce jumpiness */ + bar->rate = (last_chunk_rate + 2 * bar->rate) / 3; + if(bar->rate > 0.0) { + bar->eta = (data->total - data->downloaded) / bar->rate; } else { - fill_progress(file_percent, file_percent, cols - infolen); + bar->eta = UINT_MAX; + } + + /* Total size is received after the download starts. */ + bar->total_size = data->total; + bar->xfered = data->downloaded; + + cursor_goto_bar(index); + draw_pacman_progress_bar(bar); + fflush(stdout); +} + +/* download completed */ +static void dload_complete_event(const char *filename, alpm_download_event_completed_t *data) +{ + int index; + struct pacman_progress_bar *bar; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + bar->completed = true; + + /* This may not have been initialized if the download finished before + * an alpm_download_event_progress_t event happened */ + bar->total_size = data->total; + + if(data->result == 1) { + cursor_goto_bar(index); + printf(_(" %s is up to date"), bar->filename); + /* The line contains text from previous status. Erase these leftovers. */ + console_erase_line(); + } else if(data->result == 0) { + /* compute final values */ + bar->xfered = bar->total_size; + timediff = get_time_ms() - bar->init_time; + + /* if transfer was too fast, treat it as a 1ms transfer, for the sake + * of the rate calculation */ + if(timediff < 1) + timediff = 1; + + bar->rate = (double)bar->xfered / (timediff / 1000.0); + /* round elapsed time (in ms) to the nearest second */ + bar->eta = (unsigned int)(timediff + 500) / 1000; + + if(multibar_ui.move_completed_up && index != 0) { + /* If this item completed then move it to the top. + * Swap 0-th bar data with `index`-th one + */ + struct pacman_progress_bar *former_topbar = multibar_ui.active_downloads->data; + alpm_list_t *baritem = alpm_list_nth(multibar_ui.active_downloads, index); + multibar_ui.active_downloads->data = bar; + baritem->data = former_topbar; + + cursor_goto_bar(index); + draw_pacman_progress_bar(former_topbar); + + index = 0; + } + + cursor_goto_bar(index); + draw_pacman_progress_bar(bar); + } else { + cursor_goto_bar(index); + printf(_(" %s failed to download"), bar->filename); + console_erase_line(); + } + fflush(stdout); + + /* If the first bar is completed then there is no reason to keep it + * in the list as we are not going to redraw it anymore. + */ + while(multibar_ui.active_downloads) { + alpm_list_t *head = multibar_ui.active_downloads; + struct pacman_progress_bar *j = head->data; + if(j->completed) { + multibar_ui.cursor_lineno--; + multibar_ui.active_downloads_num--; + multibar_ui.active_downloads = alpm_list_remove_item( + multibar_ui.active_downloads, head); + free(head); + free(j); + } else { + break; + } } - return; } +/* Callback to handle display of download progress */ void cb_download(const char *filename, alpm_download_event_type_t event, void *data) { - if(event == ALPM_DOWNLOAD_PROGRESS) { - alpm_download_event_progress_t *progress = data; - dload_progress_event(filename, progress->downloaded, progress->total); + if(event == ALPM_DOWNLOAD_INIT) { + dload_init_event(filename, data); + } else if(event == ALPM_DOWNLOAD_PROGRESS) { + dload_progress_event(filename, data); + } else if(event == ALPM_DOWNLOAD_COMPLETED) { + dload_complete_event(filename, data); + } else { + pm_printf(ALPM_LOG_ERROR, _("unknown callback event type %d for %s\n"), + event, filename); } } diff --git a/src/pacman/callback.h b/src/pacman/callback.h index 6d92e86b..09d544a6 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -20,6 +20,7 @@ #ifndef PM_CALLBACK_H #define PM_CALLBACK_H +#include <stdbool.h> #include <sys/types.h> /* off_t */ #include <alpm.h> @@ -44,4 +45,7 @@ void cb_download(const char *filename, alpm_download_event_type_t event, __attribute__((format(printf, 2, 0))) void cb_log(alpm_loglevel_t level, const char *fmt, va_list args); +/* specify if multibar UI should move completed bars to the top of the screen */ +void multibar_move_completed_up(bool value); + #endif /* PM_CALLBACK_H */ diff --git a/src/pacman/sync.c b/src/pacman/sync.c index f7dcb958..a05af5da 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -35,6 +35,7 @@ #include "pacman.h" #include "util.h" #include "package.h" +#include "callback.h" #include "conf.h" static int unlink_verbose(const char *pathname, int ignore_missing) @@ -824,6 +825,7 @@ int sync_prepare_execute(void) goto cleanup; } + multibar_move_completed_up(true); if(alpm_trans_commit(config->handle, &data) == -1) { alpm_errno_t err = alpm_errno(config->handle); pm_printf(ALPM_LOG_ERROR, _("failed to commit transaction (%s)\n"), diff --git a/src/pacman/util.c b/src/pacman/util.c index 97b8e06d..03035037 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -1837,3 +1837,21 @@ char *arg_to_string(int argc, char *argv[]) strcpy(p, argv[i]); return cl_text; } + +/* Moves console cursor `lines` up */ +void console_cursor_move_up(unsigned int lines) +{ + printf("\x1B[%dF", lines); +} + +/* Moves console cursor `lines` down */ +void console_cursor_move_down(unsigned int lines) +{ + printf("\x1B[%dE", lines); +} + +/* Erases line from the current cursor position till the end of the line */ +void console_erase_line(void) +{ + printf("\x1B[K"); +} diff --git a/src/pacman/util.h b/src/pacman/util.h index 2b21f3d5..c97048fb 100644 --- a/src/pacman/util.h +++ b/src/pacman/util.h @@ -83,6 +83,10 @@ char *arg_to_string(int argc, char *argv[]); char *safe_fgets_stdin(char *s, int size); void console_cursor_hide(void); void console_cursor_show(void); +void console_cursor_move_up(unsigned int lines); +void console_cursor_move_down(unsigned int lines); +/* Erases line from the current cursor position till the end of the line */ +void console_erase_line(void); int pm_printf(alpm_loglevel_t level, const char *format, ...) __attribute__((format(printf,2,3))); int pm_asprintf(char **string, const char *format, ...) __attribute__((format(printf,2,3))); -- 2.26.2

1 1

[pacman-dev] [PATCH] Make bash-completion optional
by Wouter Wijsman 08 May '20

08 May '20

The bash completion files were the only reason pacman was not able to build as a non-root user. This patch adds the option to not install these files. This was needed for my use case, hopefully upstream someone else has a use for it as well. Signed-off-by: Wouter Wijsman <wwijsman(a)live.nl> --- meson_options.txt | 3 +++ scripts/meson.build | 44 +++++++++++++++++++++++--------------------- 2 files changed, 26 insertions(+), 21 deletions(-) diff --git a/meson_options.txt b/meson_options.txt index 4d8cc300..2927cd15 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -51,6 +51,9 @@ option('gpgme', type : 'feature', value : 'auto', option('i18n', type : 'boolean', value : true, description : 'enable localization of pacman, libalpm and scripts') +option('bash-completion', type : 'boolean', value : true, + description : 'install bash-completion files') + # tools option('file-seccomp', type: 'feature', value: 'auto', description: 'determine whether file is seccomp-enabled') diff --git a/scripts/meson.build b/scripts/meson.build index d2466523..12262e98 100644 --- a/scripts/meson.build +++ b/scripts/meson.build @@ -68,25 +68,27 @@ configure_file( output : '@BASENAME@', install_dir : join_paths(DATAROOTDIR, 'pkgconfig')) -custom_target( - 'bash_completion', - command : [ SCRIPT_EDITOR, '@INPUT@', '@OUTPUT@' ], - input : 'completion/bash_completion.in', - output : 'pacman', - install : true, - install_dir : BASHCOMPDIR) - -foreach symlink : ['pacman-key', 'makepkg'] - meson.add_install_script(MESON_MAKE_SYMLINK, - 'pacman', - join_paths(BASHCOMPDIR, symlink)) -endforeach +if get_option('bash-completion') + custom_target( + 'bash_completion', + command : [ SCRIPT_EDITOR, '@INPUT@', '@OUTPUT@' ], + input : 'completion/bash_completion.in', + output : 'pacman', + install : true, + install_dir : BASHCOMPDIR) -zsh_completion_dir = join_paths(DATAROOTDIR, 'zsh/site-functions') -custom_target( - 'zsh_completion', - command : [ SCRIPT_EDITOR, '@INPUT@', '@OUTPUT@' ], - input : 'completion/zsh_completion.in', - output : '_pacman', - install : true, - install_dir : zsh_completion_dir) + foreach symlink : ['pacman-key', 'makepkg'] + meson.add_install_script(MESON_MAKE_SYMLINK, + 'pacman', + join_paths(BASHCOMPDIR, symlink)) + endforeach + + zsh_completion_dir = join_paths(DATAROOTDIR, 'zsh/site-functions') + custom_target( + 'zsh_completion', + command : [ SCRIPT_EDITOR, '@INPUT@', '@OUTPUT@' ], + input : 'completion/zsh_completion.in', + output : '_pacman', + install : true, + install_dir : zsh_completion_dir) +endif -- 2.20.1

4 3

[pacman-dev] [PATCH v2] Implement multibar UI
by Anatol Pomozov 08 May '20

08 May '20

Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down. `active_dls` variable represents the list of active progress bars. `struct dl_progress_bar` is a data structure for a progress bar. In some cases we want to keep progress bars in order (e.g. db downloads). In some other cases (package downloads) we want to move completed items to the top of the screen. Global variable `multibar_move_completed_up` allows to configure such behavior. Signature file downloads are not shown at the UI. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/dload.c | 31 +--- lib/libalpm/dload.h | 1 - src/pacman/callback.c | 402 ++++++++++++++++++++++++++++-------------- src/pacman/callback.h | 3 + src/pacman/sync.c | 2 + 5 files changed, 286 insertions(+), 153 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 2f360ab8..1556d72c 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -100,6 +100,11 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, return 1; } + if(dlnow < 0 || dltotal <= 0 || dlnow > dltotal) { + /* bogus values : stop here */ + return 0; + } + current_size = payload->initial_size + dlnow; /* is our filesize still under any set limit? */ @@ -115,34 +120,15 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, total_size = payload->initial_size + dltotal; - if(dltotal == 0 || payload->prevprogress == total_size) { + if(payload->prevprogress == total_size) { return 0; } - /* initialize the progress bar here to avoid displaying it when - * a repo is up to date and nothing gets downloaded. - * payload->handle->dlcb will receive the remote_name - * and the following arguments: - * 0, -1: download initialized - * 0, 0: non-download event - * x {x>0}, x: download complete - * x {x>0, x<y}, y {y > 0}: download progress, expected total is known */ - if(!payload->cb_initialized) { - cb_data.downloaded = 0; - cb_data.total = -1; - payload->cb_initialized = 1; - } - if(payload->prevprogress == current_size) { - cb_data.downloaded = 0; - cb_data.total = 0; - } else { /* do NOT include initial_size since it wasn't part of the package's * download_size (nor included in the total download size callback) */ - cb_data.downloaded = dlnow; - cb_data.total = dltotal; - } + cb_data.total = dltotal; + cb_data.downloaded = dlnow; payload->handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_PROGRESS, &cb_data); - payload->prevprogress = current_size; return 0; @@ -1184,5 +1170,4 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; - payload->cb_initialized = 0; } diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 3f2fb9ea..c7fd40ee 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -42,7 +42,6 @@ struct dload_payload { int errors_ok; int unlink_on_fail; int trust_remote_name; - int cb_initialized; #ifdef HAVE_LIBCURL CURL *curl; char error_buffer[CURL_ERROR_SIZE]; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 613d59d4..5d42a274 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -18,6 +18,8 @@ * along with this program. If not, see <http://www.gnu.org/licenses/>. */ +#include <assert.h> +#include <stdbool.h> #include <stdio.h> #include <stdlib.h> #include <string.h> @@ -45,6 +47,8 @@ static off_t list_total = 0.0; static int on_progress = 0; static alpm_list_t *output = NULL; +static void cursor_goto_end(void); + /* update speed for the fill_progress based functions */ #define UPDATE_SPEED_MS 200 @@ -152,11 +156,7 @@ static void fill_progress(const int bar_percent, const int disp_percent, printf(" %3d%%", disp_percent); } - if(bar_percent == 100) { - putchar('\n'); - } else { - putchar('\r'); - } + putchar('\r'); fflush(stdout); } @@ -346,6 +346,7 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_DB_RETRIEVE_FAILED: case ALPM_EVENT_PKG_RETRIEVE_DONE: case ALPM_EVENT_PKG_RETRIEVE_FAILED: + cursor_goto_end(); flush_output_list(); on_progress = 0; break; @@ -629,6 +630,7 @@ void cb_progress(alpm_progress_t event, const char *pkgname, int percent, fill_progress(percent, percent, cols - infolen); if(percent == 100) { + putchar('\n'); flush_output_list(); on_progress = 0; } else { @@ -647,124 +649,119 @@ void cb_dl_total(off_t total) } } -/* callback to handle display of download progress */ -static void dload_progress_event(const char *filename, off_t file_xfered, off_t file_total) +static int dload_progressbar_enabled(void) { - static double rate_last; - static off_t xfered_last; - static int64_t initial_time = 0; - int infolen; - int filenamelen; - char *fname, *p; - /* used for wide character width determination and printing */ - int len, wclen, wcwid, padwid; - wchar_t *wcfname; + return !config->noprogressbar && (getcols() != 0); +} - int totaldownload = 0; - off_t xfered, total; - double rate = 0.0; - unsigned int eta_h = 0, eta_m = 0, eta_s = 0; - double rate_human, xfered_human; - const char *rate_label, *xfered_label; - int file_percent = 0, total_percent = 0; +struct dl_progress_bar { + const char *filename; + off_t xfered; + off_t total_size; + uint64_t init_time; /* Time when this download started doing any progress */ + uint64_t sync_time; /* Last time we updated the bar info */ + double rate; + unsigned int eta; /* ETA in seconds */ + bool completed; /* transfer is completed */ +}; - const unsigned short cols = getcols(); +/* List of active downloads handled by multibar UI. + * Once the first download in the list is completed it is removed + * from this list and we never redraw it anymore. + * If the download is in this list, then the UI can redraw the progress bar or change + * the order of the bars (e.g. moving completed bars to the top of the list) + */ +alpm_list_t *pacman_active_downloads = NULL; - /* Nothing has changed since last callback; stop here */ - if(file_xfered == 0 && file_total == 0) { - return; - } +/* Number of active progress bars that multibar UI handles. + * Each progress bar has a corresponding active download. + * Note that some downloads might not have a bar, e.g. optional signature files + * do not have a corresponding bar. + */ +size_t pacman_active_downloads_num = 0; - if(config->noprogressbar || cols == 0) { - if(file_xfered == 0 && file_total == -1) { - printf(_("downloading %s...\n"), filename); - fflush(stdout); - } - return; - } +int multibar_move_completed_up = 0; - infolen = cols * 6 / 10; - if(infolen < 50) { - infolen = 50; - } - /* only use TotalDownload if enabled and we have a callback value */ - if(config->totaldownload && list_total) { - /* sanity check */ - if(list_xfered + file_total <= list_total) { - totaldownload = 1; - } else { - /* bogus values : don't enable totaldownload and reset */ - list_xfered = 0; - list_total = 0; - } +/* Cursor position relative to the first active progress bar, + * e.g. 0 means the first active progress bar, pacman_active_downloads_num-1 means the last bar, + * pacman_active_downloads_num - is the line below all progress bars. + */ +int cursor_lineno = 0; + +/* Move console cursor `lines` up */ +static void cursor_line_up(unsigned int lines) +{ + assert(lines > 0); + printf("\x1B[%dF", lines); +} + +/* Move console cursor `lines` down */ +static void cursor_line_down(unsigned int lines) +{ + assert(lines > 0); + printf("\x1B[%dE", lines); +} + +/* Erase line from the cursor position till the end of the line */ +static void console_erase_line(void) +{ + printf("\x1B[K"); +} + +/* Goto the line that corresponds to num-th active download */ +static void cursor_goto_bar(int num) +{ + if(num > cursor_lineno) { + cursor_line_down(num - cursor_lineno); + } else if(num < cursor_lineno) { + cursor_line_up(cursor_lineno - num); } + cursor_lineno = num; +} - if(totaldownload) { - xfered = list_xfered + file_xfered; - total = list_total; - } else { - xfered = file_xfered; - total = file_total; - } - - /* this is basically a switch on xfered: 0, total, and - * anything else */ - if(file_xfered == 0 && file_total == -1) { - /* set default starting values, ensure we only call this once - * if TotalDownload is enabled */ - if(!totaldownload || (totaldownload && list_xfered == 0)) { - initial_time = get_time_ms(); - xfered_last = (off_t)0; - rate_last = 0.0; - get_update_timediff(1); - } - } else if(xfered > total || xfered < 0) { - /* bogus values : stop here */ - return; - } else if(file_xfered == file_total) { - /* compute final values */ - int64_t timediff = get_time_ms() - initial_time; - if(timediff > 0) { - rate = (double)xfered / (timediff / 1000.0); - /* round elapsed time (in ms) to the nearest second */ - eta_s = (unsigned int)(timediff + 500) / 1000; - } else { - eta_s = 0; - } - } else { - /* compute current average values */ - int64_t timediff = get_update_timediff(0); +/* Goto the line *after* the last active progress bar */ +static void cursor_goto_end(void) +{ + cursor_goto_bar(pacman_active_downloads_num); +} - if(timediff < UPDATE_SPEED_MS) { - /* return if the calling interval was too short */ - return; - } - rate = (double)(xfered - xfered_last) / (timediff / 1000.0); - /* average rate to reduce jumpiness */ - rate = (rate + 2 * rate_last) / 3; - if(rate > 0.0) { - eta_s = (total - xfered) / rate; - } else { - eta_s = UINT_MAX; +/* Returns true if element with the specified name is found, false otherwise */ +static bool find_bar_for_filename(const char *filename, int *index, struct dl_progress_bar **bar) +{ + int i = 0; + alpm_list_t *listitem = pacman_active_downloads; + for(; listitem; listitem = listitem->next, i++) { + struct dl_progress_bar *b = listitem->data; + if (strcmp(b->filename, filename) == 0) { + /* we found a progress bar with the given name */ + *index = i; + *bar = b; + return true; } - rate_last = rate; - xfered_last = xfered; } - if(file_total) { - file_percent = (file_xfered * 100) / file_total; - } else { - file_percent = 100; - } + return false; +} + +static void draw_dl_progress_bar(struct dl_progress_bar *bar) +{ + int infolen; + int filenamelen; + char *fname, *p; + /* used for wide character width determination and printing */ + int len, wclen, wcwid, padwid; + wchar_t *wcfname; + unsigned int eta_h = 0, eta_m = 0, eta_s = bar->eta; + double rate_human, xfered_human; + const char *rate_label, *xfered_label; + int file_percent = 0; - if(totaldownload) { - total_percent = ((list_xfered + file_xfered) * 100) / - list_total; + const unsigned short cols = getcols(); - /* if we are at the end, add the completed file to list_xfered */ - if(file_xfered == file_total) { - list_xfered += file_total; - } + if(bar->total_size) { + file_percent = (bar->xfered * 100) / bar->total_size; + } else { + file_percent = 100; } /* fix up time for display */ @@ -773,23 +770,20 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t eta_m = eta_s / 60; eta_s -= eta_m * 60; - len = strlen(filename); + len = strlen(bar->filename); fname = malloc(len + 1); - memcpy(fname, filename, len + 1); + memcpy(fname, bar->filename, len + 1); /* strip package or DB extension for cleaner look */ if((p = strstr(fname, ".pkg")) || (p = strstr(fname, ".db")) || (p = strstr(fname, ".files"))) { - /* tack on a .sig suffix for signatures */ - if(memcmp(&filename[len - 4], ".sig", 4) == 0) { - memcpy(p, ".sig", 4); - - /* adjust length for later calculations */ - len = p - fname + 4; - } else { - len = p - fname; - } + len = p - fname; fname[len] = '\0'; } + infolen = cols * 6 / 10; + if(infolen < 50) { + infolen = 50; + } + /* 1 space + filenamelen + 1 space + 6 for size + 1 space + 3 for label + * + 2 spaces + 4 for rate + 1 space + 3 for label + 2 for /s + 1 space + * 8 for eta, gives us the magic 33 */ @@ -824,8 +818,8 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t } - rate_human = humanize_size((off_t)rate, '\0', -1, &rate_label); - xfered_human = humanize_size(xfered, '\0', -1, &xfered_label); + rate_human = humanize_size((off_t)bar->rate, '\0', -1, &rate_label); + xfered_human = humanize_size(bar->xfered, '\0', -1, &xfered_label); printf(" %ls%-*s ", wcfname, padwid, ""); /* We will show 1.62 MiB/s, 11.6 MiB/s, but 116 KiB/s and 1116 KiB/s */ @@ -850,19 +844,169 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t free(fname); free(wcfname); - if(totaldownload) { - fill_progress(file_percent, total_percent, cols - infolen); + fill_progress(file_percent, file_percent, cols - infolen); + return; +} + +static void dload_init_event(const char *filename, alpm_download_event_init_t *data) +{ + (void)data; + + if(!dload_progressbar_enabled()) { + printf(_(" %s downloading...\n"), filename); + return; + } + + struct dl_progress_bar *bar = calloc(1, sizeof(struct dl_progress_bar)); + assert(bar); + bar->filename = filename; + bar->init_time = get_time_ms(); + bar->rate = 0.0; + pacman_active_downloads = alpm_list_add(pacman_active_downloads, bar); + + cursor_goto_end(); + printf(_(" %s downloading...\n"), filename); + cursor_lineno++; + pacman_active_downloads_num++; +} + +/* Draws download progress */ +static void dload_progress_event(const char *filename, alpm_download_event_progress_t *data) +{ + int index; + struct dl_progress_bar *bar; + int64_t curr_time = get_time_ms(); + double last_chunk_rate; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + + /* compute current average values */ + timediff = curr_time - bar->sync_time; + + if(timediff < UPDATE_SPEED_MS) { + /* return if the calling interval was too short */ + return; + } + bar->sync_time = curr_time; + + last_chunk_rate = (double)(data->downloaded - bar->xfered) / (timediff / 1000.0); + /* average rate to reduce jumpiness */ + bar->rate = (last_chunk_rate + 2 * bar->rate) / 3; + if(bar->rate > 0.0) { + bar->eta = (data->total - data->downloaded) / bar->rate; } else { - fill_progress(file_percent, file_percent, cols - infolen); + bar->eta = UINT_MAX; + } + + /* Total size is received after the download starts. */ + bar->total_size = data->total; + bar->xfered = data->downloaded; + + cursor_goto_bar(index); + draw_dl_progress_bar(bar); + fflush(stdout); +} + +/* download completed */ +static void dload_complete_event(const char *filename, alpm_download_event_completed_t *data) +{ + int index; + struct dl_progress_bar *bar; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + bar->completed = true; + + /* This may not have been initialized if the download finished before + * an alpm_download_event_progress_t event happened */ + bar->total_size = data->total; + + if(data->result == 1) { + cursor_goto_bar(index); + printf(_(" %s is up to date"), bar->filename); + /* The line contains text from previous status. Erase these leftovers. */ + console_erase_line(); + } else if(data->result == 0) { + /* compute final values */ + bar->xfered = bar->total_size; + timediff = get_time_ms() - bar->init_time; + + /* if transfer was too fast, treat it as a 1ms transfer, for the sake + * of the rate calculation */ + if(timediff < 1) + timediff = 1; + + bar->rate = (double)bar->xfered / (timediff / 1000.0); + /* round elapsed time (in ms) to the nearest second */ + bar->eta = (unsigned int)(timediff + 500) / 1000; + + if(multibar_move_completed_up && index != 0) { + /* If this item completed then move it to the top. + * Swap 0-th bar data with `index`-th one + */ + struct dl_progress_bar *former_topbar = pacman_active_downloads->data; + alpm_list_t *baritem = alpm_list_nth(pacman_active_downloads, index); + pacman_active_downloads->data = bar; + baritem->data = former_topbar; + + cursor_goto_bar(index); + draw_dl_progress_bar(former_topbar); + + index = 0; + } + + cursor_goto_bar(index); + draw_dl_progress_bar(bar); + } else { + cursor_goto_bar(index); + printf(_(" %s failed to download"), bar->filename); + console_erase_line(); + } + fflush(stdout); + + /* If the first bar is completed then there is no reason to keep it + * in the list as we are not going to redraw it anymore. + */ + while(pacman_active_downloads) { + alpm_list_t *head = pacman_active_downloads; + struct dl_progress_bar *j = head->data; + if(j->completed) { + cursor_lineno--; + pacman_active_downloads_num--; + pacman_active_downloads = alpm_list_remove_item(pacman_active_downloads, head); + free(head); + free(j); + } else { + break; + } } - return; } +/* Callback to handle display of download progress */ void cb_download(const char *filename, alpm_download_event_type_t event, void *data) { - if(event == ALPM_DOWNLOAD_PROGRESS) { - alpm_download_event_progress_t *progress = data; - dload_progress_event(filename, progress->downloaded, progress->total); + if(str_endswith(filename, ".sig")) { + return; + } + + if(event == ALPM_DOWNLOAD_INIT) { + dload_init_event(filename, data); + } else if(event == ALPM_DOWNLOAD_PROGRESS) { + dload_progress_event(filename, data); + } else if(event == ALPM_DOWNLOAD_COMPLETED) { + dload_complete_event(filename, data); + } else { + pm_printf(ALPM_LOG_ERROR, _("unknown callback event type %d for %s\n"), + event, filename); } } diff --git a/src/pacman/callback.h b/src/pacman/callback.h index 6d92e86b..7d1f3f08 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -44,4 +44,7 @@ void cb_download(const char *filename, alpm_download_event_type_t event, __attribute__((format(printf, 2, 0))) void cb_log(alpm_loglevel_t level, const char *fmt, va_list args); +/* specify if multibar should move complete bars to the top of the screen */ +int multibar_move_completed_up; + #endif /* PM_CALLBACK_H */ diff --git a/src/pacman/sync.c b/src/pacman/sync.c index f7dcb958..c23b8385 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -35,6 +35,7 @@ #include "pacman.h" #include "util.h" #include "package.h" +#include "callback.h" #include "conf.h" static int unlink_verbose(const char *pathname, int ignore_missing) @@ -824,6 +825,7 @@ int sync_prepare_execute(void) goto cleanup; } + multibar_move_completed_up = 1; if(alpm_trans_commit(config->handle, &data) == -1) { alpm_errno_t err = alpm_errno(config->handle); pm_printf(ALPM_LOG_ERROR, _("failed to commit transaction (%s)\n"), -- 2.26.2

2 2

[pacman-dev] [PATCH] Implement multibar UI
by Anatol Pomozov 07 May '20

07 May '20

Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down. `active_dls` variable represents the list of active progress bars. `struct dl_progress_bar` is a data structure for a progress bar. In some cases we want to keep progress bars in order (e.g. db downloads). In some other cases (package downloads) we want to move completed items to the top of the screen. Global variable `multibar_move_complete_top` allows to configure such behavior. Signature file downloads are not shown at the UI. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/dload.c | 31 +--- lib/libalpm/dload.h | 1 - src/pacman/callback.c | 393 +++++++++++++++++++++++++++++------------- src/pacman/callback.h | 3 + src/pacman/sync.c | 2 + 5 files changed, 285 insertions(+), 145 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index c09abde4..cf467ff0 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -100,6 +100,11 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, return 1; } + if(dlnow < 0 || dltotal <= 0 || dlnow > dltotal) { + /* bogus values : stop here */ + return 0; + } + current_size = payload->initial_size + dlnow; /* is our filesize still under any set limit? */ @@ -115,34 +120,15 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, total_size = payload->initial_size + dltotal; - if(dltotal == 0 || payload->prevprogress == total_size) { + if(payload->prevprogress == total_size) { return 0; } - /* initialize the progress bar here to avoid displaying it when - * a repo is up to date and nothing gets downloaded. - * payload->handle->dlcb will receive the remote_name - * and the following arguments: - * 0, -1: download initialized - * 0, 0: non-download event - * x {x>0}, x: download complete - * x {x>0, x<y}, y {y > 0}: download progress, expected total is known */ - if(!payload->cb_initialized) { - cb_data.downloaded = 0; - cb_data.total = -1; - payload->cb_initialized = 1; - } - if(payload->prevprogress == current_size) { - cb_data.downloaded = 0; - cb_data.total = 0; - } else { /* do NOT include initial_size since it wasn't part of the package's * download_size (nor included in the total download size callback) */ - cb_data.downloaded = dlnow; - cb_data.total = dltotal; - } + cb_data.total = dltotal; + cb_data.downloaded = dlnow; payload->handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_PROGRESS, &cb_data); - payload->prevprogress = current_size; return 0; @@ -1160,5 +1146,4 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; - payload->cb_initialized = 0; } diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index a40b51b7..142568e8 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -42,7 +42,6 @@ struct dload_payload { int errors_ok; int unlink_on_fail; int trust_remote_name; - int cb_initialized; #ifdef HAVE_LIBCURL CURL *curl; char error_buffer[CURL_ERROR_SIZE]; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 387319b6..6133a036 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -18,6 +18,8 @@ * along with this program. If not, see <http://www.gnu.org/licenses/>. */ +#include <assert.h> +#include <stdbool.h> #include <stdio.h> #include <stdlib.h> #include <string.h> @@ -45,6 +47,8 @@ static off_t list_total = 0.0; static int on_progress = 0; static alpm_list_t *output = NULL; +static void cursor_goto_end(void); + /* update speed for the fill_progress based functions */ #define UPDATE_SPEED_MS 200 @@ -152,11 +156,7 @@ static void fill_progress(const int bar_percent, const int disp_percent, printf(" %3d%%", disp_percent); } - if(bar_percent == 100) { - putchar('\n'); - } else { - putchar('\r'); - } + putchar('\r'); fflush(stdout); } @@ -346,6 +346,7 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_DB_RETRIEVE_FAILED: case ALPM_EVENT_RETRIEVE_DONE: case ALPM_EVENT_RETRIEVE_FAILED: + cursor_goto_end(); flush_output_list(); on_progress = 0; break; @@ -629,6 +630,7 @@ void cb_progress(alpm_progress_t event, const char *pkgname, int percent, fill_progress(percent, percent, cols - infolen); if(percent == 100) { + putchar('\n'); flush_output_list(); on_progress = 0; } else { @@ -647,124 +649,118 @@ void cb_dl_total(off_t total) } } -/* callback to handle display of download progress */ -static void dload_progress_event(const char *filename, off_t file_xfered, off_t file_total) +static int dload_progressbar_enabled(void) { - static double rate_last; - static off_t xfered_last; - static int64_t initial_time = 0; - int infolen; - int filenamelen; - char *fname, *p; - /* used for wide character width determination and printing */ - int len, wclen, wcwid, padwid; - wchar_t *wcfname; + return !config->noprogressbar && getcols() != 0; +} - int totaldownload = 0; - off_t xfered, total; - double rate = 0.0; - unsigned int eta_h = 0, eta_m = 0, eta_s = 0; - double rate_human, xfered_human; - const char *rate_label, *xfered_label; - int file_percent = 0, total_percent = 0; +struct dl_progress_bar { + const char *filename; + off_t xfered; + off_t total_size; + uint64_t init_time; /* Time when this download started doing any progress */ + uint64_t sync_time; /* Last time we updated the bar info */ + double rate; + unsigned int eta; /* ETA in seconds */ + bool completed; /* transfer is completed */ +}; - const unsigned short cols = getcols(); +/* List of active downloads handled by multibar UI. + * Once the first download in the list is completed it is removed + * from this list and we never redraw it anymore. + * If the download in this list then UI can redraw the progress bar or change + * the order of the bars (e.g. completed bar is moved to the top of the list) + */ +alpm_list_t *active_dls = NULL; - /* Nothing has changed since last callback; stop here */ - if(file_xfered == 0 && file_total == 0) { - return; - } +/* Number of active progress bars that multibar UI handles. + * Each progress bar has a corresponding active download. + * Note that some downloads might not have a bar, e.g. optional signature files + * do not have a corresponding bar. + */ +size_t active_bars_no = 0; - if(config->noprogressbar || cols == 0) { - if(file_xfered == 0 && file_total == -1) { - printf(_("downloading %s...\n"), filename); - fflush(stdout); - } - return; - } +int multibar_move_complete_top = 0; - infolen = cols * 6 / 10; - if(infolen < 50) { - infolen = 50; - } - /* only use TotalDownload if enabled and we have a callback value */ - if(config->totaldownload && list_total) { - /* sanity check */ - if(list_xfered + file_total <= list_total) { - totaldownload = 1; - } else { - /* bogus values : don't enable totaldownload and reset */ - list_xfered = 0; - list_total = 0; - } +/* Cursor position relative to the first active progress bar, + * e.g. 0 means the first active progress bar, active_bars_no-1 means the last bar, + * active_bars_no - is the line below all progress bars. + */ +int cursor_lineno = 0; + +/* Move console cursor `lines` up */ +static void cursor_line_up(unsigned int lines) +{ + assert(lines > 0); + printf("\x1B[%dF", lines); +} + +/* Move console cursor `lines` down */ +static void cursor_line_down(unsigned int lines) +{ + assert(lines > 0); + printf("\x1B[%dE", lines); +} + +/* Erase line from the cursor position till the end of the line */ +static void console_erase_line(void) +{ + printf("\x1B[K"); +} + +/* Goto the line that corresponds to num-th active download */ +static void cursor_goto_bar(int num) +{ + if(num > cursor_lineno) { + cursor_line_down(num - cursor_lineno); + } else if(num < cursor_lineno) { + cursor_line_up(cursor_lineno - num); } + cursor_lineno = num; +} - if(totaldownload) { - xfered = list_xfered + file_xfered; - total = list_total; - } else { - xfered = file_xfered; - total = file_total; - } - - /* this is basically a switch on xfered: 0, total, and - * anything else */ - if(file_xfered == 0 && file_total == -1) { - /* set default starting values, ensure we only call this once - * if TotalDownload is enabled */ - if(!totaldownload || (totaldownload && list_xfered == 0)) { - initial_time = get_time_ms(); - xfered_last = (off_t)0; - rate_last = 0.0; - get_update_timediff(1); - } - } else if(xfered > total || xfered < 0) { - /* bogus values : stop here */ - return; - } else if(file_xfered == file_total) { - /* compute final values */ - int64_t timediff = get_time_ms() - initial_time; - if(timediff > 0) { - rate = (double)xfered / (timediff / 1000.0); - /* round elapsed time (in ms) to the nearest second */ - eta_s = (unsigned int)(timediff + 500) / 1000; - } else { - eta_s = 0; - } - } else { - /* compute current average values */ - int64_t timediff = get_update_timediff(0); +/* Goto the line *after* the last active progress bar */ +static void cursor_goto_end(void) +{ + cursor_goto_bar(active_bars_no); +} - if(timediff < UPDATE_SPEED_MS) { - /* return if the calling interval was too short */ - return; - } - rate = (double)(xfered - xfered_last) / (timediff / 1000.0); - /* average rate to reduce jumpiness */ - rate = (rate + 2 * rate_last) / 3; - if(rate > 0.0) { - eta_s = (total - xfered) / rate; - } else { - eta_s = UINT_MAX; +/* Returns true if element with the specified name is found, false otherwise */ +static bool find_bar_for_filename(const char *filename, int *index, struct dl_progress_bar **bar) +{ + int i = 0; + alpm_list_t *listitem = active_dls; + for(; listitem; listitem = listitem->next, i++) { + struct dl_progress_bar *b = listitem->data; + if (b->filename == filename /* or strcmp() ? */) { + *index = i; + *bar = b; + return true; } - rate_last = rate; - xfered_last = xfered; } - if(file_total) { - file_percent = (file_xfered * 100) / file_total; - } else { - file_percent = 100; - } + return false; +} - if(totaldownload) { - total_percent = ((list_xfered + file_xfered) * 100) / - list_total; +static void draw_dl_progress_bar(struct dl_progress_bar *bar) +{ + int infolen; + int filenamelen; + char *fname, *p; + /* used for wide character width determination and printing */ + int len, wclen, wcwid, padwid; + wchar_t *wcfname; + unsigned int eta_h = 0, eta_m = 0, eta_s = bar->eta; + double rate_human, xfered_human; + const char *rate_label, *xfered_label; + int file_percent = 0; - /* if we are at the end, add the completed file to list_xfered */ - if(file_xfered == file_total) { - list_xfered += file_total; - } + const unsigned short cols = getcols(); + + if(bar->total_size) { + file_percent = (bar->xfered * 100) / bar->total_size; + } else { + file_percent = 100; } /* fix up time for display */ @@ -773,13 +769,13 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t eta_m = eta_s / 60; eta_s -= eta_m * 60; - len = strlen(filename); + len = strlen(bar->filename); fname = malloc(len + 1); - memcpy(fname, filename, len + 1); + memcpy(fname, bar->filename, len + 1); /* strip package or DB extension for cleaner look */ if((p = strstr(fname, ".pkg")) || (p = strstr(fname, ".db")) || (p = strstr(fname, ".files"))) { /* tack on a .sig suffix for signatures */ - if(memcmp(&filename[len - 4], ".sig", 4) == 0) { + if(memcmp(&bar->filename[len - 4], ".sig", 4) == 0) { memcpy(p, ".sig", 4); /* adjust length for later calculations */ @@ -790,6 +786,11 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t fname[len] = '\0'; } + infolen = cols * 6 / 10; + if(infolen < 50) { + infolen = 50; + } + /* 1 space + filenamelen + 1 space + 6 for size + 1 space + 3 for label + * + 2 spaces + 4 for rate + 1 space + 3 for label + 2 for /s + 1 space + * 8 for eta, gives us the magic 33 */ @@ -824,8 +825,8 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t } - rate_human = humanize_size((off_t)rate, '\0', -1, &rate_label); - xfered_human = humanize_size(xfered, '\0', -1, &xfered_label); + rate_human = humanize_size((off_t)bar->rate, '\0', -1, &rate_label); + xfered_human = humanize_size(bar->xfered, '\0', -1, &xfered_label); printf(" %ls%-*s ", wcfname, padwid, ""); /* We will show 1.62 MiB/s, 11.6 MiB/s, but 116 KiB/s and 1116 KiB/s */ @@ -850,19 +851,169 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t free(fname); free(wcfname); - if(totaldownload) { - fill_progress(file_percent, total_percent, cols - infolen); + fill_progress(file_percent, file_percent, cols - infolen); + return; +} + +static void dload_init_event(const char *filename, alpm_download_event_init_t *data) +{ + (void)data; + + if(!dload_progressbar_enabled()) { + printf(_(" %s downloading...\n"), filename); + return; + } + + struct dl_progress_bar *bar = calloc(1, sizeof(struct dl_progress_bar)); + assert(bar); + bar->filename = filename; + bar->init_time = get_time_ms(); + bar->rate = 0.0; + active_dls = alpm_list_add(active_dls, bar); + + cursor_goto_end(); + printf(_(" %s downloading...\n"), filename); + cursor_lineno++; + active_bars_no++; +} + +/* Draws download progress */ +static void dload_progress_event(const char *filename, alpm_download_event_progress_t *data) +{ + int index; + struct dl_progress_bar *bar; + int64_t curr_time = get_time_ms(); + double last_chunk_rate; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + + /* compute current average values */ + timediff = curr_time - bar->sync_time; + + if(timediff < UPDATE_SPEED_MS) { + /* return if the calling interval was too short */ + return; + } + bar->sync_time = curr_time; + + last_chunk_rate = (double)(data->downloaded - bar->xfered) / (timediff / 1000.0); + /* average rate to reduce jumpiness */ + bar->rate = (last_chunk_rate + 2 * bar->rate) / 3; + if(bar->rate > 0.0) { + bar->eta = (data->total - data->downloaded) / bar->rate; } else { - fill_progress(file_percent, file_percent, cols - infolen); + bar->eta = UINT_MAX; } - return; + + /* Total size is received after the download starts. */ + bar->total_size = data->total; + bar->xfered = data->downloaded; + + cursor_goto_bar(index); + draw_dl_progress_bar(bar); + fflush(stdout); } +/* download completed */ +static void dload_complete_event(const char *filename, alpm_download_event_completed_t *data) +{ + int index; + struct dl_progress_bar *bar; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + bar->completed = true; + + /* This may not have been initialized if the download finished before + * an alpm_download_event_progress_t event happened */ + bar->total_size = data->total; + + if(data->result == 1) { + cursor_goto_bar(index); + printf(_(" %s is up to date"), bar->filename); + /* The line contains text from previous status. Erase these leftovers. */ + console_erase_line(); + } else if(data->result == 0) { + /* compute final values */ + bar->xfered = bar->total_size; + timediff = get_time_ms() - bar->init_time; + + /* if transfer was too fast, treat it as a 1ms transfer, for the sake + * of the rate calculation */ + if(timediff < 1) + timediff = 1; + + bar->rate = (double)bar->xfered / (timediff / 1000.0); + /* round elapsed time (in ms) to the nearest second */ + bar->eta = (unsigned int)(timediff + 500) / 1000; + + if(multibar_move_complete_top && index != 0) { + /* If this item completed then move it to the top. + * Swap 0-th bar data with `index`-th one + */ + struct dl_progress_bar *former_topbar = active_dls->data; + alpm_list_t *baritem = alpm_list_nth(active_dls, index); + active_dls->data = bar; + baritem->data = former_topbar; + + cursor_goto_bar(index); + draw_dl_progress_bar(former_topbar); + + index = 0; + } + + cursor_goto_bar(index); + draw_dl_progress_bar(bar); + } else { + cursor_goto_bar(index); + printf(_(" %s failed to download"), bar->filename); + console_erase_line(); + } + fflush(stdout); + + /* If the first bar is completed then there is no reason to keep it + * in the list as we are not going to redraw it anymore. + */ + while(active_dls) { + alpm_list_t *head = active_dls; + struct dl_progress_bar *j = head->data; + if(j->completed) { + cursor_lineno--; + active_bars_no--; + active_dls = alpm_list_remove_item(active_dls, head); + free(head); + free(j); + } else { + break; + } + } +} + +/* Callback to handle display of download progress */ void cb_download(const char *filename, alpm_download_event_type_t event, void *data) { - if(event == ALPM_DOWNLOAD_PROGRESS) { - alpm_download_event_progress_t *progress = data; - dload_progress_event(filename, progress->downloaded, progress->total); + if(str_endswith(filename, ".sig")) { + return; + } + + if(event == ALPM_DOWNLOAD_INIT) { + dload_init_event(filename, data); + } else if(event == ALPM_DOWNLOAD_PROGRESS) { + dload_progress_event(filename, data); + } else if(event == ALPM_DOWNLOAD_COMPLETED) { + dload_complete_event(filename, data); + } else { + pm_printf(ALPM_LOG_ERROR, _("unknown callback event type %d for %s\n"), + event, filename); } } diff --git a/src/pacman/callback.h b/src/pacman/callback.h index 6d92e86b..916b9dad 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -44,4 +44,7 @@ void cb_download(const char *filename, alpm_download_event_type_t event, __attribute__((format(printf, 2, 0))) void cb_log(alpm_loglevel_t level, const char *fmt, va_list args); +/* specify if multibar should move complete bars to the top of the screen */ +int multibar_move_complete_top; + #endif /* PM_CALLBACK_H */ diff --git a/src/pacman/sync.c b/src/pacman/sync.c index f7dcb958..d5886f13 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -35,6 +35,7 @@ #include "pacman.h" #include "util.h" #include "package.h" +#include "callback.h" #include "conf.h" static int unlink_verbose(const char *pathname, int ignore_missing) @@ -824,6 +825,7 @@ int sync_prepare_execute(void) goto cleanup; } + multibar_move_complete_top = 1; if(alpm_trans_commit(config->handle, &data) == -1) { alpm_errno_t err = alpm_errno(config->handle); pm_printf(ALPM_LOG_ERROR, _("failed to commit transaction (%s)\n"), -- 2.26.2

2 5

[pacman-dev] Setting HOME in build() breaks package signing
by brainpower 07 May '20

07 May '20

Hi, a while back I came across a PKGBUILD which changed $HOME in build() as a workaround for a bad installer [1]. For some reason this failed to build with `makepkg --sign`, claiming it could not find my gpg key, see snippet below. After some time debugging this, I found changing $HOME broke the check for the key because gpg would look for it's .gnupg dir in the "new" $HOME, not the $HOME makepkg was started with. I solved this by editing the PKGBUILD to restore $HOME before exiting build(). I had thought changing $HOME was only used in that single case and didn't think much of it, but today a PKGBUILD [3] posted in a question on aur-general [2] reminded me of this and it seems this workaround is considered by packagers more commonly than I thought... This made me wonder: Is this something makepkg should take care of (e.g.by restoring $HOME after build() or ensuring gpg will use $OLDHOME/.gnupg) or should such a PKGBUILD be considered broken / invalid? $ makepkg --sign ==> Making package: broken-home 1-1 (Tue May 5 21:35:57 2020) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Retrieving sources... ==> Extracting sources... ==> Removing existing $pkgdir/ directory... ==> Starting build()... ==> Entering fakeroot environment... ==> ERROR: The key 06ABC843BA90E65B does not exist in your keyring. $ gpg --list-key 06ABC843BA90E65B :( pub rsa4096 2019-03-05 [SC] 59EB8C1AF8CFEBF4A683760206ABC843BA90E65B uid [ultimate] package signing key <brainpower(a)mailbox.org> sub rsa4096 2019-03-05 [E] $ cat PKGBUILD pkgname=broken-home pkgver=1 pkgrel=1 arch=('any') build() { export HOME="${srcdir}/tmphome" } package() { true } [1]: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=mongodb-compass-isol… [2]: https://lists.archlinux.org/pipermail/aur-general/2020-May/035729.html [3]: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=intel-opencl-sdk&id=… -- regards, brainpower

2 2