- Pacman-dev - lists.archlinux.org

[pacman-dev] [Proposal] Hold the first working mirror when downloading
by Marco Rubin 22 Mar '21

22 Mar '21

Hello, I'd like to propose a change to pacman, but I don't know if someone is already working on it for the next release. I think it would be useful if pacman could hold the first working mirror when downloading updates. Yesterday I was upgrading all my packages and, for some reason, the first five mirrors timed out after a few seconds, so every time pacman was downloading a package, it checked all the mirrors from the beginning until it found one where to grab the updated package. I had many old packages so this process quickly became a little frustrating, and to me it would have been faster if it remembered the first working mirror instead of checking all of them every time.

3 3

[pacman-dev] [PATCH] RFC: Remove "duplicate" download error messages
by Allan McRae 20 Mar '21

20 Mar '21

This is a very incomplete and hacky patch... But it does do this: allan@mando ~/arch/code/pacman (patchqueue) > pacman -Syy :: Synchronizing package databases... core 131.1 KiB 47.0 KiB/s 00:03 [######################] 100% extra 1645.7 KiB 490 KiB/s 00:03 [######################] 100% community 5.4 MiB 161 KiB/s 00:34 [######################] 100% error: failed retrieving file 'extra.db' from allanmcrae.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from allanmcrae.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from allanmcrae.com : The requested URL returned error: 404 allan@mando ~/arch/code/pacman (patchqueue) > sudo ./build/pacman -Syy [sudo] password for allan: :: Synchronizing package databases... core 131.1 KiB 51.2 KiB/s 00:03 [######################] 100% extra 1645.7 KiB 494 KiB/s 00:03 [######################] 100% community 5.4 MiB 172 KiB/s 00:32 [######################] 100% error: failed retrieving file(s) from allanmcrae.com : The requested URL returned error: 404 There are a few things to note. 1) we lose the file names for the failed downloads. And we lose information of how many downloads failed. 2) I only changed the messages for download failures so far. So (e.g.) having a file exceed its expected download size will still report the file name. 3) debug output still gives file names 4) note the "error" in the messages above, yet pacman was still successful as I have an actual mirror configured. I guess this (and others?) should be changed to warning. We get an overall error message if there is complete failure anyway. 5) I have been super lazy for this PoC, switching alpm_list_remove_dupes() to use string comparisons, rather than pointer comparisons. This will need fixed. Things to discuss: - do we need to report the number of failures from a server. Currently 1 and 200 failures are treated the same. This involves doing more than a dedup of the error list. - converting errors to warnings. They are clearly not errors until they are... Is the final line enough: error: failed to synchronize all databases (failed to retrieve some files) --- lib/libalpm/alpm_list.c | 2 +- lib/libalpm/dload.c | 12 ++++++------ src/pacman/callback.c | 7 +++++-- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/lib/libalpm/alpm_list.c b/lib/libalpm/alpm_list.c index 5a684181..bc243898 100644 --- a/lib/libalpm/alpm_list.c +++ b/lib/libalpm/alpm_list.c @@ -338,7 +338,7 @@ alpm_list_t SYMEXPORT *alpm_list_remove_dupes(const alpm_list_t *list) const alpm_list_t *lp = list; alpm_list_t *newlist = NULL; while(lp) { - if(!alpm_list_find_ptr(newlist, lp->data)) { + if(!alpm_list_find_str(newlist, lp->data)) { if(alpm_list_append(&newlist, lp->data) == NULL) { alpm_list_free(newlist); return NULL; diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 0d0936c7..702bd059 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -421,8 +421,8 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, snprintf(payload->error_buffer, sizeof(payload->error_buffer), "The requested URL returned error: %ld", payload->respcode); _alpm_log(handle, ALPM_LOG_ERROR, - _("failed retrieving file '%s' from %s : %s\n"), - payload->remote_name, hostname, payload->error_buffer); + _("failed retrieving file(s) from %s : %s\n"), + hostname, payload->error_buffer); } if(curl_retry_next_server(curlm, curl, payload) == 0) { return 2; @@ -446,8 +446,8 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, payload->unlink_on_fail = 1; handle->pm_errno = ALPM_ERR_SERVER_BAD_URL; _alpm_log(handle, ALPM_LOG_ERROR, - _("failed retrieving file '%s' from %s : %s\n"), - payload->remote_name, hostname, payload->error_buffer); + _("failed retrieving file(s) from %s : %s\n"), + hostname, payload->error_buffer); if(curl_retry_next_server(curlm, curl, payload) == 0) { return 2; } else { @@ -461,8 +461,8 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, if(!payload->errors_ok) { handle->pm_errno = ALPM_ERR_LIBCURL; _alpm_log(handle, ALPM_LOG_ERROR, - _("failed retrieving file '%s' from %s : %s\n"), - payload->remote_name, hostname, payload->error_buffer); + _("failed retrieving file(s) from %s : %s\n"), + hostname, payload->error_buffer); } else { _alpm_log(handle, ALPM_LOG_DEBUG, "failed retrieving file '%s' from %s : %s\n", diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 40973340..d07df816 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -212,9 +212,12 @@ static void fill_progress(const int bar_percent, const int disp_percent, } static void flush_output_list(void) { - alpm_list_t *i = NULL; + alpm_list_t *i = NULL, *rmdup; fflush(stdout); - for(i = output; i; i = i->next) { + + rmdup = alpm_list_remove_dupes(output); + + for(i = rmdup; i; i = i->next) { fputs((const char *)i->data, stderr); } fflush(stderr); -- 2.30.1

3 4

[pacman-dev] [PATCH v2] Remove "total download" callback in favor of generic event callback
by Anatol Pomozov 16 Mar '21

16 Mar '21

Total download callback called right before packages start downloaded. But we already have an event for such event (ALPM_EVENT_PKG_RETRIEVE_START) and it is naturally to use the event to pass information about expected download size. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- README | 1 - lib/libalpm/alpm.h | 32 ++++++++++++-------------------- lib/libalpm/dload.c | 2 +- lib/libalpm/handle.c | 13 ------------- lib/libalpm/handle.h | 1 - lib/libalpm/sync.c | 25 +++++++++---------------- src/pacman/callback.c | 9 ++------- src/pacman/callback.h | 2 -- src/pacman/conf.c | 4 ---- 9 files changed, 24 insertions(+), 65 deletions(-) diff --git a/README b/README index 6aa68374..470ccf3c 100644 --- a/README +++ b/README @@ -52,7 +52,6 @@ library is initialized. * logcb: The callback function for "log" operations. * dlcb: The callback function for download progress of each package. * fetchcb: Callback for custom download function. -* totaldlcb: The callback function for overall download progress. * eventcb: Callback for transaction messages. * questioncb: Callback for selecting amongst choices. * progresscb: Callback to handle display of transaction progress. diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 0909fe11..833df829 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -926,6 +926,16 @@ typedef struct _alpm_event_hook_run_t { size_t total; } alpm_event_hook_run_t; +/** Packages downloading about to start. */ +typedef struct _alpm_event_pkg_retrieve_t { + /** Type of event */ + alpm_event_type_t type; + /** Number of packages to download */ + size_t num; + /** Total size of packages to download */ + off_t total_size; +} alpm_event_pkg_retrieve_t; + /** Events. * This is a union passed to the callback that allows the frontend to know * which type of event was triggered (via type). It is then possible to @@ -954,6 +964,8 @@ typedef union _alpm_event_t { alpm_event_hook_t hook; /** A hook was ran */ alpm_event_hook_run_t hook_run; + /** Download packages */ + alpm_event_pkg_retrieve_t pkg_retrieve; } alpm_event_t; /** Event callback. @@ -1197,12 +1209,6 @@ typedef void (*alpm_cb_download)(const char *filename, alpm_download_event_type_t event, void *data); -/** Total Download callback. - * @param howmany the number of packages that will be downloaded during \link alpm_trans_commit \endlink. - * @param total amount that will be downloaded during \link alpm_trans_commit \endlink. - */ -typedef void (*alpm_cb_totaldl)(size_t howmany, off_t total); - /** A callback for downloading files * @param url the URL of the file to be downloaded * @param localpath the directory to which the file should be downloaded @@ -1525,20 +1531,6 @@ alpm_cb_fetch alpm_option_get_fetchcb(alpm_handle_t *handle); */ int alpm_option_set_fetchcb(alpm_handle_t *handle, alpm_cb_fetch cb); -/** Returns the callback used to report total download size. - * @param handle the context handle - * @return the currently set total download callback - */ -alpm_cb_totaldl alpm_option_get_totaldlcb(alpm_handle_t *handle); - -/** Sets the callback used to report total download size. - * @param handle the context handle - * @param cb the cb to use - * @return 0 on success, -1 on error (pm_errno is set accordingly) - */ -int alpm_option_set_totaldlcb(alpm_handle_t *handle, alpm_cb_totaldl cb); - - /** Returns the callback used for events. * @param handle the context handle * @return the currently set event callback diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 0d0936c7..0423a139 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -851,7 +851,7 @@ int SYMEXPORT alpm_fetch_pkgurl(alpm_handle_t *handle, const alpm_list_t *urls, const char *cachedir; alpm_list_t *payloads = NULL; const alpm_list_t *i; - alpm_event_t event; + alpm_event_t event = {0}; CHECK_HANDLE(handle, return -1); ASSERT(*fetched == NULL, RET_ERR(handle, ALPM_ERR_WRONG_ARGS, -1)); diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index d1ea5886..7b8cb1da 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -174,12 +174,6 @@ alpm_cb_fetch SYMEXPORT alpm_option_get_fetchcb(alpm_handle_t *handle) return handle->fetchcb; } -alpm_cb_totaldl SYMEXPORT alpm_option_get_totaldlcb(alpm_handle_t *handle) -{ - CHECK_HANDLE(handle, return NULL); - return handle->totaldlcb; -} - alpm_cb_event SYMEXPORT alpm_option_get_eventcb(alpm_handle_t *handle) { CHECK_HANDLE(handle, return NULL); @@ -327,13 +321,6 @@ int SYMEXPORT alpm_option_set_fetchcb(alpm_handle_t *handle, alpm_cb_fetch cb) return 0; } -int SYMEXPORT alpm_option_set_totaldlcb(alpm_handle_t *handle, alpm_cb_totaldl cb) -{ - CHECK_HANDLE(handle, return -1); - handle->totaldlcb = cb; - return 0; -} - int SYMEXPORT alpm_option_set_eventcb(alpm_handle_t *handle, alpm_cb_event cb) { CHECK_HANDLE(handle, return -1); diff --git a/lib/libalpm/handle.h b/lib/libalpm/handle.h index 349ef2c7..2ebaafb2 100644 --- a/lib/libalpm/handle.h +++ b/lib/libalpm/handle.h @@ -72,7 +72,6 @@ struct __alpm_handle_t { /* callback functions */ alpm_cb_log logcb; /* Log callback function */ alpm_cb_download dlcb; /* Download callback function */ - alpm_cb_totaldl totaldlcb; /* Total download callback function */ alpm_cb_fetch fetchcb; /* Download file callback function */ alpm_cb_event eventcb; alpm_cb_question questioncb; diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index d258b8d1..7fa50a13 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -749,7 +749,7 @@ static int download_files(alpm_handle_t *handle) const char *cachedir; alpm_list_t *i, *files = NULL; int ret = 0; - alpm_event_t event; + alpm_event_t event = {0}; alpm_list_t *payloads = NULL; cachedir = _alpm_filecache_setup(handle); @@ -760,21 +760,6 @@ static int download_files(alpm_handle_t *handle) goto finish; } - /* Total progress - figure out the total download size if required to - * pass to the callback. This function is called once, and it is up to the - * frontend to compute incremental progress. */ - if(handle->totaldlcb) { - off_t total_size = (off_t)0; - size_t howmany = 0; - /* sum up the download size for each package and store total */ - for(i = files; i; i = i->next) { - alpm_pkg_t *spkg = i->data; - total_size += spkg->download_size; - howmany++; - } - handle->totaldlcb(howmany, total_size); - } - if(files) { /* check for necessary disk space for download */ if(handle->checkspace) { @@ -800,6 +785,14 @@ static int download_files(alpm_handle_t *handle) } event.type = ALPM_EVENT_PKG_RETRIEVE_START; + + /* sum up the number of packages to download and its total size */ + for(i = files; i; i = i->next) { + alpm_pkg_t *spkg = i->data; + event.pkg_retrieve.total_size += spkg->download_size; + event.pkg_retrieve.num++; + } + EVENT(handle, &event); for(i = files; i; i = i->next) { alpm_pkg_t *pkg = i->data; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 40973340..a28a79a9 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -338,6 +338,8 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_PKG_RETRIEVE_START: colon_printf(_("Retrieving packages...\n")); on_progress = 1; + list_total_pkgs = event->pkg_retrieve.num; + list_total = event->pkg_retrieve.total_size; total_enabled = config->totaldownload && list_total; if(total_enabled) { init_total_progressbar(); @@ -696,13 +698,6 @@ void cb_progress(alpm_progress_t event, const char *pkgname, int percent, } } -/* callback to handle receipt of total download value */ -void cb_dl_total(size_t howmany, off_t total) -{ - list_total_pkgs = howmany; - list_total = total; -} - static int dload_progressbar_enabled(void) { return !config->noprogressbar && (getcols() != 0); diff --git a/src/pacman/callback.h b/src/pacman/callback.h index 9b5deb6f..8ac9d960 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -35,8 +35,6 @@ void cb_question(alpm_question_t *question); void cb_progress(alpm_progress_t event, const char *pkgname, int percent, size_t howmany, size_t remain); -/* callback to handle receipt of total download value */ -void cb_dl_total(size_t howmany, off_t total); /* callback to handle display of download progress */ void cb_download(const char *filename, alpm_download_event_type_t event, void *data); diff --git a/src/pacman/conf.c b/src/pacman/conf.c index 96e5790f..a4f2ba35 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -892,10 +892,6 @@ static int setup_libalpm(void) pm_printf(ALPM_LOG_WARNING, _("no '%s' configured\n"), "XferCommand"); } - if(config->totaldownload) { - alpm_option_set_totaldlcb(handle, cb_dl_total); - } - alpm_option_set_arch(handle, config->arch); alpm_option_set_checkspace(handle, config->checkspace); alpm_option_set_usesyslog(handle, config->usesyslog); -- 2.30.1

2 2

[pacman-dev] [PATCH v3] Remove "total download" callback in favor of generic event callback
by Anatol Pomozov 16 Mar '21

16 Mar '21

Total download callback called right before packages start downloaded. But we already have an event for such event (ALPM_EVENT_PKG_RETRIEVE_START) and it is naturally to use the event to pass information about expected download size. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- README | 1 - lib/libalpm/alpm.h | 32 ++++++++++++-------------------- lib/libalpm/dload.c | 3 ++- lib/libalpm/handle.c | 13 ------------- lib/libalpm/handle.h | 1 - lib/libalpm/sync.c | 25 +++++++++---------------- src/pacman/callback.c | 9 ++------- src/pacman/callback.h | 2 -- src/pacman/conf.c | 4 ---- 9 files changed, 25 insertions(+), 65 deletions(-) diff --git a/README b/README index 6aa68374..470ccf3c 100644 --- a/README +++ b/README @@ -52,7 +52,6 @@ library is initialized. * logcb: The callback function for "log" operations. * dlcb: The callback function for download progress of each package. * fetchcb: Callback for custom download function. -* totaldlcb: The callback function for overall download progress. * eventcb: Callback for transaction messages. * questioncb: Callback for selecting amongst choices. * progresscb: Callback to handle display of transaction progress. diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 0909fe11..833df829 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -926,6 +926,16 @@ typedef struct _alpm_event_hook_run_t { size_t total; } alpm_event_hook_run_t; +/** Packages downloading about to start. */ +typedef struct _alpm_event_pkg_retrieve_t { + /** Type of event */ + alpm_event_type_t type; + /** Number of packages to download */ + size_t num; + /** Total size of packages to download */ + off_t total_size; +} alpm_event_pkg_retrieve_t; + /** Events. * This is a union passed to the callback that allows the frontend to know * which type of event was triggered (via type). It is then possible to @@ -954,6 +964,8 @@ typedef union _alpm_event_t { alpm_event_hook_t hook; /** A hook was ran */ alpm_event_hook_run_t hook_run; + /** Download packages */ + alpm_event_pkg_retrieve_t pkg_retrieve; } alpm_event_t; /** Event callback. @@ -1197,12 +1209,6 @@ typedef void (*alpm_cb_download)(const char *filename, alpm_download_event_type_t event, void *data); -/** Total Download callback. - * @param howmany the number of packages that will be downloaded during \link alpm_trans_commit \endlink. - * @param total amount that will be downloaded during \link alpm_trans_commit \endlink. - */ -typedef void (*alpm_cb_totaldl)(size_t howmany, off_t total); - /** A callback for downloading files * @param url the URL of the file to be downloaded * @param localpath the directory to which the file should be downloaded @@ -1525,20 +1531,6 @@ alpm_cb_fetch alpm_option_get_fetchcb(alpm_handle_t *handle); */ int alpm_option_set_fetchcb(alpm_handle_t *handle, alpm_cb_fetch cb); -/** Returns the callback used to report total download size. - * @param handle the context handle - * @return the currently set total download callback - */ -alpm_cb_totaldl alpm_option_get_totaldlcb(alpm_handle_t *handle); - -/** Sets the callback used to report total download size. - * @param handle the context handle - * @param cb the cb to use - * @return 0 on success, -1 on error (pm_errno is set accordingly) - */ -int alpm_option_set_totaldlcb(alpm_handle_t *handle, alpm_cb_totaldl cb); - - /** Returns the callback used for events. * @param handle the context handle * @return the currently set event callback diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 0d0936c7..3e20d335 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -851,7 +851,7 @@ int SYMEXPORT alpm_fetch_pkgurl(alpm_handle_t *handle, const alpm_list_t *urls, const char *cachedir; alpm_list_t *payloads = NULL; const alpm_list_t *i; - alpm_event_t event; + alpm_event_t event = {0}; CHECK_HANDLE(handle, return -1); ASSERT(*fetched == NULL, RET_ERR(handle, ALPM_ERR_WRONG_ARGS, -1)); @@ -884,6 +884,7 @@ int SYMEXPORT alpm_fetch_pkgurl(alpm_handle_t *handle, const alpm_list_t *urls, if(payloads) { event.type = ALPM_EVENT_PKG_RETRIEVE_START; + event.pkg_retrieve.num = alpm_list_count(payloads); EVENT(handle, &event); if(_alpm_download(handle, payloads, cachedir) == -1) { _alpm_log(handle, ALPM_LOG_WARNING, _("failed to retrieve some files\n")); diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index d1ea5886..7b8cb1da 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -174,12 +174,6 @@ alpm_cb_fetch SYMEXPORT alpm_option_get_fetchcb(alpm_handle_t *handle) return handle->fetchcb; } -alpm_cb_totaldl SYMEXPORT alpm_option_get_totaldlcb(alpm_handle_t *handle) -{ - CHECK_HANDLE(handle, return NULL); - return handle->totaldlcb; -} - alpm_cb_event SYMEXPORT alpm_option_get_eventcb(alpm_handle_t *handle) { CHECK_HANDLE(handle, return NULL); @@ -327,13 +321,6 @@ int SYMEXPORT alpm_option_set_fetchcb(alpm_handle_t *handle, alpm_cb_fetch cb) return 0; } -int SYMEXPORT alpm_option_set_totaldlcb(alpm_handle_t *handle, alpm_cb_totaldl cb) -{ - CHECK_HANDLE(handle, return -1); - handle->totaldlcb = cb; - return 0; -} - int SYMEXPORT alpm_option_set_eventcb(alpm_handle_t *handle, alpm_cb_event cb) { CHECK_HANDLE(handle, return -1); diff --git a/lib/libalpm/handle.h b/lib/libalpm/handle.h index 349ef2c7..2ebaafb2 100644 --- a/lib/libalpm/handle.h +++ b/lib/libalpm/handle.h @@ -72,7 +72,6 @@ struct __alpm_handle_t { /* callback functions */ alpm_cb_log logcb; /* Log callback function */ alpm_cb_download dlcb; /* Download callback function */ - alpm_cb_totaldl totaldlcb; /* Total download callback function */ alpm_cb_fetch fetchcb; /* Download file callback function */ alpm_cb_event eventcb; alpm_cb_question questioncb; diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index d258b8d1..7fa50a13 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -749,7 +749,7 @@ static int download_files(alpm_handle_t *handle) const char *cachedir; alpm_list_t *i, *files = NULL; int ret = 0; - alpm_event_t event; + alpm_event_t event = {0}; alpm_list_t *payloads = NULL; cachedir = _alpm_filecache_setup(handle); @@ -760,21 +760,6 @@ static int download_files(alpm_handle_t *handle) goto finish; } - /* Total progress - figure out the total download size if required to - * pass to the callback. This function is called once, and it is up to the - * frontend to compute incremental progress. */ - if(handle->totaldlcb) { - off_t total_size = (off_t)0; - size_t howmany = 0; - /* sum up the download size for each package and store total */ - for(i = files; i; i = i->next) { - alpm_pkg_t *spkg = i->data; - total_size += spkg->download_size; - howmany++; - } - handle->totaldlcb(howmany, total_size); - } - if(files) { /* check for necessary disk space for download */ if(handle->checkspace) { @@ -800,6 +785,14 @@ static int download_files(alpm_handle_t *handle) } event.type = ALPM_EVENT_PKG_RETRIEVE_START; + + /* sum up the number of packages to download and its total size */ + for(i = files; i; i = i->next) { + alpm_pkg_t *spkg = i->data; + event.pkg_retrieve.total_size += spkg->download_size; + event.pkg_retrieve.num++; + } + EVENT(handle, &event); for(i = files; i; i = i->next) { alpm_pkg_t *pkg = i->data; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 40973340..a28a79a9 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -338,6 +338,8 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_PKG_RETRIEVE_START: colon_printf(_("Retrieving packages...\n")); on_progress = 1; + list_total_pkgs = event->pkg_retrieve.num; + list_total = event->pkg_retrieve.total_size; total_enabled = config->totaldownload && list_total; if(total_enabled) { init_total_progressbar(); @@ -696,13 +698,6 @@ void cb_progress(alpm_progress_t event, const char *pkgname, int percent, } } -/* callback to handle receipt of total download value */ -void cb_dl_total(size_t howmany, off_t total) -{ - list_total_pkgs = howmany; - list_total = total; -} - static int dload_progressbar_enabled(void) { return !config->noprogressbar && (getcols() != 0); diff --git a/src/pacman/callback.h b/src/pacman/callback.h index 9b5deb6f..8ac9d960 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -35,8 +35,6 @@ void cb_question(alpm_question_t *question); void cb_progress(alpm_progress_t event, const char *pkgname, int percent, size_t howmany, size_t remain); -/* callback to handle receipt of total download value */ -void cb_dl_total(size_t howmany, off_t total); /* callback to handle display of download progress */ void cb_download(const char *filename, alpm_download_event_type_t event, void *data); diff --git a/src/pacman/conf.c b/src/pacman/conf.c index 96e5790f..a4f2ba35 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -892,10 +892,6 @@ static int setup_libalpm(void) pm_printf(ALPM_LOG_WARNING, _("no '%s' configured\n"), "XferCommand"); } - if(config->totaldownload) { - alpm_option_set_totaldlcb(handle, cb_dl_total); - } - alpm_option_set_arch(handle, config->arch); alpm_option_set_checkspace(handle, config->checkspace); alpm_option_set_usesyslog(handle, config->usesyslog); -- 2.31.0

1 0

[pacman-dev] [PATCH] remove sed command lookup and hardcoding in edit-script.sh
by Eli Schwartz 14 Mar '21

14 Mar '21

We should not need to hardcode the path to sed as we simply don't care. We don't check what kind of sed we found, and we're using the same one we initially found on the PATH, which is surely still on the PATH. At one point we did care to find the system copy of sed and hardcode it in makepkg, because we also passed non-portable -i options to it and makepkg needed to continue working on macOS even if some incompatible GNU sed got installed afterward, elsewhere on the PATH. But this was never relevant to the in-tree buildsystem script running sed. In commit 3a814ee6bca9ee24a868c0dc032b321048a53e08 we removed even that, so we don't need to look it up at all. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- build-aux/edit-script.sh.in | 2 +- meson.build | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/build-aux/edit-script.sh.in b/build-aux/edit-script.sh.in index 661c22d5b..85c56cfe9 100644 --- a/build-aux/edit-script.sh.in +++ b/build-aux/edit-script.sh.in @@ -4,7 +4,7 @@ input=$1 output=$2 mode=$3 -"@SED@" \ +sed \ -e "s|@rootdir[@]|@ROOTDIR@|g" \ -e "s|@localedir[@]|@LOCALEDIR@|g" \ -e "s|@sysconfdir[@]|@sysconfdir@|g" \ diff --git a/meson.build b/meson.build index ea9bd2f7a..8ccd5c78b 100644 --- a/meson.build +++ b/meson.build @@ -28,7 +28,6 @@ LIBMAKEPKGDIR = join_paths(PREFIX, DATAROOTDIR, 'makepkg') PKGDATADIR = join_paths(PREFIX, DATAROOTDIR, meson.project_name()) PYTHON = import('python').find_installation('python3') -SED = find_program('sed') DU = find_program('du') LDCONFIG = get_option('ldconfig') MESON_MAKE_SYMLINK = join_paths(meson.source_root(), 'build-aux/meson-make-symlink.sh') @@ -251,7 +250,6 @@ carch = chost.split('-')[0] # annoyingly, we have to maintain two sets of configuration_data which is # largely identical, but which distinguishes between quoting needs. substs = configuration_data() -substs.set('SED', SED.path()) substs.set('CARCH', carch) substs.set('CHOST', chost) substs.set('PKGEXT', get_option('pkg-ext')) -- 2.30.2

1 0

[pacman-dev] Blockers for releasing pacman-6.0.0?
by Allan McRae 09 Mar '21

09 Mar '21

Hi all, I think the 6.0.0 release is roughly ready to go. Here are my known blockers: 1) Remove "total download" callback There is a patch submitted which I briefly looked at. Need to do a proper review, but looked fine apart from copy/paste issue with comment. 2) prevent spamming at the end of downloads when the top server in the mirrorlist is bad. I sent an RFC for this. It is not the best of patches, but serves as a start... The only other things I have in the roadmap on Flyspray are: FS#61497 - add a timestamp file to repos FS#65677 - [makepkg] not all source files get included in debug packages Both are nice to have, but not regressions so can be delayed a release (unless someone else propose patches in the meantime). Is there anything else that people think is essential to get in pacman-6.0.0? I know there are a few other downloader issues listed in the meta-bug, but I think these are not critical and can be future improvements. Allan

4 5

[pacman-dev] [GIT] The official pacman repository branch, master, updated. v6.0.0alpha1-76-gede73f4d
by Allan McRae 06 Mar '21

06 Mar '21

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, master has been updated via ede73f4d898946a7e54634169b3605a024553315 (commit) via c118a61f62a4b100b01504dfc5611cd08fb9e537 (commit) via 4a0891f49d29458e1b93d95f8121f8096bb02727 (commit) from 17f9911ffc79d60d1bebb4484fac82cb1ed78d2a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit ede73f4d898946a7e54634169b3605a024553315 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Tue Mar 2 23:23:55 2021 -0500 fix syntax error when running pacman-key --help In commit 0f75ab3224141a5e8a4fe72b48b4639c83c6316b some unbalanced quotes were added by the committer while editing an error message. Fixes FS#69865 Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit c118a61f62a4b100b01504dfc5611cd08fb9e537 Author: Allan McRae <allan(a)archlinux.org> Date: Tue Mar 2 10:53:06 2021 +1000 Strip LTO symbols from distributed .a/.o files GCC's LTO implementation emits bytecodes into .o files it generates. These bytecodes are _not_ considered stable from one release of GCC to the next. There we need to strip the LTO bytecode out of any .o (and .a) file that gets installed into the package. Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 4a0891f49d29458e1b93d95f8121f8096bb02727 Author: Allan McRae <allan(a)archlinux.org> Date: Mon Mar 1 14:08:18 2021 +1000 Add link time optimization support to makepkg Add the 'lto' option to enable building with link time optimization by adding '-flto' to both CFLAGS and CXXFLAGS. The 'lto' option can be specificed both in the PKGBUILD or by setting the default in makepkg.conf. Signed-off-by: Allan McRae <allan(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: doc/PKGBUILD.5.asciidoc | 4 ++++ doc/makepkg.conf.5.asciidoc | 4 ++++ etc/makepkg.conf.in | 5 +++-- .../buildenv/{makeflags.sh.in => lto.sh.in} | 20 +++++++++++--------- scripts/libmakepkg/buildenv/meson.build | 1 + scripts/libmakepkg/tidy/strip.sh.in | 13 +++++++++++++ scripts/pacman-key.sh.in | 2 +- 7 files changed, 37 insertions(+), 12 deletions(-) copy scripts/libmakepkg/buildenv/{makeflags.sh.in => lto.sh.in} (65%) hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [PATCH v2] pacman-key: --refresh-keys queries WKD before keyserver
by Morten Linderud 03 Mar '21

03 Mar '21

From: Morten Linderud <morten(a)linderud.pw> With the recent outages of the keyservers there is a possibility of `--refresh-keys` failing to fetch new keys. A lot of current key distribution is done over WKD these days, and `pacman-key` has the ability to use it for `--recv-key`. There was a hope `gpg` would end up supporting WKD for the refresh functionality, but this seems to be limited to expired keys fetched through WKD. Since this functionality isn't yet available it makes sense to stuff it into `pacman-key`. The current implementation looks over all available keyids in the keyring, attempts to fetch over WKD and then fall backs to keyservers if no email has a valid WKD available. The downside of this approach is that it takes a bit longer to refresh the keys, but it should be more robust as the distribution should be providing their own WKDs. Co-authored-by: Jonas Witschel <diabonas(a)archlinux.org> Signed-off-by: Morten Linderud <morten(a)linderud.pw> --- * Done grep -vx * Removed the redundant error since it's caught by `check_keyids_exist` * Improved the error message with the keyid scripts/pacman-key.sh.in | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/scripts/pacman-key.sh.in b/scripts/pacman-key.sh.in index c65669f5..e89d7af9 100644 --- a/scripts/pacman-key.sh.in +++ b/scripts/pacman-key.sh.in @@ -540,11 +540,36 @@ receive_keys() { } refresh_keys() { + local ret=0 ids masterkey emails + check_keyids_exist "$@" - if ! "${GPG_PACMAN[@]}" --refresh-keys "$@" ; then - error "$(gettext "A specified local key could not be updated from a keyserver.")" - exit 1 - fi + + # don't try to refresh the user's local masterkey + masterkey="$("${GPG_PACMAN[@]}" --list-keys --with-colons pacman@localhost | + awk -F: '$1 == "pub" { print $5 }')" + + mapfile -t ids < \ + <("${GPG_PACMAN[@]}" --list-keys --with-colons "$@" | + awk -F: '$1 == "pub" { print $5 }' | grep -vx "$masterkey") + + for id in "${ids[@]}"; do + mapfile -t emails < \ + <("${GPG_PACMAN[@]}" --list-keys --list-options show-only-fpr-mbox "$id" | + awk '{print $2 }') + + # first try looking up the key in a WKD (only works by email address) + for email in "${emails[@]}"; do + "${GPG_PACMAN[@]}" --locate-external-keys "$email" && break + done + + # if no key was found, fall back to using the keyservers (with the key fingerprint instead) + if (( $? )) && ! "${GPG_PACMAN[@]}" --refresh-keys "$id"; then + error "$(gettext "The following key could not be updated from WKD or keyserver: %s")" "$id" + ret=1 + fi + done + + exit $ret } verify_sig() { -- 2.30.1

3 3

[pacman-dev] [PATCH] Add link time optimization support to makepkg
by Allan McRae 03 Mar '21

03 Mar '21

Add the 'lto' option to enable building with link time optimization by adding '-flto' to both CFLAGS and CXXFLAGS. The 'lto' option can be specificed both in the PKGBUILD or by setting the default in makepkg.conf. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- TODO: - strip LTO symbols from distributed .a/.o files. e.g. https://bugzilla.redhat.com/attachment.cgi?id=1650804&action=diff - provide helper script to adjust old autoconf files for LTO support(?) e.g. https://bugzilla.redhat.com/attachment.cgi?id=1650843&action=diff Not sure this should be part of makepkg or distribution tools. doc/PKGBUILD.5.asciidoc | 4 +++ doc/makepkg.conf.5.asciidoc | 4 +++ etc/makepkg.conf.in | 5 ++-- scripts/libmakepkg/buildenv/lto.sh.in | 37 +++++++++++++++++++++++++++ 4 files changed, 48 insertions(+), 2 deletions(-) create mode 100644 scripts/libmakepkg/buildenv/lto.sh.in diff --git a/doc/PKGBUILD.5.asciidoc b/doc/PKGBUILD.5.asciidoc index 1db0b6c4..af615040 100644 --- a/doc/PKGBUILD.5.asciidoc +++ b/doc/PKGBUILD.5.asciidoc @@ -325,6 +325,10 @@ underscore and the architecture name e.g., 'replaces_x86_64=()'. When used in combination with the `strip' option, a separate package containing the debug symbols is created. + *lto*;; + Enable building packages using link time optimization. Adds '-flto' + to both CFLAGS and CXXFLAGS. + Packaging Functions ------------------- diff --git a/doc/makepkg.conf.5.asciidoc b/doc/makepkg.conf.5.asciidoc index 04cc5ea9..2c7a54db 100644 --- a/doc/makepkg.conf.5.asciidoc +++ b/doc/makepkg.conf.5.asciidoc @@ -189,6 +189,10 @@ Options DEBUG_CXXFLAGS to their counterpart buildflags. Creates a separate package containing the debug symbols when used with `strip'. + *lto*;; + Enable building packages using link time optimization. Adds '-flto' + to both CFLAGS and CXXFLAGS. + **INTEGRITY_CHECK=(**check1 ...**)**:: File integrity checks to use. Multiple checks may be specified; this affects both generation and checking. The current valid options are: diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in index b13b1d5d..43a69df6 100644 --- a/etc/makepkg.conf.in +++ b/etc/makepkg.conf.in @@ -76,7 +76,7 @@ BUILDENV=(!distcc color !ccache check !sign) # These are default values for the options=() settings ######################################################################### # -# Makepkg defaults: OPTIONS=(!strip docs libtool staticlibs emptydirs !zipman !purge !debug) +# Makepkg defaults: OPTIONS=(!strip docs libtool staticlibs emptydirs !zipman !purge !debug !lto) # A negated option will do the opposite of the comments below. # #-- strip: Strip symbols from binaries/libraries @@ -87,8 +87,9 @@ BUILDENV=(!distcc color !ccache check !sign) #-- zipman: Compress manual (man and info) pages in MAN_DIRS with gzip #-- purge: Remove files specified by PURGE_TARGETS #-- debug: Add debugging flags as specified in DEBUG_* variables +#-- lto: Add compile flags for building with link time optimization # -OPTIONS=(strip docs libtool staticlibs emptydirs zipman purge !debug) +OPTIONS=(strip docs libtool staticlibs emptydirs zipman purge !debug !lto) #-- File integrity checks to use. Valid: ck, md5, sha1, sha224, sha256, sha384, sha512, b2 INTEGRITY_CHECK=(ck) diff --git a/scripts/libmakepkg/buildenv/lto.sh.in b/scripts/libmakepkg/buildenv/lto.sh.in new file mode 100644 index 00000000..8e7be66a --- /dev/null +++ b/scripts/libmakepkg/buildenv/lto.sh.in @@ -0,0 +1,37 @@ +#!/usr/bin/bash +# +# lto.sh - Specify flags for building a package with link-time +# optimisation +# +# Copyright (c) 2021 Pacman Development Team <pacman-dev(a)archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_BUILDENV_LTO_SH" ]] && return +LIBMAKEPKG_BUILDENV_LTO_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + +source "$LIBRARY/util/option.sh" + +build_options+=('lto') +buildenv_functions+=('buildenv_lto') + +buildenv_lto() { + if check_option "lto" "y"; then + CFLAGS+=" -flto" + CXXFLAGS+=" -flto" + fi +} -- 2.30.0

1 2

[pacman-dev] [PATCH] fix syntax error when running pacman-key --help
by Eli Schwartz 03 Mar '21

03 Mar '21

In commit 0f75ab3224141a5e8a4fe72b48b4639c83c6316b some unbalanced quotes were added by the committer while editing an error message. Fixes FS#69865 Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- scripts/pacman-key.sh.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/pacman-key.sh.in b/scripts/pacman-key.sh.in index 35a3b4daa..50342649c 100644 --- a/scripts/pacman-key.sh.in +++ b/scripts/pacman-key.sh.in @@ -564,7 +564,7 @@ refresh_keys() { # if no key was found, fall back to using the keyservers (with the key fingerprint instead) if (( $? )) && ! "${GPG_PACMAN[@]}" --refresh-keys "$id"; then - error "$(gettext "Could not update key: %s") "$id" + error "$(gettext "Could not update key: %s")" "$id" ret=1 fi done -- 2.30.1

2 1