April 2020 - Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH] makepkg: Don't double-layer distcc on ccache
by Matti Niemenmaa 08 Feb '21

08 Feb '21

From: Matti Niemenmaa <matti.niemenmaa+git(a)iki.fi> buildenv is set once for build() and a second time for package(). When using both distcc and ccache, this lead to CCACHE_PREFIX="distcc distcc" in package(), which breaks PKGBUILDs that execute the compiler in package() because distcc complains: distcc[383041] (main) CRITICAL! distcc seems to have invoked itself recursively! Avoid causing this error by only adding "distcc" to CCACHE_PREFIX if it's not yet there. Signed-off-by: Matti Niemenmaa <matti.niemenmaa+git(a)iki.fi> --- scripts/libmakepkg/buildenv/compiler.sh.in | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) ncurses is an example of a package that runs into this problem. Setting buildenv twice was done on purpose in commit 02a0bf550a22e199f48537b7eee87361b112e8a0, but at a glance I'm not sure whether it provides any benefit. All the meaningful changes are to environment variables that are exported from the parent process anyway, so all the changes are either repeated or even duplicated (like in PATH, or this CCACHE_PREFIX issue). Maybe it would be better to simply remove the prepare_buildenv call from the INFAKEROOT case in makepkg.sh.in? diff --git a/scripts/libmakepkg/buildenv/compiler.sh.in b/scripts/libmakepkg/buildenv/compiler.sh.in index 69f58a29..c93c77b4 100644 --- a/scripts/libmakepkg/buildenv/compiler.sh.in +++ b/scripts/libmakepkg/buildenv/compiler.sh.in @@ -44,7 +44,9 @@ buildenv_ccache() { buildenv_distcc() { if check_buildoption "distcc" "y"; then if (( using_ccache )); then - export CCACHE_PREFIX="${CCACHE_PREFIX:+$CCACHE_PREFIX }distcc" + if ! [[ "$CCACHE_PREFIX" =~ (^| )distcc($| ) ]]; then + export CCACHE_PREFIX="${CCACHE_PREFIX:+$CCACHE_PREFIX }distcc" + fi export CCACHE_BASEDIR="$srcdir" elif [[ -d /usr/lib/distcc/bin ]]; then export PATH="/usr/lib/distcc/bin:$PATH" -- 2.24.1

3 6

[pacman-dev] Alternatives system
by Allan McRae 06 Dec '20

06 Dec '20

Hi all, I plan to finish implementing an alternatives system for pacman post 5.2 release: https://wiki.archlinux.org/index.php/User:Allan/Alternatives Any comments or suggestions? Thanks, Allan

6 7

[pacman-dev] [PATCH] Implement multibar UI
by Anatol Pomozov 07 May '20

07 May '20

Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down. `active_dls` variable represents the list of active progress bars. `struct dl_progress_bar` is a data structure for a progress bar. In some cases we want to keep progress bars in order (e.g. db downloads). In some other cases (package downloads) we want to move completed items to the top of the screen. Global variable `multibar_move_complete_top` allows to configure such behavior. Signature file downloads are not shown at the UI. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/dload.c | 31 +--- lib/libalpm/dload.h | 1 - src/pacman/callback.c | 393 +++++++++++++++++++++++++++++------------- src/pacman/callback.h | 3 + src/pacman/sync.c | 2 + 5 files changed, 285 insertions(+), 145 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index c09abde4..cf467ff0 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -100,6 +100,11 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, return 1; } + if(dlnow < 0 || dltotal <= 0 || dlnow > dltotal) { + /* bogus values : stop here */ + return 0; + } + current_size = payload->initial_size + dlnow; /* is our filesize still under any set limit? */ @@ -115,34 +120,15 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, total_size = payload->initial_size + dltotal; - if(dltotal == 0 || payload->prevprogress == total_size) { + if(payload->prevprogress == total_size) { return 0; } - /* initialize the progress bar here to avoid displaying it when - * a repo is up to date and nothing gets downloaded. - * payload->handle->dlcb will receive the remote_name - * and the following arguments: - * 0, -1: download initialized - * 0, 0: non-download event - * x {x>0}, x: download complete - * x {x>0, x<y}, y {y > 0}: download progress, expected total is known */ - if(!payload->cb_initialized) { - cb_data.downloaded = 0; - cb_data.total = -1; - payload->cb_initialized = 1; - } - if(payload->prevprogress == current_size) { - cb_data.downloaded = 0; - cb_data.total = 0; - } else { /* do NOT include initial_size since it wasn't part of the package's * download_size (nor included in the total download size callback) */ - cb_data.downloaded = dlnow; - cb_data.total = dltotal; - } + cb_data.total = dltotal; + cb_data.downloaded = dlnow; payload->handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_PROGRESS, &cb_data); - payload->prevprogress = current_size; return 0; @@ -1160,5 +1146,4 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; - payload->cb_initialized = 0; } diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index a40b51b7..142568e8 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -42,7 +42,6 @@ struct dload_payload { int errors_ok; int unlink_on_fail; int trust_remote_name; - int cb_initialized; #ifdef HAVE_LIBCURL CURL *curl; char error_buffer[CURL_ERROR_SIZE]; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 387319b6..6133a036 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -18,6 +18,8 @@ * along with this program. If not, see <http://www.gnu.org/licenses/>. */ +#include <assert.h> +#include <stdbool.h> #include <stdio.h> #include <stdlib.h> #include <string.h> @@ -45,6 +47,8 @@ static off_t list_total = 0.0; static int on_progress = 0; static alpm_list_t *output = NULL; +static void cursor_goto_end(void); + /* update speed for the fill_progress based functions */ #define UPDATE_SPEED_MS 200 @@ -152,11 +156,7 @@ static void fill_progress(const int bar_percent, const int disp_percent, printf(" %3d%%", disp_percent); } - if(bar_percent == 100) { - putchar('\n'); - } else { - putchar('\r'); - } + putchar('\r'); fflush(stdout); } @@ -346,6 +346,7 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_DB_RETRIEVE_FAILED: case ALPM_EVENT_RETRIEVE_DONE: case ALPM_EVENT_RETRIEVE_FAILED: + cursor_goto_end(); flush_output_list(); on_progress = 0; break; @@ -629,6 +630,7 @@ void cb_progress(alpm_progress_t event, const char *pkgname, int percent, fill_progress(percent, percent, cols - infolen); if(percent == 100) { + putchar('\n'); flush_output_list(); on_progress = 0; } else { @@ -647,124 +649,118 @@ void cb_dl_total(off_t total) } } -/* callback to handle display of download progress */ -static void dload_progress_event(const char *filename, off_t file_xfered, off_t file_total) +static int dload_progressbar_enabled(void) { - static double rate_last; - static off_t xfered_last; - static int64_t initial_time = 0; - int infolen; - int filenamelen; - char *fname, *p; - /* used for wide character width determination and printing */ - int len, wclen, wcwid, padwid; - wchar_t *wcfname; + return !config->noprogressbar && getcols() != 0; +} - int totaldownload = 0; - off_t xfered, total; - double rate = 0.0; - unsigned int eta_h = 0, eta_m = 0, eta_s = 0; - double rate_human, xfered_human; - const char *rate_label, *xfered_label; - int file_percent = 0, total_percent = 0; +struct dl_progress_bar { + const char *filename; + off_t xfered; + off_t total_size; + uint64_t init_time; /* Time when this download started doing any progress */ + uint64_t sync_time; /* Last time we updated the bar info */ + double rate; + unsigned int eta; /* ETA in seconds */ + bool completed; /* transfer is completed */ +}; - const unsigned short cols = getcols(); +/* List of active downloads handled by multibar UI. + * Once the first download in the list is completed it is removed + * from this list and we never redraw it anymore. + * If the download in this list then UI can redraw the progress bar or change + * the order of the bars (e.g. completed bar is moved to the top of the list) + */ +alpm_list_t *active_dls = NULL; - /* Nothing has changed since last callback; stop here */ - if(file_xfered == 0 && file_total == 0) { - return; - } +/* Number of active progress bars that multibar UI handles. + * Each progress bar has a corresponding active download. + * Note that some downloads might not have a bar, e.g. optional signature files + * do not have a corresponding bar. + */ +size_t active_bars_no = 0; - if(config->noprogressbar || cols == 0) { - if(file_xfered == 0 && file_total == -1) { - printf(_("downloading %s...\n"), filename); - fflush(stdout); - } - return; - } +int multibar_move_complete_top = 0; - infolen = cols * 6 / 10; - if(infolen < 50) { - infolen = 50; - } - /* only use TotalDownload if enabled and we have a callback value */ - if(config->totaldownload && list_total) { - /* sanity check */ - if(list_xfered + file_total <= list_total) { - totaldownload = 1; - } else { - /* bogus values : don't enable totaldownload and reset */ - list_xfered = 0; - list_total = 0; - } +/* Cursor position relative to the first active progress bar, + * e.g. 0 means the first active progress bar, active_bars_no-1 means the last bar, + * active_bars_no - is the line below all progress bars. + */ +int cursor_lineno = 0; + +/* Move console cursor `lines` up */ +static void cursor_line_up(unsigned int lines) +{ + assert(lines > 0); + printf("\x1B[%dF", lines); +} + +/* Move console cursor `lines` down */ +static void cursor_line_down(unsigned int lines) +{ + assert(lines > 0); + printf("\x1B[%dE", lines); +} + +/* Erase line from the cursor position till the end of the line */ +static void console_erase_line(void) +{ + printf("\x1B[K"); +} + +/* Goto the line that corresponds to num-th active download */ +static void cursor_goto_bar(int num) +{ + if(num > cursor_lineno) { + cursor_line_down(num - cursor_lineno); + } else if(num < cursor_lineno) { + cursor_line_up(cursor_lineno - num); } + cursor_lineno = num; +} - if(totaldownload) { - xfered = list_xfered + file_xfered; - total = list_total; - } else { - xfered = file_xfered; - total = file_total; - } - - /* this is basically a switch on xfered: 0, total, and - * anything else */ - if(file_xfered == 0 && file_total == -1) { - /* set default starting values, ensure we only call this once - * if TotalDownload is enabled */ - if(!totaldownload || (totaldownload && list_xfered == 0)) { - initial_time = get_time_ms(); - xfered_last = (off_t)0; - rate_last = 0.0; - get_update_timediff(1); - } - } else if(xfered > total || xfered < 0) { - /* bogus values : stop here */ - return; - } else if(file_xfered == file_total) { - /* compute final values */ - int64_t timediff = get_time_ms() - initial_time; - if(timediff > 0) { - rate = (double)xfered / (timediff / 1000.0); - /* round elapsed time (in ms) to the nearest second */ - eta_s = (unsigned int)(timediff + 500) / 1000; - } else { - eta_s = 0; - } - } else { - /* compute current average values */ - int64_t timediff = get_update_timediff(0); +/* Goto the line *after* the last active progress bar */ +static void cursor_goto_end(void) +{ + cursor_goto_bar(active_bars_no); +} - if(timediff < UPDATE_SPEED_MS) { - /* return if the calling interval was too short */ - return; - } - rate = (double)(xfered - xfered_last) / (timediff / 1000.0); - /* average rate to reduce jumpiness */ - rate = (rate + 2 * rate_last) / 3; - if(rate > 0.0) { - eta_s = (total - xfered) / rate; - } else { - eta_s = UINT_MAX; +/* Returns true if element with the specified name is found, false otherwise */ +static bool find_bar_for_filename(const char *filename, int *index, struct dl_progress_bar **bar) +{ + int i = 0; + alpm_list_t *listitem = active_dls; + for(; listitem; listitem = listitem->next, i++) { + struct dl_progress_bar *b = listitem->data; + if (b->filename == filename /* or strcmp() ? */) { + *index = i; + *bar = b; + return true; } - rate_last = rate; - xfered_last = xfered; } - if(file_total) { - file_percent = (file_xfered * 100) / file_total; - } else { - file_percent = 100; - } + return false; +} - if(totaldownload) { - total_percent = ((list_xfered + file_xfered) * 100) / - list_total; +static void draw_dl_progress_bar(struct dl_progress_bar *bar) +{ + int infolen; + int filenamelen; + char *fname, *p; + /* used for wide character width determination and printing */ + int len, wclen, wcwid, padwid; + wchar_t *wcfname; + unsigned int eta_h = 0, eta_m = 0, eta_s = bar->eta; + double rate_human, xfered_human; + const char *rate_label, *xfered_label; + int file_percent = 0; - /* if we are at the end, add the completed file to list_xfered */ - if(file_xfered == file_total) { - list_xfered += file_total; - } + const unsigned short cols = getcols(); + + if(bar->total_size) { + file_percent = (bar->xfered * 100) / bar->total_size; + } else { + file_percent = 100; } /* fix up time for display */ @@ -773,13 +769,13 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t eta_m = eta_s / 60; eta_s -= eta_m * 60; - len = strlen(filename); + len = strlen(bar->filename); fname = malloc(len + 1); - memcpy(fname, filename, len + 1); + memcpy(fname, bar->filename, len + 1); /* strip package or DB extension for cleaner look */ if((p = strstr(fname, ".pkg")) || (p = strstr(fname, ".db")) || (p = strstr(fname, ".files"))) { /* tack on a .sig suffix for signatures */ - if(memcmp(&filename[len - 4], ".sig", 4) == 0) { + if(memcmp(&bar->filename[len - 4], ".sig", 4) == 0) { memcpy(p, ".sig", 4); /* adjust length for later calculations */ @@ -790,6 +786,11 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t fname[len] = '\0'; } + infolen = cols * 6 / 10; + if(infolen < 50) { + infolen = 50; + } + /* 1 space + filenamelen + 1 space + 6 for size + 1 space + 3 for label + * + 2 spaces + 4 for rate + 1 space + 3 for label + 2 for /s + 1 space + * 8 for eta, gives us the magic 33 */ @@ -824,8 +825,8 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t } - rate_human = humanize_size((off_t)rate, '\0', -1, &rate_label); - xfered_human = humanize_size(xfered, '\0', -1, &xfered_label); + rate_human = humanize_size((off_t)bar->rate, '\0', -1, &rate_label); + xfered_human = humanize_size(bar->xfered, '\0', -1, &xfered_label); printf(" %ls%-*s ", wcfname, padwid, ""); /* We will show 1.62 MiB/s, 11.6 MiB/s, but 116 KiB/s and 1116 KiB/s */ @@ -850,19 +851,169 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t free(fname); free(wcfname); - if(totaldownload) { - fill_progress(file_percent, total_percent, cols - infolen); + fill_progress(file_percent, file_percent, cols - infolen); + return; +} + +static void dload_init_event(const char *filename, alpm_download_event_init_t *data) +{ + (void)data; + + if(!dload_progressbar_enabled()) { + printf(_(" %s downloading...\n"), filename); + return; + } + + struct dl_progress_bar *bar = calloc(1, sizeof(struct dl_progress_bar)); + assert(bar); + bar->filename = filename; + bar->init_time = get_time_ms(); + bar->rate = 0.0; + active_dls = alpm_list_add(active_dls, bar); + + cursor_goto_end(); + printf(_(" %s downloading...\n"), filename); + cursor_lineno++; + active_bars_no++; +} + +/* Draws download progress */ +static void dload_progress_event(const char *filename, alpm_download_event_progress_t *data) +{ + int index; + struct dl_progress_bar *bar; + int64_t curr_time = get_time_ms(); + double last_chunk_rate; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + + /* compute current average values */ + timediff = curr_time - bar->sync_time; + + if(timediff < UPDATE_SPEED_MS) { + /* return if the calling interval was too short */ + return; + } + bar->sync_time = curr_time; + + last_chunk_rate = (double)(data->downloaded - bar->xfered) / (timediff / 1000.0); + /* average rate to reduce jumpiness */ + bar->rate = (last_chunk_rate + 2 * bar->rate) / 3; + if(bar->rate > 0.0) { + bar->eta = (data->total - data->downloaded) / bar->rate; } else { - fill_progress(file_percent, file_percent, cols - infolen); + bar->eta = UINT_MAX; } - return; + + /* Total size is received after the download starts. */ + bar->total_size = data->total; + bar->xfered = data->downloaded; + + cursor_goto_bar(index); + draw_dl_progress_bar(bar); + fflush(stdout); } +/* download completed */ +static void dload_complete_event(const char *filename, alpm_download_event_completed_t *data) +{ + int index; + struct dl_progress_bar *bar; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + bar->completed = true; + + /* This may not have been initialized if the download finished before + * an alpm_download_event_progress_t event happened */ + bar->total_size = data->total; + + if(data->result == 1) { + cursor_goto_bar(index); + printf(_(" %s is up to date"), bar->filename); + /* The line contains text from previous status. Erase these leftovers. */ + console_erase_line(); + } else if(data->result == 0) { + /* compute final values */ + bar->xfered = bar->total_size; + timediff = get_time_ms() - bar->init_time; + + /* if transfer was too fast, treat it as a 1ms transfer, for the sake + * of the rate calculation */ + if(timediff < 1) + timediff = 1; + + bar->rate = (double)bar->xfered / (timediff / 1000.0); + /* round elapsed time (in ms) to the nearest second */ + bar->eta = (unsigned int)(timediff + 500) / 1000; + + if(multibar_move_complete_top && index != 0) { + /* If this item completed then move it to the top. + * Swap 0-th bar data with `index`-th one + */ + struct dl_progress_bar *former_topbar = active_dls->data; + alpm_list_t *baritem = alpm_list_nth(active_dls, index); + active_dls->data = bar; + baritem->data = former_topbar; + + cursor_goto_bar(index); + draw_dl_progress_bar(former_topbar); + + index = 0; + } + + cursor_goto_bar(index); + draw_dl_progress_bar(bar); + } else { + cursor_goto_bar(index); + printf(_(" %s failed to download"), bar->filename); + console_erase_line(); + } + fflush(stdout); + + /* If the first bar is completed then there is no reason to keep it + * in the list as we are not going to redraw it anymore. + */ + while(active_dls) { + alpm_list_t *head = active_dls; + struct dl_progress_bar *j = head->data; + if(j->completed) { + cursor_lineno--; + active_bars_no--; + active_dls = alpm_list_remove_item(active_dls, head); + free(head); + free(j); + } else { + break; + } + } +} + +/* Callback to handle display of download progress */ void cb_download(const char *filename, alpm_download_event_type_t event, void *data) { - if(event == ALPM_DOWNLOAD_PROGRESS) { - alpm_download_event_progress_t *progress = data; - dload_progress_event(filename, progress->downloaded, progress->total); + if(str_endswith(filename, ".sig")) { + return; + } + + if(event == ALPM_DOWNLOAD_INIT) { + dload_init_event(filename, data); + } else if(event == ALPM_DOWNLOAD_PROGRESS) { + dload_progress_event(filename, data); + } else if(event == ALPM_DOWNLOAD_COMPLETED) { + dload_complete_event(filename, data); + } else { + pm_printf(ALPM_LOG_ERROR, _("unknown callback event type %d for %s\n"), + event, filename); } } diff --git a/src/pacman/callback.h b/src/pacman/callback.h index 6d92e86b..916b9dad 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -44,4 +44,7 @@ void cb_download(const char *filename, alpm_download_event_type_t event, __attribute__((format(printf, 2, 0))) void cb_log(alpm_loglevel_t level, const char *fmt, va_list args); +/* specify if multibar should move complete bars to the top of the screen */ +int multibar_move_complete_top; + #endif /* PM_CALLBACK_H */ diff --git a/src/pacman/sync.c b/src/pacman/sync.c index f7dcb958..d5886f13 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -35,6 +35,7 @@ #include "pacman.h" #include "util.h" #include "package.h" +#include "callback.h" #include "conf.h" static int unlink_verbose(const char *pathname, int ignore_missing) @@ -824,6 +825,7 @@ int sync_prepare_execute(void) goto cleanup; } + multibar_move_complete_top = 1; if(alpm_trans_commit(config->handle, &data) == -1) { alpm_errno_t err = alpm_errno(config->handle); pm_printf(ALPM_LOG_ERROR, _("failed to commit transaction (%s)\n"), -- 2.26.2

2 5

[pacman-dev] [PATCH] Remove alpm_db_update() function
by Anatol Pomozov 07 May '20

07 May '20

This function got replaced by its multiplexed couterpart alpm_dbs_update(). If one needs to reproduce alpm_db_update() functionality then create a single item list and pass it to alpm_dbs_update(). Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/alpm.h | 36 --------- lib/libalpm/be_sync.c | 165 ------------------------------------------ 2 files changed, 201 deletions(-) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 8fe11a9b..70ed459c 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -1048,42 +1048,6 @@ int alpm_db_add_server(alpm_db_t *db, const char *url); int alpm_db_remove_server(alpm_db_t *db, const char *url); /** @} */ -/** Update a package database - * - * An update of the package database \a db will be attempted. Unless - * \a force is true, the update will only be performed if the remote - * database was modified since the last update. - * - * This operation requires a database lock, and will return an applicable error - * if the lock could not be obtained. - * - * Example: - * @code - * alpm_list_t *syncs = alpm_get_syncdbs(); - * for(i = syncs; i; i = alpm_list_next(i)) { - * alpm_db_t *db = alpm_list_getdata(i); - * result = alpm_db_update(0, db); - * - * if(result < 0) { - * printf("Unable to update database: %s\n", alpm_strerrorlast()); - * } else if(result == 1) { - * printf("Database already up to date\n"); - * } else { - * printf("Database updated\n"); - * } - * } - * @endcode - * - * @note After a successful update, the \link alpm_db_get_pkgcache() - * package cache \endlink will be invalidated - * @param force if true, then forces the update, otherwise update only in case - * the database isn't up to date - * @param db pointer to the package database to update - * @return 0 on success, -1 on error (pm_errno is set accordingly), 1 if up to - * to date - */ -int alpm_db_update(int force, alpm_db_t *db); - /** Update package databases * * An update of the package databases in the list \a dbs will be attempted. diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index 826c11e6..73915321 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -136,171 +136,6 @@ valid: return 0; } -int SYMEXPORT alpm_db_update(int force, alpm_db_t *db) -{ - char *syncpath; - const char *dbext; - alpm_list_t *i; - int updated = 0; - int ret = -1; - mode_t oldmask; - alpm_handle_t *handle; - int siglevel; - - /* Sanity checks */ - ASSERT(db != NULL, return -1); - handle = db->handle; - handle->pm_errno = ALPM_ERR_OK; - ASSERT(db != handle->db_local, RET_ERR(handle, ALPM_ERR_WRONG_ARGS, -1)); - ASSERT(db->servers != NULL, RET_ERR(handle, ALPM_ERR_SERVER_NONE, -1)); - - if(!(db->usage & ALPM_DB_USAGE_SYNC)) { - return 0; - } - - syncpath = get_sync_dir(handle); - if(!syncpath) { - return -1; - } - - /* force update of invalid databases to fix potential mismatched database/signature */ - if(db->status & DB_STATUS_INVALID) { - force = 1; - } - - /* make sure we have a sane umask */ - oldmask = umask(0022); - - siglevel = alpm_db_get_siglevel(db); - - /* attempt to grab a lock */ - if(_alpm_handle_lock(handle)) { - free(syncpath); - umask(oldmask); - RET_ERR(handle, ALPM_ERR_HANDLE_LOCK, -1); - } - - dbext = db->handle->dbext; - - for(i = db->servers; i; i = i->next) { - const char *server = i->data, *final_db_url = NULL; - struct dload_payload payload = {0}; - size_t len; - int sig_ret = 0; - - /* set hard upper limit of 128MiB */ - payload.max_size = 128 * 1024 * 1024; - - /* print server + filename into a buffer */ - len = strlen(server) + strlen(db->treename) + strlen(dbext) + 2; - MALLOC(payload.fileurl, len, - { - free(syncpath); - umask(oldmask); - RET_ERR(handle, ALPM_ERR_MEMORY, -1); - } - ); - snprintf(payload.fileurl, len, "%s/%s%s", server, db->treename, dbext); - payload.handle = handle; - payload.force = force; - payload.unlink_on_fail = 1; - - ret = _alpm_download(&payload, syncpath, NULL, &final_db_url); - _alpm_dload_payload_reset(&payload); - updated = (updated || ret == 0); - - if(ret != -1 && updated && (siglevel & ALPM_SIG_DATABASE)) { - /* an existing sig file is no good at this point */ - char *sigpath = _alpm_sigpath(handle, _alpm_db_path(db)); - if(!sigpath) { - ret = -1; - break; - } - unlink(sigpath); - free(sigpath); - - - /* check if the final URL from internal downloader looks reasonable */ - if(final_db_url != NULL) { - if(strlen(final_db_url) < 3 - || strcmp(final_db_url + strlen(final_db_url) - strlen(dbext), - dbext) != 0) { - final_db_url = NULL; - } - } - - /* if we downloaded a DB, we want the .sig from the same server */ - if(final_db_url != NULL) { - /* print final_db_url into a buffer (leave space for .sig) */ - len = strlen(final_db_url) + 5; - } else { - /* print server + filename into a buffer (leave space for separator and .sig) */ - len = strlen(server) + strlen(db->treename) + strlen(dbext) + 6; - } - - MALLOC(payload.fileurl, len, - { - free(syncpath); - umask(oldmask); - RET_ERR(handle, ALPM_ERR_MEMORY, -1); - } - ); - - if(final_db_url != NULL) { - snprintf(payload.fileurl, len, "%s.sig", final_db_url); - } else { - snprintf(payload.fileurl, len, "%s/%s%s.sig", server, db->treename, dbext); - } - - payload.handle = handle; - payload.force = 1; - payload.errors_ok = (siglevel & ALPM_SIG_DATABASE_OPTIONAL); - - /* set hard upper limit of 16KiB */ - payload.max_size = 16 * 1024; - - sig_ret = _alpm_download(&payload, syncpath, NULL, NULL); - /* errors_ok suppresses error messages, but not the return code */ - sig_ret = payload.errors_ok ? 0 : sig_ret; - _alpm_dload_payload_reset(&payload); - } - - if(ret != -1 && sig_ret != -1) { - break; - } - } - - if(updated) { - /* Cache needs to be rebuilt */ - _alpm_db_free_pkgcache(db); - - /* clear all status flags regarding validity/existence */ - db->status &= ~DB_STATUS_VALID; - db->status &= ~DB_STATUS_INVALID; - db->status &= ~DB_STATUS_EXISTS; - db->status &= ~DB_STATUS_MISSING; - - /* if the download failed skip validation to preserve the download error */ - if(ret != -1 && sync_db_validate(db) != 0) { - /* pm_errno should be set */ - ret = -1; - } - } - - if(ret == -1) { - /* pm_errno was set by the download code */ - _alpm_log(handle, ALPM_LOG_DEBUG, "failed to sync db: %s\n", - alpm_strerror(handle->pm_errno)); - } else { - handle->pm_errno = ALPM_ERR_OK; - } - - _alpm_handle_unlock(handle); - free(syncpath); - umask(oldmask); - return ret; -} - int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force) { char *syncpath; const char *dbext = handle->dbext; -- 2.26.1

2 2

[pacman-dev] [PATCH] Convert download packages logic to multiplexed API
by Anatol Pomozov 07 May '20

07 May '20

Create a list of dload_payloads and pass it to the new _alpm_multi_* interface. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/sync.c | 74 ++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 46 deletions(-) diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 8a9dcae8..aa417bca 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -726,47 +726,13 @@ static int find_dl_candidates(alpm_handle_t *handle, alpm_list_t **files) return 0; } -static int download_single_file(alpm_handle_t *handle, struct dload_payload *payload, - const char *cachedir) -{ - alpm_event_pkgdownload_t event = { - .type = ALPM_EVENT_PKGDOWNLOAD_START, - .file = payload->remote_name - }; - const alpm_list_t *server; - - payload->handle = handle; - payload->allow_resume = 1; - - EVENT(handle, &event); - for(server = payload->servers; server; server = server->next) { - const char *server_url = server->data; - size_t len; - - /* print server + filename into a buffer */ - len = strlen(server_url) + strlen(payload->remote_name) + 2; - MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); - snprintf(payload->fileurl, len, "%s/%s", server_url, payload->remote_name); - - if(_alpm_download(payload, cachedir, NULL, NULL) != -1) { - event.type = ALPM_EVENT_PKGDOWNLOAD_DONE; - EVENT(handle, &event); - return 0; - } - _alpm_dload_payload_reset_for_retry(payload); - } - - event.type = ALPM_EVENT_PKGDOWNLOAD_FAILED; - EVENT(handle, &event); - return -1; -} - static int download_files(alpm_handle_t *handle) { const char *cachedir; alpm_list_t *i, *files = NULL; int errors = 0; alpm_event_t event; + alpm_list_t *payloads = NULL; cachedir = _alpm_filecache_setup(handle); handle->trans->state = STATE_DOWNLOADING; @@ -814,26 +780,42 @@ static int download_files(alpm_handle_t *handle) event.type = ALPM_EVENT_RETRIEVE_START; EVENT(handle, &event); - event.type = ALPM_EVENT_RETRIEVE_DONE; for(i = files; i; i = i->next) { const alpm_pkg_t *pkg = i->data; - struct dload_payload payload = {0}; + struct dload_payload *payload = NULL; - STRDUP(payload.remote_name, pkg->filename, GOTO_ERR(handle, ALPM_ERR_MEMORY, finish)); - payload.servers = pkg->origin_data.db->servers; - payload.max_size = pkg->size; + CALLOC(payload, 1, sizeof(*payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, finish)); + STRDUP(payload->remote_name, pkg->filename, GOTO_ERR(handle, ALPM_ERR_MEMORY, finish)); + STRDUP(payload->filepath, pkg->filename, GOTO_ERR(handle, ALPM_ERR_MEMORY, finish)); + payload->max_size = pkg->size; + payload->servers = pkg->origin_data.db->servers; + payload->handle = handle; + payload->allow_resume = 1; - if(download_single_file(handle, &payload, cachedir) == -1) { - errors++; - event.type = ALPM_EVENT_RETRIEVE_FAILED; - _alpm_log(handle, ALPM_LOG_WARNING, _("failed to retrieve some files\n")); - } - _alpm_dload_payload_reset(&payload); + payloads = alpm_list_add(payloads, payload); + + /* TOASK: do we need to initiate *.sig file download here? */ + } + /* TOASK: the sequential download logic reported ALPM_EVENT_PKGDOWNLOAD_START/DONE/FAILED event. + * Do we want to report the event as well? If yes then it should be moved inside alpm_multi_*() + * function but at that point ALPM operates with dload_payload() and does not care if it downloads + * package or database file. + */ + event.type = ALPM_EVENT_RETRIEVE_DONE; + if(_alpm_multi_download(handle, payloads, cachedir) == -1) { + errors++; + event.type = ALPM_EVENT_RETRIEVE_FAILED; + _alpm_log(handle, ALPM_LOG_WARNING, _("failed to retrieve some files\n")); } EVENT(handle, &event); } finish: + if(payloads) { + alpm_list_free_inner(payloads, (alpm_list_fn_free)_alpm_dload_payload_reset); + FREELIST(payloads); + } + if(files) { alpm_list_free(files); } -- 2.26.1

2 5

[pacman-dev] [PATCH] Introduce event types for start/end database list download
by Anatol Pomozov 06 May '20

06 May '20

Multiplexed database/files downloads will use multiple progress bars. The UI logic is quite complicated and printing error messages while handling multiple progress bars is going to be challenging. Instead we are going to save all ALPM error messages to a list and flush it at the end of the download process. Use on_progress variable that blocks error messages printing. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/alpm.h | 6 ++++++ lib/libalpm/be_sync.c | 7 +++++++ src/pacman/callback.c | 13 +++++++++++-- 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 2cf20343..aee5aa0d 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -403,6 +403,12 @@ typedef enum _alpm_event_type_t { * arguments. */ ALPM_EVENT_SCRIPTLET_INFO, /** Files will be downloaded from a repository. */ + ALPM_EVENT_DB_RETRIEVE_START, + /** Files were downloaded from a repository. */ + ALPM_EVENT_DB_RETRIEVE_DONE, + /** Not all files were successfully downloaded from a repository. */ + ALPM_EVENT_DB_RETRIEVE_FAILED, + /** Files will be downloaded from a repository. */ ALPM_EVENT_RETRIEVE_START, /** Files were downloaded from a repository. */ ALPM_EVENT_RETRIEVE_DONE, diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index 1be40650..826c11e6 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -308,6 +308,7 @@ int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force int ret = -1; mode_t oldmask; alpm_list_t *payloads = NULL; + alpm_event_t event; /* Sanity checks */ CHECK_HANDLE(handle, return -1); @@ -382,10 +383,16 @@ int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force } } + event.type = ALPM_EVENT_DB_RETRIEVE_START; + EVENT(handle, &event); ret = _alpm_multi_download(handle, payloads, syncpath); if(ret < 0) { + event.type = ALPM_EVENT_DB_RETRIEVE_FAILED; + EVENT(handle, &event); goto cleanup; } + event.type = ALPM_EVENT_DB_RETRIEVE_DONE; + EVENT(handle, &event); for(i = dbs; i; i = i->next) { alpm_db_t *db = i->data; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 8fb89b39..585eae09 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -280,8 +280,12 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_SCRIPTLET_INFO: fputs(event->scriptlet_info.line, stdout); break; + case ALPM_EVENT_DB_RETRIEVE_START: + on_progress = 1; + break; case ALPM_EVENT_RETRIEVE_START: colon_printf(_("Retrieving packages...\n")); + on_progress = 1; break; case ALPM_EVENT_DISKSPACE_START: if(config->noprogressbar) { @@ -338,6 +342,13 @@ void cb_event(alpm_event_t *event) } } break; + case ALPM_EVENT_DB_RETRIEVE_DONE: + case ALPM_EVENT_DB_RETRIEVE_FAILED: + case ALPM_EVENT_RETRIEVE_DONE: + case ALPM_EVENT_RETRIEVE_FAILED: + flush_output_list(); + on_progress = 0; + break; /* all the simple done events, with fallthrough for each */ case ALPM_EVENT_FILECONFLICTS_DONE: case ALPM_EVENT_CHECKDEPS_DONE: @@ -349,8 +360,6 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_KEY_DOWNLOAD_DONE: case ALPM_EVENT_LOAD_DONE: case ALPM_EVENT_DISKSPACE_DONE: - case ALPM_EVENT_RETRIEVE_DONE: - case ALPM_EVENT_RETRIEVE_FAILED: case ALPM_EVENT_HOOK_DONE: case ALPM_EVENT_HOOK_RUN_DONE: /* we can safely ignore those as well */ -- 2.26.1

2 3

[pacman-dev] [PATCH] Add a helper macro to free() dload_payload structure
by Anatol Pomozov 05 May '20

05 May '20

It frees all the dynamically allocated fields plus the struct itself Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/dload.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index a40b51b7..3f2fb9ea 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -56,6 +56,11 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload); int _alpm_download(struct dload_payload *payload, const char *localpath, char **final_file, const char **final_url); +#define DLOAD_PAYLOAD_FREE(payload) { \ + _alpm_dload_payload_reset(payload); \ + FREE(payload); \ +} + int _alpm_multi_download(alpm_handle_t *handle, alpm_list_t *payloads /* struct dload_payload */, const char *localpath); -- 2.26.2

2 6

[pacman-dev] [PATCH] Add str_endswith() utility function
by Anatol Pomozov 05 May '20

05 May '20

It provides an easy way to check whether a string ends with a given suffix. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- src/common/util-common.c | 18 ++++++++++++++++++ src/common/util-common.h | 2 ++ 2 files changed, 20 insertions(+) diff --git a/src/common/util-common.c b/src/common/util-common.c index 7d43ac0d..11ffc705 100644 --- a/src/common/util-common.c +++ b/src/common/util-common.c @@ -198,6 +198,24 @@ char *strndup(const char *s, size_t n) } #endif +/** Helper function for check whether a string ends with a given suffix + * @param str a string + * @param suffix we want to check + * @return 0 in case if the given string does not end with the suffix, + * a non-zero value otherwise. + */ +int str_endswith(const char *str, const char *suffix) +{ + int str_len = strlen(str); + int suffix_len = strlen(suffix); + + if(suffix_len > str_len) { + return 0; + } + + return !strcmp(str + str_len - suffix_len, suffix); +} + void wordsplit_free(char **ws) { if(ws) { diff --git a/src/common/util-common.h b/src/common/util-common.h index 483d5da4..6b66be3e 100644 --- a/src/common/util-common.h +++ b/src/common/util-common.h @@ -35,6 +35,8 @@ char **wordsplit(const char *str); size_t strtrim(char *str); +int str_endswith(const char *str, const char *suffix); + #ifndef HAVE_STRNDUP char *strndup(const char *s, size_t n); #endif -- 2.26.2

2 2

[pacman-dev] [PATCH] Adds a "retry?" prompt for when transaction creation fails for user-friendlier fail support.
by Pedro 29 Apr '20

29 Apr '20

EDIT: Rewrote the prompt message and added a check for the --noconfirm flag >From what I could see, the function trans_init that comes right before the prompt has already specific messages for each operation, so I consider that this new implementation fits the purpose. My motivation to submit this patch is that it's pretty common for AUR-helpers to automatically invoke pacman after a build, but correct transaction error handling is quite rare. So I thought that to implement a handling mechanism in the user side would solve the problem not only in that case, but also for other scripts that invoke pacman. Signed-off-by: Pedro <pedrocga1(a)gmail.com> --- src/pacman/remove.c | 13 ++++++++++--- src/pacman/sync.c | 13 ++++++++++--- src/pacman/upgrade.c | 13 ++++++++++--- 3 files changed, 30 insertions(+), 9 deletions(-) diff --git a/src/pacman/remove.c b/src/pacman/remove.c index 9920f502..66bb100d 100644 --- a/src/pacman/remove.c +++ b/src/pacman/remove.c @@ -87,9 +87,16 @@ int pacman_remove(alpm_list_t *targets) } /* Step 0: create a new transaction */ - if(trans_init(config->flags, 0) == -1) { - return 1; - } + while(1) { + if(trans_init(config->flags, 1) == -1) { + if(config->noconfirm || yesno(_("Retry the operation with the same flags?")) == 0) { + return 1; + } + /* Try again */ + continue; + } + break; + } /* Step 1: add targets to the created transaction */ for(i = targets; i; i = alpm_list_next(i)) { diff --git a/src/pacman/sync.c b/src/pacman/sync.c index f7dcb958..e472d89a 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -694,9 +694,16 @@ static int sync_trans(alpm_list_t *targets) alpm_list_t *i; /* Step 1: create a new transaction... */ - if(trans_init(config->flags, 1) == -1) { - return 1; - } + while(1) { + if(trans_init(config->flags, 1) == -1) { + if(config->noconfirm == 1 || yesno(_("Retry the operation with the same flags?")) == 0) { + return 1; + } + /* Try again */ + continue; + } + break; + } /* process targets */ for(i = targets; i; i = alpm_list_next(i)) { diff --git a/src/pacman/upgrade.c b/src/pacman/upgrade.c index 5f984e44..2ea74618 100644 --- a/src/pacman/upgrade.c +++ b/src/pacman/upgrade.c @@ -79,9 +79,16 @@ int pacman_upgrade(alpm_list_t *targets) } /* Step 1: create a new transaction */ - if(trans_init(config->flags, 1) == -1) { - retval = 1; - goto fail_free; + while(1) { + if(trans_init(config->flags, 1) == -1) { + if(config-->noconfirm || yesno(_("Retry the operation with the same flags?")) == 0) { + retval = 1; + goto fail_free; + } + /* Try again */ + continue; + } + break; } printf(_("loading packages...\n")); -- 2.26.2

5 4

[pacman-dev] [PATCH] Convert downloading databases to the new multiplexed API
by Anatol Pomozov 29 Apr '20

29 Apr '20

Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- src/pacman/util.c | 27 +++++++++------------------ 1 file changed, 9 insertions(+), 18 deletions(-) diff --git a/src/pacman/util.c b/src/pacman/util.c index 6693ec75..93487e73 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -150,26 +150,17 @@ int check_syncdbs(size_t need_repos, int check_valid) int sync_syncdbs(int level, alpm_list_t *syncs) { - alpm_list_t *i; - unsigned int success = 1; - - for(i = syncs; i; i = alpm_list_next(i)) { - alpm_db_t *db = i->data; - - int ret = alpm_db_update((level < 2 ? 0 : 1), db); - if(ret < 0) { - pm_printf(ALPM_LOG_ERROR, _("failed to update %s (%s)\n"), - alpm_db_get_name(db), alpm_strerror(alpm_errno(config->handle))); - success = 0; - } else if(ret == 1) { - printf(_(" %s is up to date\n"), alpm_db_get_name(db)); - } - } + int ret; + int force = (level < 2 ? 0 : 1); - if(!success) { - pm_printf(ALPM_LOG_ERROR, _("failed to synchronize all databases\n")); + multibar_move_complete_top = 0; + ret = alpm_dbs_update(config->handle, syncs, force); + if(ret < 0) { + pm_printf(ALPM_LOG_ERROR, _("failed to synchronize all databases (%s)\n"), + alpm_strerror(alpm_errno(config->handle))); } - return (success > 0); + + return (ret >= 0); } /* discard unhandled input on the terminal's input buffer */ -- 2.26.1

2 1