- Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH] Adds a "retry?" prompt for when transaction creation fails for user-friendlier fail support.
by Pedro 27 Apr '20

27 Apr '20

Signed-off-by: Pedro <pedrocga1(a)gmail.com> --- src/pacman/remove.c | 13 ++++++++++--- src/pacman/sync.c | 13 ++++++++++--- src/pacman/upgrade.c | 13 ++++++++++--- 3 files changed, 30 insertions(+), 9 deletions(-) diff --git a/src/pacman/remove.c b/src/pacman/remove.c index 9920f502..f61c8652 100644 --- a/src/pacman/remove.c +++ b/src/pacman/remove.c @@ -87,9 +87,16 @@ int pacman_remove(alpm_list_t *targets) } /* Step 0: create a new transaction */ - if(trans_init(config->flags, 0) == -1) { - return 1; - } + while(1) { + if(trans_init(config->flags, 1) == -1) { + if(yesno(_("Retry?")) == 0) { + return 1; + } + /* Try again */ + continue; + } + break; + } /* Step 1: add targets to the created transaction */ for(i = targets; i; i = alpm_list_next(i)) { diff --git a/src/pacman/sync.c b/src/pacman/sync.c index f7dcb958..89f01061 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -694,9 +694,16 @@ static int sync_trans(alpm_list_t *targets) alpm_list_t *i; /* Step 1: create a new transaction... */ - if(trans_init(config->flags, 1) == -1) { - return 1; - } + while(1) { + if(trans_init(config->flags, 1) == -1) { + if(yesno(_("Retry?")) == 0) { + return 1; + } + /* Try again */ + continue; + } + break; + } /* process targets */ for(i = targets; i; i = alpm_list_next(i)) { diff --git a/src/pacman/upgrade.c b/src/pacman/upgrade.c index 5f984e44..406fec24 100644 --- a/src/pacman/upgrade.c +++ b/src/pacman/upgrade.c @@ -79,9 +79,16 @@ int pacman_upgrade(alpm_list_t *targets) } /* Step 1: create a new transaction */ - if(trans_init(config->flags, 1) == -1) { - retval = 1; - goto fail_free; + while(1) { + if(trans_init(config->flags, 1) == -1) { + if(yesno(_("Retry?")) == 0) { + retval = 1; + goto fail_free; + } + /* Try again */ + continue; + } + break; } printf(_("loading packages...\n")); -- 2.26.2

3 2

[pacman-dev] [PATCH] Add a utility function to check whether a file exists in the cache
by Anatol Pomozov 27 Apr '20

27 Apr '20

It is similar to _alpm_filecache_find() but does not return a dynamically allocated memory to user. Thus the user does not need to free this resource. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/util.c | 14 ++++++++++++++ lib/libalpm/util.h | 2 ++ 2 files changed, 16 insertions(+) diff --git a/lib/libalpm/util.c b/lib/libalpm/util.c index cb838e43..6307dc14 100644 --- a/lib/libalpm/util.c +++ b/lib/libalpm/util.c @@ -834,6 +834,20 @@ char *_alpm_filecache_find(alpm_handle_t *handle, const char *filename) return NULL; } +/** Check whether a filename exists in a registered alpm cachedir. + * @param handle the context handle + * @param filename name of file to find + * @return 0 if the filename was not found, 1 otherwise + */ +int _alpm_filecache_exists(alpm_handle_t *handle, const char *filename) +{ + int res; + char *fpath = _alpm_filecache_find(handle, filename); + res = !!fpath; + FREE(fpath); + return res; +} + /** Check the alpm cachedirs for existence and find a writable one. * If no valid cache directory can be found, use /tmp. * @param handle the context handle diff --git a/lib/libalpm/util.h b/lib/libalpm/util.h index 3306a022..4fc6e718 100644 --- a/lib/libalpm/util.h +++ b/lib/libalpm/util.h @@ -136,6 +136,8 @@ int _alpm_run_chroot(alpm_handle_t *handle, const char *cmd, char *const argv[], int _alpm_ldconfig(alpm_handle_t *handle); int _alpm_str_cmp(const void *s1, const void *s2); char *_alpm_filecache_find(alpm_handle_t *handle, const char *filename); +/* Checks whether a file exists in cache */ +int _alpm_filecache_exists(alpm_handle_t *handle, const char *filename); const char *_alpm_filecache_setup(alpm_handle_t *handle); /* Unlike many uses of alpm_pkgvalidation_t, _alpm_test_checksum expects * an enum value rather than a bitfield. */ -- 2.26.2

2 2

[pacman-dev] [PATCH v2] Add a utility function to check whether a file exists in the cache
by Anatol Pomozov 27 Apr '20

27 Apr '20

It is similar to _alpm_filecache_find() but does not return a dynamically allocated memory to user. Thus the user does not need to free this resource. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/util.c | 14 ++++++++++++++ lib/libalpm/util.h | 2 ++ 2 files changed, 16 insertions(+) diff --git a/lib/libalpm/util.c b/lib/libalpm/util.c index cb838e43..5c8c5ed6 100644 --- a/lib/libalpm/util.c +++ b/lib/libalpm/util.c @@ -834,6 +834,20 @@ char *_alpm_filecache_find(alpm_handle_t *handle, const char *filename) return NULL; } +/** Check whether a filename exists in a registered alpm cachedir. + * @param handle the context handle + * @param filename name of file to find + * @return 0 if the filename was not found, 1 otherwise + */ +int _alpm_filecache_exists(alpm_handle_t *handle, const char *filename) +{ + int res; + char *fpath = _alpm_filecache_find(handle, filename); + res = (fpath != NULL); + FREE(fpath); + return res; +} + /** Check the alpm cachedirs for existence and find a writable one. * If no valid cache directory can be found, use /tmp. * @param handle the context handle diff --git a/lib/libalpm/util.h b/lib/libalpm/util.h index 3306a022..4fc6e718 100644 --- a/lib/libalpm/util.h +++ b/lib/libalpm/util.h @@ -136,6 +136,8 @@ int _alpm_run_chroot(alpm_handle_t *handle, const char *cmd, char *const argv[], int _alpm_ldconfig(alpm_handle_t *handle); int _alpm_str_cmp(const void *s1, const void *s2); char *_alpm_filecache_find(alpm_handle_t *handle, const char *filename); +/* Checks whether a file exists in cache */ +int _alpm_filecache_exists(alpm_handle_t *handle, const char *filename); const char *_alpm_filecache_setup(alpm_handle_t *handle); /* Unlike many uses of alpm_pkgvalidation_t, _alpm_test_checksum expects * an enum value rather than a bitfield. */ -- 2.26.2

1 0

[pacman-dev] [PATCH] Constify some input pointers
by Rikard Falkeborn 27 Apr '20

27 Apr '20

Signed-off-by: Rikard Falkeborn <rikard.falkeborn(a)gmail.com> --- lib/libalpm/diskspace.c | 2 +- lib/libalpm/diskspace.h | 2 +- lib/libalpm/util.c | 2 +- src/pacman/util.c | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/libalpm/diskspace.c b/lib/libalpm/diskspace.c index 4cc17a12..2be98055 100644 --- a/lib/libalpm/diskspace.c +++ b/lib/libalpm/diskspace.c @@ -359,7 +359,7 @@ static int check_mountpoint(alpm_handle_t *handle, alpm_mountpoint_t *mp) } int _alpm_check_downloadspace(alpm_handle_t *handle, const char *cachedir, - size_t num_files, off_t *file_sizes) + size_t num_files, const off_t *file_sizes) { alpm_list_t *mount_points; alpm_mountpoint_t *cachedir_mp; diff --git a/lib/libalpm/diskspace.h b/lib/libalpm/diskspace.h index 4f7d00b9..5f17a878 100644 --- a/lib/libalpm/diskspace.h +++ b/lib/libalpm/diskspace.h @@ -58,6 +58,6 @@ typedef struct __alpm_mountpoint_t { int _alpm_check_diskspace(alpm_handle_t *handle); int _alpm_check_downloadspace(alpm_handle_t *handle, const char *cachedir, - size_t num_files, off_t *file_sizes); + size_t num_files, const off_t *file_sizes); #endif /* ALPM_DISKSPACE_H */ diff --git a/lib/libalpm/util.c b/lib/libalpm/util.c index cb838e43..ead03004 100644 --- a/lib/libalpm/util.c +++ b/lib/libalpm/util.c @@ -1006,7 +1006,7 @@ static int sha256_file(const char *path, unsigned char output[32]) * @return a NULL terminated string with the hexadecimal representation of * bytes or NULL on error. This string must be freed. */ -static char *hex_representation(unsigned char *bytes, size_t size) +static char *hex_representation(const unsigned char *bytes, size_t size) { static const char *hex_digits = "0123456789abcdef"; char *str; diff --git a/src/pacman/util.c b/src/pacman/util.c index 6693ec75..97b8e06d 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -515,7 +515,7 @@ void string_display(const char *title, const char *string, unsigned short cols) } static void table_print_line(const alpm_list_t *line, short col_padding, - size_t colcount, size_t *widths, int *has_data) + size_t colcount, const size_t *widths, const int *has_data) { size_t i; int need_padding = 0; -- 2.26.2

2 1

[pacman-dev] Dropping SIGPIPE workaround from dload.c
by Anatol Pomozov 20 Apr '20

20 Apr '20

Hi folks I am looking at the lib/libalpm/dload.c download codepath and found that it has a following snippet: /* Ignore any SIGPIPE signals. With libcurl, these shouldn't be happening, * but better safe than sorry. Store the old signal handler first. */ mask_signal(SIGPIPE, SIG_IGN, &orig_sig_pipe); dload_interrupted = 0; mask_signal(SIGINT, &inthandler, &orig_sig_int); My understanding that this code tries to handle DNS timeouts that are signaled as a SIGPIPE. And to avoid killing the whole app this code installs this sighandler. But my experiment (at my dev branch that removes the codesnippet above) shows that curl handles DNS timeout correctly: sudo pacman -U http://blackhole.webpagetest.org/foo.zst :: Retrieving packages... foo.zst failed to download foo.zst.sig failed to download error: failed retrieving file 'foo.zst' from blackhole.webpagetest.org : Connection timed out after 10000 milliseconds error: failed retrieving file 'foo.zst.sig' from blackhole.webpagetest.org : Connection timed out after 10000 milliseconds warning: failed to retrieve some files There is also a test that checks for SIGPIPE handling (test/pacman/tests/scriptlet-signal-reset.py) and it passes as well: 157/332 scriptlet-signal-reset.py OK 0.11832118034362793 s Which makes me believe that SIGPIPE handler trick is not needed and mCURL handles timeouts correctly. I've been using a branch without this handler for two months and never seen any signal related issues with my parallel-download pacman. I am thinking of *not* carrying this workaround to the new multiplexed function and want to check with you if you are OK with it.

2 1

[pacman-dev] [PATCH] Implement multibar UI
by Anatol Pomozov 17 Apr '20

17 Apr '20

Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down. `active_dls` variable represents the list of active downloads. `struct dl_progress_bar` is a data structure for an active progress bar. In some cases we want to keep progress bars in order (e.g. db downloads). In some other cases (package downloads) we want to move completed items to the top of the screen. Global `multibar_move_complete_top` allows to configure such behavior. Optional downloads (such as optional signature files) do not have a visible progress bar. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/dload.c | 31 +--- lib/libalpm/dload.h | 1 - src/pacman/callback.c | 401 +++++++++++++++++++++++++++++------------- src/pacman/callback.h | 3 + src/pacman/sync.c | 2 + 5 files changed, 294 insertions(+), 144 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index c09abde4..cf467ff0 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -100,6 +100,11 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, return 1; } + if(dlnow < 0 || dltotal <= 0 || dlnow > dltotal) { + /* bogus values : stop here */ + return 0; + } + current_size = payload->initial_size + dlnow; /* is our filesize still under any set limit? */ @@ -115,34 +120,15 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, total_size = payload->initial_size + dltotal; - if(dltotal == 0 || payload->prevprogress == total_size) { + if(payload->prevprogress == total_size) { return 0; } - /* initialize the progress bar here to avoid displaying it when - * a repo is up to date and nothing gets downloaded. - * payload->handle->dlcb will receive the remote_name - * and the following arguments: - * 0, -1: download initialized - * 0, 0: non-download event - * x {x>0}, x: download complete - * x {x>0, x<y}, y {y > 0}: download progress, expected total is known */ - if(!payload->cb_initialized) { - cb_data.downloaded = 0; - cb_data.total = -1; - payload->cb_initialized = 1; - } - if(payload->prevprogress == current_size) { - cb_data.downloaded = 0; - cb_data.total = 0; - } else { /* do NOT include initial_size since it wasn't part of the package's * download_size (nor included in the total download size callback) */ - cb_data.downloaded = dlnow; - cb_data.total = dltotal; - } + cb_data.total = dltotal; + cb_data.downloaded = dlnow; payload->handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_PROGRESS, &cb_data); - payload->prevprogress = current_size; return 0; @@ -1160,5 +1146,4 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; - payload->cb_initialized = 0; } diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index a40b51b7..142568e8 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -42,7 +42,6 @@ struct dload_payload { int errors_ok; int unlink_on_fail; int trust_remote_name; - int cb_initialized; #ifdef HAVE_LIBCURL CURL *curl; char error_buffer[CURL_ERROR_SIZE]; diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 387319b6..a4aafada 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -18,6 +18,8 @@ * along with this program. If not, see <http://www.gnu.org/licenses/>. */ +#include <assert.h> +#include <stdbool.h> #include <stdio.h> #include <stdlib.h> #include <string.h> @@ -45,6 +47,8 @@ static off_t list_total = 0.0; static int on_progress = 0; static alpm_list_t *output = NULL; +static void cursor_goto_end(void); + /* update speed for the fill_progress based functions */ #define UPDATE_SPEED_MS 200 @@ -152,11 +156,7 @@ static void fill_progress(const int bar_percent, const int disp_percent, printf(" %3d%%", disp_percent); } - if(bar_percent == 100) { - putchar('\n'); - } else { - putchar('\r'); - } + putchar('\r'); fflush(stdout); } @@ -346,6 +346,7 @@ void cb_event(alpm_event_t *event) case ALPM_EVENT_DB_RETRIEVE_FAILED: case ALPM_EVENT_RETRIEVE_DONE: case ALPM_EVENT_RETRIEVE_FAILED: + cursor_goto_end(); flush_output_list(); on_progress = 0; break; @@ -629,6 +630,7 @@ void cb_progress(alpm_progress_t event, const char *pkgname, int percent, fill_progress(percent, percent, cols - infolen); if(percent == 100) { + putchar('\n'); flush_output_list(); on_progress = 0; } else { @@ -647,124 +649,121 @@ void cb_dl_total(off_t total) } } -/* callback to handle display of download progress */ -static void dload_progress_event(const char *filename, off_t file_xfered, off_t file_total) +static int dload_progressbar_enabled(void) { - static double rate_last; - static off_t xfered_last; - static int64_t initial_time = 0; - int infolen; - int filenamelen; - char *fname, *p; - /* used for wide character width determination and printing */ - int len, wclen, wcwid, padwid; - wchar_t *wcfname; + return !config->noprogressbar && getcols() != 0; +} - int totaldownload = 0; - off_t xfered, total; - double rate = 0.0; - unsigned int eta_h = 0, eta_m = 0, eta_s = 0; - double rate_human, xfered_human; - const char *rate_label, *xfered_label; - int file_percent = 0, total_percent = 0; +struct dl_progress_bar { + const char *filename; + off_t xfered; + off_t total_size; + uint64_t init_time; /* Time when this download started doing any progress */ + uint64_t sync_time; /* Last time we updated the bar info */ + double rate; + unsigned int eta; /* ETA in seconds */ + bool completed; /* transfer is completed */ + bool visible; /* whether to show the bar, some downloads (e.g. db.sig files) are not shown */ +}; - const unsigned short cols = getcols(); +/* List of active downloads handled by multibar UI. + * Once the first download in the list is completed it is removed + * from this list and we never redraw it anymore. + * If the download in this list then UI can redraw the progress bar or change + * the order of the bars (e.g. completed bar is moved to the top of the list) + */ +alpm_list_t *active_dls = NULL; +/* Number of active progress bars that multibar UI handles. + * Each progress bar has a corresponding active download. + * Note that some downloads might not have a bar, e.g. optional signature files + * do not have a corresponding bar. + */ +size_t active_bars_no = 0; - /* Nothing has changed since last callback; stop here */ - if(file_xfered == 0 && file_total == 0) { - return; - } +int multibar_move_complete_top = 0; - if(config->noprogressbar || cols == 0) { - if(file_xfered == 0 && file_total == -1) { - printf(_("downloading %s...\n"), filename); - fflush(stdout); - } - return; - } +/* Cursor position relative to the first active progress bar, + * e.g. 0 means the first active progress bar, active_bars_no-1 means the last bar, + * active_bars_no - is the line below all progress bars. + */ +int cursor_lineno = 0; - infolen = cols * 6 / 10; - if(infolen < 50) { - infolen = 50; - } - /* only use TotalDownload if enabled and we have a callback value */ - if(config->totaldownload && list_total) { - /* sanity check */ - if(list_xfered + file_total <= list_total) { - totaldownload = 1; - } else { - /* bogus values : don't enable totaldownload and reset */ - list_xfered = 0; - list_total = 0; - } +/* Move console cursor `lines` up */ +static void cursor_line_up(unsigned int lines) +{ + assert(lines > 0); + printf("\x1B[%dF", lines); +} + +/* Move console cursor `lines` down */ +static void cursor_line_down(unsigned int lines) +{ + assert(lines > 0); + printf("\x1B[%dE", lines); +} + +/* Erase line from the cursor position till the end of the line */ +static void console_erase_line(void) +{ + printf("\x1B[K"); +} + +/* Goto the line that corresponds to num-th active download */ +static void cursor_goto_bar(int num) +{ + if(num > cursor_lineno) { + cursor_line_down(num - cursor_lineno); + } else if(num < cursor_lineno) { + cursor_line_up(cursor_lineno - num); } + cursor_lineno = num; +} - if(totaldownload) { - xfered = list_xfered + file_xfered; - total = list_total; - } else { - xfered = file_xfered; - total = file_total; - } - - /* this is basically a switch on xfered: 0, total, and - * anything else */ - if(file_xfered == 0 && file_total == -1) { - /* set default starting values, ensure we only call this once - * if TotalDownload is enabled */ - if(!totaldownload || (totaldownload && list_xfered == 0)) { - initial_time = get_time_ms(); - xfered_last = (off_t)0; - rate_last = 0.0; - get_update_timediff(1); - } - } else if(xfered > total || xfered < 0) { - /* bogus values : stop here */ - return; - } else if(file_xfered == file_total) { - /* compute final values */ - int64_t timediff = get_time_ms() - initial_time; - if(timediff > 0) { - rate = (double)xfered / (timediff / 1000.0); - /* round elapsed time (in ms) to the nearest second */ - eta_s = (unsigned int)(timediff + 500) / 1000; - } else { - eta_s = 0; - } - } else { - /* compute current average values */ - int64_t timediff = get_update_timediff(0); +/* Goto the line *after* the last active progress bar */ +static void cursor_goto_end(void) +{ + cursor_goto_bar(active_bars_no); +} - if(timediff < UPDATE_SPEED_MS) { - /* return if the calling interval was too short */ - return; +/* Returns true if element with the specified name is found, false otherwise */ +static bool find_bar_for_filename(const char *filename, int *index, struct dl_progress_bar **bar) +{ + int i = 0; + alpm_list_t *listitem = active_dls; + for(; listitem; listitem = listitem->next) { + struct dl_progress_bar *b = listitem->data; + if (b->filename == filename /* or strcmp() ? */) { + *index = i; + *bar = b; + return true; } - rate = (double)(xfered - xfered_last) / (timediff / 1000.0); - /* average rate to reduce jumpiness */ - rate = (rate + 2 * rate_last) / 3; - if(rate > 0.0) { - eta_s = (total - xfered) / rate; - } else { - eta_s = UINT_MAX; + if(b->visible) { + i++; } - rate_last = rate; - xfered_last = xfered; } - if(file_total) { - file_percent = (file_xfered * 100) / file_total; - } else { - file_percent = 100; - } + return false; +} + +static void draw_dl_progress_bar(struct dl_progress_bar *bar) +{ + int infolen; + int filenamelen; + char *fname, *p; + /* used for wide character width determination and printing */ + int len, wclen, wcwid, padwid; + wchar_t *wcfname; + unsigned int eta_h = 0, eta_m = 0, eta_s = bar->eta; + double rate_human, xfered_human; + const char *rate_label, *xfered_label; + int file_percent = 0; - if(totaldownload) { - total_percent = ((list_xfered + file_xfered) * 100) / - list_total; + const unsigned short cols = getcols(); - /* if we are at the end, add the completed file to list_xfered */ - if(file_xfered == file_total) { - list_xfered += file_total; - } + if(bar->total_size) { + file_percent = (bar->xfered * 100) / bar->total_size; + } else { + file_percent = 100; } /* fix up time for display */ @@ -773,13 +772,13 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t eta_m = eta_s / 60; eta_s -= eta_m * 60; - len = strlen(filename); + len = strlen(bar->filename); fname = malloc(len + 1); - memcpy(fname, filename, len + 1); + memcpy(fname, bar->filename, len + 1); /* strip package or DB extension for cleaner look */ if((p = strstr(fname, ".pkg")) || (p = strstr(fname, ".db")) || (p = strstr(fname, ".files"))) { /* tack on a .sig suffix for signatures */ - if(memcmp(&filename[len - 4], ".sig", 4) == 0) { + if(memcmp(&bar->filename[len - 4], ".sig", 4) == 0) { memcpy(p, ".sig", 4); /* adjust length for later calculations */ @@ -790,6 +789,11 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t fname[len] = '\0'; } + infolen = cols * 6 / 10; + if(infolen < 50) { + infolen = 50; + } + /* 1 space + filenamelen + 1 space + 6 for size + 1 space + 3 for label + * + 2 spaces + 4 for rate + 1 space + 3 for label + 2 for /s + 1 space + * 8 for eta, gives us the magic 33 */ @@ -824,8 +828,8 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t } - rate_human = humanize_size((off_t)rate, '\0', -1, &rate_label); - xfered_human = humanize_size(xfered, '\0', -1, &xfered_label); + rate_human = humanize_size((off_t)bar->rate, '\0', -1, &rate_label); + xfered_human = humanize_size(bar->xfered, '\0', -1, &xfered_label); printf(" %ls%-*s ", wcfname, padwid, ""); /* We will show 1.62 MiB/s, 11.6 MiB/s, but 116 KiB/s and 1116 KiB/s */ @@ -850,19 +854,176 @@ static void dload_progress_event(const char *filename, off_t file_xfered, off_t free(fname); free(wcfname); - if(totaldownload) { - fill_progress(file_percent, total_percent, cols - infolen); + fill_progress(file_percent, file_percent, cols - infolen); + return; +} + +static void dload_init_event(const char *filename, alpm_download_event_init_t *data) +{ + (void)data; + + if(!dload_progressbar_enabled()) { + printf(_(" %s downloading...\n"), filename); + return; + } + + struct dl_progress_bar *bar = calloc(1, sizeof(struct dl_progress_bar)); + if(!bar) { + return; + } + bar->filename = filename; + bar->init_time = get_time_ms(); + bar->rate = 0.0; + bar->visible = !data->optional; + active_dls = alpm_list_add(active_dls, bar); + + if(bar->visible) { + cursor_goto_end(); + printf(_(" %s downloading...\n"), filename); + cursor_lineno++; + active_bars_no++; + } +} + +/* render download progress */ +static void dload_progress_event(const char *filename, alpm_download_event_progress_t *data) +{ + int index; + struct dl_progress_bar *bar; + int64_t curr_time = get_time_ms(); + double last_chunk_rate; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + + /* compute current average values */ + timediff = curr_time - bar->sync_time; + + if(timediff < UPDATE_SPEED_MS) { + /* return if the calling interval was too short */ + return; + } + bar->sync_time = curr_time; + + last_chunk_rate = (double)(data->downloaded - bar->xfered) / (timediff / 1000.0); + /* average rate to reduce jumpiness */ + bar->rate = (last_chunk_rate + 2 * bar->rate) / 3; + if(bar->rate > 0.0) { + bar->eta = (data->total - data->downloaded) / bar->rate; } else { - fill_progress(file_percent, file_percent, cols - infolen); + bar->eta = UINT_MAX; + } + + /* Total size is received after the download starts. */ + bar->total_size = data->total; + bar->xfered = data->downloaded; + + if(bar->visible) { + cursor_goto_bar(index); + draw_dl_progress_bar(bar); + fflush(stdout); } - return; } +/* download completed */ +static void dload_complete_event(const char *filename, alpm_download_event_completed_t *data) +{ + int index; + struct dl_progress_bar *bar; + int64_t timediff; + + if(!dload_progressbar_enabled()) { + return; + } + + assert(find_bar_for_filename(filename, &index, &bar)); + bar->completed = true; + + /* This may not have been initialized if the download finished before + * an alpm_download_event_progress_t event happened */ + bar->total_size = data->total; + + if(bar->visible) { + if(data->result == 1) { + cursor_goto_bar(index); + printf(_(" %s is up to date"), bar->filename); + /* The line contains text from previous status. Erase these leftovers. */ + console_erase_line(); + } else if(data->result == 0) { + /* compute final values */ + bar->xfered = bar->total_size; + timediff = get_time_ms() - bar->init_time; + + /* if transfer was too fast, treat it as a 1ms transfer, for the sake + * of the rate calculation */ + if(timediff < 1) + timediff = 1; + + bar->rate = (double)bar->xfered / (timediff / 1000.0); + /* round elapsed time (in ms) to the nearest second */ + bar->eta = (unsigned int)(timediff + 500) / 1000; + + if(multibar_move_complete_top && index != 0) { + /* If this item completed then move it to the top. + * Swap 0-th bar data with `index`-th one + */ + struct dl_progress_bar *former_topbar = active_dls->data; + alpm_list_t *baritem = alpm_list_nth(active_dls, index); + active_dls->data = bar; + baritem->data = former_topbar; + + cursor_goto_bar(index); + draw_dl_progress_bar(former_topbar); + + index = 0; + } + + cursor_goto_bar(index); + draw_dl_progress_bar(bar); + } else { + cursor_goto_bar(index); + printf(_(" %s failed to download"), bar->filename); + console_erase_line(); + } + fflush(stdout); + } + + /* If the first bar is completed then there is no reason to keep it + * in the list as we are not going to redraw it anymore. + */ + while(active_dls) { + alpm_list_t *head = active_dls; + struct dl_progress_bar *j = head->data; + if(j->completed) { + if(j->visible) { + cursor_lineno--; + active_bars_no--; + } + active_dls = alpm_list_remove_item(active_dls, head); + free(head); + free(j); + } else { + break; + } + } +} + +/* callback to handle display of download progress */ void cb_download(const char *filename, alpm_download_event_type_t event, void *data) { - if(event == ALPM_DOWNLOAD_PROGRESS) { - alpm_download_event_progress_t *progress = data; - dload_progress_event(filename, progress->downloaded, progress->total); + if(event == ALPM_DOWNLOAD_INIT) { + dload_init_event(filename, data); + } else if(event == ALPM_DOWNLOAD_PROGRESS) { + dload_progress_event(filename, data); + } else if(event == ALPM_DOWNLOAD_COMPLETED) { + dload_complete_event(filename, data); + } else { + pm_printf(ALPM_LOG_ERROR, _("unknown callback event type %d for %s\n"), + event, filename); } } diff --git a/src/pacman/callback.h b/src/pacman/callback.h index 6d92e86b..916b9dad 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -44,4 +44,7 @@ void cb_download(const char *filename, alpm_download_event_type_t event, __attribute__((format(printf, 2, 0))) void cb_log(alpm_loglevel_t level, const char *fmt, va_list args); +/* specify if multibar should move complete bars to the top of the screen */ +int multibar_move_complete_top; + #endif /* PM_CALLBACK_H */ diff --git a/src/pacman/sync.c b/src/pacman/sync.c index f7dcb958..d5886f13 100644 --- a/src/pacman/sync.c +++ b/src/pacman/sync.c @@ -35,6 +35,7 @@ #include "pacman.h" #include "util.h" #include "package.h" +#include "callback.h" #include "conf.h" static int unlink_verbose(const char *pathname, int ignore_missing) @@ -824,6 +825,7 @@ int sync_prepare_execute(void) goto cleanup; } + multibar_move_complete_top = 1; if(alpm_trans_commit(config->handle, &data) == -1) { alpm_errno_t err = alpm_errno(config->handle); pm_printf(ALPM_LOG_ERROR, _("failed to commit transaction (%s)\n"), -- 2.26.1

1 1

[pacman-dev] [PATCH] Extend download callback interface with start/complete events
by Anatol Pomozov 17 Apr '20

17 Apr '20

With the previous download interface the callback uses the first progress event as 'download has started' signal. Unfortunately it does not work with up-to-date files that never receive 'download progress' events. Up-to-date database messages are currently handled in sync_syncdbs() after the sequential download is completed and a result from ALPM is received. But this is not going to work with multiplexed download interface that returns the result only after all files are completed. Another problem with 'first progress event is the beginning of the download' is that such events time are unpredictable. Thus the UI progress bar order might differ from what has been passed by client to alpm_dbs_update() function. We actually want to keep the dbs progress bars in a strict order. To help to solve the given problems extend the download callback to allow 2 more events - download started and completed. 'Download started' events appear in the same order as in the list given by a client. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/alpm.h | 35 ++++++++++++++++++++++++++++++++--- lib/libalpm/dload.c | 19 ++++++++++++++----- src/pacman/callback.c | 10 +++++++++- src/pacman/callback.h | 3 ++- src/pacman/conf.c | 2 +- 5 files changed, 58 insertions(+), 11 deletions(-) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index aee5aa0d..8fe11a9b 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -715,13 +715,42 @@ typedef void (*alpm_cb_progress)(alpm_progress_t, const char *, int, size_t, siz * Downloading */ +/* File download events. + * These events are reported by ALPM via download callback. + */ +typedef enum { + ALPM_DOWNLOAD_INIT, /* alpm initializes file download logic */ + ALPM_DOWNLOAD_PROGRESS, /* download progress reported */ + ALPM_DOWNLOAD_COMPLETED /* alpm is about to release data related to the file */ +} alpm_download_event_type_t; + +typedef struct { + int optional; /* whether this file is optional and thus the errors could be ignored */ +} alpm_download_event_init_t; + +typedef struct { + off_t downloaded; /* amount of data downloaded */ + off_t total; /* total amount need to be downloaded */ +} alpm_download_event_progress_t; + +typedef struct { + /* total bytes in file */ + off_t total; + /* download result code: + * 0 - download completed successfully + * 1 - the file is up-to-date + * negative - error code + */ + int result; +} alpm_download_event_completed_t; + /** Type of download progress callbacks. * @param filename the name of the file being downloaded - * @param xfered the number of transferred bytes - * @param total the total number of bytes to transfer + * @param event the event type + * @param data the event data of type alpm_download_event_*_t */ typedef void (*alpm_cb_download)(const char *filename, - off_t xfered, off_t total); + alpm_download_event_type_t event, void *data); typedef void (*alpm_cb_totaldl)(off_t total); diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 786c11a1..c09abde4 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -88,6 +88,7 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, { struct dload_payload *payload = (struct dload_payload *)file; off_t current_size, total_size; + alpm_download_event_progress_t cb_data = {0}; /* avoid displaying progress bar for redirects with a body */ if(payload->respcode >= 300) { @@ -127,16 +128,20 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, * x {x>0}, x: download complete * x {x>0, x<y}, y {y > 0}: download progress, expected total is known */ if(!payload->cb_initialized) { - payload->handle->dlcb(payload->remote_name, 0, -1); + cb_data.downloaded = 0; + cb_data.total = -1; payload->cb_initialized = 1; } if(payload->prevprogress == current_size) { - payload->handle->dlcb(payload->remote_name, 0, 0); + cb_data.downloaded = 0; + cb_data.total = 0; } else { /* do NOT include initial_size since it wasn't part of the package's * download_size (nor included in the total download size callback) */ - payload->handle->dlcb(payload->remote_name, dlnow, dltotal); + cb_data.downloaded = dlnow; + cb_data.total = dltotal; } + payload->handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_PROGRESS, &cb_data); payload->prevprogress = current_size; @@ -655,6 +660,7 @@ static int curl_multi_check_finished_download(CURLM *curlm, CURLMsg *msg, long remote_time = -1; struct stat st; char hostname[HOSTNAME_SIZE]; + alpm_download_event_completed_t cb_data = {0}; int ret = -1; curlerr = curl_easy_getinfo(curl, CURLINFO_PRIVATE, &payload); @@ -812,7 +818,9 @@ cleanup: unlink(payload->tempfile_name); } - // TODO: report that the download has been completed + cb_data.total = bytes_dl; + cb_data.result = ret; + handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_COMPLETED, &cb_data); curl_multi_remove_handle(curlm, curl); curl_easy_cleanup(curl); @@ -920,9 +928,10 @@ static int curl_multi_download_internal(alpm_handle_t *handle, for(; still_running < parallel_downloads && payloads; still_running++) { struct dload_payload *payload = payloads->data; + alpm_download_event_init_t cb_data = {.optional = payload->errors_ok}; if(curl_multi_add_payload(handle, curlm, payloads->data, localpath) == 0) { - // TODO: report that download has started + handle->dlcb(payload->remote_name, ALPM_DOWNLOAD_INIT, &cb_data); payloads = payloads->next; } else { // the payload failed to start, do not start any new downloads just wait until diff --git a/src/pacman/callback.c b/src/pacman/callback.c index 585eae09..387319b6 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -648,7 +648,7 @@ void cb_dl_total(off_t total) } /* callback to handle display of download progress */ -void cb_dl_progress(const char *filename, off_t file_xfered, off_t file_total) +static void dload_progress_event(const char *filename, off_t file_xfered, off_t file_total) { static double rate_last; static off_t xfered_last; @@ -858,6 +858,14 @@ void cb_dl_progress(const char *filename, off_t file_xfered, off_t file_total) return; } +void cb_download(const char *filename, alpm_download_event_type_t event, void *data) +{ + if(event == ALPM_DOWNLOAD_PROGRESS) { + alpm_download_event_progress_t *progress = data; + dload_progress_event(filename, progress->downloaded, progress->total); + } +} + /* Callback to handle notifications from the library */ void cb_log(alpm_loglevel_t level, const char *fmt, va_list args) { diff --git a/src/pacman/callback.h b/src/pacman/callback.h index b4bde5f2..6d92e86b 100644 --- a/src/pacman/callback.h +++ b/src/pacman/callback.h @@ -37,7 +37,8 @@ void cb_progress(alpm_progress_t event, const char *pkgname, int percent, /* callback to handle receipt of total download value */ void cb_dl_total(off_t total); /* callback to handle display of download progress */ -void cb_dl_progress(const char *filename, off_t file_xfered, off_t file_total); +void cb_download(const char *filename, alpm_download_event_type_t event, + void *data); /* callback to handle messages/notifications from pacman library */ __attribute__((format(printf, 2, 0))) diff --git a/src/pacman/conf.c b/src/pacman/conf.c index 2dcf7a5b..5a450426 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -835,7 +835,7 @@ static int setup_libalpm(void) config->handle = handle; alpm_option_set_logcb(handle, cb_log); - alpm_option_set_dlcb(handle, cb_dl_progress); + alpm_option_set_dlcb(handle, cb_download); alpm_option_set_eventcb(handle, cb_event); alpm_option_set_questioncb(handle, cb_question); alpm_option_set_progresscb(handle, cb_progress); -- 2.26.1

1 0

[pacman-dev] [GIT] The official pacman repository branch, master, updated. v5.2.1-59-g3ea7fdd9
by Allan McRae 15 Apr '20

15 Apr '20

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, master has been updated via 3ea7fdd96c874e111588a6e64ee30b75b55db785 (commit) via b323528491470ff77815fea97289c2cbf7e29689 (commit) via 6ba250e4001740ca428226abf157b25aa121c7bf (commit) via 0eda92c5d4f8810a68066ee72713ffdfa878dd8c (commit) via 1b3289745334ec31507a12b6c54b2883a521543e (commit) via 1d39557aa070d2260cfda650e59c8d190397ba01 (commit) from 87b74fcd8257e3c2fbf2f2949f0561df7517ce71 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 3ea7fdd96c874e111588a6e64ee30b75b55db785 Author: Allan McRae <allan(a)archlinux.org> Date: Tue Apr 14 23:45:24 2020 +1000 pacman.8: Fix typo Signed-off-by: Allan McRae <allan(a)archlinux.org> commit b323528491470ff77815fea97289c2cbf7e29689 Author: Carson Black <uhhadd(a)gmail.com> Date: Tue Apr 14 20:51:03 2020 -0400 Dull version colour numbers in summary Version colour numbers are dulled in the non-verbose transaction summary when colours are enabled. To prevent a regression, this patch also adds handling of strings with ANSI codes to string_length as to not break the transaction summary's output functions when colour codes are in the package name strings. Signed-off-by: Carson Black <uhhadd(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 6ba250e4001740ca428226abf157b25aa121c7bf Author: Allan McRae <allan(a)archlinux.org> Date: Mon Apr 13 22:24:23 2020 +1000 Use GOTO_ERR throughout The GOTO_ERR define was added in commit 80ae8014 for use in future commits. There are plenty of places in the code base it can be used, so convert them. Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 0eda92c5d4f8810a68066ee72713ffdfa878dd8c Author: Allan McRae <allan(a)archlinux.org> Date: Mon Apr 13 17:39:35 2020 +1000 Use STRDUP for error checking in more places Use STRDUP() over strdup() to catch memory allocation errors. There are still some instances of strdup left, but these are in functions that currently have no error path and would require a larger rework. Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 1b3289745334ec31507a12b6c54b2883a521543e Author: Rikard Falkeborn <rikard.falkeborn(a)gmail.com> Date: Mon Apr 13 07:28:48 2020 +0200 Add REALLOC macro to simplify realloc error handling realloc can fail just like the other memory allocation functions. Add a macro to simplify handling of realloc failures, similar to the already existing MALLOC, CALLOC, etc. Replace the existing realloc uses with the new macro, allowing us to move tedious error handling to the macro. Also, in be_package and be_sync, this fixes hypothetical memory leaks (and thereafter null pointer dereferences) in case realloc fails to shrink the allocated memory. Signed-off-by: Rikard Falkeborn <rikard.falkeborn(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> commit 1d39557aa070d2260cfda650e59c8d190397ba01 Author: Earnestly <zibeon(a)googlemail.com> Date: Wed Apr 1 11:29:57 2020 -0400 Use noextract with pacman-conf NoExtract Current code accidently uses noupgrade for the NoExtract directive. Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: doc/pacman.8.asciidoc | 2 +- lib/libalpm/be_local.c | 13 +++---------- lib/libalpm/be_package.c | 37 ++++++++++++++++--------------------- lib/libalpm/be_sync.c | 8 +++----- lib/libalpm/dload.c | 6 ++---- lib/libalpm/handle.c | 2 +- lib/libalpm/signing.c | 16 ++++++---------- lib/libalpm/sync.c | 5 ++--- lib/libalpm/util.c | 16 +++++----------- lib/libalpm/util.h | 1 + src/pacman/conf.c | 4 ++++ src/pacman/conf.h | 1 + src/pacman/pacman-conf.c | 2 +- src/pacman/util.c | 46 ++++++++++++++++++++++++++++++++++------------ 14 files changed, 80 insertions(+), 79 deletions(-) hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [PATCH v3] Dull version colour numbers in summary
by Carson Black 15 Apr '20

15 Apr '20

Version colour numbers are dulled in the non-verbose transaction summary when colours are enabled. To prevent a regression, this patch also adds handling of strings with ANSI codes to string_length as to not break the transaction summary's output functions when colour codes are in the package name strings. Signed-off-by: Carson Black <uhhadd(a)gmail.com> --- src/pacman/conf.c | 4 ++++ src/pacman/conf.h | 1 + src/pacman/util.c | 52 ++++++++++++++++++++++++++++++++++++----------- 3 files changed, 45 insertions(+), 12 deletions(-) diff --git a/src/pacman/conf.c b/src/pacman/conf.c index f9de386f..d0b8e135 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -62,6 +62,7 @@ config_t *config = NULL; #define BOLDMAGENTA "\033[1;35m" #define BOLDCYAN "\033[1;36m" #define BOLDWHITE "\033[1;37m" +#define FAINTBLACK "\033[38;5;243m" void enable_colors(int colors) { @@ -76,6 +77,7 @@ void enable_colors(int colors) colstr->meta = BOLDCYAN; colstr->warn = BOLDYELLOW; colstr->err = BOLDRED; + colstr->faint = FAINTBLACK; colstr->nocolor = NOCOLOR; } else { colstr->colon = ":: "; @@ -86,6 +88,7 @@ void enable_colors(int colors) colstr->meta = ""; colstr->warn = ""; colstr->err = ""; + colstr->faint = ""; colstr->nocolor = ""; } } @@ -120,6 +123,7 @@ config_t *config_new(void) newconfig->colstr.warn = ""; newconfig->colstr.err = ""; newconfig->colstr.nocolor = ""; + newconfig->colstr.faint = ""; return newconfig; } diff --git a/src/pacman/conf.h b/src/pacman/conf.h index d954e637..e1df24b7 100644 --- a/src/pacman/conf.h +++ b/src/pacman/conf.h @@ -31,6 +31,7 @@ typedef struct __colstr_t { const char *meta; const char *warn; const char *err; + const char *faint; const char *nocolor; } colstr_t; diff --git a/src/pacman/util.c b/src/pacman/util.c index a3a85bb9..5042e0e3 100644 --- a/src/pacman/util.c +++ b/src/pacman/util.c @@ -415,12 +415,40 @@ static size_t string_length(const char *s) if(!s || s[0] == '\0') { return 0; } - /* len goes from # bytes -> # chars -> # cols */ - len = strlen(s) + 1; - wcstr = calloc(len, sizeof(wchar_t)); - len = mbstowcs(wcstr, s, len); - len = wcswidth(wcstr, len); - free(wcstr); + if(strstr(s, "\033")) { + const char* source = s; + char* replaced = malloc(sizeof(char)*strlen(s)); + int mode = 0; + int iter = 0; + for(char character = *source; character != '\0'; character = *++source) { + if(mode == 0) { + if(character == '\033') { + mode = 1; + } else { + replaced[iter] = character; + iter++; + } + } else if(mode == 1) { + if (character == 'm') { + mode = 0; + } + } + } + replaced[iter] = '\0'; + len = iter; + wcstr = calloc(len, sizeof(wchar_t)); + len = mbstowcs(wcstr, replaced, len); + len = wcswidth(wcstr, len); + free(wcstr); + free(replaced); + } else { + /* len goes from # bytes -> # chars -> # cols */ + len = strlen(s) + 1; + wcstr = calloc(len, sizeof(wchar_t)); + len = mbstowcs(wcstr, s, len); + len = wcswidth(wcstr, len); + free(wcstr); + } return len; } @@ -905,14 +933,14 @@ static void _display_targets(alpm_list_t *targets, int verbose) } if(target->install) { - pm_asprintf(&str, "%s-%s", alpm_pkg_get_name(target->install), - alpm_pkg_get_version(target->install)); + pm_asprintf(&str, "%s%s-%s%s", alpm_pkg_get_name(target->install), config->colstr.faint, + alpm_pkg_get_version(target->install), config->colstr.nocolor); } else if(isize == 0) { - pm_asprintf(&str, "%s-%s", alpm_pkg_get_name(target->remove), - alpm_pkg_get_version(target->remove)); + pm_asprintf(&str, "%s%s-%s%s", alpm_pkg_get_name(target->remove), config->colstr.faint, + alpm_pkg_get_version(target->remove), config->colstr.nocolor); } else { - pm_asprintf(&str, "%s-%s [%s]", alpm_pkg_get_name(target->remove), - alpm_pkg_get_version(target->remove), _("removal")); + pm_asprintf(&str, "%s%s-%s %s[%s]%s", alpm_pkg_get_name(target->remove), config->colstr.faint, + alpm_pkg_get_version(target->remove), config->colstr.nocolor, _("removal"), config->colstr.nocolor); } names = alpm_list_add(names, str); } -- 2.26.0

2 3

[pacman-dev] [PATCH] pacman.8: Fix typo
by Allan McRae 14 Apr '20

14 Apr '20

Signed-off-by: Allan McRae <allan(a)archlinux.org> --- doc/pacman.8.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/pacman.8.asciidoc b/doc/pacman.8.asciidoc index 68b62a90..fee5aa15 100644 --- a/doc/pacman.8.asciidoc +++ b/doc/pacman.8.asciidoc @@ -464,7 +464,7 @@ Database Options (apply to '-D')[[QO]] *\--asexplicit* <package>:: Mark a package as explicitly installed; in other words, set their install - reason to be explicitly installed. This is useful it you want to keep a + reason to be explicitly installed. This is useful if you want to keep a package installed even when it was initially installed as a dependency of another package. -- 2.25.1

1 0

[pacman-dev] Fw: Pacman support for IPFS
by ＠RubenKelevra 14 Apr '20

14 Apr '20

This message was originally send to Allan, who recommended to move the discussion to the pacman-dev mailing list: I tested a bit around and setup a mirror (via a cluster-configuration) in IPFS for the packages - but it's kind of ugly. I was wondering if there's interest from you or the team to implement native IPFS support in pacman? The idea is, to extend the current database definition, by an additional field, which holds the Content-ID necessary to fetch a package from IPFS. If there's an API-endpoint configured in the pacman config, pacman will query the IPFS API to resolve a IPNS-public key[0]. The public key will lead to a folder listing, which holds the latest databases, like this path[1] does. The Content-ID of each database could be saved as symbolic link in the db-patch. If one of the Content-IDs changed, the new version is fetched. With the Content-IDs the updated packages can be fetched from the network. So if there's no API-endpoint configured, the feature wont be active. If it's been activated, pacman can fetch packages much quicker, since IPFS can fetch from multiple sources, like BitTorrent and will also connect to other peers on the local network via mDNS. That's especially interesting if you run a large amount of computers and currently need a rsync mirror server or a cache in the network or you're a home user with extremely limited bandwidth. --- To maintain a reliably redundancy of packages there's a need for a cluster installation. A IPFS cluster[2] is basically just a small daemon which runs and holds a database which Content-IDs should be maintained on which cluster peer. So if a package gets added to the database, the Content-ID would be added to the dataset of the cluster and the Content-ID of the old package get's removed. And the new version of the folder with the databases would be replaced as well. The cluster definition isn't static, everybody can join the cluster and can hold the cluster data. There are some examples of clusters which are open for copies here[3]. You can set on each element how many copies the cluster should hold, at maximum and minimum. The cluster will start to replicate the maximum amount and if the redundancy drops below minimum additional copies will be made. You can also specify -1 which means every cluster member will hold a copy - this makes sense for example for the databases. --- How to add new package versions: A package maintainer would do the normal sha256/md5/signing and additionally add the file to a IPFS client (for example on a server) or his computer. The Content-ID generated while adding the file to IPFS. The content ID will be added to the database definition. The new database file is then added to to IPFS as well, and the folder of the current database listing (in IPFS) is altered to reference the new Content-ID of the database file. This will alter the Content-ID of the folder, since the references has changed. The new Content-ID of the folder and the package then get added to the cluster, which will fetch both new files from the computer of the maintainer or the server where they was added. Now the IPNS-Record can be switched to the new Content-ID of the folder, any everyone can fetch the new database. [0] https://docs-beta.ipfs.io/concepts/ipns/ [1] https://ipfs.io/ipns/pkg.pacman.store/arch/x86_64/default/db [2] https://cluster.ipfs.io/ [3] https://collab.ipfscluster.io/ Best regards, Ruben

4 6

[pacman-dev] [PATCH v3] Implement multiplexed download using mCURL
by Anatol Pomozov 14 Apr '20

14 Apr '20

curl_multi_download_internal() is the main loop that creates up to 'ParallelDownloads' easy curl handles, adds them to mcurl and then performs curl execution. This is when the paralled downloads happens. Once any of the downloads complete the function checks its result. In case if the download fails it initiates retry with the next server from payload->servers list. At the download completion all the payload resources are cleaned up. curl_multi_handle_single_done() is essentially refactored version of curl_download_internal() adopted for multi_curl. Once mcurl porting is complete curl_download_internal() will be removed. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/dload.c | 360 +++++++++++++++++++++++++++++++++++++++++++- lib/libalpm/dload.h | 2 + 2 files changed, 358 insertions(+), 4 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index a413792a..d117a26d 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -270,6 +270,7 @@ static void curl_set_handle_opts(struct dload_payload *payload, curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, 60L); curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 60L); curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY); + curl_easy_setopt(curl, CURLOPT_PRIVATE, (void *)payload); _alpm_log(handle, ALPM_LOG_DEBUG, "url: %s\n", payload->fileurl); @@ -599,15 +600,366 @@ cleanup: return ret; } +/* Return 0 if retry was successful, -1 otherwise */ +static int curl_multi_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload *payload) +{ + const char *server; + size_t len; + alpm_handle_t *handle = payload->handle; + + payload->servers = payload->servers->next; + if(!payload->servers) { + return -1; + } + server = payload->servers->data; + + /* regenerate a new fileurl */ + free(payload->fileurl); + len = strlen(server) + strlen(payload->filepath) + 2; + MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); + snprintf(payload->fileurl, len, "%s/%s", server, payload->filepath); + + if(payload->unlink_on_fail) { + /* we keep the file for a new retry but remove its data if any */ + fflush(payload->localf); + ftruncate(fileno(payload->localf), 0); + fseek(payload->localf, 0, SEEK_SET); + } + + /* Set curl with the new URL */ + curl_easy_setopt(curl, CURLOPT_URL, payload->fileurl); + + curl_multi_remove_handle(curlm, curl); + curl_multi_add_handle(curlm, curl); + + return 0; +} + +/* Returns 2 if download retry happened + * Returns 1 if the file is up-to-date + * Returns 0 if current payload is completed successfully + * Returns -1 if an error happened + */ +static int curl_multi_check_finished_download(CURLM *curlm, CURLMsg *msg, + const char *localpath) +{ + alpm_handle_t *handle = NULL; + struct dload_payload *payload = NULL; + CURL *curl = msg->easy_handle; + CURLcode curlerr; + char *effective_url; + long timecond; + double remote_size, bytes_dl = 0; + long remote_time = -1; + struct stat st; + char hostname[HOSTNAME_SIZE]; + int ret = -1; + + curlerr = curl_easy_getinfo(curl, CURLINFO_PRIVATE, &payload); + ASSERT(curlerr == CURLE_OK, RET_ERR(handle, ALPM_ERR_LIBCURL, -1)); + handle = payload->handle; + + curl_gethost(payload->fileurl, hostname, sizeof(hostname)); + curlerr = msg->data.result; + _alpm_log(handle, ALPM_LOG_DEBUG, "curl returned result %d from transfer\n", + curlerr); + + /* was it a success? */ + switch(curlerr) { + case CURLE_OK: + /* get http/ftp response code */ + _alpm_log(handle, ALPM_LOG_DEBUG, "response code: %ld\n", payload->respcode); + if(payload->respcode >= 400) { + payload->unlink_on_fail = 1; + if(!payload->errors_ok) { + handle->pm_errno = ALPM_ERR_RETRIEVE; + /* non-translated message is same as libcurl */ + snprintf(payload->error_buffer, sizeof(payload->error_buffer), + "The requested URL returned error: %ld", payload->respcode); + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed retrieving file '%s' from %s : %s\n"), + payload->remote_name, hostname, payload->error_buffer); + } + if(curl_multi_retry_next_server(curlm, curl, payload) == 0) { + return 2; + } else { + goto cleanup; + } + } + break; + case CURLE_ABORTED_BY_CALLBACK: + /* handle the interrupt accordingly */ + if(dload_interrupted == ABORT_OVER_MAXFILESIZE) { + curlerr = CURLE_FILESIZE_EXCEEDED; + payload->unlink_on_fail = 1; + handle->pm_errno = ALPM_ERR_LIBCURL; + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed retrieving file '%s' from %s : expected download size exceeded\n"), + payload->remote_name, hostname); + } + goto cleanup; + case CURLE_COULDNT_RESOLVE_HOST: + payload->unlink_on_fail = 1; + handle->pm_errno = ALPM_ERR_SERVER_BAD_URL; + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed retrieving file '%s' from %s : %s\n"), + payload->remote_name, hostname, payload->error_buffer); + if(curl_multi_retry_next_server(curlm, curl, payload) == 0) { + return 2; + } else { + goto cleanup; + } + default: + /* delete zero length downloads */ + if(fstat(fileno(payload->localf), &st) == 0 && st.st_size == 0) { + payload->unlink_on_fail = 1; + } + if(!payload->errors_ok) { + handle->pm_errno = ALPM_ERR_LIBCURL; + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed retrieving file '%s' from %s : %s\n"), + payload->remote_name, hostname, payload->error_buffer); + } else { + _alpm_log(handle, ALPM_LOG_DEBUG, + "failed retrieving file '%s' from %s : %s\n", + payload->remote_name, hostname, payload->error_buffer); + } + if(curl_multi_retry_next_server(curlm, curl, payload) == 0) { + return 2; + } else { + goto cleanup; + } + } + + /* retrieve info about the state of the transfer */ + curl_easy_getinfo(curl, CURLINFO_FILETIME, &remote_time); + curl_easy_getinfo(curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &remote_size); + curl_easy_getinfo(curl, CURLINFO_SIZE_DOWNLOAD, &bytes_dl); + curl_easy_getinfo(curl, CURLINFO_CONDITION_UNMET, &timecond); + curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &effective_url); + + /* time condition was met and we didn't download anything. we need to + * clean up the 0 byte .part file that's left behind. */ + if(timecond == 1 && DOUBLE_EQ(bytes_dl, 0)) { + _alpm_log(handle, ALPM_LOG_DEBUG, "file met time condition\n"); + ret = 1; + unlink(payload->tempfile_name); + goto cleanup; + } + + /* remote_size isn't necessarily the full size of the file, just what the + * server reported as remaining to download. compare it to what curl reported + * as actually being transferred during curl_easy_perform() */ + if(!DOUBLE_EQ(remote_size, -1) && !DOUBLE_EQ(bytes_dl, -1) && + !DOUBLE_EQ(bytes_dl, remote_size)) { + _alpm_log(handle, ALPM_LOG_ERROR, _("%s appears to be truncated: %jd/%jd bytes\n"), + payload->remote_name, (intmax_t)bytes_dl, (intmax_t)remote_size); + GOTO_ERR(handle, ALPM_ERR_RETRIEVE, cleanup); + } + + if(payload->trust_remote_name) { + if(payload->content_disp_name) { + /* content-disposition header has a better name for our file */ + free(payload->destfile_name); + payload->destfile_name = get_fullpath(localpath, + get_filename(payload->content_disp_name), ""); + } else { + const char *effective_filename = strrchr(effective_url, '/'); + if(effective_filename && strlen(effective_filename) > 2) { + effective_filename++; + + /* if destfile was never set, we wrote to a tempfile. even if destfile is + * set, we may have followed some redirects and the effective url may + * have a better suggestion as to what to name our file. in either case, + * refactor destfile to this newly derived name. */ + if(!payload->destfile_name || strcmp(effective_filename, + strrchr(payload->destfile_name, '/') + 1) != 0) { + free(payload->destfile_name); + payload->destfile_name = get_fullpath(localpath, effective_filename, ""); + } + } + } + } + + ret = 0; + +cleanup: + /* disconnect relationships from the curl handle for things that might go out + * of scope, but could still be touched on connection teardown. This really + * only applies to FTP transfers. */ + curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L); + curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, (char *)NULL); + + if(payload->localf != NULL) { + fclose(payload->localf); + utimes_long(payload->tempfile_name, remote_time); + } + + if(ret == 0) { + if(payload->destfile_name) { + if(rename(payload->tempfile_name, payload->destfile_name)) { + _alpm_log(handle, ALPM_LOG_ERROR, _("could not rename %s to %s (%s)\n"), + payload->tempfile_name, payload->destfile_name, strerror(errno)); + ret = -1; + } + } + } + + if((ret == -1 || dload_interrupted) && payload->unlink_on_fail && + payload->tempfile_name) { + unlink(payload->tempfile_name); + } + + // TODO: report that the download has been completed + + curl_multi_remove_handle(curlm, curl); + curl_easy_cleanup(curl); + payload->curl = NULL; + + FREE(payload->fileurl); + return ret; +} + +/* Returns 0 in case if a new download transaction has been successfully started + * Returns -1 if am error happened while starting a new download + */ +static int curl_multi_add_payload(alpm_handle_t *handle, CURLM *curlm, + struct dload_payload *payload, const char *localpath) +{ + size_t len; + const char *server; + CURL *curl = NULL; + char hostname[HOSTNAME_SIZE]; + + ASSERT(payload->servers, RET_ERR(handle, ALPM_ERR_SERVER_NONE, -1)); + server = payload->servers->data; + + curl = curl_easy_init(); + payload->curl = curl; + + len = strlen(server) + strlen(payload->filepath) + 2; + MALLOC(payload->fileurl, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(payload->fileurl, len, "%s/%s", server, payload->filepath); + + payload->tempfile_openmode = "wb"; + if(!payload->remote_name) { + STRDUP(payload->remote_name, get_filename(payload->fileurl), + GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + } + if(curl_gethost(payload->fileurl, hostname, sizeof(hostname)) != 0) { + _alpm_log(handle, ALPM_LOG_ERROR, _("url '%s' is invalid\n"), payload->fileurl); + GOTO_ERR(handle, ALPM_ERR_SERVER_BAD_URL, cleanup); + } + + if(payload->remote_name && strlen(payload->remote_name) > 0) { + payload->destfile_name = get_fullpath(localpath, payload->remote_name, ""); + payload->tempfile_name = get_fullpath(localpath, payload->remote_name, ".part"); + if(!payload->destfile_name || !payload->tempfile_name) { + goto cleanup; + } + } else { + /* URL doesn't contain a filename, so make a tempfile. We can't support + * resuming this kind of download; partial transfers will be destroyed */ + payload->unlink_on_fail = 1; + + payload->localf = create_tempfile(payload, localpath); + if(payload->localf == NULL) { + goto cleanup; + } + } + + curl_set_handle_opts(payload, curl, payload->error_buffer); + + if(payload->max_size == payload->initial_size) { + /* .part file is complete */ + goto cleanup; + } + + if(payload->localf == NULL) { + payload->localf = fopen(payload->tempfile_name, payload->tempfile_openmode); + if(payload->localf == NULL) { + _alpm_log(handle, ALPM_LOG_ERROR, + _("could not open file %s: %s\n"), + payload->tempfile_name, strerror(errno)); + GOTO_ERR(handle, ALPM_ERR_RETRIEVE, cleanup); + } + } + + _alpm_log(handle, ALPM_LOG_DEBUG, + "opened tempfile for download: %s (%s)\n", payload->tempfile_name, + payload->tempfile_openmode); + + curl_easy_setopt(curl, CURLOPT_WRITEDATA, payload->localf); + curl_multi_add_handle(curlm, curl); + return 0; + +cleanup: + FREE(payload->fileurl); + FREE(payload->tempfile_name); + FREE(payload->destfile_name); + FREE(payload->content_disp_name); + curl_easy_cleanup(curl); + return -1; +} + static int curl_multi_download_internal(alpm_handle_t *handle, alpm_list_t *payloads /* struct dload_payload */, const char *localpath) { - (void)handle; - (void)payloads; - (void)localpath; - return 0; + int still_running = 0; + int err = 0; + int parallel_downloads = handle->parallel_downloads; + + CURLM *curlm = handle->curlm; + CURLMsg *msg; + + while(still_running || payloads) { + int msgs_left = -1; + + for(; still_running < parallel_downloads && payloads; still_running++) { + struct dload_payload *payload = payloads->data; + + if(curl_multi_add_payload(handle, curlm, payloads->data, localpath) == 0) { + // TODO: report that download has started + payloads = payloads->next; + } else { + // the payload failed to start, do not start any new downloads just wait until + // active one complete. + _alpm_log(handle, ALPM_LOG_ERROR, "failed to setup a download payload for %s\n", payload->remote_name); + payloads = NULL; + err = -1; + } + } + + CURLMcode mc = curl_multi_perform(curlm, &still_running); + + if(mc != CURLM_OK) { + _alpm_log(handle, ALPM_LOG_ERROR, "curl returned error %d from transfer\n", mc); + payloads = NULL; + err = -1; + } + + while((msg = curl_multi_info_read(curlm, &msgs_left))) { + if(msg->msg == CURLMSG_DONE) { + int done_code = curl_multi_check_finished_download(curlm, msg, localpath); + if(done_code == 2) { + /* in case of a retry increase the counter of active requests + * to avoid exiting the loop early + */ + still_running++; + } + } else { + _alpm_log(handle, ALPM_LOG_ERROR, "curl curl_multi_info_read error %d\n", msg->msg); + } + } + if(still_running) { + curl_multi_wait(curlm, NULL, 0, 1000, NULL); + } + } + + return err; } + #endif /** Download a file given by a URL to a local directory. diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index e87b6a93..a40b51b7 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -45,6 +45,8 @@ struct dload_payload { int cb_initialized; #ifdef HAVE_LIBCURL CURL *curl; + char error_buffer[CURL_ERROR_SIZE]; + FILE *localf; /* temp download file */ #endif }; -- 2.26.0

2 2

[pacman-dev] [PATCH v2] Implement _alpm_multi_download
by Anatol Pomozov 14 Apr '20

14 Apr '20

It is an equivalent of _alpm_download but accepts a list of payloads. curl_multi_download_internal() is a stub at this moment and will be implemented in the later commits of this patch series. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/dload.c | 50 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 4c81b11f..a413792a 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -598,6 +598,16 @@ cleanup: return ret; } + +static int curl_multi_download_internal(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath) +{ + (void)handle; + (void)payloads; + (void)localpath; + return 0; +} #endif /** Download a file given by a URL to a local directory. @@ -634,10 +644,42 @@ int _alpm_multi_download(alpm_handle_t *handle, alpm_list_t *payloads /* struct dload_payload */, const char *localpath) { - (void)handle; - (void)payloads; - (void)localpath; - return 0; + if(handle->fetchcb == NULL) { +#ifdef HAVE_LIBCURL + return curl_multi_download_internal(handle, payloads, localpath); +#else + RET_ERR(handle, ALPM_ERR_EXTERNAL_DOWNLOAD, -1); +#endif + } else { + alpm_list_t *p; + for(p = payloads; p; p = p->next) { + struct dload_payload *payload = p->data; + alpm_list_t *s; + int success = 0; + + for(s = payload->servers; s; s = s->next) { + const char *server = s->data; + char *fileurl; + int ret; + + size_t len = strlen(server) + strlen(payload->filepath) + 2; + MALLOC(fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); + snprintf(fileurl, len, "%s/%s", server, payload->filepath); + + ret = handle->fetchcb(fileurl, localpath, payload->force); + free(fileurl); + + if (ret != -1) { + success = 1; + break; + } + } + if(!success && !payload->errors_ok) { + RET_ERR(handle, ALPM_ERR_EXTERNAL_DOWNLOAD, -1); + } + } + return 0; + } } static char *filecache_find_url(alpm_handle_t *handle, const char *url) -- 2.26.0

2 1

[pacman-dev] [PATCH] Implement multiplexed download using mCURL
by Anatol Pomozov 14 Apr '20

14 Apr '20

curl_multi_download_internal() is the main loop that creates up to 'ParallelDownloads' easy curl handles, adds them to mcurl and then performs curl execution. This is when the paralled downloads heppen. Once any of the downloads complete the function checks its result. In case if the download fails it initiates retry with the next server from payload->servers list. At the download completion all the payload resources are cleaned up. curl_multi_handle_single_done() is essentially refactored version of curl_download_internal() adopted for multi_curl. Once mcurl porting is complete curl_download_internal() will be removed. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/dload.c | 361 +++++++++++++++++++++++++++++++++++++++++++- lib/libalpm/dload.h | 2 + 2 files changed, 359 insertions(+), 4 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 3570e234..06bce330 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -18,6 +18,7 @@ * along with this program. If not, see <http://www.gnu.org/licenses/>. */ +#include <assert.h> #include <stdlib.h> #include <stdio.h> #include <errno.h> @@ -271,6 +272,8 @@ static void curl_set_handle_opts(struct dload_payload *payload, curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, 60L); curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY); + curl_easy_setopt(curl, CURLOPT_PRIVATE, (void *)payload); + _alpm_log(handle, ALPM_LOG_DEBUG, "url: %s\n", payload->fileurl); if(payload->max_size) { @@ -601,15 +604,365 @@ cleanup: return ret; } +/* Return 0 if retry was sucessfull, -1 otherwise */ +static int curl_multi_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload *payload) +{ + const char *server; + size_t len; + alpm_handle_t *handle = payload->handle; + + payload->servers = payload->servers->next; + if(!payload->servers) { + return -1; + } + server = payload->servers->data; + + /* regenerate a new fileurl */ + free(payload->fileurl); + len = strlen(server) + strlen(payload->filepath) + 2; + MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); + snprintf(payload->fileurl, len, "%s/%s", server, payload->filepath); + + if(payload->unlink_on_fail) { + /* we keep the file for a new retry but remove its data if any */ + fflush(payload->localf); + ftruncate(fileno(payload->localf), 0); + fseek(payload->localf, 0, SEEK_SET); + } + + /* Set curl with the new URL */ + curl_easy_setopt(curl, CURLOPT_URL, payload->fileurl); + + curl_multi_remove_handle(curlm, curl); + curl_multi_add_handle(curlm, curl); + + return 0; +} + +/* Returns 2 if download retry happened + * Returns 1 if the file is up-to-date + * Returns 0 if current payload is completed sucessfully + * Returns -1 if an error happened + */ +static int curl_multi_handle_single_done(CURLM *curlm, CURLMsg *msg, const char *localpath) +{ + alpm_handle_t *handle = NULL; + struct dload_payload *payload = NULL; + CURL *curl = msg->easy_handle; + CURLcode curlerr; + char *effective_url; + long timecond; + double remote_size, bytes_dl = 0; + long remote_time = -1; + struct stat st; + char hostname[HOSTNAME_SIZE]; + int ret = -1; + + curlerr = curl_easy_getinfo(curl, CURLINFO_PRIVATE, &payload); + assert(curlerr == CURLE_OK); + handle = payload->handle; + + curl_gethost(payload->fileurl, hostname, sizeof(hostname)); + curlerr = msg->data.result; + _alpm_log(handle, ALPM_LOG_DEBUG, "curl returned result %d from transfer\n", + curlerr); + + /* was it a success? */ + switch(curlerr) { + case CURLE_OK: + /* get http/ftp response code */ + _alpm_log(handle, ALPM_LOG_DEBUG, "response code: %ld\n", payload->respcode); + if(payload->respcode >= 400) { + payload->unlink_on_fail = 1; + if(!payload->errors_ok) { + handle->pm_errno = ALPM_ERR_RETRIEVE; + /* non-translated message is same as libcurl */ + snprintf(payload->error_buffer, sizeof(payload->error_buffer), + "The requested URL returned error: %ld", payload->respcode); + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed retrieving file '%s' from %s : %s\n"), + payload->remote_name, hostname, payload->error_buffer); + } + if(curl_multi_retry_next_server(curlm, curl, payload) == 0) { + return 2; + } else { + goto cleanup; + } + } + break; + case CURLE_ABORTED_BY_CALLBACK: + /* handle the interrupt accordingly */ + if(dload_interrupted == ABORT_OVER_MAXFILESIZE) { + curlerr = CURLE_FILESIZE_EXCEEDED; + payload->unlink_on_fail = 1; + handle->pm_errno = ALPM_ERR_LIBCURL; + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed retrieving file '%s' from %s : expected download size exceeded\n"), + payload->remote_name, hostname); + } + goto cleanup; + case CURLE_COULDNT_RESOLVE_HOST: + payload->unlink_on_fail = 1; + handle->pm_errno = ALPM_ERR_SERVER_BAD_URL; + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed retrieving file '%s' from %s : %s\n"), + payload->remote_name, hostname, payload->error_buffer); + if(curl_multi_retry_next_server(curlm, curl, payload) == 0) { + return 2; + } else { + goto cleanup; + } + default: + /* delete zero length downloads */ + if(fstat(fileno(payload->localf), &st) == 0 && st.st_size == 0) { + payload->unlink_on_fail = 1; + } + if(!payload->errors_ok) { + handle->pm_errno = ALPM_ERR_LIBCURL; + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed retrieving file '%s' from %s : %s\n"), + payload->remote_name, hostname, payload->error_buffer); + } else { + _alpm_log(handle, ALPM_LOG_DEBUG, + "failed retrieving file '%s' from %s : %s\n", + payload->remote_name, hostname, payload->error_buffer); + } + if(curl_multi_retry_next_server(curlm, curl, payload) == 0) { + return 2; + } else { + goto cleanup; + } + } + + /* retrieve info about the state of the transfer */ + curl_easy_getinfo(curl, CURLINFO_FILETIME, &remote_time); + curl_easy_getinfo(curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &remote_size); + curl_easy_getinfo(curl, CURLINFO_SIZE_DOWNLOAD, &bytes_dl); + curl_easy_getinfo(curl, CURLINFO_CONDITION_UNMET, &timecond); + curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &effective_url); + + /* time condition was met and we didn't download anything. we need to + * clean up the 0 byte .part file that's left behind. */ + if(timecond == 1 && DOUBLE_EQ(bytes_dl, 0)) { + _alpm_log(handle, ALPM_LOG_DEBUG, "file met time condition\n"); + ret = 1; + unlink(payload->tempfile_name); + goto cleanup; + } + + /* remote_size isn't necessarily the full size of the file, just what the + * server reported as remaining to download. compare it to what curl reported + * as actually being transferred during curl_easy_perform() */ + if(!DOUBLE_EQ(remote_size, -1) && !DOUBLE_EQ(bytes_dl, -1) && + !DOUBLE_EQ(bytes_dl, remote_size)) { + _alpm_log(handle, ALPM_LOG_ERROR, _("%s appears to be truncated: %jd/%jd bytes\n"), + payload->remote_name, (intmax_t)bytes_dl, (intmax_t)remote_size); + GOTO_ERR(handle, ALPM_ERR_RETRIEVE, cleanup); + } + + if(payload->trust_remote_name) { + if(payload->content_disp_name) { + /* content-disposition header has a better name for our file */ + free(payload->destfile_name); + payload->destfile_name = get_fullpath(localpath, + get_filename(payload->content_disp_name), ""); + } else { + const char *effective_filename = strrchr(effective_url, '/'); + if(effective_filename && strlen(effective_filename) > 2) { + effective_filename++; + + /* if destfile was never set, we wrote to a tempfile. even if destfile is + * set, we may have followed some redirects and the effective url may + * have a better suggestion as to what to name our file. in either case, + * refactor destfile to this newly derived name. */ + if(!payload->destfile_name || strcmp(effective_filename, + strrchr(payload->destfile_name, '/') + 1) != 0) { + free(payload->destfile_name); + payload->destfile_name = get_fullpath(localpath, effective_filename, ""); + } + } + } + } + + ret = 0; + +cleanup: + /* disconnect relationships from the curl handle for things that might go out + * of scope, but could still be touched on connection teardown. This really + * only applies to FTP transfers. */ + curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L); + curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, (char *)NULL); + + if(payload->localf != NULL) { + fclose(payload->localf); + utimes_long(payload->tempfile_name, remote_time); + } + + if(ret == 0) { + if(payload->destfile_name) { + if(rename(payload->tempfile_name, payload->destfile_name)) { + _alpm_log(handle, ALPM_LOG_ERROR, _("could not rename %s to %s (%s)\n"), + payload->tempfile_name, payload->destfile_name, strerror(errno)); + ret = -1; + } + } + } + + if((ret == -1 || dload_interrupted) && payload->unlink_on_fail && + payload->tempfile_name) { + unlink(payload->tempfile_name); + } + + // TODO: report that the download has been completed + + curl_multi_remove_handle(curlm, curl); + curl_easy_cleanup(curl); + payload->curl = NULL; + + FREE(payload->fileurl); + return ret; +} + +/* Returns 0 in case if a new download transaction has been successfully started + * Returns -1 if am error happened while starting a new download + */ +static int curl_multi_add_payload(alpm_handle_t *handle, CURLM *curlm, + struct dload_payload *payload, const char *localpath) +{ + size_t len; + const char *server; + CURL *curl = NULL; + char hostname[HOSTNAME_SIZE]; + + ASSERT(payload->servers, RET_ERR(handle, ALPM_ERR_SERVER_NONE, -1)); + server = payload->servers->data; + + curl = curl_easy_init(); + payload->curl = curl; + + len = strlen(server) + strlen(payload->filepath) + 2; + MALLOC(payload->fileurl, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(payload->fileurl, len, "%s/%s", server, payload->filepath); + + payload->tempfile_openmode = "wb"; + if(!payload->remote_name) { + STRDUP(payload->remote_name, get_filename(payload->fileurl), + GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + } + if(curl_gethost(payload->fileurl, hostname, sizeof(hostname)) != 0) { + _alpm_log(handle, ALPM_LOG_ERROR, _("url '%s' is invalid\n"), payload->fileurl); + GOTO_ERR(handle, ALPM_ERR_SERVER_BAD_URL, cleanup); + } + + if(payload->remote_name && strlen(payload->remote_name) > 0) { + payload->destfile_name = get_fullpath(localpath, payload->remote_name, ""); + payload->tempfile_name = get_fullpath(localpath, payload->remote_name, ".part"); + if(!payload->destfile_name || !payload->tempfile_name) { + goto cleanup; + } + } else { + /* URL doesn't contain a filename, so make a tempfile. We can't support + * resuming this kind of download; partial transfers will be destroyed */ + payload->unlink_on_fail = 1; + + payload->localf = create_tempfile(payload, localpath); + if(payload->localf == NULL) { + goto cleanup; + } + } + + curl_set_handle_opts(payload, curl, payload->error_buffer); + + if(payload->max_size == payload->initial_size) { + /* .part file is complete */ + goto cleanup; + } + + if(payload->localf == NULL) { + payload->localf = fopen(payload->tempfile_name, payload->tempfile_openmode); + if(payload->localf == NULL) { + _alpm_log(handle, ALPM_LOG_ERROR, + _("could not open file %s: %s\n"), + payload->tempfile_name, strerror(errno)); + GOTO_ERR(handle, ALPM_ERR_RETRIEVE, cleanup); + } + } + + _alpm_log(handle, ALPM_LOG_DEBUG, + "opened tempfile for download: %s (%s)\n", payload->tempfile_name, + payload->tempfile_openmode); + + curl_easy_setopt(curl, CURLOPT_WRITEDATA, payload->localf); + curl_multi_add_handle(curlm, curl); + return 0; + +cleanup: + FREE(payload->fileurl); + FREE(payload->tempfile_name); + FREE(payload->destfile_name); + FREE(payload->content_disp_name); + curl_easy_cleanup(curl); + return -1; +} + static int curl_multi_download_internal(alpm_handle_t *handle, alpm_list_t *payloads /* struct dload_payload */, const char *localpath) { - (void)handle; - (void)payloads; - (void)localpath; - return 0; + int still_running = 0; + int err = 0; + int parallel_downloads = handle->parallel_downloads; + + CURLM *curlm = handle->curlm; + CURLMsg *msg; + + while(still_running || payloads) { + int msgs_left = -1; + + for(; still_running < parallel_downloads && payloads; still_running++) { + struct dload_payload *payload = payloads->data; + + if(curl_multi_add_payload(handle, curlm, payloads->data, localpath) == 0) { + // TODO: report that download has started + payloads = payloads->next; + } else { + // the payload failed to start, do not start any new downloads just wait until + // active one complete. + _alpm_log(handle, ALPM_LOG_ERROR, "failed to setup a download payload for %s\n", payload->remote_name); + payloads = NULL; + err = -1; + } + } + + CURLMcode mc = curl_multi_perform(curlm, &still_running); + + if(mc != CURLM_OK) { + _alpm_log(handle, ALPM_LOG_ERROR, "curl returned error %d from transfer\n", mc); + payloads = NULL; + err = -1; + } + + while((msg = curl_multi_info_read(curlm, &msgs_left))) { + if(msg->msg == CURLMSG_DONE) { + int done_code = curl_multi_handle_single_done(curlm, msg, localpath); + if(done_code == 2) { + /* in case of a retry increase the counter of active requests + * to avoid exiting the loop early + */ + still_running++; + } + } else { + _alpm_log(handle, ALPM_LOG_ERROR, "curl curl_multi_info_read error %d\n", msg->msg); + } + } + if(still_running) { + curl_multi_wait(curlm, NULL, 0, 1000, NULL); + } + } + + return err; } + #endif /** Download a file given by a URL to a local directory. diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index e87b6a93..a40b51b7 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -45,6 +45,8 @@ struct dload_payload { int cb_initialized; #ifdef HAVE_LIBCURL CURL *curl; + char error_buffer[CURL_ERROR_SIZE]; + FILE *localf; /* temp download file */ #endif }; -- 2.25.1

2 5

Re: [pacman-dev] [PATCH] Implement _alpm_multi_download
by Anatol Pomozov 14 Apr '20

14 Apr '20

Hello On Mon, Mar 16, 2020 at 7:00 AM Allan McRae <allan(a)archlinux.org> wrote: > > On 10/3/20 6:19 am, Anatol Pomozov wrote: > > It is equivalent of _alpm_download but accepts list of payloads. > > > > Expand the commit message to mention curl_multi_download_internal is > just not functional at the moment. Done. > > Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> > > --- > > lib/libalpm/dload.c | 50 +++++++++++++++++++++++++++++++++++++++++---- > > 1 file changed, 46 insertions(+), 4 deletions(-) > > > > diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c > > index 7cd3e3a4..bec3ff5e 100644 > > --- a/lib/libalpm/dload.c > > +++ b/lib/libalpm/dload.c > > @@ -604,6 +604,16 @@ cleanup: > > > > return ret; > > } > > + > > +static int curl_multi_download_internal(alpm_handle_t *handle, > > + alpm_list_t *payloads /* struct dload_payload */, > > + const char *localpath) > > +{ > > + (void)handle; > > + (void)payloads; > > + (void)localpath; > > + return 0; > > +} > > #endif > > > > /** Download a file given by a URL to a local directory. > > @@ -640,10 +650,42 @@ int _alpm_multi_download(alpm_handle_t *handle, > > alpm_list_t *payloads /* struct dload_payload */, > > const char *localpath) > > { > > - (void)handle; > > - (void)payloads; > > - (void)localpath; > > - return 0; > > + if(handle->fetchcb == NULL) { > > +#ifdef HAVE_LIBCURL > > + return curl_multi_download_internal(handle, payloads, localpath); > > +#else > > You need some (void)foo to get rid of unused variable warnings when > building without libcurl. > > (void)payloads; > (void)localpath; These 2 variables used below in the else{} branch. Thus there should not be any "unsed variable" warnings. > > > + RET_ERR(handle, ALPM_ERR_EXTERNAL_DOWNLOAD, -1); > > +#endif > > + } else { > > + alpm_list_t *p; > > + for(p = payloads; p; p = p->next) { > > + struct dload_payload *payload = p->data; > > + alpm_list_t *s; > > + int success = 0; > > + > > + for(s = payload->servers; s; s = s->next) { > > + const char *server = s->data; > > + char *fileurl; > > + int ret; > > + > > + size_t len = strlen(server) + strlen(payload->filepath) + 2; > > + MALLOC(fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); > > + snprintf(fileurl, len, "%s/%s", server, payload->filepath); > > + > > + ret = handle->fetchcb(fileurl, localpath, payload->force); > > + free(fileurl); > > + > > + if (ret != -1) { > > + success = 1; > > + break; > > + } > > + } > > + if(!success && !payload->errors_ok) { > > + RET_ERR(handle, ALPM_ERR_EXTERNAL_DOWNLOAD, -1); > > + } > > + } > > + return 0; > > + } > > } > > > > static char *filecache_find_url(alpm_handle_t *handle, const char *url) > >

1 0

[pacman-dev] [PATCH v2] Introduce alpm_dbs_update() function for parallel db updates
by Anatol Pomozov 14 Apr '20

14 Apr '20

This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/alpm.h | 29 ++++++++++ lib/libalpm/be_sync.c | 128 ++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/dload.c | 12 ++++ lib/libalpm/dload.h | 5 ++ 4 files changed, 174 insertions(+) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 5d559db1..c985429d 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -1049,6 +1049,35 @@ int alpm_db_remove_server(alpm_db_t *db, const char *url); */ int alpm_db_update(int force, alpm_db_t *db); +/** Update package databases + * + * An update of the package databases \a dbs will be attempted. Unless + * \a force is true, the update will only be performed if the remote + * databases were modified since the last update. + * + * This operation requires a database lock, and will return an applicable error + * if the lock could not be obtained. + * + * Example: + * @code + * alpm_list_t *dbs = alpm_get_syncdbs(); + * ret = alpm_dbs_update(config->handle, dbs, force); + * if(ret < 0) { + * pm_printf(ALPM_LOG_ERROR, _("failed to synchronize all databases (%s)\n"), + * alpm_strerror(alpm_errno(config->handle))); + * } + * @endcode + * + * @note After a successful update, the \link alpm_db_get_pkgcache() + * package cache \endlink will be invalidated + * @param handle the context handle + * @param dbs list of package databases to update + * @param force if true, then forces the update, otherwise update only in case + * the databases aren't up to date + * @return 0 on success, -1 on error (pm_errno is set accordingly) + */ +int alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force); + /** Get a package entry from a package database. * @param db pointer to the package database to get the package from * @param name of the package diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index aafed15d..497e1054 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -301,6 +301,134 @@ int SYMEXPORT alpm_db_update(int force, alpm_db_t *db) return ret; } +int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force) { + char *syncpath; + const char *dbext = handle->dbext; + alpm_list_t *i; + int ret = -1; + mode_t oldmask; + alpm_list_t *payloads = NULL; + + /* Sanity checks */ + ASSERT(dbs != NULL, return -1); + handle->pm_errno = ALPM_ERR_OK; + + syncpath = get_sync_dir(handle); + ASSERT(syncpath != NULL, return -1); + + /* make sure we have a sane umask */ + oldmask = umask(0022); + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + int dbforce = force; + struct dload_payload *payload = NULL; + size_t len; + int siglevel; + + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + ASSERT(db != handle->db_local, RET_ERR(handle, ALPM_ERR_WRONG_ARGS, -1)); + ASSERT(db->servers != NULL, RET_ERR(handle, ALPM_ERR_SERVER_NONE, -1)); + + /* force update of invalid databases to fix potential mismatched database/signature */ + if(db->status & DB_STATUS_INVALID) { + dbforce = 1; + } + + CALLOC(payload, 1, sizeof(*payload), RET_ERR(handle, ALPM_ERR_MEMORY, -1)); + + /* set hard upper limit of 128MiB */ + payload->max_size = 128 * 1024 * 1024; + ASSERT(db->servers != NULL, RET_ERR(handle, ALPM_ERR_SERVER_NONE, -1)); + payload->servers = db->servers; + + /* print server + filename into a buffer */ + len = strlen(db->treename) + strlen(dbext) + 1; + MALLOC(payload->filepath, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(payload->filepath, len, "%s%s", db->treename, dbext); + payload->handle = handle; + payload->force = dbforce; + payload->unlink_on_fail = 1; + + payloads = alpm_list_add(payloads, payload); + + siglevel = alpm_db_get_siglevel(db); + if(siglevel & ALPM_SIG_DATABASE) { + struct dload_payload *sig_payload; + CALLOC(sig_payload, 1, sizeof(*sig_payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + + /* print filename into a buffer (leave space for separator and .sig) */ + len = strlen(db->treename) + strlen(dbext) + 5; + MALLOC(sig_payload->filepath, len, goto cleanup); + snprintf(sig_payload->filepath, len, "%s%s.sig", db->treename, dbext); + + sig_payload->handle = handle; + sig_payload->force = dbforce; + sig_payload->errors_ok = (siglevel & ALPM_SIG_DATABASE_OPTIONAL); + + /* set hard upper limit of 16KiB */ + sig_payload->max_size = 16 * 1024; + sig_payload->servers = db->servers; + + payloads = alpm_list_add(payloads, sig_payload); + } + } + + /* attempt to grab a lock */ + if(_alpm_handle_lock(handle)) { + GOTO_ERR(handle, ALPM_ERR_HANDLE_LOCK, cleanup); + } + + ret = _alpm_multi_download(handle, payloads, syncpath); + if(ret < 0) { + goto cleanup; + } + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + /* Cache needs to be rebuilt */ + _alpm_db_free_pkgcache(db); + + /* clear all status flags regarding validity/existence */ + db->status &= ~DB_STATUS_VALID; + db->status &= ~DB_STATUS_INVALID; + db->status &= ~DB_STATUS_EXISTS; + db->status &= ~DB_STATUS_MISSING; + + /* if the download failed skip validation to preserve the download error */ + if(sync_db_validate(db) != 0) { + /* pm_errno should be set */ + ret = -1; + } + } + +cleanup: + _alpm_handle_unlock(handle); + + if(ret == -1) { + /* pm_errno was set by the download code */ + _alpm_log(handle, ALPM_LOG_DEBUG, "failed to sync dbs: %s\n", + alpm_strerror(handle->pm_errno)); + } else { + handle->pm_errno = ALPM_ERR_OK; + } + + if(payloads) { + alpm_list_free_inner(payloads, (alpm_list_fn_free)_alpm_dload_payload_reset); + FREELIST(payloads); + } + free(syncpath); + umask(oldmask); + return ret; +} + /* Forward decl so I don't reorganize the whole file right now */ static int sync_db_read(alpm_db_t *db, struct archive *archive, struct archive_entry *entry, alpm_pkg_t **likely_pkg); diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 670da03d..7cd3e3a4 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -636,6 +636,16 @@ int _alpm_download(struct dload_payload *payload, const char *localpath, } } +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath) +{ + (void)handle; + (void)payloads; + (void)localpath; + return 0; +} + static char *filecache_find_url(alpm_handle_t *handle, const char *url) { const char *filebase = strrchr(url, '/'); @@ -738,6 +748,7 @@ void _alpm_dload_payload_reset(struct dload_payload *payload) FREE(payload->destfile_name); FREE(payload->content_disp_name); FREE(payload->fileurl); + FREE(payload->filepath); *payload = (struct dload_payload){0}; } @@ -746,6 +757,7 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) ASSERT(payload, return); FREE(payload->fileurl); + FREE(payload->filepath); payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 1e8f75f3..3eb7fbe1 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -31,6 +31,7 @@ struct dload_payload { char *destfile_name; char *content_disp_name; char *fileurl; + char *filepath; /* download URL path */ alpm_list_t *servers; long respcode; off_t initial_size; @@ -53,4 +54,8 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload); int _alpm_download(struct dload_payload *payload, const char *localpath, char **final_file, const char **final_url); +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath); + #endif /* ALPM_DLOAD_H */ -- 2.25.1

2 6

[pacman-dev] [PATCH] Add NoProgressbar to pacman.conf options
by escondida＠iff.ink 13 Apr '20

13 Apr '20

From: Ivy Foster <escondida(a)iff.ink> Signed-off-by: Ivy Foster <escondida(a)iff.ink> --- doc/pacman.conf.5.asciidoc | 3 +++ etc/pacman.conf.in | 1 + src/pacman/conf.c | 2 ++ src/pacman/pacman-conf.c | 3 +++ 4 files changed, 9 insertions(+) diff --git a/doc/pacman.conf.5.asciidoc b/doc/pacman.conf.5.asciidoc index b297e332..73f57eeb 100644 --- a/doc/pacman.conf.5.asciidoc +++ b/doc/pacman.conf.5.asciidoc @@ -186,6 +186,9 @@ Options *Color*:: Automatically enable colors only when pacman's output is on a tty. +*NoProgressbar*:: + Disables progressbars. + *TotalDownload*:: When downloading, display the amount downloaded, download rate, ETA, and completed percentage of the entire download list rather diff --git a/etc/pacman.conf.in b/etc/pacman.conf.in index 7446944f..fb473fe9 100644 --- a/etc/pacman.conf.in +++ b/etc/pacman.conf.in @@ -31,6 +31,7 @@ Architecture = auto # Misc options #UseSyslog #Color +#NoProgressbar #TotalDownload CheckSpace #VerbosePkgLists diff --git a/src/pacman/conf.c b/src/pacman/conf.c index f9de386f..92614860 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -579,6 +579,8 @@ static int _parse_options(const char *key, char *value, config->color = isatty(fileno(stdout)) ? PM_COLOR_ON : PM_COLOR_OFF; enable_colors(config->color); } + } else if(strcmp(key, "NoProgressbar") == 0) { + config->noprogressbar = 1; } else if(strcmp(key, "DisableDownloadTimeout") == 0) { config->disable_dl_timeout = 1; } else { diff --git a/src/pacman/pacman-conf.c b/src/pacman/pacman-conf.c index 6e25d243..48152697 100644 --- a/src/pacman/pacman-conf.c +++ b/src/pacman/pacman-conf.c @@ -259,6 +259,7 @@ static void dump_config(void) show_bool("VerbosePkgLists", config->verbosepkglists); show_bool("DisableDownloadTimeout", config->disable_dl_timeout); show_bool("ILoveCandy", config->chomp); + show_bool("NoProgressbar", config->noprogressbar); show_cleanmethod("CleanMethod", config->cleanmethod); @@ -368,6 +369,8 @@ static int list_directives(void) show_bool("VerbosePkgLists", config->verbosepkglists); } else if(strcasecmp(i->data, "DisableDownloadTimeout") == 0) { show_bool("DisableDownloadTimeout", config->disable_dl_timeout); + } else if(strcasecmp(i->data, "NoProgressbar") == 0) { + show_bool("NoProgressbar", config->noprogressbar); } else if(strcasecmp(i->data, "CleanMethod") == 0) { show_cleanmethod("CleanMethod", config->cleanmethod); -- 2.26.0

2 2

[pacman-dev] [PATCH 1/2] Use STRDUP for error checking in more places
by Allan McRae 13 Apr '20

13 Apr '20

Use STRDUP() over strdup() to catch memory allocation errors. There are still some instances of strdup left, but these are in functions that currently have no error path and would require a larger rework. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- lib/libalpm/be_package.c | 10 +++++++--- lib/libalpm/handle.c | 2 +- lib/libalpm/util.c | 3 ++- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/lib/libalpm/be_package.c b/lib/libalpm/be_package.c index 4832966b..38ba365d 100644 --- a/lib/libalpm/be_package.c +++ b/lib/libalpm/be_package.c @@ -201,11 +201,15 @@ static int parse_descfile(alpm_handle_t *handle, struct archive *a, alpm_pkg_t * } else if(strcmp(key, "pkgdesc") == 0) { STRDUP(newpkg->desc, ptr, return -1); } else if(strcmp(key, "group") == 0) { - newpkg->groups = alpm_list_add(newpkg->groups, strdup(ptr)); + char *tmp = NULL; + STRDUP(tmp, ptr, return -1); + newpkg->groups = alpm_list_add(newpkg->groups, tmp); } else if(strcmp(key, "url") == 0) { STRDUP(newpkg->url, ptr, return -1); } else if(strcmp(key, "license") == 0) { - newpkg->licenses = alpm_list_add(newpkg->licenses, strdup(ptr)); + char *tmp = NULL; + STRDUP(tmp, ptr, return -1); + newpkg->licenses = alpm_list_add(newpkg->licenses, tmp); } else if(strcmp(key, "builddate") == 0) { newpkg->builddate = _alpm_parsedate(ptr); } else if(strcmp(key, "packager") == 0) { @@ -660,7 +664,7 @@ alpm_pkg_t *_alpm_pkg_load_internal(alpm_handle_t *handle, /* internal fields for package struct */ newpkg->origin = ALPM_PKG_FROM_FILE; - newpkg->origin_data.file = strdup(pkgfile); + STRDUP(newpkg->origin_data.file, pkgfile, goto error); newpkg->ops = get_file_pkg_ops(); newpkg->handle = handle; newpkg->infolevel = INFRQ_BASE | INFRQ_DESC | INFRQ_SCRIPTLET; diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index 8ae08d11..e3b2c911 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -105,7 +105,7 @@ int _alpm_handle_lock(alpm_handle_t *handle) ASSERT(handle->lockfd < 0, return 0); /* create the dir of the lockfile first */ - dir = strdup(handle->lockfile); + STRDUP(dir, handle->lockfile, return -1); ptr = strrchr(dir, '/'); if(ptr) { *ptr = '\0'; diff --git a/lib/libalpm/util.c b/lib/libalpm/util.c index cebc87ec..cb838e43 100644 --- a/lib/libalpm/util.c +++ b/lib/libalpm/util.c @@ -350,7 +350,8 @@ int _alpm_unpack(alpm_handle_t *handle, const char *path, const char *prefix, /* If specific files were requested, skip entries that don't match. */ if(list) { - char *entry_prefix = strdup(entryname); + char *entry_prefix = NULL; + STRDUP(entry_prefix, entryname, ret = 1; goto cleanup); char *p = strstr(entry_prefix,"/"); if(p) { *(p + 1) = '\0'; -- 2.25.1

1 1

[pacman-dev] [PATCH] Touch up proto files
by Daniel M. Capella 13 Apr '20

13 Apr '20

- split: Fix erroneous $pkgname calls to use $pkgbase - VCS: - Add missing fields - Remove unnecessary $srcdir's - Move ./autogen.sh to prepare() - 80 char line length for comments and make license note stand out - Add ./autogen.sh to the other proto's Signed-off-by: Daniel M. Capella <polyzen(a)archlinux.org> --- proto/PKGBUILD-split.proto | 18 ++++++++++-------- proto/PKGBUILD-vcs.proto | 21 +++++++++++++-------- proto/PKGBUILD.proto | 1 + 3 files changed, 24 insertions(+), 16 deletions(-) diff --git a/proto/PKGBUILD-split.proto b/proto/PKGBUILD-split.proto index 9898ef81..6fc91a56 100644 --- a/proto/PKGBUILD-split.proto +++ b/proto/PKGBUILD-split.proto @@ -1,7 +1,8 @@ -# This is an example of a PKGBUILD for splitting packages. Use this as a -# start to creating your own, and remove these comments. For more information, -# see 'man PKGBUILD'. NOTE: Please fill out the license field for your package! -# If it is unknown, then please put 'unknown'. +# This is an example of a PKGBUILD for splitting packages. Use this as a start +# to creating your own, and remove these comments. For more information, see +# 'man PKGBUILD'. +# NOTE: Please fill out the license field for your package! If it is unknown, +# then please put 'unknown'. # Maintainer: Your Name <youremail(a)domain.com> pkgname=('pkg1' 'pkg2') @@ -26,14 +27,15 @@ options=() install= changelog= source=("$pkgbase-$pkgver.tar.gz" - "$pkgname-$pkgver.patch") + "$pkgbase-$pkgver.patch") noextract=() md5sums=() validpgpkeys=() prepare() { - cd "$pkgname-$pkgver" - patch -p1 -i "$srcdir/$pkgname-$pkgver.patch" + cd "$pkgbase-$pkgver" + patch -p1 -i "$srcdir/$pkgbase-$pkgver.patch" + ./autogen.sh } build() { @@ -43,7 +45,7 @@ build() { } check() { - cd "$pkgname-$pkgver" + cd "$pkgbase-$pkgver" make -k check } diff --git a/proto/PKGBUILD-vcs.proto b/proto/PKGBUILD-vcs.proto index ae9956a9..01eddf98 100644 --- a/proto/PKGBUILD-vcs.proto +++ b/proto/PKGBUILD-vcs.proto @@ -1,5 +1,6 @@ -# This is an example PKGBUILD file. Use this as a start to creating your own, -# and remove these comments. For more information, see 'man PKGBUILD'. +# This is an example of a PKGBUILD for VCS packages. Use this as a start to +# creating your own, and remove these comments. For more information, see 'man +# PKGBUILD'. # NOTE: Please fill out the license field for your package! If it is unknown, # then please put 'unknown'. @@ -10,6 +11,7 @@ pkgname=NAME-VCS # '-bzr', '-git', '-hg' or '-svn' pkgver=VERSION pkgrel=1 +epoch= pkgdesc="" arch=() url="" @@ -17,21 +19,24 @@ license=('GPL') groups=() depends=() makedepends=('VCS_PACKAGE') # 'bzr', 'git', 'mercurial' or 'subversion' +checkdepends=() provides=("${pkgname%-VCS}") conflicts=("${pkgname%-VCS}") replaces=() backup=() options=() install= +changelog= source=('FOLDER::VCS+URL#FRAGMENT') noextract=() md5sums=('SKIP') +validpgpkeys=() # Please refer to the 'USING VCS SOURCES' section of the PKGBUILD man page for # a description of each element in the source array. pkgver() { - cd "$srcdir/${pkgname%-VCS}" + cd "${pkgname%-VCS}" # The examples below are not absolute and need to be adapted to each repo. The # primary goal is to generate version numbers that will increase according to @@ -56,23 +61,23 @@ pkgver() { } prepare() { - cd "$srcdir/${pkgname%-VCS}" + cd "${pkgname%-VCS}" patch -p1 -i "$srcdir/${pkgname%-VCS}.patch" + ./autogen.sh } build() { - cd "$srcdir/${pkgname%-VCS}" - ./autogen.sh + cd "${pkgname%-VCS}" ./configure --prefix=/usr make } check() { - cd "$srcdir/${pkgname%-VCS}" + cd "${pkgname%-VCS}" make -k check } package() { - cd "$srcdir/${pkgname%-VCS}" + cd "${pkgname%-VCS}" make DESTDIR="$pkgdir/" install } diff --git a/proto/PKGBUILD.proto b/proto/PKGBUILD.proto index a2c600d5..e72b1aa8 100644 --- a/proto/PKGBUILD.proto +++ b/proto/PKGBUILD.proto @@ -33,6 +33,7 @@ validpgpkeys=() prepare() { cd "$pkgname-$pkgver" patch -p1 -i "$srcdir/$pkgname-$pkgver.patch" + ./autogen.sh } build() { -- 2.26.0

2 1

[pacman-dev] [PATCH] Use noextract with pacman-conf NoExtract
by Eli Schwartz 13 Apr '20

13 Apr '20

From: Earnestly <zibeon(a)googlemail.com> Current code accidently uses noupgrade for the NoExtract directive. https://medium.com/@Code_Analysis/the-last-line-effect-7b1cb7f0d2a1 Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- Previously fixed in https://github.com/andrewgregory/pacutils/commit/64c7d0aca1a3aa3be5b7789f10… but the unfixed version got merged here. :/ src/pacman/pacman-conf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pacman/pacman-conf.c b/src/pacman/pacman-conf.c index 939cbdbd..f67ec21f 100644 --- a/src/pacman/pacman-conf.c +++ b/src/pacman/pacman-conf.c @@ -348,7 +348,7 @@ static int list_directives(void) } else if(strcasecmp(i->data, "NoUpgrade") == 0) { show_list_str("NoUpgrade", config->noupgrade); } else if(strcasecmp(i->data, "NoExtract") == 0) { - show_list_str("NoExtract", config->noupgrade); + show_list_str("NoExtract", config->noextract); } else if(strcasecmp(i->data, "Architecture") == 0) { -- 2.26.0

3 3

[pacman-dev] [PATCH] Add REALLOC macro to simplify realloc error handling
by Rikard Falkeborn 13 Apr '20

13 Apr '20

realloc can fail just like the other memory allocation functions. Add a macro to simplify handling of realloc failures, similar to the already existing MALLOC, CALLOC, etc. Replace the existing realloc uses with the new macro, allowing us to move tedious error handling to the macro. Also, in be_package and be_sync, this fixes hypothetical memory leaks (and thereafter null pointer dereferences) in case realloc fails to shrink the allocated memory. Signed-off-by: Rikard Falkeborn <rikard.falkeborn(a)gmail.com> --- lib/libalpm/be_local.c | 7 +------ lib/libalpm/be_package.c | 3 +-- lib/libalpm/be_sync.c | 2 +- lib/libalpm/util.c | 13 +++---------- lib/libalpm/util.h | 1 + 5 files changed, 7 insertions(+), 19 deletions(-) diff --git a/lib/libalpm/be_local.c b/lib/libalpm/be_local.c index 9ebdfa40..689f0102 100644 --- a/lib/libalpm/be_local.c +++ b/lib/libalpm/be_local.c @@ -843,12 +843,7 @@ static int local_db_read(alpm_pkg_t *info, int inforeq) } /* attempt to hand back any memory we don't need */ if(files_count > 0) { - alpm_file_t *newfiles; - - newfiles = realloc(files, sizeof(alpm_file_t) * files_count); - if(newfiles != NULL) { - files = newfiles; - } + REALLOC(files, sizeof(alpm_file_t) * files_count, (void)0); } else { FREE(files); } diff --git a/lib/libalpm/be_package.c b/lib/libalpm/be_package.c index 7a118d2a..4832966b 100644 --- a/lib/libalpm/be_package.c +++ b/lib/libalpm/be_package.c @@ -669,8 +669,7 @@ alpm_pkg_t *_alpm_pkg_load_internal(alpm_handle_t *handle, if(full) { if(newpkg->files.files) { /* attempt to hand back any memory we don't need */ - newpkg->files.files = realloc(newpkg->files.files, - sizeof(alpm_file_t) * newpkg->files.count); + REALLOC(newpkg->files.files, sizeof(alpm_file_t) * newpkg->files.count, (void)0); /* "checking for conflicts" requires a sorted list, ensure that here */ _alpm_log(handle, ALPM_LOG_DEBUG, "sorting package filelist for %s\n", pkgfile); diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index aafed15d..5f457122 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -689,7 +689,7 @@ static int sync_db_read(alpm_db_t *db, struct archive *archive, } /* attempt to hand back any memory we don't need */ if(files_count > 0) { - files = realloc(files, sizeof(alpm_file_t) * files_count); + REALLOC(files, sizeof(alpm_file_t) * files_count, (void)0); } else { FREE(files); } diff --git a/lib/libalpm/util.c b/lib/libalpm/util.c index c46b1397..cebc87ec 100644 --- a/lib/libalpm/util.c +++ b/lib/libalpm/util.c @@ -1427,22 +1427,15 @@ int _alpm_fnmatch(const void *pattern, const void *string) */ void *_alpm_realloc(void **data, size_t *current, const size_t required) { - char *newdata; - - newdata = realloc(*data, required); - if(!newdata) { - _alpm_alloc_fail(required); - return NULL; - } + REALLOC(*data, required, return NULL); if (*current < required) { /* ensure all new memory is zeroed out, in both the initial * allocation and later reallocs */ - memset(newdata + *current, 0, required - *current); + memset((char*)*data + *current, 0, required - *current); } *current = required; - *data = newdata; - return newdata; + return *data; } /** This automatically grows data based on current/required. diff --git a/lib/libalpm/util.h b/lib/libalpm/util.h index 675eedec..3306a022 100644 --- a/lib/libalpm/util.h +++ b/lib/libalpm/util.h @@ -53,6 +53,7 @@ void _alpm_alloc_fail(size_t size); #define MALLOC(p, s, action) do { p = malloc(s); if(p == NULL) { _alpm_alloc_fail(s); action; } } while(0) #define CALLOC(p, l, s, action) do { p = calloc(l, s); if(p == NULL) { _alpm_alloc_fail(l * s); action; } } while(0) +#define REALLOC(p, s, action) do { void* np = realloc(p, s); if(np == NULL) { _alpm_alloc_fail(s); action; } else { p = np; } } while(0) /* This strdup macro is NULL safe- copying NULL will yield NULL */ #define STRDUP(r, s, action) do { if(s != NULL) { r = strdup(s); if(r == NULL) { _alpm_alloc_fail(strlen(s)); action; } } else { r = NULL; } } while(0) #define STRNDUP(r, s, l, action) do { if(s != NULL) { r = strndup(s, l); if(r == NULL) { _alpm_alloc_fail(l); action; } } else { r = NULL; } } while(0) -- 2.26.0

2 1

[pacman-dev] [PATCH v3] Introduce alpm_dbs_update() function for parallel db updates
by Anatol Pomozov 13 Apr '20

13 Apr '20

This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- lib/libalpm/alpm.h | 29 ++++++++++ lib/libalpm/be_sync.c | 128 ++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/dload.c | 12 ++++ lib/libalpm/dload.h | 5 ++ 4 files changed, 174 insertions(+) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 5d559db1..2cf20343 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -1049,6 +1049,35 @@ int alpm_db_remove_server(alpm_db_t *db, const char *url); */ int alpm_db_update(int force, alpm_db_t *db); +/** Update package databases + * + * An update of the package databases in the list \a dbs will be attempted. + * Unless \a force is true, the update will only be performed if the remote + * databases were modified since the last update. + * + * This operation requires a database lock, and will return an applicable error + * if the lock could not be obtained. + * + * Example: + * @code + * alpm_list_t *dbs = alpm_get_syncdbs(); + * ret = alpm_dbs_update(config->handle, dbs, force); + * if(ret < 0) { + * pm_printf(ALPM_LOG_ERROR, _("failed to synchronize all databases (%s)\n"), + * alpm_strerror(alpm_errno(config->handle))); + * } + * @endcode + * + * @note After a successful update, the \link alpm_db_get_pkgcache() + * package cache \endlink will be invalidated + * @param handle the context handle + * @param dbs list of package databases to update + * @param force if true, then forces the update, otherwise update only in case + * the databases aren't up to date + * @return 0 on success, -1 on error (pm_errno is set accordingly) + */ +int alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force); + /** Get a package entry from a package database. * @param db pointer to the package database to get the package from * @param name of the package diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index aafed15d..7ab52301 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -301,6 +301,134 @@ int SYMEXPORT alpm_db_update(int force, alpm_db_t *db) return ret; } +int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force) { + char *syncpath; + const char *dbext = handle->dbext; + alpm_list_t *i; + int ret = -1; + mode_t oldmask; + alpm_list_t *payloads = NULL; + + /* Sanity checks */ + CHECK_HANDLE(handle, return -1); + ASSERT(dbs != NULL, return -1); + handle->pm_errno = ALPM_ERR_OK; + + syncpath = get_sync_dir(handle); + ASSERT(syncpath != NULL, return -1); + + /* make sure we have a sane umask */ + oldmask = umask(0022); + + /* attempt to grab a lock */ + if(_alpm_handle_lock(handle)) { + GOTO_ERR(handle, ALPM_ERR_HANDLE_LOCK, cleanup); + } + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + int dbforce = force; + struct dload_payload *payload = NULL; + size_t len; + int siglevel; + + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + ASSERT(db != handle->db_local, GOTO_ERR(handle, ALPM_ERR_WRONG_ARGS, cleanup)); + ASSERT(db->servers != NULL, GOTO_ERR(handle, ALPM_ERR_SERVER_NONE, cleanup)); + + /* force update of invalid databases to fix potential mismatched database/signature */ + if(db->status & DB_STATUS_INVALID) { + dbforce = 1; + } + + CALLOC(payload, 1, sizeof(*payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + + /* set hard upper limit of 128 MiB */ + payload->max_size = 128 * 1024 * 1024; + payload->servers = db->servers; + + /* print server + filename into a buffer */ + len = strlen(db->treename) + strlen(dbext) + 1; + MALLOC(payload->filepath, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(payload->filepath, len, "%s%s", db->treename, dbext); + payload->handle = handle; + payload->force = dbforce; + payload->unlink_on_fail = 1; + + payloads = alpm_list_add(payloads, payload); + + siglevel = alpm_db_get_siglevel(db); + if(siglevel & ALPM_SIG_DATABASE) { + struct dload_payload *sig_payload; + CALLOC(sig_payload, 1, sizeof(*sig_payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + + /* print filename into a buffer (leave space for separator and .sig) */ + len = strlen(db->treename) + strlen(dbext) + 5; + MALLOC(sig_payload->filepath, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(sig_payload->filepath, len, "%s%s.sig", db->treename, dbext); + + sig_payload->handle = handle; + sig_payload->force = 1; + sig_payload->errors_ok = (siglevel & ALPM_SIG_DATABASE_OPTIONAL); + + /* set hard upper limit of 16 KiB */ + sig_payload->max_size = 16 * 1024; + sig_payload->servers = db->servers; + + payloads = alpm_list_add(payloads, sig_payload); + } + } + + ret = _alpm_multi_download(handle, payloads, syncpath); + if(ret < 0) { + goto cleanup; + } + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + /* Cache needs to be rebuilt */ + _alpm_db_free_pkgcache(db); + + /* clear all status flags regarding validity/existence */ + db->status &= ~DB_STATUS_VALID; + db->status &= ~DB_STATUS_INVALID; + db->status &= ~DB_STATUS_EXISTS; + db->status &= ~DB_STATUS_MISSING; + + /* if the download failed skip validation to preserve the download error */ + if(sync_db_validate(db) != 0) { + /* pm_errno should be set */ + ret = -1; + } + } + +cleanup: + _alpm_handle_unlock(handle); + + if(ret == -1) { + /* pm_errno was set by the download code */ + _alpm_log(handle, ALPM_LOG_DEBUG, "failed to sync dbs: %s\n", + alpm_strerror(handle->pm_errno)); + } else { + handle->pm_errno = ALPM_ERR_OK; + } + + if(payloads) { + alpm_list_free_inner(payloads, (alpm_list_fn_free)_alpm_dload_payload_reset); + FREELIST(payloads); + } + free(syncpath); + umask(oldmask); + return ret; +} + /* Forward decl so I don't reorganize the whole file right now */ static int sync_db_read(alpm_db_t *db, struct archive *archive, struct archive_entry *entry, alpm_pkg_t **likely_pkg); diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 670da03d..7cd3e3a4 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -636,6 +636,16 @@ int _alpm_download(struct dload_payload *payload, const char *localpath, } } +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath) +{ + (void)handle; + (void)payloads; + (void)localpath; + return 0; +} + static char *filecache_find_url(alpm_handle_t *handle, const char *url) { const char *filebase = strrchr(url, '/'); @@ -738,6 +748,7 @@ void _alpm_dload_payload_reset(struct dload_payload *payload) FREE(payload->destfile_name); FREE(payload->content_disp_name); FREE(payload->fileurl); + FREE(payload->filepath); *payload = (struct dload_payload){0}; } @@ -746,6 +757,7 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) ASSERT(payload, return); FREE(payload->fileurl); + FREE(payload->filepath); payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 1e8f75f3..3eb7fbe1 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -31,6 +31,7 @@ struct dload_payload { char *destfile_name; char *content_disp_name; char *fileurl; + char *filepath; /* download URL path */ alpm_list_t *servers; long respcode; off_t initial_size; @@ -53,4 +54,8 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload); int _alpm_download(struct dload_payload *payload, const char *localpath, char **final_file, const char **final_url); +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath); + #endif /* ALPM_DLOAD_H */ -- 2.26.0

2 2

[pacman-dev] [PATCH] Use C18 language standard
by Anatol Pomozov 05 Apr '20

05 Apr '20

C18 is the latest released version of the language spec. The toolchains support it starting with GCC 8.1.0 and Clang 7. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- configure.ac | 2 +- meson.build | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/configure.ac b/configure.ac index 9db39232..84869ce5 100644 --- a/configure.ac +++ b/configure.ac @@ -183,7 +183,7 @@ AC_SUBST(LFS_CFLAGS) # Checks for programs. AC_PROG_AWK -AC_PROG_CC_C99 +AC_PROG_CC_C18 AC_PROG_INSTALL AC_CHECK_PROGS([PYTHON], [python3 python], [false]) AC_PATH_PROGS([BASH_SHELL], [bash bash4], [false]) diff --git a/meson.build b/meson.build index fc81fa27..2f4e7cc4 100644 --- a/meson.build +++ b/meson.build @@ -3,7 +3,7 @@ project('pacman', version : '5.2.1', license : 'GPLv2+', default_options : [ - 'c_std=gnu99', + 'c_std=c18', 'prefix=/usr', 'sysconfdir=/etc', 'localstatedir=/var', -- 2.26.0

3 6

[pacman-dev] [PATCH v2] meson: use 'pedantic' compiler warning level by default
by Anatol Pomozov 27 Mar '20

27 Mar '20

This sets parity with the autotools version of the build that has -pedantic flag enbled unconditionally. Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- meson.build | 1 + 1 file changed, 1 insertion(+) diff --git a/meson.build b/meson.build index 572526b2..fc81fa27 100644 --- a/meson.build +++ b/meson.build @@ -7,6 +7,7 @@ project('pacman', 'prefix=/usr', 'sysconfdir=/etc', 'localstatedir=/var', + 'warning_level=3', ], meson_version : '>= 0.51') -- 2.26.0

2 1

[pacman-dev] [PATCH] meson: use 'pedantic' compiler warning level
by Anatol Pomozov 27 Mar '20

27 Mar '20

Signed-off-by: Anatol Pomozov <anatol.pomozov(a)gmail.com> --- meson.build | 1 + 1 file changed, 1 insertion(+) diff --git a/meson.build b/meson.build index 572526b2..fc81fa27 100644 --- a/meson.build +++ b/meson.build @@ -7,6 +7,7 @@ project('pacman', 'prefix=/usr', 'sysconfdir=/etc', 'localstatedir=/var', + 'warning_level=3', ], meson_version : '>= 0.51') -- 2.25.1

3 13