- Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH v2] libalpm: fix error when downloading up to date files
by morganamilo 03 May '21

03 May '21

--- lib/libalpm/sync.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 2419cc69..bdfab635 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -1128,7 +1128,7 @@ int _alpm_sync_load(alpm_handle_t *handle, alpm_list_t **data) uint64_t total_bytes = 0; alpm_trans_t *trans = handle->trans; - if(download_files(handle)) { + if(download_files(handle) == -1) { return -1; } -- 2.31.1

2 1

Re: [pacman-dev] [PATCH] libalpm: fix alpm_fetch_pkgurl with a fetch callback
by Guillaume Benoit 02 May '21

02 May '21

Le 02/05/2021 à 11:47, Allan McRae a écrit :> On 2/5/21 12:33 am, Guillaume Benoit wrote:>> Le 01/05/2021 à 05:53, Andrew Gregory a écrit :>>> On 04/30/21 at 12:09pm, Guillaume Benoit wrote:>>>> After download, alpm_fetch_pkgurl uses payload->destfile_name>>>> and payload->tempfile_name in order to find the downloaded file path.>>>> Those fields are not set if a custom fetch callback is defined.>>>> Use filecache_find_url instead, like in the beginning of the function.>>>>>>>> --->>>> lib/libalpm/dload.c | 13 ++++--------->>>> 1 file changed, 4 insertions(+), 9 deletions(-)>>>>>>>> diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c>>>> index 6f33451a..d45c7707 100644>>>> --- a/lib/libalpm/dload.c>>>> +++ b/lib/libalpm/dload.c>>>> @@ -1007,16 +1007,11 @@ int SYMEXPORT alpm_fetch_pkgurl(alpm_handle_t>>>> *handle, const alpm_list_t *urls,>>>> EVENT(handle, &event);>>>> }>>>> - for(i = payloads; i; i = i->next) {>>>> - struct dload_payload *payload = i->data;>>>> - char *filepath;>>>> + for(i = urls; i; i = i->next) {>>>> + char *url = i->data;>>>> - if(payload->destfile_name) {>>>> - const char *filename =>>>> mbasename(payload->destfile_name);>>>> - filepath = _alpm_filecache_find(handle, filename);>>>> - } else {>>>> - STRDUP(filepath, payload->tempfile_name,>>>> GOTO_ERR(handle, ALPM_ERR_MEMORY, err));>>>> - }>>>> + /* attempt again to find the file in our pkgcache */>>>> + char *filepath = filecache_find_url(handle, url);>>>> if(filepath) {>>>> alpm_list_append(fetched, filepath);>>>> } else {>>>>>> Without testing since this needs to be rebased now; this looks broken.>>> filecache_find_url with fail for any url where destfile_name doesn't>>> match the file name in the url. Notably, that includes the download>>> links on Arch's package pages:>>> https://archlinux.org/packages/core/x86_64/pacman/download/>>>>> Yes, with or without this patch, the current implementation doesn't work>> for this kind of url when using a fetch callback.>> The solution I see is to change the fetch callback signature by making>> it return the path of the downloaded file.>> Otherwise this patch works for standard mirror urls.>> .> > While this patch does improve the current situation, it is a step> backwards in terms the full fix, which will need to pass the full> payload to the front-end so it can fill in the needed values.> > On that basis, I will not be accepting this patch.> > Given fetching a URL with XferCommand has not worked for a long time (or> ever?), this issue is not a blocker for 6.0.> > Allan> Yes, with pacman 5, fetching a URL with XferCommand doesn't work.With pacman 6, it doesn't work with the current code but it works with this patch,except for the special URLs given by Andrew Gregory.I can try to provide a different patch that pass the full payload to the front-endbut it can't help making pacman 6 correctly fetching a URL with XferCommand when thefilename can be guessed from the URL.

3 2

[pacman-dev] [PATCH] libalpm: fix error when downloading up to date files
by morganamilo 01 May '21

01 May '21

--- lib/libalpm/sync.c | 2 +- lib/libalpm/trans.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 2419cc69..bdfab635 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -1128,7 +1128,7 @@ int _alpm_sync_load(alpm_handle_t *handle, alpm_list_t **data) uint64_t total_bytes = 0; alpm_trans_t *trans = handle->trans; - if(download_files(handle)) { + if(download_files(handle) == -1) { return -1; } diff --git a/lib/libalpm/trans.c b/lib/libalpm/trans.c index 939ab05a..e608555b 100644 --- a/lib/libalpm/trans.c +++ b/lib/libalpm/trans.c @@ -185,7 +185,7 @@ int SYMEXPORT alpm_trans_commit(alpm_handle_t *handle, alpm_list_t **data) } if(trans->add) { - if(_alpm_sync_load(handle, data) != 0) { + if(_alpm_sync_load(handle, data) == -1) { /* pm_errno is set by _alpm_sync_load() */ return -1; } -- 2.31.1

1 0

Re: [pacman-dev] Patch Policy Inquiry
by Sijie Bu 01 May '21

01 May '21

Hello, Sorry for the late reply. I was a bit busy for the past two days. The specific Arch Linux 32 changes in question, though I have not confirmed yet, seems to be detecting Pentium 4 CPUs and optimizing accordingly. I will have to discuss with my project mentor to see if we need these too (because we're targeting the R6 sub-architecture, which is relatively new). But otherwise, looks like the pacman code should compile and run cleanly. Thank you for the clarifications. Sincerely

1 0

[pacman-dev] [GIT] The official pacman repository branch, master, updated. v6.0.0alpha1-106-g72238aa0
by Allan McRae 01 May '21

01 May '21

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "The official pacman repository". The branch, master has been updated via 72238aa046b8a8de1c7b9110726986c7538023b5 (commit) via eb1a63a516cb3791d0084b0d05b074c963aa01e9 (commit) via 0ff94ae85d40efdc177e1852ef58848f82d14156 (commit) via 8faf749fd19aff0eb35eb7108d2fd379d3c9c3b9 (commit) via ab622b4881e6b971c62de4323c72c87a6da51323 (commit) via e7fa35baa22bf6710a904815456d3ff679005fc7 (commit) via 523c393e9e6175e632d355c9440abb0be164a362 (commit) from 558d08d821a6797c972acdf02e03aa1a1ff49917 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 72238aa046b8a8de1c7b9110726986c7538023b5 Author: Andrew Gregory <andrew.gregory.8(a)gmail.com> Date: Tue Apr 27 19:11:27 2021 -0700 call download progress callback for signatures pacman may not care about them, but other front-ends do. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> commit eb1a63a516cb3791d0084b0d05b074c963aa01e9 Author: Andrew Gregory <andrew.gregory.8(a)gmail.com> Date: Mon Apr 26 20:47:58 2021 -0700 alpm_db_update: indicate if dbs were up to date Restore the prior indicator whether or not databases were up to date. 0 is used to indicate if *any* db was actually updated as callers are more likely to care about that than if *all* dbs were updated. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> commit 0ff94ae85d40efdc177e1852ef58848f82d14156 Author: Andrew Gregory <andrew.gregory.8(a)gmail.com> Date: Mon Apr 26 19:37:52 2021 -0700 fix downloading multiple urls with XferCommand An extra break causes _alpm_download to break out of the payload loop as soon as it sees a successful url download with XferCommand. Fixes: FS#70608 - -U fails to download all files with XferCommand Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> commit 8faf749fd19aff0eb35eb7108d2fd379d3c9c3b9 Author: Andrew Gregory <andrew.gregory.8(a)gmail.com> Date: Mon Apr 26 19:34:14 2021 -0700 pmserve: allow serving byte objects directly Useful for serving in-memory package files. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> commit ab622b4881e6b971c62de4323c72c87a6da51323 Author: Andrew Gregory <andrew.gregory.8(a)gmail.com> Date: Mon Apr 26 19:32:57 2021 -0700 pmpkg: add makepkg_bytes Builds the package file in memory. Useful with the built-in server. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> commit e7fa35baa22bf6710a904815456d3ff679005fc7 Author: Andrew Gregory <andrew.gregory.8(a)gmail.com> Date: Sun Apr 25 11:03:54 2021 -0700 add front-end provided context to callbacks Our callbacks require front-ends to maintain state in order to provide reasonable output. The new download callback in particular requires much more complex state information to be saved. Without the ability to provide context, state must be saved globally, which may not be possible for all front-ends. Scripting language bindings in particular have no way to register per-handle callbacks without some form of context. Implements: FS#12721 Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> commit 523c393e9e6175e632d355c9440abb0be164a362 Author: Eli Schwartz <eschwartz(a)archlinux.org> Date: Fri Apr 23 14:34:47 2021 -0400 makepkg: document BUILDTOOL and BUILDTOOLVER environmental variables These were added without documentation in commit a33cdac10bf1a68279ea1b9baa26aa09126f1ca4 Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> Signed-off-by: Allan McRae <allan(a)archlinux.org> ----------------------------------------------------------------------- Summary of changes: doc/makepkg.8.asciidoc | 8 +++ lib/libalpm/alpm.h | 76 ++++++++++++++++++---- lib/libalpm/dload.c | 61 ++++++++--------- lib/libalpm/handle.c | 54 +++++++++++++-- lib/libalpm/handle.h | 12 +++- lib/libalpm/log.c | 2 +- lib/libalpm/sync.c | 2 +- src/pacman/callback.c | 30 +++++++-- src/pacman/callback.h | 14 ++-- src/pacman/conf.c | 18 ++--- src/util/testpkg.c | 7 +- test/pacman/meson.build | 2 + test/pacman/pmpkg.py | 14 ++-- test/pacman/pmserve.py | 2 + .../tests/upgrade-download-with-xfercommand.py | 26 ++++++++ 15 files changed, 246 insertions(+), 82 deletions(-) create mode 100644 test/pacman/tests/upgrade-download-with-xfercommand.py hooks/post-receive -- The official pacman repository

1 0

[pacman-dev] [PATCH] libalpm: download signatures with a fetch callback
by Guillaume Benoit 30 Apr '21

30 Apr '21

Remove an unwanted break and download signatures when using a custom fetch callback. --- lib/libalpm/dload.c | 43 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 37 insertions(+), 6 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index a4c42f8d..6f33451a 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -877,9 +877,26 @@ int _alpm_download(alpm_handle_t *handle, int success = 0; if(payload->fileurl) { - if (handle->fetchcb(payload->fileurl, localpath, payload->force) != -1) { - success = 1; - break; + int ret; + + ret = handle->fetchcb(payload->fileurl, localpath, payload->force); + + if (ret != -1) { + /* Let's check if client requested downloading accompanion *.sig file */ + if(payload->download_signature) { + char *sig_fileurl; + + size_t sig_len = strlen(payload->fileurl) + 5; + MALLOC(sig_fileurl, sig_len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); + snprintf(sig_fileurl, sig_len, "%s.sig", payload->fileurl); + + ret = handle->fetchcb(sig_fileurl, localpath, payload->force); + + free(sig_fileurl); + } + if (ret != -1) { + success = 1; + } } } else { for(s = payload->servers; s; s = s->next) { @@ -892,12 +909,26 @@ int _alpm_download(alpm_handle_t *handle, snprintf(fileurl, len, "%s/%s", server, payload->filepath); ret = handle->fetchcb(fileurl, localpath, payload->force); - free(fileurl); if (ret != -1) { - success = 1; - break; + /* Let's check if client requested downloading accompanion *.sig file */ + if(payload->download_signature) { + char *sig_fileurl; + + size_t sig_len = strlen(fileurl) + 5; + MALLOC(sig_fileurl, sig_len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); + snprintf(sig_fileurl, sig_len, "%s.sig", fileurl); + + ret = handle->fetchcb(sig_fileurl, localpath, payload->force); + + free(sig_fileurl); + } + if (ret != -1) { + success = 1; + break; + } } + free(fileurl); } } if(!success && !payload->errors_ok) { -- 2.31.1

1 0

[pacman-dev] [PATCH] call download progress callback for signatures
by Andrew Gregory 30 Apr '21

30 Apr '21

pacman may not care about them, but other front-ends do. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> --- lib/libalpm/dload.c | 18 +++++++----------- src/pacman/callback.c | 12 ++++++++++++ 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 935f5418..8fa46bb2 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -166,11 +166,6 @@ static int dload_progress_cb(void *file, curl_off_t dltotal, curl_off_t dlnow, off_t current_size, total_size; alpm_download_event_progress_t cb_data = {0}; - /* do not print signature files progress bar */ - if(payload->signature) { - return 0; - } - /* avoid displaying progress bar for redirects with a body */ if(payload->respcode >= 300) { return 0; @@ -669,7 +664,7 @@ cleanup: unlink(payload->tempfile_name); } - if(handle->dlcb && !payload->signature) { + if(handle->dlcb) { alpm_download_event_completed_t cb_data = {0}; cb_data.total = bytes_dl; cb_data.result = ret; @@ -782,6 +777,12 @@ static int curl_add_payload(alpm_handle_t *handle, CURLM *curlm, curl_easy_setopt(curl, CURLOPT_WRITEDATA, payload->localf); curl_multi_add_handle(curlm, curl); + + if(handle->dlcb) { + alpm_download_event_init_t cb_data = {.optional = payload->errors_ok}; + handle->dlcb(handle->dlcb_ctx, payload->remote_name, ALPM_DOWNLOAD_INIT, &cb_data); + } + return 0; cleanup: @@ -810,11 +811,6 @@ static int curl_download_internal(alpm_handle_t *handle, struct dload_payload *payload = payloads->data; if(curl_add_payload(handle, curlm, payload, localpath) == 0) { - if(handle->dlcb && !payload->signature) { - alpm_download_event_init_t cb_data = {.optional = payload->errors_ok}; - handle->dlcb(handle->dlcb_ctx, payload->remote_name, ALPM_DOWNLOAD_INIT, &cb_data); - } - payloads = payloads->next; } else { /* The payload failed to start. Do not start any new downloads. diff --git a/src/pacman/callback.c b/src/pacman/callback.c index c3563af3..ae7e1bed 100644 --- a/src/pacman/callback.c +++ b/src/pacman/callback.c @@ -1070,10 +1070,22 @@ static void dload_complete_event(const char *filename, alpm_download_event_compl } } +static int strendswith(const char *haystack, const char *needle) +{ + size_t hlen = strlen(haystack), nlen = strlen(needle); + return hlen >= nlen && strcmp(haystack + hlen - nlen, needle) == 0; +} + /* Callback to handle display of download progress */ void cb_download(void *ctx, const char *filename, alpm_download_event_type_t event, void *data) { (void)ctx; + + /* do not print signature files progress bar */ + if(strendswith(filename, ".sig")) { + return; + } + if(event == ALPM_DOWNLOAD_INIT) { dload_init_event(filename, data); } else if(event == ALPM_DOWNLOAD_PROGRESS) { -- 2.31.1

1 0

[pacman-dev] Patch Policy Inquiry
by Sijie Bu 29 Apr '21

29 Apr '21

Hello, Not sure if I'm posting it to the right place, but this is a developer-related question so hopefully it's in the right place. I will be starting an internship project with CIP United, a company specialized in MIPS processors, and will be porting Arch to MIPS64 R6. And my project supervisor would like to know pacman's patch policies for architecture-specific modifications. I have already asked on arch-ports list and was told by people from Arch Linux 32 that the pacman codebase is generally portable, but I see them maintaining a fork for certain i486-specific optimizations. I have also looked at the readme files (HACKING etc.) but was unable to find a clear statement of your patch policies. Therefore, I would like to know some details about your patch policies, especially the ones regarding non-amd64 architectures. Thank you for your time and have a nice day. Sincerely Tom

3 2

[pacman-dev] [PATCH] alpm_db_update: indicate if dbs were up to date
by Andrew Gregory 27 Apr '21

27 Apr '21

Restore the prior indicator whether or not databases were up to date. 0 is used to indicate if *any* db was actually updated as callers are more likely to care about that than if *all* dbs were updated. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> --- Please include this for v6.0. Pacutils uses the up-to-date indicator. lib/libalpm/alpm.h | 3 ++- lib/libalpm/dload.c | 35 +++++++++++++++++++++-------------- lib/libalpm/sync.c | 2 +- 3 files changed, 24 insertions(+), 16 deletions(-) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 2e99d4d8..f47ddc1d 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -1356,7 +1356,8 @@ int alpm_db_remove_server(alpm_db_t *db, const char *url); * @param dbs list of package databases to update * @param force if true, then forces the update, otherwise update only in case * the databases aren't up to date - * @return 0 on success, -1 on error (pm_errno is set accordingly) + * @return 0 on success, -1 on error (pm_errno is set accordingly), + * 1 if all databases are up to to date */ int alpm_db_update(alpm_handle_t *handle, alpm_list_t *dbs, int force); diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 57d25383..00a729ec 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -788,6 +788,10 @@ cleanup: return ret; } +/* Returns -1 if an error happened for a required file + * Returns 0 if a payload was actually downloaded + * Returns 1 if no files were downloaded and all errors were non-fatal + */ static int curl_download_internal(alpm_handle_t *handle, alpm_list_t *payloads /* struct dload_payload */, const char *localpath) @@ -795,6 +799,7 @@ static int curl_download_internal(alpm_handle_t *handle, int active_downloads_num = 0; int err = 0; int max_streams = handle->parallel_downloads; + int updated = 0; /* was a file actually updated */ CURLM *curlm = handle->curlm; while(active_downloads_num > 0 || payloads) { @@ -846,6 +851,8 @@ static int curl_download_internal(alpm_handle_t *handle, */ payloads = NULL; err = -1; + } else if(ret == 0) { + updated = 1; } } else { _alpm_log(handle, ALPM_LOG_ERROR, _("curl transfer error: %d\n"), msg->msg); @@ -854,11 +861,15 @@ static int curl_download_internal(alpm_handle_t *handle, } _alpm_log(handle, ALPM_LOG_DEBUG, "curl_download_internal return code is %d\n", err); - return err; + return err ? -1 : updated ? 0 : 1; } #endif +/* Returns -1 if an error happened for a required file + * Returns 0 if a payload was actually downloaded + * Returns 1 if no files were downloaded and all errors were non-fatal + */ int _alpm_download(alpm_handle_t *handle, alpm_list_t *payloads /* struct dload_payload */, const char *localpath) @@ -871,20 +882,18 @@ int _alpm_download(alpm_handle_t *handle, #endif } else { alpm_list_t *p; + int updated = 0; for(p = payloads; p; p = p->next) { struct dload_payload *payload = p->data; alpm_list_t *s; - int success = 0; + int ret = -1; if(payload->fileurl) { - if (handle->fetchcb(payload->fileurl, localpath, payload->force) != -1) { - success = 1; - } + ret = handle->fetchcb(payload->fileurl, localpath, payload->force); } else { - for(s = payload->servers; s; s = s->next) { + for(s = payload->servers; s && ret == -1; s = s->next) { const char *server = s->data; char *fileurl; - int ret; size_t len = strlen(server) + strlen(payload->filepath) + 2; MALLOC(fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); @@ -892,18 +901,16 @@ int _alpm_download(alpm_handle_t *handle, ret = handle->fetchcb(fileurl, localpath, payload->force); free(fileurl); - - if (ret != -1) { - success = 1; - break; - } } } - if(!success && !payload->errors_ok) { + + if(ret == -1 && !payload->errors_ok) { RET_ERR(handle, ALPM_ERR_EXTERNAL_DOWNLOAD, -1); + } else if(ret == 0) { + updated = 1; } } - return 0; + return updated ? 0 : 1; } } diff --git a/lib/libalpm/sync.c b/lib/libalpm/sync.c index 7fa50a13..2419cc69 100644 --- a/lib/libalpm/sync.c +++ b/lib/libalpm/sync.c @@ -815,7 +815,7 @@ static int download_files(alpm_handle_t *handle) } ret = _alpm_download(handle, payloads, cachedir); - if(ret != 0) { + if(ret == -1) { event.type = ALPM_EVENT_PKG_RETRIEVE_FAILED; EVENT(handle, &event); _alpm_log(handle, ALPM_LOG_WARNING, _("failed to retrieve some files\n")); -- 2.31.1

1 0

[pacman-dev] [PATCH 1/3] pmpkg: add makepkg_bytes
by Andrew Gregory 27 Apr '21

27 Apr '21

Builds the package file in memory. Useful with the built-in server. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> --- test/pacman/pmpkg.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/test/pacman/pmpkg.py b/test/pacman/pmpkg.py index 8ff1ec1e..01ed4e0a 100644 --- a/test/pacman/pmpkg.py +++ b/test/pacman/pmpkg.py @@ -98,7 +98,12 @@ def parse_filename(name): filename, extra = filename.split("|") return filename - def makepkg(self, path): + def makepkg_bytes(self): + buf = BytesIO(); + self.makepkg(fileobj=buf) + return buf.getvalue() + + def makepkg(self, path=None, fileobj=None): """Creates an Arch Linux package archive. A package archive is generated in the location 'path', based on the data @@ -138,11 +143,12 @@ def makepkg(self, path): if any(self.install.values()): archive_files.append((".INSTALL", self.installfile())) - self.path = os.path.join(path, self.filename()) - util.mkdir(os.path.dirname(self.path)) + if path: + self.path = os.path.join(path, self.filename()) + util.mkdir(os.path.dirname(self.path)) # Generate package metadata - tar = tarfile.open(self.path, "w:gz", format=tarfile.GNU_FORMAT) + tar = tarfile.open(name=self.path, fileobj=fileobj, mode="w:gz", format=tarfile.GNU_FORMAT) for name, data in archive_files: info = tarfile.TarInfo(name) info.size = len(data) -- 2.31.1

1 2