- Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH] libalpm: prevent download error pages ending up in package files
by Allan McRae 02 Jun '21

02 Jun '21

Some servers respond with error pages (e.g. 404.html) when a package is not present. These were getting written to packages before moving onto the next server. Reset the download progress on 400+ error conditions to avoid this. Signed-off-by: Allan McRae <allan(a)archlinux.org> --- This is a replacement for Morgan's patch: libalpm: fix 404 pages ending up inside package downloads This approach is less heavy handed, allowing previously partially downloaded packages to be continued. I'm still not sure if there is total download amount tracking issue, but error downloads should be small... lib/libalpm/dload.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 2d8b4d6d..3eea998b 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -510,6 +510,16 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, payload->remote_name, hostname, payload->error_buffer); server_soft_error(handle, payload->fileurl); } + + fflush(payload->localf); + if(fstat(fileno(payload->localf), &st) == 0 && st.st_size != payload->initial_size) { + /* an html error page was written to the file, reset it */ + if(ftruncate(fileno(payload->localf), payload->initial_size)) { + RET_ERR(handle, ALPM_ERR_SYSTEM, -1); + } + fseek(payload->localf, payload->initial_size, SEEK_SET); + } + if(curl_retry_next_server(curlm, curl, payload) == 0) { (*active_downloads_num)++; return 2; -- 2.31.1

1 0

[pacman-dev] Questions about stderr/stdout behaviour on failures
by Oskar Roesler 02 Jun '21

02 Jun '21

Hello, when file conflicts in packages occur, why does the actual conflict get written to stdout and only an non-informative error message to stderr? Example: Stderr only: warning: dependency cycle detected: warning: libglvnd will be installed before its mesa dependency error: failed to commit transaction (conflicting files) ==> ERROR: 'pacman' failed to install missing dependencies. warning: dependency cycle detected: warning: harfbuzz will be installed before its freetype2 dependency warning: dependency cycle detected: warning: libglvnd will be installed before its mesa dependency error: failed to commit transaction (conflicting files) ==> ERROR: 'pacman' failed to install missing dependencies. Both outputs combined: :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... error: failed to commit transaction (conflicting files) /opt/ros/melodic/lib/python3.9/site-packages/libgtest.so exists in both 'ros-melodic-moveit-core' and 'ros-melodic-tf' /opt/ros/melodic/lib/python3.9/site-packages/libgtest.so.1.10.0 exists in both 'ros-melodic-moveit-core' and 'ros-melodic-tf' Errors occurred, no packages were upgraded. ==> ERROR: 'pacman' failed to install missing dependencies. ==> Missing dependencies: -> ros-build-tools -> ros-melodic-message-filters -> ros-melodic-moveit-core -> ros-melodic-moveit-msgs As e.g. the dependency cycle errors get fully written to stderr, is this a bug or itended? Imho this one line more is important to stderr, otherwise the whole stderr log helps me as much as the returncode. Regards, Oskar

2 2

[pacman-dev] [PATCH 1/1] RFC: handle soft failure in downloads
by Christian Hesse 02 Jun '21

02 Jun '21

From: Christian Hesse <mail(a)eworm.de> A server can indicate a soft failure by returning 412 (Precondition Failed). Only a warning (instead of error) message is logged and download is retried with next server, without counting server error. This http code can be used by servers that are not expected to be complete, for example when serving a local cache. Signed-off-by: Christian Hesse <mail(a)eworm.de> --- lib/libalpm/dload.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index f6a4f012..7116cb03 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -515,13 +515,23 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, if(!payload->errors_ok) { handle->pm_errno = ALPM_ERR_RETRIEVE; - /* non-translated message is same as libcurl */ - snprintf(payload->error_buffer, sizeof(payload->error_buffer), + if(payload->respcode == 412) { + /* A server can indicate a soft failure by returning 412 + (Precondition Failed). Only a debug message is logged and + download is retried with next server, without counting + server error. */ + _alpm_log(handle, ALPM_LOG_WARNING, + _("file '%s' is currently not available from %s\n"), + payload->remote_name, hostname); + } else { + /* non-translated message is same as libcurl */ + snprintf(payload->error_buffer, sizeof(payload->error_buffer), "The requested URL returned error: %ld", payload->respcode); - _alpm_log(handle, ALPM_LOG_ERROR, + _alpm_log(handle, ALPM_LOG_ERROR, _("failed retrieving file '%s' from %s : %s\n"), payload->remote_name, hostname, payload->error_buffer); - server_soft_error(handle, payload->fileurl); + server_soft_error(handle, payload->fileurl); + } } if(curl_retry_next_server(curlm, curl, payload) == 0) { (*active_downloads_num)++;

6 19

[pacman-dev] [PATCH v2] libalpm: fix 404 pages ending up inside package downloads
by morganamilo 21 May '21

21 May '21

Previously successfull but non 200 responses would still be written to disk before trying the next server. This would lead to the next download resuming with the html response in the file. --- v2: ensure allow_resume is passed to callback Signed-off-by: morganamilo <morganamilo(a)archlinux.org> --- lib/libalpm/dload.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 2d8b4d6d..048cec5e 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -404,7 +404,7 @@ static FILE *create_tempfile(struct dload_payload *payload, const char *localpat } /* Return 0 if retry was successful, -1 otherwise */ -static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload *payload) +static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload *payload, int allow_resume) { const char *server; size_t len; @@ -428,10 +428,9 @@ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); snprintf(payload->fileurl, len, "%s/%s", server, payload->filepath); - fflush(payload->localf); - if(payload->allow_resume && stat(payload->tempfile_name, &st) == 0) { + if(allow_resume && stat(payload->tempfile_name, &st) == 0) { /* a previous partial download exists, resume from end of file. */ payload->tempfile_openmode = "ab"; curl_easy_setopt(curl, CURLOPT_RESUME_FROM_LARGE, (curl_off_t)st.st_size); @@ -449,7 +448,7 @@ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload if(handle->dlcb && !payload->signature) { alpm_download_event_retry_t cb_data; - cb_data.resume = payload->allow_resume; + cb_data.resume = allow_resume; handle->dlcb(handle->dlcb_ctx, payload->remote_name, ALPM_DOWNLOAD_RETRY, &cb_data); } @@ -510,7 +509,7 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, payload->remote_name, hostname, payload->error_buffer); server_soft_error(handle, payload->fileurl); } - if(curl_retry_next_server(curlm, curl, payload) == 0) { + if(curl_retry_next_server(curlm, curl, payload, false) == 0) { (*active_downloads_num)++; return 2; } else { @@ -537,7 +536,7 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, _("failed retrieving file '%s' from %s : %s\n"), payload->remote_name, hostname, payload->error_buffer); server_hard_error(handle, payload->fileurl); - if(curl_retry_next_server(curlm, curl, payload) == 0) { + if(curl_retry_next_server(curlm, curl, payload, payload->allow_resume) == 0) { (*active_downloads_num)++; return 2; } else { @@ -556,7 +555,7 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, "failed retrieving file '%s' from %s : %s\n", payload->remote_name, hostname, payload->error_buffer); } - if(curl_retry_next_server(curlm, curl, payload) == 0) { + if(curl_retry_next_server(curlm, curl, payload, payload->allow_resume) == 0) { (*active_downloads_num)++; return 2; } else { -- 2.31.1

1 1

Re: [pacman-dev] Translation of man pages
by Mario Blättermann 21 May '21

21 May '21

Hello, see pt_BR.po attached. Best Regards, Mario

1 0

Re: [pacman-dev] Translation of man pages
by Mario Blättermann 21 May '21

21 May '21

Hello, see fr.po attached. Best Regards, Mario

1 0

[pacman-dev] [PATCH] libalpm: call retry events for sig downlods
by morganamilo 21 May '21

21 May '21

Around the same time retry events were added, there was a patch to pass sig download events to the frontend. The retry code was not updated to account for this. Signed-off-by: morganamilo <morganamilo(a)archlinux.org> diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 048cec5e..37e8ae34 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -446,7 +446,7 @@ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload fseek(payload->localf, 0, SEEK_SET); } - if(handle->dlcb && !payload->signature) { + if(handle->dlcb) { alpm_download_event_retry_t cb_data; cb_data.resume = allow_resume; handle->dlcb(handle->dlcb_ctx, payload->remote_name, ALPM_DOWNLOAD_RETRY, &cb_data); -- 2.31.1

1 0

[pacman-dev] [PATCH 1/1] doc/pacman.conf: ParallelDownloads expects a value
by Christian Hesse 21 May '21

21 May '21

From: Christian Hesse <mail(a)eworm.de> This can not be specified on its own but requires a value. Signed-off-by: Christian Hesse <mail(a)eworm.de> --- doc/pacman.conf.5.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/pacman.conf.5.asciidoc b/doc/pacman.conf.5.asciidoc index fd765f3d..77a3907f 100644 --- a/doc/pacman.conf.5.asciidoc +++ b/doc/pacman.conf.5.asciidoc @@ -202,7 +202,7 @@ Options Disable defaults for low speed limit and timeout on downloads. Use this if you have issues downloading files with proxy and/or security gateway. -*ParallelDownloads*:: +*ParallelDownloads =* ...:: Specifies number of concurrent download streams. The value needs to be a positive integer. If this config option is not set then only one download stream is used (i.e. downloads happen sequentially).

2 1

[pacman-dev] [PATCH] pactest: add test cause for servers returning 404 with body
by morganamilo 20 May '21

20 May '21

Signed-off-by: morganamilo <morganamilo(a)archlinux.org> --- test/pacman/meson.build | 1 + .../tests/sync-failover-404-with-body.py | 24 +++++++++++++++++++ 2 files changed, 25 insertions(+) create mode 100644 test/pacman/tests/sync-failover-404-with-body.py diff --git a/test/pacman/meson.build b/test/pacman/meson.build index 0e75b3ee..ecab75b2 100644 --- a/test/pacman/meson.build +++ b/test/pacman/meson.build @@ -173,6 +173,7 @@ pacman_tests = [ 'tests/symlink012.py', 'tests/symlink020.py', 'tests/symlink021.py', + 'tests/sync-failover-404-with-body.py', 'tests/sync-install-assumeinstalled.py', 'tests/sync-nodepversion01.py', 'tests/sync-nodepversion02.py', diff --git a/test/pacman/tests/sync-failover-404-with-body.py b/test/pacman/tests/sync-failover-404-with-body.py new file mode 100644 index 00000000..8022461d --- /dev/null +++ b/test/pacman/tests/sync-failover-404-with-body.py @@ -0,0 +1,24 @@ +self.description = "server failover after 404" +self.require_capability("curl") + +p1 = pmpkg('pkg') +self.addpkg2db('sync', p1) + +url_broke = self.add_simple_http_server({ + '/{}'.format(p1.filename()): { + 'code': 404, + 'body': 'a', + } +}) +url_good = self.add_simple_http_server({ + '/{}'.format(p1.filename()): p1.makepkg_bytes(), +}) + +self.db['sync'].option['Server'] = [ url_broke, url_good ] +self.db['sync'].syncdir = False +self.cachepkgs = False + +self.args = '-S pkg' + +self.addrule("PACMAN_RETCODE=0") +self.addrule("PKG_EXIST=pkg") -- 2.31.1

1 0

[pacman-dev] [PATCH] libalpm: fix 404 pages ending up inside package downloads
by morganamilo 20 May '21

20 May '21

Previously successfull but non 200 responses would still be written to disk before trying the next server. This would lead to the next download resuming with the html response in the file. Signed-off-by: morganamilo <morganamilo(a)archlinux.org> --- lib/libalpm/dload.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 2d8b4d6d..f6a4f012 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -403,6 +403,17 @@ static FILE *create_tempfile(struct dload_payload *payload, const char *localpat return fp; } +/* truncates tempfile without deleting it */ +static int clear_tempfile(struct dload_payload *payload) { + fflush(payload->localf); + if(ftruncate(fileno(payload->localf), 0)) { + return -1; + } + fseek(payload->localf, 0, SEEK_SET); + return 0; +} + + /* Return 0 if retry was successful, -1 otherwise */ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload *payload) { @@ -428,11 +439,9 @@ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); snprintf(payload->fileurl, len, "%s/%s", server, payload->filepath); - - fflush(payload->localf); - if(payload->allow_resume && stat(payload->tempfile_name, &st) == 0) { /* a previous partial download exists, resume from end of file. */ + fflush(payload->localf); payload->tempfile_openmode = "ab"; curl_easy_setopt(curl, CURLOPT_RESUME_FROM_LARGE, (curl_off_t)st.st_size); _alpm_log(handle, ALPM_LOG_DEBUG, @@ -441,10 +450,9 @@ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload payload->initial_size = st.st_size; } else { /* we keep the file for a new retry but remove its data if any */ - if(ftruncate(fileno(payload->localf), 0)) { + if(clear_tempfile(payload)) { RET_ERR(handle, ALPM_ERR_SYSTEM, -1); } - fseek(payload->localf, 0, SEEK_SET); } if(handle->dlcb && !payload->signature) { @@ -500,6 +508,11 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, _alpm_log(handle, ALPM_LOG_DEBUG, "%s: response code %ld\n", payload->remote_name, payload->respcode); if(payload->respcode >= 400) { + /* make sure 404 pages and similar don't end up in the file */ + if(clear_tempfile(payload)) { + RET_ERR(handle, ALPM_ERR_SYSTEM, -1); + } + if(!payload->errors_ok) { handle->pm_errno = ALPM_ERR_RETRIEVE; /* non-translated message is same as libcurl */ -- 2.31.1

3 2