September 2021 - Pacman-dev - lists.archlinux.org

[pacman-dev] [Git][pacman/pacman][master] 2 commits: Update NEWS for pacman-6.0.1 release
by Allan McRae (＠allan) 04 Sep '21

04 Sep '21

Allan McRae pushed to branch master at Pacman / Pacman Commits: d5e2c0a5 by Allan McRae at 2021-09-04T19:36:20+10:00 Update NEWS for pacman-6.0.1 release Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 0a6fecd0 by Allan McRae at 2021-09-04T19:36:22+10:00 Release pacman-6.0.1 Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 3 changed files: - NEWS - doc/index.asciidoc - meson.build View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/70d0b2c4b92443a96217be… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/70d0b2c4b92443a96217be… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[pacman-dev] [PATCH] Update NEWS for pacman-6.0.1 release
by Allan McRae 04 Sep '21

04 Sep '21

Signed-off-by: Allan McRae <allan(a)archlinux.org> --- NEWS | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/NEWS b/NEWS index ae936661..223f84b6 100644 --- a/NEWS +++ b/NEWS @@ -1,5 +1,20 @@ VERSION DESCRIPTION ----------------------------------------------------------------------------- +6.0.1 - Prevent download error pages ending up in package files + (FS#71083) + - Give -U downloads a random .part file name if needed + W(FS#71464) + - Fix downloading signatures with redirecting URLs (FS#71148) + - Fix double free when importing PGP keys (FS#71107) + - Ensure signature files are named after original file + following redirects (FS#71274) + - Order downloads by size - largest to smallest (FS#70172) + - Fix reproducibility of man pages (FS#71154) + - makepkg: + - Fix stripping debug symbols with binutils 2.37 (FS#71722) + - Export PYTHONHASHSEED for reproducible python packages + - pacman-key: + - Quieten trust db checks. 6.0.0 - internal downloader can retrieve files in parallel (FS#20056) - an additional progress bar is added to track total download progress. This replaces the previous TotalDownload option. -- 2.33.0

1 0

[pacman-dev] [Git][pacman/pacman][master] Include az_AZ translation files
by Allan McRae (＠allan) 04 Sep '21

04 Sep '21

Allan McRae pushed to branch master at Pacman / Pacman Commits: 70d0b2c4 by Allan McRae at 2021-09-04T18:02:39+10:00 Include az_AZ translation files Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 3 changed files: - + lib/libalpm/po/az_AZ.po - + scripts/po/az_AZ.po - + src/pacman/po/az_AZ.po View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/70d0b2c4b92443a96217be0… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/70d0b2c4b92443a96217be0… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[pacman-dev] [Git][pacman/pacman][master] 3 commits: pacman-key: Add --quiet to a few more gpg invocations
by Allan McRae (＠allan) 04 Sep '21

04 Sep '21

Allan McRae pushed to branch master at Pacman / Pacman Commits: 160f5bec by Daan De Meyer at 2021-09-04T13:34:58+10:00 pacman-key: Add --quiet to a few more gpg invocations Currently, when running pacman-key --populate, gpg prints the trustdb check output once for each locally signed and revoked key. When bootstrapping a new container image, about 50 keys get signed and revoked which leads to a huge amount of output when running pacman-key which is repeated 50x. To avoid overloading the user with gpg output, we add --quiet to the gpg calls generating the trustdb checking output to silence those calls which gets rid of the trustdb check output on the terminal. Signed-off-by: Daan De Meyer <daan.j.demeyer(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - c7e4f459 by Daan De Meyer at 2021-09-04T13:43:26+10:00 pacman-key: Reduce gpg trustdb checks Every time we modify gpg's state by signing or revoking a key, gpg marks the trustdb as stale and rechecks it the next time key_is_lsigned() or key_is_revoked() is called. Currently, we alternate calls signing of keys and calling key_is_lsigned() (idem for revoking) which means that for each key we sign (or revoke), gpg will check the trustdb once. To avoid checking the trustb so many times, we can simply do all the key_is_lsigned() and key_is_revoked() checks upfront. Inbetween read operations the trustdb is not marked stale and inbetween write operations the trustdb is also not marked stale. This reduces the amount of trustdb checks from 50 to 1. Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 4c3c0e06 by Allan McRae at 2021-09-04T16:16:20+10:00 Update translations from Transifex Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 25 changed files: - lib/libalpm/po/LINGUAS - lib/libalpm/po/ar.po - lib/libalpm/po/ast.po - lib/libalpm/po/bg.po - lib/libalpm/po/br.po - lib/libalpm/po/ca.po - lib/libalpm/po/cs.po - lib/libalpm/po/da.po - lib/libalpm/po/de.po - lib/libalpm/po/el.po - lib/libalpm/po/en_GB.po - lib/libalpm/po/eo.po - lib/libalpm/po/es.po - lib/libalpm/po/es_419.po - lib/libalpm/po/eu.po - lib/libalpm/po/eu_ES.po - lib/libalpm/po/fi.po - lib/libalpm/po/fr.po - lib/libalpm/po/gl.po - lib/libalpm/po/hi.po - lib/libalpm/po/hr.po - lib/libalpm/po/hu.po - lib/libalpm/po/id.po - lib/libalpm/po/it.po - lib/libalpm/po/ja.po View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/5da4af2b5dcb6f214a93fd… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/5da4af2b5dcb6f214a93fd… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[pacman-dev] [PATCH] pacman-key: Reorder key_is_lsigned() and key_is_revoked() checks to reduce gpg trustdb checks
by Daan De Meyer 04 Sep '21

04 Sep '21

Every time we modify gpg's state by signing or revoking a key, gpg marks the trustdb as stale and rechecks it the next time key_is_lsigned() or key_is_revoked() is called. Currently, we alternate calls signing of keys and calling key_is_lsigned() (idem for revoking) which means that for each key we sign (or revoke), gpg will check the trustdb once. When populating the keyring, this means the trustdb is checked roughly 50 times. Each time the trustdb is checked, we get the following output once: ``` gpg: checking the trustdb gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 1EB2638FF56C0C53: no user ID for key signature packet of class 10 gpg: key 1EB2638FF56C0C53: no user ID for key signature packet of class 10 gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 6 trust: 0-, 0q, 0n, 0m, 0f, 1u gpg: depth: 1 valid: 6 signed: 83 trust: 0-, 0q, 0n, 6m, 0f, 0u gpg: depth: 2 valid: 78 signed: 25 trust: 78-, 0q, 0n, 0m, 0f, 0u gpg: next trustdb check due at 2021-12-01 ``` This repeated 50x leads to incredibly verbose output from pacman-key. To avoid checking the trustb so many times, we can simply do all the key_is_lsigned() and key_is_revoked() checks upfront. Inbetween read operations the trustdb is not marked stale and inbetween write operations the trustdb is also not marked stale. This reduces the amount of trustdb checks from 50 to 1. The output of pacman-key --populate now looks as follows: ``` gpg: /var/tmp/mkosi-pqii6f64/root/etc/pacman.d/gnupg/trustdb.gpg: trustdb created gpg: no ultimately trusted keys found gpg: starting migration from earlier GnuPG versions gpg: porting secret keys from '/var/tmp/mkosi-pqii6f64/root/etc/pacman.d/gnupg/secring.gpg' to gpg-agent gpg: migration succeeded ==> Generating pacman master key. This may take some time. gpg: Generating pacman keyring master key... gpg: key D429469316A97B49 marked as ultimately trusted gpg: directory '/var/tmp/mkosi-pqii6f64/root/etc/pacman.d/gnupg/openpgp-revocs.d' created gpg: revocation certificate stored as '/var/tmp/mkosi-pqii6f64/root/etc/pacman.d/gnupg/openpgp-revocs.d/059BC618296EDA8B4614FAD4D429469316A97B49.rev' gpg: Done ==> Updating trust database... gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u ==> Appending keys from archlinux.gpg... ==> Locally signing trusted keys in keyring... -> Locally signed 6 keys. ==> Importing owner trust values... gpg: setting ownertrust to 4 gpg: setting ownertrust to 4 gpg: setting ownertrust to 4 gpg: setting ownertrust to 4 gpg: inserting ownertrust of 4 gpg: setting ownertrust to 4 ==> Disabling revoked keys in keyring... -> Disabled 44 keys. ==> Updating trust database... gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10 gpg: key 1EB2638FF56C0C53: no user ID for key signature packet of class 10 gpg: key 1EB2638FF56C0C53: no user ID for key signature packet of class 10 gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 6 trust: 0-, 0q, 0n, 0m, 0f, 1u gpg: depth: 1 valid: 6 signed: 83 trust: 0-, 0q, 0n, 6m, 0f, 0u gpg: depth: 2 valid: 78 signed: 25 trust: 78-, 0q, 0n, 0m, 0f, 0u gpg: next trustdb check due at 2021-12-01 ``` --- scripts/pacman-key.sh.in | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/scripts/pacman-key.sh.in b/scripts/pacman-key.sh.in index 50342649..721079b7 100644 --- a/scripts/pacman-key.sh.in +++ b/scripts/pacman-key.sh.in @@ -333,12 +333,29 @@ populate_keyring() { # skip blank lines, comments; these are valid in this file [[ -z $key_id || ${key_id:0:1} = \# ]] && continue + if key_is_lsigned "$key_id" ; then + continue + fi + # Mark this key to be lsigned trusted_ids[$key_id]=$keyring done < "${KEYRING_IMPORT_DIR}/${keyring}-trusted" fi done + local -A revoked_ids + for keyring in "${KEYRINGIDS[@]}"; do + if [[ -s $KEYRING_IMPORT_DIR/$keyring-revoked ]]; then + while read -r key_id; do + if key_is_revoked "$key_id" ; then + continue + fi + + revoked_ids["$key_id"]=1 + done <"$KEYRING_IMPORT_DIR/$keyring-revoked" + fi + done + if (( ${#trusted_ids[@]} > 0 )); then msg "$(gettext "Locally signing trusted keys in keyring...")" lsign_keys "${!trusted_ids[@]}" @@ -350,22 +367,10 @@ populate_keyring() { done fi - local -A revoked_ids - for keyring in "${KEYRINGIDS[@]}"; do - if [[ -s $KEYRING_IMPORT_DIR/$keyring-revoked ]]; then - while read -r key_id; do - revoked_ids["$key_id"]=1 - done <"$KEYRING_IMPORT_DIR/$keyring-revoked" - fi - done - if (( ${#revoked_ids[@]} > 0 )); then local key_count=0 msg "$(gettext "Disabling revoked keys in keyring...")" for key_id in "${!revoked_ids[@]}"; do - if key_is_revoked "$key_id" ; then - continue - fi if (( VERBOSE )); then msg2 "$(gettext "Disabling key %s...")" "${key_id}" fi @@ -485,9 +490,6 @@ lsign_keys() { local ret=0 local key_count=0 for key_id in "$@"; do - if key_is_lsigned "$key_id" ; then - continue - fi if (( VERBOSE )); then msg2 "$(gettext "Locally signing key %s...")" "${key_id}" fi -- 2.33.0

2 2

[pacman-dev] [Git][pacman/pacman][master] 7 commits: libmakepkg: reproducibilty for python packages
by Allan McRae (＠allan) 04 Sep '21

04 Sep '21

Allan McRae pushed to branch master at Pacman / Pacman Commits: 1c5a5688 by Allan McRae at 2021-08-08T22:49:32+10:00 libmakepkg: reproducibilty for python packages Arch Linux has been setting PYTHONHASHSEED=0 to create deterministic .pyc files. After a thorough review by the Arch Security Team, setting this variable was determined not to generated vulnerable .pyc files, as when the loader loads the .pyc file and unmarshalls it, the internal runtime will just populate the unordered data structures and use a new runtime hash for them. Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - c0026caa by morganamilo at 2021-09-04T10:33:51+10:00 libalpm: Give -U downloads a random .part name if needed archweb's download links all ended in /download. This cause all the temp files to be named download.part. With parallel downloads this results in multiple downloads to go to the same temp file and breaks the transaction. Assign random temporary filenames to downloads from URLs that are either missing a filename, or if the filename does not contain at least three hyphens (as a well formed package filename does). While this approach to determining when to use a temporary filename is not 100% foolproof, it does keep nice looking download progress bar names when a proper package filename is given. The only downside of not using temporary files when provided with a filename with three or more hyphens is URLs created specifically to bypass temporary filename usage can not be downloaded in parallel. We probably do not want to download packages from such URLs anyway. Fixes FS#71464 Modified-by: Allan McRae (do not use temporary files for realish URLs) Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 2ec6de96 by morganamilo at 2021-09-04T10:34:00+10:00 only use effective url for urls containing .db or .pkg Github and other sites redirect their downloads to a cdn. So the download http://foo.org/myrepo.db may redirect to something like https://cdn.foo.org/83749327439. This then causes pacman to try and download the sig as https://cdn.foo.org/83749327439.sig which is incorrect. In this case pacman should append .sig to the original url. However urls like https://archlinux.org/packages/community/x86_64/0ad/download/ Redirect to the mirror, so .sig has to appended after the redirects and not before. So we decide if we should append .sig on the original or effective url based on if the effective url (minus the query part) has .db or .pkg in it. Fixes FS#71148 --- v2: move variable decleration to start of block v3: use dbext instead of db - - - - - f951282b by morganamilo at 2021-09-04T10:34:00+10:00 pactest: add tests for downloading packages from a cdn Test for downloads that redirect to some sort of cdn where the redirected url does not relate to the original filename. Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - efb714b3 by Charlie Sale at 2021-09-04T10:34:00+10:00 Order downloads by descending max_size When downloading in parallel, sort by package size so that the larger packages are queued first to fully leverage parallelism. Addresses FS#70172 Signed-off-by: Charlie Sale <softwaresale01(a)gmail.com> Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - cf923e73 by Hugo Osvaldo Barrera at 2021-09-04T10:34:00+10:00 Update broken links pointing to git.archlinux.org All of these links are broken since the recent move to gitlab.archlinux.org. A few projects are, apparently, only available on GitHub, so I've linked to that source (hopefully that's only temporary). For git-clone URLs, I've opted for the https URLs since those can be used by anyone -- whereas the ssh URLs require the user to be registered on the gitlab instance which is not open to the public yet. Signed-off-by: Hugo Osvaldo Barrera <hugo(a)barrera.io> Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 5da4af2b by Hugo Osvaldo Barrera at 2021-09-04T10:34:00+10:00 Delete the "Other Utilities" section Signed-off-by: Hugo Osvaldo Barrera <hugo(a)barrera.io> Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 10 changed files: - doc/index.asciidoc - doc/submitting-patches.asciidoc - doc/translation-help.asciidoc - lib/libalpm/dload.c - lib/libalpm/dload.h - scripts/libmakepkg/meson.build - + scripts/libmakepkg/reproducible.sh.in - + scripts/libmakepkg/reproducible/meson.build - + scripts/libmakepkg/reproducible/python.sh.in - test/pacman/tests/upgrade-download-pkg-and-sig-with-filename.py Changes: ===================================== doc/index.asciidoc ===================================== @@ -59,11 +59,11 @@ configuration files dealing with pacman. Changelog ~~~~~~~~~ For a good idea of what is going on in pacman development, take a look at the -link:https://git.archlinux.org/pacman.git/[Git summary page] for the +link:https://gitlab.archlinux.org/pacman/pacman[Git summary page] for the project. See the most recent -link:https://git.archlinux.org/pacman.git/tree/NEWS[NEWS] +link:https://gitlab.archlinux.org/pacman/pacman/-/blob/master/NEWS[NEWS] file for a not-as-frequently-updated list of changes. However, this should contain the biggest changes in a format more concise than the commit log. @@ -220,12 +220,11 @@ these trees). The current development tree can be fetched with the following command: - git clone git://git.archlinux.org/pacman.git pacman + git clone https://gitlab.archlinux.org/pacman/pacman.git which will fetch the full development history into a directory named pacman. You can browse the source as well using -link:https://git.archlinux.org/pacman.git/[cgit]. HTTP/HTTPS URLs are also -available for cloning purposes; these URLs are listed at the above page. +link:https://gitlab.archlinux.org/pacman/pacman/[gitlab]. If you are interested in hacking on pacman, it is highly recommended you join the mailing list mentioned above, as well as take a quick glance at our @@ -237,20 +236,6 @@ you speak a foreign language, you can help by either creating or updating a translation file for your native language. Instructions can be found in link:translation-help.html[translation-help]. -Other Utilities -~~~~~~~~~~~~~~~ -Although the package manager itself is quite simple, many scripts have been -developed that help automate building and installing packages. These are used -extensively in link:https://archlinux.org/[Arch Linux]. Most of these utilities -are available in the Arch Linux projects -link:https://git.archlinux.org/[code browser]. - -Utilities available: - -* link:https://git.archlinux.org/dbscripts.git/[dbscripts] - scripts used by Arch Linux to manage the main package repositories -* link:https://git.archlinux.org/devtools.git/[devtools] - tools to assist in packaging and dependency checking -* link:https://git.archlinux.org/namcap.git/[namcap] - a package analysis utility written in python - Bugs ---- If you find bugs (which is quite likely), please email them to the pacman-dev ===================================== doc/submitting-patches.asciidoc ===================================== @@ -20,7 +20,7 @@ started with GIT if you have not worked with it before. The pacman code can be fetched using the following command: - git clone git://git.archlinux.org/pacman.git + git clone https://gitlab.archlinux.org/pacman/pacman.git Creating your patch ===================================== doc/translation-help.asciidoc ===================================== @@ -78,7 +78,7 @@ Incremental Updates If you have more advanced needs you will have to get a copy of the pacman repository. - git clone git://git.archlinux.org/pacman.git pacman + git clone https://gitlab.archlinux.org/pacman/pacman.git Next, you will need to run `./autogen.sh` and `./configure` in the base directory to generate the correct Makefiles. At this point, all necessary ===================================== lib/libalpm/dload.c ===================================== @@ -613,12 +613,33 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, /* Let's check if client requested downloading accompanion *.sig file */ if(!payload->signature && payload->download_signature && curlerr == CURLE_OK && payload->respcode < 400) { struct dload_payload *sig = NULL; - + char *url = payload->fileurl; + char *_effective_filename; + const char *effective_filename; + char *query; + const char *dbext = alpm_option_get_dbext(handle); const char* realname = payload->destfile_name ? payload->destfile_name : payload->tempfile_name; - int len = strlen(effective_url) + 5; + int len; + + STRDUP(_effective_filename, effective_url, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + effective_filename = get_filename(_effective_filename); + query = strrchr(effective_filename, '?'); + + if(query) { + query[0] = '\0'; + } + + /* Only use the effective url for sig downloads if the effective_url contains .dbext or .pkg */ + if(strstr(effective_filename, dbext) || strstr(effective_filename, ".pkg")) { + url = effective_url; + } + + free(_effective_filename); + + len = strlen(url) + 5; CALLOC(sig, 1, sizeof(*sig), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); MALLOC(sig->fileurl, len, FREE(sig); GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); - snprintf(sig->fileurl, len, "%s.sig", effective_url); + snprintf(sig->fileurl, len, "%s.sig", url); if(payload->trust_remote_name) { /* In this case server might provide a new name for the main payload. @@ -767,7 +788,7 @@ static int curl_add_payload(alpm_handle_t *handle, CURLM *curlm, GOTO_ERR(handle, ALPM_ERR_SERVER_BAD_URL, cleanup); } - if(payload->remote_name && strlen(payload->remote_name) > 0) { + if(!payload->random_partfile && payload->remote_name && strlen(payload->remote_name) > 0) { if(!payload->destfile_name) { payload->destfile_name = get_fullpath(localpath, payload->remote_name, ""); } @@ -776,8 +797,9 @@ static int curl_add_payload(alpm_handle_t *handle, CURLM *curlm, goto cleanup; } } else { - /* URL doesn't contain a filename, so make a tempfile. We can't support - * resuming this kind of download; partial transfers will be destroyed */ + /* We want a random filename or the URL does not contain a filename, so download to a + * temporary location. We can not support resuming this kind of download; any partial + * transfers will be destroyed */ payload->unlink_on_fail = 1; payload->localf = create_tempfile(payload, localpath); @@ -825,6 +847,19 @@ cleanup: return ret; } +/* + * Use to sort payloads by max size in decending order (largest -> smallest) + */ +static int compare_dload_payload_sizes(const void *left_ptr, const void *right_ptr) +{ + struct dload_payload *left, *right; + + left = (struct dload_payload *) left_ptr; + right = (struct dload_payload *) right_ptr; + + return right->max_size - left->max_size; +} + /* Returns -1 if an error happened for a required file * Returns 0 if a payload was actually downloaded * Returns 1 if no files were downloaded and all errors were non-fatal @@ -838,6 +873,10 @@ static int curl_download_internal(alpm_handle_t *handle, int max_streams = handle->parallel_downloads; int updated = 0; /* was a file actually updated */ CURLM *curlm = handle->curlm; + size_t payloads_size = alpm_list_count(payloads); + + /* Sort payloads by package size */ + payloads = alpm_list_msort(payloads, payloads_size, &compare_dload_payload_sizes); while(active_downloads_num > 0 || payloads) { CURLMcode mc; @@ -986,11 +1025,20 @@ int SYMEXPORT alpm_fetch_pkgurl(alpm_handle_t *handle, const alpm_list_t *urls, alpm_list_append(fetched, filepath); } else { struct dload_payload *payload = NULL; + char *c; ASSERT(url, GOTO_ERR(handle, ALPM_ERR_WRONG_ARGS, err)); CALLOC(payload, 1, sizeof(*payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, err)); STRDUP(payload->fileurl, url, FREE(payload); GOTO_ERR(handle, ALPM_ERR_MEMORY, err)); - payload->allow_resume = 1; + + c = strrchr(url, '/'); + if(strstr(c, ".pkg")) { + /* we probably have a usable package filename to download to */ + payload->allow_resume = 1; + } else { + payload->random_partfile = 1; + } + payload->handle = handle; payload->trust_remote_name = 1; payload->download_signature = (handle->siglevel & ALPM_SIG_PACKAGE); ===================================== lib/libalpm/dload.h ===================================== @@ -44,6 +44,7 @@ struct dload_payload { off_t prevprogress; int force; int allow_resume; + int random_partfile; int errors_ok; int unlink_on_fail; int trust_remote_name; ===================================== scripts/libmakepkg/meson.build ===================================== @@ -5,6 +5,7 @@ libmakepkg_modules = [ { 'name' : 'lint_config', 'has_subdir' : true }, { 'name' : 'lint_package', 'has_subdir' : true }, { 'name' : 'lint_pkgbuild', 'has_subdir' : true }, + { 'name' : 'reproducible', 'has_subdir' : true }, { 'name' : 'source', 'has_subdir' : true }, { 'name' : 'srcinfo', }, { 'name' : 'tidy', 'has_subdir' : true }, ===================================== scripts/libmakepkg/reproducible.sh.in ===================================== @@ -0,0 +1,29 @@ +#!/bin/bash +# +# reproducible.sh - utilities for improving package reproducibility +# +# Copyright (c) 2021 Pacman Development Team <pacman-dev(a)archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_REPRODUCIBLE_SH" ]] && return +LIBMAKEPKG_REPRODUCIBLE_SH=1 + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + + +for lib in "$LIBRARY/reproducible/"*.sh; do + source "$lib" +done ===================================== scripts/libmakepkg/reproducible/meson.build ===================================== @@ -0,0 +1,17 @@ +libmakepkg_module = 'reproducible' + +sources = [ + 'python.sh.in', +] + +foreach src : sources + output_dir = join_paths(get_option('datadir'), 'makepkg', libmakepkg_module) + + custom_target( + libmakepkg_module + '_' + src.underscorify(), + command : [ SCRIPT_EDITOR, '@INPUT@', '@OUTPUT@' ], + input : src, + output : '@BASENAME@', + install : true, + install_dir : output_dir) +endforeach ===================================== scripts/libmakepkg/reproducible/python.sh.in ===================================== @@ -0,0 +1,29 @@ +#!/bin/bash +# +# python.sh - creating reproducible python packages +# +# Copyright (c) 2021 Pacman Development Team <pacman-dev(a)archlinux.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +[[ -n "$LIBMAKEPKG_REPRODUCIBLE_PYTHON_SH" ]] && return +LIBMAKEPKG_REPRODUCIBLE_PYTHON_SH=1 + + +LIBRARY=${LIBRARY:-'@libmakepkgdir@'} + + +# disable hash randomization when creating .pyc files +export PYTHONHASHSEED=0 ===================================== test/pacman/tests/upgrade-download-pkg-and-sig-with-filename.py ===================================== @@ -22,6 +22,12 @@ '/redir-dest.pkg': 'redir-dest', '/redir-dest.pkg.sig': 'redir-dest.sig', + # redirect cdn + '/redir-cdn.pkg': { 'code': 303, 'headers': { 'Location': '/cdn-1' } }, + '/redir-cdn.pkg.sig': { 'code': 303, 'headers': { 'Location': '/cdn-2' } }, + '/cdn-1': 'redir-dest', + '/cdn-2': 'redir-dest.sig', + # content-disposition and redirect '/cd-redir.pkg': { 'code': 303, 'headers': { 'Location': '/cd-redir-dest.pkg' } }, '/cd-redir-dest.pkg': { @@ -30,6 +36,18 @@ }, '/cd-redir-dest.pkg.sig': 'cd-redir-dest.sig', + # content-disposition and redirect to cdn + '/cd-redir-cdn.pkg': { 'code': 303, 'headers': { 'Location': '/cdn-3' } }, + '/cd-redir-cdn.pkg.sig': { 'code': 303, 'headers': { 'Location': '/cdn-4' } }, + '/cdn-3': { + 'headers': { 'Content-Disposition': 'attachment; filename="cdn-alt.pkg"' }, + 'body': 'cdn-alt' + }, + '/cdn-4': { + 'headers': { 'Content-Disposition': 'attachment; filename="cdn-alt.pkg.sig"' }, + 'body': 'cdn-alt.sig' + }, + # TODO: absolutely terrible hack to prevent pacman from attempting to # validate packages, which causes failure under --valgrind thanks to # a memory leak in gpgme that is too general for inclusion in valgrind.supp @@ -38,7 +56,7 @@ '': 'fallback', }) -self.args = '-Uw {url}/simple.pkg {url}/cd.pkg {url}/redir.pkg {url}/cd-redir.pkg {url}/404'.format(url=url) +self.args = '-Uw {url}/simple.pkg {url}/cd.pkg {url}/redir.pkg {url}/redir-cdn.pkg {url}/cd-redir.pkg {url}/cd-redir-cdn.pkg {url}/404'.format(url=url) # packages/sigs are not valid, error is expected self.addrule('!PACMAN_RETCODE=0') @@ -59,3 +77,6 @@ self.addrule('!CACHE_FEXISTS=cd-redir-dest.pkg') self.addrule('CACHE_FCONTENTS=cd-redir-dest-alt.pkg|cd-redir-dest') self.addrule('CACHE_FCONTENTS=cd-redir-dest-alt.pkg.sig|cd-redir-dest.sig') + +self.addrule('CACHE_FCONTENTS=cdn-alt.pkg|cdn-alt') +self.addrule('CACHE_FCONTENTS=cdn-alt.pkg.sig|cdn-alt.sig') View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/fc7986485cad9c93df7e59… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/fc7986485cad9c93df7e59… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[pacman-dev] [PATCH] libalpm: Give -U downloads a random .part name if needed
by Allan McRae 04 Sep '21

04 Sep '21

From: morganamilo <morganamilo(a)archlinux.org> archweb's download links all ended in /download. This cause all the temp files to be named download.part. With parallel downloads this results in multiple downloads to go to the same temp file and breaks the transaction. Assign random temporary filenames to downloads from URLs that are either missing a filename, or if the filename does not contain at least three hyphens (as a well formed package filename does). While this approach to determining when to use a temporary filename is not 100% foolproof, it does keep nice looking download progress bar names when a proper package filename is given. The only downside of not using temporary files when provided with a filename with three or more hyphens is URLs created specifically to bypass temporary filename usage can not be downloaded in parallel. We probably do not want to download packages from such URLs anyway. Fixes FS#71464 Modified-by: Allan McRae (do not use temporary files for realish URLs) Signed-off-by: Allan McRae <allan(a)archlinux.org> --- lib/libalpm/dload.c | 26 ++++++++++++++++++++++---- lib/libalpm/dload.h | 1 + 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index ca6be7b6..d2448349 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -767,7 +767,7 @@ static int curl_add_payload(alpm_handle_t *handle, CURLM *curlm, GOTO_ERR(handle, ALPM_ERR_SERVER_BAD_URL, cleanup); } - if(payload->remote_name && strlen(payload->remote_name) > 0) { + if(!payload->random_partfile && payload->remote_name && strlen(payload->remote_name) > 0) { if(!payload->destfile_name) { payload->destfile_name = get_fullpath(localpath, payload->remote_name, ""); } @@ -776,8 +776,9 @@ static int curl_add_payload(alpm_handle_t *handle, CURLM *curlm, goto cleanup; } } else { - /* URL doesn't contain a filename, so make a tempfile. We can't support - * resuming this kind of download; partial transfers will be destroyed */ + /* We want a random filename or the URL does not contain a filename, so download to a + * temporary location. We can not support resuming this kind of download; any partial + * transfers will be destroyed */ payload->unlink_on_fail = 1; payload->localf = create_tempfile(payload, localpath); @@ -986,11 +987,28 @@ int SYMEXPORT alpm_fetch_pkgurl(alpm_handle_t *handle, const alpm_list_t *urls, alpm_list_append(fetched, filepath); } else { struct dload_payload *payload = NULL; + size_t hyphen_count = 0; + char *c; ASSERT(url, GOTO_ERR(handle, ALPM_ERR_WRONG_ARGS, err)); CALLOC(payload, 1, sizeof(*payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, err)); STRDUP(payload->fileurl, url, FREE(payload); GOTO_ERR(handle, ALPM_ERR_MEMORY, err)); - payload->allow_resume = 1; + + c = strrchr(url, '/'); + while(*c) { + if(*c == '-') { + hyphen_count++; + } + c++; + } + + if(hyphen_count > 2) { + /* we probably have a usable package filename to download to */ + payload->allow_resume = 1; + } else { + payload->random_partfile = 1; + } + payload->handle = handle; payload->trust_remote_name = 1; payload->download_signature = (handle->siglevel & ALPM_SIG_PACKAGE); diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 8f3d17b4..dfeb7902 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -44,6 +44,7 @@ struct dload_payload { off_t prevprogress; int force; int allow_resume; + int random_partfile; int errors_ok; int unlink_on_fail; int trust_remote_name; -- 2.33.0

1 1

[pacman-dev] [PATCH] Update broken links pointing to git.archlinux.org
by Hugo Osvaldo Barrera 03 Sep '21

03 Sep '21

All of these links are broken since the recent move to gitlab.archlinux.org. A few projects are, apparently, only available on GitHub, so I've linked to that source (hopefully that's only temporary). For git-clone URLs, I've opted for the https URLs since those can be used by anyone -- whereas the ssh URLs require the user to be registered on the gitlab instance which is not open to the public yet. Signed-off-by: Hugo Osvaldo Barrera <hugo(a)barrera.io> --- doc/index.asciidoc | 19 ++++++++++--------- doc/submitting-patches.asciidoc | 2 +- doc/translation-help.asciidoc | 2 +- 3 files changed, 12 insertions(+), 11 deletions(-) diff --git a/doc/index.asciidoc b/doc/index.asciidoc index 47295f9b..a1bc4cc8 100644 --- a/doc/index.asciidoc +++ b/doc/index.asciidoc @@ -59,11 +59,11 @@ configuration files dealing with pacman. Changelog ~~~~~~~~~ For a good idea of what is going on in pacman development, take a look at the -link:https://git.archlinux.org/pacman.git/[Git summary page] for the +link:https://gitlab.archlinux.org/pacman/pacman[Git summary page] for the project. See the most recent -link:https://git.archlinux.org/pacman.git/tree/NEWS[NEWS] +link:https://gitlab.archlinux.org/pacman/pacman/-/blob/master/NEWS[NEWS] file for a not-as-frequently-updated list of changes. However, this should contain the biggest changes in a format more concise than the commit log. @@ -220,12 +220,13 @@ these trees). The current development tree can be fetched with the following command: - git clone git://git.archlinux.org/pacman.git pacman + git clone https://gitlab.archlinux.org/pacman/pacman.git which will fetch the full development history into a directory named pacman. You can browse the source as well using -link:https://git.archlinux.org/pacman.git/[cgit]. HTTP/HTTPS URLs are also -available for cloning purposes; these URLs are listed at the above page. +link:https://gitlab.archlinux.org/pacman/pacman/-/tree/master[gitlab]. +HTTP/HTTPS URLs are also available for cloning purposes; these URLs are listed +at the above page. If you are interested in hacking on pacman, it is highly recommended you join the mailing list mentioned above, as well as take a quick glance at our @@ -243,13 +244,13 @@ Although the package manager itself is quite simple, many scripts have been developed that help automate building and installing packages. These are used extensively in link:https://archlinux.org/[Arch Linux]. Most of these utilities are available in the Arch Linux projects -link:https://git.archlinux.org/[code browser]. +link:https://gitlab.archlinux.org/archlinux[code browser]. Utilities available: -* link:https://git.archlinux.org/dbscripts.git/[dbscripts] - scripts used by Arch Linux to manage the main package repositories -* link:https://git.archlinux.org/devtools.git/[devtools] - tools to assist in packaging and dependency checking -* link:https://git.archlinux.org/namcap.git/[namcap] - a package analysis utility written in python +* link:https://github.com/archlinux/dbscripts[dbscripts] - scripts used by Arch Linux to manage the main package repositories +* link:https://github.com/archlinux/devtools[devtools] - tools to assist in packaging and dependency checking +* link:https://gitlab.archlinux.org/pacman/namcap[namcap] - a package analysis utility written in python Bugs ---- diff --git a/doc/submitting-patches.asciidoc b/doc/submitting-patches.asciidoc index 88b58ca5..14b80b61 100644 --- a/doc/submitting-patches.asciidoc +++ b/doc/submitting-patches.asciidoc @@ -20,7 +20,7 @@ started with GIT if you have not worked with it before. The pacman code can be fetched using the following command: - git clone git://git.archlinux.org/pacman.git + git clone https://gitlab.archlinux.org/pacman/pacman.git Creating your patch diff --git a/doc/translation-help.asciidoc b/doc/translation-help.asciidoc index 5a67a2d6..4301852d 100644 --- a/doc/translation-help.asciidoc +++ b/doc/translation-help.asciidoc @@ -78,7 +78,7 @@ Incremental Updates If you have more advanced needs you will have to get a copy of the pacman repository. - git clone git://git.archlinux.org/pacman.git pacman + git clone https://gitlab.archlinux.org/pacman/pacman.git Next, you will need to run `./autogen.sh` and `./configure` in the base directory to generate the correct Makefiles. At this point, all necessary -- 2.32.0

2 8

Re: [pacman-dev] [PATCH] Order downloads by descending max_size
by Charlie Sale 03 Sep '21

03 Sep '21

>This sort is unlikely to optimal, but optimality would be much harder >to achieve. Do you have anything in mind that would be more optimal? Should I take a different strategy? > The ideal situation is to have 1 large download running to soak up bandwidth Would it be better to search for any really large packages (much larger than the rest), put those at the top of the list, and then leave everything else unsorted? Would that be too slow as well? Cheers

4 5

[pacman-dev] [PATCH 1/2] only use effective url for urls containing .db or .pkg
by morganamilo 03 Sep '21

03 Sep '21

Github and other sites redirect their downloads to a cdn. So the download http://foo.org/myrepo.db may redirect to something like https://cdn.foo.org/83749327439. This then causes pacman to try and download the sig as https://cdn.foo.org/83749327439.sig which is incorrect. In this case pacman should append .sig to the original url. However urls like https://archlinux.org/packages/community/x86_64/0ad/download/ Redirect to the mirror, so .sig has to appended after the redirects and not before. So we decide if we should append .sig on the original or effective url based on if the effective url has .db or .pkg in it. Fixes FS#71148 --- lib/libalpm/dload.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 2c14841f..72e9cfcd 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -613,11 +613,28 @@ static int curl_check_finished_download(CURLM *curlm, CURLMsg *msg, /* Let's check if client requested downloading accompanion *.sig file */ if(!payload->signature && payload->download_signature && curlerr == CURLE_OK && payload->respcode < 400) { struct dload_payload *sig = NULL; + char *url = payload->fileurl; + char *_effective_filename; - int len = strlen(effective_url) + 5; + STRDUP(_effective_filename, effective_url, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + const char *effective_filename = get_filename(_effective_filename); + char *query = strrchr(effective_filename, '?'); + + if(query) { + query[0] = '\0'; + } + + /* Only use the effective url for sig downloads if the effective_url contains .db or .pkg */ + if(strstr(effective_filename, ".db") || strstr(effective_filename, ".pkg")) { + url = effective_url; + } + + free(_effective_filename); + + int len = strlen(url) + 5; CALLOC(sig, 1, sizeof(*sig), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); MALLOC(sig->fileurl, len, FREE(sig); GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); - snprintf(sig->fileurl, len, "%s.sig", effective_url); + snprintf(sig->fileurl, len, "%s.sig", url); if(payload->trust_remote_name) { /* In this case server might provide a new name for the main payload. -- 2.32.0

5 10