- Pacman-dev - lists.archlinux.org

[PATCH] download smallest files first but always use one parallel job to download large files concurrently
by Thomas Reichel 16 Feb '24

16 Feb '24

When downloading small files the download time seems dominated by connection latency rather than bandwidth. Downloading several small files in parallel is an effective way to reduce the impact of latency. However, downloading many small files in parallel does not always saturate bandwidth, which is inefficient. This commit attempts to fully utilize parallelism to download small files while maintaining high bandwidth utilization. This is accomplished by downloading the smallest files first using all but 1 parallel connection while downloading a large file using the remaining parallel connection. The result seems to maintain a more stable download speed throughout entire transactions. This is in contrast to the usual behavior I observed when downloading many packages, where the download speed progressively declines as smaller packages are downloaded. When my entire cache is deleted and all packages are redownloaded using ` pacman -Qqn | sudo pacman -Sw -`, the mean download speed is 47.8 MiB/s. After this patch, the mean download speed is 54.0 MiB/s. In terms of time savings, this patch causes a 14.9 GiB download to go from 5 minutes 20 seconds to 4 minutes 43 seconds on my system and network. Your mileage may vary on different systems, networks, and selections of packages. I expect there to be virtually no effect on selections of packages that are all fairly large. Signed-off-by: Thomas Reichel <tom.p.reichel(a)gmail.com> --- lib/libalpm/dload.c | 39 +++++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 7c344760..4d8777f9 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -884,7 +884,7 @@ cleanup: } /* - * Use to sort payloads by max size in decending order (largest -> smallest) + * Use to sort payloads by max size in ascending order (smallest -> largest) */ static int compare_dload_payload_sizes(const void *left_ptr, const void *right_ptr) { @@ -893,7 +893,7 @@ static int compare_dload_payload_sizes(const void *left_ptr, const void *right_p left = (struct dload_payload *) left_ptr; right = (struct dload_payload *) right_ptr; - return right->max_size - left->max_size; + return left->max_size - right->max_size; } /* Returns -1 if an error happened for a required file @@ -910,18 +910,39 @@ static int curl_download_internal(alpm_handle_t *handle, int updated = 0; /* was a file actually updated */ CURLM *curlm = handle->curlm; size_t payloads_size = alpm_list_count(payloads); + /* download a single large file along with smaller files */ + struct dload_payload *large_file = NULL; + /* a reverse iterator for payloads */ + alpm_list_t* payloads_reverse; /* Sort payloads by package size */ payloads = alpm_list_msort(payloads, payloads_size, &compare_dload_payload_sizes); + payloads_reverse = alpm_list_last(payloads); + while(active_downloads_num > 0 || payloads) { CURLMcode mc; for(; active_downloads_num < max_streams && payloads; active_downloads_num++) { - struct dload_payload *payload = payloads->data; + struct dload_payload *payload; + if (large_file == NULL) { + payload = payloads_reverse->data; + } else { + payload = payloads->data; + } if(curl_add_payload(handle, curlm, payload, localpath) == 0) { - payloads = payloads->next; + if (payloads_reverse == payloads) { + /* pointers met in the middle, we finished by advancing either */ + payloads = NULL; + break; + } + if (large_file == NULL) { + large_file = payloads_reverse->data; + payloads_reverse = payloads_reverse->prev; + } else { + payloads = payloads->next; + } } else { /* The payload failed to start. Do not start any new downloads. * Wait until all active downloads complete. @@ -959,6 +980,16 @@ static int curl_download_internal(alpm_handle_t *handle, payloads = NULL; err = -1; } else if(ret == 0) { + struct dload_payload *finished_payload; + CURLcode curlerr; + /* which file downloaded? */ + curlerr = curl_easy_getinfo(msg->easy_handle, CURLINFO_PRIVATE, &finished_payload); + ASSERT(curlerr == CURLE_OK, RET_ERR(handle, ALPM_ERR_LIBCURL, -1)); + /* did we just finish the large file that was downloading? */ + if (large_file != NULL && finished_payload == large_file) { + large_file = NULL; + } + updated = 1; } } else { -- 2.43.1

3 7

[Git][pacman/pacman][master] libmakepkg: skip stripping guile-2.2 files
by Allan McRae (＠allan) 16 Feb '24

16 Feb '24

Allan McRae pushed to branch master at Pacman / Pacman Commits: 48729f8e by Allan McRae at 2024-02-15T23:34:05+00:00 libmakepkg: skip stripping guile-2.2 files Guile 2.2 uses ELF format for its byte-compiled files. These are not normal executables, and are not strippable in the normal sense. Given these are ELF files and detected by "file" as non-stripped binaries, it is only possible to skip these using the file path. Fixes #73 Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 1 changed file: - scripts/libmakepkg/tidy/strip.sh.in View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/48729f8eccc99a09ebd2f1a… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/48729f8eccc99a09ebd2f1a… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[Git][pacman/pacman][master] Update "submitting-patches.asciidoc" for the move to gitlab
by Allan McRae (＠allan) 16 Feb '24

16 Feb '24

Allan McRae pushed to branch master at Pacman / Pacman Commits: fface900 by Allan McRae at 2024-02-15T23:19:20+00:00 Update "submitting-patches.asciidoc" for the move to gitlab Also include a section encouraging discussion of large changes as there have been a number of case of rejected features being reinvented. Fixes #34 Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 1 changed file: - doc/submitting-patches.asciidoc View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/fface9001a0b1a386b07622… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/fface9001a0b1a386b07622… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[Git][pacman/pacman][master] makepkg.conf.5: Add ellipsis to OPTIONS array description
by Allan McRae (＠allan) 16 Feb '24

16 Feb '24

Allan McRae pushed to branch master at Pacman / Pacman Commits: f74daa39 by Allan McRae at 2024-02-15T23:14:28+00:00 makepkg.conf.5: Add ellipsis to OPTIONS array description We have added more options to makepkg, but adding them to the man page entry would result in line wrapping on a standard width terminal. Instead, trucate and add ellipsis to indicate more members (as described in the section below). Fixes #91 Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 1 changed file: - doc/makepkg.conf.5.asciidoc View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/f74daa39e8a4e4806196261… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/f74daa39e8a4e4806196261… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[Git][pacman/pacman][master] Fix format string
by Allan McRae (＠allan) 16 Feb '24

16 Feb '24

Allan McRae pushed to branch master at Pacman / Pacman Commits: 22c043d4 by Allan McRae at 2024-02-16T09:09:45+10:00 Fix format string Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 1 changed file: - scripts/libmakepkg/integrity/verify_signature.sh.in View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/22c043d4c38daec03c28077… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/22c043d4c38daec03c28077… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[Git][pacman/pacman][master] 5 commits: Add function to check if lists are equal
by Allan McRae (＠allan) 16 Feb '24

16 Feb '24

Allan McRae pushed to branch master at Pacman / Pacman Commits: 196de7e9 by morganamilo at 2024-02-15T23:02:15+00:00 Add function to check if lists are equal - - - - - 386125fc by morganamilo at 2024-02-15T23:02:15+00:00 Validate extra data when comparing dbpkg to pkgfile - - - - - 62095d91 by morganamilo at 2024-02-15T23:02:15+00:00 Prompt to delete invalid package - - - - - 7a43c6fe by morganamilo at 2024-02-15T23:02:15+00:00 Speedup comparing lists if they happen to be in the same order - - - - - abc6dd74 by morganamilo at 2024-02-15T23:02:15+00:00 libalpm: check calloc in alpm_list_cmp_unsorted - - - - - 3 changed files: - lib/libalpm/alpm_list.c - lib/libalpm/alpm_list.h - lib/libalpm/sync.c View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/e1dc609939cc5025213a51… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/e1dc609939cc5025213a51… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[Git][pacman/pacman][master] libalpm: print warnings for unknown keys in databases/packages
by Allan McRae (＠allan) 10 Feb '24

10 Feb '24

Allan McRae pushed to branch master at Pacman / Pacman Commits: e1dc6099 by morganamilo at 2024-02-10T11:31:12+10:00 libalpm: print warnings for unknown keys in databases/packages Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 3 changed files: - lib/libalpm/be_local.c - lib/libalpm/be_package.c - lib/libalpm/be_sync.c View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/e1dc609939cc5025213a51b… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/commit/e1dc609939cc5025213a51b… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

[Git][pacman/pacman][master] 2 commits: libalpm: don't use atio for pkgreason
by Allan McRae (＠allan) 09 Feb '24

09 Feb '24

Allan McRae pushed to branch master at Pacman / Pacman Commits: 6e6d3f18 by morganamilo at 2024-02-09T11:14:38+10:00 libalpm: don't use atio for pkgreason atio's behaviour is undefined if the input is not valid. Also it does all sorts of whitespace and prefix handling which we don't need for pkgreason. Instead of going into UB on invalid input we now return EXPLICIT as the fallback and print an error. However we don't actually error out as the DB parsing tries to be error tolerant. Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 0649a66e by Allan McRae at 2024-02-09T11:14:38+10:00 Add ALPM_PKG_REASON_UNKNOWN type Return ALPM_PKG_REASON_UNKNOWN when parsing of %REASON% in the local database fails. Signed-off-by: Allan McRae <allan(a)archlinux.org> - - - - - 3 changed files: - lib/libalpm/alpm.h - lib/libalpm/be_local.c - lib/libalpm/deps.c View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/0a394144b22f02b5956e89… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/0a394144b22f02b5956e89… You're receiving this email because of your account on gitlab.archlinux.org.

1 0

pacman-7.0 release planning
by Allan McRae 09 Feb '24

09 Feb '24

Hi all, I have finally had enough time to finish the privsep work. We now download all files using a dedicated user. This is ready to be committed (following any final review) [1]. Note that this currently only separates downloads - more work is needed for (e.g.) package verification to also occur in a separate process. I consider this marking the milestone for the next package release. I have started a tracker for what is needed to be addressed before the release [2]. There are more things that need added to the list... It just represents my bare minimum before release. Realistically, given my upcoming commitments, I think the release will happen close to the end of the year. Cheers, Allan [1] https://gitlab.archlinux.org/pacman/pacman/-/merge_requests/23 [2] https://gitlab.archlinux.org/pacman/pacman/-/milestones/1

2 3

[Git][pacman/pacman][master] 4 commits: allow freeing partial db package cache
by Allan McRae (＠allan) 07 Feb '24

07 Feb '24

Allan McRae pushed to branch master at Pacman / Pacman Commits: 5c75a55c by Andrew Gregory at 2024-02-07T12:27:26+00:00 allow freeing partial db package cache The free function was checking DB_STATUS_PKGCACHE, which is only set once the package cache has been fully built. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> - - - - - edd57c8b by Andrew Gregory at 2024-02-07T12:27:26+00:00 perform cleanup on sync db parsing errors Cleanup was only being performed when libarchive failed to actually read the file. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> - - - - - fde59b99 by Andrew Gregory at 2024-02-07T12:27:26+00:00 be_package: delay freeing archive resource The error path uconditinally tries to free the archive, leading to a double-free segmentation fault if the error path is triggered after already freeing it. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> - - - - - 0a394144 by Andrew Gregory at 2024-02-07T12:27:26+00:00 validate package metadata after loading alpm has certain requirements for package metadata necessary for proper functioning, name and version in particular. These requirements are already enforced in makepkg, but nowhere in alpm. Exceptions are treated as errors for non-local packages because they cannot be installed without potentially resulting in undefined behavior. Exceptions for local packages are treated as warnings because they are already installed, so any damage has already been done, and the user would otherwise have no way to uninstall them. Signed-off-by: Andrew Gregory <andrew.gregory.8(a)gmail.com> - - - - - 10 changed files: - lib/libalpm/be_local.c - lib/libalpm/be_package.c - lib/libalpm/be_sync.c - lib/libalpm/db.c - lib/libalpm/package.c - lib/libalpm/package.h - test/pacman/meson.build - + test/pacman/tests/pkg-meta-invalid-name-file.py - + test/pacman/tests/pkg-meta-invalid-name-local.py - + test/pacman/tests/pkg-meta-invalid-name-sync.py View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/56f1eeef4b97733db3d2b3… -- View it on GitLab: https://gitlab.archlinux.org/pacman/pacman/-/compare/56f1eeef4b97733db3d2b3… You're receiving this email because of your account on gitlab.archlinux.org.

1 0