[pacman-dev] Download options
There was many thread and patches regarding this, so I am starting another! So far there are two (maybe three) things people want options for to do with the downloader. I see most of these options being set in pacman.conf and rarely specified on the command line. Argue with me if I am wrong here... Based on that assumption, here are my preferred option names (for both pacman.conf and the command line): 1) Disable low speed timeout DisableDownloadTimeout --disable-download-timeout I see no need to separate out speed and time timeouts. 2) Set maximum download speed MaxDownloadSpeed --max-download-speed I would also accept replacing speed with limit. 3) (in a glorious future) Set maximum concurrent downloads MaxConcurrentDownloads --max-concurrent-downloads Any argument about these option names will need to be very clearly justified. I don't own a bike so have no need for a bikeshed. Allan
Allan McRae <allan@archlinux.org> on Fri, 2016/12/02 16:47:
There was many thread and patches regarding this, so I am starting another!
So far there are two (maybe three) things people want options for to do with the downloader. I see most of these options being set in pacman.conf and rarely specified on the command line. Argue with me if I am wrong here...
Based on that assumption, here are my preferred option names (for both pacman.conf and the command line):
1) Disable low speed timeout
DisableDownloadTimeout --disable-download-timeout
I see no need to separate out speed and time timeouts.
This is the one I was interested in. As you want a switch only I will take "Introduce a 'stupid-proxy' option" [0], rename "stupid-proxy" to "disable-download-timeout" and resend the patch. Thanks! [0] https://lists.archlinux.org/pipermail/pacman-dev/2016-November/021672.html -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Best regards my address: */=0;b=c[a++];) putchar(b-1/(/* Chris cc -ox -xc - && ./x */b/42*2-3)*42);}
From: Christian Hesse <mail@eworm.de> Add command line option ('--disable-download-timeout') and config file option ('DisableDownloadTimeout') to disable defaults for low speed limit and timeout on downloads. Use this if you have issues downloading files with proxy and/or security gateway. Signed-off-by: Christian Hesse <mail@eworm.de> --- doc/pacman.8.txt | 4 ++++ doc/pacman.conf.5.txt | 4 ++++ lib/libalpm/alpm.h | 2 ++ lib/libalpm/dload.c | 6 ++++-- lib/libalpm/handle.c | 10 ++++++++++ lib/libalpm/handle.h | 1 + src/pacman/conf.c | 4 ++++ src/pacman/conf.h | 4 +++- src/pacman/pacman.c | 6 ++++++ 9 files changed, 38 insertions(+), 3 deletions(-) diff --git a/doc/pacman.8.txt b/doc/pacman.8.txt index 0fa727e..b761560 100644 --- a/doc/pacman.8.txt +++ b/doc/pacman.8.txt @@ -193,6 +193,10 @@ Options *\--confirm*:: Cancels the effects of a previous '\--noconfirm'. +*\--disable-download-timeout*:: + Disable defaults for low speed limit and timeout on downloads. Use this + if you have issues downloading files with proxy and/or security gateway. + Transaction Options (apply to '-S', '-R' and '-U') -------------------------------------------------- diff --git a/doc/pacman.conf.5.txt b/doc/pacman.conf.5.txt index c665870..f92ee20 100644 --- a/doc/pacman.conf.5.txt +++ b/doc/pacman.conf.5.txt @@ -209,6 +209,10 @@ Options Displays name, version and size of target packages formatted as a table for upgrade, sync and remove operations. +*DisableDownloadTimeout*:: + Disable defaults for low speed limit and timeout on downloads. Use this + if you have issues downloading files with proxy and/or security gateway. + Repository Sections ------------------- diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 2d2491d..1166385 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -926,6 +926,8 @@ int alpm_option_set_local_file_siglevel(alpm_handle_t *handle, int level); int alpm_option_get_remote_file_siglevel(alpm_handle_t *handle); int alpm_option_set_remote_file_siglevel(alpm_handle_t *handle, int level); +int alpm_option_set_disable_dl_timeout(alpm_handle_t *handle, unsigned short disable_dl_timeout); + /** @} */ /** @addtogroup alpm_api_databases Database Functions diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 9d80358..28f1990 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -305,8 +305,10 @@ static void curl_set_handle_opts(struct dload_payload *payload, curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L); curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, dload_progress_cb); curl_easy_setopt(curl, CURLOPT_XFERINFODATA, (void *)payload); - curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1L); - curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 10L); + if(!handle->disable_dl_timeout) { + curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1L); + curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 10L); + } curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, dload_parseheader_cb); curl_easy_setopt(curl, CURLOPT_WRITEHEADER, (void *)payload); curl_easy_setopt(curl, CURLOPT_NETRC, CURL_NETRC_OPTIONAL); diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index 28e8148..f760748 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -844,4 +844,14 @@ int SYMEXPORT alpm_option_get_remote_file_siglevel(alpm_handle_t *handle) } } +int SYMEXPORT alpm_option_set_disable_dl_timeout(alpm_handle_t *handle, + unsigned short disable_dl_timeout) +{ + CHECK_HANDLE(handle, return -1); +#ifdef HAVE_LIBCURL + handle->disable_dl_timeout = disable_dl_timeout; +#endif + return 0; +} + /* vim: set noet: */ diff --git a/lib/libalpm/handle.h b/lib/libalpm/handle.h index 652f17d..469aecc 100644 --- a/lib/libalpm/handle.h +++ b/lib/libalpm/handle.h @@ -60,6 +60,7 @@ struct __alpm_handle_t { #ifdef HAVE_LIBCURL /* libcurl handle */ CURL *curl; /* reusable curl_easy handle */ + unsigned short disable_dl_timeout; #endif #ifdef HAVE_LIBGPGME diff --git a/src/pacman/conf.c b/src/pacman/conf.c index d8d64fb..de7715c 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -498,6 +498,8 @@ static int _parse_options(const char *key, char *value, config->color = isatty(fileno(stdout)) ? PM_COLOR_ON : PM_COLOR_OFF; enable_colors(config->color); } + } else if(strcmp(key, "DisableDownloadTimeout") == 0) { + config->disable_dl_timeout = 1; } else { pm_printf(ALPM_LOG_WARNING, _("config file %s, line %d: directive '%s' in section '%s' not recognized.\n"), @@ -815,6 +817,8 @@ static int setup_libalpm(void) alpm_option_set_noupgrades(handle, config->noupgrade); alpm_option_set_noextracts(handle, config->noextract); + alpm_option_set_disable_dl_timeout(handle, config->disable_dl_timeout); + for(i = config->assumeinstalled; i; i = i->next) { char *entry = i->data; alpm_depend_t *dep = alpm_dep_from_string(entry); diff --git a/src/pacman/conf.h b/src/pacman/conf.h index 945de7c..4d39271 100644 --- a/src/pacman/conf.h +++ b/src/pacman/conf.h @@ -55,6 +55,7 @@ typedef struct __config_t { unsigned short checkspace; unsigned short usesyslog; unsigned short color; + unsigned short disable_dl_timeout; double deltaratio; char *arch; char *print_format; @@ -203,7 +204,8 @@ enum { OP_VERBOSE, OP_DOWNLOADONLY, OP_REFRESH, - OP_ASSUMEINSTALLED + OP_ASSUMEINSTALLED, + OP_DISABLEDLTIMEOUT }; /* clean method */ diff --git a/src/pacman/pacman.c b/src/pacman/pacman.c index be52d1b..61e8b37 100644 --- a/src/pacman/pacman.c +++ b/src/pacman/pacman.c @@ -221,6 +221,8 @@ static void usage(int op, const char * const myname) addlist(_(" --logfile <path> set an alternate log file\n")); addlist(_(" --noconfirm do not ask for any confirmation\n")); addlist(_(" --confirm always ask for confirmation\n")); + addlist(_(" --disable-download-timeout\n" + " use relaxed timeouts for download\n")); } list = alpm_list_msort(list, alpm_list_count(list), options_cmp); for(i = list; i; i = alpm_list_next(i)) { @@ -443,6 +445,9 @@ static int parsearg_global(int opt) free(config->rootdir); config->rootdir = strdup(optarg); break; + case OP_DISABLEDLTIMEOUT: + config->disable_dl_timeout = 1; + break; case OP_VERBOSE: case 'v': (config->verbose)++; @@ -934,6 +939,7 @@ static int parseargs(int argc, char *argv[]) {"gpgdir", required_argument, 0, OP_GPGDIR}, {"dbonly", no_argument, 0, OP_DBONLY}, {"color", required_argument, 0, OP_COLOR}, + {"disable-download-timeout", no_argument, 0, OP_DISABLEDLTIMEOUT}, {0, 0, 0, 0} }; -- 2.10.2
On Fri, 2 Dec 2016 16:47:41 +1000 Allan McRae <allan@archlinux.org> wrote:
There was many thread and patches regarding this, so I am starting another!
So far there are two (maybe three) things people want options for to do with the downloader. I see most of these options being set in pacman.conf and rarely specified on the command line. Argue with me if I am wrong here...
Based on that assumption, here are my preferred option names (for both pacman.conf and the command line):
1) Disable low speed timeout
DisableDownloadTimeout --disable-download-timeout
I see no need to separate out speed and time timeouts.
2) Set maximum download speed
MaxDownloadSpeed --max-download-speed
I would also accept replacing speed with limit.
3) (in a glorious future) Set maximum concurrent downloads
MaxConcurrentDownloads --max-concurrent-downloads
Any argument about these option names will need to be very clearly justified. I don't own a bike so have no need for a bikeshed.
Well, I could re-state how "Dl" is shorter than "Download" which might be nice, especially on command-line option -- but also looks to me like all those options would be about the longest ones yet, so far options can have "shortcuts" in their names, e.g. --dbpath not --databasepath or --nodeps not --nodependencies It is possible those might be rarely specified on command line, doesn't mean they have to be as lengthy as possible because of it. Clear but short is good, and I feel that can be achieved here. Doesn't have to apply to options in pacman.conf, and I guess they don't have to be named the same; There's already difference in case style, and it wouldn't even be the first time, e.g. --ignore vs IgnorePkg Why use "disable" as prefix while every other options usually goes with "no" instead? (--nodeps, --noconfirm, --noscriplet, etc) And while on the topic of consistency, let me add that almost all of pacman's options use a "single word style," i.e. no dashes in them: --ignoregroup; --noprogressbar; --downloadonly; etc Why not keep that up? Yes, there are already expections to the rule (namely --print-format & --assume-installed), but those are the only two (I believe), I'm not sure it's good to make more. Consistency is a nice thing. So at least for command line, --nodltimeout and --maxdlspeed (and for the glorious future simply --maxdownloads) seem better to me. (Though if you don't like the "dl" bit, --nodownloadtimeout & --maxdownloadspeed would work too I guess; I do prefer the shorter versions myself, which I find still pretty clear/readable, but your choice.) Either way we can use NoDownloadTimeout & MaxDownloadSpeed (and MawDownloads, or even MaxConcurrentDownloads) in pacman.conf; Yes the "No" prefix is also used there, so again feels better/more consistent that way. Hopefully there were some clear justifications in there. :) Cheers, -j
On 03/12/16 06:31, Olivier Brunel wrote:
On Fri, 2 Dec 2016 16:47:41 +1000 Allan McRae <allan@archlinux.org> wrote:
There was many thread and patches regarding this, so I am starting another!
So far there are two (maybe three) things people want options for to do with the downloader. I see most of these options being set in pacman.conf and rarely specified on the command line. Argue with me if I am wrong here...
Based on that assumption, here are my preferred option names (for both pacman.conf and the command line):
1) Disable low speed timeout
DisableDownloadTimeout --disable-download-timeout
I see no need to separate out speed and time timeouts.
2) Set maximum download speed
MaxDownloadSpeed --max-download-speed
I would also accept replacing speed with limit.
3) (in a glorious future) Set maximum concurrent downloads
MaxConcurrentDownloads --max-concurrent-downloads
Any argument about these option names will need to be very clearly justified. I don't own a bike so have no need for a bikeshed.
Well, I could re-state how "Dl" is shorter than "Download" which might be nice, especially on command-line option -- but also looks to me like all those options would be about the longest ones yet, so far options can have "shortcuts" in their names, e.g. --dbpath not --databasepath or --nodeps not --nodependencies
It is possible those might be rarely specified on command line, doesn't mean they have to be as lengthy as possible because of it. Clear but short is good, and I feel that can be achieved here.
Doesn't have to apply to options in pacman.conf, and I guess they don't have to be named the same; There's already difference in case style, and it wouldn't even be the first time, e.g. --ignore vs IgnorePkg
Why use "disable" as prefix while every other options usually goes with "no" instead? (--nodeps, --noconfirm, --noscriplet, etc)
And while on the topic of consistency, let me add that almost all of pacman's options use a "single word style," i.e. no dashes in them: --ignoregroup; --noprogressbar; --downloadonly; etc Why not keep that up?
Yes, there are already expections to the rule (namely --print-format & --assume-installed), but those are the only two (I believe), I'm not sure it's good to make more. Consistency is a nice thing.
So at least for command line, --nodltimeout and --maxdlspeed (and for the glorious future simply --maxdownloads) seem better to me. (Though if you don't like the "dl" bit, --nodownloadtimeout & --maxdownloadspeed would work too I guess; I do prefer the shorter versions myself, which I find still pretty clear/readable, but your choice.)
Either way we can use NoDownloadTimeout & MaxDownloadSpeed (and MawDownloads, or even MaxConcurrentDownloads) in pacman.conf; Yes the "No" prefix is also used there, so again feels better/more consistent that way.
And we hit bikeshed again... I'm done. All patches adding new options will not be considered until further notice. Allan
On Sat, 3 Dec 2016 09:41:13 +1000 Allan McRae <allan@archlinux.org> wrote:
I'm done. All patches adding new options will not be considered until further notice.
hmm, so it seems I totally misunderstood (the intent) of your earlier mail, all apologies. The intent never was to annoy/waste anyone's time.
On Fri, Dec 02, 2016 at 04:47:41PM +1000, Allan McRae wrote:
There was many thread and patches regarding this, so I am starting another!
So far there are two (maybe three) things people want options for to do with the downloader. I see most of these options being set in pacman.conf and rarely specified on the command line. Argue with me if I am wrong here...
Based on that assumption, here are my preferred option names (for both pacman.conf and the command line):
1) Disable low speed timeout
DisableDownloadTimeout --disable-download-timeout
As a user, I would have no idea what this means. As the person who wrote the code that this affects, the thing which is most closely associated with time in this equation can hardly be called a "timeout". It's more of a threshold. Some other suggestions, because my shed is better than your shed: --disabledeadnetworkcheck --nocheckdeadnetwork --nodeadconncheck --nodeadconnectioncheck --namingishard
I see no need to separate out speed and time timeouts.
2) Set maximum download speed
MaxDownloadSpeed --max-download-speed
I think I've pointed out before that this is a lie at best. You aren't actually throttling anything, you're just adding artificial delay. You can also do this from outside of pacman: https://lists.archlinux.org/pipermail/pacman-dev/2016-September/021367.html FWIW, it's no longer true that curl uses a flat average -- it's now a rolling window which should provide a smoother limiting mechanism. However, that's a very recent change (7.50.2), and requires a libcurl far newer than what we need for compilation (7.32.0).
I would also accept replacing speed with limit.
3) (in a glorious future) Set maximum concurrent downloads
MaxConcurrentDownloads --max-concurrent-downloads
Any argument about these option names will need to be very clearly justified. I don't own a bike so have no need for a bikeshed.
Allan
On 03/12/16 06:54, Dave Reisner wrote:
On Fri, Dec 02, 2016 at 04:47:41PM +1000, Allan McRae wrote:
There was many thread and patches regarding this, so I am starting another!
So far there are two (maybe three) things people want options for to do with the downloader. I see most of these options being set in pacman.conf and rarely specified on the command line. Argue with me if I am wrong here...
Based on that assumption, here are my preferred option names (for both pacman.conf and the command line):
1) Disable low speed timeout
DisableDownloadTimeout --disable-download-timeout
As a user, I would have no idea what this means. As the person who wrote the code that this affects, the thing which is most closely associated with time in this equation can hardly be called a "timeout". It's more of a threshold.
Some other suggestions, because my shed is better than your shed:
--disabledeadnetworkcheck --nocheckdeadnetwork --nodeadconncheck --nodeadconnectioncheck --namingishard
I think timeout captures what this is trying to do and is the commonly used term for this kind of functionality in various downloaders. What it actually captures (10sec at less than 1kb/s) can be documented in the man page.
I see no need to separate out speed and time timeouts.
2) Set maximum download speed
MaxDownloadSpeed --max-download-speed
I think I've pointed out before that this is a lie at best. You aren't actually throttling anything, you're just adding artificial delay. You can also do this from outside of pacman:
https://lists.archlinux.org/pipermail/pacman-dev/2016-September/021367.html
FWIW, it's no longer true that curl uses a flat average -- it's now a rolling window which should provide a smoother limiting mechanism. However, that's a very recent change (7.50.2), and requires a libcurl far newer than what we need for compilation (7.32.0).
I realise this. It is the intended effect of the option that I am naming. The fact that curl does not implement it very well should be documented in the man page for that option. And despite our assertion that pacman is cross distro, almost every usage of it will have recent curl...
I would also accept replacing speed with limit.
3) (in a glorious future) Set maximum concurrent downloads
MaxConcurrentDownloads --max-concurrent-downloads
Any argument about these option names will need to be very clearly justified. I don't own a bike so have no need for a bikeshed.
Allan .
participants (4)
-
Allan McRae
-
Christian Hesse
-
Dave Reisner
-
Olivier Brunel