[arch-general] pacman doesn't show download progress
pacman -Syu show database download progress but not packages being downloaded. They do get downloaded and installation progress is rightly displayed. This can be annoying when connection speed is poor. This has been observed on three machines. I am obviously requesting a solution and am ready to provide any additional required information. Thanks.
On 08/15/2017 03:37 PM, SET wrote:
pacman -Syu show database download progress but not packages being downloaded. They do get downloaded and installation progress is rightly displayed. This can be annoying when connection speed is poor. This has been observed on three machines.
I am obviously requesting a solution and am ready to provide any additional required information.
I'm not sure what to say, other than that I see progress every time I download packages, unless they are already cached and therefore do not need to be downloaded. Have you set an external downloader? -- Eli Schwartz
Le mardi 15 août 2017 21:43:46 CEST Eli Schwartz a écrit :
Have you set an external downloader?
No, I'm using the default downloader. [options] #RootDir = / #DBPath = /var/lib/pacman/ #CacheDir = /var/cache/pacman/pkg/ #LogFile = /var/log/pacman.log #GPGDir = /etc/pacman.d/gnupg/ HoldPkg = pacman glibc #XferCommand = /usr/bin/curl -C - -f %u > %o #XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u #CleanMethod = KeepInstalled #UseDelta = 0.7 Architecture = auto IgnoreGroup = xorg IgnorePkg = xf86-input-keyboard xf86-input-mouse xf86-video-vmware IgnorePkg = qt5-doc minuet ocaml #NoUpgrade = #NoExtract = # Misc options #UseSyslog Color #TotalDownload CheckSpace
Can you explain in more detail what the problem is, and show actual and expected output? On 08/15/2017 02:04 PM, SET wrote:
Le mardi 15 août 2017 21:43:46 CEST Eli Schwartz a écrit :
Have you set an external downloader?
No, I'm using the default downloader.
[options] #RootDir = / #DBPath = /var/lib/pacman/ #CacheDir = /var/cache/pacman/pkg/ #LogFile = /var/log/pacman.log #GPGDir = /etc/pacman.d/gnupg/ HoldPkg = pacman glibc #XferCommand = /usr/bin/curl -C - -f %u > %o #XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u #CleanMethod = KeepInstalled #UseDelta = 0.7 Architecture = auto IgnoreGroup = xorg IgnorePkg = xf86-input-keyboard xf86-input-mouse xf86-video-vmware IgnorePkg = qt5-doc minuet ocaml #NoUpgrade = #NoExtract = # Misc options #UseSyslog Color #TotalDownload CheckSpace
(Tue, Aug 15, 2017 at 02:05:44PM -0700) Moses Miller via arch-general :
Can you explain in more detail what the problem is, and show actual and expected output?
Hello, I have the same "problem" for a few weeks too. Here is the output of pacman -Syu I got just now on two machines: https://paste.ee/p/hqCTu https://paste.ee/p/eLu01 In the first one, you see download progress on the packages. In the second one you don't see them. Here you don't see them at all (probably because there are only a few packages), but it happens that I only see a few of them. The behavior is completely random, and on both machines: it's just luck that I could get the two different examples today The only line that differs between both machines in pacman.conf is IgnorePkg: ---------------------------------- [options] HoldPkg = pacman glibc IgnorePkg = (...) Architecture = auto Color CheckSpace SigLevel = Required DatabaseOptional LocalFileSigLevel = Optional [core] Include = /etc/pacman.d/mirrorlist [extra] Include = /etc/pacman.d/mirrorlist [community] Include = /etc/pacman.d/mirrorlist [multilib] Include = /etc/pacman.d/mirrorlist ------------------------------------ [NB: I "think" it's related to curl (didn't check if pacman uses it though) since I have a similar experience once with that command in a non-related use.] Hope it can help. -- Ismael
Could you also share the first few lines of your mirrorlist file?
I don't have much idea about how `pacman` is implemented, so what I'm
saying next is pure guesswork and speculation.
My guess is that there exist some mirror(s) that return a bad HTTP
Response, that is a response without a file-size. A HTTP/1.1 response,
almost always includes a "Content-Length" header which tells the client
program what the total size of the file is. This data is also obviously
important for drawing an accurate progress bar. Now, if the
"Content-Length" header was missing, pacman cannot know the final size
of the file before hand and hence it is not drawing the progress bars.
As a test, I would request that the next time you see something like
this, go through these steps:
1. Delete the just downloaded package from the cache, so that pacman
will download it again
2. Replace the XferCommand line with this:
`XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S
-o %o.log -O %o %u`
3. Run pacman again
This will create a log file containing the Server's response next to the
package in the pacman cache. Take a look at the log file and also please
share the contents here with us. If my guess is correct, we should be
able to see this in the log file, else, we find another way to debug
this
* Ismael Bouya
(Tue, Aug 15, 2017 at 02:05:44PM -0700) Moses Miller via arch-general :
Can you explain in more detail what the problem is, and show actual and expected output?
Hello, I have the same "problem" for a few weeks too. Here is the output of pacman -Syu I got just now on two machines: https://paste.ee/p/hqCTu https://paste.ee/p/eLu01
In the first one, you see download progress on the packages. In the second one you don't see them. Here you don't see them at all (probably because there are only a few packages), but it happens that I only see a few of them. The behavior is completely random, and on both machines: it's just luck that I could get the two different examples today
The only line that differs between both machines in pacman.conf is IgnorePkg: ---------------------------------- [options] HoldPkg = pacman glibc IgnorePkg = (...) Architecture = auto Color CheckSpace SigLevel = Required DatabaseOptional LocalFileSigLevel = Optional [core] Include = /etc/pacman.d/mirrorlist [extra] Include = /etc/pacman.d/mirrorlist [community] Include = /etc/pacman.d/mirrorlist [multilib] Include = /etc/pacman.d/mirrorlist ------------------------------------
[NB: I "think" it's related to curl (didn't check if pacman uses it though) since I have a similar experience once with that command in a non-related use.]
Hope it can help. -- Ismael
-- Thanking You, Darshit Shah PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6
Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S -o %o.log -O %o %u
Where will be the log file ? current working directory ?
* nmset@netcourrier.com
Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
XferCommand = /usr/bin/wget --show-progress -q --passive-ftp -c -S -o %o.log -O %o %u
Where will be the log file ? current working directory ?
No, right next to the packages, in the package cache (/var/cache/pacman/pkg by default) -- Thanking You, Darshit Shah PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6
Le jeudi 17 août 2017 06:46:37 CEST Darshit Shah via arch-general a écrit :
No, right next to the packages, in the package cache (/var/cache/pacman/pkg by default)
Though the problem is resolved by choosing another mirror, here is a sample log file : oxygen-icons-1:5.37.0-1-any.pkg.tar.xz.part.log HTTP/1.1 200 OK Server: nginx/1.6.2 Date: Thu, 17 Aug 2017 18:14:56 GMT Content-Type: application/octet-stream X-Varnish: 537330300 Age: 0 Via: 1.1 varnish-v4 Transfer-Encoding: chunked Accept-Ranges: bytes X-IPLB-Instance: 5865 I don't know how to understand it.
On 08/17/2017 02:23 PM, SET wrote:
Le jeudi 17 août 2017 06:46:37 CEST Darshit Shah via arch-general a écrit :
No, right next to the packages, in the package cache (/var/cache/pacman/pkg by default)
Though the problem is resolved by choosing another mirror, here is a sample log file : oxygen-icons-1:5.37.0-1-any.pkg.tar.xz.part.log
HTTP/1.1 200 OK Server: nginx/1.6.2 Date: Thu, 17 Aug 2017 18:14:56 GMT Content-Type: application/octet-stream X-Varnish: 537330300 Age: 0 Via: 1.1 varnish-v4 Transfer-Encoding: chunked Accept-Ranges: bytes X-IPLB-Instance: 5865
I don't know how to understand it.
Well, look at this: ``` [eschwartz@arch ~]$ pacman -Sp linux http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i6... [eschwartz@arch ~]$ wget -P /tmp/ http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i6... --2017-08-17 14:31:17-- http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i6... Resolving archlinux.mirrors.ovh.net... 213.32.5.7, 2001:41d0:202:100:213:32:5:7 Connecting to archlinux.mirrors.ovh.net|213.32.5.7|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/octet-stream] Saving to: ‘/tmp/linux-4.12.7-1-i686.pkg.tar.xz’ linux-4.12.7-1-i686.pkg.tar.xz [ <=> ] 1.37M 912KB/s ``` Length is unspecified, and wget does its <=> slidy-back-and-forth thing. And it seems pacman just gives up on showing progress information and instead shows nothing (which isn't unreasonable). As for your log, that's just the HTTP headers, something you can also see with curl -I http://... Notice how the Content-Length: header is missing. -- Eli Schwartz
On 08/17/2017 02:43 PM, Eli Schwartz wrote:
Well, look at this:
``` [eschwartz@arch ~]$ pacman -Sp linux http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i6... [eschwartz@arch ~]$ wget -P /tmp/ http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i6... --2017-08-17 14:31:17-- http://archlinux.mirrors.ovh.net/archlinux/testing/os/i686/linux-4.12.7-1-i6... Resolving archlinux.mirrors.ovh.net... 213.32.5.7, 2001:41d0:202:100:213:32:5:7 Connecting to archlinux.mirrors.ovh.net|213.32.5.7|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [application/octet-stream] Saving to: ‘/tmp/linux-4.12.7-1-i686.pkg.tar.xz’
linux-4.12.7-1-i686.pkg.tar.xz [ <=> ] 1.37M 912KB/s
```
Length is unspecified, and wget does its <=> slidy-back-and-forth thing. And it seems pacman just gives up on showing progress information and instead shows nothing (which isn't unreasonable).
As for your log, that's just the HTTP headers, something you can also see with curl -I http://...
Notice how the Content-Length: header is missing.
... and it's probably worth mentioning that sometime in the last 10 minutes they seem to have fixed this. -- Eli Schwartz
On 08/17/2017 02:46 PM, Eli Schwartz wrote:
... and it's probably worth mentioning that sometime in the last 10 minutes they seem to have fixed this.
Or not. It's erratically switching between working and not-working. -- Eli Schwartz
Le jeudi 17 août 2017 20:43:33 CEST Eli Schwartz a écrit :
As for your log, that's just the HTTP headers, something you can also see with curl -I http://...
Notice how the Content-Length: header is missing.
Thanks for the clear explanations, at least I understand what's happening, it's up to remote to fix.
Le vendredi 18 août 2017 11:55:41 CEST mpan a écrit :
Don’t write them to fix that, as nothing is broken: Content-Length is not required.
Of course I won't, ;-)
Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
Could you also share the first few lines of your mirrorlist file?
The first few lines verbatim : Server = http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch #Server = http://mirror.archlinux.ikoula.com/archlinux/$repo/os/$arch Server = http://archlinux.polymorf.fr/$repo/os/$arch ## ## Arch Linux repository mirrorlist ## Generated on 2016-03-10 ## ## Worldwide #Server = http://mirror.rackspace.com/archlinux/$repo/os/$arch ## Australia #Server = http://mirror.aarnet.edu.au/pub/archlinux/$repo/os/$arch
Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
My guess is that there exist some mirror(s) that return a bad HTTP Response, that is a response without a file-size.
You're probably right. pacman has expected output if I change remote mirror. It had to download 150 MB, I interrupted the download and restarted many times. No output while downloading with one mirror, and normal output with another one. At least up to now. Thanks.
On 2017-08-16 20:41, SET wrote:
Le mercredi 16 août 2017 17:09:11 CEST Darshit Shah via arch-general a écrit :
My guess is that there exist some mirror(s) that return a bad HTTP Response, that is a response without a file-size.
You're probably right. pacman has expected output if I change remote mirror. It had to download 150 MB, I interrupted the download and restarted many times. No output while downloading with one mirror, and normal output with another one. At least up to now.
Thanks.
Which mirror was it? We will try to contact them about the issue. B
Le mercredi 16 août 2017 20:48:59 CEST Bartłomiej Piotrowski a écrit :
Which mirror was it? We will try to contact them about the issue.
B
It is : http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch Today at least, the following mirror is working fine : http://archlinux.polymorf.fr/$repo/os/$arch Thx.
On 16.08.2017 21:47, SET wrote:
Le mercredi 16 août 2017 20:48:59 CEST Bartłomiej Piotrowski a écrit :
Which mirror was it? We will try to contact them about the issue.
B
It is : http://archlinux.mirrors.ovh.net/archlinux/$repo/os/$arch
I've notified the mirror admin. I'll post here again if I know more. Florian
On Tue, 15 Aug 2017 23:04:35 +0200, SET wrote:
# Misc options #UseSyslog Color #TotalDownload CheckSpace
Here a progress bar is displayed, but I'm using the "ILoveCandy" Easter egg. [root@archlinux ~]# grep Misc\ options -A8 /etc/pacman.conf # Misc options #UseSyslog #UseDelta #TotalDownload CheckSpace #VerbosePkgLists Color ILoveCandy [root@archlinux ~]# pacman -Q pacman pacman 5.0.2-2
participants (10)
-
Bartłomiej Piotrowski
-
Darshit Shah
-
Eli Schwartz
-
Florian Pritz
-
Ismael Bouya
-
Moses Miller
-
mpan
-
nmset＠netcourrier.com
-
Ralf Mardorf
-
SET