[aur-general] Problem downloading a source package with new curl version
Hi,
one of my aur packages (eccodes) has a problem getting the source package using the new version of curl (7.79.0).
The output of the old curl version looks like: ----------------------------------------------------------------------------- jankoh@kohni-mobil ~/projects/eccodes $ curl -v https://confluence.ecmwf.int/ download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2 * Trying 136.156.180.232:443... * Connected to confluence.ecmwf.int (136.156.180.232) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: jurisdictionC=GB; businessCategory=Non-Commercial Entity; serialNumber=ISBN: 9780101820028; C=GB; ST=West Berkshire; L=Reading; O=European Cen tre for Medium-Range Weather Forecasts; CN=confluence.ecmwf.int * start date: Mar 29 16:16:45 2021 GMT * expire date: Mar 29 16:26:00 2022 GMT * subjectAltName: host "confluence.ecmwf.int" matched cert's "confluence.ecmwf.int" * issuer: C=NL; O=QuoVadis Trustlink B.V.; CN=QuoVadis Europe EV SSL CA G1 * SSL certificate verify ok.
GET /download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2 HTTP/
1.1
Host: confluence.ecmwf.int User-Agent: curl/7.78.0 Accept: */*
* Mark bundle as not supporting multiuse < HTTP/1.1 200 200 < Date: Sun, 19 Sep 2021 19:48:11 GMT < Server: Apache < Cache-Control: no-cache, must-revalidate < Expires: Thu, 01 Jan 1970 00:00:00 GMT < X-Confluence-Request-Time: 1632080871142 < Set-Cookie: JSESSIONID=659BDBFF1F03F2DC7848A7BEECF3CCC3; Path=/; HttpOnly < Last-Modified: Mon, 19 Jan 1970 20:46:07 GMT < Accept-Ranges: bytes < X-Content-Type-Options: nosniff < Content-Disposition: inline; filename="eccodes-2.23.0-Source.tar.gz" < Content-Type: application/x-gzip;charset=UTF-8 < Content-Length: 12037258 < Strict-Transport-Security: max-age=15552000 < Warning: Binary output can mess up your terminal. Use "--output -" to tell Warning: curl to output it to your terminal anyway, or consider "--output Warning: <FILE>" to save to a file. * Failure writing output to destination * Closing connection 0 * TLSv1.2 (OUT), TLS alert, close notify (256): jankoh@kohni-mobil ~/projects/eccodes $ -----------------------------------------------------------------------------
The new version outputs: ----------------------------------------------------------------------------- jankoh@kohni-mobil ~/projects/eccodes $ curl -v -i https:// confluence.ecmwf.int/download/attachments/45757960/eccodes-2.23.0- Source.tar.gz?api=v2 * Trying 136.156.180.232:443... * Connected to confluence.ecmwf.int (136.156.180.232) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: jurisdictionC=GB; businessCategory=Non-Commercial Entity; serialNumber=ISBN: 9780101820028; C=GB; ST=West Berkshire; L=Reading; O=European Centre for Medium-Range Weather Forecasts; CN=confluence.ecmwf.int * start date: Mar 29 16:16:45 2021 GMT * expire date: Mar 29 16:26:00 2022 GMT * subjectAltName: host "confluence.ecmwf.int" matched cert's "confluence.ecmwf.int" * issuer: C=NL; O=QuoVadis Trustlink B.V.; CN=QuoVadis Europe EV SSL CA G1 * SSL certificate verify ok.
GET /download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2 HTTP/
1.1
Host: confluence.ecmwf.int User-Agent: curl/7.79.0 Accept: */*
* Unsupported response code in HTTP response * Closing connection 0 * TLSv1.2 (OUT), TLS alert, close notify (256): curl: (1) Unsupported response code in HTTP response jankoh@kohni-mobil ~/projects/eccodes $ -----------------------------------------------------------------------------
I see the response code is 200 200, which has an additional errournous 200. Since the upgrade my automated daily test builds fail, so I'd like to know if I should file a bug report either upstream or the arch bugtracker.
I can download the file using chrome or wget, so it might be a workaroud if I could tell makepkg to use wget instead of curl.
However, executing ----------------------------------------------------------------------------- sudo sed -i -r "s/(http[s]?::)/usr/bin/curl\ -qgb\ ""\ -fLC\ -\ --retry\ 3\ --retry-delay\ 3 -o/\1/usr/sbin/wget\ --no-cookies\ --retry-connrefused\ -t\ 3\ --waitretry\ 3 -O/" /etc/makepkg.conf ----------------------------------------------------------------------------- inside the build container will not help users that upgrade or install the package. I could post the workaround to the packages page, but this will also affect other packages to be build, and wget needs to be installed before that workaround will actually work.
Any tips on how to go on?
TIA
Did you merge the changes on "/etc/makepkg.conf"? There were curl specific changes coming with pacman 6.0.1-2... I can't tell about your issues, but for other packages, I've had no issues, but I did the merge as soon as it was available...
-- Javier
On 9/20/21 13:50, Jan Kohnert via aur-general wrote:
Hi,
one of my aur packages (eccodes) has a problem getting the source package using the new version of curl (7.79.0).
The output of the old curl version looks like:
jankoh@kohni-mobil ~/projects/eccodes $ curl -v https://confluence.ecmwf.int/ download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2
- Trying 136.156.180.232:443...
- Connected to confluence.ecmwf.int (136.156.180.232) port 443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
- CApath: none
- TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.2 (IN), TLS handshake, Certificate (11):
- TLSv1.2 (IN), TLS handshake, Server key exchange (12):
- TLSv1.2 (IN), TLS handshake, Server finished (14):
- TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
- TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.2 (OUT), TLS handshake, Finished (20):
- TLSv1.2 (IN), TLS handshake, Finished (20):
- SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
- ALPN, server did not agree to a protocol
- Server certificate:
- subject: jurisdictionC=GB; businessCategory=Non-Commercial Entity;
serialNumber=ISBN: 9780101820028; C=GB; ST=West Berkshire; L=Reading; O=European Cen tre for Medium-Range Weather Forecasts; CN=confluence.ecmwf.int
- start date: Mar 29 16:16:45 2021 GMT
- expire date: Mar 29 16:26:00 2022 GMT
- subjectAltName: host "confluence.ecmwf.int" matched cert's
"confluence.ecmwf.int"
- issuer: C=NL; O=QuoVadis Trustlink B.V.; CN=QuoVadis Europe EV SSL CA G1
- SSL certificate verify ok.
GET /download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2 HTTP/
1.1
Host: confluence.ecmwf.int User-Agent: curl/7.78.0 Accept: */*
- Mark bundle as not supporting multiuse
< HTTP/1.1 200 200 < Date: Sun, 19 Sep 2021 19:48:11 GMT < Server: Apache < Cache-Control: no-cache, must-revalidate < Expires: Thu, 01 Jan 1970 00:00:00 GMT < X-Confluence-Request-Time: 1632080871142 < Set-Cookie: JSESSIONID=659BDBFF1F03F2DC7848A7BEECF3CCC3; Path=/; HttpOnly < Last-Modified: Mon, 19 Jan 1970 20:46:07 GMT < Accept-Ranges: bytes < X-Content-Type-Options: nosniff < Content-Disposition: inline; filename="eccodes-2.23.0-Source.tar.gz" < Content-Type: application/x-gzip;charset=UTF-8 < Content-Length: 12037258 < Strict-Transport-Security: max-age=15552000 < Warning: Binary output can mess up your terminal. Use "--output -" to tell Warning: curl to output it to your terminal anyway, or consider "--output Warning: <FILE>" to save to a file.
- Failure writing output to destination
- Closing connection 0
- TLSv1.2 (OUT), TLS alert, close notify (256):
jankoh@kohni-mobil ~/projects/eccodes $
The new version outputs:
jankoh@kohni-mobil ~/projects/eccodes $ curl -v -i https:// confluence.ecmwf.int/download/attachments/45757960/eccodes-2.23.0- Source.tar.gz?api=v2
- Trying 136.156.180.232:443...
- Connected to confluence.ecmwf.int (136.156.180.232) port 443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
- CApath: none
- TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.2 (IN), TLS handshake, Certificate (11):
- TLSv1.2 (IN), TLS handshake, Server key exchange (12):
- TLSv1.2 (IN), TLS handshake, Server finished (14):
- TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
- TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.2 (OUT), TLS handshake, Finished (20):
- TLSv1.2 (IN), TLS handshake, Finished (20):
- SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
- ALPN, server did not agree to a protocol
- Server certificate:
- subject: jurisdictionC=GB; businessCategory=Non-Commercial Entity;
serialNumber=ISBN: 9780101820028; C=GB; ST=West Berkshire; L=Reading; O=European Centre for Medium-Range Weather Forecasts; CN=confluence.ecmwf.int
- start date: Mar 29 16:16:45 2021 GMT
- expire date: Mar 29 16:26:00 2022 GMT
- subjectAltName: host "confluence.ecmwf.int" matched cert's
"confluence.ecmwf.int"
- issuer: C=NL; O=QuoVadis Trustlink B.V.; CN=QuoVadis Europe EV SSL CA G1
- SSL certificate verify ok.
GET /download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2 HTTP/
1.1
Host: confluence.ecmwf.int User-Agent: curl/7.79.0 Accept: */*
- Unsupported response code in HTTP response
- Closing connection 0
- TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (1) Unsupported response code in HTTP response jankoh@kohni-mobil ~/projects/eccodes $
I see the response code is 200 200, which has an additional errournous 200. Since the upgrade my automated daily test builds fail, so I'd like to know if I should file a bug report either upstream or the arch bugtracker.
I can download the file using chrome or wget, so it might be a workaroud if I could tell makepkg to use wget instead of curl.
However, executing
sudo sed -i -r "s/(http[s]?::)/usr/bin/curl\ -qgb\ ""\ -fLC\ -\ --retry\ 3\ --retry-delay\ 3 -o/\1/usr/sbin/wget\ --no-cookies\ --retry-connrefused\
-t\ 3\ --waitretry\ 3 -O/" /etc/makepkg.conf
inside the build container will not help users that upgrade or install the package. I could post the workaround to the packages page, but this will also affect other packages to be build, and wget needs to be installed before that workaround will actually work.
Any tips on how to go on?
TIA
On 2021-09-20 21:50, Jan Kohnert via aur-general wrote:
Hi,
one of my aur packages (eccodes) has a problem getting the source package using the new version of curl (7.79.0).
The output of the old curl version looks like:
jankoh@kohni-mobil ~/projects/eccodes $ curl -v https://confluence.ecmwf.int/ download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2
- Trying 136.156.180.232:443...
- Connected to confluence.ecmwf.int (136.156.180.232) port 443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
- CApath: none
- TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.2 (IN), TLS handshake, Certificate (11):
- TLSv1.2 (IN), TLS handshake, Server key exchange (12):
- TLSv1.2 (IN), TLS handshake, Server finished (14):
- TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
- TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.2 (OUT), TLS handshake, Finished (20):
- TLSv1.2 (IN), TLS handshake, Finished (20):
- SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
- ALPN, server did not agree to a protocol
- Server certificate:
- subject: jurisdictionC=GB; businessCategory=Non-Commercial Entity;
serialNumber=ISBN: 9780101820028; C=GB; ST=West Berkshire; L=Reading; O=European Cen tre for Medium-Range Weather Forecasts; CN=confluence.ecmwf.int
- start date: Mar 29 16:16:45 2021 GMT
- expire date: Mar 29 16:26:00 2022 GMT
- subjectAltName: host "confluence.ecmwf.int" matched cert's
"confluence.ecmwf.int"
- issuer: C=NL; O=QuoVadis Trustlink B.V.; CN=QuoVadis Europe EV SSL CA G1
- SSL certificate verify ok.
GET /download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2 HTTP/
1.1
Host: confluence.ecmwf.int User-Agent: curl/7.78.0 Accept: */*
- Mark bundle as not supporting multiuse
< HTTP/1.1 200 200 < Date: Sun, 19 Sep 2021 19:48:11 GMT < Server: Apache < Cache-Control: no-cache, must-revalidate < Expires: Thu, 01 Jan 1970 00:00:00 GMT < X-Confluence-Request-Time: 1632080871142 < Set-Cookie: JSESSIONID=659BDBFF1F03F2DC7848A7BEECF3CCC3; Path=/; HttpOnly < Last-Modified: Mon, 19 Jan 1970 20:46:07 GMT < Accept-Ranges: bytes < X-Content-Type-Options: nosniff < Content-Disposition: inline; filename="eccodes-2.23.0-Source.tar.gz" < Content-Type: application/x-gzip;charset=UTF-8 < Content-Length: 12037258 < Strict-Transport-Security: max-age=15552000 < Warning: Binary output can mess up your terminal. Use "--output -" to tell Warning: curl to output it to your terminal anyway, or consider "--output Warning: <FILE>" to save to a file.
- Failure writing output to destination
- Closing connection 0
- TLSv1.2 (OUT), TLS alert, close notify (256):
jankoh@kohni-mobil ~/projects/eccodes $
The new version outputs:
jankoh@kohni-mobil ~/projects/eccodes $ curl -v -i https:// confluence.ecmwf.int/download/attachments/45757960/eccodes-2.23.0- Source.tar.gz?api=v2
- Trying 136.156.180.232:443...
- Connected to confluence.ecmwf.int (136.156.180.232) port 443 (#0)
- ALPN, offering h2
- ALPN, offering http/1.1
- successfully set certificate verify locations:
- CAfile: /etc/ssl/certs/ca-certificates.crt
- CApath: none
- TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.2 (IN), TLS handshake, Certificate (11):
- TLSv1.2 (IN), TLS handshake, Server key exchange (12):
- TLSv1.2 (IN), TLS handshake, Server finished (14):
- TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
- TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
- TLSv1.2 (OUT), TLS handshake, Finished (20):
- TLSv1.2 (IN), TLS handshake, Finished (20):
- SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
- ALPN, server did not agree to a protocol
- Server certificate:
- subject: jurisdictionC=GB; businessCategory=Non-Commercial Entity;
serialNumber=ISBN: 9780101820028; C=GB; ST=West Berkshire; L=Reading; O=European Centre for Medium-Range Weather Forecasts; CN=confluence.ecmwf.int
- start date: Mar 29 16:16:45 2021 GMT
- expire date: Mar 29 16:26:00 2022 GMT
- subjectAltName: host "confluence.ecmwf.int" matched cert's
"confluence.ecmwf.int"
- issuer: C=NL; O=QuoVadis Trustlink B.V.; CN=QuoVadis Europe EV SSL CA G1
- SSL certificate verify ok.
GET /download/attachments/45757960/eccodes-2.23.0-Source.tar.gz?api=v2 HTTP/
1.1
Host: confluence.ecmwf.int User-Agent: curl/7.79.0 Accept: */*
- Unsupported response code in HTTP response
- Closing connection 0
- TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (1) Unsupported response code in HTTP response jankoh@kohni-mobil ~/projects/eccodes $
I see the response code is 200 200, which has an additional errournous 200. Since the upgrade my automated daily test builds fail, so I'd like to know if I should file a bug report either upstream or the arch bugtracker.
I can download the file using chrome or wget, so it might be a workaroud if I could tell makepkg to use wget instead of curl.
However, executing
sudo sed -i -r "s/(http[s]?::)/usr/bin/curl\ -qgb\ ""\ -fLC\ -\ --retry\ 3\ --retry-delay\ 3 -o/\1/usr/sbin/wget\ --no-cookies\ --retry-connrefused\
-t\ 3\ --waitretry\ 3 -O/" /etc/makepkg.conf
inside the build container will not help users that upgrade or install the package. I could post the workaround to the packages page, but this will also affect other packages to be build, and wget needs to be installed before that workaround will actually work.
Any tips on how to go on?
TIA
This is likely due to a recent change in curl, being stricter about response codes.
HTTP 1.1: disallow >3-digit response codes The HTTP protocol is defined to only allow three-digit numbers and
now curl enforces that check stricter. This was in part made to align behavior when curl is built to use hyper.
ref
https://daniel.haxx.se/blog/2021/09/15/curl-7-79-0-secure-local-cookies/
So the arch bugtracker is probably the appropriate one.
- v
Hi Javier,
Am Montag, 20. September 2021, 22:07:15 CEST schrieb Javier via aur-general:
Did you merge the changes on "/etc/makepkg.conf"? There were curl specific changes coming with pacman 6.0.1-2...
thanks for your suggestion. I'm using the official arch-build docker container (archlinux:base-devel) as base for my testbuilds, so I assumed the changes have been made therein.
But I had a closer look. The relevant changes were: +++ /etc/makepkg.conf.pacnew 2021-09-15 15:38:35.000000000 +0200 @@ -9,10 +9,10 @@ # #-- The download utilities that makepkg should use to acquire sources # Format: 'protocol::agent' -DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u' - 'ftp::/usr/bin/curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' - 'http::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'https::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' +DLAGENTS=('file::/usr/bin/curl -qgC - -o %o %u' + 'ftp::/usr/bin/curl -qgfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' + 'http::/usr/bin/curl -qgb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'https::/usr/bin/curl -qgb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'rsync::/usr/bin/rsync --no-motd -z %u %o' 'scp::/usr/bin/scp -C %u %o')
so it's basically just reordering -g and -q options. I made the config merge on my local system and it does not change the behaviour (as excpected). Also a short investigation in the build container showed the changes have been made by the container image builders.
Hi Victor,
Am Montag, 20. September 2021, 22:16:11 CEST schrieb Victor Häggqvist via aur- general:
On 2021-09-20 21:50, Jan Kohnert via aur-general wrote:
Hi,
one of my aur packages (eccodes) has a problem getting the source package using the new version of curl (7.79.0).
[...]
This is likely due to a recent change in curl, being stricter about response codes.
HTTP 1.1: disallow >3-digit response codes The HTTP protocol is defined to only allow three-digit numbers and
now curl enforces that check stricter. This was in part made to align behavior when curl is built to use hyper.
ref
https://daniel.haxx.se/blog/2021/09/15/curl-7-79-0-secure-local-cookies/
So the arch bugtracker is probably the appropriate one.
thanks, I'll report an issue then.
Am Montag, 20. September 2021, 22:46:49 CEST schrieb Jan Kohnert via aur- general:
Am Montag, 20. September 2021, 22:16:11 CEST schrieb Victor Häggqvist via aur- general:
So the arch bugtracker is probably the appropriate one.
thanks, I'll report an issue then.
Reported as https://bugs.archlinux.org/task/72186
participants (3)
-
Jan Kohnert
-
Javier
-
Victor Häggqvist