I'm toying with a clamav-lts for the extended support of 0.104 and I have a strange problem downloading the signature file that results in a 403 permission issue from curl, but downloading directly from the clamav directory (with the same link) works?
Short version:
pkgname=clamav-lts pkgver=0.103.4 ... source=(https://www.clamav.net/downloads/production/$%7Bpkgname%-lts%7D-$%7Bpkgver%7... ...
taken directly from the 0.103.3 PKGBUILD.
From the command line both curl and wget fail with the same issue.
The official Arch clamav for 0.104.1 retrieved tonight with asp export shows the same behavior when trying to build the package.
Am I missing a key or signature. I thought that any user should be able to asp export clamav and then 'makepkg -g' and retrieve the source files?
IIRC it has something to do with network configuration.
On my machine /etc/resolv.conf isn't a link against something systemd related:
[rocketmouse@archlinux tmp]$ /bin/ls -l /etc/resolv.conf -rw-r--r-- 1 root root 94 Nov 16 15:44 /etc/resolv.conf
IIRC this might be the culprit.
1. ####
I experienced the same issue with several PKGBUILDs from AUR. An example from 2018, I needed to change
url="http://search.cpan.org/dist/Goo-Canvas/"
to
url="https://metacpan.org/pod/Goo::Canvas"
and
source=("http://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-$pkgver.tar....")
to
source=("https://cpan.metacpan.org/authors/id/Y/YE/YEWENBIN/Goo-Canvas-$pkgver.tar.gz")
to get rid of "403 Forbidden".
2. ####
If I copy the clamav signature link from a Firefox download, then wget works:
[rocketmouse@archlinux test]$ wget "https://clamav-site.s3.amazonaws.com/production/release_files/files/000/000/..." The destination name is too long (292), reducing to 236 --2021-11-20 07:39:05-- https://clamav-site.s3.amazonaws.com/production/release_files/files/000/000/... SSL_INIT Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt' Resolving clamav-site.s3.amazonaws.com (clamav-site.s3.amazonaws.com)... 52.217.38.156 Connecting to clamav-site.s3.amazonaws.com (clamav-site.s3.amazonaws.com)|52.217.38.156|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 801 [] Saving to: ‘clamav-0.103.4.tar.gz.sig?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU7AK5ITMGOEV4EFM%2F20211120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211120T063226Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=33b8bab0’
clamav-0.103.4.tar.gz.sig?X-Amz-Algorithm=AWS4- 100%[======================================================================================================>] 801 --.-KB/s in 0s
2021-11-20 07:39:05 (7.76 MB/s) - ‘clamav-0.103.4.tar.gz.sig?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU7AK5ITMGOEV4EFM%2F20211120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211120T063226Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=33b8bab0’ saved [801/801]
[rocketmouse@archlinux test]$ cat clamav-0.103.4.tar.gz.sig?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU7AK5ITMGOEV4EFM%2F20211120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211120T063226Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=33b8bab0
On 11/20/21 1:06 AM, Ralf Mardorf via arch-general wrote:
- ####
If I copy the clamav signature link from a Firefox download, then wget works:
[rocketmouse@archlinux test]$ wget "https://clamav-site.s3.amazonaws.com/production/release_files/files/000/000/..." The destination name is too long (292), reducing to 236 --2021-11-20 07:39:05-- https://clamav-site.s3.amazonaws.com/production/release_files/files/000/000/... SSL_INIT Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt' Resolving clamav-site.s3.amazonaws.com (clamav-site.s3.amazonaws.com)... 52.217.38.156 Connecting to clamav-site.s3.amazonaws.com (clamav-site.s3.amazonaws.com)|52.217.38.156|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 801 [] Saving to: ‘clamav-0.103.4.tar.gz.sig?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU7AK5ITMGOEV4EFM%2F20211120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211120T063226Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=33b8bab0’
clamav-0.103.4.tar.gz.sig?X-Amz-Algorithm=AWS4- 100%[======================================================================================================>] 801 --.-KB/s in 0s
Holy Cow!
Thank you Ralf. I too downloaded in FF to build clamav, but I didn't go back and see what FF actually used as when I went to the site the status-bar url was the same as I was trying to download in source=(...).
So the PKGBUILD source=(...) seems to be choking on some internal redirect at the clamav site that is actually some clamav-site.s3.amazonaws.com site.?.?
I asked about this problem on the clamav IRC (well the discord thing that replaced IRC), and they don't seem to know anything about the redirect issue.
Is there any standard "short-url" conversion magic that can be used when this sort of thing crops up? At least from my network, I can't even use the current Arch clamav source to build due to this problem.
Hello,
I asked about this problem on the clamav IRC (well the discord thing that replaced IRC), and they don't seem to know anything about the redirect issue.
For some reason clamav blocks the curl user agent for .sig files ... This could be worked around by using a custom user agent for curl like
curl -A makepkg -L "https://www.clamav.net/downloads/production/clamav-0.103.4.tar.gz.sig"
The curl commands in makepkg.conf are not doing that, maybe that should be changed.
Is there any standard "short-url" conversion magic that can be used when this sort of thing crops up? At least from my network, I can't even use the current Arch clamav source to build due to this problem.
Something like this works at least in this case:
curl -so /dev/null --max-filesize 1 -H "Range: bytes=0-0" -A makepkg -L "https://www.clamav.net/downloads/production/clamav-0.103.2.tar.gz.sig" -w %{url_effective}
-- Andreas Bosch
On Sat, 20 Nov 2021 09:54:28 +0100, Andreas Bosch wrote:
curl -A makepkg -L "https://www.clamav.net/downloads/production/clamav-0.103.4.tar.gz.sig"
Years back I tested it with "http://search.cpan.org/" and similar URLs. Since it didn't work, I haven't tested it today. However, you a right, it works. Without user-agent [1], with user-agent [2]. The "http://search.cpan.org/" issue still isn't solved [3].
[1] #### #### #### ####
[rocketmouse@archlinux extra-x86_64]$ grep curl /etc/makepkg.conf DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u' 'ftp::/usr/bin/curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' 'http::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'https::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' [rocketmouse@archlinux extra-x86_64]$ makepkg -s ==> Making package: clamav 0.104.1-1 (Sat 20 Nov 2021 11:40:27 CET) [snip] curl: (22) The requested URL returned error: 403 ==> ERROR: Failure while downloading https://www.clamav.net/downloads/production/clamav-0.104.1.tar.gz.sig Aborting...
[2] #### #### #### ####
[rocketmouse@archlinux extra-x86_64]$ grep curl /etc/makepkg.conf DLAGENTS=('file::/usr/bin/curl --user-agent archlinux -gqC - -o %o %u' 'ftp::/usr/bin/curl --user-agent archlinux -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' 'http::/usr/bin/curl --user-agent archlinux -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'https::/usr/bin/curl --user-agent archlinux -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' [rocketmouse@archlinux extra-x86_64]$ makepkg -s ==> Making package: clamav 0.104.1-1 (Sat 20 Nov 2021 11:46:16 CET) [snip] -> Downloading clamav-0.104.1.tar.gz.sig... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 467 0 467 0 0 1223 0 --:--:-- --:--:-- --:--:-- 1225 100 801 100 801 0 0 933 0 --:--:-- --:--:-- --:--:-- 2206 -> Found clamav.logrotate -> Found clamav.tmpfiles -> Found clamav.sysusers ==> Validating source files with sha512sums... clamav-0.104.1.tar.gz ... Passed clamav-0.104.1.tar.gz.sig ... Skipped clamav.logrotate ... Passed clamav.tmpfiles ... Passed clamav.sysusers ... Passed ==> Verifying source file signatures with gpg... clamav-0.104.1.tar.gz ... Passed ==> Extracting sources... -> Extracting clamav-0.104.1.tar.gz with bsdtar ==> Starting prepare()... ==> Starting build()... [snip]
[3] #### #### #### ####
[rocketmouse@archlinux extra-x86_64]$ curl -L "https://cpan.metacpan.org/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output Goo-Canvas-0.06.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 103k 100 103k 0 0 782k 0 --:--:-- --:--:-- --:--:-- 778k [rocketmouse@archlinux extra-x86_64]$ curl --user-agent archlinux -L "http://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output Goo-Canvas-0.06.tar.gz-2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 162 100 162 0 0 2605 0 --:--:-- --:--:-- --:--:-- 2612 [rocketmouse@archlinux extra-x86_64]$ cat Goo-Canvas-0.06.tar.gz-2 <html> <head><title>403 Forbidden</title></head> <body bgcolor="white"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> [rocketmouse@archlinux extra-x86_64]$ tar xfv Goo-Canvas-0.06.tar.gz Goo-Canvas-0.06/ Goo-Canvas-0.06/Changes Goo-Canvas-0.06/META.yml Goo-Canvas-0.06/maps Goo-Canvas-0.06/MANIFEST Goo-Canvas-0.06/xs/ [snip]
Am 20.11.21 um 12:10 schrieb Ralf Mardorf via arch-general:
The "http://search.cpan.org/" issue still isn't solved [3].
Interesting. I have no problems with search.cpan.org at all, not even with the default agent. I'd guess somehow they identified your ip or ip range as some problematic for too many automated downloads?
$ curl -L "http://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output goo-1.tar.gz $ curl -L "https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output goo-2-https.tar.gz $ curl -L "https://cpan.metacpan.org/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output goo-3-https-metacpan.tar.gz $ curl -A archlinux -L "https://cpan.metacpan.org/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output goo-4-agent-https-metacpan.tar.gz $ curl -A archlinux -L "https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output goo-5-agent-https.tar.gz $ curl -A archlinux -L "http://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output goo-6-agent.tar.gz $ md5sum goo-* 7dfe0be8c17bfd641d18384d4fd8fb23 goo-1.tar.gz 7dfe0be8c17bfd641d18384d4fd8fb23 goo-2-https.tar.gz 7dfe0be8c17bfd641d18384d4fd8fb23 goo-3-https-metacpan.tar.gz 7dfe0be8c17bfd641d18384d4fd8fb23 goo-4-agent-https-metacpan.tar.gz 7dfe0be8c17bfd641d18384d4fd8fb23 goo-5-agent-https.tar.gz 7dfe0be8c17bfd641d18384d4fd8fb23 goo-6-agent.tar.gz $ bsdtar -tf goo-1.tar.gz | head -n2 Goo-Canvas-0.06/ Goo-Canvas-0.06/Changes
By the way, here is a better command for resolving URL redirections:
$ curl -A archlinux -IXGET -Lso /dev/null -w %{url_effective} "http://test.greenbytes.de/tech/tc/httpredirects/t307loc.asis"
or with --max-redirs if there are multiple redirects and you only want the first stage:
$ curl -IXGET --max-redirs 1 -Lso /dev/null -w %{url_effective} "http://test.greenbytes.de/tech/tc/httpredirects/t307loc.asis"
On Sat, 20 Nov 2021 13:11:41 +0100, Andreas Bosch via arch-general wrote:
Am 20.11.21 um 12:10 schrieb Ralf Mardorf via arch-general:
The "http://search.cpan.org/" issue still isn't solved [3].
Interesting. I have no problems with search.cpan.org at all, not even with the default agent. I'd guess somehow they identified your ip or ip range as some problematic for too many automated downloads?
I don't know the reason for this issue.
$ curl -A archlinux
curl -ArchLinux
:D
When I updated my current /etc/makepkg.conf today I kept "--user-agent archlinux" to avoid confusion.
Thanks to this thread I run meld /etc/makepkg.conf.pacnew /etc/makepkg.conf and noticed that /etc/makepkg.conf needed a few updates.
On Sat, 20 Nov 2021 at 13:22, Ralf Mardorf via arch-general < arch-general@lists.archlinux.org> wrote:
On Sat, 20 Nov 2021 13:11:41 +0100, Andreas Bosch via arch-general wrote:
Am 20.11.21 um 12:10 schrieb Ralf Mardorf via arch-general:
The "http://search.cpan.org/" issue still isn't solved [3].
I think this is just a case of HSTS. You are trying to fetch resources
over HTTP, whereas it's set to enforce the use of https
Just make sure you use https everywhere and you should be fine
On Sat, 20 Nov 2021 16:05:07 +0000, Andy Pieters via arch-general wrote:
On Sat, 20 Nov 2021 at 13:22, Ralf Mardorf via arch-general < arch-general@lists.archlinux.org> wrote:
On Sat, 20 Nov 2021 13:11:41 +0100, Andreas Bosch via arch-general wrote:
Am 20.11.21 um 12:10 schrieb Ralf Mardorf via arch-general:
The "http://search.cpan.org/" issue still isn't solved [3].
I think this is just a case of HSTS. You are trying to fetch resources
over HTTP, whereas it's set to enforce the use of https
Just make sure you use https everywhere and you should be fine
That's what I did. I replaced a PKGBUILD's
http://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz
by
https://cpan.metacpan.org/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz
so I removed the search and migrated from http to https.
However, using the search URL with https does still return a 403.
[rocketmouse@archlinux tmp]$ curl --user-agent archlinux -L "https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output 1_Goo-Canvas-0.06.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. [rocketmouse@archlinux tmp]$ curl --user-agent archlinux --insecure -L "https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output 2_Goo-Canvas-0.06.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 162 100 162 0 0 1321 0 --:--:-- --:--:-- --:--:-- 1327 [rocketmouse@archlinux tmp]$ tar xf 2_Goo-Canvas-0.06.tar.gz | head -5 tar: This does not look like a tar archive
gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now [rocketmouse@archlinux tmp]$ strings 2_Goo-Canvas-0.06.tar.gz | head -5 <html> <head><title>403 Forbidden</title></head> <body bgcolor="white"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center>
On Sat, 20 Nov 2021 at 16:25, Ralf Mardorf via arch-general < arch-general@lists.archlinux.org> wrote:
[rocketmouse@archlinux tmp]$ curl --user-agent archlinux -L " https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output 1_Goo-Canvas-0.06.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.se/docs/sslcerts.html
hmm except it isn't a self-signed certificate at all. SSL Server Certificate Common Name (CN) *.metacpan.org Organisation (O) <Not part of certificate> Organisational Unit (OU) <Not part of certificate> Common Name (CN) GlobalSign Atlas R3 DV TLS CA 2020 Organisation (O) GlobalSign nv-sa Organisational Unit (OU) <Not part of certificate> Issued On Monday, 22 February 2021 at 18:31:
Something fishy is going on in your internet?
[Andy@A-Pieters tmp]$ curl --user-agent archlinux -L " https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output 1_Goo-Canvas-0.06.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed .. Previous command exit with status code: 0
On Sat, 20 Nov 2021 16:39:08 +0000, Andy Pieters via arch-general wrote:
Something fishy is going on in your internet?
But to my knowledge this is the only issue. Some day I'll test https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz with my NomadBSD USB stick and/or with my Ventoy Xubuntu 20.04 LTS USB stick and report back.
Hi Ralf,
That's what I did. I replaced a PKGBUILD's http://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz by https://cpan.metacpan.org/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz so I removed the search and migrated from http to https.
However, using the search URL with https does still return a 403.
$ curl --user-agent archlinux -L
"https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output 1_Goo-Canvas-0.06.tar.gz
That URL isn't either of the two you initially give. :-)
curl: (60) SSL certificate problem: self signed certificate
...
$ curl --user-agent archlinux --insecure -L
"https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz" --output 2_Goo-Canvas-0.06.tar.gz
...
$ strings 2_Goo-Canvas-0.06.tar.gz | head -5
<html> <head><title>403 Forbidden</title></head>
Try precisely this:
curl -sSvg -L \ https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz \ >foo.tar.gz
This will give more detail on each connection and how the certificates fare. I get a valid tar file.
$ b2sum -l32 foo.tar.gz 3a2a9dd1 foo.tar.gz $ tar tf foo.tar.gz | wc -l 47
On Sun, 21 Nov 2021 08:00:31 +0000, Ralph Corderoy wrote:
Try precisely this:
curl -sSvg -L \ https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz \ >foo.tar.gz
Hi Ralph,
done:
[rocketmouse@archlinux curl_search_cpan_01]$ curl -sSvg -L https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz >foo.tar.gz * Trying 46.43.35.68:443... * Connected to search.cpan.org (46.43.35.68) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [108 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [1202 bytes data] * TLSv1.2 (OUT), TLS alert, unknown CA (560): } [2 bytes data] * SSL certificate problem: self signed certificate * Closing connection 0 curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. [rocketmouse@archlinux curl_search_cpan_01]$
Regards, Ralf
Hi Ralf,
$ curl -sSvg -L https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz >foo.tar.gz
- Trying 46.43.35.68:443...
That's a different IP address than what I get.
$ dig +short @8.8.8.8 search.cpan.org. cdn-fastly-sni.cpan.org. dualstack.j.sni.global.fastly.net. 151.101.2.132 151.101.66.132 151.101.130.132 151.101.194.132 $ $ dig +short @8.8.4.4 search.cpan.org. cdn-fastly-sni.cpan.org. dualstack.j.sni.global.fastly.net. 151.101.2.132 151.101.66.132 151.101.130.132 151.101.194.132 $ $ dig +short @1.1.1.1 search.cpan.org. cdn-fastly-sni.cpan.org. dualstack.j.sni.global.fastly.net. 151.101.2.132 151.101.66.132 151.101.130.132 151.101.194.132 $
Ralph Corderoy via arch-general arch-general@lists.archlinux.org wrote:
Hi Ralf,
$ curl -sSvg -L https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz >foo.tar.gz
- Trying 46.43.35.68:443...
For me, u34, 46.43.35.68 points to bm-n1.metacpan.org. bm-n1.metacpan.org can not be resolved (NXDOMAIN). metacpan.org presents itself as a search engine for CPAN.
-- u34
That's a different IP address than what I get.
$ dig +short @8.8.8.8 search.cpan.org. cdn-fastly-sni.cpan.org. dualstack.j.sni.global.fastly.net. 151.101.2.132 151.101.66.132 151.101.130.132 151.101.194.132 $ $ dig +short @8.8.4.4 search.cpan.org. cdn-fastly-sni.cpan.org. dualstack.j.sni.global.fastly.net. 151.101.2.132 151.101.66.132 151.101.130.132 151.101.194.132 $ $ dig +short @1.1.1.1 search.cpan.org. cdn-fastly-sni.cpan.org. dualstack.j.sni.global.fastly.net. 151.101.2.132 151.101.66.132 151.101.130.132 151.101.194.132 $-- Cheers, Ralph.
Am 21.11.21 um 09:17 schrieb Ralf Mardorf via arch-general:
done:
[rocketmouse@archlinux curl_search_cpan_01]$ curl -sSvg -L https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz >foo.tar.gz
- Trying 46.43.35.68:443...
Hi Ralf,
Somehow you seem get an endpoint that has only an internal certificate. I can duplicate your error with this comand:
$ curl --resolve search.cpan.org:443:46.43.35.68 -sSvg -L https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz >foo.tar.gz
I get the same IPs as Ralph and those do work and have Let's Encrypt certificates:
curl -4 -sSvg -L https://search.cpan.org/CPAN/authors/id/Y/YE/YEWENBIN/Goo-Canvas-0.06.tar.gz >foo.tar.gz * Trying 151.101.114.132:443... * Connected to search.cpan.org (151.101.114.132) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [19 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [4019 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [52 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [52 bytes data] * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.cpan.org * start date: Sep 30 05:43:29 2021 GMT * expire date: Dec 29 05:43:28 2021 GMT * subjectAltName: host "search.cpan.org" matched cert's "*.cpan.org" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. * Using HTTP2, server supports multiplexing * Connection state changed (HTTP/2 confirmed)
Here is the wrong internal certificate:
$ openssl s_client --servername search.cpan.org 46.43.35.68:443 CONNECTED(00000003) depth=0 C = PL, ST = Perl Lane, L = Perl City, O = MetaCPAN, OU = NOC, CN = api.metacpan.org, emailAddress = noc@metacpan.org verify error:num=18:self signed certificate verify return:1 depth=0 C = PL, ST = Perl Lane, L = Perl City, O = MetaCPAN, OU = NOC, CN = api.metacpan.org, emailAddress = noc@metacpan.org verify return:1 --- Certificate chain 0 s:C = PL, ST = Perl Lane, L = Perl City, O = MetaCPAN, OU = NOC, CN = api.metacpan.org, emailAddress = noc@metacpan.org i:C = PL, ST = Perl Lane, L = Perl City, O = MetaCPAN, OU = NOC, CN = api.metacpan.org, emailAddress = noc@metacpan.org --- Server certificate -----BEGIN CERTIFICATE----- MIIEpDCCA4ygAwIBAgIJAKx9b4awQ9JxMA0GCSqGSIb3DQEBCwUAMIGSMQswCQYD VQQGEwJQTDESMBAGA1UECBMJUGVybCBMYW5lMRIwEAYDVQQHEwlQZXJsIENpdHkx ETAPBgNVBAoTCE1ldGFDUEFOMQwwCgYDVQQLEwNOT0MxGTAXBgNVBAMTEGFwaS5t ZXRhY3Bhbi5vcmcxHzAdBgkqhkiG9w0BCQEWEG5vY0BtZXRhY3Bhbi5vcmcwHhcN MTYxMTE5MTgyODU4WhcNMzMwNDI0MTgyODU4WjCBkjELMAkGA1UEBhMCUEwxEjAQ BgNVBAgTCVBlcmwgTGFuZTESMBAGA1UEBxMJUGVybCBDaXR5MREwDwYDVQQKEwhN ZXRhQ1BBTjEMMAoGA1UECxMDTk9DMRkwFwYDVQQDExBhcGkubWV0YWNwYW4ub3Jn MR8wHQYJKoZIhvcNAQkBFhBub2NAbWV0YWNwYW4ub3JnMIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEA900TvKRxw8jLouxOGSXksFs1LqOy+kws79dmzUPS 4O6pD1Aj/iY3Sy37IaIR55TAKzGzeaB/r4V0eVgfGq6s8uuRfqsDPVFbtON5JA1V bV9ZkNUlOa74TPuDex5NRbmnom3Nwff/4R8uvT7Z13fcG85OESo3eAtJnGmMeg40 sTgJilqTPPeb9aXEgBZDP1a8WRX+Tp7/0XLZbSkLO4jCdhVyH7ZHpjIX4RbsrGao yTGkR6w/w4SU2E2WcRuXDI1KU3O/uyNW67gDZPkPaW55oqkAAfMgSfF6I8qZpb0W wYu7eSbNBAJbfPJM8wf3uYUtws1gdJzUEsOlDavbcSdVJwIDAQABo4H6MIH3MB0G A1UdDgQWBBS9iUaroj5Ag/Z8EqfHEE8tQD9qPzCBxwYDVR0jBIG/MIG8gBS9iUar oj5Ag/Z8EqfHEE8tQD9qP6GBmKSBlTCBkjELMAkGA1UEBhMCUEwxEjAQBgNVBAgT CVBlcmwgTGFuZTESMBAGA1UEBxMJUGVybCBDaXR5MREwDwYDVQQKEwhNZXRhQ1BB TjEMMAoGA1UECxMDTk9DMRkwFwYDVQQDExBhcGkubWV0YWNwYW4ub3JnMR8wHQYJ KoZIhvcNAQkBFhBub2NAbWV0YWNwYW4ub3JnggkArH1vhrBD0nEwDAYDVR0TBAUw AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAlVmTatV61RdmrM7Uoj8OcPg7w9qbfCJy gAg9OQfj8qA3dloSKSIl+3q7Bkgr1IsmpR5uKQu8HLLbyZMs2G6t40etHaI1o+AJ Q0uY63FksbVwmEIp/4pdrt6VMqASG6OfzO2SJbd6wb7GGoFZmvPA5CQ50Jv8wEkn T6+IT8VynV+ZR2kWBUXANexuIrThwBTGihyej9rvZgYTJ3aKGgCT9Teov1T6A7Ed dUbpdIJG9QdFliBmPO049ej7h3N79EarmUyuN5Z0tQDwrXZLJ1gxrAsSXw/InEao TO2m3xZXEDGRfQSHf1YJt/sgpoYwGPQ1KOWuPIBmp1mD5h7CcMGpIg== -----END CERTIFICATE----- subject=C = PL, ST = Perl Lane, L = Perl City, O = MetaCPAN, OU = NOC, CN = api.metacpan.org, emailAddress = noc@metacpan.org
issuer=C = PL, ST = Perl Lane, L = Perl City, O = MetaCPAN, OU = NOC, CN = api.metacpan.org, emailAddress = noc@metacpan.org
--- No client certificate CA names sent Peer signing digest: SHA512 Peer signature type: RSA Server Temp Key: X25519, 253 bits --- SSL handshake has read 1670 bytes and written 410 bytes Verification error: self signed certificate ^C
-- Andreas
Hi,
at the moment I don't know how to override the nameserver assigned by the O2 HomeBox 6641. Some information claims that it is impossible.
:(
Regards, Ralf
Op zo 21 nov. 2021 12:35 schreef Ralf Mardorf via arch-general < arch-general@lists.archlinux.org>:
Hi,
at the moment I don't know how to override the nameserver assigned by the O2 HomeBox 6641. Some information claims that it is impossible.
As long as you control the host itself, this shouldn't be too big a problem. DHCP is convenient, not obligatory.
Worst case: edit /etc/resolv.conv directly.
Mvg, Guus
On Sun, 21 Nov 2021 13:35:20 +0100, Guus Snijders wrote:
Op zo 21 nov. 2021 12:35 schreef Ralf Mardorf:
at the moment I don't know how to override the nameserver assigned by the O2 HomeBox 6641. Some information claims that it is impossible.
As long as you control the host itself, this shouldn't be too big a problem. DHCP is convenient, not obligatory.
Worst case: edit /etc/resolv.conv directly.
Hi,
do you think of PPPoE pass-through or something else?
Regards, Ralf
Ralf Mardorf via arch-general arch-general@lists.archlinux.org wrote:
On Sun, 21 Nov 2021 13:35:20 +0100, Guus Snijders wrote:
Op zo 21 nov. 2021 12:35 schreef Ralf Mardorf:
at the moment I don't know how to override the nameserver assigned by the O2 HomeBox 6641. Some information claims that it is impossible.
As long as you control the host itself, this shouldn't be too big a problem. DHCP is convenient, not obligatory.
Worst case: edit /etc/resolv.conv directly.
Hi,
do you think of PPPoE pass-through or something else?
Regards, Ralf
My understanding to his suggestion is to start with
nameserver 8.8.8.8 nameserver 8.8.4.4
in /etc/resolv.conf. Just those 2 lines.
-- u34
On Sun, 21 Nov 2021 14:06:53 +0000, u34--- wrote:
Ralf Mardorf via arch-general arch-general@lists.archlinux.org wrote:
On Sun, 21 Nov 2021 13:35:20 +0100, Guus Snijders wrote:
Op zo 21 nov. 2021 12:35 schreef Ralf Mardorf:
at the moment I don't know how to override the nameserver assigned by the O2 HomeBox 6641. Some information claims that it is impossible.
As long as you control the host itself, this shouldn't be too big a problem. DHCP is convenient, not obligatory.
Worst case: edit /etc/resolv.conv directly.
do you think of PPPoE pass-through or something else?
My understanding to his suggestion is to start with
nameserver 8.8.8.8 nameserver 8.8.4.4
in /etc/resolv.conf. Just those 2 lines.
Hi,
the connection with the O2 HomeBox 6641 is done by
nameserver 192.168.1.1
the O2 HomeBox 6641 then does use an O2 nameserver. It's a long time ago that I used PPP [1], I even don't remember, if I've ever used it with this router. However, when neither using DHCP nor PPPoE pass-through, how do I connect with the router?
Regards, Ralf
[1] [rocketmouse@archlinux ~]$ /bin/ls -hAl /etc/ppp/peers/ total 4.0K -rw-r--r-- 1 root root 98 Feb 18 2013 alice lrwxrwxrwx 1 root root 20 Apr 22 2015 provider -> /etc/ppp/peers/alice [rocketmouse@archlinux ~]$ head -2 /etc/ppp/peers/alice plugin rp-pppoe.so enp3s0 [rocketmouse@archlinux ~]$ /bin/ls -hAl /usr/local/sbin/alice* -rwxr-xr-x 1 root root 2.1K Apr 13 2016 /usr/local/sbin/alice -rwxr-xr-x 1 root root 1.8K Apr 13 2016 /usr/local/sbin/alice-dhcp [rocketmouse@archlinux ~]$ grep pppoe_on( -A12 /usr/local/sbin/alice pppoe_on() { if [ "$(pidof pppd)" ]; then printf "\nError: An instance of pppd is already running.\n\n"; exit 1 else echo ; modprobe -v pppoe; ip link set enp3s0 up d="" if [ ! "$(grep Arch /etc/issue)" ]; then d="dsl-provider" fi pon $d printf "\nEstablishing the connection might take a while.\n\n" fi }
That's how I establish an Internet connection nowadays:
[rocketmouse@archlinux ~]$ grep dhcpcd_on( -A2 /usr/local/sbin/alice-dhcp dhcpcd_on() { echo ; dhcpcd $(basename $(ls -d /sys/class/net/enp?s0)) ; echo }
Ralf Mardorf via arch-general arch-general@lists.archlinux.org wrote:
On Sun, 21 Nov 2021 14:06:53 +0000, u34--- wrote:
Ralf Mardorf via arch-general arch-general@lists.archlinux.org wrote:
On Sun, 21 Nov 2021 13:35:20 +0100, Guus Snijders wrote:
Op zo 21 nov. 2021 12:35 schreef Ralf Mardorf:
at the moment I don't know how to override the nameserver assigned by the O2 HomeBox 6641. Some information claims that it is impossible.
As long as you control the host itself, this shouldn't be too big a problem. DHCP is convenient, not obligatory.
Worst case: edit /etc/resolv.conv directly.
do you think of PPPoE pass-through or something else?
My understanding to his suggestion is to start with
nameserver 8.8.8.8 nameserver 8.8.4.4
in /etc/resolv.conf. Just those 2 lines.
Hi,
the connection with the O2 HomeBox 6641 is done by
nameserver 192.168.1.1the O2 HomeBox 6641 then does use an O2 nameserver. It's a long time ago that I used PPP [1], I even don't remember, if I've ever used it with this router. However, when neither using DHCP nor PPPoE pass-through, how do I connect with the router?
Regards, Ralf
[1] [rocketmouse@archlinux ~]$ /bin/ls -hAl /etc/ppp/peers/ total 4.0K -rw-r--r-- 1 root root 98 Feb 18 2013 alice lrwxrwxrwx 1 root root 20 Apr 22 2015 provider -> /etc/ppp/peers/alice [rocketmouse@archlinux ~]$ head -2 /etc/ppp/peers/alice plugin rp-pppoe.so enp3s0 [rocketmouse@archlinux ~]$ /bin/ls -hAl /usr/local/sbin/alice* -rwxr-xr-x 1 root root 2.1K Apr 13 2016 /usr/local/sbin/alice -rwxr-xr-x 1 root root 1.8K Apr 13 2016 /usr/local/sbin/alice-dhcp [rocketmouse@archlinux ~]$ grep pppoe_on( -A12 /usr/local/sbin/alice pppoe_on() { if [ "$(pidof pppd)" ]; then printf "\nError: An instance of pppd is already running.\n\n"; exit 1 else echo ; modprobe -v pppoe; ip link set enp3s0 up d="" if [ ! "$(grep Arch /etc/issue)" ]; then d="dsl-provider" fi pon $d printf "\nEstablishing the connection might take a while.\n\n" fi }
That's how I establish an Internet connection nowadays:
[rocketmouse@archlinux ~]$ grep dhcpcd_on( -A2 /usr/local/sbin/alice-dhcp dhcpcd_on() { echo ; dhcpcd $(basename $(ls -d /sys/class/net/enp?s0)) ; echo }
I am confused by that output. I don't know the answers to your questions. Never the less, my suggestion is to
# mv -v /etc/resolv.conf /etc/resolv.conf.bak
If that fails because /etc/resolv.conf doesn't exist in the first place, then ignore the failure, and
# echo "nameserver 8.8.8.8\nnameserver 8.8.4.4" > /etc/resolv.conf
as root. Then
$ ping -c1 search.cpan.org
And see what IP it pings to.
Once again: even though I don't know the answers to your questions, my suggestion is to issue these commands after the router is connected in the usual way. Just try it, and see if that helps. I assume it should not interfer with your internet connection. It is supposed to improve your name resolution by bypassing the router process of name resolution. Hopefully it will override only the router name resolution, and nothing else. Just try it. Hopefully, it will work.
-- u34
On 11/20/21 2:54 AM, Andreas Bosch via arch-general wrote:
For some reason clamav blocks the curl user agent for .sig files ... This could be worked around by using a custom user agent for curl like
curl -A makepkg -L "https://www.clamav.net/downloads/production/clamav-0.103.4.tar.gz.sig"The curl commands in makepkg.conf are not doing that, maybe that should be changed.
Thanks Andreas & Ralf,
After Ralf pointed me to the redirect that was occurring and failing, I let the clamav folks know and apparently there are a few things they are looking into as to why their source is being treated differently from their sig from the cloud provider.
I suspect this type of issue isn't uncommon in the "it's hosted in someone else's cloud" days we live in. Instead of only having one site involved, there may be many with the referral from the first (or URL magic from some provider's DNS entry) in the middle.
I'll work through the different curl options and see what works here.
On 11/20/21 2:54 AM, Andreas Bosch via arch-general wrote:
For some reason clamav blocks the curl user agent for .sig files ... This could be worked around by using a custom user agent for curl like
curl -A makepkg -L "https://www.clamav.net/downloads/production/clamav-0.103.4.tar.gz.sig"The curl commands in makepkg.conf are not doing that, maybe that should be changed.
The folks at clamav have now fixed the signature redirect problem that caused makepkg (curl) to fail to download the .sig file for 0.103.4. (micah_s and sfirefench on discord).
I suspect most of the problems like this are due to incorrect configurations for the redirects from the package site to wherever the actual file lives on a cloud service provider host.
At least here, a quick post to the upstream site was able to get this resolved.
Foreseeable growing pains that are the side-effect of giving up control of where you data actually lives...
On Sat, 20 Nov 2021 02:11:33 -0600, David C. Rankin wrote:
Holy Cow!
[rocketmouse@archlinux ~]$ apt moo (__) (oo) /------/ / | || * /---/\ ~~ ~~ ..."Have you mooed today?"...
Is there any standard "short-url" conversion magic that can be used when this sort of thing crops up? At least from my network, I can't even use the current Arch clamav source to build due to this problem.
To my knowledge there isn't. You can install apt and try to run "apt moo" with root privileges at full moon 42 times.
IIRC a long time ago I made /etc/resolv.conf a file again to access the Internet when using systemd-nspawn without the -b, --boot option.
I don't know if this is related to the issue or if something else is the culprit. Seemingly something else does cause this issue.
After "enabling" /run/systemd/resolve/stub-resolv.conf, I tried wget and curl with different options still to no avail, before running dhcpd. Then I run dhcpd and after that makepkg.
After this test I stopped systemd-networkd.service and systemd-resolved.service and replaced the link by a resolv.conf file again ;). systemd-networkd.service is probably not required to use systemd-resolved.service.
I don't know how to get rid of this issue.
1. ####
[rocketmouse@archlinux etc]$ /bin/ls -ltr resolv* -rw-r--r-- 1 root root 255 Dec 30 2020 resolvconf.conf -rw-r--r-- 1 root root 65 Nov 11 10:14 resolv.conf.pacnew -rw-r--r-- 1 root root 701 Nov 16 15:35 resolv.conf.bak -rw-r--r-- 1 root root 94 Nov 16 15:44 resolv.conf [rocketmouse@archlinux etc]$ sudo mv resolv.conf resolv.conf-2021-11-20 [rocketmouse@archlinux etc]$ sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf [rocketmouse@archlinux tmp]$ cd /tmp/ [rocketmouse@archlinux tmp]$ sudo nano /etc/systemd/network/20-wired.network [rocketmouse@archlinux tmp]$ cat /etc/systemd/network/20-wired.network [Match] Name=enp1s0
[Network] DHCP=yes [rocketmouse@archlinux tmp]$ systemctl status systemd-networkd.service [rocketmouse@archlinux tmp]$ sudo systemctl start systemd-resolved.service
2. ####
[rocketmouse@archlinux tmp]$ sudo dhcpcd -x $(basename $(ls -d /sys/class/net/enp?s0)) dhcpcd not running [rocketmouse@archlinux tmp]$ sudo dhcpcd $(basename $(ls -d /sys/class/net/enp?s0)) sending commands to dhcpcd process [rocketmouse@archlinux extra-x86_64]$ /bin/ls -ltr /etc/resolv* /run/systemd/resolve/ -rw-r--r-- 1 root root 255 Dec 30 2020 /etc/resolvconf.conf -rw-r--r-- 1 root root 65 Nov 11 10:14 /etc/resolv.conf.pacnew -rw-r--r-- 1 root root 701 Nov 16 15:35 /etc/resolv.conf.bak -rw-r--r-- 1 root root 94 Nov 16 15:44 /etc/resolv.conf-2021-11-20 lrwxrwxrwx 1 root root 37 Nov 20 09:47 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
/run/systemd/resolve/: total 8 -rw-r--r-- 1 systemd-resolve systemd-resolve 920 Nov 20 10:08 stub-resolv.conf -rw-r--r-- 1 systemd-resolve systemd-resolve 981 Nov 20 10:08 resolv.conf srw-rw-rw- 1 systemd-resolve systemd-resolve 0 Nov 20 10:08 io.systemd.Resolve [rocketmouse@archlinux extra-x86_64]$ diff PKGBUILD PKGBUILD.bak 7,8c7,8 < pkgname=clamav-lts < pkgver=0.103.4 ---
pkgname=clamav pkgver=0.104.1
23c23 < source=(https://www.clamav.net/downloads/production/$%7Bpkgname%-lts%7D-$%7Bpkgver%7... ---
source=(https://www.clamav.net/downloads/production/$%7Bpkgname%7D-$%7Bpkgver%7D.tar...
27c27 < sha512sums=('SKIP' ---
sha512sums=('2cd4f73de73a2bbc002e1aa85326ea30cce0073fc1a2d5d7d220465217a84eb97fac759010ae0af54d2f0ed725112a51a65a486491fa52388cd7652d7b5cfa5a'
29,31c29,31 < 'SKIP' < 'SKIP' < 'SKIP') ---
'9cb168c1c16bb43c99900d7ef34456e3f3b593d4d1943c875a0306bc86fd3872cb78e9e1413dcba93579e01b96d466c9eea1975e24190193663b7986c4525d48' 'c5443634399bd87fe0d0192518538ffdb7296a8437b5b0160a0fbd58696b01285de3237e3feb552c0095c49e576832dec2e2b2107eef2be42424ed7edd13cd19' 'b984836f6c34d97b90d81fa5d17361a2e3f8c0cc709e3350a4d25cf088dc04f7bf2504359980c8be489c96b1b8798c60e6da533069d3378d49d4f50f929a2c90')
[rocketmouse@archlinux extra-x86_64]$ makepkg -s ==> Making package: clamav-lts 0.103.4-1 (Sat 20 Nov 2021 10:39:07 CET) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Retrieving sources... -> Downloading clamav-0.103.4.tar.gz... [snip] --:--:-- 0 curl: (22) The requested URL returned error: 403 ==> ERROR: Failure while downloading https://www.clamav.net/downloads/production/clamav-0.103.4.tar.gz.sig Aborting...
3. ####
[rocketmouse@archlinux etc]$ sudo systemctl stop systemd-networkd.service; sudo systemctl stop systemd-resolved.service [rocketmouse@archlinux etc]$ sudo mv -i resolv.conf resolv.conf.dangling_link && sudo mv -i resolv.conf-2021-11-20 resolv.conf
And just in case I rebooted :D.
arch-general@lists.archlinux.org