- Arch-mirrors - lists.archlinux.org

Re: [arch-mirrors] [sysadmin] CDN based/caching mirror?
by Kristian Klausen 03 Feb '20

03 Feb '20

On 30.01.2020 17.04, Konstantin Ryabitsev wrote: > On Sun, 26 Jan 2020 at 11:19, Kristian Klausen via arch-mirrors > <arch-mirrors(a)archlinux.org> wrote: >> I'm considering setting up a Arch Linux mirror and I'm considering a >> different design. >> >> So instead of mirroring the whole thing, the idea is to mirror only the >> database files (core.db etc) and download the packages on demand from a >> Tier 1 mirror (and let nginx cache them). By doing it that way, I only >> download requested packages from the Tier 1 mirrors, instead of >> downloading the whole thing (saving Tier 1 bandwidth). >> >> To provide even better performance a CDN (ex: Cloudflare) could be used >> to provide more caching. So we end up with a setup like this: >> Cloudflare -> Nginx cache -> Tier1 mirrors (nginx with multiple upstream) >> >> Do I miss something? Is this a bad idea? > If you are trying to save Tier1 some bandwidth, you'll probably > actually end up causing them more problems due to increased random > seek waits. Tier1 mirrors may not necessarily have fast storage -- for > example, all kernel.org mirror nodes have terabytes of spinning rust > and about half-a-TB of ssd used via lvm-cache. It works great for > Tier1 setups because most Tier2 mirrors want the same set of recent > updates that are served out of ssd cache. If a new mirror comes along > and wants to slurp and entire distro, that is fine too, because even > if there's higher iowait latency, the Tier2 mirror isn't working > against any HTTP timeouts or impatient clients and doesn't care if the > data arrives at a slower rate due to higher iowait. Tier1 can also > tell Tier2 mirror "I'm overloaded right now, please try again later" > and it'll be fine as most Tier2 mirrors can wait an hour or two before > receiving updates. > > Making Tier1 mirrors a "cold cache" for your setup will likely cause > more disk thrash for them, but will also result in poorer service for > people using your mirror due to the reasons I listed above. Tier 1 mirrors is also used directly by end-users (correct me if I'm wrong) So worst-case (cache miss) my SSD-backed shared cache won't be noticeable slower than pulling directly from the Tier 1 mirror. Best-case (cache hit) I'm saving the Tier 1 mirror some bandwidth and disk usage. My idea is basically tiered caching (CDN -> Nginx SSD-backed shared cache -> Tier 1 mirror(s)), is that worse than status quo? :) > If someone > tries to install a package and watches their download bar sit at 0 for > half a minute due to backend proxies fetching data from Tier1 origin, > that's going to result in frustrated people. Nginx streams the data as it is received from the upstream server, so worst-case (cache miss) the data can be delivered as fast as received from the upstream server. > TL;DR: If you can afford CDN-fronting your mirror, that should be > mostly fine, but I would recommend against using Tier1 as your > cache-miss backend. Storage is cheap and most Tier1 mirrors have > unlimited bandwidth, so just run a Tier2 mirror (with slow/fast > storage caching) and keep local copies of everything. > > -K > (mirrors.kernel.org administrator)

1 0

Re: [arch-mirrors] [syscom] CDN based/caching mirror?
by Kristian Klausen 26 Jan '20

26 Jan '20

On 26.01.2020 17.40, Yunxiang Li wrote: > https://wiki.archlinux.org/index.php/Pacman/Tips_and_tricks#Network_shared_… I'm aware of that. https://wiki.archlinux.org/index.php/Pacman/Tips_and_tricks#Dynamic_reverse… is more or less the idea. The idea is basically to mirror the packages when requested, instead of mirroring everything upfront. Is Arch Linux interested is that kind of mirrors? > On January 26, 2020 4:19:10 PM UTC, Kristian Klausen via arch-mirrors <arch-mirrors(a)archlinux.org> wrote: >> Hi >> >> I'm considering setting up a Arch Linux mirror and I'm considering a >> different design. >> >> So instead of mirroring the whole thing, the idea is to mirror only the >> database files (core.db etc) and download the packages on demand from a >> Tier 1 mirror (and let nginx cache them). By doing it that way, I only >> download requested packages from the Tier 1 mirrors, instead of >> downloading the whole thing (saving Tier 1 bandwidth). >> >> To provide even better performance a CDN (ex: Cloudflare) could be used >> to provide more caching. So we end up with a setup like this: >> Cloudflare -> Nginx cache -> Tier1 mirrors (nginx with multiple upstream) >> >> Do I miss something? Is this a bad idea? >> If I do setup a mirror like that, is there any chance it could be added >> as a official mirror? >> >> Best regards >> Kristian Klausen

1 0

Re: [arch-mirrors] archlinux.mirror.digitalpacific.com.au IP change / tier1 query.
by Matthew Taylor 20 Jan '20

20 Jan '20

Hi team, archlinux.mirror.digitalpacific.com.au has been migrated to a new server, and the link speed has been bumped to 20Gbps. If there are any ACLs which need updating, then please add our new IPs: 101.0.120.90 2401:fc00:0:201::4 Also, just as a question on the side.. we've been a Tier2 mirror in Australia for quite a long time (since 2015) -- https://www.archlinux.org/mirrors/digitalpacific.com.au/ ..can you please advise on what the procedure is on becoming a Tier1 mirror for AU? At this point, there's only two other Tier 1 mirrors in AU. Thanks! Regards, Matthew.

3 3

[arch-mirrors] Fwd: [FAR] Aktivoi tunnuksesi
by Amin Vakil 08 Jan '20

08 Jan '20

I've announced a downtime in arch-mirrors-announce mailing list and this email has been sent to me afterwards. Should I do something about this, is it spam or what? -------- Forwarded Message -------- Subject: [FAR] Aktivoi tunnuksesi Date: Wed, 8 Jan 2020 11:22:00 +0000 (UTC) From: FAR ry / Jäsenpalvelut <yllapito(a)far.fi> To: info(a)aminvakil.com Olemme vastaanottaneet tukipyyntösi. Sinulle on luotu tunnus FAR ry / Jäsenpalvelut -palveluportaaliin. Jos halua kirjautua palveluportaaliin, tulee sinun aktivoida käyttäjätunnuksesi. http://jira.far.fi/servicedesk/customer/portal/2/user/visitportal?username=… --------------------------------------------------------------------------------- FAR ry / Jäsenpalvelut lähetti sinulle tämän viestin, käyttäen sovellusta JIRA Service Desk http://www.atlassian.com/software/jira/service-desk?utm_medium=email&utm_so…

2 1

[arch-mirrors] DoS- mirror.chaoticum.net
by Andreas Pfister 23 Oct '19

23 Oct '19

Hi everyone, Today, my logfile (apache2) was full with thousands of thousands of requests like this: 85.14.109.184 - - [21/Oct/2019:14:57:33 +0200] "\xad|\xf8*!\xc7\xf4%\xb4\x0e\x8aj\xc2\xa80\xc2k\xbbh\xdd\xfa\x06\xc3b\x0e\xd8L\x87\xd4\xbd\xd0\x02\x86\xfc\xc6\xe6\xd2\xc1\xad8\v0\r\xfb\xb83\x9d\xca^\xa8h\x97\x99\xad\x9a\xfd\xed\xe1\xd4\xbf^'\xfeg\xbe#\xf0\x9d\x80qM\xb2\xe3A\x8a$Z\x94\xc1*\xae\x11\xf4\x82\xe9\xd14wV\xef\x0ez\xe0\x83\xfe\x07\xab\x86d\xdfN\xb0N6\v\xa8\x1e{\xb0\xc1\xe9\xa3(\xd7E\xc7\xa2\x17\xce\xe5X\xdd@\xc3\x12\xc5\xa8f\x84\xa7\x0e\xe9\xe3:\"\xb89\xb3\xa4u0\x91\xe4\xac\xe2\xb4P\v\x8c\n" 400 0 "-" "-" For this reason, my mirror was not reachable much time. Sorry. For me looks like a dos attack, but i am not sure. Anyone see this anytime in his logfiles or have any further idea/information? Now, i solved the problem by blocking 851 different ip's and i think now running stable. Greetings Andi Pfister

3 2

[arch-mirrors] Expected Downtime of mirror.orbit-os.com
by Daniel Laube 18 Oct '19

18 Oct '19

Hello, because of another server migration the mirror mirror.orbit-os.com [mirror.orbit-os.com] will experience a downtime of ~12-24 hours starting later today. Greetings Daniel Laube

1 0

[arch-mirrors] mirrors.lug.mtu.edu ISOs
by Zachary Jones 25 Sep '19

25 Sep '19

We have worked with our university IT and they discovered an issue with their IPS. They have fixed the issue and we are now syncing ISO's properly. You can add us back as an ISO mirror. Thanks, Zach -- -Zachary Jones Michigan Technological University Computer Networks and System Administration Major Linux Users Group at Michigan Tech: Server Admin IT Oxygen: Operations Husky Amateur Radio Club: President Michigan Tech IT: Ops - Linux Team

2 1

[arch-mirrors] Issue with https://mirror.osbeck.com
by STR . 30 Aug '19

30 Aug '19

I discovered this when I ran pacman -Syu today and it threw errors with the 'libhandy' package. Did some troubleshooting with help from #archlinux and it was suggested I report it here. The file is empty AND the server reports empty `Last-Modified` header. $ wget --debug ' https://mirror.osbeck.com/archlinux/community/os/x86_64/libhandy-0.0.11-1-x…' DEBUG output created by Wget 1.20.3 on linux-gnu. Reading HSTS entries from /home/strykar/.wget-hsts URI encoding = ‘UTF-8’ Converted file name 'libhandy-0.0.11-1-x86_64.pkg.tar.xz' (UTF-8) -> 'libhandy-0.0.11-1-x86_64.pkg.tar.xz' (UTF-8) --2019-08-30 12:54:28-- https://mirror.osbeck.com/archlinux/community/os/x86_64/libhandy-0.0.11-1-x… Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt' Certificates loaded: 139 Resolving mirror.osbeck.com (mirror.osbeck.com)... 104.25.194.31, 104.25.195.31, 2606:4700:20::6819:c21f, ... Caching mirror.osbeck.com => 104.25.194.31 104.25.195.31 2606:4700:20::6819:c21f 2606:4700:20::6819:c31f Connecting to mirror.osbeck.com (mirror.osbeck.com)|104.25.194.31|:443... connected. Created socket 3. Releasing 0x000056402163a210 (new refcount 1). ---request begin--- GET /archlinux/community/os/x86_64/libhandy-0.0.11-1-x86_64.pkg.tar.xz HTTP/1.1 User-Agent: Wget/1.20.3 (linux-gnu) Accept: */* Accept-Encoding: identity Host: mirror.osbeck.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Fri, 30 Aug 2019 07:24:29 GMT Content-Type: application/x-xz Content-Length: 0 Connection: keep-alive Set-Cookie: __cfduid=d455f3bdca693a4bf3eac7aaf482622401567149869; expires=Sat, 29-Aug-20 07:24:29 GMT; path=/; domain=.osbeck.com; HttpOnly; Secure CF-Cache-Status: HIT Cache-Control: public, s-maxage=1200000, stale-while-revalidate=7200 CF-Ray: 50e5067c6951df6f-BOM Accept-Ranges: bytes Age: 5304 etag: "0-0" Expect-CT: max-age=604800, report-uri=" https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" Last-Modified: Strict-Transport-Security: max-age=2592000; includeSubDomains; preload Vary: Accept-Encoding X-Content-Type-Options: nosniff x-modified: 1567144565141 Server: cloudflare ---response end--- 200 OK cdm: 1 Stored cookie osbeck.com -1 (ANY) / <permanent> <secure> [expiry 2020- 08-29 12:54:29] __cfduid d455f3bdca693a4bf3eac7aaf482622401567149869 Registered socket 3 for persistent reuse. Parsed Strict-Transport-Security max-age = 2592000, includeSubDomains = true Added new HSTS host: mirror.osbeck.com:443 (max-age: 2592000, includeSubdomains: true) Length: 0 [application/x-xz] Saving to: ‘libhandy-0.0.11-1-x86_64.pkg.tar.xz’ libhandy-0.0.11-1-x86_64.pkg.tar.xz [ <=> ] 0 --.- KB/s in 0s Last-modified header invalid -- time-stamp ignored. 2019-08-30 12:54:29 (0.00 B/s) - ‘libhandy-0.0.11-1-x86_64.pkg.tar.xz’ saved [0/0] Saving HSTS entries to /home/strykar/.wget-hsts

2 1

[arch-mirrors] [mirror.lty.me] Emergency Maintenance Notice
by Tianyu Liu 26 Aug '19

26 Aug '19

Hello, mirror.lty.me is expecting to migrate to a new server in the next few hours. The mirror might be down during migration. The service is expected to be fully restored within 24 hours. Regards, -- Tianyu Liu lty1993(a)gmail.com

1 0

[arch-mirrors] Change of contact email for mirrors.uk2.net
by David Precious 26 Jul '19

26 Jul '19

Hi, I'm the chap responsible for mirrors.uk2.net, but the email address you will have on record for our Arch mirror is likely mirrors(a)uk2.net which is no longer active due to a restructuring of our mail. Please update your records to replace mirrors(a)uk2.net with my work address, david.precious(a)uk2group.com in case you need to reach me (although I'm subscribed to the arch-mirrors list with my personal email address anyway). Many thanks Dave P

2 1