Hi all, I don't know if this is an intended behavior for pacman. Today I ran `pacman -Syyu` against the mirror hosted by me, and pacman output 404 for all the packages: ``` Total Download Size: 395.48 MiB Total Installed Size: 2273.64 MiB Net Upgrade Size: 133.43 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... noto-fonts-cjk-20230817-1-any is up to date linux-6.4.12.arch1-1-x86_64 is up to date ktuberling-23.08.0-1-x86_64 is up to date marble-common-23.08.0-1-x86_64 is up to date klettres-23.08.0-1-x86_64 is up to date linux-headers-6.4.12.arch1-1-x86_64 is up to date minuet-23.08.0-1-x86_64 is up to date kalzium-23.08.0-1-x86_64 is up to date linux-docs-6.4.12.arch1-1-x86_64 is up to date archlinux-appstream-data-20230827-2-any is up to date Total ( 10/217) 395.5 MiB 22.7 GiB/s 00:00 [#######################################################################################] 100% error: failed retrieving file 'noto-fonts-cjk-20230817-1-any.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'linux-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'archlinux-appstream-data-20230827-2-any.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 warning: too many errors from repo.jing.rocks, skipping for the remainder of this transaction error: failed retrieving file 'ktuberling-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'linux-headers-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'marble-common-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'kalzium-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'klettres-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'minuet-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'linux-docs-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 warning: failed to retrieve some files error: failed to commit transaction (failed to retrieve some files) Errors occurred, no packages were upgraded. ``` I thought my mirror was out of date at first, so I ran the syncrepo script manually, but it looks like only the file lists were transferred. I searched on the arch forum, but in their cases, they have out-of-date package index or they change the mirrorlist to solve this problem. Not exactly my case. So I looked at the nginx access.log: ``` - - - [29/Aug/2023:08:06:42 +0900] "GET /archlinux/core/os/x86_64/core.db HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:42 +0900] "GET /archlinux/multilib/os/x86_64/multilib.db HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:42 +0900] "GET /archlinux/extra/os/x86_64/extra.db HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/noto-fonts-cjk-20230817-1-any.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/core/os/x86_64/linux-6.4.12.arch1-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/ktuberling-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/marble-common-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/klettres-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/core/os/x86_64/linux-headers-6.4.12.arch1-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/minuet-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/kalzium-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/core/os/x86_64/linux-docs-6.4.12.arch1-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/archlinux-appstream-data-20230827-2-any.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" ``` After a little bit of research, I found that nginx seems to conform to the standard according to my setup: gzip off, telling the browser (in this case, pacman) to use the client side local cache, returning a 304. Note that my mirror site is set up like this: a load-balancer/reverse proxy (repo.jing.rocks) in front of two web servers (repo1 and repo2) connected to the same NFS storage backend. Caching to disk is not enabled in nginx. After that I decided to put these three lines into my location block so that nginx always returns 200: ``` add_header Cache-Control "no-cache"; etag off; if_modified_since off; ``` It looks all good now, except I can't reproduce this pacman bug (?) anymore. Now that arch's gitlab closed new account registration, I thought I should probably look for experienced mirror operators' input before submitting a bug report. What do y'all think? Is this a bug? (Also my mirror hasn't been accepted after 2 months https://bugs.archlinux.org/task/78991 looks like the mirror admin team is taking their sweet sweet time?) BR, -- Jing Luo About me: https://jing.rocks/about/ PGP Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC