Hi all, I was able to reproduce this issue, but at the same time I looked closer at the logs and I found the cause of it: it was a misconfigured nginx server block. All the files that were reported 404 by pacman are *.sig, and I had a location block config that turns off access logging for *.sig, and this config shouldn't have been on the reverse proxy at all which causes the reverse proxy to respond 404 to *.sig file requests instead of passing the requests to upstreams, that's why the 404s didn't appear in the access log, which had me confused for a few days. Sorry everyone... stupid mistakes here. <-- That's why getting enough sleep is important, otherwise things like this happen :( On Wed, Sep 6, 2023 at 6:16 AM Tyler Dence wrote:

As a potential cause/possible test condition - I'd be curious if there was some sort of time desync between the machine and mirror. Theoretically, if the pacman client were "in the future" compared to the mirror, an if-modified-since header could result in a 304.

libalpm/dload.c:362 potentially supports this line of thinking, since the timestamp used to query is the mtime of the file on the local filesystem as returned by the stat call on 359 (looking at v6.0.1-133-gf9d8beef).

On Tue, Sep 5, 2023, 14:30 pitastrudl wrote:

...

On 8/29/23 12:12, Jing Luo wrote:

...

Hi all,

I don't know if this is an intended behavior for pacman. Today I ran `pacman -Syyu` against the mirror hosted by me, and pacman output 404 for all the packages:

``` Total Download Size: 395.48 MiB Total Installed Size: 2273.64 MiB Net Upgrade Size: 133.43 MiB

:: Proceed with installation? [Y/n] :: Retrieving packages... noto-fonts-cjk-20230817-1-any is up to date linux-6.4.12.arch1-1-x86_64 is up to date ktuberling-23.08.0-1-x86_64 is up to date marble-common-23.08.0-1-x86_64 is up to date klettres-23.08.0-1-x86_64 is up to date linux-headers-6.4.12.arch1-1-x86_64 is up to date minuet-23.08.0-1-x86_64 is up to date kalzium-23.08.0-1-x86_64 is up to date linux-docs-6.4.12.arch1-1-x86_64 is up to date archlinux-appstream-data-20230827-2-any is up to date Total ( 10/217) 395.5 MiB 22.7 GiB/s 00:00 [#######################################################################################] 100% error: failed retrieving file 'noto-fonts-cjk-20230817-1-any.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'linux-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'archlinux-appstream-data-20230827-2-any.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 warning: too many errors from repo.jing.rocks, skipping for the remainder of this transaction error: failed retrieving file 'ktuberling-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'linux-headers-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'marble-common-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'kalzium-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'klettres-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'minuet-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 error: failed retrieving file 'linux-docs-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The requested URL returned error: 404 warning: failed to retrieve some files error: failed to commit transaction (failed to retrieve some files) Errors occurred, no packages were upgraded. ```

I thought my mirror was out of date at first, so I ran the syncrepo script manually, but it looks like only the file lists were transferred. I searched on the arch forum, but in their cases, they have out-of-date package index or they change the mirrorlist to solve this problem. Not exactly my case. So I looked at the nginx access.log:

``` - - - [29/Aug/2023:08:06:42 +0900] "GET /archlinux/core/os/x86_64/core.db HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:42 +0900] "GET /archlinux/multilib/os/x86_64/multilib.db HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:42 +0900] "GET /archlinux/extra/os/x86_64/extra.db HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/noto-fonts-cjk-20230817-1-any.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/core/os/x86_64/linux-6.4.12.arch1-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/ktuberling-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/marble-common-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/klettres-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/core/os/x86_64/linux-headers-6.4.12.arch1-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/minuet-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/kalzium-23.08.0-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/core/os/x86_64/linux-docs-6.4.12.arch1-1-x86_64.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" - - - [29/Aug/2023:08:06:43 +0900] "GET /archlinux/extra/os/x86_64/archlinux-appstream-data-20230827-2-any.pkg.tar.zst HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" ```

After a little bit of research, I found that nginx seems to conform to the standard according to my setup: gzip off, telling the browser (in this case, pacman) to use the client side local cache, returning a 304. Note that my mirror site is set up like this: a load-balancer/reverse proxy (repo.jing.rocks) in front of two web servers (repo1 and repo2) connected to the same NFS storage backend. Caching to disk is not enabled in nginx.

After that I decided to put these three lines into my location block so that nginx always returns 200:

``` add_header Cache-Control "no-cache"; etag off; if_modified_since off; ```

It looks all good now, except I can't reproduce this pacman bug (?) anymore. Now that arch's gitlab closed new account registration, I thought I should probably look for experienced mirror operators' input before submitting a bug report. What do y'all think? Is this a bug?

(Also my mirror hasn't been accepted after 2 months https://bugs.archlinux.org/task/78991 looks like the mirror admin team is taking their sweet sweet time?)

BR,

Hey,

I myself have not seen this error in my time being here and it seems it would be a good idea to open an issue on gitlab if you'd want someone to possibly look into it but if you cannot reliably reproduce it ,you can still ask for any ideas. Were there any updates between the problem and any possible upgrades that might fix it? The registration is disabled due to spam issues that affect generally most public gitlab instances and you can email accountsupport@archlinux.org to open an account and open a bugreport there and someone who has more indepth knowledge of pacman might be able to help.

Regards, Arun

-- Jing Luo About me: https://jing.rocks/about/ PGP Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC