On 05/05/2023 09:25, Morten Linderud wrote:

Hi,

The AUR is currently unavailable because of some broken refs in the underlying Git repository that needs to be fixed.

We are working on solving the issue but it might take a couple more hours.

Hello,

The AUR experienced some issues last night with the backend git repo which serves all packages. This resulted in some reflogs in the git repo being corrupted. We used the Github [1] copy of the repo to rebuild the corrupted objects in the git repo and confirmed the repo is OK and we haven't lost history from it.

Please do let us know if you notice further issues or updates that are missing from the UI.

Apologies for any inconvenient this has caused you.

You can find more information on the issue and what we think happened and cause that problem below.

We have a systemd service [1] which is used to push to github a copy of the underlying aur git repo [2]. This kept failing yesterday night due to some commits missing a committer email [3]. The commits without a committed email are done by a bot which auto-updates the AUR package. We had that before and we set the repo to reject such commits [4]. We overwritten the history of the commits which were invalid in the offended package [2] setting the user as `user@example.com` and left a comment to maintainer to stop using that bot. The service which syncs with Github kept failing so we initiated a `git gc --prune=now --aggressive` operation to remove the unused reflogs. We believe that during that operation and while git was trying to repack objects the system was live and accepting package updates the corruption of the objects happened. We were able to identify which objects were corrupted and rebuild them from Github which had them intact.

We don't believe there was a failure from the hardware / software. Going forward we would be a bit more careful on how we handle such operations and schedule them during maintenance, even if it means the Github mirror would be out of sync until that maintenance.

It would be nice though to not have to handle corrupted commits due to automatic package update bots committing with wrong committer email. We also intend to investigate why our fail-safe didn't catch these [3].

Finally I'd like to thank DevOps team for assistance in this and moson [5] especially which helped immensely in resolving the issue.

Best Regards,

[1]: https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/roles/au... [2]: https://github.com/archlinux/aur [3]: https://aur.archlinux.org/packages/blender-plugin-luxcorerender-latest-bin [4]: https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/465b835da6311... [5]: https://gitlab.archlinux.org/moson