[arch-general] pacman Segfault on full system update

I have an Arch guest in VirtualBox running on an Arch server. I installed virtualbox-bin 1-34-2 from AUR with 5.18 patch and the install on the host went fine. I access the guest headless using rdesktop over the LAN. (have done so for years). Doing a full system update from 5.17.7 to 5.18.6 (about 25 days of updates) on guest. Pacman segfaulted leaving post transaction hooks unrun. I reinstalled the kernel and systemd to force the initrd to be made so the guest is bootable. However, I'm not sure what additional hooks were missed. The guest window when X is started is 1440 x 864 but the only portion of the desktop shown is 1024 x 768 (rest is black and not shown, taskbar unable to be seen, etc.) After the pacman segfault, now the console is also missing the last 7-8 lines. I have to hit clear and start over at the top to see what I'm typing. How can I recover this system? I was able to rsync the last 1000 lines of pacman.log from the guest to the host, but the shared folders no longer work. The pacman.log summary of how the issue occurred is: [2022-06-24T02:06:26-0500] [PACMAN] Running 'pacman -Sy --needed archlinux-keyring' [2022-06-24T02:06:26-0500] [PACMAN] synchronizing package lists [2022-06-24T02:07:03-0500] [PACMAN] Running 'pacman -Su' [2022-06-24T02:07:03-0500] [PACMAN] starting full system upgrade [2022-06-24T02:07:30-0500] [ALPM] running '60-mkinitcpio-remove.hook'... [2022-06-24T02:07:30-0500] [ALPM] transaction started [2022-06-24T02:07:30-0500] [ALPM] upgraded iana-etc (20220427-1 -> 20220603-1) [2022-06-24T02:07:30-0500] [ALPM] warning: /etc/locale.gen installed as /etc/locale.gen.pacnew [2022-06-24T02:07:31-0500] [ALPM] upgraded glibc (2.35-5 -> 2.35-6) [2022-06-24T02:07:31-0500] [ALPM-SCRIPTLET] Generating locales... [2022-06-24T02:07:32-0500] [ALPM-SCRIPTLET] en_US.UTF-8... done [2022-06-24T02:07:32-0500] [ALPM-SCRIPTLET] Generation complete. <snipped lots of packages installed> ... [2022-06-24T02:09:04-0500] [ALPM] upgraded linux-firmware-whence (20220509.b19cbdc-1 -> 20220610.7b71b75-1) Segfault Remove db.lock and continue: [2022-06-24T02:12:57-0500] [PACMAN] Running 'pacman -Syu' [2022-06-24T02:12:57-0500] [PACMAN] synchronizing package lists [2022-06-24T02:12:59-0500] [PACMAN] starting full system upgrade [2022-06-24T02:13:16-0500] [ALPM] transaction started [2022-06-24T02:13:17-0500] [ALPM] error: could not extract /usr/share/licenses/linux-firmware/LICENCE.nvidia (Zstd decompression failed: Restored data doesn't match checksum) [2022-06-24T02:13:17-0500] [ALPM] error: problem occurred while upgrading linux-firmware [2022-06-24T02:13:17-0500] [ALPM] upgraded linux-firmware (20220509.b19cbdc-1 -> 20220610.7b71b75-1) [2022-06-24T02:13:17-0500] [ALPM] transaction failed So I delete the firmware file from /var/cache/pacman/pkg and run the upgrade again. To my surprise, it doesn't download the firmware package again, it just considers it installed and proceeds to other packages, e.g. [2022-06-24T02:15:49-0500] [PACMAN] Running 'pacman -Syu' [2022-06-24T02:15:49-0500] [PACMAN] synchronizing package lists [2022-06-24T02:15:51-0500] [PACMAN] starting full system upgrade [2022-06-24T02:16:03-0500] [ALPM] transaction started [2022-06-24T02:16:03-0500] [ALPM] upgraded python (3.10.4-1 -> 3.10.5-1) [2022-06-24T02:16:09-0500] [ALPM] upgraded linux-headers (5.17.7.arch1-1 -> 5.18.6.arch1-1) [2022-06-24T02:16:11-0500] [ALPM] error: could not extract /usr/lib/mysql/plugin/client_ed25519.so (Zstd decompression failed: Restored data doesn't match checksum) [2022-06-24T02:16:11-0500] [ALPM] error: problem occurred while upgrading mariadb-libs [2022-06-24T02:16:11-0500] [ALPM] upgraded mariadb-libs (10.7.3-1 -> 10.8.3-1) [2022-06-24T02:16:11-0500] [ALPM] transaction failed Crash again and delete mariadb-libs from /var/cache... and try again: [2022-06-24T02:19:16-0500] [PACMAN] Running 'pacman -Syu' [2022-06-24T02:19:16-0500] [PACMAN] synchronizing package lists [2022-06-24T02:19:17-0500] [PACMAN] starting full system upgrade [2022-06-24T02:19:26-0500] [ALPM] transaction started [2022-06-24T02:19:26-0500] [ALPM] upgraded mariadb-clients (10.7.3-1 -> 10.8.3-1) [2022-06-24T02:19:27-0500] [ALPM] warning: directory permissions differ on /usr/lib/mysql/plugin/auth_pam_tool_dir/ filesystem: 700 package: 755 [2022-06-24T02:19:27-0500] [ALPM] upgraded mariadb (10.7.3-1 -> 10.8.3-1) [2022-06-24T02:19:27-0500] [ALPM-SCRIPTLET] :: MariaDB was updated to a new feature release. To update the data run: [2022-06-24T02:19:27-0500] [ALPM-SCRIPTLET] systemctl restart mariadb.service && mariadb-upgrade -u root -p [2022-06-24T02:19:27-0500] [ALPM] warning: /etc/pacman.d/mirrorlist installed as /etc/pacman.d/mirrorlist.pacnew [2022-06-24T02:19:27-0500] [ALPM] upgraded pacman-mirrorlist (20220501-1 -> 20220605-1) <snip lots more packages installed> ... [2022-06-24T02:19:34-0500] [ALPM] upgraded tmux (3.2_a-1 -> 3.3_a-2) [2022-06-24T02:19:35-0500] [ALPM] error: could not extract /usr/share/doc/valgrind/valgrind_manual.ps (Zstd decompression failed: Restored data doesn't match checksum) [2022-06-24T02:19:35-0500] [ALPM] error: problem occurred while upgrading valgrind [2022-06-24T02:19:35-0500] [ALPM] upgraded valgrind (3.19.0-3 -> 3.19.0-4) [2022-06-24T02:19:35-0500] [ALPM] transaction failed Crash again. Rinse, repeat and try again: [2022-06-24T02:20:14-0500] [PACMAN] Running 'pacman -Syu' [2022-06-24T02:20:14-0500] [PACMAN] synchronizing package lists [2022-06-24T02:20:15-0500] [PACMAN] starting full system upgrade [2022-06-24T02:20:20-0500] [ALPM] transaction started [2022-06-24T02:20:21-0500] [ALPM] upgraded vim-runtime (8.2.4827-1 -> 8.2.5046-2) [2022-06-24T02:20:21-0500] [ALPM] upgraded vim (8.2.4827-1 -> 8.2.5046-2) [2022-06-24T02:20:21-0500] [ALPM] upgraded xbitmaps (1.1.2-2 -> 1.1.2-3) [2022-06-24T02:20:21-0500] [ALPM] upgraded xcb-util-cursor (0.1.3-3 -> 0.1.3-4) [2022-06-24T02:20:21-0500] [ALPM] upgraded xcursor-themes (1.0.6-2 -> 1.0.6-3) [2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-bdftopcf (1.1-2 -> 1.1-3) [2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-font-util (1.3.2-2 -> 1.3.2-3) [2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-server-common (21.1.3-6 -> 21.1.3-7) [2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-server (21.1.3-6 -> 21.1.3-7) [2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-xcursorgen (1.0.7-2 -> 1.0.7-3) [2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-xmessage (1.0.5-2 -> 1.0.5-3) [2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-xmodmap (1.0.10-2 -> 1.0.10-3) [2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-xsetroot (1.1.2-2 -> 1.1.2-3) [2022-06-24T02:20:21-0500] [ALPM] upgraded xterm (372-1 -> 372-2) [2022-06-24T02:20:21-0500] [ALPM] upgraded zsh (5.8.1-2 -> 5.9-1) [2022-06-24T02:20:24-0500] [ALPM] transaction completed [2022-06-24T02:20:24-0500] [ALPM] running '30-systemd-update.hook'... [2022-06-24T02:20:24-0500] [ALPM] running 'fontconfig.hook'... [2022-06-24T02:20:25-0500] [ALPM] running 'gtk-update-icon-cache.hook'... [2022-06-24T02:20:25-0500] [ALPM] running 'update-desktop-database.hook'... [2022-06-24T02:20:26-0500] [ALPM] running 'xorg-mkfontscale.hook'... Now system has many hooks not run, at this point if upgrade the kernel to force an image to be created and it succeeds: [2022-06-24T02:24:31-0500] [PACMAN] Running 'pacman -U /var/cache/pacman/pkg/linux-5.18.6.arch1-1-x86_64.pkg.tar.zst' [2022-06-24T02:24:37-0500] [ALPM] transaction started [2022-06-24T02:24:38-0500] [ALPM] reinstalled linux (5.18.6.arch1-1) [2022-06-24T02:24:40-0500] [ALPM] transaction completed [2022-06-24T02:24:41-0500] [ALPM] running '30-systemd-update.hook'... [2022-06-24T02:24:41-0500] [ALPM] running '60-depmod.hook'... [2022-06-24T02:24:46-0500] [ALPM] running '90-mkinitcpio-install.hook'... [2022-06-24T02:25:05-0500] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-fall back.img [2022-06-24T02:25:06-0500] [ALPM-SCRIPTLET] ==> Image generation successful Reinstall systemd to force it to run its hooks and vim to see if I can't fix "can't allocate for color Orange", etc... [2022-06-24T02:46:41-0500] [PACMAN] Running 'pacman -S systemd systemd-libs vim vim-runtime' [2022-06-24T02:46:53-0500] [ALPM] transaction started [2022-06-24T02:46:53-0500] [ALPM] reinstalled systemd-libs (251.2-1) [2022-06-24T02:46:54-0500] [ALPM] reinstalled systemd (251.2-1) [2022-06-24T02:46:55-0500] [ALPM] reinstalled vim-runtime (8.2.5046-2) [2022-06-24T02:46:55-0500] [ALPM] reinstalled vim (8.2.5046-2) [2022-06-24T02:46:56-0500] [ALPM] transaction completed [2022-06-24T02:46:56-0500] [ALPM] running '20-systemd-sysusers.hook'... [2022-06-24T02:46:56-0500] [ALPM] running '30-systemd-catalog.hook'... [2022-06-24T02:46:56-0500] [ALPM] running '30-systemd-daemon-reload.hook'... [2022-06-24T02:46:56-0500] [ALPM] running '30-systemd-hwdb.hook'... [2022-06-24T02:46:57-0500] [ALPM] running '30-systemd-sysctl.hook'... [2022-06-24T02:46:57-0500] [ALPM] running '30-systemd-tmpfiles.hook'... [2022-06-24T02:46:57-0500] [ALPM] running '30-systemd-udev-reload.hook'... [2022-06-24T02:46:57-0500] [ALPM] running '30-systemd-update.hook'... [2022-06-24T02:46:57-0500] [ALPM] running '90-mkinitcpio-install.hook'... So after all of this manual running, I can start the VM and log in as user or root, but I'm not sure what got missed? Is there any way to know what hooks remain unrun or what packages the system thinks were installed but that were not actually installed? Why does X and the console only show 1024 x 768 of the guest even though the window (which is normally filled with the VM) is 1440 x 864? What do I need to look at/test to determine what shape my install is in? Sorry for the long post, but without the logs, it would have been impossible to explain what happened or where it happened in the upgrade process. Let me know if there is anything else I can post that will help. -- David C. Rankin, J.D.,P.E.