[arch-general] pacman Segfault on full system update

[David C. Rankin]

I have an Arch guest in VirtualBox running on an Arch server. I installed 
virtualbox-bin 1-34-2 from AUR with 5.18 patch and the install on the host 
went fine.

I access the guest headless using rdesktop over the LAN. (have done so for 
years). Doing a full system update from 5.17.7 to 5.18.6 (about 25 days of 
updates) on guest.

Pacman segfaulted leaving post transaction hooks unrun. I reinstalled the 
kernel and systemd to force the initrd to be made so the guest is bootable.

However, I'm not sure what additional hooks were missed. The guest window when 
X is started is 1440 x 864 but the only portion of the desktop shown is 1024 x 
768 (rest is black and not shown, taskbar unable to be seen, etc.) After the 
pacman segfault, now the console is also missing the last 7-8 lines. I have to 
hit clear and start over at the top to see what I'm typing.

How can I recover this system? I was able to rsync the last 1000 lines of 
pacman.log from the guest to the host, but the shared folders no longer work. 
The pacman.log summary of how the issue occurred is:

[2022-06-24T02:06:26-0500] [PACMAN] Running 'pacman -Sy --needed 
archlinux-keyring'
[2022-06-24T02:06:26-0500] [PACMAN] synchronizing package lists
[2022-06-24T02:07:03-0500] [PACMAN] Running 'pacman -Su'
[2022-06-24T02:07:03-0500] [PACMAN] starting full system upgrade
[2022-06-24T02:07:30-0500] [ALPM] running '60-mkinitcpio-remove.hook'...
[2022-06-24T02:07:30-0500] [ALPM] transaction started
[2022-06-24T02:07:30-0500] [ALPM] upgraded iana-etc (20220427-1 -> 20220603-1)
[2022-06-24T02:07:30-0500] [ALPM] warning: /etc/locale.gen installed as 
/etc/locale.gen.pacnew
[2022-06-24T02:07:31-0500] [ALPM] upgraded glibc (2.35-5 -> 2.35-6)
[2022-06-24T02:07:31-0500] [ALPM-SCRIPTLET] Generating locales...
[2022-06-24T02:07:32-0500] [ALPM-SCRIPTLET]   en_US.UTF-8... done
[2022-06-24T02:07:32-0500] [ALPM-SCRIPTLET] Generation complete.
<snipped lots of packages installed>
...
[2022-06-24T02:09:04-0500] [ALPM] upgraded linux-firmware-whence 
(20220509.b19cbdc-1 -> 20220610.7b71b75-1)
Segfault

Remove db.lock and continue:

[2022-06-24T02:12:57-0500] [PACMAN] Running 'pacman -Syu'
[2022-06-24T02:12:57-0500] [PACMAN] synchronizing package lists
[2022-06-24T02:12:59-0500] [PACMAN] starting full system upgrade
[2022-06-24T02:13:16-0500] [ALPM] transaction started
[2022-06-24T02:13:17-0500] [ALPM] error: could not extract 
/usr/share/licenses/linux-firmware/LICENCE.nvidia (Zstd decompression failed: 
Restored data doesn't match checksum)
[2022-06-24T02:13:17-0500] [ALPM] error: problem occurred while upgrading 
linux-firmware
[2022-06-24T02:13:17-0500] [ALPM] upgraded linux-firmware (20220509.b19cbdc-1 
-> 20220610.7b71b75-1)
[2022-06-24T02:13:17-0500] [ALPM] transaction failed

So I delete the firmware file from /var/cache/pacman/pkg and run the upgrade 
again. To my surprise, it doesn't download the firmware package again, it just 
considers it installed and proceeds to other packages, e.g.

[2022-06-24T02:15:49-0500] [PACMAN] Running 'pacman -Syu'
[2022-06-24T02:15:49-0500] [PACMAN] synchronizing package lists
[2022-06-24T02:15:51-0500] [PACMAN] starting full system upgrade
[2022-06-24T02:16:03-0500] [ALPM] transaction started
[2022-06-24T02:16:03-0500] [ALPM] upgraded python (3.10.4-1 -> 3.10.5-1)
[2022-06-24T02:16:09-0500] [ALPM] upgraded linux-headers (5.17.7.arch1-1 -> 
5.18.6.arch1-1)
[2022-06-24T02:16:11-0500] [ALPM] error: could not extract 
/usr/lib/mysql/plugin/client_ed25519.so (Zstd decompression failed: Restored 
data doesn't match checksum)
[2022-06-24T02:16:11-0500] [ALPM] error: problem occurred while upgrading 
mariadb-libs
[2022-06-24T02:16:11-0500] [ALPM] upgraded mariadb-libs (10.7.3-1 -> 10.8.3-1)
[2022-06-24T02:16:11-0500] [ALPM] transaction failed

Crash again and delete mariadb-libs from /var/cache... and try again:

[2022-06-24T02:19:16-0500] [PACMAN] Running 'pacman -Syu'
[2022-06-24T02:19:16-0500] [PACMAN] synchronizing package lists
[2022-06-24T02:19:17-0500] [PACMAN] starting full system upgrade
[2022-06-24T02:19:26-0500] [ALPM] transaction started
[2022-06-24T02:19:26-0500] [ALPM] upgraded mariadb-clients (10.7.3-1 -> 10.8.3-1)
[2022-06-24T02:19:27-0500] [ALPM] warning: directory permissions differ on 
/usr/lib/mysql/plugin/auth_pam_tool_dir/
filesystem: 700  package: 755
[2022-06-24T02:19:27-0500] [ALPM] upgraded mariadb (10.7.3-1 -> 10.8.3-1)
[2022-06-24T02:19:27-0500] [ALPM-SCRIPTLET] :: MariaDB was updated to a new 
feature release. To update the data run:
[2022-06-24T02:19:27-0500] [ALPM-SCRIPTLET]    systemctl restart 
mariadb.service && mariadb-upgrade -u root -p
[2022-06-24T02:19:27-0500] [ALPM] warning: /etc/pacman.d/mirrorlist installed 
as /etc/pacman.d/mirrorlist.pacnew
[2022-06-24T02:19:27-0500] [ALPM] upgraded pacman-mirrorlist (20220501-1 -> 
20220605-1)
<snip lots more packages installed>
...
[2022-06-24T02:19:34-0500] [ALPM] upgraded tmux (3.2_a-1 -> 3.3_a-2)
[2022-06-24T02:19:35-0500] [ALPM] error: could not extract 
/usr/share/doc/valgrind/valgrind_manual.ps (Zstd decompression failed: 
Restored data doesn't match checksum)
[2022-06-24T02:19:35-0500] [ALPM] error: problem occurred while upgrading valgrind
[2022-06-24T02:19:35-0500] [ALPM] upgraded valgrind (3.19.0-3 -> 3.19.0-4)
[2022-06-24T02:19:35-0500] [ALPM] transaction failed

Crash again. Rinse, repeat and try again:

[2022-06-24T02:20:14-0500] [PACMAN] Running 'pacman -Syu'
[2022-06-24T02:20:14-0500] [PACMAN] synchronizing package lists
[2022-06-24T02:20:15-0500] [PACMAN] starting full system upgrade
[2022-06-24T02:20:20-0500] [ALPM] transaction started
[2022-06-24T02:20:21-0500] [ALPM] upgraded vim-runtime (8.2.4827-1 -> 8.2.5046-2)
[2022-06-24T02:20:21-0500] [ALPM] upgraded vim (8.2.4827-1 -> 8.2.5046-2)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xbitmaps (1.1.2-2 -> 1.1.2-3)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xcb-util-cursor (0.1.3-3 -> 0.1.3-4)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xcursor-themes (1.0.6-2 -> 1.0.6-3)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-bdftopcf (1.1-2 -> 1.1-3)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-font-util (1.3.2-2 -> 1.3.2-3)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-server-common (21.1.3-6 -> 
21.1.3-7)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-server (21.1.3-6 -> 21.1.3-7)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-xcursorgen (1.0.7-2 -> 1.0.7-3)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-xmessage (1.0.5-2 -> 1.0.5-3)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-xmodmap (1.0.10-2 -> 1.0.10-3)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xorg-xsetroot (1.1.2-2 -> 1.1.2-3)
[2022-06-24T02:20:21-0500] [ALPM] upgraded xterm (372-1 -> 372-2)
[2022-06-24T02:20:21-0500] [ALPM] upgraded zsh (5.8.1-2 -> 5.9-1)
[2022-06-24T02:20:24-0500] [ALPM] transaction completed
[2022-06-24T02:20:24-0500] [ALPM] running '30-systemd-update.hook'...
[2022-06-24T02:20:24-0500] [ALPM] running 'fontconfig.hook'...
[2022-06-24T02:20:25-0500] [ALPM] running 'gtk-update-icon-cache.hook'...
[2022-06-24T02:20:25-0500] [ALPM] running 'update-desktop-database.hook'...
[2022-06-24T02:20:26-0500] [ALPM] running 'xorg-mkfontscale.hook'...

Now system has many hooks not run, at this point if upgrade the kernel to 
force an image to be created and it succeeds:

[2022-06-24T02:24:31-0500] [PACMAN] Running 'pacman -U 
/var/cache/pacman/pkg/linux-5.18.6.arch1-1-x86_64.pkg.tar.zst'
[2022-06-24T02:24:37-0500] [ALPM] transaction started
[2022-06-24T02:24:38-0500] [ALPM] reinstalled linux (5.18.6.arch1-1)
[2022-06-24T02:24:40-0500] [ALPM] transaction completed
[2022-06-24T02:24:41-0500] [ALPM] running '30-systemd-update.hook'...
[2022-06-24T02:24:41-0500] [ALPM] running '60-depmod.hook'...
[2022-06-24T02:24:46-0500] [ALPM] running '90-mkinitcpio-install.hook'...
<snip -- succeeds without issue>
[2022-06-24T02:25:05-0500] [ALPM-SCRIPTLET] ==> Creating zstd-compressed 
initcpio image: /boot/initramfs-linux-fall
back.img
[2022-06-24T02:25:06-0500] [ALPM-SCRIPTLET] ==> Image generation successful

Reinstall systemd to force it to run its hooks and vim to see if I can't fix 
"can't allocate for color Orange", etc...

[2022-06-24T02:46:41-0500] [PACMAN] Running 'pacman -S systemd systemd-libs 
vim vim-runtime'
[2022-06-24T02:46:53-0500] [ALPM] transaction started
[2022-06-24T02:46:53-0500] [ALPM] reinstalled systemd-libs (251.2-1)
[2022-06-24T02:46:54-0500] [ALPM] reinstalled systemd (251.2-1)
[2022-06-24T02:46:55-0500] [ALPM] reinstalled vim-runtime (8.2.5046-2)
[2022-06-24T02:46:55-0500] [ALPM] reinstalled vim (8.2.5046-2)
[2022-06-24T02:46:56-0500] [ALPM] transaction completed
[2022-06-24T02:46:56-0500] [ALPM] running '20-systemd-sysusers.hook'...
[2022-06-24T02:46:56-0500] [ALPM] running '30-systemd-catalog.hook'...
[2022-06-24T02:46:56-0500] [ALPM] running '30-systemd-daemon-reload.hook'...
[2022-06-24T02:46:56-0500] [ALPM] running '30-systemd-hwdb.hook'...
[2022-06-24T02:46:57-0500] [ALPM] running '30-systemd-sysctl.hook'...
[2022-06-24T02:46:57-0500] [ALPM] running '30-systemd-tmpfiles.hook'...
[2022-06-24T02:46:57-0500] [ALPM] running '30-systemd-udev-reload.hook'...
[2022-06-24T02:46:57-0500] [ALPM] running '30-systemd-update.hook'...
[2022-06-24T02:46:57-0500] [ALPM] running '90-mkinitcpio-install.hook'...

So after all of this manual running, I can start the VM and log in as user or 
root, but I'm not sure what got missed? Is there any way to know what hooks 
remain unrun or what packages the system thinks were installed but that were 
not actually installed? Why does X and the console only show 1024 x 768 of the 
guest even though the window (which is normally filled with the VM) is 1440 x 864?

What do I need to look at/test to determine what shape my install is in?

Sorry for the long post, but without the logs, it would have been impossible 
to explain what happened or where it happened in the upgrade process. Let me 
know if there is anything else I can post that will help.


-- 
David C. Rankin, J.D.,P.E.