[arch-general] Xen GPU Passthrough

[Jay Seth]

Hello, I need help & I would appreciate any help to get my setup working.

When I try to start a domU on Arch/Xen, I am getting the following error:

xl create w7.cfg

Parsing config from w7.cfg
libxl: error: libxl_pci.c:1235:libxl__device_pci_add: PCI device 0:1:0.0 is
not assignable
libxl: error: libxl_create.c:1566:domcreate_attach_pci:
libxl_device_pci_add failed: -3
libxl: error: libxl.c:1583:libxl__destroy_domid: non-existant domain 2
libxl: error: libxl.c:1542:domain_destroy_callback: unable to destroy guest
with domid 2
libxl: error: libxl.c:1471:domain_destroy_cb: destruction of domain 2 failed

Checking the GPU with lspci, I get:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108M
[GeForce GT 525M] [10de:0df5] (rev a1)
        Subsystem: Dell Device [1028:0446]
        Kernel driver in use: nouveau
        Kernel modules: nouveau


Background...

I am relatively new to Arch and Xen (coming from Fedora and KVM) but I
decided to see if Xen on Arch would support GPU passthrough for my Dell XPS
15z. It has Nvidia GeForce GT 525M. On Fedora/KVM, GPU passthrough works
but there is the nasty error code 43. So that proved that the GPU can be
passed through. I am trying Xen to see if the HVM will hide whatever it is
that Nvidia uses to identify a VM platform and then disable drivers, giving
error code 43. I have successfully installed xen and created a test VM. The
config is as follows:

name = 'w7'
builder = 'hvm'
vcpus = '2'
memory = '2048'
boot = 'd'
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'destroy'
device_model_version = 'qemu-xen-traditional'
disk = [ '/home/tim/Downloads/w7.iso,,hdc,cdrom',
'/home/tim/vms/w7/w7.img,raw,hda,rw' ]
vif = ['mac=00:16:3e:54:86:18,bridge=xenbr0']
acpi = '1'
apic = '1'
serial = 'pty'
localtime = 1

# Keyboard and mouse passthrough
usb = '1'
usbdevice = ['tablet']

# VGA and sound passthrough
viridian = '1'
xen_platform_pci = '1'
gfx_passthru = '1'
pci_permissive = '1'
pci = ['01:00.0,rdm_policy=relaxed']

# Remote vnc access
vnc = '1'
# vnclisten = 'x.x.x.x'

The BDF of the GPU is 01:00.0 and it is what I have set for the pci
parameter.

I have also set intel_iommu=on in my /etc/default/grub file. I am booting
the Xen kernel. I have also set the vfio-pci id for the GPU in
/etc/modprobe.d/vfio.conf as:
options vfio--pci ids=10de:0df5

After that, I set the vfio modules to load first in /etc/mkinitcpio.conf as
follows:
MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd i915 nouveau"
HOOKS="base udev autodetect modconf block filesystems keyboard fsck"

After that, I did:
mkinitcpio -p linux
grub-mkconfig -o /boot/grb/grub.cfg
reboot

After reboot, I ran the following checks:

dmesg | grep vfio

[    3.856580] vfio-pci: probe of 0000:01:00.0 failed with error -22
[    3.856584] vfio_pci: add [10de:0df5[ffff:ffff]] class 0x000000/00000000
[   18.960859]  xen_pciback xen_netback xenfs xen_privcmd xen_blkback
xen_gntalloc xen_gntdev xen_evtchn ip_tables x_tables ext4 crc16 jbd2
fscrypto mbcache sd_mod serio_raw atkbd libps2 ahci libahci xhci_pci
xhci_hcd libata ehci_pci ehci_hcd scsi_mod usbcore usb_common i8042 serio
nouveau mxm_wmi wmi ttm i915 video button intel_gtt i2c_algo_bit
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm vfio_pci
irqbypass vfio_virqfd vfio_iommu_type1 vfio


dmesg | egrep -i iommu

[    0.000000] Command line: /boot/vmlinuz-linux
root=UUID=d8ee1f55-14c6-4401-bc22-24b78eaf8863 rw intel_iommu=on quiet
console=tty0
[    0.000000] Kernel command line: /boot/vmlinuz-linux
root=UUID=d8ee1f55-14c6-4401-bc22-24b78eaf8863 rw intel_iommu=on quiet
console=tty0
[    0.000000] DMAR: IOMMU enabled
[   18.960859]  xen_pciback xen_netback xenfs xen_privcmd xen_blkback
xen_gntalloc xen_gntdev xen_evtchn ip_tables x_tables ext4 crc16 jbd2
fscrypto mbcache sd_mod serio_raw atkbd libps2 ahci libahci xhci_pci
xhci_hcd libata ehci_pci ehci_hcd scsi_mod usbcore usb_common i8042 serio
nouveau mxm_wmi wmi ttm i915 video button intel_gtt i2c_algo_bit
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm vfio_pci
irqbypass vfio_virqfd vfio_iommu_type1 vfio

Is it possible that the root of the problem might the vfio-pci error -22 at
boot?

Thanks