mirror of
https://review.haiku-os.org/haiku
synced 2024-11-23 07:18:40 +01:00
3rdparty/kallisti5: Update iommu guide for vfio
This commit is contained in:
Alexander von Gluck IV
parent
65ff9f27e1
commit
acf75a66d6
130
3rdparty/kallisti5/howto-iommu-old.md
vendored
Normal file
130
3rdparty/kallisti5/howto-iommu-old.md
vendored
Normal file
@ -0,0 +1,130 @@
|
||||
Haiku PCI Driver Development Under QEMU on Linux
|
||||
====================================================
|
||||
|
||||
> WARNING: This is an older guide under kernel 3.x using pci-stub
|
||||
> On newer kernels, you have to use vfio.
|
||||
|
||||
Developing Haiku drivers for PCI and PCI Express cards is now a lot easier
|
||||
given advancements in IOMMU under Linux. You can effectively detach PCI cards
|
||||
from their host operating system and attach them to guest VM's resulting in
|
||||
true hardware emulation. In this guide we will be configuring a secondary
|
||||
graphics card to be attached to a Haiku virtual machine.
|
||||
|
||||
**Warning**: Any device attached to a VM will be unavailable to the host operating
|
||||
system. This means you **cannot** use your primary graphics card, network device,
|
||||
etc within a VM and the host operating system at the same time. In this
|
||||
example, we have two graphics cards installed in the Linux system.
|
||||
|
||||
IOMMU Setup
|
||||
-----------------------
|
||||
You will need to have IOMMU hardware support on your motherboard for this
|
||||
to function. Most modern AMD A3 socket chips and Intel i3/i5/i7 devices
|
||||
have IOMMU built in. If your board does indeed have IOMMU, you will likely
|
||||
need to enable IOMMU within the bios of your motherboard before proceeding.
|
||||
|
||||
Linux Setup
|
||||
-----------------------
|
||||
Now that you have an IOMMU enabled system, you will need to tell Linux to
|
||||
fully utilize IOMMU. To do this, you will need to add a few kernel boot
|
||||
parameters. Depending on how your system is configured, there may be a few
|
||||
places to do this. Here are some example config files:
|
||||
|
||||
* `/etc/default/grub` (`GRUB_CMDLINE_LINUX_DEFAULT`)
|
||||
* `/boot/grub/grub.cfg`
|
||||
* `/boot/grub/menu.lst`
|
||||
* `/boot/refind_linux.conf`
|
||||
|
||||
Enabling OS support IOMMU for IOMMU involves adding one of the following
|
||||
kernel boot parameters:
|
||||
|
||||
**AMD:**
|
||||
```
|
||||
iommu=pt iommu=1
|
||||
```
|
||||
**Intel:**
|
||||
```
|
||||
intel_iommu=on
|
||||
```
|
||||
|
||||
Now, all we need to do is to reserve the PCI device. We want to make sure
|
||||
no host drivers attempt to attach to the PCI device in question.
|
||||
|
||||
First we need to find the PCIID for the device in question. We can find
|
||||
this through lcpci. Running lspci shows a bunch of devices. I've identified
|
||||
this device as my target:
|
||||
|
||||
```
|
||||
07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000]
|
||||
```
|
||||
|
||||
Now, to get the PCI ID, I run lspci again with the -n flag (lspci -n). We find
|
||||
the matching BUS ID and we get our PCI ID:
|
||||
|
||||
```
|
||||
07:00.0 0300: 1002:68f9
|
||||
```
|
||||
|
||||
Now that we have our target PCI ID (`1002:68f9`), we can bind this device to
|
||||
a special pci-stub driver.
|
||||
|
||||
We will create two files for this graphics card:
|
||||
|
||||
**`/lib/modprobe.d/pci-stub.conf`:**
|
||||
```
|
||||
options pci-stub ids=1002:68f9
|
||||
```
|
||||
|
||||
**`/lib/modprobe.d/drm.conf`:**
|
||||
```
|
||||
softdep drm pre: pci-stub
|
||||
```
|
||||
|
||||
The first line tells the pci-stub driver to bind to the device in question.
|
||||
The second line tells DRM (graphics driver stack) that it should make sure
|
||||
pci-stub loads before DRM (ensuring the device is stubbed and not loaded by
|
||||
DRM).
|
||||
|
||||
Now we reboot and cross our fingers.
|
||||
|
||||
On my AMD Linux system, we can see that IOMMU is active and functional:
|
||||
|
||||
```
|
||||
kallisti5@eris ~ $ dmesg | grep AMD-Vi
|
||||
[ 0.119400] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
|
||||
[ 0.119406] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
|
||||
[ 0.119409] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found
|
||||
[ 0.119412] AMD-Vi: Disabling interrupt remapping
|
||||
[ 1.823122] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
|
||||
[ 1.823253] AMD-Vi: Initialized for Passthrough Mode
|
||||
```
|
||||
|
||||
And checking for pci-stub we can see it successfully took over my graphics card:
|
||||
```
|
||||
kallisti5@eris ~ $ dmesg | grep pci-stub
|
||||
[ 3.685970] pci-stub: add 1002:68F9 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
|
||||
[ 3.686002] pci-stub 0000:07:00.0: claimed by stub
|
||||
```
|
||||
|
||||
On every boot, the device will be available for attachment to VM's
|
||||
Now, we simply attach the device to a VM:
|
||||
|
||||
```
|
||||
sudo qemu-system-x86_64 --enable-kvm -hda haiku-nightly-anyboot.image -m 2048 -device pci-assign,host=07:00.0
|
||||
```
|
||||
|
||||
If you experience any problems, try looking at kvm messages:
|
||||
```
|
||||
kallisti5@eris ~ $ dmesg | grep kvm
|
||||
```
|
||||
|
||||
If you're doing this for a graphics card generally the qemu window will
|
||||
lock up at the bootsplash and the video will appear on the second window.
|
||||
Click the qemu window to control the Haiku machine.
|
||||
|
||||
If things go well you will see:
|
||||
```
|
||||
[ 1966.132176] kvm: Nested Virtualization enabled
|
||||
[ 1966.132185] kvm: Nested Paging enabled
|
||||
[ 1972.212231] pci-stub 0000:07:00.0: kvm assign device
|
||||
[ 1974.186382] kvm: zapping shadow pages for mmio generation wraparound
|
||||
```
|
||||
119
3rdparty/kallisti5/howto-iommu.md
vendored
119
3rdparty/kallisti5/howto-iommu.md
vendored
@ -22,26 +22,7 @@ need to enable IOMMU within the bios of your motherboard before proceeding.
|
||||
Linux Setup
|
||||
-----------------------
|
||||
Now that you have an IOMMU enabled system, you will need to tell Linux to
|
||||
fully utilize IOMMU. To do this, you will need to add a few kernel boot
|
||||
parameters. Depending on how your system is configured, there may be a few
|
||||
places to do this. Here are some example config files:
|
||||
|
||||
* `/etc/default/grub` (`GRUB_CMDLINE_LINUX_DEFAULT`)
|
||||
* `/boot/grub/grub.cfg`
|
||||
* `/boot/grub/menu.lst`
|
||||
* `/boot/refind_linux.conf`
|
||||
|
||||
Enabling OS support IOMMU for IOMMU involves adding one of the following
|
||||
kernel boot parameters:
|
||||
|
||||
**AMD:**
|
||||
```
|
||||
iommu=pt iommu=1
|
||||
```
|
||||
**Intel:**
|
||||
```
|
||||
intel_iommu=on
|
||||
```
|
||||
fully utilize IOMMU and reserve the PCI cards for IOMMU use.
|
||||
|
||||
Now, all we need to do is to reserve the PCI device. We want to make sure
|
||||
no host drivers attempt to attach to the PCI device in question.
|
||||
@ -51,77 +32,91 @@ this through lcpci. Running lspci shows a bunch of devices. I've identified
|
||||
this device as my target:
|
||||
|
||||
```
|
||||
07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000]
|
||||
$ lspci -nn | egrep "VGA|Audio"
|
||||
28:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/580] [1002:67df] (rev c7)
|
||||
28:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
|
||||
|
||||
29:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Redwood XT GL [FirePro V4800] [1002:68c8]
|
||||
29:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] [1002:aa60]
|
||||
|
||||
2b:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]
|
||||
```
|
||||
|
||||
Now, to get the PCI ID, I run lspci again with the -n flag (lspci -n). We find
|
||||
the matching BUS ID and we get our PCI ID:
|
||||
Now that we have our target PCI IDs (in this case, `1002:68c8,1002:aa60`), we can bind this device to
|
||||
the vfio-pci driver.
|
||||
|
||||
**vfio-pci module**
|
||||
|
||||
If your distro ships vfio-pci as a module, you will need to add the vfio drivers to the initial ramdisk
|
||||
to leverage them as early as possible in the boot process.
|
||||
|
||||
Below is an example on Fedora:
|
||||
|
||||
```
|
||||
07:00.0 0300: 1002:68f9
|
||||
echo 'add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"' > /etc/dracut.conf.d/vfio.conf
|
||||
dracut -f --kver `uname -r`
|
||||
```
|
||||
|
||||
Now that we have our target PCI ID (`1002:68f9`), we can bind this device to
|
||||
a special pci-stub driver.
|
||||
**vfio reservation**
|
||||
|
||||
We will create two files for this graphics card:
|
||||
Now that the requirements are met, we can attach the target GPU to the vfio driver using the information
|
||||
we have collected so far.
|
||||
|
||||
Below, i've leveraged the PCI ID's collected above, and provided them to the vfio-pci driver via the kernel
|
||||
parameters at boot.
|
||||
|
||||
> Be sure to replace <CPU> with amd or intel depending on your system.
|
||||
|
||||
**`/lib/modprobe.d/pci-stub.conf`:**
|
||||
```
|
||||
options pci-stub ids=1002:68f9
|
||||
rd.driver.pre=vfio-pci vfio-pci.ids=1002:68c8,1002:aa60 <CPU>_iommu=on
|
||||
```
|
||||
|
||||
**`/lib/modprobe.d/drm.conf`:**
|
||||
```
|
||||
softdep drm pre: pci-stub
|
||||
```
|
||||
> YMMV: These steps differ a lot between distros
|
||||
|
||||
The first line tells the pci-stub driver to bind to the device in question.
|
||||
The second line tells DRM (graphics driver stack) that it should make sure
|
||||
pci-stub loads before DRM (ensuring the device is stubbed and not loaded by
|
||||
DRM).
|
||||
On my EFI Fedora 26 system, I appended the line above to my /etc/sysconfig/grub config under GRUB_CMDLINE_LINUX.
|
||||
Then, regenerated my grub.cfg via ```grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg```
|
||||
|
||||
|
||||
Attach GPU to VM
|
||||
-----------------
|
||||
|
||||
Now we reboot and cross our fingers.
|
||||
|
||||
> If displays attached to the card are now black, then things are working as designed. If you see your
|
||||
> desktop on the target GPU, the vfio driver didn't properly bind to your card and something was done
|
||||
> incorrectly.
|
||||
|
||||
On my AMD Linux system, we can see that IOMMU is active and functional:
|
||||
|
||||
```
|
||||
kallisti5@eris ~ $ dmesg | grep AMD-Vi
|
||||
[ 0.119400] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
|
||||
[ 0.119406] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
|
||||
[ 0.119409] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found
|
||||
[ 0.119412] AMD-Vi: Disabling interrupt remapping
|
||||
[ 1.823122] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
|
||||
[ 1.823253] AMD-Vi: Initialized for Passthrough Mode
|
||||
kallisti5@eris ~ $ dmesg | egrep "IOMMU|AMD-Vi"
|
||||
[ 0.650138] AMD-Vi: IOMMU performance counters supported
|
||||
[ 0.652201] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
|
||||
[ 0.652201] AMD-Vi: Extended features (0xf77ef22294ada):
|
||||
[ 0.652204] AMD-Vi: Interrupt remapping enabled
|
||||
[ 0.652204] AMD-Vi: virtual APIC enabled
|
||||
[ 0.652312] AMD-Vi: Lazy IO/TLB flushing enabled
|
||||
[ 0.653841] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
|
||||
[ 4.114847] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
|
||||
|
||||
```
|
||||
|
||||
And checking for pci-stub we can see it successfully took over my graphics card:
|
||||
And checking for vfio we can see it successfully took over my graphics card:
|
||||
```
|
||||
kallisti5@eris ~ $ dmesg | grep pci-stub
|
||||
[ 3.685970] pci-stub: add 1002:68F9 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
|
||||
[ 3.686002] pci-stub 0000:07:00.0: claimed by stub
|
||||
kallisti5@eris ~ $ dmesg | grep vfio
|
||||
[ 3.928695] vfio-pci 0000:29:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
|
||||
[ 3.940222] vfio_pci: add [1002:68c8[ffff:ffff]] class 0x000000/00000000
|
||||
[ 3.952302] vfio_pci: add [1002:aa60[ffff:ffff]] class 0x000000/00000000
|
||||
[ 35.629861] vfio-pci 0000:29:00.0: enabling device (0000 -> 0003)
|
||||
```
|
||||
|
||||
On every boot, the device will be available for attachment to VM's
|
||||
Now, we simply attach the device to a VM:
|
||||
|
||||
```
|
||||
sudo qemu-system-x86_64 --enable-kvm -hda haiku-nightly-anyboot.image -m 2048 -device pci-assign,host=07:00.0
|
||||
```
|
||||
|
||||
If you experience any problems, try looking at kvm messages:
|
||||
```
|
||||
kallisti5@eris ~ $ dmesg | grep kvm
|
||||
sudo qemu-system-x86_64 --enable-kvm -hda haiku-nightly-anyboot.image -m 2048 -device pci-assign,host=29:00.0
|
||||
```
|
||||
|
||||
If you're doing this for a graphics card generally the qemu window will
|
||||
lock up at the bootsplash and the video will appear on the second window.
|
||||
Click the qemu window to control the Haiku machine.
|
||||
|
||||
If things go well you will see:
|
||||
```
|
||||
[ 1966.132176] kvm: Nested Virtualization enabled
|
||||
[ 1966.132185] kvm: Nested Paging enabled
|
||||
[ 1972.212231] pci-stub 0000:07:00.0: kvm assign device
|
||||
[ 1974.186382] kvm: zapping shadow pages for mmio generation wraparound
|
||||
```
|
||||
|
||||
Loading…
Reference in New Issue
Block a user