Saturday, January 25, 2020

PCI Passthrough without IOMMU, AKA Gaming on Qubes 4.0.3

Step Zero: Know that Qubes is designed for security, not for fun times, not for ease of use, not for a pretty UI. Qubes is a fantastic and unique OS that everyone should give an honest try to, but it's also kind of ugly and a pain to use. Before reading my words any further, you should read up on the "correct" way to use and think about Qubes. If this doesn't scare you off, then read how the author of Qubes uses it as a daily driver. The flow charts help a lot. It'll be much easier to use Qubes if you're in the right mindset. For example: when you boot up Qubes and log in, you're sitting at a desktop without any network access and you should never install any programs on it. This sounds crazy until you read more about how everything is virtualized. Your email? Three different VMs: a different VM for each email account (optionally). It starts to feel a lot smoother and more natural after you get into that mindset. Your tasks aren't divided into programs, they're divided into... social circles, shall we say? You communicate differently with your boss, your parents, your friends, and your spouse. You'd share different information with each of those groups of people, and even different friends you'd treat differently. Some of my closest friends are terrible with money, and I'd sooner extend a loan to my boss than some friends. It's the same with your software. I trust Steam more than Origin, and neither of them enough to have full, unfettered access to my documents and music. I don't even trust my music player to have write access to my music, ever since iTunes destroyed all of my ID3 tags decades ago. Normally we do everything on our computer in one context. As I type this I have 135 tabs open in one browser. Some of them are personal, some are professional, some are ephemeral, some sites are solid and statically generated, and some sites are slightly suspect because they're written in PHP or Cold Fusion--basically a neon sign reading 'easy to hack'. As usual, XKCD puts it best in #1200 pointing out how our whole lives are in one box. You could carry around a work phone, a personal phone, a work computer, a personal computer, an even more personal computer that's never connected to the Internet because it contains all of your passwords and pictures of your drivers license and Social Security card because even knowing those numbers makes your life so much easier if you were to ever lose the cards themselves. You could segregate your life onto multiple computers, or you could have one computer containing a lot of tiny fake computers. That's Qubes.

Step One: Mandatory: you must have at least two monitors, each one plugged into a different GPU. Or the same monitor with two inputs, each plugged into a different GPU, but that'll make the initial setup more difficult. There comes a point later on when your Windows qube will appear to hang or crash, but actually it's sending everything to your other monitor while leaving its last frame on its qube window/frame buffer thing. If you only have one GPU then plug the other into your motherboard and let your CPU run that one. One of these GPUs and all of its monitors will be passed through to a guest OS. A PS/2 Mouse and Keyboard and a separate USB mouse and keyboard is heavily recommended, but not mandatory. Your life will be better with a set of PS/2 and USB inputs (two keyboards and two mice) but you can live without them.

Step Two: Install Qubes 4.0.3 or later on a large disk. SSD/NVMe is recommended, but more important is size. There's probably a way to use multiple hard drives with Qubes, but I haven't gone looking and the GUI doesn't offer any help here. A fast drive will make everything better, but you'll want to prioritize size here; 200GB will be small for a daily driver (maybe too small) and 100GB will be a painful squeeze for even a brief test machine.

Step Three: After installation you reboot, and you're greeted with some configuration checkboxes. Even on an SSD this part takes forever. No choices here will mean life or death, it's 100% preference. I leave the defaults and also check the box for a separate USB qube, but I don't combine Net and USB into the same qube. I like the idea of every USB device being plugged in being immediately isolated from the main system by default. It's trivial to turn this USB isolation off after the fact.

Step Four: You're finally at a desktop. It's ugly. First up is to hit the "Applications Menu" at the top right (or right-click the desktop) go to "System Tools" and launch "Qubes Updater" which takes forever to run the first time. Check all the boxes and run those updates. If you've opted to have a Whonix qube, then you should now go back into the Applications Menu (or right-click the desktop) and hover over any qube containing the name "whonix" and run a program in there. Any program might work, but I use WhonixCheck. The point is to be asked how you want to connect, and make sure it's working. When the Whonix VM is getting updated in that Qubes updater program you're currently running, it'll need to know how to connect to the Internet, and that's what you're doing now. Once your own tests have succeeded, you can close that knowing the Qubes updater will succeed when it eventually makes its slow way down to the two Whonix VMs.

Step Five: Make XFCE less ugly. Beyond the ascetic advantages, there's also a security benefit to this. Applications > System Tools > Appearance will keep your eyes from bleeding, but Applications > System Tools > Window Manager will help you use your system more safely and more easily. Change the window decorations to something that lets you see a lot of blue. Don't browse around too long, just find something that shows off the blue in the window decoration. Now open something from the qube named "personal" or "disposable DVM". Notice how the lock icons in the Applications menu are colored Yellow and Red respectively. Notice how your Window Manager settings window is blue. If you've followed my advice and seen Joanna's flow charts, these colors will already make sense to you. Now, with a "personal" and "Disposable DVM" program each open, continue deciding which window decoration is prettiest while also immediately communicating which context you're in. Oh, also make sure the GPU and monitor you're wanting to pass through isn't currently being used to display your desktop. Applications > System Tools > Displays will let you easily disable that screen. I haven't tried blacklisting the GPU from the kernel, so know that isn't necessary... I think.

Step Six: When the Qubes Update program has completed successfully, reboot. Now the real work of PCI passthrough and gaming begins. Log back into Qubes and open up Applications > System Tools > Qube Manager. You'll be using this program a lot.

Step Seven: Click the top-left plus icon: Create New Qube

Step Eight: Gaming on Linux works great, it's easy to set up, easy to use, and as such is boring and doesn't need this step-by-step guide. I recommend basing your Gaming-on-Linux qubes on Debian because Fedora comes and goes so fast, the template that came with Qubes is probably already past its End Of Life support. This is Qubes' fault, not Fedora's, but it's still your problem to deal with. If you want to install Steam in Debian on Qubes, then you should also run these two lines in your gaming qube to install some needed 32-bit libraries.
sudo dpkg --add-architecture i386
sudo apt install libgl1-mesa-glx:i386
Fun fact about using templates in Qubes: userdata is persistent, but any system-level stuff (anything that required sudo, like installing Steam) is gone as soon as the VM is shut down. If you want system stuff to persist, you'll have to do it to the template. Applications > Template: debian-10 > Terminal will give you access to that template. Any changes you make in there will stick around forever and be available to all qubes based on that template. For this reason, when making your Linux Gaming qube, you should select Type: "Standalone qube copied from a template". I know what you're thinking: why not install Manjaro or whatever myself and forget the template? The reason is the tight and fluid integration already set up and working with these Debian and Fedora templates. Notice how you choose a random program from a qube that isn't running, it then starts that qube, shows you just that program, and you could be convinced it's all happening locally on your main OS? That kind of automagical X11 forwarding is already set up and working, and anyway there's nothing wrong with Debain 10--that thing will be supported until mid 2024.

Step Nine: Gaming on Windows on Linux, on the other hand, will consume the rest of this tutorial. If you couldn't tell by the Fedora version, Qubes is a bit stuck in the past. Qubes prefers to work with Windows 7, but in theory all of this will work with any future version--Windows 10 certainly installs and runs fine, and I got most of the PCI passthrough working without a hitch. If you want Windows 7 but don't want to be on the unpatched side of the big bad Internet, then this is where Qubes shines: you can set very specific firewall rules for ingress (locked down by default) and egress (wide open by default) network traffic, and have those rules be unique for each qube even if they're all sharing the same NIC. You just want to play games on Windows 7, right? Install, configure, and update everything as far as you can, then lock down all traffic coming in or out of the box except for steampowered.com or install all the games you want and then cut off all networking. Whatever your plans here, they start with clicking that Create New Qube button in the Qube Manager program, and selecting Type: "Empty standalone qube" and checking the box "launch settings after creation"

Step Ten: First you'll want enough disk space for the "System storage max size". This number can be increased any time the qube is off, but can never be decreased ever. Be careful with your typing because it saves automatically, you only get one try at this.

Step Eleven: That Rubicon crossed, click over to the "Advanced" tab and give yourself some Initial memory to work with. Don't "Include in memory balancing". Your gaming machine won't "Provide network" either, so make sure both of those boxes are unchecked. Virtualization Mode needs to be "HVM" for PCI passthrough, so set that.

Step Twelve: Firewall rules can be ignored for now, so click over to the "Devices" tab. Here's where you find out which of your GPUs are which. Call me superstitious, but I don't think you should cut your GPU in half: give both the "VGA controller" and the "Audio device" halves of your GPU to the qube.

Step Thirteen: Applications and servers can be written off as "not an option" for Windows qubes, which isn't actually correct but close enough. Click "Apply" and then click back to the Advanced tab and select "Boot qube from CDROM". Leave that tiny window open while you think about where your ISOs are stored. Mine are on a NAS. If yours are too then now's a time admire some of the polish put into Qubes. In one of your Fedora or Debian based qubes, install cifs-utils or whatever package for your given protocol, and connect any one of your qubes to your ISOs share. Once you've done that, go back to the tiny "Boot qube from device" window, and select the bottom option "from file in qube" and browse for your iso. If that file picker window is iris-scaldingly bright, then open a terminal in that file picker's qube and run
[ -f ~/.config/gtk-3.0/settings.ini ] && printf "gtk-application-prefer-dark-theme=true" >> ~/.config/gtk-3.0/settings.ini || mkdir -p ~/.config/gtk-3.0/ && printf "[Settings]\ngtk-application-prefer-dark-theme=true" > ~/.config/gtk-3.0/settings.ini
That long command will check to see if you've already been configuring those settings; if so it'll add a request for dark themes to the end of the config file, if not it'll create the config folder and file with a request for dark themes. Remember that user settings are preserved in and unique to each qube. System level changes (like stuff requiring sudo) are only preserved when made in the template qube, and are then available to all dependent qubes.

Step Fourteen: After the normal OS installation, go about configuring and updating everything. You'll need to watch the Qube Manager, because every reboot only gets half way and ends up shut down, so you'll have to right-click your Windows qube each time and select "Start/Resume qube". Several hours and reboots later you should have everything updated and drivers for hard drives and PCI devices installed and functional. By now you'll have noticed your passed through GPU and its attached monitor are working for your guest qube, even without IOMMU support. That was pretty easy. You can even mouse around the guest qube's monitor from the tiny window displayed on your host OS; neat.

Step Fifteen: The final piece of the puzzle is giving your guest OS a dedicated USB mouse and keyboard, and only having a PS/2 mouse and keyboard for your host OS. You can use two USB-to-PS/2 adapters if you don't have any dusty old ball mice around. If you don't go for this 'two mice and two keyboard' recommendation of mine, you'll keep accidentally sliding your mouse out of the little window on your host OS and lose all of your input into the guest Qube and therefore lose your game you loser. To dedicate a mouse and keyboard that will be locked inside your guest qube, first make sure you don't currently have a "sys-usb" qube running, hoovering up all of your USB devices. If you do then shut that qube down (immediately losing all of your USB devices) and right click it in the Qube Manager, select "Qube Settings" and click the "Devices" tab and remove all of its USB controllers. In theory you could give some controllers to this sys-usb qube and some to your Windows qube, but in practice I've found that difficult but I also didn't try too hard. Click "OK" and go back to your Windows qube. Give it some or all of your USB controllers and start it up. Does it fail and mention "libxl"? Here's the secret sauce: "Configure strict reset for PCI devices". In the "Devices" tab of your Windows qube's settings is a button with that label. Click that long button and hold down the Control key to select all of your USB controllers. Now it should boot up with no issue, and have those USB devices working fine.

Step Sixteen: Maybe you want to limit your new qube to only talk to Steam? Until I figure out how to get it communicating with its host, the Windows qube can't accept any changes while it's running, they require the qube be shut down and then started back up before they're applied. Because of this, you should install Steam in a Linux qube for rapid testing of these firewall rules, because I still haven't cracked this nut. Hope you have better luck than me with this info:

https://support.steampowered.com/kb_article.php?ref=8571-GLVN-8711
http://whois.arin.net/rest/org/VC-2/nets

162.254.192.0/21
192.69.96.0/22
205.196.6.0/24
208.64.200.0/22
208.78.164.0/22