[arch-dev-public] Xen domU kernel - what to do?

Allan McRae allan at archlinux.org
Fri Oct 30 21:03:19 EDT 2009

Dan McGee wrote:
> On Fri, Oct 30, 2009 at 7:22 PM, Aaron Griffin <aaronmgriffin at gmail.com> wrote:
>> On Fri, Oct 30, 2009 at 6:51 PM, Dan McGee <dpmcgee at gmail.com> wrote:
>>> On Thu, Oct 29, 2009 at 7:59 PM, Thomas Bächler <thomas at archlinux.org> wrote:
>>>> Okay, it is obvious that we need a kernel which can run on Xen domU for our
>>>> servers. It would also be very convenient for those running Arch on hosting
>>>> services that have pv-grub available.
>>>> Until now, I compiled a modified version of the kernel26 package with
>>>> pv_ops/Xern enabled which - in theory - should also run on a real machine.
>>>> If all the external modules compile properly (which they probably do), we
>>>> could add these options to our normal stock kernel. I haven't tried running
>>>> it on a real machine though and tpowa said in the past the nvidia module
>>>> failed. This has to be tested.
>>>> However, compiling Xen on a 32 bit kernel requires enabling PAE which we
>>>> refused to do in the past for performance reasons (no idea if these are
>>>> valid concerns) and due to the fact that we want to encourage people to run
>>>> 64 bit if they have large amounts of memory.
>>>> Assuming all of untested things above work, what would you prefer:
>>>> 1) Switch to PAE on the 32 bit kernel, ship Xen domU support with kernel26
>>>> on both architectures.
>>>> 2) Ship Xen domU support on the 64 bit kernel, and don't on the 32 bit
>>>> kernel (my favorite so far).
>>>> 3) Leave kernel26 unchanged and create a separate kernel26-xen package with
>>>> domU support.
>>>> The advantage of 3) is that the Xen kernel could be very slim, as almost all
>>>> hardware drivers could be disabled. The advantage of 1) and 2) is the lower
>>>> maintenance cost.
>>>> On a different note, I'd like to try to switch over to pv-grub on our
>>>> servers. Right now, upgrading the kernel requires copying the kernel to the
>>>> dom0 manually, thus dom0 access is required to do a (potentially security
>>>> relevant) kernel update. With pv-grub, upgrading the kernel and rebooting
>>>> would be handled just like with a normal machine, as the dom0 only loads the
>>>> pv-grub image which then runs inside the domU and can load the right kernel
>>>> from there.
>>>> The pv-grub is only available from Xen 3.3 and later, but we have Xen 3.2 -
>>>> I compiled pv-grub from 3.4 though and guess it should work with the older
>>>> hypervisor as well, but I haven't tested that yet.
>>>> Anyway, please post your opinion on the issues above.
>>> I'm -1 on the PAE, leaving options 2 and 3 as viable. 2 is the easiest
>>> for maintenance; 3 sounds like the best solution long term as it would
>>> be a very lightweight and well-suited kernel package.
>>> If not for the maintenance issues, 3 seems by far the best. Is there
>>> anyway we can automate the creation of this package to the point where
>>> it doesn't take much extra effort? e.g. try to share resources with
>>> the primary kernel package.
>> Can you illustrate why you're -1 on PAE? I can't find exact info on
>> the performance hit caused by enabling that, and with machines having
>> more and more RAM, I'm sure we'll eventually need to enable PAE. I
>> _know_ we're trying to push people to 64bit, but that shouldn't mean
>> we leave the i686 arch in the dust.
>> I'm just sure we're EVENTUALLY going to need to enable this. Why not now?
> Completely disagree, PAE is stupid. If you want to stick with 32 bit
> and have more memory available, at least run a 64 bit kernel and stick
> with a 32 bit userspace:
> http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/01128.html
> http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/01131.html

I do that, and the are some usability issues.  Nothing insurmountable.  
Like needing to use "linux32" before ./configure's, makepkg's, etc. 
Nothing a graphical user would notice, but would be more noticeable in 
Arch where building from the AUR is common. 

 From memory, installing a 64 bit kernel when appropriate was a release 
goal for Fedora 11 but got dropped.  I do not know why, but I guess the 
small annoyances I find are probably the reason.

> And those numbers for performance problems? 12%:
> http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/00959.html
> Leaving i686 in the dust is hardly the concern here; why add
> unnecessary overhead to those machines that sit well under the 4GB
> mark that need that extra 12% of performance the most?

Red Hat did a more comprehensive survey of PAE (a few years ago now).  
They found about the same thing: ~0% for some operations, 10% for the 
worst (which reflects the test case Dan linked).  It averaged out to 
~1%.  See http://people.redhat.com/nmurray/RHEL-2.1-VM-whitepaper.pdf 
(top page 10).


More information about the arch-dev-public mailing list