[arch-dev-public] Xen domU kernel - what to do?
Okay, it is obvious that we need a kernel which can run on Xen domU for our servers. It would also be very convenient for those running Arch on hosting services that have pv-grub available. Until now, I compiled a modified version of the kernel26 package with pv_ops/Xern enabled which - in theory - should also run on a real machine. If all the external modules compile properly (which they probably do), we could add these options to our normal stock kernel. I haven't tried running it on a real machine though and tpowa said in the past the nvidia module failed. This has to be tested. However, compiling Xen on a 32 bit kernel requires enabling PAE which we refused to do in the past for performance reasons (no idea if these are valid concerns) and due to the fact that we want to encourage people to run 64 bit if they have large amounts of memory. Assuming all of untested things above work, what would you prefer: 1) Switch to PAE on the 32 bit kernel, ship Xen domU support with kernel26 on both architectures. 2) Ship Xen domU support on the 64 bit kernel, and don't on the 32 bit kernel (my favorite so far). 3) Leave kernel26 unchanged and create a separate kernel26-xen package with domU support. The advantage of 3) is that the Xen kernel could be very slim, as almost all hardware drivers could be disabled. The advantage of 1) and 2) is the lower maintenance cost. On a different note, I'd like to try to switch over to pv-grub on our servers. Right now, upgrading the kernel requires copying the kernel to the dom0 manually, thus dom0 access is required to do a (potentially security relevant) kernel update. With pv-grub, upgrading the kernel and rebooting would be handled just like with a normal machine, as the dom0 only loads the pv-grub image which then runs inside the domU and can load the right kernel from there. The pv-grub is only available from Xen 3.3 and later, but we have Xen 3.2 - I compiled pv-grub from 3.4 though and guess it should work with the older hypervisor as well, but I haven't tested that yet. Anyway, please post your opinion on the issues above.
2009/10/29, Thomas Bächler <thomas@archlinux.org>:
3) Leave kernel26 unchanged and create a separate kernel26-xen package with domU support.
I would prefer the third solution. -- Arch Linux Developer http://www.archlinux.org http://www.archlinux.it
Am Freitag 30 Oktober 2009 01:59:19 schrieb Thomas Bächler:
1) Switch to PAE on the 32 bit kernel, ship Xen domU support with kernel26 on both architectures.
2) Ship Xen domU support on the 64 bit kernel, and don't on the 32 bit kernel (my favorite so far).
3) Leave kernel26 unchanged and create a separate kernel26-xen package with domU support.
What about an 4th option: Add xen support to the lts kernel which is meant for servers anyway. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Fri, 2009-10-30 at 02:12 +0100, Pierre Schmitz wrote:
What about an 4th option: Add xen support to the lts kernel which is meant for servers anyway.
Which is probably broken because pvops/xen matured after 2.6.27?
Jan de Groot schrieb:
On Fri, 2009-10-30 at 02:12 +0100, Pierre Schmitz wrote:
What about an 4th option: Add xen support to the lts kernel which is meant for servers anyway.
Which is probably broken because pvops/xen matured after 2.6.27?
That, and I doubt the lts kernel will be around very long as packages like udev (and more) are developping quite fast, and lts stays ancient. I'd like to have something recent, at least on our server.
Am Freitag 30 Oktober 2009 09:21:13 schrieb Thomas Bächler:
Jan de Groot schrieb:
On Fri, 2009-10-30 at 02:12 +0100, Pierre Schmitz wrote:
What about an 4th option: Add xen support to the lts kernel which is meant for servers anyway.
Which is probably broken because pvops/xen matured after 2.6.27?
That, and I doubt the lts kernel will be around very long as packages like udev (and more) are developping quite fast, and lts stays ancient. I'd like to have something recent, at least on our server.
Fair enough. So as long as this change does not introduce püroblems or decrease of performance for the average desktop user I am fine with whatever you like. -- Pierre Schmitz, https://users.archlinux.de/~pierre
Pierre Schmitz schrieb:
Fair enough. So as long as this change does not introduce püroblems or decrease of performance for the average desktop user I am fine with whatever you like.
The performance question is hard to answer - however, as we have pv_ops enabled anyway, I guess there will be no difference anymore. Even the difference between pv_ops and no pv_ops is very small.
Thomas Bächler wrote:
Okay, it is obvious that we need a kernel which can run on Xen domU for our servers. It would also be very convenient for those running Arch on hosting services that have pv-grub available.
What about dumping Xen in favor of KVM and move libvirt and virt-manager to extra? Glenn
RedShift schrieb:
Thomas Bächler wrote:
Okay, it is obvious that we need a kernel which can run on Xen domU for our servers. It would also be very convenient for those running Arch on hosting services that have pv-grub available.
What about dumping Xen in favor of KVM and move libvirt and virt-manager to extra?
KVM needs hardware support, while Xen is a para-virtualization environment and doesn't need any special hardware. I don't know about libvirt and what is is. However, Xen works just great (and is deployed by many commercial hosters) and the main Arch servers have relied on it for over 2 months. Xen is very stable and well-tested, I doubt you can say the last on other para-virtualization environments.
On Thu, Oct 29, 2009 at 7:59 PM, Thomas Bächler <thomas@archlinux.org> wrote:
Okay, it is obvious that we need a kernel which can run on Xen domU for our servers. It would also be very convenient for those running Arch on hosting services that have pv-grub available.
Until now, I compiled a modified version of the kernel26 package with pv_ops/Xern enabled which - in theory - should also run on a real machine.
If all the external modules compile properly (which they probably do), we could add these options to our normal stock kernel. I haven't tried running it on a real machine though and tpowa said in the past the nvidia module failed. This has to be tested.
However, compiling Xen on a 32 bit kernel requires enabling PAE which we refused to do in the past for performance reasons (no idea if these are valid concerns) and due to the fact that we want to encourage people to run 64 bit if they have large amounts of memory.
Assuming all of untested things above work, what would you prefer:
1) Switch to PAE on the 32 bit kernel, ship Xen domU support with kernel26 on both architectures.
2) Ship Xen domU support on the 64 bit kernel, and don't on the 32 bit kernel (my favorite so far).
3) Leave kernel26 unchanged and create a separate kernel26-xen package with domU support.
The advantage of 3) is that the Xen kernel could be very slim, as almost all hardware drivers could be disabled. The advantage of 1) and 2) is the lower maintenance cost.
On a different note, I'd like to try to switch over to pv-grub on our servers. Right now, upgrading the kernel requires copying the kernel to the dom0 manually, thus dom0 access is required to do a (potentially security relevant) kernel update. With pv-grub, upgrading the kernel and rebooting would be handled just like with a normal machine, as the dom0 only loads the pv-grub image which then runs inside the domU and can load the right kernel from there. The pv-grub is only available from Xen 3.3 and later, but we have Xen 3.2 - I compiled pv-grub from 3.4 though and guess it should work with the older hypervisor as well, but I haven't tested that yet.
Anyway, please post your opinion on the issues above.
I'm -1 on the PAE, leaving options 2 and 3 as viable. 2 is the easiest for maintenance; 3 sounds like the best solution long term as it would be a very lightweight and well-suited kernel package. If not for the maintenance issues, 3 seems by far the best. Is there anyway we can automate the creation of this package to the point where it doesn't take much extra effort? e.g. try to share resources with the primary kernel package. -Dan
On Fri, Oct 30, 2009 at 6:51 PM, Dan McGee <dpmcgee@gmail.com> wrote:
On Thu, Oct 29, 2009 at 7:59 PM, Thomas Bächler <thomas@archlinux.org> wrote:
Okay, it is obvious that we need a kernel which can run on Xen domU for our servers. It would also be very convenient for those running Arch on hosting services that have pv-grub available.
Until now, I compiled a modified version of the kernel26 package with pv_ops/Xern enabled which - in theory - should also run on a real machine.
If all the external modules compile properly (which they probably do), we could add these options to our normal stock kernel. I haven't tried running it on a real machine though and tpowa said in the past the nvidia module failed. This has to be tested.
However, compiling Xen on a 32 bit kernel requires enabling PAE which we refused to do in the past for performance reasons (no idea if these are valid concerns) and due to the fact that we want to encourage people to run 64 bit if they have large amounts of memory.
Assuming all of untested things above work, what would you prefer:
1) Switch to PAE on the 32 bit kernel, ship Xen domU support with kernel26 on both architectures.
2) Ship Xen domU support on the 64 bit kernel, and don't on the 32 bit kernel (my favorite so far).
3) Leave kernel26 unchanged and create a separate kernel26-xen package with domU support.
The advantage of 3) is that the Xen kernel could be very slim, as almost all hardware drivers could be disabled. The advantage of 1) and 2) is the lower maintenance cost.
On a different note, I'd like to try to switch over to pv-grub on our servers. Right now, upgrading the kernel requires copying the kernel to the dom0 manually, thus dom0 access is required to do a (potentially security relevant) kernel update. With pv-grub, upgrading the kernel and rebooting would be handled just like with a normal machine, as the dom0 only loads the pv-grub image which then runs inside the domU and can load the right kernel from there. The pv-grub is only available from Xen 3.3 and later, but we have Xen 3.2 - I compiled pv-grub from 3.4 though and guess it should work with the older hypervisor as well, but I haven't tested that yet.
Anyway, please post your opinion on the issues above.
I'm -1 on the PAE, leaving options 2 and 3 as viable. 2 is the easiest for maintenance; 3 sounds like the best solution long term as it would be a very lightweight and well-suited kernel package.
If not for the maintenance issues, 3 seems by far the best. Is there anyway we can automate the creation of this package to the point where it doesn't take much extra effort? e.g. try to share resources with the primary kernel package.
Can you illustrate why you're -1 on PAE? I can't find exact info on the performance hit caused by enabling that, and with machines having more and more RAM, I'm sure we'll eventually need to enable PAE. I _know_ we're trying to push people to 64bit, but that shouldn't mean we leave the i686 arch in the dust. I'm just sure we're EVENTUALLY going to need to enable this. Why not now?
On Fri, Oct 30, 2009 at 7:22 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Fri, Oct 30, 2009 at 6:51 PM, Dan McGee <dpmcgee@gmail.com> wrote:
On Thu, Oct 29, 2009 at 7:59 PM, Thomas Bächler <thomas@archlinux.org> wrote:
Okay, it is obvious that we need a kernel which can run on Xen domU for our servers. It would also be very convenient for those running Arch on hosting services that have pv-grub available.
Until now, I compiled a modified version of the kernel26 package with pv_ops/Xern enabled which - in theory - should also run on a real machine.
If all the external modules compile properly (which they probably do), we could add these options to our normal stock kernel. I haven't tried running it on a real machine though and tpowa said in the past the nvidia module failed. This has to be tested.
However, compiling Xen on a 32 bit kernel requires enabling PAE which we refused to do in the past for performance reasons (no idea if these are valid concerns) and due to the fact that we want to encourage people to run 64 bit if they have large amounts of memory.
Assuming all of untested things above work, what would you prefer:
1) Switch to PAE on the 32 bit kernel, ship Xen domU support with kernel26 on both architectures.
2) Ship Xen domU support on the 64 bit kernel, and don't on the 32 bit kernel (my favorite so far).
3) Leave kernel26 unchanged and create a separate kernel26-xen package with domU support.
The advantage of 3) is that the Xen kernel could be very slim, as almost all hardware drivers could be disabled. The advantage of 1) and 2) is the lower maintenance cost.
On a different note, I'd like to try to switch over to pv-grub on our servers. Right now, upgrading the kernel requires copying the kernel to the dom0 manually, thus dom0 access is required to do a (potentially security relevant) kernel update. With pv-grub, upgrading the kernel and rebooting would be handled just like with a normal machine, as the dom0 only loads the pv-grub image which then runs inside the domU and can load the right kernel from there. The pv-grub is only available from Xen 3.3 and later, but we have Xen 3.2 - I compiled pv-grub from 3.4 though and guess it should work with the older hypervisor as well, but I haven't tested that yet.
Anyway, please post your opinion on the issues above.
I'm -1 on the PAE, leaving options 2 and 3 as viable. 2 is the easiest for maintenance; 3 sounds like the best solution long term as it would be a very lightweight and well-suited kernel package.
If not for the maintenance issues, 3 seems by far the best. Is there anyway we can automate the creation of this package to the point where it doesn't take much extra effort? e.g. try to share resources with the primary kernel package.
Can you illustrate why you're -1 on PAE? I can't find exact info on the performance hit caused by enabling that, and with machines having more and more RAM, I'm sure we'll eventually need to enable PAE. I _know_ we're trying to push people to 64bit, but that shouldn't mean we leave the i686 arch in the dust.
I'm just sure we're EVENTUALLY going to need to enable this. Why not now?
Completely disagree, PAE is stupid. If you want to stick with 32 bit and have more memory available, at least run a 64 bit kernel and stick with a 32 bit userspace: http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/01128.html http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/01131.html And those numbers for performance problems? 12%: http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/00959.html Leaving i686 in the dust is hardly the concern here; why add unnecessary overhead to those machines that sit well under the 4GB mark that need that extra 12% of performance the most? -Dan
Dan McGee wrote:
On Fri, Oct 30, 2009 at 7:22 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Fri, Oct 30, 2009 at 6:51 PM, Dan McGee <dpmcgee@gmail.com> wrote:
On Thu, Oct 29, 2009 at 7:59 PM, Thomas Bächler <thomas@archlinux.org> wrote:
Okay, it is obvious that we need a kernel which can run on Xen domU for our servers. It would also be very convenient for those running Arch on hosting services that have pv-grub available.
Until now, I compiled a modified version of the kernel26 package with pv_ops/Xern enabled which - in theory - should also run on a real machine.
If all the external modules compile properly (which they probably do), we could add these options to our normal stock kernel. I haven't tried running it on a real machine though and tpowa said in the past the nvidia module failed. This has to be tested.
However, compiling Xen on a 32 bit kernel requires enabling PAE which we refused to do in the past for performance reasons (no idea if these are valid concerns) and due to the fact that we want to encourage people to run 64 bit if they have large amounts of memory.
Assuming all of untested things above work, what would you prefer:
1) Switch to PAE on the 32 bit kernel, ship Xen domU support with kernel26 on both architectures.
2) Ship Xen domU support on the 64 bit kernel, and don't on the 32 bit kernel (my favorite so far).
3) Leave kernel26 unchanged and create a separate kernel26-xen package with domU support.
The advantage of 3) is that the Xen kernel could be very slim, as almost all hardware drivers could be disabled. The advantage of 1) and 2) is the lower maintenance cost.
On a different note, I'd like to try to switch over to pv-grub on our servers. Right now, upgrading the kernel requires copying the kernel to the dom0 manually, thus dom0 access is required to do a (potentially security relevant) kernel update. With pv-grub, upgrading the kernel and rebooting would be handled just like with a normal machine, as the dom0 only loads the pv-grub image which then runs inside the domU and can load the right kernel from there. The pv-grub is only available from Xen 3.3 and later, but we have Xen 3.2 - I compiled pv-grub from 3.4 though and guess it should work with the older hypervisor as well, but I haven't tested that yet.
Anyway, please post your opinion on the issues above.
I'm -1 on the PAE, leaving options 2 and 3 as viable. 2 is the easiest for maintenance; 3 sounds like the best solution long term as it would be a very lightweight and well-suited kernel package.
If not for the maintenance issues, 3 seems by far the best. Is there anyway we can automate the creation of this package to the point where it doesn't take much extra effort? e.g. try to share resources with the primary kernel package.
Can you illustrate why you're -1 on PAE? I can't find exact info on the performance hit caused by enabling that, and with machines having more and more RAM, I'm sure we'll eventually need to enable PAE. I _know_ we're trying to push people to 64bit, but that shouldn't mean we leave the i686 arch in the dust.
I'm just sure we're EVENTUALLY going to need to enable this. Why not now?
Completely disagree, PAE is stupid. If you want to stick with 32 bit and have more memory available, at least run a 64 bit kernel and stick with a 32 bit userspace: http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/01128.html http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/01131.html
I do that, and the are some usability issues. Nothing insurmountable. Like needing to use "linux32" before ./configure's, makepkg's, etc. Nothing a graphical user would notice, but would be more noticeable in Arch where building from the AUR is common. From memory, installing a 64 bit kernel when appropriate was a release goal for Fedora 11 but got dropped. I do not know why, but I guess the small annoyances I find are probably the reason.
And those numbers for performance problems? 12%: http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/00959.html
Leaving i686 in the dust is hardly the concern here; why add unnecessary overhead to those machines that sit well under the 4GB mark that need that extra 12% of performance the most?
Red Hat did a more comprehensive survey of PAE (a few years ago now). They found about the same thing: ~0% for some operations, 10% for the worst (which reflects the test case Dan linked). It averaged out to ~1%. See http://people.redhat.com/nmurray/RHEL-2.1-VM-whitepaper.pdf (top page 10). Allan
participants (8)
-
Aaron Griffin
-
Allan McRae
-
Dan McGee
-
Giovanni Scafora
-
Jan de Groot
-
Pierre Schmitz
-
RedShift
-
Thomas Bächler