I've been using Arch Linux for about 4 years now. I have it on a few important systems at work and it has been doing very well.
This morning I saw "/usr is not mounted. This is not supported." in my boot up after a recent rc.sysinit update.
What is this, bait and switch? I've been running Linux and BSD systems since 1996 and typically always have /usr in a separate partition (as well as /var, /home/ and /tmp, but lately been using a ram /tmp).
Why does /usr even exist if it can't be on a separate partition? Why not just combine /usr/lib and /lib? And /usr/bin and /bin? And /usr/sbin and /sbin? Why have the distinction at all if it can't be on separate partition?
I understand that historically that /usr often use to be on different drive, and that is not really an issue nowadays. Only this year have I started not putting /usr into separate partitions because I've been making thumbdrive installs, and did not really see any benefit to making so many partitions (automatically created anyways, with a custom install script).
Does this "/usr is not mounted. This is not supported." mean I'm going to have to eventually fix (dump/fix/restore) all my systems that are now currently running fine (and that I and others are depending on at my work) because Arch Linux no longer supports /usr on a different partition (due to rc.sysinit failing, not just printing an error message)? I run Arch Linux on more than 10 systems, and about 6 or 7 of those are at work (where Arch has been working out very well).
I'm not looking forward to redoing all these systems that are running fine if this is where Arch is headed and rc.sysinit is not fixed to take out this new requirement.
I know this a bit of a rant, but this "/usr is not mounted. This is not supported." error message is definitely not getting this day off to a good start...
Definitely not wanting to give up Arch for such a simple issue....
Dwight
This is not a new thing, it has been "broken" for quite a while.
http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken
Sander
On Mon, Oct 24, 2011 at 9:42 AM, Dwight Schauer dschauer@gmail.com wrote:
I've been using Arch Linux for about 4 years now. I have it on a few important systems at work and it has been doing very well.
This morning I saw "/usr is not mounted. This is not supported." in my boot up after a recent rc.sysinit update.
What is this, bait and switch? I've been running Linux and BSD systems since 1996 and typically always have /usr in a separate partition (as well as /var, /home/ and /tmp, but lately been using a ram /tmp).
Why does /usr even exist if it can't be on a separate partition? Why not just combine /usr/lib and /lib? And /usr/bin and /bin? And /usr/sbin and /sbin? Why have the distinction at all if it can't be on separate partition?
I understand that historically that /usr often use to be on different drive, and that is not really an issue nowadays. Only this year have I started not putting /usr into separate partitions because I've been making thumbdrive installs, and did not really see any benefit to making so many partitions (automatically created anyways, with a custom install script).
Does this "/usr is not mounted. This is not supported." mean I'm going to have to eventually fix (dump/fix/restore) all my systems that are now currently running fine (and that I and others are depending on at my work) because Arch Linux no longer supports /usr on a different partition (due to rc.sysinit failing, not just printing an error message)? I run Arch Linux on more than 10 systems, and about 6 or 7 of those are at work (where Arch has been working out very well).
I'm not looking forward to redoing all these systems that are running fine if this is where Arch is headed and rc.sysinit is not fixed to take out this new requirement.
I know this a bit of a rant, but this "/usr is not mounted. This is not supported." error message is definitely not getting this day off to a good start...
Definitely not wanting to give up Arch for such a simple issue....
Dwight
On 2011-10-24 17:42, Dwight Schauer wrote:
This morning I saw "/usr is not mounted. This is not supported." in my boot up after a recent rc.sysinit update.
What is this, bait and switch? I've been running Linux and BSD systems since 1996 and typically always have /usr in a separate partition (as well as /var, /home/ and /tmp, but lately been using a ram /tmp).
See http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken for an explanation on why booting without a separate /usr does not really work, as well as this thread http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/1337.
Note I said "booting". If /usr is mounted by your initramfs, it's perfectly fine.
Why does /usr even exist if it can't be on a separate partition? Why not just combine /usr/lib and /lib? And /usr/bin and /bin? And /usr/sbin and /sbin? Why have the distinction at all if it can't be on separate partition?
I remember reading a few mailing list posts about this, but can't find them right now. http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/3480 appears to be relevant -- it's easier to snapshot a single /usr than /bin+/lib+/sbin+...:
| The point is not to have 6-10 top-level dirs for the system to manage, | but only a single one. We need a single point to snapshot or share.
Am 24.10.2011 16:42, schrieb Dwight Schauer:
I've been using Arch Linux for about 4 years now. I have it on a few important systems at work and it has been doing very well.
This morning I saw "/usr is not mounted. This is not supported." in my boot up after a recent rc.sysinit update.
Basically, we will not fix any bugs in the future that are the result of a separate /usr, we will instantly close them instead. The link Sander posted gives a few examples.
If everything works fine for you, then nothing changes for you.
Does this "/usr is not mounted. This is not supported." mean I'm going to have to eventually fix (dump/fix/restore) all my systems
No. Even if things fail entirely, we will provide a hook for mkinitcpio that mounts /usr before switching to the real root filesystem. This will eliminate all your potential bugs. However - this hook hasn't been written yet.
Thomas Bächler wrote:
No. Even if things fail entirely, we will provide a hook for mkinitcpio that mounts /usr before switching to the real root filesystem. This will eliminate all your potential bugs. However
- this hook hasn't been written yet.
Imagine somebody with a desktop arch-linux. For some reason his system "fails entirely". Should he or she now mess with a live-CD to backport the yet unwritten hook once it is finished and doodle on his thumbs waiting for it? Reinstall everything?
clemens
On Tue, Oct 25, 2011 at 2:18 PM, clemens fischer ino-news@spotteswoode.dnsalias.org wrote:
Thomas Bächler wrote:
No. Even if things fail entirely, we will provide a hook for mkinitcpio that mounts /usr before switching to the real root filesystem. This will eliminate all your potential bugs. However
- this hook hasn't been written yet.
Imagine somebody with a desktop arch-linux. For some reason his system "fails entirely". Should he or she now mess with a live-CD to backport the yet unwritten hook once it is finished and doodle on his thumbs waiting for it? Reinstall everything?
Clarifying again: nothing is more broken now than it used to be, and no one are forced to do anything. We are just now pointing out that things _are_ broken, so people should beware.
-t
On Mon, Oct 24, 2011 at 4:42 PM, Dwight Schauer dschauer@gmail.com wrote:
I've been using Arch Linux for about 4 years now. I have it on a few important systems at work and it has been doing very well.
This morning I saw "/usr is not mounted. This is not supported." in my boot up after a recent rc.sysinit update.
What is this, bait and switch? I've been running Linux and BSD systems since 1996 and typically always have /usr in a separate partition (as well as /var, /home/ and /tmp, but lately been using a ram /tmp).
Why does /usr even exist if it can't be on a separate partition? Why not just combine /usr/lib and /lib? And /usr/bin and /bin? And /usr/sbin and /sbin? Why have the distinction at all if it can't be on separate partition?
Several people already pointed out the relevant information about this, but I thought I should offer my two cents as I was responsible for the error message.
The situation is currently that lots of tools break silently if /usr is not mounted at boot. In most cases you will not notice, and in the cases you do notice it is really difficult to tell what is actually going on.
From time to time we get bug reports that are really difficult to
debug, and that eventually turn out to be due to a separate /usr. Once we figure out the cause, we usually end up having to say, sorry, there is nothing we can do about that, but in the meantime we have wasted a lot of time. Therefore, we really want to get the message out there that at the moment things simply don't work as they should with a separate /usr. Then people will at least know that this is a likely cause of any weird problems they experience.
There are two ways to solve this: either merge your / and your /usr partitions, or make your initramfs mount /usr so init won't even know that /usr is separate.
We are currently working on adding support for the second approach, but we are not there yet (I have some patches against mkinitcpio to add this, but they rely on a patch by Thomas against busybox that has not yet landed upstream).
Cheers,
Tom
On Mon, Oct 24, 2011 at 1:18 PM, Tom Gundersen teg@jklm.no wrote:
There are two ways to solve this: either merge your / and your /usr partitions, or make your initramfs mount /usr so init won't even know that /usr is separate.
We are currently working on adding support for the second approach, but we are not there yet (I have some patches against mkinitcpio to add this, but they rely on a patch by Thomas against busybox that has not yet landed upstream).
Ok, so it is not as much of a problem as I initially thought it would be.
On new large media installs I'll try to not use /usr until this gets resolved. On current installs they all seem to be working fine (I've not noticed any lack of functionality) I'll just wait until the mkinitcpio patches are completed and mkinitcpio is released with /usr mount support.
Thanks, Dwight Schauer
Tom Gundersen wrote:
From time to time we get bug reports that are really difficult to debug, and that eventually turn out to be due to a separate /usr. Once we figure out the cause, we usually end up having to say, sorry, there is nothing we can do about that, but in the meantime we have wasted a lot of time. Therefore, we really want to get the message out there that at the moment things simply don't work as they should with a separate /usr. Then people will at least know that this is a likely cause of any weird problems they experience.
There are two ways to solve this: either merge your / and your /usr partitions, or make your initramfs mount /usr so init won't even know that /usr is separate.
We are currently working on adding support for the second approach, but we are not there yet (I have some patches against mkinitcpio to add this, but they rely on a patch by Thomas against busybox that has not yet landed upstream).
What patch would that be? THE-FAVOURITE-SEARCH-ENGINE didn't pull anything useful for "patch Thomas-Bächler busybox". Can somebody point us to the relevant code, please?
A hook to mount the users /usr would presumably go right before
"if [ -x /lib/udev/udevd ]; then"
in "/lib/initcpio/init"?
Are the symlinks in "/dev/disk/by-label/" by then? I guess not, since udevd rules are responsible for setting them up, right?
So how do the boot-loaders do this? My first thought was to have a kernel command line parameter "mnt_usr_from=/dev/disk/by-label/my-usr" or whatever and then use busybox' ``mount "$mnt_usr_from" /usr''.
I do find it annoying to have /usr with all the bulky GUI crap, audio-tools and whatnot in "/". FreeBSD has a clean separation between what's needed to bring up, build and run a basic system ( -> /usr ) and user-toys, including all the GUI and multimedia stuff ( -> /usr/local ).
I always regret linux cramming everything into /usr. Some vital programs needed at startup are in /[s]bin, some in /lib, but look at the rules in /lib/udev/rules.d/: while vboxdrv and alsa-restore could equally well run when /usr is finally up, the device-mapper rules check for dmsetup residing in usr/sbin, although they are needed early!
clemens
Am 25.10.2011 14:12, schrieb clemens fischer:
We are currently working on adding support for the second approach, but we are not there yet (I have some patches against mkinitcpio to add this, but they rely on a patch by Thomas against busybox that has not yet landed upstream).
What patch would that be? THE-FAVOURITE-SEARCH-ENGINE didn't pull anything useful for "patch Thomas-Bächler busybox". Can somebody point us to the relevant code, please?
I am not sure which patch he means. I wrote something recently, but I only shared that with Tom in private conversations and it is not related to this issue.
A hook to mount the users /usr would presumably go right before
"if [ -x /lib/udev/udevd ]; then"
in "/lib/initcpio/init"?
Are the symlinks in "/dev/disk/by-label/" by then? I guess not, since udevd rules are responsible for setting them up, right?
Mounting /usr needs to go to the initramfs. It is possible to implement a mount handler for this. At this stage, the by-label symlinks exist already.
On Tue, Oct 25, 2011 at 8:59 AM, Thomas Bächler thomas@archlinux.org wrote:
Mounting /usr needs to go to the initramfs. It is possible to implement a mount handler for this. At this stage, the by-label symlinks exist already.
Would the /usr location be determined when the initramfs is created, or would it determine the location at runtime via /etc/fstab? Just wanted to make sure it is the latter.
By label? Is that assuming /usr location can be guessed by the label? That would not work on a lot of setups. Even if a mountpoint in label was used, that does not account for multiple disks attached, all with Linux installs on them.
Or by by-label do you mean because /etc/fstab is being used, possibly with a label in the entries (I use by-uuid entries) that the correct one will be chosen?
Am 25.10.2011 16:19, schrieb Dwight Schauer:
Would the /usr location be determined when the initramfs is created, or would it determine the location at runtime via /etc/fstab? Just wanted to make sure it is the latter.
Yes, you mounted root, so you can now look at fstab.
By label? Is that assuming /usr location can be guessed by the label? That would not work on a lot of setups. Even if a mountpoint in label was used, that does not account for multiple disks attached, all with Linux installs on them.
Just saying, you can use whatever your fstab says - be it a device, uuid, label, or other symlink.
Thomas Bächler wrote:
Am 25.10.2011 14:12, schrieb clemens fischer:
We are currently working on adding support for the second approach, but we are not there yet (I have some patches against mkinitcpio to add this, but they rely on a patch by Thomas against busybox that has not yet landed upstream).
What patch would that be? THE-FAVOURITE-SEARCH-ENGINE didn't pull anything useful for "patch Thomas-Bächler busybox". Can somebody point us to the relevant code, please?
I am not sure which patch he means. I wrote something recently, but I only shared that with Tom in private conversations and it is not related to this issue.
A hook to mount the users /usr would presumably go right before
"if [ -x /lib/udev/udevd ]; then"
in "/lib/initcpio/init"?
Are the symlinks in "/dev/disk/by-label/" by then? I guess not, since udevd rules are responsible for setting them up, right?
Mounting /usr needs to go to the initramfs. It is possible to implement a mount handler for this. At this stage, the by-label symlinks exist already.
AFAIU udevd(8) is responsible for setting up those symlinks. How can they exist _before_ udevd(8) is started?
clemens
Am 25.10.2011 20:12, schrieb clemens fischer:
Mounting /usr needs to go to the initramfs. It is possible to implement a mount handler for this. At this stage, the by-label symlinks exist already.
AFAIU udevd(8) is responsible for setting up those symlinks. How can they exist _before_ udevd(8) is started?
I'll let you go back and read everything again, what you are saying makes no sense whatsoever, and I have no idea how to reply to it.
Thomas Bächler wrote:
Am 25.10.2011 20:12, schrieb clemens fischer:
Mounting /usr needs to go to the initramfs. It is possible to implement a mount handler for this. At this stage, the by-label symlinks exist already.
AFAIU udevd(8) is responsible for setting up those symlinks. How can they exist _before_ udevd(8) is started?
I'll let you go back and read everything again, what you are saying makes no sense whatsoever, and I have no idea how to reply to it.
Lucky you, I have a way to explain it: There are udev rules referencing stuff in /usr. If people mount /usr by-label or by-uuid, udev must have completed to setup those symlinks. Catch-22.
clemens
On Wed, Oct 26, 2011 at 12:57 PM, clemens fischer ino-news@spotteswoode.dnsalias.org wrote:
Thomas Bächler wrote:
Am 25.10.2011 20:12, schrieb clemens fischer:
Mounting /usr needs to go to the initramfs. It is possible to implement a mount handler for this. At this stage, the by-label symlinks exist already.
AFAIU udevd(8) is responsible for setting up those symlinks. How can they exist _before_ udevd(8) is started?
I'll let you go back and read everything again, what you are saying makes no sense whatsoever, and I have no idea how to reply to it.
Lucky you, I have a way to explain it: There are udev rules referencing stuff in /usr. If people mount /usr by-label or by-uuid, udev must have completed to setup those symlinks. Catch-22.
If you want to understand this I suggest you look at the udev hook in initramfs. There is no problem.
-t
On Wed, Oct 26, 2011 at 3:13 PM, Tom Gundersen wrote:
On Wed, Oct 26, 2011 at 12:57 PM, clemens fischer wrote:
Lucky you, I have a way to explain it: There are udev rules referencing stuff in /usr. If people mount /usr by-label or by-uuid, udev must have completed to setup those symlinks. Catch-22.
If you want to understand this I suggest you look at the udev hook in initramfs. There is no problem.
Well, when I started this thread I did not know it get so much discussion.
Anyways, I'm not worried about it any more.
My root= on my kernel boot line is using /dev/by-uuid/ so if the initramfs can find the root device, I'm sure it can find the /usr device from the rootfs /etc/fstab.
I've not noticed any breakage on all my system's that have a seperate /usr, apart from the message doing boot.
As long as the message and any potential problems from a seperate /usr go away when the initramfs hook is completed, I'll be happy.
-das
Dwight Schauer wrote:
My root= on my kernel boot line is using /dev/by-uuid/ so if the initramfs can find the root device, I'm sure it can find the /usr device from the rootfs /etc/fstab.
I've not noticed any breakage on all my system's that have a seperate /usr, apart from the message doing boot.
Don't you have a boot message saying "minilogd not found" or somesuch?
$ which minilogd -> /usr/sbin/minilogd
By the time /etc/rc.sysinit starts minilogd, /usr is not available, so there's no minilogd and hence, no log of early boot messages.
I'm thinking of solving this particular problem like this:
minilogd requires the following libs:
-rwxr-xr-x 1 root root 11K Oct 18 18:34 /usr/sbin/minilogd lrwxrwxrwx 1 root root 12 Sep 9 01:23 /usr/lib32/libc.so.6 -> libc-2.14.so -rwxr-xr-x 1 root root 1.4M Sep 9 01:23 /usr/lib32/libc-2.14.so /usr/lib32/libc.so.6: lrwxrwxrwx 1 root root 26 Sep 9 01:23 /lib/ld-linux.so.2 -> ../usr/lib32/ld-linux.so.2 -rwxr-xr-x 1 root root 141K Sep 9 01:23 /usr/lib32/ld-2.14.so
I don't know why it asks for libs out of usr/lib32, this output is from (the recursive use of) readelf(1). This is on a 64bit PC.
So I could "mount -B / /mnt/root" and copy the needed file hierarchy to /mnt/root/usr/.
On the other hand, rc.sysinit also invokes /sbin/bootlogd, which leaves most of the interesting stuff in var/log/boot, so this would be an academic exercise ...
clemens
Am 27.10.2011 10:38, schrieb clemens fischer:
minilogd requires the following libs:
-rwxr-xr-x 1 root root 11K Oct 18 18:34 /usr/sbin/minilogd lrwxrwxrwx 1 root root 12 Sep 9 01:23 /usr/lib32/libc.so.6 -> libc-2.14.so -rwxr-xr-x 1 root root 1.4M Sep 9 01:23 /usr/lib32/libc-2.14.so /usr/lib32/libc.so.6: lrwxrwxrwx 1 root root 26 Sep 9 01:23 /lib/ld-linux.so.2 -> ../usr/lib32/ld-linux.so.2 -rwxr-xr-x 1 root root 141K Sep 9 01:23 /usr/lib32/ld-2.14.so
I don't know why it asks for libs out of usr/lib32, this output is from (the recursive use of) readelf(1). This is on a 64bit PC.
Where do you get that from?
$ ldd /usr/sbin/minilogd linux-vdso.so.1 => (0x00007fff09d0c000) libc.so.6 => /lib/libc.so.6 (0x00007f9c3fc9b000) /lib/ld-linux-x86-64.so.2 (0x00007f9c40022000) $ readelf -d /usr/sbin/minilogd | grep library 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
Thomas Bächler wrote:
Am 27.10.2011 10:38, schrieb clemens fischer:
minilogd requires the following libs:
-rwxr-xr-x 1 root root 11K Oct 18 18:34 /usr/sbin/minilogd lrwxrwxrwx 1 root root 12 Sep 9 01:23 /usr/lib32/libc.so.6 -> libc-2.14.so -rwxr-xr-x 1 root root 1.4M Sep 9 01:23 /usr/lib32/libc-2.14.so /usr/lib32/libc.so.6: lrwxrwxrwx 1 root root 26 Sep 9 01:23 /lib/ld-linux.so.2 -> ../usr/lib32/ld-linux.so.2 -rwxr-xr-x 1 root root 141K Sep 9 01:23 /usr/lib32/ld-2.14.so
I don't know why it asks for libs out of usr/lib32, this output is from (the recursive use of) readelf(1). This is on a 64bit PC.
Where do you get that from?
$ ldd /usr/sbin/minilogd linux-vdso.so.1 => (0x00007fff09d0c000) libc.so.6 => /lib/libc.so.6 (0x00007f9c3fc9b000) /lib/ld-linux-x86-64.so.2 (0x00007f9c40022000) $ readelf -d /usr/sbin/minilogd | grep library 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
I'm really glad you asked! I have a script called "missing-libs.sh", and quite a number of self compiled programs. Whenever pacman upgrades my system, some libs required by my programs may vanish, so "missing-libs.sh" is used to scan my entire usr/local/{{s,}bin,lib,libexec} recursively to check that the programs get to keep their dependencies. It uses "readelf -d" instead of ldd(1), because the latter actually runs any binary, which I can't afford for security reasons. Readelf prints a "NEEDED" entry for dependencies besides other info, but (normally) without a path, so the script has an init stage where all libs reported by "ldconfig -p" are cashed in an array to have path info at hand. I have to do this, because some lib may require further libs which have to be checked recursively.
$ readelf -d /usr/sbin/minilogd
Dynamic section at offset 0x1e50 contains 20 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] ... $ readelf -d /lib/libc.so.6
Dynamic section at offset 0x159b40 contains 27 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2] ...
"ld-linux-x86-64.so.2" had no entry "NEEDED", so my script stops there.
Now your question points out a bug: the first lib found is displayed, and this may be a 32bit lib! I'll have to come up with a fix for that that avoids ldd(1). My system was converted from 32 bit to 64 bit not long ago, the script had been working for so long, it just didn't cross my mind that the caching was at fault.
I just see that:
$ readelf -h /usr/sbin/minilogd ELF Header: Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 Class: ELF64 ...
So I'll have to account for that "Class" attribute of binaries. Oh well, I need to avoid ldd, so it has to be done. Hmm, unless I can efficiently sandbox ldd, entire directories need to be scanned afterall ...
clemens
Am 27.10.2011 21:15, schrieb clemens fischer:
$ readelf -d /usr/sbin/minilogd
Dynamic section at offset 0x1e50 contains 20 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] ... $ readelf -d /lib/libc.so.6
Dynamic section at offset 0x159b40 contains 27 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2] ...
"ld-linux-x86-64.so.2" had no entry "NEEDED", so my script stops there.
Your system has two versions of libc.so.6: A 32 bit and a 64 bit version (/usr/lib32/libc.so.6 vs. /lib/libc.so.6), and your script cannot distinguish them. Actually, using readelf, this is not so easy to solve:
1) A 64 bit binary always loads only 64 bit libraries, so the ELF class of the library can help. 2) The length of the address at the beginning of the line is 64 bit here, not 32 bit.
I think namcap has code for this, but I am not sure.
Thomas Bächler wrote:
Your system has two versions of libc.so.6: A 32 bit and a 64 bit version (/usr/lib32/libc.so.6 vs. /lib/libc.so.6), and your script cannot distinguish them. Actually, using readelf, this is not so easy to solve:
- A 64 bit binary always loads only 64 bit libraries, so the ELF class
of the library can help. 2) The length of the address at the beginning of the line is 64 bit here, not 32 bit.
I used the ELF "Class:" line of "readelf -h". Missing-libs.sh works now.
clemens
On Thu, Oct 27, 2011 at 3:38 AM, clemens fischer wrote:
Dwight Schauer wrote:
My root= on my kernel boot line is using /dev/by-uuid/ so if the initramfs can find the root device, I'm sure it can find the /usr device from the rootfs /etc/fstab.
I've not noticed any breakage on all my system's that have a seperate /usr, apart from the message doing boot.
Don't you have a boot message saying "minilogd not found" or somesuch?
I don't see that in /var/log/boot. I do see the "/usr is not mounted. This is not supported." in /var/log/boot.
On the other hand, rc.sysinit also invokes /sbin/bootlogd, which leaves most of the interesting stuff in var/log/boot, so this would be an academic exercise ...
And this works even with a separate /var, but that is beside the point.
clemens fischer wrote:
Dwight Schauer wrote:
My root= on my kernel boot line is using /dev/by-uuid/ so if the initramfs can find the root device, I'm sure it can find the /usr device from the rootfs /etc/fstab.
I've not noticed any breakage on all my system's that have a seperate /usr, apart from the message doing boot.
Don't you have a boot message saying "minilogd not found" or somesuch?
$ which minilogd -> /usr/sbin/minilogd
By the time /etc/rc.sysinit starts minilogd, /usr is not available, so there's no minilogd and hence, no log of early boot messages.
I'm thinking of solving this particular problem like this:
minilogd requires the following libs:
-rwxr-xr-x 1 root root 11K Oct 18 18:34 /usr/sbin/minilogd lrwxrwxrwx 1 root root 12 Sep 9 01:23 /usr/lib32/libc.so.6 -> libc-2.14.so -rwxr-xr-x 1 root root 1.4M Sep 9 01:23 /usr/lib32/libc-2.14.so /usr/lib32/libc.so.6: lrwxrwxrwx 1 root root 26 Sep 9 01:23 /lib/ld-linux.so.2 -> ../usr/lib32/ld-linux.so.2 -rwxr-xr-x 1 root root 141K Sep 9 01:23 /usr/lib32/ld-2.14.so
I don't know why it asks for libs out of usr/lib32, this output is from (the recursive use of) readelf(1). This is on a 64bit PC.
So I could "mount -B / /mnt/root" and copy the needed file hierarchy to /mnt/root/usr/.
Mkinitcpio already seems to be smart enough:
BINARIES="/usr/sbin/minilogd"
in etc/mkinitcpio.conf should do the trick.
clemens
On Thu, Oct 27, 2011 at 9:18 PM, clemens fischer ino-news@spotteswoode.dnsalias.org wrote:
clemens fischer wrote:
Dwight Schauer wrote:
My root= on my kernel boot line is using /dev/by-uuid/ so if the initramfs can find the root device, I'm sure it can find the /usr device from the rootfs /etc/fstab.
I've not noticed any breakage on all my system's that have a seperate /usr, apart from the message doing boot.
Don't you have a boot message saying "minilogd not found" or somesuch?
$ which minilogd -> /usr/sbin/minilogd
By the time /etc/rc.sysinit starts minilogd, /usr is not available, so there's no minilogd and hence, no log of early boot messages.
I'm thinking of solving this particular problem like this:
minilogd requires the following libs:
-rwxr-xr-x 1 root root 11K Oct 18 18:34 /usr/sbin/minilogd lrwxrwxrwx 1 root root 12 Sep 9 01:23 /usr/lib32/libc.so.6 -> libc-2.14.so -rwxr-xr-x 1 root root 1.4M Sep 9 01:23 /usr/lib32/libc-2.14.so /usr/lib32/libc.so.6: lrwxrwxrwx 1 root root 26 Sep 9 01:23 /lib/ld-linux.so.2 -> ../usr/lib32/ld-linux.so.2 -rwxr-xr-x 1 root root 141K Sep 9 01:23 /usr/lib32/ld-2.14.so
I don't know why it asks for libs out of usr/lib32, this output is from (the recursive use of) readelf(1). This is on a 64bit PC.
So I could "mount -B / /mnt/root" and copy the needed file hierarchy to /mnt/root/usr/.
Mkinitcpio already seems to be smart enough:
BINARIES="/usr/sbin/minilogd"
in etc/mkinitcpio.conf should do the trick.
clemens
Sorry if I'm terribly mixing things up, but isn't minilogd dead? http://mailman.archlinux.org/pipermail/arch-projects/2011-October/001982.htm...
Karol Blazewicz wrote:
On Thu, Oct 27, 2011 at 9:18 PM, clemens fischer <...> wrote:
Mkinitcpio already seems to be smart enough:
BINARIES="/usr/sbin/minilogd"
in etc/mkinitcpio.conf should do the trick.
Sorry if I'm terribly mixing things up, but isn't minilogd dead? http://mailman.archlinux.org/pipermail/arch-projects/2011-October/001982.htm...
Ok, so if bootlogd is superior to minilogd, and minilogd is buggy, why are both started right next to each other in etc/rc.sysinit?
Anyway, I took out minilogd out of BINARIES. Bootlogd does better.
clemens
On Fri, Oct 28, 2011 at 3:20 PM, clemens fischer ino-news@spotteswoode.dnsalias.org wrote:
Karol Blazewicz wrote:
On Thu, Oct 27, 2011 at 9:18 PM, clemens fischer <...> wrote:
Mkinitcpio already seems to be smart enough:
BINARIES="/usr/sbin/minilogd"
in etc/mkinitcpio.conf should do the trick.
Sorry if I'm terribly mixing things up, but isn't minilogd dead? http://mailman.archlinux.org/pipermail/arch-projects/2011-October/001982.htm...
Ok, so if bootlogd is superior to minilogd, and minilogd is buggy, why are both started right next to each other in etc/rc.sysinit?
Read the whole patch:
diff --git a/rc.sysinit b/rc.sysinit index f82368a..d802f3f 100755 --- a/rc.sysinit +++ b/rc.sysinit @@ -31,8 +31,7 @@ findmnt / --options ro &>/dev/null ||
run_hook sysinit_start
-# start up our mini logger until syslog takes over -minilogd +# log all console messages bootlogd -p /run/bootlogd.pid
minilogd is going to be removed.
On Fri, Oct 28, 2011 at 3:20 PM, clemens fischer ino-news@spotteswoode.dnsalias.org wrote:
Karol Blazewicz wrote:
On Thu, Oct 27, 2011 at 9:18 PM, clemens fischer <...> wrote:
Mkinitcpio already seems to be smart enough:
BINARIES="/usr/sbin/minilogd"
in etc/mkinitcpio.conf should do the trick.
Sorry if I'm terribly mixing things up, but isn't minilogd dead? http://mailman.archlinux.org/pipermail/arch-projects/2011-October/001982.htm...
Ok, so if bootlogd is superior to minilogd, and minilogd is buggy, why are both started right next to each other in etc/rc.sysinit?
As Karol pointed out, they will not be any longer. However, the original reason was that they are designed to do completely different things (I'm sure Google can explain better than me exactly what they do).
-t
Tom Gundersen wrote:
On Wed, Oct 26, 2011 at 12:57 PM, clemens fischer <...> wrote:
Thomas Bächler wrote:
Am 25.10.2011 20:12, schrieb clemens fischer:
Mounting /usr needs to go to the initramfs. It is possible to implement a mount handler for this. At this stage, the by-label symlinks exist already.
AFAIU udevd(8) is responsible for setting up those symlinks. How can they exist _before_ udevd(8) is started?
I'll let you go back and read everything again, what you are saying makes no sense whatsoever, and I have no idea how to reply to it.
Lucky you, I have a way to explain it: There are udev rules referencing stuff in /usr. If people mount /usr by-label or by-uuid, udev must have completed to setup those symlinks. Catch-22.
If you want to understand this I suggest you look at the udev hook in initramfs. There is no problem.
I think I have to take back what I said about a catch-22. I didn't have a clear understanding about the switch over from the initramfs to the full system.
Correct me if I'm wrong:
1. The initramfs provides what's necessary to run a full system. It loads the needed modules and lets udev install the systems devices.
2. See switch_root(8): "switch_root moves already mounted /proc, /dev and /sys to newroot and makes newroot the new root filesystem and starts init process."
3. The full system is made operational by eg. etc/rc.sysinit and friends.
So indeed a carefully crafted "mount /usr" can rely on labels and uuids of filesystems to be available very early.
clemens
On Wed, Oct 26, 2011 at 5:57 AM, clemens fischer ino-news@spotteswoode.dnsalias.org wrote:
Thomas Bächler wrote:
Am 25.10.2011 20:12, schrieb clemens fischer:
Mounting /usr needs to go to the initramfs. It is possible to implement a mount handler for this. At this stage, the by-label symlinks exist already.
AFAIU udevd(8) is responsible for setting up those symlinks. How can they exist _before_ udevd(8) is started?
I'll let you go back and read everything again, what you are saying makes no sense whatsoever, and I have no idea how to reply to it.
Lucky you, I have a way to explain it: There are udev rules referencing stuff in /usr. If people mount /usr by-label or by-uuid, udev must have completed to setup those symlinks. Catch-22.
most of the time initramfs install scripts will inject everything you need, even if stuff is in /usr, just fine, and it will all work correctly ... in initramfs. udev has everything it needs to run properly in that environment (once the real init takes over these assurances *POOF*) ... keep in mind udev will actually be ran twice, the first being killed off before real init exec's and takes over.
the reasons for not supporting separate /usr are very similar to the reasons for not supporting /run and /run/lock on within var anymore -- define "system binary". the answer is transient, subjective, and contextual -- my system is not your system is not their system -- eg, systemd needs udev at step ONE ... nobody would have guessed init would dep dbus 5+ years ago (except the visionaries of course ;-). separate usr is mostly pointless in my experience, though there are probably several appropriate niche cases.
per your "udev references /usr", if you look at the rule, it tries /sbin and /usr/sbin -- in that order -- and it just so happens that:
# which dmsetup /sbin/dmsetup
udev doesnt do much beyond actually creating the symlinks, the UUIDs/LABELs -> blockdev mappings are discovered by blkid (take a look at the rules). like most things "*nix-ey", udevd is more-or-less a dumb shell that provides a convenient environment for *other* tools to do the heavy lifting. it's "personal" task is to create nodes with proper permissions + names + links as the kernel finds them ... that's about it.
Tom/Thomas, I would be interested in experimenting with this as time allows. I'm not sure i grasp the problem though ... after switch_root and mount --move why can't initiscripts or systemd simply notice the mount and tear it down? i must be overlooking something critical because it's like 4am, but this seems related to a problem ive been having w/systemd (on GPT -> MDRAID -> LVM2 ->EXT4) where at the last second systemd flashing a message about not being about to deactivate one device (the root?, i have swap/boot/root all on that array)
On Tue, Oct 25, 2011 at 2:12 PM, clemens fischer ino-news@spotteswoode.dnsalias.org wrote:
What patch would that be? THE-FAVOURITE-SEARCH-ENGINE didn't pull anything useful for "patch Thomas-Bächler busybox". Can somebody point us to the relevant code, please?
Thomas is right, that is not strictly speaking needed (but see below).
A hook to mount the users /usr would presumably go right before
"if [ -x /lib/udev/udevd ]; then"
in "/lib/initcpio/init"?
This is what I had in mind: https://github.com/teg/mkinitcpio/commit/76dacf8b9de9cc0409741840b1d8e449862fc846.
To make it work nicely, we should also add the dual logic to shutdown: https://github.com/teg/mkinitcpio/commit/42610beb5a317d63dc76dabb72a9061a799b280b, which will pivot back to the initramfs and unmount /usr cleanly. This is where we need some busybox work, and I guess this patch should be discussed a bit more, maybe it is not the best way to do it.
Are the symlinks in "/dev/disk/by-label/" by then? I guess not, since udevd rules are responsible for setting them up, right?
So how do the boot-loaders do this? My first thought was to have a kernel command line parameter "mnt_usr_from=/dev/disk/by-label/my-usr" or whatever and then use busybox' ``mount "$mnt_usr_from" /usr''.
I do find it annoying to have /usr with all the bulky GUI crap, audio-tools and whatnot in "/". FreeBSD has a clean separation between what's needed to bring up, build and run a basic system ( -> /usr ) and user-toys, including all the GUI and multimedia stuff ( -> /usr/local ).
I always regret linux cramming everything into /usr. Some vital programs needed at startup are in /[s]bin, some in /lib, but look at the rules in /lib/udev/rules.d/: while vboxdrv and alsa-restore could equally well run when /usr is finally up, the device-mapper rules check for dmsetup residing in usr/sbin, although they are needed early!
It is currently a mess. I agree with the people advocating putting everything in /usr (and symlink /bin, /sbin and /lib for compatibility). That obviously requires being able to mount /usr from initramfs.
-t
Am 25.10.2011 17:51, schrieb Tom Gundersen:
This is what I had in mind: https://github.com/teg/mkinitcpio/commit/76dacf8b9de9cc0409741840b1d8e449862fc846.
Just two problems: 1) You overmount the initramfs' /usr for a short time. Nobody knows what that might lead to. 2) You mount /usr in read-write mode, this will conflict with fsck.
On Tue, Oct 25, 2011 at 6:31 PM, Thomas Bächler thomas@archlinux.org wrote:
Am 25.10.2011 17:51, schrieb Tom Gundersen:
This is what I had in mind: https://github.com/teg/mkinitcpio/commit/76dacf8b9de9cc0409741840b1d8e449862fc846.
Just two problems:
Good points.
- You overmount the initramfs' /usr for a short time. Nobody knows what
that might lead to.
1) This was a bit of a hack, need to look into doing it right (mounting with a "prefix").
- You mount /usr in read-write mode, this will conflict with fsck.
2) We should probably fsck bouth /usr and / before mounting them in the initramfs (doesn't really make sense to boot from a potentially broken disk, only to fsck it afterwards with a fsck binary stored on the disk itself...). Otherwise, we could just mount /usr ro and remount rw in rc.sysinit.
-t
Tom Gundersen wrote:
On Tue, Oct 25, 2011 at 2:12 PM, clemens fischer ... wrote:
(Removed email. No need to atract spammers!)
What patch would that be? THE-FAVOURITE-SEARCH-ENGINE didn't pull anything useful for "patch Thomas-Bächler busybox". Can somebody point us to the relevant code, please?
Thomas is right, that is not strictly speaking needed (but see below).
A hook to mount the users /usr would presumably go right before
"if [ -x /lib/udev/udevd ]; then"
in "/lib/initcpio/init"?
This is what I had in mind: https://github.com/teg/mkinitcpio/commit/76dacf8b9de9cc0409741840b1d8e449862fc846.
But what is this?
10 +In order for this to work, /usr needs to be in your /etc/fstab and it 11 +should be marked for not being fsck'ed (the last option should be 0).
This is getting weird. Who/what is going to fsck(8) /usr then?
To make it work nicely, we should also add the dual logic to shutdown: https://github.com/teg/mkinitcpio/commit/42610beb5a317d63dc76dabb72a9061a799b280b, which will pivot back to the initramfs and unmount /usr cleanly. This is where we need some busybox work, and I guess this patch should be discussed a bit more, maybe it is not the best way to do it.
Pardon me, but I'm not sure I want to keep all of the initramfs around after regular operation of the system commences.
Are the symlinks in "/dev/disk/by-label/" by then? I guess not, since udevd rules are responsible for setting them up, right?
I object to prohibit the use of LABEL/UUID to identify /usr (or any other fs for that matter) if that's what all this boils down to. I keep a backup and a rescue system around, with some filesystems shared between them. Sometimes I shuffle them around, eg. for testing alternatives like btrfs etc. For consistently identifying them they get labels, and I'm sure other people rely on labels as well.
So how do the boot-loaders do this? My first thought was to have a kernel command line parameter "mnt_usr_from=/dev/disk/by-label/my-usr" or whatever and then use busybox' ``mount "$mnt_usr_from" /usr''.
I do find it annoying to have /usr with all the bulky GUI crap, audio-tools and whatnot in "/". FreeBSD has a clean separation between what's needed to bring up, build and run a basic system ( -> /usr ) and user-toys, including all the GUI and multimedia stuff ( -> /usr/local ).
I always regret linux cramming everything into /usr. Some vital programs needed at startup are in /[s]bin, some in /lib, but look at the rules in /lib/udev/rules.d/: while vboxdrv and alsa-restore could equally well run when /usr is finally up, the device-mapper rules check for dmsetup residing in usr/sbin, although they are needed early!
It is currently a mess. I agree with the people advocating putting everything in /usr (and symlink /bin, /sbin and /lib for compatibility). That obviously requires being able to mount /usr from initramfs.
... which is more complicated than I thought. We should agree on a clean future-proof concept.
clemens
On Tue, Oct 25, 2011 at 8:09 PM, clemens fischer ino-news@spotteswoode.dnsalias.org wrote:
10 +In order for this to work, /usr needs to be in your /etc/fstab and it 11 +should be marked for not being fsck'ed (the last option should be 0).
This is getting weird. Who/what is going to fsck(8) /usr then?
This is work in progress. The comment is there to show that we need to sort it out before we push this to users (I only linked to it since you asked for the code...).
To make it work nicely, we should also add the dual logic to shutdown: https://github.com/teg/mkinitcpio/commit/42610beb5a317d63dc76dabb72a9061a799b280b, which will pivot back to the initramfs and unmount /usr cleanly. This is where we need some busybox work, and I guess this patch should be discussed a bit more, maybe it is not the best way to do it.
Pardon me, but I'm not sure I want to keep all of the initramfs around after regular operation of the system commences.
This seems to be the simplest and most robust way of doing it, so you'd need a better argument than "I don't want it". The only cost is a couple of megabytes of ram, which will be swapped out soon enough and not bother you until shutdown.
Are the symlinks in "/dev/disk/by-label/" by then? I guess not, since udevd rules are responsible for setting them up, right?
I object to prohibit the use of LABEL/UUID to identify /usr
This is not happening. Udev is included in the initramfs (if you enable the "udev" hook) and will set up all the links for you.
It is currently a mess. I agree with the people advocating putting everything in /usr (and symlink /bin, /sbin and /lib for compatibility). That obviously requires being able to mount /usr from initramfs.
... which is more complicated than I thought. We should agree on a clean future-proof concept.
I don't think it gets much simpler than "mount /usr from initramfs". We would no longer have to worry about which files may be needed during early boot or late shutdown and patch lots of packages to put the files in the right place. Instead the fix is done in only one package (also it is the same as done by other distros). Forget what I said about moving everything to /usr, this is not relevant to the current discussion.
I think we should take the rest of the discussion to the projects ML and get back when we have something more complete to show.
Cheers,
Tom
arch-general@lists.archlinux.org