And I'm still trying to find out why my FireWire/FFADO (MOTU UltraLite) works on Fedora and not on Arch... Arch udev mysteries...

On Tue, Jul 19, 2011 at 05:15:46PM +0200, Tom Gundersen wrote:

On Tue, Jul 19, 2011 at 4:37 PM, Fons Adriaensen wrote:

...

After a routine upgrade it seems I don't have any sound devices any more.

More in detail:

* The driver for my card (RME HDSP MADI, snd_hdpsm) is loaded. * aplay -L tells me there is only the 'null' device. * (re)starting /etc/rc.d/alsa doesn't change things. * There is *no* /dev/pcm at all. * /sys/class/sound only has a link to a timer device. * No traces in dmesg or /var/log/messages of anything going wrong.

I suspect udev fails to do the right things.

* /etc/udev/rules.d is empty * /etc/udev/udev.conf contains only the line  udev_log="err"

This is a professional studio and I'm expecting clients tomorrow at 09:00.

You could try udev from testing (I agree that probably udev is the culprit, though it has not been updated in core for some time).

But the whole boot system seems to have changed - this machine boots incredibly fast now (not that it matters, it's rebooted at most once in a week). Some new info: I get complaints from udev in /var/log/errors.log about files not being found in /lib/udev/rules.d. This would explain things I guess. But when I look in /lib/udev/rules.d those files *do* exist. So maybe things go a bit *too* fast (or out of order). If that is the case, what can be done about it ? I have to do a fresh install on 5 new machines (similar HW) later this week. I guess they will fail as well... P.S. I also tried udev .172 from AUR, same result. -- FA

On Tue, Jul 19, 2011 at 06:59:46PM +0200, Tom Gundersen wrote:

...

Some new info: I get complaints from udev in /var/log/errors.log about files not being found in /lib/udev/rules.d. This would explain things I guess. But when I look in /lib/udev/rules.d those files *do* exist. So maybe things go a bit *too* fast (or out of order). If that is the case, what can be done about it ?

Could you attach your fstab and the relevant parts of your logs ("grep udev /var/log/errors.log")?

I'm at a different site ATM, so this will have to wait. But the errors are all about .rules files not being found (which one seems to be different each time), and the fstab would be almost identical to what I have here: # # /etc/fstab: static file system information # # <file system> <dir> <type> <options> <dump> <pass> none /dev/pts devpts defaults 0 0 none /dev/shm tmpfs defaults 0 0 /dev/cdrom /media/cd auto ro,user,noauto,unhide 0 0 /dev/dvd /media/dvd auto ro,user,noauto,unhide 0 0 /dev/sda3 / ext3 defaults 0 1 UUID=0abe80fe-78a0-4d8a-bb1e-1ea78f64f8d7 /boot ext2 defaults 0 1 UUID=e4bde8a3-1e79-4217-836b-c07ad8cc409b swap swap defaults 0 0 UUID=f5e985c2-45e2-4e1a-a0b3-a07dfab18206 /audio ext2 defaults 0 1 UUID=fc21faea-7cac-4132-9c7e-01201c1bf59e /data ext3 defaults 0 1 # Allowed USB pens etc. LABEL=/pp1 /usb/pp1 ext2 noauto,user 0 0 LABEL=/pp2 /usb/pp2 ext2 noauto,user 0 0 LABEL=/mt1 /usb/mt1 ext2 noauto,user 0 0 LABEL=/cr1 /usb/cr1 ext2 noauto,user 0 0 LABEL=/sony1 /usb/so1 ext2 noauto,user 0 0 LABEL=/sony2 /usb/so2 ext2 noauto,user 0 0 # Backup on nfs DLINK-6B7BF3:/mnt/HD_a2 /mnt/backup/v1 nfs noauto,user,rsize=32768,wsize=32768 0 0 DLINK-6B7BF3:/mnt/HD_b2 /mnt/backup/v2 nfs noauto,user,rsize=32768,wsize=32768 0 0 Note: /audio is on /dev/sdb, the others on /dev/sda -- FA

On Wed, Jul 20, 2011 at 02:38:58AM +0800, Ray Rashif wrote:

...

* There is *no* /dev/pcm at all.

Do you mean /dev/snd/pcm? Either way, NOT good.

Yes.

...

* /etc/udev/rules.d is empty

This is expected; have you ever placed anything there yourself before?

No.

Do you know about the the rc.conf changes WRT (1) blacklisting modules and (2) time?

Yes.

Or, was the last time you updated before this, quite a while back?

According to pacman.log it seems the last full update was in March.

I understand getting the RME to work initially was no trouble, and just the standard alsa installation (and the driver gets loaded automatically)?

Yes, never had any problem with it. It seems I have *two* machines having this problem, both updated yesterday. I didn't test the audio on the first one as everything seemed to be OK. The second one was actually updated using pacserve, with only a few packages coming from the mirror, the rest being provided by the first updated machine. Also this went so smoothly that I forgot to mention it. Today, on both machines, the 'missing file' errors from udev have disappeared. But you'll find yesterday's ones, rc.conf, pacman.log and fstab here: http://kokkinizita.linuxaudio.org/archinfo.tar.bz2. Use wget, there are no links to this. Ciao, -- FA

On 20 July 2011 21:11, Fons Adriaensen wrote:

It seems I have *two* machines having this problem, both updated yesterday. I didn't test the audio on the first one as everything seemed to be OK. The second one was actually updated using pacserve, with only a few packages coming from the mirror, the rest being provided by the first updated machine. Also this went so smoothly that I forgot to mention it.

Today, on both machines, the 'missing file' errors from udev have disappeared. But you'll find yesterday's ones, rc.conf, pacman.log and fstab here: http://kokkinizita.linuxaudio.org/archinfo.tar.bz2. Use wget, there are no links to this.

Here are the most significant packages: [2011-07-19 15:47] upgraded pciutils (3.1.7-3 -> 3.1.7-4) [2011-07-19 15:47] upgraded udev (166-2 -> 171-2) [2011-07-19 15:47] warning: /etc/inittab installed as /etc/inittab.pacnew [2011-07-19 15:47] warning: /etc/rc.conf installed as /etc/rc.conf.pacnew [2011-07-19 15:47] Blacklisting of modules is no longer supported in rc.conf, [2011-07-19 15:47] please add blacklist entries to /etc/modprobe.d/ instead. [2011-07-19 15:47] upgraded initscripts (2011.02.1-1 -> 2011.06.4-1) [2011-07-19 15:47] upgraded linux-firmware (20110227-1 -> 20110512-2) [2011-07-19 15:47] upgraded mkinitcpio-busybox (1.18.2-1 -> 1.18.4-1) [2011-07-19 15:47] upgraded mkinitcpio (0.6.8-2 -> 0.7.2-1) [2011-07-19 15:48] upgraded kernel26 (2.6.37.4-1 -> 2.6.39.3-1) [2011-07-19 15:48] upgraded kernel26-headers (2.6.37.4-1 -> 2.6.39.3-1) I see nothing unusual in rc.conf. Please downgrade these packages [1] and test. You can do this one by one, starting with the kernel + mkinitcpio + linux-firmware, then test, then if no difference, go on to downgrading the rest in that list. Of particular interest is initscripts. Watch out what happens when you downgrade initscripts (and if possible, restore previous rc.conf, inittab). A lot of changes have come in during the last few months, so the cause here is not going to be obvious. We only know that it is either udev (nothing to determine it is fully responsible), init or kernel related. Whether something else is involved, needs further troubleshooting after the downgrade test. * Also, provide us with a package list comparison between the 2 affected machines, and a third, unaffected one. You may just provide the full package lists produced with pacman -Q. * Provide us a list of modules with the older packages and one with newer packages (produced by lsmod) immediately after boot-up. You will need to reboot twice or more for this. [1] https://wiki.archlinux.org/index.php/Downgrading_Packages -- GPG/PGP ID: 8AADBB10

On Wed, Jul 20, 2011 at 10:15:32PM +0800, Ray Rashif wrote:

Please downgrade these packages [1] and test. You can do this one by one, starting with the kernel + mkinitcpio + linux-firmware, then test, then if no difference, go on to downgrading the rest in that list. Of particular interest is initscripts. Watch out what happens when you downgrade initscripts (and if possible, restore previous rc.conf, inittab).

OK I tried: 1. Downgrading firmware, kernel, mkinitcpio: --> no improvement. 2. Downgrading initscripts: --> no improvement. 3. Downgrading udev: --> no improvement. 4. Downgrading pciutils: --> no improvement. 5. Upgrading everything again: --> still the same result. In all cases downgrading was to the previously used version, for initscripts rc.conf was reverted as well. -- FA

On Wed, Jul 20, 2011 at 5:58 PM, Fons Adriaensen wrote:

When I downgrade initscripts and mkinitcpio (to versions of around february), run mkinitcpio -p kernel26, and reboot, shouldn't I get the 'old style' boot messages (without the timing in []) ?

But I don't...

Are we forgetting something ?

That's decided by the kernel. Could you try downgrading your kernel too? Possibly install the kernel26-lts? -t

On Wed, Jul 20, 2011 at 06:04:00PM +0200, Tom Gundersen wrote:

On Wed, Jul 20, 2011 at 5:58 PM, Fons Adriaensen wrote:

...

When I downgrade initscripts and mkinitcpio (to versions of around february), run mkinitcpio -p kernel26, and reboot, shouldn't I get the 'old style' boot messages (without the timing in []) ?

But I don't...

Are we forgetting something ?

That's decided by the kernel. Could you try downgrading your kernel too? Possibly install the kernel26-lts?

Are you sure ? Even with lts I get the new format. And it doesn't help. -- FA

On Thu, Jul 21, 2011 at 12:23:47AM +0800, Ray Rashif wrote:

Just to be double sure. Did you reboot each time you downgraded something? If not, downgrade all of them at one go, then reboot.

Of course. mkinitcpio -p kernel26 and reboot.

1. Now we really need a comparison between your working systems and non-working systems, looking at the differences between hardware and software.

Since both systems worked perfectly until the upgrade yesterday, that difference is the set of packages that were upgraded yesterday, wich you find in pacman.log.

Just thinking out loud:

A diff with the previous udev yields no difference in 78-sound-card.rules, only an ammendment of our system ruleset.

-# SOUND addon modules -SUBSYSTEM=="sound", RUN+="/lib/udev/load-modules.sh snd-pcm-oss" -SUBSYSTEM=="sound", RUN+="/lib/udev/load-modules.sh snd-seq-oss"

-# miscellaneous -KERNEL=="rtc|rtc0", GROUP="audio", MODE="0664"

And 50-udev-default.rules.

# sound -SUBSYSTEM=="sound", GROUP="audio" +SUBSYSTEM=="sound", GROUP="audio", \ + OPTIONS+="static_node=snd/seq", OPTIONS+="static_node=snd/timer"

2. At this point I'd recommend using a Fedora Live medium to test if your sound works there.

I'm still unsure whether this is an upstream or local problem. Tom would be a better judge on this, especially regarding the 'old style' boot messages. If it's not the kernel, not alsa, not udev, not init, then I don't know what else it could be (RME card/PCI bus?).

*Two cards* going defective at the same time, and that time coincides with an upgrade ? That could only mean that the new system has destroyed them :-) Ciao, -- FA

On Thu, Jul 21, 2011 at 12:50:20AM +0800, Ray Rashif wrote:

On 21 July 2011 00:38, Fons Adriaensen wrote:

...

*Two cards* going defective at the same time, and that time coincides with an upgrade ? That could only mean that the new system has destroyed them :-)

Yes, but the obvious culprits have been tried. What's left is to figure out what else could cause this.

I'm not at the studio ATM so I can't try, but what about the sysvinit package ? It provides /sbin/init, and it was not in your list of 'usual suspects'. And I'm still trying to find out why after reverting kernel, mkinitcpio, udev, etc. to versions known to work I still get those 'new style' boot messages - indicating that something else should be reverted as well to return to the working configuration. -- FA

On Thu, Jul 21, 2011 at 03:52:04PM +0800, Ray Rashif wrote:

On 21 July 2011 15:19, Fons Adriaensen wrote:

...

I'm not at the studio ATM so I can't try, but what about the sysvinit package ? It provides /sbin/init, and it was not in your list of 'usual suspects'.

The last time that was updated was in Oct 2010 [1].

...

to find out why after reverting kernel, mkinitcpio, udev, etc. to versions known to work I still get those 'new style' boot messages - indicating that something else should be reverted as well to return to the working configuration.

How are these "new style" boot messages actually?

Each line starts with the time (since boot) in seconds and microseconds, like e.g. [ 3.141592] -- FA

On Wed, Jul 20, 2011 at 06:50:32PM +0200, Tom Gundersen wrote:

"modprobe -rv snd_hdpsm && modprobe -v snd_hdpsm", and then post the output to the console and any dmesg output (and verify that your devices are not in /dev/snd)?

This becomes interesting: [root@c252 fons]# modprobe -rv snd_hdspm rmmod /lib/modules/2.6.39-ARCH/kernel/sound/pci/rme9652/snd-hdspm.ko.gz rmmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-rawmidi.ko.gz rmmod /lib/modules/2.6.39-ARCH/kernel/sound/core/seq/snd-seq-device.ko.gz rmmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-pcm.ko.gz rmmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-timer.ko.gz rmmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-page-alloc.ko.gz rmmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-hwdep.ko.gz rmmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd.ko.gz rmmod /lib/modules/2.6.39-ARCH/kernel/sound/soundcore.ko.gz [root@c252 fons]# modprobe -v snd_hdspm insmod /lib/modules/2.6.39-ARCH/kernel/sound/soundcore.ko.gz insmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd.ko.gz insmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-hwdep.ko.gz insmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-page-alloc.ko.gz insmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-timer.ko.gz insmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-pcm.ko.gz insmod /lib/modules/2.6.39-ARCH/kernel/sound/core/seq/snd-seq-device.ko.gz insmod /lib/modules/2.6.39-ARCH/kernel/sound/core/snd-rawmidi.ko.gz insmod /lib/modules/2.6.39-ARCH/kernel/sound/pci/rme9652/snd-hdspm.ko.gz [root@c252 fons]# dmesg | tail [ 12.447653] hrtimer: interrupt took 2962 ns [ 13.231271] CE: hpet2 increased min_delta_ns to 7500 nsec [ 13.231277] CE: hpet2 increased min_delta_ns to 11250 nsec [ 13.962310] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx [ 13.962463] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 14.346672] generic-usb 0003:05AC:9232.0002: hiddev0,hidraw1: USB HID v1.11 Device [Apple Computer, Inc. Apple Cinema HD Display] on usb-0000:00:1d.0-1.4.2/input0 [ 19.405476] CE: hpet6 increased min_delta_ns to 7500 nsec [ 19.405482] CE: hpet6 increased min_delta_ns to 11250 nsec [ 24.736440] eth0: no IPv6 routers present [ 48.057757] HDSPM: unknown firmware revision cc Clearly the firmware revision hasn't changed... So either it is not read correctly, or the driver has changed. -- FA

On Thu, Jul 21, 2011 at 12:21:23PM +0200, Tom Gundersen wrote:

...

[   48.057757] HDSPM: unknown firmware revision cc

Clearly the firmware revision hasn't changed... So either it is not read correctly, or the driver has changed.

Where do you get your firmware from? Is it in the kernel26-firmware package, or from a third party? Maybe you need to downgrade the firmware package to make things work.

Also, could you please verify this with an earlier kernel? The commit [0] aborting on unknown firware is recent, so maybe earlier kernels give a better error message.

AFAIK the HDSP Madi does not require a firmware download, the firmware is in an EEPROM on the card (a Xilinx FPGA configuration actually). The firmware revision is used to identify the card type (MADI, AES, Raydat, etc.). Some more news: yesterday when downgrading various packages I only looked at the output of 'aplay -L' to check if things were OK, not at /dev/snd. Today I tried again downgrading the kernel to 2.6.37, to check if the firmware revision error would occur. It doesn't. And: * aplay -L still doesn't show any devices, * but they DO exist in /dev/snd, all those I expect for this card, * but there is no ALSA sequencer device (/dev/snd/seq). And the audio does actually work. Tentative conclusion: the snd_hdspm driver in 2.6.39 is not OK. Ciao, -- FA

On Thu, Jul 21, 2011 at 1:26 PM, Kacper Żuk wrote:

2011/7/21 Fons Adriaensen :

...

* but there is no ALSA sequencer device (/dev/snd/seq).

I think that for sequencer you need to: modprobe snd-seq

This should not be necessary. There was a bug to do with this, but with the newest udev (from testing), this should have been resolved. The static device should be created automatically and the module loaded on demand. That said, do you get the seq device when you manually modprobe? -t

Adrian, Takashi: I added you to cc, as you last touched this code, sorry if I should have cc'ed someone else. It appears that the HDSPM module does not recognise cards with firmware revision 0xCC: On Thu, Jul 21, 2011 at 1:23 PM, Fons Adriaensen wrote:

...

...

...

...

* The driver for my card (RME HDSP MADI, snd_hdpsm) is loaded. [   48.057757] HDSPM: unknown firmware revision cc [...] Today I tried again downgrading the kernel to 2.6.37, to check if the firmware revision error would occur. It doesn't. [...] Tentative conclusion: the snd_hdspm driver in 2.6.39 is not OK.

Ah, I see. Thanks for the clarification. This means that the kernel is not able to recognise your card. Pre-.39 it would load anyway, but since .39 it gives the warning you posted and refuses to load. Your firmware revision is 0xCC, which is not recognised. If your card is MADI, the kernel expects the firmware revision to be 0xD2 (or starting in 3.0, it also accepts 0xCF). I guess this can be fixed easily by adding your firmware revision to the list of accepted revs. Adrian: is any more info needed to add this rev? Cheers, Tom