On 2008-02-14, Jim Beard wrote:
> Anyone know what will cause the following?
>
> When running shutdown -r now or shutdown -h now, I get
> as the last set of messages scrolling by:
>
> Turning off swap
> Unmounting file systems: umount2: Device or resource busy
> umount: /usr: device is busy
> umount2: Device or resource busy
> umount: /usr: device is busy
> [FAILED]
> /usr: m
> INIT: no more processes left in this runlevel
>
>
>
> At at this point the system hangs. I can hit the
> warm reboot button or power down and restart, and
> all seems to boot and work fine. But an attempt to
> shutdown results in repetition of the above.
>
> FWIW, the occasion for the reboot initially was
> installation of
>
> kernel-desktop-devel-2.6.22.18-1mdv-1-1mdv2008.0
> kernel-desktop-2.6.22.18-1mdv-1-1mdv2008.0

Some process has a file open on /usr. All nonessential
processes should have been killed prior to the attempt to
u(n)mount file systems, but it seems Mandriva's init scripts
have order problems (especially with parallel init).

One thing that might help, if only to make things
deterministic, is to disable parallel init by adding the
nopinit option to the kernel boot options.

Checking the order of things in /etc/rc.d/rc[0-6].d may be
informative.

Two tools to see what processes have files open are fuser
and lsof.

HTH

--
Robert Riches
spamtrap42@verizon.net
(Yes, that is one of my email addresses.)

On Wed, 13 Feb 2008 21:59:39 -0500, Jim Beard wrote:

> Unmounting file systems: umount2: Device or resource busy
> umount: /usr: device is busy

Temporarily modify /etc/rc.d/init.d/halt. Add "lsof > /root/openfiles.txt",
after the hardware clock sync, before the unmounting of tmpfs. After
a reboot, check the /root/openfiles.txt, to see which programs are causing
the problem. Then you have to figure out why they are not terminating.

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

David W. Hodgins wrote:
> On Wed, 13 Feb 2008 21:59:39 -0500, Jim Beard wrote:
>
>> Unmounting file systems: umount2: Device or resource busy
>> umount: /usr: device is busy
>
> Temporarily modify /etc/rc.d/init.d/halt. Add "lsof > /root/openfiles.txt",
> after the hardware clock sync, before the unmounting of tmpfs. After
> a reboot, check the /root/openfiles.txt, to see which programs are causing
> the problem. Then you have to figure out why they are not terminating.

The following are the lines from openfiles.txt that involve
usr (the busy device). S01reboot is a symbolic link to
/etc/init.d/halt. Apart from presumably being needed, it is
in the same sequence as it was in 2007.1.

S01reboot 12005 root mem REG 8,22 15656 611760
/usr/lib64/libtermcap.so.2.0.8
S01reboot 12005 root mem REG 8,22 1364424 707786
/usr/lib64/libc-2.6.1.so
S01reboot 12005 root mem REG 8,22 17376 756750
/usr/lib64/libdl-2.6.1.so
S01reboot 12005 root mem REG 8,22 43440 822289
/usr/lib64/libnss_files-2.6.1.so


I am in over my head in reading /etc/init.d/halt, but it
looks to me like the script should kill everything except
kernel processes and session processes for the shell that
it is invoked in, and any disk not umounted should be
remounted read-only. This is not happening.

Any ideas about what is special about the above that it is
not getting killed?

Cheers!

jim b.

--
UNIX is not user-unfriendly; it merely
expects users to be computer-friendly.

On Thu, 14 Feb 2008 22:58:10 -0500, Jim Beard wrote:

> The following are the lines from openfiles.txt that involve
> usr (the busy device). S01reboot is a symbolic link to
> S01reboot 12005 root mem REG 8,22 15656 611760
> /usr/lib64/libtermcap.so.2.0.8

It looks like the halt script was written for i586, where the
library routines in use are in /lib, which must be on the
root device.

Try moving the in use libraries to /lib, and then running ldconfig.
If that fixes the problem, you should file a bug report.

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

It appears the problem was ldconfig loading
/usr/lib64 twice, and /etc/init.d/halt did not handle that well.

This may have implications for my bugreport 37670, as the symbol
not found is contained in a library in /usr/lib64.

David W. Hodgins wrote:
> On Thu, 14 Feb 2008 22:58:10 -0500, Jim Beard wrote:
>
>> The following are the lines from openfiles.txt that involve
>> usr (the busy device). S01reboot is a symbolic link to
>> S01reboot 12005 root mem REG 8,22 15656 611760
>> /usr/lib64/libtermcap.so.2.0.8
>
> It looks like the halt script was written for i586, where the
> library routines in use are in /lib, which must be on the
> root device.
>
> Try moving the in use libraries to /lib, and then running ldconfig.
> If that fixes the problem, you should file a bug report.

I was a little reluctant to try that, as the libraries exist in
both 32-bit and 64-bit versions, and both are used, depending on
the application running.

I ran ldconfig -v and wrote the output into junk, and a error
message complained that /usr/lib64 appeared twice.

So I started looking around, and found /etc/ld.so.conf.d/vmware.conf
which contained one line: /usr/lib64

ldconfig automatically loads /usr/lib64, and vmware.conf apparently
got it loaded a second time. I moved vmware.conf to
..vmware.conf to get it out of sight, ran ldconfig, rebooted,
ran ldconfig again, rebooted again, and reboot came off without
a hitch.

Moved the original /etc/init.d/halt script back into place, and
all works well.

But there are some odd features I ran across. The multiple instances
of the 64-bit libraries below are not only multiple copies, not
symlinks, but duplicates. I am not sure whether I should delete
the extras (e.g. delete /usr/lib64/libdl-2.6.1.so in favor of
/lib64/libdl-2.6.1.so), change the extras to symlinks, or leave
things as they are. Opinions?

[jim@jb etc]$ locate libtermcap.so.2.0.8
/lib64/libtermcap.so.2.0.8
/usr/lib64/libtermcap.so.2.0.8

[jim@jb etc]$ locate libc-2.6.1.so
/lib/libc-2.6.1.so
/lib64/libc-2.6.1.so
/usr/lib64/libc-2.6.1.so

[jim@jb etc]$ locate libdl-2.6.1.so
/lib/libdl-2.6.1.so
/lib64/libdl-2.6.1.so
/usr/lib64/libdl-2.6.1.so

[jim@jb etc]$ locate libnss_files-2.6.1.so
/lib/libnss_files-2.6.1.so
/lib64/libnss_files-2.6.1.so
/usr/lib64/libnss_files-2.6.1.so
/var/spool/postfix/lib64/libnss_files-2.6.1.so

Cheers!

jim b.

--
UNIX is not user-unfriendly; it merely
expects users to be computer-friendly.

On Fri, 15 Feb 2008 22:32:24 -0500, Jim Beard wrote:

> I ran ldconfig -v and wrote the output into junk, and a error
> message complained that /usr/lib64 appeared twice.

It doesn't matter how many times a path is included in the order
defined in /etc/ld.so.conf, and included from there ld.so.conf.d/*.conf.

What does matter, is the order.

> [jim@jb etc]$ locate libtermcap.so.2.0.8
> /lib64/libtermcap.so.2.0.8
> /usr/lib64/libtermcap.so.2.0.8

Change /etc/ld.so.conf. Add /lib and /lib64 before the include, or
any other paths.

Regards, Dave Hodgins

--
Change nomail.afraid.org to ody.ca to reply by email.
(nomail.afraid.org has been set up specifically for
use in usenet. Feel free to use it yourself.)

David W. Hodgins wrote:
> On Fri, 15 Feb 2008 22:32:24 -0500, Jim Beard wrote:
>
>> I ran ldconfig -v and wrote the output into junk, and a error
>> message complained that /usr/lib64 appeared twice.
>
> It doesn't matter how many times a path is included in the order
> defined in /etc/ld.so.conf, and included from there ld.so.conf.d/*.conf.
>
> What does matter, is the order.
>
>> [jim@jb etc]$ locate libtermcap.so.2.0.8
>> /lib64/libtermcap.so.2.0.8
>> /usr/lib64/libtermcap.so.2.0.8
>
> Change /etc/ld.so.conf. Add /lib and /lib64 before the include, or
> any other paths.

/lib and /lib64 should be loaded automatically at the very start,
and this seems to be happening. The problem was the second
invocation of /usr/lib64, so just deleting that should suffice.

Cheers!

jim b.

--
UNIX is not user-unfriendly; it merely
expects users to be computer-friendly.