No crypto in firefox

I just dug out a laptop that I hadn't used for several months and ran
portsnap/portupgrade on it. Afterwards, I had no crypto in firefox!
If I tried go to some https: page, I would get a response "Unexpected
response from server / Firefox doesn't know how to communicate with
the server" and suggesting I check to make sure my system has the
Personal Security Manager installed. Now I don't know how to do that
check, but if I go to preferences, everything crypto related seems to
be gone: There are no certificates listed, password manager doesn't
list any stored passwords, and I have no security devices.

So I figured big deal, something went really bad with the portupgrade,
so I started a

portupgrade -fR firefox --batch

and went to bed. This morning, firefox worked like it should, after a
mere 13442 seconds of wall clock time spent compiling stuff.

But now, after a shutdown, it behaves like before: No crypto.

My current hypothesis is that this is somehow ldconfig related: Maybe
firefox links to the wrong libraries somehow? I do not specify any
ldconfig related stuff in /etc/rc.config, just using what's in
/etc/default/rc.config:

ldconfig_paths="/usr/lib/compat /usr/X11R6/lib /usr/local/lib /usr/local/lib/compat/pkg"

Any idea how I may proceed to figure out what is going on?
Or alternative hypotheses as to the reason for this?
Or a better place to ask than here?

I am totally stumped on this one.

--
* Harald Hanche-Olsen <URL:http://www.math.ntnu.no/~hanche/>
- It is undesirable to believe a proposition
when there is no ground whatsoever for supposing it is true.
-- Bertrand Russell

Harald Hanche-Olsen wrote:
[color=blue]
> I just dug out a laptop that I hadn't used for several months and ran
> portsnap/portupgrade on it. Afterwards, I had no crypto in firefox!
> If I tried go to some https: page, I would get a response "Unexpected
> response from server / Firefox doesn't know how to communicate with
> the server" and suggesting I check to make sure my system has the
> Personal Security Manager installed. Now I don't know how to do that
> check, but if I go to preferences, everything crypto related seems to
> be gone: There are no certificates listed, password manager doesn't
> list any stored passwords, and I have no security devices.
>
> So I figured big deal, something went really bad with the portupgrade,
> so I started a
>
> portupgrade -fR firefox --batch
>
> and went to bed. This morning, firefox worked like it should, after a
> mere 13442 seconds of wall clock time spent compiling stuff.
>
> But now, after a shutdown, it behaves like before: No crypto.
>
> My current hypothesis is that this is somehow ldconfig related: Maybe
> firefox links to the wrong libraries somehow? I do not specify any
> ldconfig related stuff in /etc/rc.config, just using what's in
> /etc/default/rc.config:
>
> ldconfig_paths="/usr/lib/compat /usr/X11R6/lib /usr/local/lib
> /usr/local/lib/compat/pkg"
>
> Any idea how I may proceed to figure out what is going on?
> Or alternative hypotheses as to the reason for this?
> Or a better place to ask than here?
>
> I am totally stumped on this one.
>[/color]

See if you have /usr/ports/security/nss/ installed. If not, install this and
rebuild Firefox. It's a dependency and should have been included
automagically.

Also, you can check what libraries an app has been built against with the
ldd command: ldd <firefox-executable-binary (not the startup script) file>

-Jason

Well, to make a long story short, after messing around with ktrace and
lsof I found a workaround: I symlinked everything in
/usr/local/lib/nss/ to /usr/local/lib/, and now things work once more.

As to why this should be necessary, well I don't know ...

(If you're morbidly curious and/or think you can provide a clue as to
what is happening and why, read on. Otherwise, you may as well stop
here.)

+ Jason Bourne <j_bourne_treadstone@hotmail.com>:
[color=blue]
> See if you have /usr/ports/security/nss/ installed. If not, install
> this and rebuild Firefox. It's a dependency and should have been
> included automagically.[/color]

It's already there.
[color=blue]
> Also, you can check what libraries an app has been built against
> with the ldd command: ldd <firefox-executable-binary (not the
> startup script) file>[/color]

Yeah, and I get a long list, which is identical on the machine at work
(where firefox works as it should) and on the laptop (where it
doesn't) apart from the hex numbers at the end of each line.

No nss libs are listed, though. Apparently, firefox loads them on its
own: I can use lsof on the machine where it works to see that firefox
has linked in various libraries from /usr/local/lib/nss/, but not so
on the laptop. (The libraries are all present on the machine.)

So I start playing with ktrace. I see firefox looking for and
finding various libraries in /usr/local/lib/nss/, including
libnss3.so.1. But then it searches for libnss3.so.1 /again/. On the
good machine it finds it, but doesn't open it. This happens twice.
On the bad machine it doesn't find it, as it for some reason skips the
directory where it has found it before. Then follow four rapid
mumnap() calls, which seem to unmap various library files in
/usr/local/lib/nss/, I guess as a result of not being able to find
that library file again.

Recall that firefox worked fine after the rebuild, before I rebooted.
All I did was reboot, and it no longer works.

Here is where it looks for libnss3.so.1 the first time, when it does
find it:

/usr/local/lib/firefox/libnss3.so.1
/usr/local/lib/firefox/plugins/libnss3.so.1
/usr/local/lib/browser_plugins/libnss3.so.1
/usr/local/lib/browser_linux_plugins/libnss3.so.1
/usr/local/lib/firefox/libnss3.so.1
/usr/local/lib/firefox/libnss3.so.1
/usr/local/lib/nss/libnss3.so.1 (success!)

And here is where it looks the second time:

/usr/local/lib/firefox/libnss3.so.1
/usr/local/lib/firefox/plugins/libnss3.so.1
/usr/local/lib/browser_plugins/libnss3.so.1
/usr/local/lib/browser_linux_plugins/libnss3.so.1
/usr/local/lib/firefox/libnss3.so.1
(here it skips trying /usr/local/lib/nss/ and scans the directories
listed in /var/run/ld-elf.so.hints)
/lib/libnss3.so.1
/usr/lib/libnss3.so.1
/usr/lib/compat/libnss3.so.1
/usr/X11R6/lib/libnss3.so.1
/usr/local/lib/libnss3.so.1
/usr/local/lib/compat/pkg/libnss3.so.1
(and then these two in what I assume is an act of desperation)
/lib/libnss3.so.1
/usr/lib/libnss3.so.1

Oh, well. Way past bedtime agin.

--
* Harald Hanche-Olsen <URL:http://www.math.ntnu.no/~hanche/>
- It is undesirable to believe a proposition
when there is no ground whatsoever for supposing it is true.
-- Bertrand Russell

+ Harald Hanche-Olsen <hanche@math.ntnu.no>:
[color=blue]
> Well, to make a long story short, after messing around with ktrace and
> lsof I found a workaround: I symlinked everything in
> /usr/local/lib/nss/ to /usr/local/lib/, and now things work once more.
>
> As to why this should be necessary, well I don't know ...[/color]

But I just had a little "duh!" moment here: As I have explained
before, my recompiled firefox works fine on my office machine, while
it stopped working on my home machine after rebooting. (My office
machine has been up for 193 days ...) So I looked in
/var/run/ld-elf.so.hints on the office machine, and it does indeed
list /usr/local/lib/nss/. Surely, that will go away when I reboot,
and then firefox will stop working again.

Does the ports system run ldconfig without making provisions for the
same to happen at boot? Nah, that would be too dumb. Must be
something I messed up in my configuration. But what? Where?
(I have not touched ldconfig_paths in /etc/rc.conf.)

--
* Harald Hanche-Olsen <URL:http://www.math.ntnu.no/~hanche/>
- It is undesirable to believe a proposition
when there is no ground whatsoever for supposing it is true.
-- Bertrand Russell

On Tue, 15 Jan 2008 12:18:59 +0100, Harald Hanche-Olsen wrote:
[color=blue]
> But I just had a little "duh!" moment here: As I have explained
> before, my recompiled firefox works fine on my office machine, while
> it stopped working on my home machine after rebooting. (My office
> machine has been up for 193 days ...) So I looked in
> /var/run/ld-elf.so.hints on the office machine, and it does indeed
> list /usr/local/lib/nss/. Surely, that will go away when I reboot,
> and then firefox will stop working again.[/color]
[color=blue]
> Does the ports system run ldconfig without making provisions for the
> same to happen at boot? Nah, that would be too dumb. Must be
> something I messed up in my configuration. But what? Where?
> (I have not touched ldconfig_paths in /etc/rc.conf.)[/color]

Yes, it does! You should find a file /usr/local/libdata/ldconfig/nss

cat /usr/local/libdata/ldconfig/nss
/usr/local/lib/nss

(if it has been erased on your machine for some reason, just re-create
it)

This will be read by /etc/rc.d/ldconfig.
--
Th. Thomas.

+ Thierry Thomas <tthomas@mail.dotcom.fr>:
[color=blue]
> On Tue, 15 Jan 2008 12:18:59 +0100, Harald Hanche-Olsen wrote:
>[color=green]
>> Does the ports system run ldconfig without making provisions for the
>> same to happen at boot? Nah, that would be too dumb. Must be
>> something I messed up in my configuration. But what? Where?
>> (I have not touched ldconfig_paths in /etc/rc.conf.)[/color]
>
> Yes, it does! You should find a file /usr/local/libdata/ldconfig/nss
>
> cat /usr/local/libdata/ldconfig/nss
> /usr/local/lib/nss[/color]

Yes, it is present and contains what you say.
[color=blue]
> This will be read by /etc/rc.d/ldconfig.[/color]

It doesn't look like it did:

; strings /var/run/ld-elf.so.hints
Ehnt
/lib:/usr/lib:/usr/lib/compat:/usr/X11R6/lib:/usr/local/lib:/usr/local/lib/compat/pkg

And reading /etc/rc.d/ldconfig I don't quite understand how that is
supposed to happen. Where is the magic, that I may find out why it
doesn't work?

Let me add that I am running 6.2-RELEASE, via a source upgrade done
quite a long time ago from 5.something, and /etc/rc.d/ldconfig is

# $FreeBSD: src/etc/rc.d/ldconfig,v 1.14 2005/01/16 08:34:30 obrien Exp $

which looks kind of old (?).

--
* Harald Hanche-Olsen <URL:http://www.math.ntnu.no/~hanche/>
- It is undesirable to believe a proposition
when there is no ground whatsoever for supposing it is true.
-- Bertrand Russell

On Wed, 16 Jan 2008 22:46:29 +0100, Harald Hanche-Olsen wrote:
[color=blue]
> And reading /etc/rc.d/ldconfig I don't quite understand how that is
> supposed to happen. Where is the magic, that I may find out why it
> doesn't work?[/color]
[color=blue]
> Let me add that I am running 6.2-RELEASE, via a source upgrade done
> quite a long time ago from 5.something, and /etc/rc.d/ldconfig is[/color]
[color=blue]
> # $FreeBSD: src/etc/rc.d/ldconfig,v 1.14 2005/01/16 08:34:30 obrien Exp $[/color]
[color=blue]
> which looks kind of old (?).[/color]

Did you run mergemaster? I have no more 6.2, and I've not checked the
CVS, but on 6.3-PRERELEASE I have this one:

ident /etc/rc.d/ldconfig
/etc/rc.d/ldconfig:
$NetBSD: ldconfig,v 1.5 2002/03/22 04:33:58 thorpej Exp $
$FreeBSD: src/etc/rc.d/ldconfig,v 1.14.2.3 2007/10/25 14:36:11 mtm Exp $

The part that should interest you is these lines:

for i in ${ldconfig_local_dirs}; do
if [ -d "${i}" ]; then
_files=`find ${i} -type f`
if [ -n "${_files}" ]; then
ldconfig_paths="${ldconfig_paths} `cat ${_files} | sort -u`"
fi
fi
done

By default (in /etc/defaults/rc.conf), ldconfig_local_dirs is set to
/usr/local/libdata/ldconfig.
--
Th. Thomas.

+ Thierry Thomas <tthomas@mail.dotcom.fr>:
[color=blue]
> Did you run mergemaster?[/color]

Yes, all according to the upgrade instructions. But I must have done
it wrong, for none of the stuff you mention is in my files:

/etc/rc.d/ldconfig never mentions ldconfig_local_dirs and neither does
/etc/defaults/rc.conf.

It's been a while now, but as I recall, mergemaster asks a /lot/ of
questions, and it seems easy to answer them wrong. (A situation akin
to pkgdb. I always look forward to any pgkdb -F run with dread,
because I don't really understand what its questions mean.)

Ah, I see now that /usr/src/etc/ has much newer versions of these
files. I should use them to replace the old moldy ones, then. With
care presumably, but surely it must be done.

Thanks for pointing me in the right direction. I could probably have
been able to figure this one out eventually, but surely you saved me
some time.

--
* Harald Hanche-Olsen <URL:http://www.math.ntnu.no/~hanche/>
- It is undesirable to believe a proposition
when there is no ground whatsoever for supposing it is true.
-- Bertrand Russell

On Thu, 17 Jan 2008 23:32:52 +0100, Harald Hanche-Olsen wrote:
[color=blue]
> It's been a while now, but as I recall, mergemaster asks a /lot/ of
> questions, and it seems easy to answer them wrong. (A situation akin
> to pkgdb. I always look forward to any pgkdb -F run with dread,
> because I don't really understand what its questions mean.)[/color]

Check mergemaster's options -i and -U to eliminate a lot of superfluous
questions...
--
Th. Thomas.

Harald Hanche-Olsen <hanche@math.ntnu.no> writes:
[color=blue]
> + Thierry Thomas <tthomas@mail.dotcom.fr>:
>[color=green]
> > Did you run mergemaster?[/color]
>
> Yes, all according to the upgrade instructions. But I must have done
> it wrong, for none of the stuff you mention is in my files:
>
> /etc/rc.d/ldconfig never mentions ldconfig_local_dirs and neither does
> /etc/defaults/rc.conf.
>
> It's been a while now, but as I recall, mergemaster asks a /lot/ of
> questions, and it seems easy to answer them wrong. (A situation akin
> to pkgdb. I always look forward to any pgkdb -F run with dread,
> because I don't really understand what its questions mean.)[/color]

What can make it easier is commenting with your initials and date
every change to the config files. When mergemaster shows differences,
it makes it much clearer what are changes that you made and may need
to be carried over vs. what's a change in the standard files that can
probably be accepted in the new version.

I don't use the editor built-in to mergemaster at all. If I'm in
doubt about the changes, I leave the old version and merge them by
hand with my favorite editor.

-- Patrick