long wait in boot sequence, shorewall issue? - Mandriva

This is a discussion on long wait in boot sequence, shorewall issue? - Mandriva ; Mandriva 2007.1, vanilla 2.6.21.1 kernel. I'm sometimes seeing a really long wait (2.5 minutes, roughly) on reboot for the system to start up fully. Other times it boots quickly. The problem seems to happen after (during?) shorewall startup, as the ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: long wait in boot sequence, shorewall issue?

  1. long wait in boot sequence, shorewall issue?

    Mandriva 2007.1, vanilla 2.6.21.1 kernel.

    I'm sometimes seeing a really long wait (2.5 minutes, roughly)
    on reboot for the system to start up fully. Other times it boots
    quickly. The problem seems to happen after (during?) shorewall
    startup, as the last messages on the screen while it's pausing are all
    from shorewall. Here are some examples from /var/log/messages files:

    Jun 22 14:54:04 saf03 kernel: ClusterIP Version 0.8 loaded successfully
    Jun 22 14:54:05 saf03 avahi-daemon[2931]: Service "SFTP File Transfer on
    saf03" (/etc/avahi/services/sftp-ssh.service) successfully established.
    Jun 22 14:54:05 saf03 avahi-daemon[2931]: Service "Remote Access on
    saf03" (/etc/avahi/services/openssh.service) successfully established.
    Jun 22 14:54:07 saf03 logger: Shorewall started
    Jun 22 14:56:54 saf03 sshd[4314]: Server listening on :: port 22.
    Jun 22 14:56:54 saf03 sshd[4314]: error: Bind to port 22 on 0.0.0.0
    failed: Address already in use.


    notice the pause before sshd shows up. (Ignore the sshd error, it does
    that no matter what.)

    Here's another one from a clone of the first machine. Again, notice the
    long wait after the "shorewall started" message, this time crond comes
    up next, as sshd started before shorewall.

    Jun 22 14:57:51 saf02 avahi-daemon[2961]: Service "SFTP File Transfer on
    saf02" (/etc/avahi/services/sftp-ssh.service) successfully established.
    Jun 22 14:57:51 saf02 avahi-daemon[2961]: Service "Remote Access on
    saf02" (/etc/avahi/services/openssh.service) successfully established.
    Jun 22 14:57:53 saf02 logger: Shorewall started
    Jun 22 15:01:01 saf02 crond[4361]: (root) CMD (nice -n 19 run-parts
    --report /etc/cron.hourly)
    Jun 22 15:01:02 saf02 msec: changed mode of /var/log/wtmp from 664 to 640
    Jun 22 15:01:02 saf02 msec: changed group of /var/log/wtmp from utmp to adm

    Anybody care to guess where the delay is? Naturally it doesn't do it
    every time. There's not a whole lot of information here to work
    with. It looks like something is stalling the boot sequence but
    not being kind enough to leave a message to indicate the nature of the
    problem.

    Thanks,

    David Mathog

  2. Re: long wait in boot sequence, shorewall issue?

    David Mathog wrote:
    > Mandriva 2007.1, vanilla 2.6.21.1 kernel.


    also 2.6.17-14mdv kernel.

    I finally managed to read the messages on the screen once, after it had
    stuck after shorewall but before it cleared and put up the login prompt.
    The shorewall messages got down to here:

    processing /etc/shorewall/start ...
    processing /etc/shorewall/started ...
    done.

    where it sat for almost 3 minutes. So it looks like shorewall started
    all the way. Then something flashed by about an NFS mount having
    failed, and something about a superblock, I think on remote file system.
    (Sorry I can't be more specific, did I mention "flashed by"?)

    When I logged onto the system the portmapper wasn't running.
    For the tasks I've been doing on these lately I would not have noticed
    that. Neither was ypbind. Looking through /var/log/messages there were
    a lot of these:

    Jun 22 16:28:29 saf02 portmap[2910]: cannot bind udp: Permission denied

    Once the system finally came alive it was possible to do

    service portmap start
    service ypbind start
    service nfslock start
    mount -a

    etc and they all worked. So for some reason or other the portmapper
    isn't starting and then it's all downhill from there. I'm not sure why
    the system hangs in NFS mount as the mount parameters are:

    server:/u4/pdb /mnt/server/pdb nfs ro,bg,hard,intr 0 0
    server:/u1 /mnt/server/u1 nfs rw,bg,hard,intr 0 0

    If it couldn't mount it should have backgrounded. Perhaps the timeout
    before this happens is long?

    That portmap error message used to appear on systems when the portmapper
    didn't run as root (because it needed to grab 111) but I don't think
    that's what's causing it here. I'll have to force the portmap startup
    script to log all steps somewhere, with time stamps, so that I can
    follow this better. Next week though.

    Thanks,

    David Mathog

  3. Re: long wait in boot sequence, shorewall issue?

    On Fri, 22 Jun 2007 16:07:27 -0700, David Mathog wrote:
    > Mandriva 2007.1, vanilla 2.6.21.1 kernel.
    >
    > I'm sometimes seeing a really long wait (2.5 minutes, roughly)
    > on reboot for the system to start up fully. Other times it boots
    > quickly. The problem seems to happen after (during?) shorewall
    > startup, as the last messages on the screen while it's pausing are all
    > from shorewall. Here are some examples from /var/log/messages files:


    Have you noticed the delay between
    shorewall [ ]
    or after shorewall [ OK ]


    > Jun 22 14:54:05 saf03 avahi-daemon[2931]: Service "SFTP File Transfer on
    > saf03" (/etc/avahi/services/sftp-ssh.service) successfully established.


    Yuck, are you running the machines in a Micro$oft LAN, and needing M$
    ip info. If not, you can disable the avahi-daemon On Boot. I would
    think it would make sense to move SFTP startup after Shorewall

    > Anybody care to guess where the delay is? Naturally it doesn't do it
    > every time.


    Not enough information about your connection setup, remote file systems,....

    Mandriva has switched to a parallel boot sequence. Processes could
    be hanging waiting for route to be established. Slow dhcp server could
    be a little late and other startup process are in control at the time
    the network startup process finally get control to finish network setup.

    > Jun 22 15:01:01 saf02 crond[4361]:


    Cron starting is pretty far down on the starup list.

    You can see your startup order depending on runlevel, with
    ls -al /etc/rc.d/rc3.d/S*
    or ls -al /etc/rc.d/rc5.d/S* <--- automatic gui starup/login sequence

    Then you get the required service list with
    grep Required-Start /etc/init.d/*

    Now, you can research which init script running after network maybe
    slowing boot.

    OR, you could disable On Boot services past Network, reboot, enable a
    service, reboot, repeat,reboot,repeat,reboot,repeat,reboot, untill you
    narrow down the service that is slow.


  4. Re: long wait in boot sequence, shorewall issue?

    On Fri, 22 Jun 2007 16:58:00 -0700, David Mathog wrote:
    > David Mathog wrote:


    I would disable avahi-daemon on boot.

    > I'm not sure why
    > the system hangs in NFS mount as the mount parameters are:


    Another mix maybe the servers are taking their time to figure out if
    you system is allowd to complete mount.

  5. Re: long wait in boot sequence, shorewall issue?

    On Jun 22, 8:14 pm, Bit Twister wrote:
    > On Fri, 22 Jun 2007 16:58:00 -0700, David Mathog wrote:
    > > David Mathog wrote:

    >
    > I would disable avahi-daemon on boot.
    >

    Is there a webpage describing these new daemons (from a 10.1 pov) and
    what they do? I would like to conserve resources and maybe disable
    unnecessary tasks from running all day. I know it runs this avahi and
    clamav and cupsd and so on. I don't need say cups running since the
    receipt printer isn't supported here anyway; neither is the usb webcam
    afaik. My 'driva PC is wired to the ethernet port of a wireless router
    that's hooked up to a dsl modem, and has some Win2k paritions on a
    dual boot system.

    tia


  6. Re: long wait in boot sequence, shorewall issue?

    Bit Twister wrote:
    > On Fri, 22 Jun 2007 16:58:00 -0700, David Mathog wrote:
    >> David Mathog wrote:

    >
    > I would disable avahi-daemon on boot.


    Sounds like a good idea, it wasn't the problem though.

    >
    >> I'm not sure why
    >> the system hangs in NFS mount as the mount parameters are:

    >
    > Another mix maybe the servers are taking their time to figure out if
    > you system is allowd to complete mount.


    Figured it out. Something or other rewrote my /etc/shorewall/rules
    file back from the way I had it to the way Mandriva 2007.1 initialized
    it, which was with one active line:

    INCLUDE rules.drakx

    So, I took that out, put the 4 lines I had in there before back
    (1 to provide local net sshd access, 1 to stealth 113 (identd), one to
    provide access from the NFS server to the portmapper on 111, and one for
    experimenting) and problem resolved. No delay following shorewall
    start. At least for now, until the next time something or other decides
    to erase my shorewall configuration. Now I have to figure out what
    stepped on my rules file. The most likely candidates are:

    A. urpmi --auto-update --auto
    B. yet another of the bleeping "configure it for you and lose your
    settings, too bad!" scripts which now infest Mandriva.

    Pretty sure I didn't do this via a typo, since there would have been no
    reason to ever do anything remotely like:

    cd /etc/shorewall ; cp rules.dist rules

    Anyway, there was apparently a race condition going on, and if the
    portmapper started and completed its work before the (broken) shorewall
    kicked in, then everything appeared to be fine. The sshd script or
    daemon itself was punching a hole into the firewall, so the absence of
    the explicit configuration in my rules file wasn't showing up. Looking
    at

    /etc/rc.d/rc3.d/S53shorewall
    /etc/rc.d/rc3.d/S53portmap

    one finds that while both have a required-start for $network, they don't
    say anything about each other, so with the multi-startup methodology
    currently employed they are free to come up in any order. They are
    both S53, so that doesn't help any. Makes me wonder what other kinds
    of race conditions exist with respect to the firewall starting and the
    various network services. xinetd, for instance, also only depends on
    $network and not shorewall.

    Regards,

    David Mathog

  7. Re: long wait in boot sequence, shorewall issue?

    On Sun, 24 Jun 2007 16:13:58 -0700, David Mathog wrote:
    >
    > Figured it out. Something or other rewrote my /etc/shorewall/rules
    > file back from the way I had it to the way Mandriva 2007.1 initialized
    > it, which was with one active line:
    >
    > INCLUDE rules.drakx


    With the include feature, you can now add the line
    INCLUDE my.rules
    with your settings in my.rules
    On new installs, just
    ln -s /mydir/my.rules /etc/shorewall
    and add your INCLUDE my.rules line.


    > At least for now, until the next time something or other decides
    > to erase my shorewall configuration. Now I have to figure out what
    > stepped on my rules file. The most likely candidates are:
    >
    > A. urpmi --auto-update --auto
    > B. yet another of the bleeping "configure it for you and lose your
    > settings, too bad!" scripts which now infest Mandriva.


    Then there is,
    C. Running the Mandriva Control Center which tend to wipe
    out custom settings, or almost all, depending on task


  8. Re: long wait in boot sequence, shorewall issue?

    Bit Twister wrote:

    > Then there is,
    > C. Running the Mandriva Control Center which tend to wipe
    > out custom settings, or almost all, depending on task



    MCC is definitely capable of doing this. However I've been very
    careful NOT to run MCC on these systems lately, having now been
    bitten by that particular problem enough times to have learned my
    lesson. So MCC could not have been the problem this time.

    I'm considering keeping a set of files in a /root/etc hierarchy.
    These would be copies of all the configuration files I have
    modified, so that I can more rapidly detect when Mandriva
    has been "helpful", and then recover from it. Sort of a "tripwire
    lite", with recovery capability.

    Regards,

    David Mathog


  9. Re: long wait in boot sequence, shorewall issue?

    On 2007-06-25, David Mathog wrote:
    > Bit Twister wrote:
    >
    >> Then there is,
    >> C. Running the Mandriva Control Center which tend to wipe
    >> out custom settings, or almost all, depending on task

    >
    >
    > MCC is definitely capable of doing this. However I've been very
    > careful NOT to run MCC on these systems lately, having now been
    > bitten by that particular problem enough times to have learned my
    > lesson. So MCC could not have been the problem this time.
    >
    > I'm considering keeping a set of files in a /root/etc hierarchy.
    > These would be copies of all the configuration files I have
    > modified, so that I can more rapidly detect when Mandriva
    > has been "helpful", and then recover from it. Sort of a "tripwire
    > lite", with recovery capability.


    That's a good idea to keep a backup set of configuration
    files. A script to check all of the files in one shot can
    make it convenient to check for 'tampering'.

    To take it one step further, it can be useful to keep an RCS
    repository of the config files. Set a tag on each file for
    the initial and final version for each OS release. Then,
    when you upgrade to 2008, you can compare the initial 2007
    version (the one supplied by Mandriva), the final 2007
    version (the one with your 2007 customizations), and the
    initial 2008 version to help you decide what you want to
    have for your final 2008 version.

    --
    Robert Riches
    spamtrap42@verizon.net
    (Yes, that is one of my email addresses.)

+ Reply to Thread