Openserver 5.0.6 ping floods router - SCO

This is a discussion on Openserver 5.0.6 ping floods router - SCO ; Hello, I have a client with an SCO OSR 5.0.6 box that is flooding their router with pings (hundreds per second). The router log shows: SRCIP: 192.168.1.10 (SCO box) -> DESTIP: 192.168.1.3 (router). I've done a "ps -fe" and looked ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 27

Thread: Openserver 5.0.6 ping floods router

  1. Openserver 5.0.6 ping floods router

    Hello,

    I have a client with an SCO OSR 5.0.6 box that is flooding their
    router with pings (hundreds per second). The router log shows: SRCIP:
    192.168.1.10 (SCO box) -> DESTIP: 192.168.1.3 (router).

    I've done a "ps -fe" and looked for obvious pings with no luck. I did
    a custom -V (to verify that nothing has been changed from the standard
    install) -- no luck there, either. I've searched google, AP Lawrence,
    etc. for SCO rootkits, and nothing seems to fit my situation.

    The system normally runs in init state 2. One experiment we're going
    to try is to reboot into single user mode and see if it's still
    pinging wildly. If it is, that should make it easier to track down
    the process that's causing the problem.

    Questions for the group:

    1. Is there an administrative command I can use that shows network
    usage by process? I've looked at all the options for "ps", but
    nothing really shows network I/O.
    2. Has anyone else run into this?
    3. Does anyone have an idea of something I can try to track this down?

    This is a production system, and it's wreaking havoc with the client's
    VPNs and network connections...

    Thanks,

    RLW

  2. Re: Openserver 5.0.6 ping floods router

    On Mon, Jul 07, 2008, RLW wrote:
    >Hello,
    >
    >I have a client with an SCO OSR 5.0.6 box that is flooding their
    >router with pings (hundreds per second). The router log shows: SRCIP:
    >192.168.1.10 (SCO box) -> DESTIP: 192.168.1.3 (router).
    >
    >I've done a "ps -fe" and looked for obvious pings with no luck. I did
    >a custom -V (to verify that nothing has been changed from the standard
    >install) -- no luck there, either. I've searched google, AP Lawrence,
    >etc. for SCO rootkits, and nothing seems to fit my situation.


    I would use ``lsof -n -i | grep 192.168.1.3'' to see what
    process(es) are connecting to the router box. Once you find the
    process pid, you can use ``lsof -p pid'' to see what that process
    is using.

    Bill
    --
    INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC
    URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way
    Voice: (206) 236-1676 Mercer Island, WA 98040-0820
    Fax: (206) 232-9186

    "I do not feel obliged to believe that the same God who has endowed us
    with sense, reason, and intellect has intended us to forego their use."
    -- Galileo Galilei

  3. Re: Openserver 5.0.6 ping floods router

    On Jul 7, 12:33*pm, Bill Campbell wrote:
    > ...
    > I would use ``lsof -n -i | grep 192.168.1.3'' to see what
    > process(es) are connecting to the router box. *Once you find the
    > process pid, you can use ``lsof -p pid'' to see what that process
    > is using.
    >
    > Bill
    > --


    Bill,

    Thanks for the response. This system doesn't have a development
    system installed (or licensed). I dug around the Purdue ftp server,
    but binaries aren't available -- there's a strong note in the README
    that they shouldn't be trusted. By any chance, do you have an
    OSR5.0.6 binary of lsof you could get to me?

    Thanks again for the tip.

    RLW

  4. Re: Openserver 5.0.6 ping floods router

    Bill,

    Never mind -- I found it on the Skunkware site.

    Thanks again,

    RLW


  5. Re: Openserver 5.0.6 ping floods router

    Bill Campbell typed (on Mon, Jul 07, 2008 at 09:33:33AM -0700):
    | On Mon, Jul 07, 2008, RLW wrote:
    | >Hello,
    | >
    | >I have a client with an SCO OSR 5.0.6 box that is flooding their
    | >router with pings (hundreds per second). The router log shows: SRCIP:
    | >192.168.1.10 (SCO box) -> DESTIP: 192.168.1.3 (router).
    | >
    | >I've done a "ps -fe" and looked for obvious pings with no luck. I did
    | >a custom -V (to verify that nothing has been changed from the standard
    | >install) -- no luck there, either. I've searched google, AP Lawrence,
    | >etc. for SCO rootkits, and nothing seems to fit my situation.
    |
    | I would use ``lsof -n -i | grep 192.168.1.3'' to see what
    | process(es) are connecting to the router box. Once you find the
    | process pid, you can use ``lsof -p pid'' to see what that process
    | is using.

    No need for the 'grep'; instead, run ``lsof -ni @192.168.1.3''.

    --
    JP

  6. Re: Openserver 5.0.6 ping floods router

    Bill,

    OK, I downloaded and installed the SkunkWorks version -- it's for
    OSR5.0.5, not OSR5.0.6, but seems to run anyway.

    It's not finding anything talking to 192.168.1.3, however. I put it
    in repeat mode with:

    /usr/local/bin/lsof -r 5 -n -i @192.168.1.3

    Still coming up empty.

    I'm certain that the SCO box is the source of the pings, since they
    stopped when we disconnected the network cable from the system (the
    SCO box).

    An "ndstat" shows that there's an overwhelming amount of traffic going
    out of the server:

    Device MAC address in use Factory MAC Address
    ------ ------------------ -------------------
    /dev/net1 00:02:55:07:dc:9a 00:02:55:07:dc:9a

    Multicast address table
    -----------------------
    01:00:5e:00:00:01

    FRAMES
    Unicast Multicast Broadcast Error Octets Queue Length
    ---------- --------- --------- ------ ----------- ------------
    In: 915523883 0 105059 0 1725295301 0
    Out: 915516597 0 8 0 3962704890 0

    I'm running out of ideas...

    Thanks again for your help.

    RLW


  7. Re: Openserver 5.0.6 ping floods router

    RLW typed (on Mon, Jul 07, 2008 at 10:45:10AM -0700):
    | Bill,
    |
    | OK, I downloaded and installed the SkunkWorks version -- it's for
    | OSR5.0.5, not OSR5.0.6, but seems to run anyway.
    |
    | It's not finding anything talking to 192.168.1.3, however. I put it
    | in repeat mode with:
    |
    | /usr/local/bin/lsof -r 5 -n -i @192.168.1.3
    |
    | Still coming up empty.
    |
    | I'm certain that the SCO box is the source of the pings, since they
    | stopped when we disconnected the network cable from the system (the
    | SCO box).
    |
    | An "ndstat" shows that there's an overwhelming amount of traffic going
    | out of the server:
    |
    | Device MAC address in use Factory MAC Address
    | ------ ------------------ -------------------
    | /dev/net1 00:02:55:07:dc:9a 00:02:55:07:dc:9a
    |
    | Multicast address table
    | -----------------------
    | 01:00:5e:00:00:01
    |
    | FRAMES
    | Unicast Multicast Broadcast Error Octets Queue Length
    | ---------- --------- --------- ------ ----------- ------------
    | In: 915523883 0 105059 0 1725295301 0
    | Out: 915516597 0 8 0 3962704890 0
    |
    | I'm running out of ideas...

    Does netstat -a help at all?



    --
    JP

  8. Re: Openserver 5.0.6 ping floods router

    RLW wrote (on Mon, Jul 07, 2008 at 10:45:10AM -0700):
    > Bill,
    >
    > OK, I downloaded and installed the SkunkWorks version -- it's for
    > OSR5.0.5, not OSR5.0.6, but seems to run anyway.
    >
    > It's not finding anything talking to 192.168.1.3, however. I put it
    > in repeat mode with:
    >
    > /usr/local/bin/lsof -r 5 -n -i @192.168.1.3
    >
    > Still coming up empty.
    >
    > I'm certain that the SCO box is the source of the pings, since they
    > stopped when we disconnected the network cable from the system (the
    > SCO box).
    >
    > An "ndstat" shows that there's an overwhelming amount of traffic going
    > out of the server:
    >
    > Device MAC address in use Factory MAC Address
    > ------ ------------------ -------------------
    > /dev/net1 00:02:55:07:dc:9a 00:02:55:07:dc:9a
    >
    > Multicast address table
    > -----------------------
    > 01:00:5e:00:00:01
    >
    > FRAMES
    > Unicast Multicast Broadcast Error Octets Queue Length
    > ---------- --------- --------- ------ ----------- ------------
    > In: 915523883 0 105059 0 1725295301 0
    > Out: 915516597 0 8 0 3962704890 0
    >
    > I'm running out of ideas...
    >
    > Thanks again for your help.
    >
    > RLW


    Run tcpdump? I suppose you'd run it on a neighboring linux/windows
    box, as it's a difficult SCO install, but maybe it would provide
    you a clue.

    --
    _________________________________________
    Nachman Yaakov Ziskind, FSPA, LLM awacs@ziskind.us
    Attorney and Counselor-at-Law http://ziskind.us
    Economic Group Pension Services http://egps.com
    Actuaries and Employee Benefit Consultants

  9. Re: Openserver 5.0.6 ping floods router

    Jean-Pierre Radley wrote (on Mon, Jul 07, 2008 at 02:15:21PM -0400):
    > RLW typed (on Mon, Jul 07, 2008 at 10:45:10AM -0700):
    > | Bill,
    > |
    > | OK, I downloaded and installed the SkunkWorks version -- it's for
    > | OSR5.0.5, not OSR5.0.6, but seems to run anyway.
    > |
    > | It's not finding anything talking to 192.168.1.3, however. I put it
    > | in repeat mode with:
    > |
    > | /usr/local/bin/lsof -r 5 -n -i @192.168.1.3
    > |
    > | Still coming up empty.
    > |
    > | I'm certain that the SCO box is the source of the pings, since they
    > | stopped when we disconnected the network cable from the system (the
    > | SCO box).
    > |
    > | An "ndstat" shows that there's an overwhelming amount of traffic going
    > | out of the server:
    > |
    > | Device MAC address in use Factory MAC Address
    > | ------ ------------------ -------------------
    > | /dev/net1 00:02:55:07:dc:9a 00:02:55:07:dc:9a
    > |
    > | Multicast address table
    > | -----------------------
    > | 01:00:5e:00:00:01
    > |
    > | FRAMES
    > | Unicast Multicast Broadcast Error Octets Queue Length
    > | ---------- --------- --------- ------ ----------- ------------
    > | In: 915523883 0 105059 0 1725295301 0
    > | Out: 915516597 0 8 0 3962704890 0
    > |
    > | I'm running out of ideas...
    >
    > Does netstat -a help at all?
    > --
    > JP


    Would netstat track ICMP at all? I just tried flood ping on my 506
    box, and netstat -a showed nothing.

    --
    _________________________________________
    Nachman Yaakov Ziskind, FSPA, LLM awacs@ziskind.us
    Attorney and Counselor-at-Law http://ziskind.us
    Economic Group Pension Services http://egps.com
    Actuaries and Employee Benefit Consultants

  10. Re: Openserver 5.0.6 ping floods router

    On Jul 7, 11:24*am, "N. Yaakov Ziskind" wrote:
    > Jean-Pierre Radley wrote (on Mon, Jul 07, 2008 at 02:15:21PM -0400):
    >
    >
    >
    > > RLW typed (on Mon, Jul 07, 2008 at 10:45:10AM -0700):
    > > | Bill,
    > > |
    > > | OK, I downloaded and installed the SkunkWorks version -- it's for
    > > | OSR5.0.5, not OSR5.0.6, but seems to run anyway.
    > > |
    > > | It's not finding anything talking to 192.168.1.3, however. *I put it
    > > | in repeat mode with:
    > > |
    > > | * /usr/local/bin/lsof -r 5 -n -i @192.168.1.3
    > > |
    > > | Still coming up empty.
    > > |
    > > | I'm certain that the SCO box is the source of the pings, since they
    > > | stopped when we disconnected the network cable from the system (the
    > > | SCO box).
    > > |
    > > | An "ndstat" shows that there's an overwhelming amount of traffic going
    > > | out of the server:
    > > |
    > > | Device * * * MAC address in use * *Factory MAC Address
    > > | ------ * * * ------------------ * *-------------------
    > > | /dev/net1 * *00:02:55:07:dc:9a * * 00:02:55:07:dc:9a
    > > |
    > > | * * * * * * * * Multicast address table
    > > | * * * * * * * * -----------------------
    > > | * * * * * * * * 01:00:5e:00:00:01
    > > |
    > > | * * * * * * * * * * * * * * *FRAMES
    > > | * * * Unicast *Multicast Broadcast *Error * *Octets *Queue Length
    > > | * * ---------- --------- --------- ------ ----------- ------------
    > > | In: *915523883 * * * * 0 * *105059 * * *0 *1725295301 * * * * * *0
    > > | Out: 915516597 * * * * 0 * * * * 8 * * *0 *3962704890 * * * * * *0
    > > |
    > > | I'm running out of ideas...

    >
    > > Does netstat -a help at all?
    > > --
    > > JP

    >
    > Would netstat track ICMP at all? I just tried flood ping on my 506
    > box, and netstat -a showed nothing.


    I doubt it. In any case, any process that is cranking out that much
    traffic should be using quite a bit of processor time, and should show
    up as such in "top" (or "ps" for that matter).

    You can easily (somewhat non-intrusively) test what process is doing
    it by using kill SIGSTOP/SIGCONT.

    Keep in mind that some applications do not take kindling to being
    SIGSTOP'd but in general, it should be ok if you SIGCONT quickly
    enough.


  11. Re: Openserver 5.0.6 ping floods router

    RLW wrote:
    > Hello,
    >
    > I have a client with an SCO OSR 5.0.6 box that is flooding their
    > router with pings (hundreds per second). The router log shows: SRCIP:
    > 192.168.1.10 (SCO box) -> DESTIP: 192.168.1.3 (router).
    >
    > I've done a "ps -fe" and looked for obvious pings with no luck. I did
    > a custom -V (to verify that nothing has been changed from the standard
    > install) -- no luck there, either. I've searched google, AP Lawrence,
    > etc. for SCO rootkits, and nothing seems to fit my situation.
    >
    > The system normally runs in init state 2. One experiment we're going
    > to try is to reboot into single user mode and see if it's still
    > pinging wildly. If it is, that should make it easier to track down
    > the process that's causing the problem.
    >
    > Questions for the group:
    >
    > 1. Is there an administrative command I can use that shows network
    > usage by process? I've looked at all the options for "ps", but
    > nothing really shows network I/O.
    > 2. Has anyone else run into this?
    > 3. Does anyone have an idea of something I can try to track this down?
    >
    > This is a production system, and it's wreaking havoc with the client's
    > VPNs and network connections...
    >
    > Thanks,
    >
    > RLW


    Perhaps the SCO boxes NIC has bit the big one - a hardware glitch
    causing the pings wouldn't show using SCO tools.

    Could also be a bad cable with the read/write leads touching each other.

    Also shine a light in the female RJ-45b connection on the router and SCO
    side - maybe the little copper pins are touching.

    --
    ----------------------------------------------------
    Pat Welch, UBB Computer Services, a WCS Affiliate
    SCO Authorized Partner
    Microlite BackupEdge Certified Reseller
    Unix/Linux/Windows/Hardware Sales/Support
    (209) 745-1401 Cell: (209) 251-9120
    E-mail: patubb@inreach.com
    ----------------------------------------------------

  12. Re: Openserver 5.0.6 ping floods router

    Thanks for all the replies -- I've tried most of them, still, to no
    avail.

    We did an experiment with the system today -- rebooted it and watched
    the CPU Load on the router while it came up. Just at the point where
    the boot process says (I'm paraphrasing) "Type D to continue, or
    enter root password:", the CPU Load jumped back up to 100%.

    ==============================
    Note to Pat Welch, thanks for the idea, but I've never heard of a NIC
    that can do a ping flood by itself. Have you actually seen something
    like that happen? Also, if the NIC was causing the problem, I'd think
    that it would have started flooding before the OS was all the way up,
    assuming it was a hardware problem. I still don't see how it would
    remember the IP for the default gateway between reboots.
    ==============================

    I had my customer, Tom, enter the root password to go into init state
    1 (Single User mode). I then had him do a "ps -fe > /usr/rlw/
    singleusr.txt" to capture a list of all of the processes running at
    the time. Here's the list:

    UID PID PPID C STIME TTY TIME CMD
    root 0 0 0 12:06:49 ? 00:00:00 sched
    root 1 0 0 12:06:49 ? 00:00:00 /etc/init
    root 2 0 0 12:06:49 ? 00:00:00 vhand
    root 3 0 0 12:06:49 ? 00:00:12 bdflush
    root 4 0 0 12:06:49 ? 00:00:00 kmdaemon
    root 5 1 0 12:06:49 ? 00:00:08 htepi_daemon /
    root 6 0 0 12:06:49 ? 00:00:13 strd
    root 92 1 2 12:14:25 console 00:00:01 /bin/ksh -o vi
    root 53 1 0 12:14:24 ? 00:00:00 /etc/ifor_pmd
    root 54 53 0 12:14:24 ? 00:00:00 /etc/ifor_pmd
    root 50 1 0 12:14:23 ? 00:00:36 /etc/syslogd
    root 41 1 0 12:14:23 ? 00:00:00 htepi_daemon /
    stand
    root 103 92 2 12:19:09 console 00:00:00 ps -fe
    root 58 54 0 12:14:24 ? 00:00:00 /etc/sco_cpd
    root 59 54 0 12:14:24 ? 00:00:22 /etc/ifor_sld
    root 79 1 0 12:14:24 ? 00:00:00 strerr
    root 93 1 0 12:14:25 ? 00:00:00 /var/scohttp/
    scohttpd -d /var/scohttp

    I had run a "custom -V" to verify the entire setup, and captured the
    output in another file. I then grepped all of the filenames in the ps
    list above in that file, and came up empty handed. This implies to me
    that the listed processes aren't being seen by custom as having been
    compromised. A very smart trojan (it'd have to be REALLY smart) could
    conceivably change the checksum in the custom database for a
    particular file, and could restore the dates on the file. Has anyone
    ever HEARD of a trojan Denial Of Service for SCO Openserver? I've
    searched Google high and low for something like that -- haven't found
    anything.

    I stopped scohttp and ns_http (SCO web server on port 457, I think,
    and the Netscape "FastTrack" web server) -- we don't use those. I
    haven't killed all of the X windows stuff, but it wasn't running in
    Single User mode, so I don't think that's it.

    Here are my next questions for the group:

    Which of the above processes can I kill (in single user mode) to see
    if the pings stop?
    How do I disable the X Windows stuff? -- We never use it on this box.

    Thanks again for all the help everyone's trying to give me.

    Thanks,

    RLW


  13. Re: Openserver 5.0.6 ping floods router

    RLW wrote:
    > Thanks for all the replies -- I've tried most of them, still, to no
    > avail.
    >
    > We did an experiment with the system today -- rebooted it and watched
    > the CPU Load on the router while it came up. Just at the point where
    > the boot process says (I'm paraphrasing) "Type D to continue, or
    > enter root password:", the CPU Load jumped back up to 100%.


    This is just an off the wall seat of the pants fix that we did once when
    we thought that the Unix Server(ip XXX.XXX.XXX.92) was causing flooding
    of the network, all tests showed it to be so). We finally discovered
    somehow(I cannot recollect exactly how we discovered it) that if we
    disconnected a certain windows 2003 Server that had an ip addreee of
    XXX.XXX.XXX.97 the flooding stopped. Without further ado or inquiry, we
    just reset the ip address of the windows server(the easiest to do) and
    the problem has not recurred in two or so years. Someday I may try to
    find out what was happening but so far time constrants and a degree of
    sloth have kept me from doing so, especially since it appeard to be more
    closely related to Microsoft than Unix.
    >
    > ==============================
    > Note to Pat Welch, thanks for the idea, but I've never heard of a NIC
    > that can do a ping flood by itself. Have you actually seen something
    > like that happen? Also, if the NIC was causing the problem, I'd think
    > that it would have started flooding before the OS was all the way up,
    > assuming it was a hardware problem. I still don't see how it would
    > remember the IP for the default gateway between reboots.
    > ==============================
    >
    > I had my customer, Tom, enter the root password to go into init state
    > 1 (Single User mode). I then had him do a "ps -fe > /usr/rlw/
    > singleusr.txt" to capture a list of all of the processes running at
    > the time. Here's the list:
    >
    > UID PID PPID C STIME TTY TIME CMD
    > root 0 0 0 12:06:49 ? 00:00:00 sched
    > root 1 0 0 12:06:49 ? 00:00:00 /etc/init
    > root 2 0 0 12:06:49 ? 00:00:00 vhand
    > root 3 0 0 12:06:49 ? 00:00:12 bdflush
    > root 4 0 0 12:06:49 ? 00:00:00 kmdaemon
    > root 5 1 0 12:06:49 ? 00:00:08 htepi_daemon /
    > root 6 0 0 12:06:49 ? 00:00:13 strd
    > root 92 1 2 12:14:25 console 00:00:01 /bin/ksh -o vi
    > root 53 1 0 12:14:24 ? 00:00:00 /etc/ifor_pmd
    > root 54 53 0 12:14:24 ? 00:00:00 /etc/ifor_pmd
    > root 50 1 0 12:14:23 ? 00:00:36 /etc/syslogd
    > root 41 1 0 12:14:23 ? 00:00:00 htepi_daemon /
    > stand
    > root 103 92 2 12:19:09 console 00:00:00 ps -fe
    > root 58 54 0 12:14:24 ? 00:00:00 /etc/sco_cpd
    > root 59 54 0 12:14:24 ? 00:00:22 /etc/ifor_sld
    > root 79 1 0 12:14:24 ? 00:00:00 strerr
    > root 93 1 0 12:14:25 ? 00:00:00 /var/scohttp/
    > scohttpd -d /var/scohttp
    >
    > I had run a "custom -V" to verify the entire setup, and captured the
    > output in another file. I then grepped all of the filenames in the ps
    > list above in that file, and came up empty handed. This implies to me
    > that the listed processes aren't being seen by custom as having been
    > compromised. A very smart trojan (it'd have to be REALLY smart) could
    > conceivably change the checksum in the custom database for a
    > particular file, and could restore the dates on the file. Has anyone
    > ever HEARD of a trojan Denial Of Service for SCO Openserver? I've
    > searched Google high and low for something like that -- haven't found
    > anything.
    >
    > I stopped scohttp and ns_http (SCO web server on port 457, I think,
    > and the Netscape "FastTrack" web server) -- we don't use those. I
    > haven't killed all of the X windows stuff, but it wasn't running in
    > Single User mode, so I don't think that's it.
    >
    > Here are my next questions for the group:
    >
    > Which of the above processes can I kill (in single user mode) to see
    > if the pings stop?
    > How do I disable the X Windows stuff? -- We never use it on this box.
    >
    > Thanks again for all the help everyone's trying to give me.
    >
    > Thanks,
    >
    > RLW
    >
    >
    >
    >



  14. Re: Openserver 5.0.6 ping floods router

    RLW wrote (on Wed, Jul 09, 2008 at 10:39:31AM -0700):
    > Thanks for all the replies -- I've tried most of them, still, to no
    > avail.
    >
    > How do I disable the X Windows stuff? -- We never use it on this box.


    'scologin disable.'

    --
    _________________________________________
    Nachman Yaakov Ziskind, FSPA, LLM awacs@ziskind.us
    Attorney and Counselor-at-Law http://ziskind.us
    Economic Group Pension Services http://egps.com
    Actuaries and Employee Benefit Consultants

  15. Re: Openserver 5.0.6 ping floods router

    wrote:

    > UID PID PPID C STIME TTY TIME CMD
    > root 0 0 0 12:06:49 ? 00:00:00 sched
    > root 1 0 0 12:06:49 ? 00:00:00 /etc/init
    > root 2 0 0 12:06:49 ? 00:00:00 vhand
    > root 3 0 0 12:06:49 ? 00:00:12 bdflush
    > root 4 0 0 12:06:49 ? 00:00:00 kmdaemon
    > root 5 1 0 12:06:49 ? 00:00:08 htepi_daemon /
    > root 6 0 0 12:06:49 ? 00:00:13 strd
    > root 92 1 2 12:14:25 console 00:00:01 /bin/ksh -o vi
    > root 53 1 0 12:14:24 ? 00:00:00 /etc/ifor_pmd
    > root 54 53 0 12:14:24 ? 00:00:00 /etc/ifor_pmd
    > root 50 1 0 12:14:23 ? 00:00:36 /etc/syslogd
    > root 41 1 0 12:14:23 ? 00:00:00 htepi_daemon /stand
    > root 103 92 2 12:19:09 console 00:00:00 ps -fe
    > root 58 54 0 12:14:24 ? 00:00:00 /etc/sco_cpd
    > root 59 54 0 12:14:24 ? 00:00:22 /etc/ifor_sld
    > root 79 1 0 12:14:24 ? 00:00:00 strerr
    > root 93 1 0 12:14:25 ? 00:00:00 /var/scohttp/scohttpd -d /var/scohttp


    > Which of the above processes can I kill (in single user mode) to see
    > if the pings stop?


    Most of those are kernel threads that you won't succeed in killing even
    if you try.

    You don't really need to kill them, just pause them. Use `kill -STOP`.

    The ones that are killable at all are:

    init
    ifor_pmd (x2)
    sco_cpd
    ifor_sld
    syslogd
    strerr
    scohttpd
    ksh (but that was created when you entered single-user mode, it
    isn't the culprit; and you can't kill it and continue testing
    anyway)

    Procedure:

    1. observe network monitor to see whether pings stop
    2. kill -STOP 1 ## pause init
    3. check network monitor
    4. kill -CONT 1 ## continue init
    5. repeat with each other killable process ID
    m. tell us which process it was
    n. it will turn out to be none of the processes; tell us that.

    >Bela<


  16. Re: Openserver 5.0.6 ping floods router

    On Jul 10, 7:06 pm, Bela Lubkin wrote:
    > Procedure:
    >
    > 1. observe network monitor to see whether pings stop
    > 2. kill -STOP 1 ## pause init
    > 3. check network monitor
    > 4. kill -CONT 1 ## continue init
    > 5. repeat with each other killable process ID
    > m. tell us which process it was
    >
    > >Bela<


    Bela,

    > n. it will turn out to be none of the processes; tell us that.


    You were right. We shut the box down and brought it back up in single
    user mode. I did a "kill -STOP x" on each of the "killable" processes
    -- the ping still kept on coming.

    While I was there, I verfied that disconnecting the SCO box from the
    network causes the pings to stop. In a vain attempt to see if some
    other process on some other system was "keeping an eye" on the SCO box
    and spoofing pings to the router, we disconnected the entire network
    from the router, plugged ONLY the SCO box into it, and the pings came
    back.

    One odd behavior, it took about a minute or so for the pings to start
    up again after we plugged the SCO box back in to the router. The
    message saying that the NIC was connected appeared on the console
    immediately.

    We're going to try putting another NIC in the machine and disabling
    the onboard NIC, if I can't think of anything else that could be doing
    this. If that doesn't work a kernel restore from a 6 month old tape
    might be a thing to try (the indication of this problem, mainly a
    slowdown through the VPNs on the router, started about 4 months ago).

    Thanks again for everyone's help, I'm still open to suggestions!

    RLW


  17. Re: Openserver 5.0.6 ping floods router

    On Mon, 7 Jul 2008 09:12:16 -0700 (PDT), RLW
    wrote:

    >I have a client with an SCO OSR 5.0.6 box that is flooding their
    >router with pings (hundreds per second). The router log shows: SRCIP:
    >192.168.1.10 (SCO box) -> DESTIP: 192.168.1.3 (router).


    Did this system previously have an HP4 LaserJet printer with an old
    JetDirect card installed? I've seen something like this when the
    print spooler was somehow screwed up. It's a rather wild guess
    because for this to be the culprit, the HP JetDirect card would need
    to be mis-configured to be outside the LAN netmask (thus sending its
    traffic to the router). The pings were just the way the driver was
    checking to see if the printer was available.

    Incidentally, the fix turned out to be updating the firmware on the
    JetDirect card which made no sense because the server was originating
    the garbage. However, the problem never re-appeared so I'll assume
    that there was some connection.



    --
    # Jeff Liebermann 150 Felker St #D Santa Cruz CA 95060
    # 831-336-2558 jeffl@comix.santa-cruz.ca.us
    # http://802.11junk.com jeffl@cruzio.com
    # http://www.LearnByDestroying.com AE6KS

  18. Re: Openserver 5.0.6 ping floods router

    On Jul 15, 7:55*pm, Jeff Liebermann wrote:
    > Did this system previously have an HP4 LaserJet printer with an old
    > JetDirect card installed?

    Jeff,

    Nope -- they've had a couple of {Linksys|Netgear|Dlink|Hawking|RedHat
    6.1} print servers, but no HP JetDirects.

    I'm thinkin' more'n'more it's got some kind of DOS rootkit/trojan on
    it.

    Next week, we're gonna try a new NIC -- see if that changes anything.

    Thanks for the suggestion,

    RLW

  19. Re: Openserver 5.0.6 ping floods router

    wrote:

    > > 1. observe network monitor to see whether pings stop
    > > 2. kill -STOP 1 ## pause init
    > > 3. check network monitor
    > > 4. kill -CONT 1 ## continue init
    > > 5. repeat with each other killable process ID
    > > m. tell us which process it was
    > > n. it will turn out to be none of the processes; tell us that.

    >
    > You were right. We shut the box down and brought it back up in single
    > user mode. I did a "kill -STOP x" on each of the "killable" processes
    > -- the ping still kept on coming.
    >
    > While I was there, I verfied that disconnecting the SCO box from the
    > network causes the pings to stop. In a vain attempt to see if some
    > other process on some other system was "keeping an eye" on the SCO box
    > and spoofing pings to the router, we disconnected the entire network
    > from the router, plugged ONLY the SCO box into it, and the pings came
    > back.
    >
    > One odd behavior, it took about a minute or so for the pings to start
    > up again after we plugged the SCO box back in to the router. The
    > message saying that the NIC was connected appeared on the console
    > immediately.


    I know you're just trying to solve the immediate problem, while I am
    responding out of intellectual curiousity and a desire to actually
    _solve_ the problem, i.e. fully understand the root cause. On that
    basis, I have some further suggestions to try before you bring down the
    sledgehammer by replacing the NIC.

    One is: capture those ping packets with your sniffer; then deliberately
    issue a similar ping request from the command line (i.e. ping -s [same
    size as those packets] [same host they are pinging]); compare the
    resulting packets. Any interesting differences? (You might have to
    fiddle the size +/- 8 bytes due to how OSR5 `ping` calculates the packet
    size vs. your requested size.) Part of the idea here is to try to craft
    your own ping packet to be as similar as possible to the ones you're
    trying to eliminate. Vary the `ping` args until you match it as closely
    as possible: the remaining differences will be instructive. (Also the
    differences you had to introduce to make the approximate match.) Post
    that information + 1ea decoded packet dumps of yours + theirs.

    Another is: capture output of `netstat -i` and `ndstat -l` before and
    during the ping flooding. The fact that it doesn't start happening for
    a minute or so gives you a fine opportunity to record a baseline:

    netstat -i > netstat-i.before
    ndstat -l > ndstat-l.before

    Then let it ping away for a while, capture another set. Then another a
    few minutes later. `netstat -i` gives us a crude picture of how much
    net I/O the NIC driver claims to have done. `ndstat -l` gives a lot
    more details. Interesting results might include: if neither claims any
    packets are being sent (or substantially less than the number you're
    seeing flow by on the sniffer), the action must be autonomous inside the
    NIC. Or you may see some obscure error counter incrementing in `ndstat
    -l`, once per ping. For instance, the machine I'm looking at says:

    SQE Test Errors 0 Number of Signal Quality Error Test
    signals that were detected by the adapter

    Well, who knows what the heck that is. But if it went up by 20000
    during a time span which also saw 20000 ping packets issued by the NIC,
    it would be pretty interesting...

    Another tack also relates to your observation of a 1-minute delay before
    the pings start. That implies that some user-level process is
    initiating them. Perhaps it's a transient process, something that
    somehow starts an in-kernel ping which continues past the death of the
    initiating process. How can you catch such a thing?

    Turn on process accounting, then monitor it:

    # accton /usr/adm/pacct
    # tail -32cf /usr/adm/pacct | acctcom -i

    Since you only have a minute or so between reaching the first
    single-user mode prompt and the pings starting, you'll want to have
    those commands saved in a script. Get to SU mode and immediately run
    the script. Watch your net monitor and the `acctcom` output at the same
    time. Does some process go by just before the pings start?

    `acctcom` only records the name of the program that was run, not its
    parameters. So even if you see something suspicious go by, there will
    be some further sleuthing to do. As a first step on that trail, I'll
    mention that the file /usr/adm/pacct is building up and can be
    re-examined with further `acctcom` commands (read the man page).

    > We're going to try putting another NIC in the machine and disabling
    > the onboard NIC, if I can't think of anything else that could be doing
    > this. If that doesn't work a kernel restore from a 6 month old tape
    > might be a thing to try (the indication of this problem, mainly a
    > slowdown through the VPNs on the router, started about 4 months ago).


    I'd be interested in the results of the NIC switch too, I just hope
    you'll try some of these software probes first ('cause I know that if a
    new NIC "fixes" it, the press of circumstances will prevent you from
    doing any further research on the original problem...)

    >Bela<


  20. Re: Openserver 5.0.6 ping floods router

    On Jul 16, 4:11*am, Bela Lubkin wrote:
    > I know you're just trying to solve the immediate problem, while I am
    > responding out of intellectual curiousity and a desire to actually
    > _solve_ the problem, i.e. fully understand the root cause.


    Bela,

    I want to understand the root cause, too. Unfortunately, if I can
    solve the immediate problem, the client will be off my back. I'm
    still going to try to track it down, however.

    > One is: capture those ping packets with your sniffer; then deliberately
    > issue a similar ping request


    We're not using a sniffer -- just watching the CPU Utilization graph
    on the router's web interface, and viewing the log file.

    > Another is: capture output of `netstat -i` and `ndstat -l` before and
    > during the ping flooding. *The fact that it doesn't start happening for
    > a minute or so gives you a fine opportunity to record a baseline:
    >
    > * netstat -i > netstat-i.before
    > * ndstat -l > ndstat-l.before
    >
    > Then let it ping away for a while, capture another set.


    That's a good idea, and something I can do with what I have
    available...

    > Another tack also relates to your observation of a 1-minute delay before
    > the pings start. *That implies that some user-level process is
    > initiating them. *...[snip]...


    How about I build a little script file, stick it on /stand, and put a
    reference to it in /etc/rc1? Or would a better trick be to create a
    script called something like /etc/rc1.d/S00TryToCatchAPinger so it'll
    be run first thing when rc1.d gets processed?

    I was digging around in the /etc/rc1 script yesterday, but was a tad
    hesitant to stick anything in there -- I don't want to "brick" the SCO
    box by putting something in rc1 that won't let it get to the "Type
    D to continue..." message.

    Thanks for the suggestions, I'll try 'em and post the results back
    here.

    [BEGIN OFF_TOPIC]
    It's been years since I used comp.unix.sco.misc (I used to post and
    reply as bob (AT) infinet (DOT) com, back in the 90s) -- I forgot what
    a great resource it is. One of the ironic problems I have with OSR5
    is that once it's set up, it just pretty much works. That has the
    effect of making my SCO skills atrophy. I've got a couple of clients
    still chuggin' along on pre-Y2K boxes. I'm playing around with moving
    those folks to a VMWare-hosted SCO on a desktop box. It'll be easier
    to backup to 4GB memory sticks than to find a working 4GB HP DAT
    drive! For anyone that's interested in doing so with OSR5.0.5, the
    trick is to boot off an OSR5.0.7 boot diskette, and feed it the
    OSR5.0.5 CD. Also, use the AMD PCNET PCI Adapter driver for the NIC -
    Bus #0/Dev #0/Func #0. I've verified that works on OSR5.0.4...
    [END OFF_TOPIC]

    Thanks again for all the help!

    RLW

+ Reply to Thread
Page 1 of 2 1 2 LastLast