hidden processes - how can I investigate - Security

This is a discussion on hidden processes - how can I investigate - Security ; Hello, I just found out that I have some hidden processes on my computer. I found that using chkrootkit ( www.chkrootkit.org ). The case is that when I do "ps" or "ls /proc" I can't see, for instance, process 9802. ...

+ Reply to Thread
Results 1 to 12 of 12

Thread: hidden processes - how can I investigate

  1. hidden processes - how can I investigate

    Hello,

    I just found out that I have some hidden processes on my computer. I found
    that using chkrootkit (www.chkrootkit.org). The case is that when I do "ps"
    or "ls /proc" I can't see, for instance, process 9802. But when I
    do "cd /proc/9802" I can found out that such process actually exist.

    The strange thing is that those hidden processes seem to be absolutely
    innocent: that are just six or so instances
    of /usr/lib/openoffice/program/soffice.bin.

    I checked soffice.bin's MD5 checksum against the checksum from the
    distribution, and seems it is not modified.

    I wrote a simple kernel module that lists all processes:
    for (pos = next_task(current); pos != (current); pos = next_task(pos))
    printk("process: %d\n", pos->pid);

    This module also have not shown that hidden processes - seems like something
    has removed them from kernel's list of processes! I guessed it was some
    malicious kernel module that has removed them. But when I booted the
    machine with another kernel that strange processes are still hidden. Hm, so
    it can't be a malicious module - it wouldn't load with another version of
    kernel! So maybe something patches live kernel, for instance
    using /dev/kmem? But what can do that? I checked the filesystem with
    look_for_hidden_files (
    http://freshmeat.net/projects/look_for_hidden_files/ ) and found out that
    no files are hidden. Hm, so maybe I should check all boot scripts?


    So, my question is: do you have any suggestions about where should I
    investigate now?

    --
    http://www.piotrsobolewski.w.pl/
    obfus_01@gazeta.pl obfus_02(at)gazeta.pl obfus_03[at]gazeta.pl

  2. Re: hidden processes - how can I investigate

    On Wed, 19 Apr 2006 16:18:07 +0200, piotr_sobolewski wrote:

    > Hello,
    >
    > I just found out that I have some hidden processes on my computer. I found
    > that using chkrootkit (www.chkrootkit.org). The case is that when I do
    > "ps" or "ls /proc" I can't see, for instance, process 9802. But when I do
    > "cd /proc/9802" I can found out that such process actually exist.
    >
    > The strange thing is that those hidden processes seem to be absolutely
    > innocent: that are just six or so instances of
    > /usr/lib/openoffice/program/soffice.bin.
    >
    > I checked soffice.bin's MD5 checksum against the checksum from the
    > distribution, and seems it is not modified.
    >
    > I wrote a simple kernel module that lists all processes: for (pos =
    > next_task(current); pos != (current); pos = next_task(pos))
    > printk("process: %d\n", pos->pid);
    >
    > This module also have not shown that hidden processes - seems like
    > something has removed them from kernel's list of processes! I guessed it
    > was some malicious kernel module that has removed them. But when I booted
    > the machine with another kernel that strange processes are still hidden.
    > Hm, so it can't be a malicious module - it wouldn't load with another
    > version of kernel! So maybe something patches live kernel, for instance
    > using /dev/kmem? But what can do that? I checked the filesystem with
    > look_for_hidden_files (
    > http://freshmeat.net/projects/look_for_hidden_files/ ) and found out that
    > no files are hidden. Hm, so maybe I should check all boot scripts?
    >
    >
    > So, my question is: do you have any suggestions about where should I
    > investigate now?


    Your expertise level is already pretty good...

    A few points.

    You could double check with RootKitHunter (rkhunter)
    available at :
    http://www.rkhunter.org/

    Now, if you have reasons to believe that a rootkit is owning your server,
    you cannot have any confidence in your server. Period...

    Something else you could look for : some BootCD distributions of Linux

    I have Great respect for Knoppix and Helix.

    Available at :

    http://www.knopper.net/knoppix/

    and

    http://www.e-fense.com/helix/

    Once you have booted with the CD Rom, you can look for hidden files,
    strange directory ("..." come to mind )

    Another I would look for : are these occurences of soffice.bin are either
    being used at the time or are they zombies process ?
    (a 'Z' in the third column of the command "ps wax")

    Hope it help...


  3. Re: hidden processes - how can I investigate

    noEMA writes:

    >On Wed, 19 Apr 2006 16:18:07 +0200, piotr_sobolewski wrote:


    >> Hello,
    >>
    >> I just found out that I have some hidden processes on my computer. I found
    >> that using chkrootkit (www.chkrootkit.org). The case is that when I do
    >> "ps" or "ls /proc" I can't see, for instance, process 9802. But when I do
    >> "cd /proc/9802" I can found out that such process actually exist.


    Sounds to me like you have been rooted.

    If you have an rpm based distro, try
    rpm -Vf /bin/ps
    rpm -Vf /bin/ls

    >>
    >> The strange thing is that those hidden processes seem to be absolutely
    >> innocent: that are just six or so instances of
    >> /usr/lib/openoffice/program/soffice.bin.
    >>
    >> I checked soffice.bin's MD5 checksum against the checksum from the
    >> distribution, and seems it is not modified.


    That is just a name.

    >>
    >> I wrote a simple kernel module that lists all processes: for (pos =
    >> next_task(current); pos != (current); pos = next_task(pos))
    >> printk("process: %d\n", pos->pid);
    >>
    >> This module also have not shown that hidden processes - seems like
    >> something has removed them from kernel's list of processes! I guessed it
    >> was some malicious kernel module that has removed them. But when I booted
    >> the machine with another kernel that strange processes are still hidden.
    >> Hm, so it can't be a malicious module - it wouldn't load with another
    >> version of kernel! So maybe something patches live kernel, for instance
    >> using /dev/kmem? But what can do that? I checked the filesystem with
    >> look_for_hidden_files (
    >> http://freshmeat.net/projects/look_for_hidden_files/ ) and found out that
    >> no files are hidden. Hm, so maybe I should check all boot scripts?
    >>
    >>
    >> So, my question is: do you have any suggestions about where should I
    >> investigate now?


    >Your expertise level is already pretty good...


    >A few points.


    >You could double check with RootKitHunter (rkhunter)
    >available at :
    >http://www.rkhunter.org/


    >Now, if you have reasons to believe that a rootkit is owning your server,
    >you cannot have any confidence in your server. Period...


    >Something else you could look for : some BootCD distributions of Linux


    >I have Great respect for Knoppix and Helix.


    >Available at :


    >http://www.knopper.net/knoppix/


    >and


    >http://www.e-fense.com/helix/


    >Once you have booted with the CD Rom, you can look for hidden files,
    >strange directory ("..." come to mind )


    >Another I would look for : are these occurences of soffice.bin are either
    >being used at the time or are they zombies process ?
    >(a 'Z' in the third column of the command "ps wax")


    >Hope it help...



  4. Re: hidden processes - how can I investigate

    noEMA wrote:

    > Now, if you have reasons to believe that a rootkit is owning your server,
    > you cannot have any confidence in your server. Period...


    Sure, I know it, and probably the most safe would be to just wipe hard disk
    and reinstall everything, but it is my home machine, so I don't risk *so*
    much, and this case is so interesting for me that I couldn't sleep if I
    haven't found what actually goes on...

    And thanks a lot for all your advice!

    --
    http://www.piotrsobolewski.w.pl/
    obfus_01@gazeta.pl obfus_02(at)gazeta.pl obfus_03[at]gazeta.pl

  5. Re: hidden processes - how can I investigate

    On Thu, 20 Apr 2006 21:51:08 +0200, piotr_sobolewski wrote:

    > noEMA wrote:
    >
    >> Now, if you have reasons to believe that a rootkit is owning your
    >> server, you cannot have any confidence in your server. Period...

    >
    > Sure, I know it, and probably the most safe would be to just wipe hard
    > disk and reinstall everything, but it is my home machine, so I don't risk
    > *so* much, and this case is so interesting for me that I couldn't sleep if
    > I haven't found what actually goes on...
    >
    > And thanks a lot for all your advice!


    Actually no, it's not just a problem for your own machine...
    It mean that your cracker can still use "your" PC as his mean of attack.

    Just imagine for a second if your computer was "used" by someone to send
    an death threat via E-Mail... Or your PC storing a few thousands files
    of kiddie pr0n...

    Imagine doors kicked at 3am... Police in full combat gear...

    So, backup of data files, and flush disk...





  6. Re: hidden processes - how can I investigate

    piotr_sobolewski@nospampse-o2.pl wrote:
    > Hello,
    >
    > I just found out that I have some hidden processes on my computer. I found
    > that using chkrootkit (www.chkrootkit.org).

    ....
    >
    > The strange thing is that those hidden processes seem to be absolutely
    > innocent: that are just six or so instances
    > of /usr/lib/openoffice/program/soffice.bin.


    Piotr, may I ask you how did you find these hidden processes? For me,
    chkrootkit (0.46a) shows the following with openoffice started:

    Checking `chkutmp'... The tty of the following user process(es) were
    not found
    in /var/run/utmp !
    ! mz 8380 pts/1 /bin/sh /usr/bin/soffice M1.doc
    ! mz 8381 pts/1 /opt/openoffice.org2.0/program/soffice.bin M1.doc
    ! mz 8382 pts/1 /opt/openoffice.org2.0/program/soffice.bin M1.doc
    ! mz 8383 pts/1 /opt/openoffice.org2.0/program/soffice.bin M1.doc
    ! mz 8384 pts/1 /opt/openoffice.org2.0/program/soffice.bin M1.doc
    ! mz 8385 pts/1 /opt/openoffice.org2.0/program/soffice.bin M1.doc
    ....

    I don't see anything unusual with this because a number of other
    programs generate similar warnings.

    Regards,
    Mikhail


  7. Re: hidden processes - how can I investigate

    In article ,
    Unruh wrote:

    >noEMA writes:
    >
    >>On Wed, 19 Apr 2006 16:18:07 +0200, piotr_sobolewski wrote:
    >>>
    >>> I checked soffice.bin's MD5 checksum against the checksum from the
    >>> distribution, and seems it is not modified.

    >
    >That is just a name.


    What do you mean? The OP was checking the MD5 of the file's contents,
    not its name.

  8. Re: hidden processes - how can I investigate

    In article ,
    piotr_sobolewski@nospampse-o2.pl wrote:

    >I just found out that I have some hidden processes on my computer. I found
    >that using chkrootkit (www.chkrootkit.org). The case is that when I do "ps"
    >or "ls /proc" I can't see, for instance, process 9802. But when I
    >do "cd /proc/9802" I can found out that such process actually exist.
    >
    >I wrote a simple kernel module that lists all processes:
    >for (pos = next_task(current); pos != (current); pos = next_task(pos))
    > printk("process: %d\n", pos->pid);
    >
    >This module also have not shown that hidden processes - seems like something
    >has removed them from kernel's list of processes!


    Interesting. Looking at the implementation of procfs
    , the list of PID directories
    is filled in by proc_pid_readdir, which gets a block of PIDs at a time
    by calling get_tgid_list (same source file), which collects PIDs by
    starting from init_task and following next_task links.

    Whereas when you cd directly to a particular PID directory, that would
    be found through proc_task_lookup, which calls find_task_by_pid, which
    is a macro expanding to find_task_by_pid_type
    , which calls find_pid (same
    source file), which does some kind of hash table lookup.

    So your symptoms are consistent with somebody screwing around with those
    next_task links. Note that on Linux, ps gets its information from procfs
    anyway.

  9. Re: hidden processes - how can I investigate

    Lawrence D'Oliveiro wrote:

    > > I just found out that I have some hidden
    > > processes on my computer. (...)

    >
    > Interesting. Looking at the implementation of procfs
    > , (...)


    Thanks a lot! That saved me a huge lot of time! Now I can go on
    investinating *what* (what process, what module, what program) screws
    around with that.

    --
    http://www.piotrsobolewski.w.pl/
    obfus_01@gazeta.pl obfus_02(at)gazeta.pl obfus_03[at]gazeta.pl

  10. Re: hidden processes - how can I investigate

    Mikhail Zotov wrote:

    >> I just found out that I have some hidden processes on my computer. I
    >> found that using chkrootkit (www.chkrootkit.org).

    > ...
    >>
    >> The strange thing is that those hidden processes seem to be absolutely
    >> innocent: that are just six or so instances
    >> of /usr/lib/openoffice/program/soffice.bin.

    >
    > Piotr, may I ask you how did you find these hidden processes?


    $ ./chkproc -v -v

    PID 5116(/proc/5116): not in readdir output
    PID 5116: not in ps output
    CWD 5116: /home/piotr
    EXE 5116: /usr/lib/openoffice/program/soffice.bin

    And, indeed, that processes don't appear in /proc - as I described in
    previous post.

    --
    http://www.piotrsobolewski.w.pl/
    obfus_01@gazeta.pl obfus_02(at)gazeta.pl obfus_03[at]gazeta.pl

  11. Re: hidden processes - how can I investigate

    piotr_sobolewski@nospampse-o2.pl wrote:
    > Mikhail Zotov wrote:
    > > Piotr, may I ask you how did you find these hidden processes?

    >
    > $ ./chkproc -v -v


    Thank you!

    --
    Mikhail


  12. Re: hidden processes - how can I investigate

    If the only processes that chkproc can find are multithreaded procs,
    you probably have an old version of chkproc and those ids are threads.

    Use ps -LAF to find the LWP (a.k.a. SPID) numbers.


+ Reply to Thread