any way to confirm break-in? - Security

This is a discussion on any way to confirm break-in? - Security ; Hi all, My Ubuntu machine, hooked up with the Internet with sshd etc running for access from outside, has suddenly stopped functioning -- doesn't get booted up with the normal runlevel, complaining several library files are missing for basic daemons. ...

+ Reply to Thread
Results 1 to 20 of 20

Thread: any way to confirm break-in?

  1. any way to confirm break-in?

    Hi all,

    My Ubuntu machine, hooked up with the Internet with sshd etc running
    for access from outside, has suddenly stopped functioning -- doesn't
    get booted up with the normal runlevel, complaining several library
    files are missing for basic daemons. I checked the log and found out
    that, just around the time of breakdown, hundreds of attempts were
    made through ssh to access the machine from a couple of IP addresses.
    Before embarking on the repair, therefore, I'd like to find out if the
    machine was in fact broken in somehow. I wonder if anybody could
    advise as to how I could investigate.

    I cannot see any 'successful' login attempt on the authentication log.
    Is it still possible for an outsider to manipulate the system somehow?
    (I guess it is...) There are only three open ports to my machine -- by
    means of rooter firewall but by this means only -- HTTP, VNC and ssh.

    This may be a very basic and vague question, but any help would be
    appreciated. I'd be happy to provide further info if needed.

    Regards,
    Yo


  2. Re: any way to confirm break-in?

    On 29 Apr 2007 02:57:02 -0700, yosato_uk wrote:

    >Hi all,
    >
    >My Ubuntu machine, hooked up with the Internet with sshd etc running
    >for access from outside, has suddenly stopped functioning -- doesn't
    >get booted up with the normal runlevel, complaining several library
    >files are missing for basic daemons. I checked the log and found out
    >that, just around the time of breakdown, hundreds of attempts were
    >made through ssh to access the machine from a couple of IP addresses.
    >Before embarking on the repair, therefore, I'd like to find out if the
    >machine was in fact broken in somehow. I wonder if anybody could
    >advise as to how I could investigate.


    First have a look at http://en.wikipedia.org/wiki/Rootkit it might
    explain what the hacker has used, if they did gain access.

    If your server has been compromised, and it sounds like it might, then
    don't trust *anything*, least of all what the OS is telling you. Spend
    your time backing up your valuable data first then if you want to, try
    downloading a rootkit scanner and see what it detects. But keep the
    machine disconnected from the internet as much as possible - you don't
    know what it has been hijacked to do and you could be relaying
    megabytes of spam.

    >I cannot see any 'successful' login attempt on the authentication log.


    If a hacker did log in successfully you might assume he has covered
    his tracks and removed any trace of that.

    >Is it still possible for an outsider to manipulate the system somehow?


    Highly likely. I am not an expert on Ubuntu and I know it does quite a
    lot to restrict unnecesary root access but if the hacker cracked one
    of your logins and managed to gain root-level access then he could
    have rootkitted you and installed any software he wants.

    I have seen servers with the same symptoms relaying spam and the first
    thing the owners knew was when their IP was blocked by SpamCop.

    >(I guess it is...) There are only three open ports to my machine -- by
    >means of rooter firewall but by this means only -- HTTP, VNC and ssh.


    Before enabling ssh again I would do a few days research on how this
    can be done safely. If you leave ssh configured on the default port 22
    and allow logins to 'root' and do nothing to deny multiple password
    attempts and have relatively weak passwords then it's only a matter of
    time before you get hacked.

    Here is a page taken from Google to get you started:

    http://hostingfu.com/article/ssh-dic...-with-iptables

    Good luck!
    Chris R.

  3. Re: any way to confirm break-in?

    yosato_uk writes:

    >Hi all,


    >My Ubuntu machine, hooked up with the Internet with sshd etc running
    >for access from outside, has suddenly stopped functioning -- doesn't
    >get booted up with the normal runlevel, complaining several library
    >files are missing for basic daemons. I checked the log and found out
    >that, just around the time of breakdown, hundreds of attempts were
    >made through ssh to access the machine from a couple of IP addresses.
    >Before embarking on the repair, therefore, I'd like to find out if the
    >machine was in fact broken in somehow. I wonder if anybody could
    >advise as to how I could investigate.


    Your passwords have to be pretty weak for them to break in this way, since
    they try many many user names and only a few tries for each.
    f they do break in as root they can always erase the log files, so gaps in
    the logs is suspicious. Does apt have the "verify" option that rpm has ( ie
    check the installed files against the md5 sums in the database of installed
    programs?)

    >I cannot see any 'successful' login attempt on the authentication log.
    >Is it still possible for an outsider to manipulate the system somehow?
    >(I guess it is...) There are only three open ports to my machine -- by
    >means of rooter firewall but by this means only -- HTTP, VNC and ssh.


    http is probably the most vulnerable.

    >This may be a very basic and vague question, but any help would be
    >appreciated. I'd be happy to provide further info if needed.


    >Regards,
    >Yo



  4. Re: any way to confirm break-in?

    On Sun, 29 Apr 2007, in the Usenet newsgroup comp.os.linux.security, in article
    , Cheb wrote:

    >On 29 Apr 2007 02:57:02 -0700, yosato_uk wrote:


    >>My Ubuntu machine, hooked up with the Internet with sshd etc running
    >>for access from outside, has suddenly stopped functioning -- doesn't
    >>get booted up with the normal runlevel, complaining several library
    >>files are missing for basic daemons.


    An ordinary user can't screw this up very easily - it takes 'root'
    permissions, and Ubuntu places some limits on root access. Was the
    system being kept up to date?

    >>I checked the log and found out that, just around the time of breakdown,
    >>hundreds of attempts were made through ssh to access the machine from a
    >>couple of IP addresses.


    Welcome to the world of 'dictionary attacks' on SSH daemons - VERY common
    and a reason to place restrictions on access to the SSH ports to those
    address ranges that you believe will have a legitimate reason to connect
    to your SSH server.

    >>Before embarking on the repair, therefore, I'd like to find out if the
    >>machine was in fact broken in somehow. I wonder if anybody could
    >>advise as to how I could investigate.


    Depends on your skill levels - if you are comfortable at the command
    line, there are many things you can look at. The GUI tools are more
    easily compromised to prevent showing the bad news.

    >If your server has been compromised, and it sounds like it might, then
    >don't trust *anything*, least of all what the OS is telling you.


    Absolutely agreed

    >Spend your time backing up your valuable data first


    Yes

    >then if you want to, try downloading a rootkit scanner and see what it
    >detects.


    Here, I disagree. There are two commonly used rootkit scanners available
    (and possibly included in your distribution), which are 'chkrootkit'
    (http://www.chkrootkit.org/) and 'rkhunter' (http://www.rootkit.nl/).
    Neither are worth the CPU cycles it takes to download them, never mind
    the time/effort to run them. They are far more likely to give false alarms
    than find an actual rook kit. While 'rkhunter' attempts to go further,
    both "tools" are looking for symptoms that have been seen in the past as
    indications of an intrusion. However, both do this in an extremely
    simplistic manner, EASILY CIRCUMVENTED by the most inexperienced skript
    kiddy. For example, both look for the presence of a file named
    "/tmp/.../a" or "/tmp/.../r" and on finding either declare that the
    box has a 2003 worm called "55308". Should the skript kiddiez have renamed
    the file to "/tmp/.../A" (or indeed _anything_ else), the worm won't be
    detected. Another example, the tools look for certain strings in certain
    binaries - using the commands "strings $TARGETFILE | grep $BADSTRING",
    and often cause false alarms by finding $BADSTRING as part of a longer
    string (finding "found" in "Newfoundland"). Again, and rootkit author
    can circumvent this by changing the string ("Found" is not the same as
    "found" or "FOUND").

    Another recent poster to this group on the 12th or 13th of this month
    (look back about 50 articles from "now" for a thread "How does one test
    chkrootkit?") discovered that he could _replace_ a binary (/bin/date ?)
    with something else, and chkrootkit didn't notice the substitution. The
    'rkhunter' tool _might_ have detected this, as it does checksum/hashcheck
    some binaries, but again this is subject to LOTS of false alarms. See
    responses in that thread for other solutions.

    >>I cannot see any 'successful' login attempt on the authentication log.

    >
    >If a hacker did log in successfully you might assume he has covered
    >his tracks and removed any trace of that.


    which is why the rootkit detectors are also likely to fail to find a
    break in. The tools are available with source (and are mainly shell
    scripts anyway), and (as I've been stressing) are EASY to circumvent.

    >Highly likely. I am not an expert on Ubuntu and I know it does quite a
    >lot to restrict unnecesary root access but if the hacker cracked one
    >of your logins and managed to gain root-level access then he could
    >have rootkitted you and installed any software he wants.


    Yes - in years past, one need only add a line to /etc/passwd for a
    root-equivalent account, and the box is 0wn3d!!!

    >Before enabling ssh again I would do a few days research on how this
    >can be done safely.


    Definitely agreed again. Both of you are posting from UK providers,
    suggesting you are in Britain. Is there any reason you can think of
    that you will need to log in to your system from South America, Asia,
    or even the continent? Use a firewall rule to block access to the
    ssh port and only allow access from IP ranges you may expect you will
    actually be using. (Note: IP address ranges are not arranged in nice
    convenient order - the best you can do is to substantially reduce the
    number of chances. See http://www.iana.org/assignments/ipv4-address-space
    and you own logs for clues of what you can block.) Likewise, are you
    allowing ssh access to people who are to unskilled to know how to tell
    their clients to use a different port number than the default '22'? If
    they have the skills, DON'T USE THE DEFAULT PORT 22. "Well known ports"
    exist so that people who don't know better can find a service. If you
    don't want people to automatically find your server, move it elsewhere
    (preferably above port 1024, and not a s3kr3t number only a 3L33t h4x0r
    kiddie can remember, like 12345). What's a good port number to use?

    [compton ~]$ head -2 /dev/random | mimencode | tr -d 'A-z/+' | column
    93285798 057134614 41126395029913 67274730
    03984161308677 379491685 36299082976038 146914677
    68815375230003 4144452 90087622526 34724849==
    [compton ~]$

    pick a port number between 1025 and 65500 from those digits. Some may
    think of this as "security by obscurity", and in a way it is. But because
    you still require the correct username and authentication token(s) on
    the new port number, the only thing moving the port is going to do is to
    avoid nearly all of the automatic/scripted attacks that target the default
    port number.

    Old guy

  5. Re: any way to confirm break-in?



    >My Ubuntu machine, hooked up with the Internet with sshd etc running
    >for access from outside, has suddenly stopped functioning -- doesn't
    >get booted up with the normal runlevel, complaining several library
    >files are missing for basic daemons. I checked the log and found out
    >that, just around the time of breakdown, hundreds of attempts were
    >made through ssh to access the machine from a couple of IP addresses.

    ....
    >I cannot see any 'successful' login attempt on the authentication log.
    >Is it still possible for an outsider to manipulate the system somehow?
    >(I guess it is...) There are only three open ports to my machine -- by
    >means of rooter firewall but by this means only -- HTTP, VNC and ssh.


    Were any of your passwords guessable? If not, ssh probably isn't
    the problem.

    --
    These are my opinions, not necessarily my employer's. I hate spam.


  6. Re: any way to confirm break-in?

    On Sun, 29 Apr 2007 15:58:44 -0500
    ibuprofin@painkiller.example.tld (Moe Trin) wrote:

    > >On 29 Apr 2007 02:57:02 -0700, yosato_uk wrote:

    >
    > >>My Ubuntu machine, hooked up with the Internet with sshd etc running
    > >>for access from outside, has suddenly stopped functioning -- doesn't
    > >>get booted up with the normal runlevel, complaining several library
    > >>files are missing for basic daemons.

    ....
    > Definitely agreed again. Both of you are posting from UK providers,
    > suggesting you are in Britain. Is there any reason you can think of
    > that you will need to log in to your system from South America, Asia,
    > or even the continent? Use a firewall rule to block access to the
    > ssh port and only allow access from IP ranges you may expect you will
    > actually be using. (Note: IP address ranges are not arranged in nice
    > convenient order - the best you can do is to substantially reduce the
    > number of chances. See http://www.iana.org/assignments/ipv4-address-space
    > and you own logs for clues of what you can block.)


    This site can be helpful if you need to find IP addresses used in
    a particular country:

    http://software77.net/cgi-bin/ip-country/geo-ip.pl

    Be aware though that information isn't always "exact" because
    1) there are domains like "EU", 2) a server registered in, say, UK,
    can be physically located in another country, etc.

    --
    M.


  7. Re: any way to confirm break-in?

    Thanks for the info from quite a few people in a short space of time
    -- all of which is very useful.

    The inevitable conclusion seems to be it is not so straightforward
    even just to ascertain a break-in, if the attacker's as clever as is
    expected, though there is strong circumstantial evidence... And I've
    got the feeling that ssh is *not* the route through which the attacker
    gained access, if they indeed did --- my passwds are much stronger
    than the names tried in the failed attempts, and I'm sure that they
    would have remove *all* the log files. So... exluding pure
    coincidence, my guess is the same attacker that tried the ssh route
    found some other route.

    Now I'm not so much interested in hunting for the culprit as
    preventing a further problem. I will try and enhance the security the
    various ways suggested, but the remaining worry is that there might be
    some malware or worm that is left on the system. If the usual tools
    may fail to detect them, is there any better way? If there's no way to
    be absolutely sure, I'd in fact clean-reinstall the system altogether
    and recover the backed up data. But I presume there still would remain
    some risk, if some malware's mixed into the data directory... so
    probably a rather naive question again: any (fast enough) way to
    transfer data during which you can verify safety?

    Regards,
    Yo


  8. Re: any way to confirm break-in?

    On 30 Apr 2007 03:58:38 -0700, yosato_uk wrote:
    >my guess is the same attacker that tried the ssh route
    >found some other route.


    Actually, working from log evidence is very hard. SSH is attacked
    routinely my botnets and script-kiddies all around the world, so if
    you have ssh on port 22 you will always have a full log file of silly
    attempts. However, if someone gets in they might be clever enough to
    cover all of their tracks and leave the 'normal' ssh probes in the log
    files just to make it look like any other day.

    >may fail to detect them, is there any better way? If there's no way to
    >be absolutely sure, I'd in fact clean-reinstall the system altogether
    >and recover the backed up data.


    This is the only safe way to do it.

    If you are worried about the folders of data you could: back it all
    up; reinstall the OS; donload a free Linux antivirus prog (eg.
    AntiVir) and then virus-scan the data as it comes down. However,
    copying down some data off a DVD won't activate any malware - you'd
    have to execute something there, like a Trojan Horse. So, I'd just
    download it to a temporary folder - scan it thoroughly and then move
    it into the 'live' folder structure when you are happy it is clean.

    Chris R.

  9. Re: any way to confirm break-in?

    yosato_uk writes:

    >Thanks for the info from quite a few people in a short space of time
    >-- all of which is very useful.


    >The inevitable conclusion seems to be it is not so straightforward
    >even just to ascertain a break-in, if the attacker's as clever as is
    >expected, though there is strong circumstantial evidence... And I've
    >got the feeling that ssh is *not* the route through which the attacker
    >gained access, if they indeed did --- my passwds are much stronger
    >than the names tried in the failed attempts, and I'm sure that they
    >would have remove *all* the log files. So... exluding pure
    >coincidence, my guess is the same attacker that tried the ssh route
    >found some other route.


    >Now I'm not so much interested in hunting for the culprit as
    >preventing a further problem. I will try and enhance the security the
    >various ways suggested, but the remaining worry is that there might be
    >some malware or worm that is left on the system. If the usual tools
    >may fail to detect them, is there any better way? If there's no way to
    >be absolutely sure, I'd in fact clean-reinstall the system altogether
    >and recover the backed up data. But I presume there still would remain
    >some risk, if some malware's mixed into the data directory... so
    >probably a rather naive question again: any (fast enough) way to
    >transfer data during which you can verify safety?



    As a minimum scan the backup for suid programs. There almost certainly
    should be none. At least examine any very carefully.
    find /backup -perm /6000 -ls


  10. Re: any way to confirm break-in?

    On Mon, 30 Apr 2007, in the Usenet newsgroup comp.os.linux.security, in article
    <20070430114646.31fab8f3.invalid_muxaul@lenta.ru>, Mikhail Zotov wrote:

    >ibuprofin@painkiller.example.tld (Moe Trin) wrote:


    >> (Note: IP address ranges are not arranged in nice convenient order
    >> - the best you can do is to substantially reduce the number of
    >> chances. See http://www.iana.org/assignments/ipv4-address-space and
    >> your own logs for clues of what you can block.)

    >
    >This site can be helpful if you need to find IP addresses used in
    >a particular country:
    >
    >http://software77.net/cgi-bin/ip-country/geo-ip.pl


    Looks like it's based on RIR data, which can be tricky. I normally
    use a whois query to the "appropriate" whois server (see that IANA
    web page).

    Everyone picks on China or Korea as a source of bad packets, and you
    are frequently recommended to block those addresses. There are problems.
    1. The IP addresses assigned/allocated to country X are not contiguous,
    but are scattered.

    [compton ~]$ zgrep CN IP.ADDR/stats/[ALR]* | cut -d' ' -f2 | cut -d'.'
    -f1 | sort -n | uniq -c | column
    43 58 23 122 1 161 1 198 41 219
    32 59 44 123 1 162 320 202 16 220
    38 60 63 124 1 166 92 203 63 221
    84 61 38 125 1 167 72 210 64 222
    24 116 1 134 1 168 41 211
    34 121 1 159 4 192 64 218
    [compton ~]$

    First octet of the addresses for China. Korea is even worse:

    [compton ~]$ zgrep KR IP.ADDR/stats/[ALR]* | cut -d' ' -f2 | cut -d'.'
    -f1 | sort -n | uniq -c | column
    15 58 15 125 2 152 10 165 85 211
    6 59 1 128 1 154 4 166 10 218
    1 60 1 129 1 155 8 168 2 219
    17 61 1 134 1 156 1 169 10 220
    2 116 1 137 1 157 24 192 8 221
    23 121 1 141 1 158 32 202 7 222
    23 122 1 143 1 161 56 203
    18 123 4 147 3 163 1 206
    22 124 3 150 2 164 81 210
    [compton ~]$

    But when you look at those /8s, you find something like::

    [compton ~]$ zgrep -h ' 58\.' IP.ADDR/stats/[ALR]* | cut -d' ' -f1 |
    sort | uniq -c | column
    1 AF 4 HK 15 KR 5 PK 1 VN
    24 AU 3 ID 4 MY 6 SG
    5 BD 3 IN 2 NZ 8 TH
    43 CN 29 JP 2 PH 4 TW
    [compton ~]$

    So it's not simple. (The files I'm grepping here are locally massaged
    versions of the five RIR zone files.) Part of this is due to the
    different dates that the RIRs were formed - ARIN came first, followed by
    RIPE and APNIC. LACNIC was split out of ARIN, and AFRINIC out of RIPE.
    Part is due to the old Classful allocations (there is no Class A, B, or
    C any more - hasn't been since Classless Inter-Domain Routing was
    implemented in 1993). Part is because the Internet is to _facilitate_
    connections, not prevent them. Thus. again looking at the first octet of
    the IP address, we find

    24 ARIN LACNIC RIPE 62 AFRINIC RIPE
    64 AFRINIC ARIN LACNIC 66 AFRINIC ARIN LACNIC
    69 AFRINIC ARIN 80 AFRINIC RIPE
    81 AFRINIC RIPE 82 AFRINIC RIPE
    83 AFRINIC RIPE 84 AFRINIC RIPE
    128 APNIC ARIN RIPE 129 APNIC ARIN LACNIC RIPE
    130 APNIC ARIN RIPE 131 APNIC ARIN LACNIC RIPE
    132 APNIC ARIN LACNIC RIPE 134 APNIC ARIN RIPE
    135 ARIN RIPE 136 APNIC ARIN RIPE
    137 AFRINIC APNIC ARIN RIPE 138 APNIC ARIN RIPE
    139 AFRINIC APNIC ARIN LACNIC RIPE 140 APNIC ARIN LACNIC RIPE
    141 APNIC ARIN RIPE 143 AFRINIC APNIC ARIN LACNIC RIPE
    144 APNIC ARIN LACNIC RIPE 146 AFRINIC APNIC ARIN LACNIC RIPE
    147 AFRINIC APNIC ARIN LACNIC RIPE 148 APNIC ARIN LACNIC RIPE
    149 APNIC ARIN RIPE 150 APNIC ARIN LACNIC RIPE
    151 APNIC ARIN RIPE 152 AFRINIC APNIC ARIN LACNIC RIPE
    153 APNIC ARIN RIPE 154 APNIC ARIN RIPE
    155 AFRINIC APNIC ARIN LACNIC RIPE 156 AFRINIC APNIC ARIN LACNIC RIPE
    157 APNIC ARIN LACNIC RIPE 158 APNIC ARIN LACNIC RIPE
    159 APNIC ARIN LACNIC RIPE 160 AFRINIC APNIC ARIN RIPE
    161 APNIC ARIN LACNIC RIPE 162 APNIC ARIN LACNIC RIPE
    163 AFRINIC APNIC ARIN LACNIC RIPE 164 AFRINIC APNIC ARIN LACNIC RIPE
    165 AFRINIC APNIC ARIN LACNIC RIPE 166 AFRINIC APNIC ARIN LACNIC RIPE
    167 APNIC ARIN LACNIC RIPE 168 AFRINIC APNIC ARIN LACNIC RIPE
    169 AFRINIC APNIC ARIN LACNIC 170 APNIC ARIN LACNIC RIPE
    171 ARIN RIPE 192 AFRINIC APNIC ARIN LACNIC RIPE
    193 AFRINIC RIPE 194 AFRINIC RIPE
    195 AFRINIC RIPE 196 AFRINIC APNIC ARIN LACNIC RIPE
    198 AFRINIC APNIC ARIN LACNIC RIPE 199 ARIN LACNIC
    200 AFRINIC ARIN LACNIC 202 AFRINIC APNIC
    204 AFRINIC ARIN LACNIC 205 AFRINIC ARIN LACNIC
    206 AFRINIC ARIN LACNIC 207 ARIN LACNIC
    209 AFRINIC ARIN LACNIC 212 AFRINIC RIPE
    213 AFRINIC RIPE 216 AFRINIC ARIN LACNIC
    217 AFRINIC RIPE

    2. "You can block by top level domain name, right?" No, for two reasons.
    Contrary to popular belief, ".com", ".edu". ".net' and ".org" (never mind
    the other domains like '.biz", ".info" and so on) are not restricted to
    the USA. There are ".com" domains in virtually _every_ country. Second.
    and this is particularly true in countries with "bad" reputations, a lot
    of those network administrators don't think it necessary to follow the
    requirements of RFC1034, 2050 and 2181, and don't bother creating PTR
    records (rDNS, or "IP to hostname"). In fact, many mail servers are now
    configured to reject _connections_ from hosts that don't have resolvable
    IP addresses, and some people are going further to checking that the PTR
    record does match the A (name to IP) record. That trick alone cut my
    spam load by 70 percent as it eliminated a lot of 'zombie' mail.

    >Be aware though that information isn't always "exact" because
    >1) there are domains like "EU",


    [compton ~]$ zgrep -hE '(AP|EU)' [ALR]* | cut -d' ' -f1 | sort | uniq -c |
    column
    87 AP 4697 EU
    [compton ~]$

    That's out of a total of about 80,000 listings.

    >2) a server registered in, say, UK, can be physically located in another
    >country, etc.


    The domain I work at is registered in New York, but if you were to
    traceroute to my system, the last host you see before the black hole
    is a backbone router near San Francisco, even though I am near Phoenix
    (600 KM East of Los Angeles) and we have subnets on six continents and
    over forty countries. The next subnet above mine is in France. IP
    address to geographical location is EXTREMELY haphazard and not defined.

    So - what's the firewall solution? Do not _allow_ everything by
    default. I don't know about you, but I don't offer SSH to everyone
    in the world. For SSH at home, I accept connections from exactly 1545
    IP addresses (a /22. two /24s, and nine specific hosts) ONLY. If I'm
    going on a trip and need to allow access from other addresses, I can
    do so - but for now - 1545 addresses (out of 3.71e+09 IPv4 Internet
    addresses, 2.47e+09 actually assigned/allocated world wide). That
    cuts down the "noise" considerably.

    Old guy

  11. Re: any way to confirm break-in?

    On 30 Apr 2007, in the Usenet newsgroup comp.os.linux.security, in article
    <1177930718.337989.300510@n76g2000hsh.googlegroups. com>, yosato_uk wrote:

    >The inevitable conclusion seems to be it is not so straightforward
    >even just to ascertain a break-in, if the attacker's as clever as is
    >expected, though there is strong circumstantial evidence...


    One technique with a reasonable chance of success is to compare the
    file system with a known clean version. The original version of this
    type of test is 'tripwire' currently unsupported - and the derivatives
    such as 'aide'. Briefly, you run this application on a clean box to get
    checksums and hashes of EVERYTHING. Thereafter, you take the file system
    off line, and run the same checksums and hashes using a "trusted" copy
    of an operating system, and compare these with the original data. Is
    anything different? The two problems with this is that you have to
    create the reference data from a known clean installation, AND keep it
    up to date with any intentional changes (such as security updates),
    and that you should run the tests with a known secure O/S (which
    might be a floppy that also has the needed tools) rather than the
    "normal" O/S (which may have been compromised).

    >And I've got the feeling that ssh is *not* the route through which the
    >attacker gained access, if they indeed did --- my passwds are much
    >stronger than the names tried in the failed attempts,


    What else is running on the system? A web server, especially one with
    dynamic content/scripting is FAR more likely. Is everything up to date?

    >and I'm sure that they would have remove *all* the log files.


    It's terribly easy to edit logs - and a common skript-kiddy technique.

    >So... exluding pure coincidence, my guess is the same attacker that
    >tried the ssh route found some other route.


    Or someone else did - there's more than a few skript kiddiez out there.

    >Now I'm not so much interested in hunting for the culprit as
    >preventing a further problem. I will try and enhance the security the
    >various ways suggested,


    Good

    >but the remaining worry is that there might be some malware or worm
    >that is left on the system. If the usual tools may fail to detect them,
    >is there any better way?


    Depends. My usual trick is to use the package manager to check the
    system ('/bin/rpm -V' for an rpm based system, 'debsums -s' for a
    Debian based system --> even though the debsums man-page depreciates
    this mode). First, check the tool by substituting some other file for
    a binary that the intruder would normally alter:

    /bin/mv /bin/ps /bin/ps.original
    /bin/mv /bin/less /bin/ps
    rpm -Vf /bin/ps or debsums -s procps
    /bin/mv /bin/ps.original /bin/ps

    DON'T NEGLECT THAT LAST STEP!!! Did your package manager freak out
    when it discovered that /bin/ps doesn't look like /bin/ps _ought_ to
    look like? If no, you're toast. If yes, then move on to the next step
    in the Jedi Mind Games.

    Now, if you _had_ followed the last paragraph in the CAVEATS section of
    the 'debsums' man page

    If you are looking for an integrity checker that can run from safe
    media, do integrity checks on checksum databases and can be easily
    configured to run periodically to warn the admin of changes see other
    tools such as: aide, integrit, samhain, or tripwire.

    you might have other options. Isn't hindsight wonderful? ;-)

    >If there's no way to be absolutely sure, I'd in fact clean-reinstall
    >the system altogether and recover the backed up data.


    That is the most reliable solution. Ideally, the backed up data was
    also clean. I make backups when _I_ change things on the server, not
    as an automatic function. Thus, my backups should reflect that last
    known "good" situation (I hope). ;-)

    >But I presume there still would remain some risk, if some malware's
    >mixed into the data directory...


    Your "clean install" should reduce a lot of this risk - unless this is
    some scripting/database/what-ever that might be executable as part of
    (for example) your web page. You also have the issue that you haven't
    corrected what-ever hole was exploited.

    >so probably a rather naive question again: any (fast enough) way to
    >transfer data during which you can verify safety?


    The integrity checkers like aide and so-on can do this, BUT they need
    a "clean" starting point. By the way, now you may know why our publicly
    visible systems all run from "read-only" media, and have only the
    minimum "stuff" installed to do their job. No bells, no whistles, no
    frilly knickers.

    Aren't "learning experiences" wonderful?

    Old guy

  12. Re: any way to confirm break-in?

    Thank you for the very interesting post!

    On Mon, 30 Apr 2007 19:51:27 -0500
    ibuprofin@painkiller.example.tld (Moe Trin) wrote:
    ....
    > So - what's the firewall solution? Do not _allow_ everything by
    > default. I don't know about you, but I don't offer SSH to everyone
    > in the world. For SSH at home, I accept connections from exactly 1545
    > IP addresses (a /22. two /24s, and nine specific hosts) ONLY.


    What is your opinion about port knocking and/or single packet authorization?
    I have checked links available at portknocking.org and it seems the
    majority of projects are in a stale condition (except fwknop and a
    couple of projects at alpha stages).

    Regards,
    Mikhail

  13. Re: any way to confirm break-in?

    On 30 Apr, 16:47, Unruh wrote:
    > yosato_uk writes:
    > >Now I'm not so much interested in hunting for the culprit as
    > >preventing a further problem. I will try and enhance the security the
    > >various ways suggested, but the remaining worry is that there might be
    > >some malware or worm that is left on the system. If the usual tools
    > >may fail to detect them, is there any better way? If there's no way to
    > >be absolutely sure, I'd in fact clean-reinstall the system altogether
    > >and recover the backed up data. But I presume there still would remain
    > >some risk, if some malware's mixed into the data directory... so
    > >probably a rather naive question again: any (fast enough) way to
    > >transfer data during which you can verify safety?

    >
    > As a minimum scan the backup for suid programs. There almost certainly
    > should be none. At least examine any very carefully.
    > find /backup -perm /6000 -ls


    erk, IMHO not up to your usual quality of response, U

    If one strongly suspects their machine has been compromised and / or
    its got trashed, a format / re-install is called for. This is where
    you start wishing you'd been running an IDS like tripwire or L5. By
    all means mount the drive read-only on an already running, clean
    system and have a poke around for setuid files / rootkit detectors,
    but the quickest route to getting your system back to a trusted state
    is probably going to be by vetting stuff you carry forward to the new
    installation, not trying to find what's been tampered with in the old
    one - there's just too much to check and too much risk of missing
    something.

    OP: you can see this flurry of log entries - but it does not
    necessarilty follow that this corresponds with when your machine was
    compromised - you should do a clean install and only restore your own
    files from backup (not ones which were from the previous install) and
    you should check them for interference using a rootkit detector.

    C.


  14. Re: any way to confirm break-in?

    On 2007-04-29, yosato_uk wrote:

    > This may be a very basic and vague question, but any help would be
    > appreciated. I'd be happy to provide further info if needed.


    There may be some left on the newsstand, if you hurry. But if not,
    you might consider ordering this back issue of Linux-Pro magazine.

    http://www.linux-magazine.com/issue/77

    I don't often buy periodicals anymore, especially pricey one's like
    L-P, but this issue had some very good info on back doors and
    real-time IDS techniques (lsof). Maybe a friend has an issue.

    nb

  15. Re: any way to confirm break-in?

    "C." writes:

    >On 30 Apr, 16:47, Unruh wrote:
    >> yosato_uk writes:
    >> >Now I'm not so much interested in hunting for the culprit as
    >> >preventing a further problem. I will try and enhance the security the
    >> >various ways suggested, but the remaining worry is that there might be
    >> >some malware or worm that is left on the system. If the usual tools
    >> >may fail to detect them, is there any better way? If there's no way to
    >> >be absolutely sure, I'd in fact clean-reinstall the system altogether
    >> >and recover the backed up data. But I presume there still would remain
    >> >some risk, if some malware's mixed into the data directory... so
    >> >probably a rather naive question again: any (fast enough) way to
    >> >transfer data during which you can verify safety?

    >>
    >> As a minimum scan the backup for suid programs. There almost certainly
    >> should be none. At least examine any very carefully.
    >> find /backup -perm /6000 -ls


    >erk, IMHO not up to your usual quality of response, U


    >If one strongly suspects their machine has been compromised and / or
    >its got trashed, a format / re-install is called for. This is where
    >you start wishing you'd been running an IDS like tripwire or L5. By


    It would really really help if you read both the post and the response.
    This comment was in response to the suggestion to reinstall and then put in
    place the backup of /home. That backup itself should be scanned.

    >all means mount the drive read-only on an already running, clean
    >system and have a poke around for setuid files / rootkit detectors,


    Sorry, you are of the impression that files can run themselves? Why do you
    want to mount the drive on clean system ro?

    My adivce would be.
    a) reinstall the operating system.
    b) scan all of the stuff that you did NOT reinstall for suid/sgid files.
    All. /tmp, /var/* /home, /usr/local.
    I once found a file /tmp/banana which was a root suid shell.


    >but the quickest route to getting your system back to a trusted state
    >is probably going to be by vetting stuff you carry forward to the new
    >installation, not trying to find what's been tampered with in the old
    >one - there's just too much to check and too much risk of missing
    >something.


    Precisely what I was saying-- vet the stuff you carry forward.


    >OP: you can see this flurry of log entries - but it does not
    >necessarilty follow that this corresponds with when your machine was
    >compromised - you should do a clean install and only restore your own
    >files from backup (not ones which were from the previous install) and
    >you should check them for interference using a rootkit detector.


    Well, first you should try to make sure that your machine actually was
    complrimised. If you reinstall every time something suspicious appears in a
    log file, you will never use your computer for anything worthwhile. It will
    be a daily excercise.

    If you have a rpm based system, first do
    rpm -Va>/tmp/verify
    and then look through those files for chnged files. If you find one that
    you cannot explain, you are probably comprimised.

    Then look for suid file. Look everywhere. Make sure that those file should
    be suid files. If you find an suid root file in /dev, /tmp, etc, then you
    have been comprimised.

    Look at the log files for suspicious blanks.

    >C.



  16. Re: any way to confirm break-in?

    On Tue, 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
    <20070501074148.15276527.invalid_muxaul@lenta.ru>, Mikhail Zotov wrote:

    >Thank you for the very interesting post!


    You are welcome!

    >ibuprofin@painkiller.example.tld (Moe Trin) wrote:
    >...
    >> So - what's the firewall solution? Do not _allow_ everything by
    >> default. I don't know about you, but I don't offer SSH to everyone
    >> in the world. For SSH at home, I accept connections from exactly 1545
    >> IP addresses (a /22. two /24s, and nine specific hosts) ONLY.


    >What is your opinion about port knocking and/or single packet authorization?


    ]] If I'm going on a trip and need to allow access from other addresses,
    ]] I can do so

    Port knocking is another layer - and relatively simple to add to the
    non-standard server port number. From my Sunday post to get some numbers
    to use here:

    [compton ~]$ head -2 /dev/random | mimencode | tr -d 'A-z/+' | column
    93285798 057134614 41126395029913 67274730
    03984161308677 379491685 36299082976038 146914677
    68815375230003 4144452 90087622526 34724849==
    [compton ~]$

    The knocking scheme I've used is crude - listen for packets on port
    39841 (which is closed), and on the firewall seeing such packet, open
    port 13086 (where the SSH server is hiding) for the IP address that
    had hit the first port for one minute. Also set two "trip" ports just
    above and just below the SSH port - if the knocking address hits those
    ports, close 13086 under the premise that a port scanner hit the knock
    port, then continued to scan. The trip ports will ensure that the port
    scanner closes the SSH port before he gets to it unless the scanner is
    hitting random numbers, and the Gods of Chance are frowning at me.

    PORT KNOCKING IS NOT A SUBSTITUTE FOR PROPER AUTHENTICATION. For those
    with a knee-jerk reaction to "hiding things" being security through
    obscurity, shove it! Port knocking, like moving the server to an obscure
    port number, or using firewall rules to harshly restrict access to a
    minimum number of addresses, is an _addition_ to what ever security
    functions (proper authentication, using 'non-dictionary/non-phone-book'
    usernames, strong authentication keys, etc.) that you _should_ have
    had in place before reading this. All you are trying to do is to make
    it more difficult. Grabbing numbers out of the sky - assume there is a
    single chance in a thousand of the bad guy guessing the right username
    (you DO NOT use 'root' or 'admin' or your 'public_username'), Assume
    one chance in a thousand or the back guy independently guessing your
    non-dictionary password. Thus, a robot has one chance in a million
    of guessing both. But this will do the robot little good if it must
    also guess which of the sixty-thousand ports you've hidden the server
    at - and port knocking merely makes it harder for the bad guy to stumble
    across your hidden server. Yes, I am aware of packet sniffing, and
    there are additional steps that can be taken to reduce that risk.

    The one problem you want to avoid is making what ever scheme you use to
    complicated. You _will_ screw up, and lock yourself out - and you won't
    be the first person to do so. I encountered a situation several years
    ago where the simple port-knock wouldn't work. I gave up after the third
    try, but later investigation of the logs showed the knock was being
    intercepted by a proxy server, while my SSH attempt was going straight
    through - the proxy server had (naturally) a different IP address.

    >I have checked links available at portknocking.org and it seems the
    >majority of projects are in a stale condition (except fwknop and a
    >couple of projects at alpha stages).


    I've looked at a number of the projects (though not recently) and tend
    to agree. The scheme used doesn't have to be complicated - my original
    knocking "daemon" was a firewall rule to log to a special file any
    connection attempts to port $FOO, and a cron script that looked for
    that file - and on finding it, parsed the IP, set a firewall rule for
    port $BAR from that IP, and set a timer to clear that rule a minute
    later. The firewall 'ESTABLISHED" rule allowed the "conversation to
    continue. Security does not have to be complicated, or cute. Simple
    steps _can_ do a lot for you as long as you use some common sense. Oh,
    and you _do_ remember to change that password (and username if you are
    ultra-paranoid) on a regular basis, right? Just don't over do it.

    Old guy


  17. Re: any way to confirm break-in?

    On 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
    <1178020943.501703.59150@l77g2000hsb.googlegroups.c om>, C. wrote:

    >Unruh wrote:


    >> As a minimum scan the backup for suid programs. There almost
    >> certainly should be none. At least examine any very carefully.
    >> find /backup -perm /6000 -ls


    >erk, IMHO not up to your usual quality of response, U


    Well, he did say it was the very minimum - but yes it's preferable not
    to have to recover from the contaminated backups. Doing automated
    nightly or weekly backups may not be the best scheme. Backing up
    immediately after _you_ make some change is safer, and safer still
    is having those originals created on an isolated box, with the
    server running from 'read-only' copies of that source.

    >If one strongly suspects their machine has been compromised and / or
    >its got trashed, a format / re-install is called for. This is where
    >you start wishing you'd been running an IDS like tripwire or L5.


    Agreed

    >By all means mount the drive read-only on an already running, clean
    >system and have a poke around for setuid files / rootkit detectors,


    though this is for educational purposes (how did they do it) only.

    >but the quickest route to getting your system back to a trusted state
    >is probably going to be by vetting stuff you carry forward to the new
    >installation, not trying to find what's been tampered with in the old
    >one - there's just too much to check and too much risk of missing
    >something.


    Talking about a public server here - a work station should not be
    running public severs, and a server should not have users allowed
    to log in - the data for the server should be developed/tested on
    a separate box, whether it be web pages, files for a file server, or
    what-ever. Getting the system back to the trusted state then only
    involves a clean install of the operating system and applications,
    and then copying the data from the development system. Running the
    server application such that the data is owned by a specific
    non-privileged user AND GROUP allows additional "does this smell right"
    checks to be run.

    Old guy

  18. Re: any way to confirm break-in?

    ibuprofin@painkiller.example.tld (Moe Trin) writes:

    >On 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
    ><1178020943.501703.59150@l77g2000hsb.googlegroups.c om>, C. wrote:


    >>Unruh wrote:


    >>> As a minimum scan the backup for suid programs. There almost
    >>> certainly should be none. At least examine any very carefully.
    >>> find /backup -perm /6000 -ls


    >>erk, IMHO not up to your usual quality of response, U


    >Well, he did say it was the very minimum - but yes it's preferable not
    >to have to recover from the contaminated backups. Doing automated
    >nightly or weekly backups may not be the best scheme. Backing up
    >immediately after _you_ make some change is safer, and safer still
    >is having those originals created on an isolated box, with the
    >server running from 'read-only' copies of that source.


    The breakin could have happened a month ago and was just utilized/screwed
    up last night. If you are broken into, do not trust anything, including
    your backups. That is why scanning your backups, no matter when made, is a
    minimum.
    Note that I am NOT advocating just doing this.


  19. Re: any way to confirm break-in?

    yosato_uk (07-04-30 03:58:38):

    > The inevitable conclusion seems to be it is not so straightforward
    > even just to ascertain a break-in, if the attacker's as clever as is
    > expected, though there is strong circumstantial evidence...


    As abstract as I like to be, the problem is that for a computer, there
    is no distinction of `compromised' vs. `not compormised'. Everybody has
    a different idea of where this distinction occurs. This is why there
    can't be a perfect rootkit scanner or break-in detector.

    In other words: To check for a break-in, you need to minimize the scope
    of the effect of a possible break-in (like changed files, BIOS, system
    time, etc.) to a certain area of your computer (like all executable
    files on your hard-disk), learn how this area works, and check it
    yourself. Because this is impractical and error-prone, you'll do better
    just reinstalling your operating system on all possibly affected hosts,
    backing up all important data.


    > And I've got the feeling that ssh is *not* the route through which the
    > attacker gained access, if they indeed did --- my passwds are much
    > stronger than the names tried in the failed attempts, and I'm sure
    > that they would have remove *all* the log files. So... exluding pure
    > coincidence, my guess is the same attacker that tried the ssh route
    > found some other route.


    Perhaps your system was compromised long before that word-list attack
    happened. Perhaps it has been compromised later. Probably it hasn't
    been compormised at all. Sometimes system updates, which include OS
    components, break things.


    > Now I'm not so much interested in hunting for the culprit as
    > preventing a further problem. I will try and enhance the security the
    > various ways suggested, [...]


    You will want to use proper authentication methods, and restrict,
    restrict, restrict everything you can for people not supposed to access
    your hosts. In the best case, a random attacker (i.e. script kiddie)
    doesn't even know your network exists.


    > [...] but the remaining worry is that there might be some malware or
    > worm that is left on the system. If the usual tools may fail to detect
    > them, is there any better way? If there's no way to be absolutely
    > sure, I'd in fact clean-reinstall the system altogether and recover
    > the backed up data.


    Exactly. There is no way you can know, besides checking manually, which
    is impractical. So it's best to do drop your current installation
    completely.


    > But I presume there still would remain some risk, if some malware's
    > mixed into the data directory... so probably a rather naive question
    > again: any (fast enough) way to transfer data during which you can
    > verify safety?


    No automated way. Firstly you need to separate functional data from
    non-functional data. Shell scripts and often configuration files are
    functional. A JPEG image generally isn't (although with a suitable
    viewer and EXIF, it might).

    Whether data is functional or not is not always as obvious. Take care,
    and if uncertain, always assume functional data. For example, you
    wouldn't consider your Quake save-games to be functional, but in fact
    they are! Functional data is not necessarily executable.

    For all functional files, you need to check manually whether they have
    been manipulated. Depending on the number of functional files you have,
    this is going to take a long time. After you've finished checking, make
    sure you use secure versions of viewer programs.

    Of course this is überparanoid and certainly extremely expensive and
    time consuming, but it's the only way to make sure. Always assume that
    the attacker has had compromised your system long before you noticed
    there's something wrong.


    Regards,
    Ertugrul Söylemez.


    --
    From the fact that this CGI program has been written in Haskell, it
    follows naturally that this CGI program is perfectly secure.

  20. Re: any way to confirm break-in?

    On Tue, 01 May 2007 14:58:31 -0500
    ibuprofin@painkiller.example.tld (Moe Trin) wrote:

    > On Tue, 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
    > <20070501074148.15276527.invalid_muxaul@lenta.ru>, Mikhail Zotov wrote:

    ....
    > >What is your opinion about port knocking and/or single packet authorization?

    >

    ....
    > Port knocking is another layer - and relatively simple to add to the
    > non-standard server port number. From my Sunday post to get some numbers
    > to use here:
    >
    > [compton ~]$ head -2 /dev/random | mimencode | tr -d 'A-z/+' | column

    ....

    Cute :-)

    ....
    > The one problem you want to avoid is making what ever scheme you use to
    > complicated. You _will_ screw up, and lock yourself out - and you won't
    > be the first person to do so.


    LOL :-) This happened one, this happened twice... How more times will
    this happen?

    > ... Security does not have to be complicated, or cute. Simple
    > steps _can_ do a lot for you as long as you use some common sense. Oh,
    > and you _do_ remember to change that password (and username if you are
    > ultra-paranoid) on a regular basis, right? Just don't over do it.


    Thank you! It's a big pleasure to study your posts!

    Regards,
    Mikhail

+ Reply to Thread