On 29 Apr 2007 02:57:02 -0700, yosato_uk wrote:

>Hi all,
>
>My Ubuntu machine, hooked up with the Internet with sshd etc running
>for access from outside, has suddenly stopped functioning -- doesn't
>get booted up with the normal runlevel, complaining several library
>files are missing for basic daemons. I checked the log and found out
>that, just around the time of breakdown, hundreds of attempts were
>made through ssh to access the machine from a couple of IP addresses.
>Before embarking on the repair, therefore, I'd like to find out if the
>machine was in fact broken in somehow. I wonder if anybody could
>advise as to how I could investigate.

First have a look at http://en.wikipedia.org/wiki/Rootkit it might
explain what the hacker has used, if they did gain access.

If your server has been compromised, and it sounds like it might, then
don't trust *anything*, least of all what the OS is telling you. Spend
your time backing up your valuable data first then if you want to, try
downloading a rootkit scanner and see what it detects. But keep the
machine disconnected from the internet as much as possible - you don't
know what it has been hijacked to do and you could be relaying
megabytes of spam.

>I cannot see any 'successful' login attempt on the authentication log.

If a hacker did log in successfully you might assume he has covered
his tracks and removed any trace of that.

>Is it still possible for an outsider to manipulate the system somehow?

Highly likely. I am not an expert on Ubuntu and I know it does quite a
lot to restrict unnecesary root access but if the hacker cracked one
of your logins and managed to gain root-level access then he could
have rootkitted you and installed any software he wants.

I have seen servers with the same symptoms relaying spam and the first
thing the owners knew was when their IP was blocked by SpamCop.

>(I guess it is...) There are only three open ports to my machine -- by
>means of rooter firewall but by this means only -- HTTP, VNC and ssh.

Before enabling ssh again I would do a few days research on how this
can be done safely. If you leave ssh configured on the default port 22
and allow logins to 'root' and do nothing to deny multiple password
attempts and have relatively weak passwords then it's only a matter of
time before you get hacked.

Here is a page taken from Google to get you started:

http://hostingfu.com/article/ssh-dic...-with-iptables

Good luck!
Chris R.

yosato_uk writes:

>Hi all,

>My Ubuntu machine, hooked up with the Internet with sshd etc running
>for access from outside, has suddenly stopped functioning -- doesn't
>get booted up with the normal runlevel, complaining several library
>files are missing for basic daemons. I checked the log and found out
>that, just around the time of breakdown, hundreds of attempts were
>made through ssh to access the machine from a couple of IP addresses.
>Before embarking on the repair, therefore, I'd like to find out if the
>machine was in fact broken in somehow. I wonder if anybody could
>advise as to how I could investigate.

Your passwords have to be pretty weak for them to break in this way, since
they try many many user names and only a few tries for each.
f they do break in as root they can always erase the log files, so gaps in
the logs is suspicious. Does apt have the "verify" option that rpm has ( ie
check the installed files against the md5 sums in the database of installed
programs?)

>I cannot see any 'successful' login attempt on the authentication log.
>Is it still possible for an outsider to manipulate the system somehow?
>(I guess it is...) There are only three open ports to my machine -- by
>means of rooter firewall but by this means only -- HTTP, VNC and ssh.

http is probably the most vulnerable.

>This may be a very basic and vague question, but any help would be
>appreciated. I'd be happy to provide further info if needed.

>Regards,
>Yo

On Sun, 29 Apr 2007, in the Usenet newsgroup comp.os.linux.security, in article
, Cheb wrote:

>On 29 Apr 2007 02:57:02 -0700, yosato_uk wrote:

>>My Ubuntu machine, hooked up with the Internet with sshd etc running
>>for access from outside, has suddenly stopped functioning -- doesn't
>>get booted up with the normal runlevel, complaining several library
>>files are missing for basic daemons.

An ordinary user can't screw this up very easily - it takes 'root'
permissions, and Ubuntu places some limits on root access. Was the
system being kept up to date?

>>I checked the log and found out that, just around the time of breakdown,
>>hundreds of attempts were made through ssh to access the machine from a
>>couple of IP addresses.

Welcome to the world of 'dictionary attacks' on SSH daemons - VERY common
and a reason to place restrictions on access to the SSH ports to those
address ranges that you believe will have a legitimate reason to connect
to your SSH server.

>>Before embarking on the repair, therefore, I'd like to find out if the
>>machine was in fact broken in somehow. I wonder if anybody could
>>advise as to how I could investigate.

Depends on your skill levels - if you are comfortable at the command
line, there are many things you can look at. The GUI tools are more
easily compromised to prevent showing the bad news.

>If your server has been compromised, and it sounds like it might, then
>don't trust *anything*, least of all what the OS is telling you.

Absolutely agreed

>Spend your time backing up your valuable data first

Yes

>then if you want to, try downloading a rootkit scanner and see what it
>detects.

Here, I disagree. There are two commonly used rootkit scanners available
(and possibly included in your distribution), which are 'chkrootkit'
(http://www.chkrootkit.org/) and 'rkhunter' (http://www.rootkit.nl/).
Neither are worth the CPU cycles it takes to download them, never mind
the time/effort to run them. They are far more likely to give false alarms
than find an actual rook kit. While 'rkhunter' attempts to go further,
both "tools" are looking for symptoms that have been seen in the past as
indications of an intrusion. However, both do this in an extremely
simplistic manner, EASILY CIRCUMVENTED by the most inexperienced skript
kiddy. For example, both look for the presence of a file named
"/tmp/.../a" or "/tmp/.../r" and on finding either declare that the
box has a 2003 worm called "55308". Should the skript kiddiez have renamed
the file to "/tmp/.../A" (or indeed _anything_ else), the worm won't be
detected. Another example, the tools look for certain strings in certain
binaries - using the commands "strings $TARGETFILE | grep $BADSTRING",
and often cause false alarms by finding $BADSTRING as part of a longer
string (finding "found" in "Newfoundland"). Again, and rootkit author
can circumvent this by changing the string ("Found" is not the same as
"found" or "FOUND").

Another recent poster to this group on the 12th or 13th of this month
(look back about 50 articles from "now" for a thread "How does one test
chkrootkit?") discovered that he could _replace_ a binary (/bin/date ?)
with something else, and chkrootkit didn't notice the substitution. The
'rkhunter' tool _might_ have detected this, as it does checksum/hashcheck
some binaries, but again this is subject to LOTS of false alarms. See
responses in that thread for other solutions.

>>I cannot see any 'successful' login attempt on the authentication log.
>
>If a hacker did log in successfully you might assume he has covered
>his tracks and removed any trace of that.

which is why the rootkit detectors are also likely to fail to find a
break in. The tools are available with source (and are mainly shell
scripts anyway), and (as I've been stressing) are EASY to circumvent.

>Highly likely. I am not an expert on Ubuntu and I know it does quite a
>lot to restrict unnecesary root access but if the hacker cracked one
>of your logins and managed to gain root-level access then he could
>have rootkitted you and installed any software he wants.

Yes - in years past, one need only add a line to /etc/passwd for a
root-equivalent account, and the box is 0wn3d!!!

>Before enabling ssh again I would do a few days research on how this
>can be done safely.

Definitely agreed again. Both of you are posting from UK providers,
suggesting you are in Britain. Is there any reason you can think of
that you will need to log in to your system from South America, Asia,
or even the continent? Use a firewall rule to block access to the
ssh port and only allow access from IP ranges you may expect you will
actually be using. (Note: IP address ranges are not arranged in nice
convenient order - the best you can do is to substantially reduce the
number of chances. See http://www.iana.org/assignments/ipv4-address-space
and you own logs for clues of what you can block.) Likewise, are you
allowing ssh access to people who are to unskilled to know how to tell
their clients to use a different port number than the default '22'? If
they have the skills, DON'T USE THE DEFAULT PORT 22. "Well known ports"
exist so that people who don't know better can find a service. If you
don't want people to automatically find your server, move it elsewhere
(preferably above port 1024, and not a s3kr3t number only a 3L33t h4x0r
kiddie can remember, like 12345). What's a good port number to use?

[compton ~]$ head -2 /dev/random | mimencode | tr -d 'A-z/+' | column
93285798 057134614 41126395029913 67274730
03984161308677 379491685 36299082976038 146914677
68815375230003 4144452 90087622526 34724849==
[compton ~]$

pick a port number between 1025 and 65500 from those digits. Some may
think of this as "security by obscurity", and in a way it is. But because
you still require the correct username and authentication token(s) on
the new port number, the only thing moving the port is going to do is to
avoid nearly all of the automatic/scripted attacks that target the default
port number.

Old guy

>My Ubuntu machine, hooked up with the Internet with sshd etc running
>for access from outside, has suddenly stopped functioning -- doesn't
>get booted up with the normal runlevel, complaining several library
>files are missing for basic daemons. I checked the log and found out
>that, just around the time of breakdown, hundreds of attempts were
>made through ssh to access the machine from a couple of IP addresses.
....
>I cannot see any 'successful' login attempt on the authentication log.
>Is it still possible for an outsider to manipulate the system somehow?
>(I guess it is...) There are only three open ports to my machine -- by
>means of rooter firewall but by this means only -- HTTP, VNC and ssh.

Were any of your passwords guessable? If not, ssh probably isn't
the problem.

--
These are my opinions, not necessarily my employer's. I hate spam.

On Sun, 29 Apr 2007 15:58:44 -0500
ibuprofin@painkiller.example.tld (Moe Trin) wrote:

> >On 29 Apr 2007 02:57:02 -0700, yosato_uk wrote:
>
> >>My Ubuntu machine, hooked up with the Internet with sshd etc running
> >>for access from outside, has suddenly stopped functioning -- doesn't
> >>get booted up with the normal runlevel, complaining several library
> >>files are missing for basic daemons.
....
> Definitely agreed again. Both of you are posting from UK providers,
> suggesting you are in Britain. Is there any reason you can think of
> that you will need to log in to your system from South America, Asia,
> or even the continent? Use a firewall rule to block access to the
> ssh port and only allow access from IP ranges you may expect you will
> actually be using. (Note: IP address ranges are not arranged in nice
> convenient order - the best you can do is to substantially reduce the
> number of chances. See http://www.iana.org/assignments/ipv4-address-space
> and you own logs for clues of what you can block.)

This site can be helpful if you need to find IP addresses used in
a particular country:

http://software77.net/cgi-bin/ip-country/geo-ip.pl

Be aware though that information isn't always "exact" because
1) there are domains like "EU", 2) a server registered in, say, UK,
can be physically located in another country, etc.

--
M.

Thanks for the info from quite a few people in a short space of time
-- all of which is very useful.

The inevitable conclusion seems to be it is not so straightforward
even just to ascertain a break-in, if the attacker's as clever as is
expected, though there is strong circumstantial evidence... And I've
got the feeling that ssh is *not* the route through which the attacker
gained access, if they indeed did --- my passwds are much stronger
than the names tried in the failed attempts, and I'm sure that they
would have remove *all* the log files. So... exluding pure
coincidence, my guess is the same attacker that tried the ssh route
found some other route.

Now I'm not so much interested in hunting for the culprit as
preventing a further problem. I will try and enhance the security the
various ways suggested, but the remaining worry is that there might be
some malware or worm that is left on the system. If the usual tools
may fail to detect them, is there any better way? If there's no way to
be absolutely sure, I'd in fact clean-reinstall the system altogether
and recover the backed up data. But I presume there still would remain
some risk, if some malware's mixed into the data directory... so
probably a rather naive question again: any (fast enough) way to
transfer data during which you can verify safety?

Regards,
Yo

On 30 Apr 2007 03:58:38 -0700, yosato_uk wrote:
>my guess is the same attacker that tried the ssh route
>found some other route.

Actually, working from log evidence is very hard. SSH is attacked
routinely my botnets and script-kiddies all around the world, so if
you have ssh on port 22 you will always have a full log file of silly
attempts. However, if someone gets in they might be clever enough to
cover all of their tracks and leave the 'normal' ssh probes in the log
files just to make it look like any other day.

>may fail to detect them, is there any better way? If there's no way to
>be absolutely sure, I'd in fact clean-reinstall the system altogether
>and recover the backed up data.

This is the only safe way to do it.

If you are worried about the folders of data you could: back it all
up; reinstall the OS; donload a free Linux antivirus prog (eg.
AntiVir) and then virus-scan the data as it comes down. However,
copying down some data off a DVD won't activate any malware - you'd
have to execute something there, like a Trojan Horse. So, I'd just
download it to a temporary folder - scan it thoroughly and then move
it into the 'live' folder structure when you are happy it is clean.

Chris R.

yosato_uk writes:

>Thanks for the info from quite a few people in a short space of time
>-- all of which is very useful.

>The inevitable conclusion seems to be it is not so straightforward
>even just to ascertain a break-in, if the attacker's as clever as is
>expected, though there is strong circumstantial evidence... And I've
>got the feeling that ssh is *not* the route through which the attacker
>gained access, if they indeed did --- my passwds are much stronger
>than the names tried in the failed attempts, and I'm sure that they
>would have remove *all* the log files. So... exluding pure
>coincidence, my guess is the same attacker that tried the ssh route
>found some other route.

>Now I'm not so much interested in hunting for the culprit as
>preventing a further problem. I will try and enhance the security the
>various ways suggested, but the remaining worry is that there might be
>some malware or worm that is left on the system. If the usual tools
>may fail to detect them, is there any better way? If there's no way to
>be absolutely sure, I'd in fact clean-reinstall the system altogether
>and recover the backed up data. But I presume there still would remain
>some risk, if some malware's mixed into the data directory... so
>probably a rather naive question again: any (fast enough) way to
>transfer data during which you can verify safety?


As a minimum scan the backup for suid programs. There almost certainly
should be none. At least examine any very carefully.
find /backup -perm /6000 -ls

On Mon, 30 Apr 2007, in the Usenet newsgroup comp.os.linux.security, in article
<20070430114646.31fab8f3.invalid_muxaul@lenta.ru>, Mikhail Zotov wrote:

>ibuprofin@painkiller.example.tld (Moe Trin) wrote:

>> (Note: IP address ranges are not arranged in nice convenient order
>> - the best you can do is to substantially reduce the number of
>> chances. See http://www.iana.org/assignments/ipv4-address-space and
>> your own logs for clues of what you can block.)
>
>This site can be helpful if you need to find IP addresses used in
>a particular country:
>
>http://software77.net/cgi-bin/ip-country/geo-ip.pl

Looks like it's based on RIR data, which can be tricky. I normally
use a whois query to the "appropriate" whois server (see that IANA
web page).

Everyone picks on China or Korea as a source of bad packets, and you
are frequently recommended to block those addresses. There are problems.
1. The IP addresses assigned/allocated to country X are not contiguous,
but are scattered.

[compton ~]$ zgrep CN IP.ADDR/stats/[ALR]* | cut -d' ' -f2 | cut -d'.'
-f1 | sort -n | uniq -c | column
43 58 23 122 1 161 1 198 41 219
32 59 44 123 1 162 320 202 16 220
38 60 63 124 1 166 92 203 63 221
84 61 38 125 1 167 72 210 64 222
24 116 1 134 1 168 41 211
34 121 1 159 4 192 64 218
[compton ~]$

First octet of the addresses for China. Korea is even worse:

[compton ~]$ zgrep KR IP.ADDR/stats/[ALR]* | cut -d' ' -f2 | cut -d'.'
-f1 | sort -n | uniq -c | column
15 58 15 125 2 152 10 165 85 211
6 59 1 128 1 154 4 166 10 218
1 60 1 129 1 155 8 168 2 219
17 61 1 134 1 156 1 169 10 220
2 116 1 137 1 157 24 192 8 221
23 121 1 141 1 158 32 202 7 222
23 122 1 143 1 161 56 203
18 123 4 147 3 163 1 206
22 124 3 150 2 164 81 210
[compton ~]$

But when you look at those /8s, you find something like::

[compton ~]$ zgrep -h ' 58\.' IP.ADDR/stats/[ALR]* | cut -d' ' -f1 |
sort | uniq -c | column
1 AF 4 HK 15 KR 5 PK 1 VN
24 AU 3 ID 4 MY 6 SG
5 BD 3 IN 2 NZ 8 TH
43 CN 29 JP 2 PH 4 TW
[compton ~]$

So it's not simple. (The files I'm grepping here are locally massaged
versions of the five RIR zone files.) Part of this is due to the
different dates that the RIRs were formed - ARIN came first, followed by
RIPE and APNIC. LACNIC was split out of ARIN, and AFRINIC out of RIPE.
Part is due to the old Classful allocations (there is no Class A, B, or
C any more - hasn't been since Classless Inter-Domain Routing was
implemented in 1993). Part is because the Internet is to _facilitate_
connections, not prevent them. Thus. again looking at the first octet of
the IP address, we find

24 ARIN LACNIC RIPE 62 AFRINIC RIPE
64 AFRINIC ARIN LACNIC 66 AFRINIC ARIN LACNIC
69 AFRINIC ARIN 80 AFRINIC RIPE
81 AFRINIC RIPE 82 AFRINIC RIPE
83 AFRINIC RIPE 84 AFRINIC RIPE
128 APNIC ARIN RIPE 129 APNIC ARIN LACNIC RIPE
130 APNIC ARIN RIPE 131 APNIC ARIN LACNIC RIPE
132 APNIC ARIN LACNIC RIPE 134 APNIC ARIN RIPE
135 ARIN RIPE 136 APNIC ARIN RIPE
137 AFRINIC APNIC ARIN RIPE 138 APNIC ARIN RIPE
139 AFRINIC APNIC ARIN LACNIC RIPE 140 APNIC ARIN LACNIC RIPE
141 APNIC ARIN RIPE 143 AFRINIC APNIC ARIN LACNIC RIPE
144 APNIC ARIN LACNIC RIPE 146 AFRINIC APNIC ARIN LACNIC RIPE
147 AFRINIC APNIC ARIN LACNIC RIPE 148 APNIC ARIN LACNIC RIPE
149 APNIC ARIN RIPE 150 APNIC ARIN LACNIC RIPE
151 APNIC ARIN RIPE 152 AFRINIC APNIC ARIN LACNIC RIPE
153 APNIC ARIN RIPE 154 APNIC ARIN RIPE
155 AFRINIC APNIC ARIN LACNIC RIPE 156 AFRINIC APNIC ARIN LACNIC RIPE
157 APNIC ARIN LACNIC RIPE 158 APNIC ARIN LACNIC RIPE
159 APNIC ARIN LACNIC RIPE 160 AFRINIC APNIC ARIN RIPE
161 APNIC ARIN LACNIC RIPE 162 APNIC ARIN LACNIC RIPE
163 AFRINIC APNIC ARIN LACNIC RIPE 164 AFRINIC APNIC ARIN LACNIC RIPE
165 AFRINIC APNIC ARIN LACNIC RIPE 166 AFRINIC APNIC ARIN LACNIC RIPE
167 APNIC ARIN LACNIC RIPE 168 AFRINIC APNIC ARIN LACNIC RIPE
169 AFRINIC APNIC ARIN LACNIC 170 APNIC ARIN LACNIC RIPE
171 ARIN RIPE 192 AFRINIC APNIC ARIN LACNIC RIPE
193 AFRINIC RIPE 194 AFRINIC RIPE
195 AFRINIC RIPE 196 AFRINIC APNIC ARIN LACNIC RIPE
198 AFRINIC APNIC ARIN LACNIC RIPE 199 ARIN LACNIC
200 AFRINIC ARIN LACNIC 202 AFRINIC APNIC
204 AFRINIC ARIN LACNIC 205 AFRINIC ARIN LACNIC
206 AFRINIC ARIN LACNIC 207 ARIN LACNIC
209 AFRINIC ARIN LACNIC 212 AFRINIC RIPE
213 AFRINIC RIPE 216 AFRINIC ARIN LACNIC
217 AFRINIC RIPE

2. "You can block by top level domain name, right?" No, for two reasons.
Contrary to popular belief, ".com", ".edu". ".net' and ".org" (never mind
the other domains like '.biz", ".info" and so on) are not restricted to
the USA. There are ".com" domains in virtually _every_ country. Second.
and this is particularly true in countries with "bad" reputations, a lot
of those network administrators don't think it necessary to follow the
requirements of RFC1034, 2050 and 2181, and don't bother creating PTR
records (rDNS, or "IP to hostname"). In fact, many mail servers are now
configured to reject _connections_ from hosts that don't have resolvable
IP addresses, and some people are going further to checking that the PTR
record does match the A (name to IP) record. That trick alone cut my
spam load by 70 percent as it eliminated a lot of 'zombie' mail.

>Be aware though that information isn't always "exact" because
>1) there are domains like "EU",

[compton ~]$ zgrep -hE '(AP|EU)' [ALR]* | cut -d' ' -f1 | sort | uniq -c |
column
87 AP 4697 EU
[compton ~]$

That's out of a total of about 80,000 listings.

>2) a server registered in, say, UK, can be physically located in another
>country, etc.

The domain I work at is registered in New York, but if you were to
traceroute to my system, the last host you see before the black hole
is a backbone router near San Francisco, even though I am near Phoenix
(600 KM East of Los Angeles) and we have subnets on six continents and
over forty countries. The next subnet above mine is in France. IP
address to geographical location is EXTREMELY haphazard and not defined.

So - what's the firewall solution? Do not _allow_ everything by
default. I don't know about you, but I don't offer SSH to everyone
in the world. For SSH at home, I accept connections from exactly 1545
IP addresses (a /22. two /24s, and nine specific hosts) ONLY. If I'm
going on a trip and need to allow access from other addresses, I can
do so - but for now - 1545 addresses (out of 3.71e+09 IPv4 Internet
addresses, 2.47e+09 actually assigned/allocated world wide). That
cuts down the "noise" considerably.

Old guy

On 30 Apr 2007, in the Usenet newsgroup comp.os.linux.security, in article
<1177930718.337989.300510@n76g2000hsh.googlegroups. com>, yosato_uk wrote:

>The inevitable conclusion seems to be it is not so straightforward
>even just to ascertain a break-in, if the attacker's as clever as is
>expected, though there is strong circumstantial evidence...

One technique with a reasonable chance of success is to compare the
file system with a known clean version. The original version of this
type of test is 'tripwire' currently unsupported - and the derivatives
such as 'aide'. Briefly, you run this application on a clean box to get
checksums and hashes of EVERYTHING. Thereafter, you take the file system
off line, and run the same checksums and hashes using a "trusted" copy
of an operating system, and compare these with the original data. Is
anything different? The two problems with this is that you have to
create the reference data from a known clean installation, AND keep it
up to date with any intentional changes (such as security updates),
and that you should run the tests with a known secure O/S (which
might be a floppy that also has the needed tools) rather than the
"normal" O/S (which may have been compromised).

>And I've got the feeling that ssh is *not* the route through which the
>attacker gained access, if they indeed did --- my passwds are much
>stronger than the names tried in the failed attempts,

What else is running on the system? A web server, especially one with
dynamic content/scripting is FAR more likely. Is everything up to date?

>and I'm sure that they would have remove *all* the log files.

It's terribly easy to edit logs - and a common skript-kiddy technique.

>So... exluding pure coincidence, my guess is the same attacker that
>tried the ssh route found some other route.

Or someone else did - there's more than a few skript kiddiez out there.

>Now I'm not so much interested in hunting for the culprit as
>preventing a further problem. I will try and enhance the security the
>various ways suggested,

Good

>but the remaining worry is that there might be some malware or worm
>that is left on the system. If the usual tools may fail to detect them,
>is there any better way?

Depends. My usual trick is to use the package manager to check the
system ('/bin/rpm -V' for an rpm based system, 'debsums -s' for a
Debian based system --> even though the debsums man-page depreciates
this mode). First, check the tool by substituting some other file for
a binary that the intruder would normally alter:

/bin/mv /bin/ps /bin/ps.original
/bin/mv /bin/less /bin/ps
rpm -Vf /bin/ps or debsums -s procps
/bin/mv /bin/ps.original /bin/ps

DON'T NEGLECT THAT LAST STEP!!! Did your package manager freak out
when it discovered that /bin/ps doesn't look like /bin/ps _ought_ to
look like? If no, you're toast. If yes, then move on to the next step
in the Jedi Mind Games.

Now, if you _had_ followed the last paragraph in the CAVEATS section of
the 'debsums' man page

If you are looking for an integrity checker that can run from safe
media, do integrity checks on checksum databases and can be easily
configured to run periodically to warn the admin of changes see other
tools such as: aide, integrit, samhain, or tripwire.

you might have other options. Isn't hindsight wonderful? ;-)

>If there's no way to be absolutely sure, I'd in fact clean-reinstall
>the system altogether and recover the backed up data.

That is the most reliable solution. Ideally, the backed up data was
also clean. I make backups when _I_ change things on the server, not
as an automatic function. Thus, my backups should reflect that last
known "good" situation (I hope). ;-)

>But I presume there still would remain some risk, if some malware's
>mixed into the data directory...

Your "clean install" should reduce a lot of this risk - unless this is
some scripting/database/what-ever that might be executable as part of
(for example) your web page. You also have the issue that you haven't
corrected what-ever hole was exploited.

>so probably a rather naive question again: any (fast enough) way to
>transfer data during which you can verify safety?

The integrity checkers like aide and so-on can do this, BUT they need
a "clean" starting point. By the way, now you may know why our publicly
visible systems all run from "read-only" media, and have only the
minimum "stuff" installed to do their job. No bells, no whistles, no
frilly knickers.

Aren't "learning experiences" wonderful?

Old guy

Thank you for the very interesting post!

On Mon, 30 Apr 2007 19:51:27 -0500
ibuprofin@painkiller.example.tld (Moe Trin) wrote:
....
> So - what's the firewall solution? Do not _allow_ everything by
> default. I don't know about you, but I don't offer SSH to everyone
> in the world. For SSH at home, I accept connections from exactly 1545
> IP addresses (a /22. two /24s, and nine specific hosts) ONLY.

What is your opinion about port knocking and/or single packet authorization?
I have checked links available at portknocking.org and it seems the
majority of projects are in a stale condition (except fwknop and a
couple of projects at alpha stages).

Regards,
Mikhail

On 30 Apr, 16:47, Unruh wrote:
> yosato_uk writes:
> >Now I'm not so much interested in hunting for the culprit as
> >preventing a further problem. I will try and enhance the security the
> >various ways suggested, but the remaining worry is that there might be
> >some malware or worm that is left on the system. If the usual tools
> >may fail to detect them, is there any better way? If there's no way to
> >be absolutely sure, I'd in fact clean-reinstall the system altogether
> >and recover the backed up data. But I presume there still would remain
> >some risk, if some malware's mixed into the data directory... so
> >probably a rather naive question again: any (fast enough) way to
> >transfer data during which you can verify safety?
>
> As a minimum scan the backup for suid programs. There almost certainly
> should be none. At least examine any very carefully.
> find /backup -perm /6000 -ls

erk, IMHO not up to your usual quality of response, U

If one strongly suspects their machine has been compromised and / or
its got trashed, a format / re-install is called for. This is where
you start wishing you'd been running an IDS like tripwire or L5. By
all means mount the drive read-only on an already running, clean
system and have a poke around for setuid files / rootkit detectors,
but the quickest route to getting your system back to a trusted state
is probably going to be by vetting stuff you carry forward to the new
installation, not trying to find what's been tampered with in the old
one - there's just too much to check and too much risk of missing
something.

OP: you can see this flurry of log entries - but it does not
necessarilty follow that this corresponds with when your machine was
compromised - you should do a clean install and only restore your own
files from backup (not ones which were from the previous install) and
you should check them for interference using a rootkit detector.

C.

On 2007-04-29, yosato_uk wrote:

> This may be a very basic and vague question, but any help would be
> appreciated. I'd be happy to provide further info if needed.

There may be some left on the newsstand, if you hurry. But if not,
you might consider ordering this back issue of Linux-Pro magazine.

http://www.linux-magazine.com/issue/77

I don't often buy periodicals anymore, especially pricey one's like
L-P, but this issue had some very good info on back doors and
real-time IDS techniques (lsof). Maybe a friend has an issue.

nb

"C." writes:

>On 30 Apr, 16:47, Unruh wrote:
>> yosato_uk writes:
>> >Now I'm not so much interested in hunting for the culprit as
>> >preventing a further problem. I will try and enhance the security the
>> >various ways suggested, but the remaining worry is that there might be
>> >some malware or worm that is left on the system. If the usual tools
>> >may fail to detect them, is there any better way? If there's no way to
>> >be absolutely sure, I'd in fact clean-reinstall the system altogether
>> >and recover the backed up data. But I presume there still would remain
>> >some risk, if some malware's mixed into the data directory... so
>> >probably a rather naive question again: any (fast enough) way to
>> >transfer data during which you can verify safety?
>>
>> As a minimum scan the backup for suid programs. There almost certainly
>> should be none. At least examine any very carefully.
>> find /backup -perm /6000 -ls

>erk, IMHO not up to your usual quality of response, U

>If one strongly suspects their machine has been compromised and / or
>its got trashed, a format / re-install is called for. This is where
>you start wishing you'd been running an IDS like tripwire or L5. By

It would really really help if you read both the post and the response.
This comment was in response to the suggestion to reinstall and then put in
place the backup of /home. That backup itself should be scanned.

>all means mount the drive read-only on an already running, clean
>system and have a poke around for setuid files / rootkit detectors,

Sorry, you are of the impression that files can run themselves? Why do you
want to mount the drive on clean system ro?

My adivce would be.
a) reinstall the operating system.
b) scan all of the stuff that you did NOT reinstall for suid/sgid files.
All. /tmp, /var/* /home, /usr/local.
I once found a file /tmp/banana which was a root suid shell.


>but the quickest route to getting your system back to a trusted state
>is probably going to be by vetting stuff you carry forward to the new
>installation, not trying to find what's been tampered with in the old
>one - there's just too much to check and too much risk of missing
>something.

Precisely what I was saying-- vet the stuff you carry forward.


>OP: you can see this flurry of log entries - but it does not
>necessarilty follow that this corresponds with when your machine was
>compromised - you should do a clean install and only restore your own
>files from backup (not ones which were from the previous install) and
>you should check them for interference using a rootkit detector.

Well, first you should try to make sure that your machine actually was
complrimised. If you reinstall every time something suspicious appears in a
log file, you will never use your computer for anything worthwhile. It will
be a daily excercise.

If you have a rpm based system, first do
rpm -Va>/tmp/verify
and then look through those files for chnged files. If you find one that
you cannot explain, you are probably comprimised.

Then look for suid file. Look everywhere. Make sure that those file should
be suid files. If you find an suid root file in /dev, /tmp, etc, then you
have been comprimised.

Look at the log files for suspicious blanks.

>C.

On Tue, 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
<20070501074148.15276527.invalid_muxaul@lenta.ru>, Mikhail Zotov wrote:

>Thank you for the very interesting post!

You are welcome!

>ibuprofin@painkiller.example.tld (Moe Trin) wrote:
>...
>> So - what's the firewall solution? Do not _allow_ everything by
>> default. I don't know about you, but I don't offer SSH to everyone
>> in the world. For SSH at home, I accept connections from exactly 1545
>> IP addresses (a /22. two /24s, and nine specific hosts) ONLY.

>What is your opinion about port knocking and/or single packet authorization?

]] If I'm going on a trip and need to allow access from other addresses,
]] I can do so

Port knocking is another layer - and relatively simple to add to the
non-standard server port number. From my Sunday post to get some numbers
to use here:

[compton ~]$ head -2 /dev/random | mimencode | tr -d 'A-z/+' | column
93285798 057134614 41126395029913 67274730
03984161308677 379491685 36299082976038 146914677
68815375230003 4144452 90087622526 34724849==
[compton ~]$

The knocking scheme I've used is crude - listen for packets on port
39841 (which is closed), and on the firewall seeing such packet, open
port 13086 (where the SSH server is hiding) for the IP address that
had hit the first port for one minute. Also set two "trip" ports just
above and just below the SSH port - if the knocking address hits those
ports, close 13086 under the premise that a port scanner hit the knock
port, then continued to scan. The trip ports will ensure that the port
scanner closes the SSH port before he gets to it unless the scanner is
hitting random numbers, and the Gods of Chance are frowning at me.

PORT KNOCKING IS NOT A SUBSTITUTE FOR PROPER AUTHENTICATION. For those
with a knee-jerk reaction to "hiding things" being security through
obscurity, shove it! Port knocking, like moving the server to an obscure
port number, or using firewall rules to harshly restrict access to a
minimum number of addresses, is an _addition_ to what ever security
functions (proper authentication, using 'non-dictionary/non-phone-book'
usernames, strong authentication keys, etc.) that you _should_ have
had in place before reading this. All you are trying to do is to make
it more difficult. Grabbing numbers out of the sky - assume there is a
single chance in a thousand of the bad guy guessing the right username
(you DO NOT use 'root' or 'admin' or your 'public_username'), Assume
one chance in a thousand or the back guy independently guessing your
non-dictionary password. Thus, a robot has one chance in a million
of guessing both. But this will do the robot little good if it must
also guess which of the sixty-thousand ports you've hidden the server
at - and port knocking merely makes it harder for the bad guy to stumble
across your hidden server. Yes, I am aware of packet sniffing, and
there are additional steps that can be taken to reduce that risk.

The one problem you want to avoid is making what ever scheme you use to
complicated. You _will_ screw up, and lock yourself out - and you won't
be the first person to do so. I encountered a situation several years
ago where the simple port-knock wouldn't work. I gave up after the third
try, but later investigation of the logs showed the knock was being
intercepted by a proxy server, while my SSH attempt was going straight
through - the proxy server had (naturally) a different IP address.

>I have checked links available at portknocking.org and it seems the
>majority of projects are in a stale condition (except fwknop and a
>couple of projects at alpha stages).

I've looked at a number of the projects (though not recently) and tend
to agree. The scheme used doesn't have to be complicated - my original
knocking "daemon" was a firewall rule to log to a special file any
connection attempts to port $FOO, and a cron script that looked for
that file - and on finding it, parsed the IP, set a firewall rule for
port $BAR from that IP, and set a timer to clear that rule a minute
later. The firewall 'ESTABLISHED" rule allowed the "conversation to
continue. Security does not have to be complicated, or cute. Simple
steps _can_ do a lot for you as long as you use some common sense. Oh,
and you _do_ remember to change that password (and username if you are
ultra-paranoid) on a regular basis, right? Just don't over do it.

Old guy

On 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
<1178020943.501703.59150@l77g2000hsb.googlegroups.c om>, C. wrote:

>Unruh wrote:

>> As a minimum scan the backup for suid programs. There almost
>> certainly should be none. At least examine any very carefully.
>> find /backup -perm /6000 -ls

>erk, IMHO not up to your usual quality of response, U

Well, he did say it was the very minimum - but yes it's preferable not
to have to recover from the contaminated backups. Doing automated
nightly or weekly backups may not be the best scheme. Backing up
immediately after _you_ make some change is safer, and safer still
is having those originals created on an isolated box, with the
server running from 'read-only' copies of that source.

>If one strongly suspects their machine has been compromised and / or
>its got trashed, a format / re-install is called for. This is where
>you start wishing you'd been running an IDS like tripwire or L5.

Agreed

>By all means mount the drive read-only on an already running, clean
>system and have a poke around for setuid files / rootkit detectors,

though this is for educational purposes (how did they do it) only.

>but the quickest route to getting your system back to a trusted state
>is probably going to be by vetting stuff you carry forward to the new
>installation, not trying to find what's been tampered with in the old
>one - there's just too much to check and too much risk of missing
>something.

Talking about a public server here - a work station should not be
running public severs, and a server should not have users allowed
to log in - the data for the server should be developed/tested on
a separate box, whether it be web pages, files for a file server, or
what-ever. Getting the system back to the trusted state then only
involves a clean install of the operating system and applications,
and then copying the data from the development system. Running the
server application such that the data is owned by a specific
non-privileged user AND GROUP allows additional "does this smell right"
checks to be run.

Old guy

ibuprofin@painkiller.example.tld (Moe Trin) writes:

>On 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
><1178020943.501703.59150@l77g2000hsb.googlegroups.c om>, C. wrote:

>>Unruh wrote:

>>> As a minimum scan the backup for suid programs. There almost
>>> certainly should be none. At least examine any very carefully.
>>> find /backup -perm /6000 -ls

>>erk, IMHO not up to your usual quality of response, U

>Well, he did say it was the very minimum - but yes it's preferable not
>to have to recover from the contaminated backups. Doing automated
>nightly or weekly backups may not be the best scheme. Backing up
>immediately after _you_ make some change is safer, and safer still
>is having those originals created on an isolated box, with the
>server running from 'read-only' copies of that source.

The breakin could have happened a month ago and was just utilized/screwed
up last night. If you are broken into, do not trust anything, including
your backups. That is why scanning your backups, no matter when made, is a
minimum.
Note that I am NOT advocating just doing this.

yosato_uk (07-04-30 03:58:38):

> The inevitable conclusion seems to be it is not so straightforward
> even just to ascertain a break-in, if the attacker's as clever as is
> expected, though there is strong circumstantial evidence...

As abstract as I like to be, the problem is that for a computer, there
is no distinction of `compromised' vs. `not compormised'. Everybody has
a different idea of where this distinction occurs. This is why there
can't be a perfect rootkit scanner or break-in detector.

In other words: To check for a break-in, you need to minimize the scope
of the effect of a possible break-in (like changed files, BIOS, system
time, etc.) to a certain area of your computer (like all executable
files on your hard-disk), learn how this area works, and check it
yourself. Because this is impractical and error-prone, you'll do better
just reinstalling your operating system on all possibly affected hosts,
backing up all important data.


> And I've got the feeling that ssh is *not* the route through which the
> attacker gained access, if they indeed did --- my passwds are much
> stronger than the names tried in the failed attempts, and I'm sure
> that they would have remove *all* the log files. So... exluding pure
> coincidence, my guess is the same attacker that tried the ssh route
> found some other route.

Perhaps your system was compromised long before that word-list attack
happened. Perhaps it has been compromised later. Probably it hasn't
been compormised at all. Sometimes system updates, which include OS
components, break things.


> Now I'm not so much interested in hunting for the culprit as
> preventing a further problem. I will try and enhance the security the
> various ways suggested, [...]

You will want to use proper authentication methods, and restrict,
restrict, restrict everything you can for people not supposed to access
your hosts. In the best case, a random attacker (i.e. script kiddie)
doesn't even know your network exists.


> [...] but the remaining worry is that there might be some malware or
> worm that is left on the system. If the usual tools may fail to detect
> them, is there any better way? If there's no way to be absolutely
> sure, I'd in fact clean-reinstall the system altogether and recover
> the backed up data.

Exactly. There is no way you can know, besides checking manually, which
is impractical. So it's best to do drop your current installation
completely.


> But I presume there still would remain some risk, if some malware's
> mixed into the data directory... so probably a rather naive question
> again: any (fast enough) way to transfer data during which you can
> verify safety?

No automated way. Firstly you need to separate functional data from
non-functional data. Shell scripts and often configuration files are
functional. A JPEG image generally isn't (although with a suitable
viewer and EXIF, it might).

Whether data is functional or not is not always as obvious. Take care,
and if uncertain, always assume functional data. For example, you
wouldn't consider your Quake save-games to be functional, but in fact
they are! Functional data is not necessarily executable.

For all functional files, you need to check manually whether they have
been manipulated. Depending on the number of functional files you have,
this is going to take a long time. After you've finished checking, make
sure you use secure versions of viewer programs.

Of course this is überparanoid and certainly extremely expensive and
time consuming, but it's the only way to make sure. Always assume that
the attacker has had compromised your system long before you noticed
there's something wrong.


Regards,
Ertugrul Söylemez.


--
From the fact that this CGI program has been written in Haskell, it
follows naturally that this CGI program is perfectly secure.

On Tue, 01 May 2007 14:58:31 -0500
ibuprofin@painkiller.example.tld (Moe Trin) wrote:

> On Tue, 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
> <20070501074148.15276527.invalid_muxaul@lenta.ru>, Mikhail Zotov wrote:
....
> >What is your opinion about port knocking and/or single packet authorization?
>
....
> Port knocking is another layer - and relatively simple to add to the
> non-standard server port number. From my Sunday post to get some numbers
> to use here:
>
> [compton ~]$ head -2 /dev/random | mimencode | tr -d 'A-z/+' | column
....

Cute :-)

....
> The one problem you want to avoid is making what ever scheme you use to
> complicated. You _will_ screw up, and lock yourself out - and you won't
> be the first person to do so.

LOL :-) This happened one, this happened twice... How more times will
this happen?

> ... Security does not have to be complicated, or cute. Simple
> steps _can_ do a lot for you as long as you use some common sense. Oh,
> and you _do_ remember to change that password (and username if you are
> ultra-paranoid) on a regular basis, right? Just don't over do it.

Thank you! It's a big pleasure to study your posts!

Regards,
Mikhail