[nzlug] Urgent: Load balancing / network monitoring / security

[Nick 'Zaf' Clifford]

Hosting Direct Ltd wrote:
> BUT GUESS WHAT!? My servers still got hacked, 20% of my network is still
> down, my site foxy hub site still wont run due to a DOS and to many MYSQL
> connection.
>
>   
Just want to reiterate what others have said:

First, you must consider your system totally and 100% compromised.
Rootkits can hide very very effectively. In fact, the harder it is to
figure out how they got in, the more likely it is that the system is
rootkitted. There are rootkit "detectors", but they only pick up the
script kiddy rootkits. Most competent Linux engineers could build their
own rootkit that won't be picked up by a "detector".

My advise is to look at your most recent backup, and seperate the data
from the programs. With websites and php scripts, this can be very hard,
but its very important. Rebuild *ALL* of your systems from trusted media
(eg original OS CDs), then restore only the data. When it comes to data
that is also a executed (eg word documents, php scripts, custom bash
scripts, etc), you have a tough choice to make. You have to decide
whether its worth the effort to go through each and every script, and
manually look for further exploits installed by the hacker. (Unless of
course you can guarentee the exact date of intrusion, and you have
backups from before then).

It helps to imagine that you yourself are the hacker, and you have
infinite knowledge about the system, and "super" skills in crafting
programs. How would you not only compromise the current system, but
ensure that you continue to have access after the admin has "restored"
the server?

All of the above is important, but here's the real key:
Security is a process, not a product -- Bruce Schneier

You can't install a program on a server that makes it secure. Even if
you could, even if there was a magic bullet that guaranteed a server was
"unhackable"....  What about your workstation? If the hacker hacked your
workstation*, he could install a keylogger that happily logged every key
(including your root password when you login to your server) that you
typed. That would effectively defeat all of your security in one foul
swoop, million dollar security be damned.

The entire system, and all of the people who access it, have to be taken
into account as a whole.

If you don't know enough about security, then you shouldn't be doing it.
You should hire someone to, on a continuing basis, recommend and advise
on the security of the whole system. This should be more than just a
contractor who comes in, looks over your current system, and "rubber
stamps" the lot. You need to know not only of any actual holes in the
security, but of the risks of the entire system. POP3 mail available?
Then there is a risk of passwords in plain-text and man-in-the-middle
interceptions getting the password. There are thousands of risks. Even
if some of the risks are unsolvable, you need to know they exist so you
can mitigate them. If whoever you get doesn't give you a list a mile
long of security risks and problems, then they aren't worth their fee.



Nick Clifford


* Yes, I suppose you could install the magic bullet software on your
workstation too, but unfortunately magic bullets don't exist in real
life. Sorry.