[nzlug] Urgent: Load balancing / network monitoring / security

[Mark Foster]

> For security we use dual firewalls at network entry with IP Tables, and SNMP
> monitoring back to a server, have MRTG bandwidth monitoring setup also etc.
> We use Nagios2 to keep an eye on all the servers in the racks.
> We use ultra monkey to do the HA load balancing for foxy hub.
> BUT GUESS WHAT!? My servers still got hacked, 20% of my network is still
> down, my site foxy hub site still wont run due to a DOS and to many MYSQL
> connection.
> Can you believe that!? Anyway it's a Sunday and I need to make a big move
> this week to really secure my network services and future proof them, My
> FreeB hosting servers have also hacked to death, the servers are dead or
> dieing and I can't jump stat it's heart the usual way!! Lol! :( (mad panic!)
>
> Anyway, down to the reason of this mail, I am looking for some advise from
> someone who knows what to do in these situations, I am happy to pay for your
> time for consulting...

Ben,
I am not an expert by any means, but heres some advice...

Instead of saying 'Can you believe that!?' you need to understand exactly 
how you got hacked.  'hacking' is usually about someone being able to 
exploit a vulnerability in your system, whether its:

- A daemon you have running that is vulnerable,
- A user with an insecure password,
- A poor security policy that allows user-space accounts to invade areas 
they shouldnt,
- Failure to keep updated,
- Failure to be considerate of the risks of running hosting services of 
different kinds on the one system.

Thus its a whole-of-system approach required.


> At the moment I am looking into exinda.com appliances to take over nagios2's
> job and policy control, snmp and network security, is this the right move?
> Go for the corporate solutions EEK!?

There is no golden bullet.  Moving to paid-for commercial solutions may 
not prevent a repeat occurrance.  You need to identify where you've 
slipped up, and learn from it - or it'll happen again.

Speaking as someone whos had this happen more than once before himself, 
its a humbling experience... but it does teach you a few things.

Simple ones like:

- If you are going to offer php access to users, understand the limits 
within which they can operate.  If they can drop out of their own user 
directory you have a problem.

- If you are going to offer php (or any other sort of active content 
option) to a web client, what are they going to run? What applications, 
what CMS? Are they going to keep it patched? What happens if they dont?

- Have you considered virtual machines? Jails? How to segregate your areas 
of risk...

- The more daemons, the more vulnerable you are.  KISS applies.

Mark.