[nzlug] Urgent: Load balancing / network monitoring / security

[Steve Holdoway]

n Sun, 16 Sep 2007 12:43:04 +1200
"Hosting Direct Ltd" <ben at hostingdirect.co.nz> wrote:

> Hi everyone.
> I run Hosting Direct Ltd, some may know that I run the FreeB web hosting
> plan, I also run a web site called Foxy Hub, this is a auction web site.
> I am having some real issues at the moment with the way I did my setup and
> managing my 45RU of servers, I did the network setup with some skilled Linux
> friends:
> For security we use dual firewalls at network entry with IP Tables, and SNMP
> monitoring back to a server, have MRTG bandwidth monitoring setup also etc.
> We use Nagios2 to keep an eye on all the servers in the racks.
> We use ultra monkey to do the HA load balancing for foxy hub.
> BUT GUESS WHAT!? My servers still got hacked, 20% of my network is still
> down, my site foxy hub site still wont run due to a DOS and to many MYSQL
> connection.
> Can you believe that!? Anyway it's a Sunday and I need to make a big move
> this week to really secure my network services and future proof them, My
> FreeB hosting servers have also hacked to death, the servers are dead or
> dieing and I can't jump stat it's heart the usual way!! Lol! :( (mad panic!)
> 
> Anyway, down to the reason of this mail, I am looking for some advise from
> someone who knows what to do in these situations, I am happy to pay for your
> time for consulting...
> 
> At the moment I am looking into exinda.com appliances to take over nagios2's
> job and policy control, snmp and network security, is this the right move?
> Go for the corporate solutions EEK!?
> 
> Any advise, suggestions, or phone calls really appreciated.
> 
> Thanks in advance.
> ________________________________
> 
> Kind Regards,
> Ben Simpson
> Managing Director
> Phone: 09 834 2560
> 
> 
> 
> 
> 
> _______________________________________________
> NZLUG mailing list NZLUG at linux.net.nz
> http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

1. disconnect your servers from the net. If you're remote, then leave ssh / vpn open.
2. Identify the problem(s)
3. Remove it.
4. Bring back up.

It's not rocket science, and don't believe anyone who says it is. Just DON'T PANIC! Use all the information from mtrg, snmp, etc... what's it telling you?

If you're being dosed, then blacklist the relevant IP address(es) / get your isp to do it further upstream.
If you're being ddosed then all you can really do is to weather the storm, or buy *MUCH* more bandwidth.
If they've got in through ftp ( crap passwords/sniffing ), then disable the relevant accounts, and remove any software they installed. 
If they're using sql injection, then fix the code / update 3rd party packages.
and so on...

If your firewall software is out of date, then update it as well.

If you want me to take a look, then email me...


Steve