[nzlug] Mailing list and website(s) outage

[Nic Bellamy]

Hi all,
    firstly, if you're not familiar with my name, I'm Nic Bellamy, one 
of the pair (the other being Dylan Reeve) that started NZLUG way back 
when in the lounge of our flat.

Since then, we've been hosting various Linux-related mailing lists and 
websites, mostly using donated hosting.

For those affected by the outage, my apologies for the rather extended 
nature of it - they were out from the wee hours of Saturday morning 
until late last night.

What happened?

At about 1am on Saturday the 5th, some little <expletive> managed to 
gain unprivileged access to the account on our server used by Apache. 
This was due to the combination of a vulnerable PHP script, and an 
oversight by myself in the rush to get things going again in February 
when we had a rather catastrophic hardware failure: I'd forgotten to set 
a number of PHP configuration options to improve server security - the 
one that bit us in this case was allow_url_fopen.

While the compromise wasn't especially serious in terms of the security 
of the server itself, what the attacker then did caused us no end of 
grief. They started a packet flood. A _big_ packet flood. Over 
200Mbit/second was shooting out of our machine destined for some poor 
sods IP in the USA.

This had a serious detrimental effect on the company that was donating 
the hosting to us, and also their upstream provider. Following good 
industry practise, they of course quickly disconnected the machine.

After this, however, things got a bit painful: we had to wait for quite 
a while to gain access to the machine again, which we really needed to 
do as the most recent offsite backup I had was nearly a month old - 
forever in Internet time. Those helping us did the best they could, but 
obviously they had their own problems they had deal with at a higher 
priority than us; primarily their upstream provider being rather unhappy 
with them.

We've now arranged paid hosting on a virtual server and moved mail over, 
and so far the main linux.net.nz website. Others sites will trickle in 
later as I restore from backups. We'll have to figure out how to keep 
this paid for in the longer term.

Anyway, a big big thanks to Pronet (our most recent host) for their 
donation of hardware and bandwith, and a big public apology to them for 
causing them so much grief.

Thanks must also go out to our previous host Orcon, who were our home 
for many years.

And now, with luck, tonight perhaps I can get to bed before it's time to 
be getting up again :-)

Cheers,
    Nic.

-- 
Nic Bellamy <nic at bellamy.co.nz>