What a Day!

Matt

Well, I was just signing off my PC at work, and "POP"......everything went Dark.

We had a total power outage at work. Now, because of the type of stuff we do, we should have total failover....this didn't work. The diesel generator didn't come on!

So, about 80 servers, a mainframe, all the PC's in the building, mail server for the whole company, THE LOT went off.

Spent 3 hours bringing it all back up, and POP it went again at 21:00. FUCKING HELL.

So, I've literally just got home, and I should have finished at 18:30. One consolation is that we all got Pizza's bought for us by Senior Management.

Andy_Coupe

Sounds like you had fun then Matt :lol:

Matt

Well one thing for certain, it's been an eventful evening. There is normally only 3 staff on after 7pm, we had about 20 people in upto me leaving at 11:45

dodger

don't you have any UPS so that help the servers shutdown correctly. i have a UPS at home when i used to run a PC 24/7 and still use it for my broadband router, firewall and wireless AP.

Matt

Yeah we do have UPS, but they are investigating why they didn't kick in. There appears to be an issue somewhere, because they weren't being charged.

dodger

the batteries do need replacing every five years IIRC, i have had to change the battery on my one it wasn't expensive

Matt

To be honest, it would normally be a worst case scenario if they were to go onto the UPS. As normally the diesel generator should kick in, and we shouldn't even notice the switch. Yesterday was the biggest outage we've ever had at my work.

Sjdickso

Sounds like you had a good day. You would need some serious UPS for all those servers though, + as you probably know you will only get around 5 - 10 mins depending on load.

Bet you are Glad it's all sorted now though hey.

I had a similar issue about 6 months ago one one site every week at the same time power was tripped in area where the servers where UPS's did kick in but by the time we had got the power back on the servers had shut down. Turned out to be some faulty electric heater that some one was using.

Matt

Bloody hell, thats not good. I'm not back in work 'till 10am, so not sure if they have diagnosed why we lost everything yet.

Edit: Even if the UPS did kick in, without the Airdale units cooling the server room down, they would have all overheated anyway before we managed to sort the problem.

vocky

from my experience with ups systems the switchover pcb is probably fried

without that nothing happens :lol:

Matt

Well we've figured out basically what happened. We lost power to the building, UPS kicked in while the generator starts up........and the generator never started. We've got engineers in now looking at why it never started.

vocky

that's the second most common fault :lol:

flat battery or the control board melts :lol:

Matt

Well we now have some instructions for bringing the gen manually. One of the big points is "Make sure the main is OFF. Having the Gen and the Mains on will blow the UPS up!"