Just read it on an italian news site.
SBG2 is the “cooked” site, 3 and 4 were abruptly shut down, SBG1 has some damages.
Can this be called a SPOF?
My Datacenter got toasted…
(No, I’m not using OVH, though I know about them…)
“Well done”. Mostly.
Another sign not to have all your data in 1 DC. Make sure you have your fallback- and/or backup servers in a DC in another town.
I have my VPS in Germany and my backup in the Netherlands.
Usually those datacenter have offsite backup, dont they?!
hum not sure in their even worst scenario they could imagine this
Can I configure identical NS-Backups for different backup destinations?
You can use multiple backup. Can create two backup sets in Cockpit interface pointing two different destinations.
Then we can proactively cover such risks.
But never before have I identified such a risk.
Life proves once again, worse is always possible.
You can, but it isn’t necessary to mitigate this risk. My Neth host is a Contabo VPS in Nürnberg; it backs up to Digital Ocean spaces in .us–both of those aren’t going to burn down at the same time. I also have it backing up to my TrueNAS box at home, but now we’re back to the multiple backup scenario.
I use a Netcup server and my backup server is also Netcup. I don’t know where the Severs are located.
In any case, it is an unhealthy constellation.
" Fire alarm system, targeted fire fighting
Smoke alarms in raised flooring and air conditioning units can detect fires very early on and trigger an alarm. The local fire brigade is familiar with the data centre layout and can target firefighting to avoid excessive damage from fire extinguishing materials."
If I remember correctly someone was afraid of being hit by a bus !
In 2000 the swiss national Telco (Now Swisscom) set up a brand new NOC to centralize connections for the whole east Switzerland, with connection to the national east-west fiber glass backbone.
The building and rooms were impressive: 50 full size racks filled with UPS alone…
From the outset, the building had a defective fire alarm which would set of false alarms to the fire dept & police… This several times a week…
By pure chance, I had helped a friend working there set up his first monitoring system, then Big Brother. My friend was based in central Switzerland, like me, but worked for the Telco and set up monitoring for the site.
Well, as you can imagine, it was a month or two of live operation, when another alarm went off, at 01:00 AM in the early morning. Only this time, the dude at the fire dept shut off the alarm “Not another false alarm at this time!” - and also informed the police!
My friend noticed one server after the other going from green to red - a whole row in a couple of minutes… He also got the alarm call we had set up, and called his superiors… The sh*t hit the fan. Eastern Switzerland had disrupted telephone and Internet for 2-3 days, as the old systems couldn’t handle the additional traffic…
He showed me photos of the rooms there. Brand new servers, ruined not only by fire, but by battery acid, from exploding UPSs… Some not plugged in, but ready to use. Not a nice view at all!
Anyone ever seen molten fiber glass?
My 2 cents
2003 september, full nation blackout during the night. Northern italy came alive at 10am, southern took a lot more hours.
2012 into northern italy a quite big hit of heartquake starting 20/5 till 29/5 gave a little “nudge” to a lot of buildings…
A lot were old or not quake-proof, but in any case was quite a shot.
Things goes sideways everyday.
2003: That was the “famous” tree that fell during a severe storm. It collapsed over one of the International Power Lines connecting Italy with Europe. The Tree was on the swiss Lukmanier pass, connecting north and south obver the Alps.
(Search Google for “lukmanier 2003 stromausfall”).
The other two International lines feeding Italy couldn’t handle the power surge - Italy went dark in minutes!
If I recall right, it happened around 01:00 AM…
I still recall that one!
My 2 cents
Looking at the photos some people joked about the data being uploaded (literally) to the cloud or that the Data Center moved to the cloud.
Normally a fire alarm can detect really early the smoke inside the cooler or in the rooms, they use some specific think called VESDA (xtralis brand) and they use some inert gas to extinguish fires
I would be curious to understand how a building can burn like this, normally the system should stop the fire at the floor level, even at the room level.
So clearly there was something bad or not working in their system.
Maybe the Windows Monitoring System was doing an update…
In reality, one would be surprised how much “Bloatware” exists in european Public Infrastructure…
Here, in public busses, there’s a Monitor in front showing the next stops and the timetable.
Instead of a thin client, repeating it’s instructions, most swiss transport use a full blown Windows 10 System. Bloatware pure, especially in view of what that box is doing…
Occasionally, you can see them manipulating it using Teamviewer.
But VERY often, it just shows a traditional blue screen!
How many fire monitoring systems are badly maintained Windows Systems?
I think anyone purchasing a Windows based alarm system should ask what happens, if Windows is doing an update / reboot when it should sound an alarm…