As vice president of operations and engineering at US-based data centre operator Peak 10, Jeff Biggs is well versed in protecting IT facilities against power outages caused by extreme weather.
For example, Biggs has taken numerous steps to harden Peak 10’s operations in Florida against the annual threat of hurricanes — like making sure its Jacksonville co-location facility taps into the city’s underground power lines in two places, in case one substation or line goes down. He also bought a massive 1,500-kilowatt diesel generator for Peak 10’s Tampa data centre and signed emergency refuelling contracts with two separate suppliers in case of an extended outage.
But Biggs acknowledged that recent storm-related power outages in Denver, Seattle and St. Louis, all of which left parts of those cities dark for a week or longer, would have tested and perhaps overwhelmed Peak 10’s precautions. “An outage that long — oh my God, it would catch even my fuel suppliers off guard,” he said.
IT managers have focused much attention on efforts to cut the spiralling costs of powering and cooling all the servers in their data centres. But in many cases, they have devoted less thought to better protecting the facilities from power outages, according to some data centre managers and services firms.
In a recent example, a Seattle data centre that supports the reservations systems of a major airline went dark for four hours on 15 December when its backup generator failed to turn on after windstorm-induced blackouts, according to Mark Svenkeson, president of Hypertect, a company that builds data centres.
“They had all of the right pieces in place,” he said. “It just wasn’t well implemented, so it shut the business down.”
Weighing the costs
Although universal power supplies are pretty much a necessity for any data centre, not every company has a backup petrol or diesel generator, which can easily run into six figures.
The Michigan Schools and Government Credit Union in Clinton Township has an uninterruptible power supply that can provide up to three hours of backup power. But for now, it lacks a generator, said CIO Scott Townsend.
“We have enough to ride out short-term outages, which is what we mostly have,” Townsend said. He added, though, that the credit union lost power for a day and a half in August 2003, when a massive outage in the US north-east and mid-west left 50 million people without electricity.
The credit union relies on a disaster recovery vendor for data backups and the ability to quickly migrate its core banking system to a data centre in a different state. But it is scouting out locations for a new data centre, for which Townsend plans to buy a generator.
Svenkeson recommended that data centres have two generators so they have “a backup for their backup.” But he said that because of new US Environmental Protection Agency emission requirements that went into effect at the start of this year, enterprise-class generators have become pricier and harder to find.
The number of back orders at many generator manufacturers is so large that it could take a year to get a device delivered if a company places an order now, Biggs said.
Gary McAuliffe, a vice president at Hosted Solutions, said the operator of collocation facilities bought a data centre at the intersection of two power grids in Boston. Having dual power inlets is “a huge advantage,” said McAuliffe, who manages the Boston facility. “It’s a best practice, clearly.”
But having faced ice storms in North Carolina that caused two-day power outages there, McAuliffe maintains 4,000 gallons of diesel fuel for the generator at his data centre. He also has contracts with multiple fuel suppliers. In preparing for outages, he said, “you have to plan for being able to provide continuous power for more than a week.”