A cascading series of problems with a datacentre consolidation project at web hosting company NaviSite left about 165,000 websites offline last week, with some of the sites remaining unreachable for six or more days.
The problems began on 3 November, when NaviSite tried to shift processing done on 850 servers at a datacentre in Baltimore to its headquarters in Andover, Mass. NaviSite acquired the Baltimore facility in August, when it bought Alabanza, another hosting vendor.
Rathin Sinha, NaviSite's chief marketing officer, said the company decided to move 200 of the 850 servers to Andover and migrate the data from the rest of the systems to new machines at the Massachusetts datacentre.
NaviSite told affected customers that their sites would be down for less than a day. But Sinha said that when the company attempted to transfer the data from the 650 servers still in Baltimore, it encountered synchronisation failures that kept mushrooming.
Company officials then decided to transfer another 200 servers to Andover to help reduce the scope of the virtual migration and speed up the data transfers.
But NaviSite ran into more problems. As the host servers finally started coming up, URLs didn't work, and customers were able to access their websites only by using the underlying IP addresses, Sinha said.
Then, he added, NaviSite's network became overloaded because of all the customers trying to get online, further exacerbating the situation.
On Thursday, NaviSite said that 90 percent of the websites were back online. The next morning, the company said it was "down to a final set of [trouble] tickets."
But the length of the outages prompted some NaviSite clients to vow in forum postings that they would switch to other hosting vendors once they could retrieve their data.
Others threatened class-action lawsuits, and New York-based law firm Wolf Popper LLP confirmed that it was looking into possible legal action after being contacted by NaviSite clients.
Cynthia Brumfield, president of Washington-based Emerging Media Dynamics, wrote in a blog posting that the datacentre consolidation was "poorly planned" and had "slipped from mere incompetence to outrageous indifference" to the needs of companies like hers.
Find your next job with techworld jobs