Marvell Semiconductor was struggling with a classic problem: how to protect and secure data at remote sites without putting a technical support person at each site.

Giving end users responsibility, as was done at several offices, inevitably resulted in irregular backups, or even no backups if a departing employee forgot to delegate the job, says Walter Curd, the company's IT director.

To address the problem, Marvell decided to centralise remote data resources and backup operations as much as possible. Sales offices now access their files on Network Appliance Filers residing at the corporate data centre in Sunnyvale, California.

The chip design centres, whose applications are too bandwidth-intensive to support remote access, retained their local servers and storage. However, their data too is centrally stored and backed up via remote replication. NetApp's SnapMirror automatically replicates files to the data centre, where they are backed up from NAS disk to tape using NetApp's SnapVault.

All this put control of remote data where it belongs: in the hands of IT professionals, Curd notes. However, before Marvell could implement its consolidation strategy, it needed to deal with latency issues that were bogging down data transfers over wide-area connections.

The company's solution was to deploy Riverbed Technology's Steelhead, one of an emerging body of products sometimes called Wide-Area File Services (WAFS) or Wide-Area Data Services (WADS). Such offerings use a variety of technologies — ranging from compression to application acceleration — to optimise bandwidth usage and accelerate throughput over long-distance links.

Internal tests by Marvell found that Steelhead devices, deployed at the data centre and at remote sites, performed a first-time file transfer of a 15MB Word document almost instantaneously. The same transfer, done over an ordinary remote connection, took approximately five minutes, Curd reports.

Time is of the essence
Boosting WAN throughput was critical to the success of Marvell's consolidation initiative, Curd says: "Without that acceleration, it would take too long to send data back here" from a remote site.

Demand for products to boost throughput and bandwidth efficiency of WAN networks has exploded, according to Brad O'Neill, a senior analyst at The Taneja Group, who declares, "I'm blown away by demand for this product category in the past year; it eclipses all forecasts."

In a recent survey of 170 IT executives, commissioned by WAN optimisation vendor Netex, 65.3 percent agreed or strongly agreed that their WAN throughput requirements for business continuity/disaster recovery were not always met. Furthermore, 61.8 percent either agreed or strongly agreed that "throwing more bandwidth at the problem does not seem to fix throughput issues."

A Forrester report titled "The WAN Traffic Compression Market in 2005," published in March, says that corporate WAN performance is one of the top five issues confronting enterprise IT infrastructure managers at $1 billion-plus companies. Driving these numbers is a burgeoning corporate movement to consolidate IT resources at the data centre.

In another recent Forrester survey, 80 percent of respondents had either consolidated their remote office infrastructure already or were planning to do so within the coming year.

Today's enterprises are highly distributed. Nemertes Research recently reported that more than 80 percent of employees now work outside of corporate headquarters. Technical decision-makers see consolidation as a means to protect, maintain and administer remote data resources more reliably and cost-effectively.

"Protecting data across a distributed enterprise constitutes some of the largest hidden IT costs," says O'Neill.

However, enterprise consolidation efforts have often been stymied in the past by sluggish throughput. The common response, adding another T1 line or two, doesn't solve the problem, which stems from latencies inherent in TCP/IP and popular applications used to transfer data.

WADS and WAFS platforms are typically dedicated devices that sit at the data centre and at remote sites. They act as gateways, optimising and accelerating communications to provide LAN-like speeds over wide-area links. Once they're deployed, "users don't see any performance difference with the local computing they had," O'Neill says.

Moving up the protocol stack
Until recently, most WAN optimisation products functioned at the network layer. Peribit, Expand Networks, Packeteer, Riverbed Technology, Exinda Networks and Swan Labs are among the vendors whose products apply compression and caching to boost throughput and use bandwidth more efficiently.

A WAN compression device recognises repeated traffic patterns, such as those associated with much-requested files, and stores them in main memory, so that they can be referenced and accessed locally instead of having to travel over the WAN each time.

Forrester Research expects WAN traffic compression revenues to jump from $150 million in 2004 to $250 million this year.

Caching performs the same basic functions as compression, but with this technique, data is stored in disk rather than in main memory. This makes it possible to store larger byte patterns for longer periods of time. Examples include a bulky file or Web page that gets accessed about once a week.

More recently, WAN optimisation products have moved up the protocol stack, using acceleration techniques to solve latency problems inherent in TCP/IP and popular network applications like Microsoft's CIFS.

TCP/IP slows traffic down because it requires data to be sent in "windows" whose maximum size is 64KB. As the receiving device must acknowledge delivery of each window before the next one is sent, the transfer of a large file over long distances can take many minutes, regardless of the size of the pipe.

Companies such as Peribit, Riverbed, Tacit and Netex address the problem by terminating TCP/IP traffic on dedicated devices at each end of the link. The devices then transfer data using a proprietary method that employs much bigger windows, allowing more data to be sent over the pipeline concurrently.

A growing number of vendors, including Riverbed, Tacit, Cisco and Peribit, provide products that address specific latencies in popular file transfer protocols, such as CIFS and NFS, and applications such as Microsoft's Exchange and Lotus Notes.

Latency inherent in these applications stems from the large amount of back-and-forth "chat" involved in a typical file request. "CIFS is not designed for the WAN; it generates lots of round trips in order for client and server to set up a simple file transfer," says Alan Saldich, vice president of marketing at Riverbed. "It can take many, many minutes to open a file remotely because of this background conversation." MAPI is nearly as bad, he adds.

HTTP poses a similar problem, because "a Web page can have hundreds of elements, or objects, that must be fetched serially," Saldich says.

Application accelerators address the problem by identifying common traffic patterns and anticipating requests. For example, a request for the first block of a file generally means the client wants the whole file. Data can then be delivered without waiting for each request and acknowledgement to travel across the WAN.

A multifaceted approach
The last year has seen WADS/WAFS vendors aggressively expanding their platforms to address WAN latency problems on multiple levels. Recent developments in this area include the following:

Cisco introduced a WAFS platform File Engine — the first fruits of its Actona acquisition — which accelerates NFS and CIFS applications over wide-area connections. The vendor is now in the process of acquiring Fineground Networks, whose Velocity-FS WAFS platform is said to accelerate CIFS and Microsoft Distributed File System over the WAN without the need for special hardware at remote offices.

Peribit recently expanded its Perisphere platform (which provides compression, caching and TCP/IP acceleration) with Appflow, which addresses latency problems in CIFS, MAPI, Exchange and HTTP.

Swan recently announced iSCSI acceleration for its WAN Optimizer platform, which boosts TCP/IP throughput over WANs.

Combining different acceleration and optimisation technologies has resulted in some impressive throughput improvements.

Riverbed's Steelhead, for example, accelerates TCP/IP performance through a combination of windows scaling and scalable data referencing, which involves identifying and storing frequent traffic patterns locally, and reusing them rather than resending them each time over the WAN. The combined techniques can boost throughput by a ratio of multiple hundreds to one, Riverbed claims.

Tacit uses a combination of compression, caching and file-aware differencing — sending only those pieces of files that have been changed — to boost throughput for CIFS and NFS traffic.

In a test performed by the University of New Hampshire InterOperability Laboratory (UNH-IOL), a 1.6MB Word file travelled 78 times faster between New York and San Francisco when Tacit's iShared appliances were deployed, than it did over an ordinary WAN connection.

The real deal
Customers, too, report impressive throughput gains from their WAFS deployments. For example, when German chemical distributor Brenntag decided to centralise IT operations at branch offices, remote users began experiencing response time problems. They would typically wait 27 or 28 seconds to open up a 1.5MB Word document, "which to a user feels like an eternity," says managing director Michael Langborg.

Upgrading WAN connections from 0.5Mbit/s to 2Mbit/s resulted in no significant performance improvement, Langborg reports. However, when Brenntag installed Tacit's iShared platform, an initial file transfer took only three seconds. "The experience for the user is like a LAN," says Langborg.

Through its IT consolidation project, Brenntag has realised savings in manpower, hardware, software and bandwidth connections to the tune of about $603,000, Langborg says. He estimates that the Tacit deployment saved about $241,000 out of that total. Furthermore, he expects that an upcoming project, the deployment of VoIP traffic over the remote links, will require no increase in bandwidth because of the efficiencies provided by iShared.

By making the consolidation of IT resources more cost-effective and therefore feasible, WAN optimisation products are helping to "fundamentally change the data centre architecture," O'Neill says. "Once you learn to optimise throughput across a large number of sites, it opens up an entirely new approach" to managing IT resources for remote offices. Geography will become less and less of an issue when it comes to deploying new IT projects. Ultimately, he says, distinctions between distributed and centralised offices will disappear.