When you've lived with VoIP for more than seven years, as Mike Shisko has, you learn a thing or two about what works and what doesn't. Inadequate prioritisation techniques can derail a deployment, for example. So can choosing the wrong security gear. Despite some setbacks, the director of IT for Hitachi Consulting wouldn't do voice any other way.

"I cannot imagine going back to an old TDM world," says Shisko, who talked about his VoIP experiences at the recent Network World IT Roadmap Conference & Expo in Dallas. "VoIP makes things so much easier to manage and to use, and is so much more functional for the end-users."

Hitachi Consulting's VoIP network offers its 1,500 employees spread across 18 US and four European offices not only seamless voice, but also state-of-the-art presence, mobility, conferencing and collaboration capabilities (see Four VoIP must-haves). Plus its long-distance calling bill - for the entire global operation - is less than $2,000 a month.

But things weren't always so rosy for Shisko and his team. When the services arm of electronics giant Hitachi first implemented ShoreTel's VoIP technology in 2000, Hitachi Consulting had just 14 offices, all in the United States, and 500 employees, all linked by a WAN with Ethernet links to each building.

"We weren't pushing the network to the limit by any means," says Shisko, who at the time was the lone IT person supporting the Dallas-based organisation. "We hadn't done any testing. We just put the voice on the network, prioritised it and it worked. And it worked because it was a good, fast network and it was utilised to a level that made it available for voice."

VPN speed bump

That all changed a few years later, when Hitachi Consulting decided to handle its VPN services in-house. The firm's Ethernet provider had gone out of business, prompting Shisko to move to a private T-1 network from Qwest Communications.

"Qwest had a managed service where they built our VPN at their core, and it worked very well," Shisko explains. "But we moved away from that because we thought that for what we're paying them to manage it, we can do it ourselves better and less expensively. And that was the beginning of our network problems."

After examining the options, Hitachi's decision came down to two VPN vendors: Cisco and Netscreen. The company went with Cisco, primarily because it was the least expensive option. It installed new Cisco routers, complete with hardware VPN acceleration, and had Cisco engineers come on-site to help with the deployment.

"It was a disaster from Day 1," Shisko says. "Their engineers tweaked and prodded the config, and it got marginally better. But it was still pretty much unusable."

Three months of struggling later, Hitachi decided to do a trial with Netscreen. "We put in the Netscreen and it really was like somebody had turned on a switch - it went from unusable voice over the network to perfect voice over the network," he says. "We didn't replace the Cisco routers - we just turned them into the most expensive CSU/DSUs anywhere in the country."

With the VPN problem solved, Hitachi's VoIP network hummed along for another three years - until the company began experiencing huge growth, in part from acquisitions. "We went from offices that were designed to seat 50 and maybe had five or six people, to having 20 or 30 people in them on a regular basis. And the traffic on the network started to change."

Because Hitachi was handling the VPN on-premises, it could only control voice prioritisation inbound, not outbound. "The voice started to break up — one way, the call would be great, but the other way, you could start to hear the robotics and static. It was a problem."

Moving to MPLS

At that point, the company decided to implement an MPLS network between its sites. "MPLS lets us prioritise voice in both directions on the local loop, which is really where the bulk of the contention comes in, on that T-1 that goes between us and the core," Shisko says. "And life is once again good on the VoIP network."

Mystery traffic

Over the years, Shisko says the firm has bulked up its VoIP management toolbox, and now relies on a variety of tools, most notably Packeteer's line of WAN bandwidth management gear.

The Packeteer tools came in especially handy when the company experienced mysterious traffic spikes from several remote offices. At one site, the company installed an additional T-1 line, only to see it filled up immediately. "They went from using 1.5M to using 3M overnight," he says.

The company, in lieu of bonuses one year, had purchased Apple iPods for everyone. Because of that, the firm was forced to allow downloads from iTunes on the corporate network.

"People were on iTunes buying albums, and for two or three minutes, they were using up a whole T-1 and knocking everything else off," Shisko says. The Packeteer tools helped the firm get a handle on that by enabling it to dial up or down the bandwidth allotted to certain applications. "Now we say, ‘OK you can use iTunes, but we're going to limit you to 128K.' It may take a bit longer to download, but the network doesn't come to its knees."

Over time, Hitachi Consulting has worked through its network and growing pains to the point where it is confident in its ability to successfully manage the VoIP network. "Over the last seven years, we've acquired a few tools, and we've made many improvements on the network," Shisko says. "Early on, we were a bit lucky. Now, we know we can make intelligent decisions as the business evolves."

Four VoIP must-haves

For Hitachi Consulting, four key features of its IP phone system stand out:

Presence. Hitachi's ShoreTel system can indicate presence, so that when users click on an employee's name in the online company phone directory, they can quickly see if the person is on the phone, in a meeting or otherwise unavailable. "It lets us use one receptionist to handle the whole company," says Mike Shisko, director of IT at Hitachi Consulting. "Everyone knows instantly the status of the person they're trying to transfer to, so we don't do blind transfers anymore."

Mobility. A key VoIP selling factor for Hitachi, which primarily employs consultants who travel frequently between offices, is its mobility. "You can be all over the world, but log into your phone and have the exact same contact info," Shisko says. "When I'm in London or Madrid, I just log into the phone there and when somebody calls my 214 number in Dallas, it rings on my desk in Europe."

Conferencing. The VoIP system enables Hitachi to do in-house audio conferencing. The firm had been using an outside vendor to handle 100,000 minutes of conference calls a month at 15 cents a minute. When it moved that capability in-house, via software integrated with the ShoreTel system, the cost went to just 6 cents a minute.

Although usage blossomed to 400,000 minutes per month shortly after the move, the company easily saw an eight-month ROI. In addition, the VoIP conferencing system provides for easier collaboration. "Right from the ShoreTel interface, we can launch a conference, share presentations and do whiteboarding," Shisko says.

Least-cost routing. The ShoreTel system does automated least-cost routing, a feature that helps the company keep its long-distance calling bill less than $2,000 a month. The feature also saves in intra-state calling. Because calls intra-state cost more than long-distance calls, Shisko sets up each switch with a list of "near" area codes. "We don't have an office in Austin," Shisko says. "But I can set up my Chicago phone switch so that it thinks the Austin 512 area code is nearby. So when I'm in Dallas and I call the 512 area code, it actually goes to Chicago and is placed as a long-distance call out of Chicago. And it costs about a cent a minute vs the 4 cents a minute that I would pay calling from Dallas."