Davidson Healthcare got a wake-up call recently when a vulnerability scan discovered that the network was missing more than 4,000 cumulative patches on its 30 servers and 500 workstations.

"I had this bad feeling about those patches," says Kevin Buchanan, who runs Davidson's 10-person IT department. "The problem was that we couldn't keep up with the volume of Microsoft's patches. They were releasing them way too often, there were too many of them, and a staff like ours had no way to manage this. Yet if we didn't, we knew we'd be at risk."

Buchanan quickly secured funding for automated patch management software from Shavlik Technologies.

Adam Hansen, manager of security at Chicago law firm Sonnenschein Nath & Rosenthal, has a similar tale. With nearly 2,000 servers, desktops and laptops spread across 11 US offices, Hansen knew his firm had to get an automated patching solution, and quick.

"About a year and a half ago, this all came to a head," Hansen says. "We did a vulnerability assessment and found we were only about 15 per cent in compliance in terms of patching." He began looking for something that could automate patching and provide real-time reporting.

Sonnenschein Nath & Rosenthal went with PatchLink, another of the pure-play patch vendors that were first to market with automated patch tools.

"The old way of doing things, deploying patch by patch, is not effective in the long term," Hansen says.

There's widespread agreement on that point among enterprise IT executives, analysts and vendors. Any lag between detecting a vulnerability and correcting it leaves an organisation open to attack. And with their own set of automated tools, hackers can strike almost as soon as new vulnerabilities are discovered.

But users are finding that even with automated patch management tools, patching can be a complicated, laborious and often problem-causing process because patches have been known to break applications.

First things first
According to Meta Group analyst Peter Firstbrook, the first step is a network assessment. "My biggest piece of advice to customers evaluating patch management solutions is to take a step back and evaluate your own organisation first," Firstbrook says. "So much of what has to be done is process and procedure."

Even the best tools won't save you if you don't have the right processes in place and the people and computing resources to back them up, he adds.

Davidson used to patch machines when they needed to be serviced or when a new image was pushed out. Now the patching is an ongoing part of IT maintenance, typically performed in off-peak hours, Buchanan says.

The patching path not taken
The issue for IT executives is not whether to automate, but which product to buy. There are the pure-play patch products from Shavlik, PatchLink and others. There are patch tools that are part of larger software suites that include life-cycle management, change management, security management and configuration management. Plus, there are Windows-only tools from Microsoft.

Angela Triola, an infrastructure analyst at ENT Federal Credit Union tackles patching with Enterprise Configuration Manager (ECM) from Configuresoft. ECM performs a variety of functions, including vulnerability assessment, change management, compliance, remediation and patching. Triola says everything from finding patches to testing to deployment to verification has now become manageable.

"We rely heavily on Microsoft," Triola adds, "so the fact that Configuresoft works well with Microsoft was very important to us." She says that Microsoft's own patching advances, including the forthcoming Windows Update Services (WUS) patch management software, will not change the need for third-party tools.

Other IT professionals agree. "Essentially, even with [Software Update Service], WUS or [Systems Management Server], you're still taking a patch-by-patch approach," Hansen says. "You still need the entire platform to see this through from start to finish."

At Sonnenschein Nath & Rosenthal, Hansen has migrated from PatchLink to Citadel's Hercules, which performs network discovery, vulnerability assessments and ongoing compliance audits.

"What I like about Hercules is that it frees you from worrying about the fine-grained details of patching. You define a baseline, and it becomes the product's responsibility to get your network devices in step," Hansen says. "We needed a solution that did more than just pushing patches, which is what a lot of products on the market do."

Hansen says he likes Hercules' strong reporting and policy-enforcement features, which he argues are essential to any successful patching strategy. "Even if a properly patched image is pushed out to a new server or desktop, who's to say that something doesn't get changed along the way, a vulnerable port opened, an unpredictable service turned back on?" Hansen asks.

An agent-based solution, Hercules keeps tabs on those devices, noting any deviation and, if the vulnerability warrants it, quarantining devices that are not compliant. With more mobile devices entering the network, this feature is critical.

Hansen says Hercules enables a lightning-quick response to vulnerabilities, with most of the patches tested and pushed out in less than 24 hours. Moreover, even with a large mobile user base, the firm's patch-compliance ratio is close to 80 per cent, with the network getting more airtight each day.

The process of integrating Hercules wasn't problem-free, however. "We had issues with getting the Hercules agent to work with certain images, but Citadel's customer support was responsive and worked with us to resolve those issues," Hansen says.

EDS opts for Opsware
Larry Lozon, the vice president of hosting services at Electronic Data Systems (EDS), has 70,000 distributed servers and a massive number of other devices. Not only that, but EDS also now requires that any device that enters its network must have end-to-end configuration and patch management.

EDS turned to data centre automation vendor Opsware, which provides server management, provisioning, configuration, change management and patch management. Lozon adds that the next problem on EDS' patch and configuration management radar screen is mobile devices.

Patch payoffs
The payoff from patch management tools can show up almost immediately. According to Davidson, it realised ROI with a single patch cycle. "If you consider our previous process, where we had two technicians walking from machine to machine and patching manually, the cost of the Shavlik product was equal to what their salaries would have been during that period of time," Buchanan says.

Another Shavlik customer, Indiana University, agrees. "The cost savings was so immediate and obvious, that we didn't even bother to quantify it in order to justify the purchase," says Jim Kippenbrok, manager of local support provider services for the Bloomington school.

Hansen estimates that Citadel's Hercules saved about the equivalent of a mid-tier full-time IT employee's salary, or about $60,000. "And this doesn't include the savings in terms of reduced risk," Hansen adds.

Automation is the way to go
Customers say they're generally happy with their patching tools, but there are still issues of concern, such as the ever-shrinking window between when a patch is issued and when it needs to be deployed, the need for good internal testing, interoperability issues, the importance of nailing down a workable patch cycle and the occasional customer service problem.

Because many companies have home-grown or niche applications running in a mission-critical capacity, the patching vendor must work with them to ensure that the patches won't cause more harm than good.

However, when assessing the risk of not patching, most customers agree that patches have become so stable that the risk of leaving a vulnerability unchecked vastly outweighs any interoperability risks.

"Let me go out on a limb here and say that if you are an IT manager and you ignore patch management, you are negligent," Buchanan says.