The engineers at vulnerability testing tool vendor nCircle Network Security spend US$100 per month at the coffee shop in the lobby of their office building in downtown San Francisco. But there is one day each month when a trip to the cafe is more urgent than at any other time: Patch Tuesday.

As anyone involved in running a Windows network knows, this falls on the second Tuesday of the month and is the day when Microsoft announces product vulnerabilities and releases patches for them. For the nCircle engineers, it's the start of a very long day that's filled with coffee, work and more work.

"We know we won't get any sleep when it's Patch Tuesday," says Michael Murray, director of nCircle's Vulnerability & Exposure Research Team (VERT), whose engineers rarely go home before 2 a.m. on the big day.

Microsoft last year released 45 security bulletins and patches; 18 of them deemed "critical" (the highest rating) and 19 "important" (the second-highest). October was the busiest and most crucial month, as Microsoft hit customers with 10 patches, seven rated "critical."

As soon as Microsoft releases its patches, nCircle's programmers in San Francisco and Toronto begin unravelling the fixes. It's a race between them and hackers who might be doing the same. The engineers have 24 hours to meet service-level agreements with their customers to determine what has changed in the software and to deliver tests that the customers can use to decide whether their systems need to be patched.

Microsoft doesn't publicise the changes, so that exploits for the vulnerabilities can't be created - not immediately anyway.

Knowing Microsoft's patch schedule lets the programmers mentally prepare for Patch Day. Murray is based at nCircle's Toronto office, but for Patch Tuesday on Dec. 14, 2004, he's at the San Francisco facility.

Early for once
The day begins on a promising note: Microsoft is 5 minutes early, issuing its patches at 10:05 a.m. Four VERT engineers in San Francisco gather at the company's "war room," a tiny office that's barely large enough to house a table that sits eight people. Nine engineers in Toronto wait for Murray to open the phone bridge, which will stay open all day.

Microsoft has issued five patches, each rated "important." Murray, his PowerBook perched on the table amid cables, two digital projectors and the conference call unit, clicks between the pages Microsoft has posted about the vulnerabilities and patches. He scribbles on a whiteboard the vulnerabilities and their Microsoft-assigned numbers, "041 Wordpad, 042 DHCP, 043 Hyperterminal, 044 Kernel/LSSAS, 045 WINs." He puts a cross next to each of the five to denote that they are all "locals," meaning that they could be exploited via e-mail or by people on the system. Next to the Dynamic Host Configuration Protocol (DHCP) and WINs vulnerabilities are two more crosses, denoting that they are also "remotes" - services that are accessed remotely.

"I'll buy lunch for anyone who writes an exploit for Wordpad," quips Christian Hunt, manager of VERT in San Francisco. Murray assigns teams; all the "locals" will go to the Toronto engineers, and San Francisco will take the "remotes." The teams are scheduled to meet every three hours.

Back at their cubicles just outside the war room, the engineers download the patches onto an instance of the affected operating system that they've created using VMware's emulation software. They grab the binaries from the patches and load them into the IDA Pro disassembler tool from DataRescue. IDA reveals the lines of code written by Microsoft programmers, and nCircle engineers pore over the evidence to pick out the differences in the patched and unpatched software.

The engineers jump back and forth from their cubes to the war room, where Murray and Jeremy Cooper, a senior software engineer, are sitting, swapping notes and advice. Sometimes the engineers will communicate via Internet Relay Chat or cell phones - even though they are in the same building. Other times they un-mute the bridge with Toronto. By 11:45 a.m., the tests for 041 and 043 are ready for quality assurance, and Murray orders pizza.

Cooper and Hunt are working on the DHCP vulnerability. Cooper has attached his laptop to the projector and is looking at the DHCP patch that IDA has disassembled. He notices that there are 41 procedures of the DHCP that can be queried directly from the network, but he sees something else that could lead to his first clue. Procedure No. 28 is "Get DHCP version number." He knows Microsoft often renames version numbers in the executables of patched software, and if the patched and unpatched versions of the DHCP show different DHCP daemon version numbers that would be a huge turning point for Cooper and Hunt.

But then Hunt discovers that the unpatched version of DHCP has only 29 procedures that can be queried. These are the "a-ha" moments that keep the programmers interested. "This isn't work; it's fun with a paycheck," says Cooper, the resident disassembling expert.

By 12:50 p.m., the Toronto team has finished creating the rules that will make up the tests for all the "locals" and is waiting to put them through quality assurance. The Canadian team thinks the tests will be customer-ready two hours after they pass quality assurance.

By 2:05 p.m., the air is getting dry in the San Francisco war room; projectors and laptops are spewing hot air, and pizza crusts and empty soda cans litter the table. Hunt is sitting beside Cooper and both have their laptop screens projected on the wall to show the code for the unpatched and patched DHCP. They've finished writing a program to call up Procedure 28 and are waiting for another engineer to create a DHCP test environment.

NCircle keeps a range of popular operating systems, including Linux, NetWare and Solaris, running in its test lab, but it has to be able to mirror most customer environments, however antiquated. The company scours eBay and flea markets for old software, and Murray says the oddest recent customer request has been to test a NetWare 3 environment. "But getting old software is less difficult than finding old hardware to run it," he says.

At 2:55 p.m., the test packages for 041 and 042 are through quality assurance and tests for the rest of the locals are in quality assurance. The rules for the remotes of 045 and 042 are still being developed. Murray says the teams don't compete with each other; rather they compete with the times of the previous months. The record for tests to be finished and ready for shipping is four hours for locals and six hours for remotes, Murray says.

Hunt and Cooper think the DHCP daemon version number has changed from 1.1 in the unpatched version to 4.1 in the patched. They use their test program to call up Procedure 28 - but wait! They discover a huge mistake that would wreak havoc on customers' networks if the customers made the same error. The test has delivered a denial-of-service attack to the test system by calling up the wrong procedure. Everyone runs to the test machine at an engineer's cube to view the dreaded Dr. Watson screen. "Dude, we found another vulnerability!" they exclaim.

The second attempt at calling Procedure 28 on the unpatched version is successful: The version number is showing 1.1. But the test of the patched version is also showing 1.1. That's not good. It turns out human error was to blame again, as the wrong patch had been applied to the test environment. The next attempt is successful, and everyone is relieved. "OK, dump it and make it into a rule," Murray says to Cooper.

Finishing the test for WINS proved to take the longest. The engineers created a test early in the evening, but after Hunt and Cooper finished their work on the DHCP they set about helping to look for a better test of the WINS vulnerability. "We had extra energy," Murray says. The engineers in San Francisco finished at 2 a.m. After catching a few hours' sleep at home, they came back into the office to see the test through quality assurance, and everything was shipped to customers just before 10 a.m.

Patch Tuesday in December came and went. Tomorrow will no doubt be another late one for the nCircle engineers and other such teams across the industry.