One of the little-discussed realities of software development is that many packages ship without having been fully tested. Despite the new emphasis on continuous regression testing and the sophistication of quality-assurance tools, there's an intractable portion of code that just isn't exercised.
This untested code generally lives in routines that handle rare errors, such as disks being full, memory being exhausted, or network services suddenly going offline. These problems are exceedingly hard to duplicate in a testing lab, so the code that handles these exceptions evades thorough testing.
Compuware's DPFS (DevPartner Fault Simulator) 1.0 is the first product to tackle this problem head-on. It simulates numerous hard-to-create errors by intercepting calls to the Windows OS and returning an error indication or failure event. By these means, software is made to exercise all the code that handles exceptions. It's an excellent concept. Unfortunately, DPFS is hurt by its limited support for Windows code and lack of integration with other tools.
DPFS comes in two flavours: a command-line utility and a plug-in for Microsoft Visual Studio .Net. In both, testers specify which faults they want to simulate, where in the program the fault should occur, and how many passes through the code should occur before the fault is triggered. This information is recorded in a file that then drives DPFS.
Once the application starts up, DPFS monitors it and jumps in where and when the fault should be duplicated. DPFS records the results in a log file, which can be examined to glean insights into the program's error-handling mechanisms.
DPFS tests for two broad categories of faults: environmental problems (such as disk full, lack of memory, missing files, or insufficient access privileges) and .Net-specific exceptions. The latter provides a wide palette of possibilities, ranging from I/O errors and network-access problems to XML errors, exceptions in various collections, and even client SQL issues. Unmanaged code triggers only environmental faults, whereas managed .Net code sets off both kinds.
We were considerably excited by the prospect of working on little-tested code. However, as we began testing, we began to rethink this. First, lots of exception code is plain vanilla stuff: You post an error dialog and you close things down gracefully. Only with complex shutdowns or deeply nested exceptions is there cause to worry that code isn't provably correct by code review.
The second problem is that testing the code requires lots of different standalone runs of DPFS, because the first fatal fault shuts down the program under test. To test all possible exceptions, we had to run lots of individual tests and pore over the results.
The most important results of exception testing tend to be those you see as a result of your own code (the dialog box warning, the shutdown, the log file, and so forth). DPFS provides additional information about what's going on, including call-stack traces and the error-handler data on .Net code "catch" and "finally" blocks. This data is useful for tracing the path taken in the exception processing.
Unfortunately, DPFS stops here. The data it generates cannot be integrated with any known code-coverage tool - not even Compuware's own DevPartner product. Hence, a principal benefit of DPFS - demonstrable 100 percent code coverage - is unattainable.
In addition to this lack of data integration, DPFS isn't operationally integrated with the rest of the DevPartner line. DPFS cannot run when any other DevPartner tool is running. And because these tools rely on Windows services, disabling and re-enabling them requires a tiresome sequence of steps. You quickly grow weary of it and simply choose to work with whatever tool is most important to you in that session.
A further problem is DPFS' claim to work with unmanaged Visual C++ code. It appears that "unmanaged Visual C++" means one thing to us but something different to Compuware. To us, it means C++ code compiled with Microsoft Visual C++ or Visual Studio .Net. This isn't the case with DPFS: It won't monitor straight-ahead C++ code unless the code employs Windows-style structured exception handling. So, ported code that relies on return codes to identify errors will realise no benefit from DPFS until the ported code is rewritten.
DPFS delivers the most value in a pure, managed-code .Net application (C# or VB.Net). The Visual Studio .Net plug-in recognises code that can be tested by DPFS and highlights it with squiggly underlines. When you double-click one of these squiggly lines, a dialog box pops up, inviting you to enter specific parameters for the simulated fault, including a description line of your choosing.
The list of generated .Net faults is long, so you can easily re-create many types of problems (such as an unresponsive network or a mangled registry entry) without unplugging the test systems or dickering with registry settings.
This is the context in which DPFS is most useful, although I was frustrated by the lack of support for simulating faults with threads. Such faults are important to simulate for server applications, and increasingly important for desktop contexts, especially in light of the upcoming release of dual-core desktop processors.
Fault simulation is one of those great ideas that make you wonder why no one thought of it sooner. By that criterion, DPFS should be a great package. However, its poor integration with code-profiling tools, conflicts with other Compuware products, and poor support for unmanaged code undercut its value. It might seem unfair to complain about these problems in a 1.0 release, but in view of the $6,000-per-user price tag, it's reasonable to expect a well-designed, well-thought-out, and well-implemented product. DPFS is not that tool, yet.
A great idea that solves a difficult and rarely addressed problem: testing rarely used program-exception code. Unfortunately, this 1.0 release is hampered by lack of integration with Compuware's other tools and limited coverage. Wait for the next release.