It's not playing nice, not when you run an SPC-1 benchmark on a competitor's box and trounce it with your own kit, which is cheaper. Neat marketing, NetApp.
The SPC storage benchmarks, imperfect as they may be, are the only independent way of comparing different storage array products using standardised workloads. 3PAR, HP, IBM, LSI, NetApp, Sun and others have submitted their kit to SPEC benchmarks, such as the transaction processing-oriented SPC-1. EMC has not.
It believes that real world performance varies markedly from benchmarked performance because every storage user is unique. However EMC did apparently contribute to SPC's benchmark work in the early days. It then walked away. Famously EMC's Chuck Hollis, VP for technical alliances, blogged: "We've never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it."
Some words come back to haunt you. NetApp, in a flush of marketing zeal and, no doubt, frustrated by EMC's constant stream of product announcements and pre-announcements, took Chuck at his word and did just that. Ouch!
The results from NetApp's point of view were spectacular. Performance was compared feeding blocks to servers in SAN environment with snapshotting turned on and off. This, for NetApp, was performance comparison time directly on EMC's SAN turf, not on a file-serving basis.
NetApp's FAS3040 produced 30,985.9 SPC-1 IOPS. EMC's CLARiiON CX3-40 delivered 24,997.48, around 6,000 less, a 20 percent performance deficit on the NetApp machine.
With snapshots enabled the comparison was dramatically skewed in NetApp's favour: its box delivered 29,958.6 IOPS and the CLARiiON a meagre 8,997.17, less than a third of the FAS3040's number. For EMC it was a crushing defeat. The difference is so stark that common sense says the FAS3040 will outperform the CLARiiON CX3-40 in any real world application with snapshots enabled.
Snapshotting trims FAS3040 performance by a trivial 3 percent. It slows the CX3-40 to a third of the performance speed with no snapshots. Obviously NetApp has done a far better engineering job with its snapshot implementation.
To rub salt in the EMC wound, NetApp says the results also show that FAS3040 users can expect 68 percent better disk capacity utilisation with its RAID-DP (double parity RAID 6) than CX3-40 users.
Hollis's view of NetApp's coup is rueful acknowledgement of a smart marketing move combined with more debunking of benchmarks. He doesn't deny the benchmark results but asserts, once more, that real world performance will vary.
Another EMC view comes from an EMC blogger who writes, in a comment to an IBM blog: "NetApp boxes perform quite well when utilising less than 10 percent of the total capacity. But that performance does not scale with utilisation - run the same benchmark at 70+ percent capacity utilisation on a NetApp box, and everyone knows that response time goes to heck in a handbasket."
"The benchmark is thus totally useless as a predictor of "real world" performance, much less as a tool to compare different systems."
NetApp points out that its product has a lower cost than the CX3-40 so the effective cost/GB stored is also markedly in NetApp's favour. Whatever EMC says, NetApp has clearly substantially increased its SAN performance storage credentials with this exercise.