Cisco used a December 2006 ESG report to show that its SAN gear was greener, more energy-efficient, than an un-named competitor. In a blog on this I deduced the competitor was Brocade.

Brocade used an April 2007 ESG Validation Report to show that its SAN gear was greener than Cisco's. Both ESG reports were written by the same analyst, Brian Garret.

The blog suggested it might be difficult to understand ESG's position on the relative greenness of Brocade and Cisco.

Brian Babineau, a senior analyst at Enterprise Strategy Group (ESG), has kindly sent Techworld this information:-

"I wanted to clarify some possible confusion surrounding two different ESG Lab Validation Reports - 'Building Power Efficient Solutions With Cisco MDS 9000 Directors' (December 2006), and 'Brocade Virtual Fabrics and Director Power Efficiency: Consolidated Connectivity and Power Savings' (April 2007).

"There appears to be some confusion based on some of the statements I have heard regarding test methodologies utilized in each report, so I am hoping to set the record straight."

"As stated in the Cisco MDS 9000 report, ESG did not physically test either the Brocade product nor the Cisco product and drew results from public sources including product specification documents. The configuration data presented (also specifically listed within the report) did not present a Brocade configuration utilizing Brocade's Virtual Fabric feature, as it was not generally available at the time this report was written."

"In regards to the Brocade Lab Validation Report, ESG did perform actual physical testing of the Brocade system - those results are factual and validated. ESG did test the Cisco configuration - however, we decided to maintain consistency and utilized the same data from the public sources as were shown in the Cisco report. ESG also incorporated Brocade's Virtual Fabric feature as it was then generally available at the time this Lab Validation Report was written - hence enabling the testing of a completely different configuration."

"As always, ESG recommends IT customers to test products themselves within their specific environment to compare features, performance characteristics and power consumption metrics. ESG Lab reports always clearly articulate exactly what is tested, and how it is tested, in order to provide extremely accurate data - however we always encourage customers to perform real-world testing using their own operating criteria in order to determine how products will behave in the real world. ESG Lab reports are typically tested in average configurations with simulated workloads, designed to emulate specific operating environments to provide an accurate representation of how a product will perform under various loads in different application situations."

Thank you Brian. You state that 'ESG Lab reports always clearly articulate exactly what is tested,' however the December 2006 Cisco report did not say what competing products were tested. The text mentions 'competitive solutions' and 'a functionally equivalent competitive solution' but it does not name the competitive solution supplier nor does it identify the actual products used.

When suppliers come to us bearing analyst's reports that compare their products with a competitor's products then sup with a long spoon and read the caveats carefully. As government gas figures note, 'actual mileage may vary.'