For the second time in less than a month, Techworld has spent several days chasing down a series of half-truths, unsubstantiated accusations and mysterious goings-on surrounding a sponsored survey. In fact, this time it was three sponsored surveys, all on the exact same topic.
What a coincidence, you say, that three groups of people decided to test the same products - anti-phishing toolbars - at the same time. None of it. Phishing is high on the hype-meter these days, and anti-phishing is a selling point for the two new browsers - Explorer 7 and Firefox 2. It makes good sense then to run these tests covering every product in the market.
And that is what one group did. Carnegie Mellon University in Pittsburgh tested all the main products and came to an extremely important conclusion: none of them were particularly good and they were all much of a muchness.
Compare that to the conclusion drawn by 3Sharp in its tests however. It concluded that there was a winner. One that was better than the others. And it was Microsoft's Internet Explorer.
There was also a winner in SmartWare's tests. It wanted to make sure you knew that you are best protected from scam websites by Firefox. Firefox is best. Use Firefox.
Money, money, money
It will come as no surprise to anyone that has worked in the IT press for long enough that the first survey was independent and that the second two were "sponsored surveys". Sponsored by who? Why, by Microsoft and Mozilla respectively, of course.
The audacity of companies in trying to pull this sort of cheap manipulation is staggering, and it is getting worse as time goes on. There is now some bland kind of acceptance that such surveys, even though they are sponsored by the company that always "wins", are still valid. That they're worth reporting on. And that's what everyone does. So long as they keep up the pretence that the tests are done by an independent third-party, everyone is happy to go along with the fact that the results are sort-of right.
But fortuntately this particular glut of anti-phishing surveys has demonstrated that they are not sort-of right, they are in fact sort-of bought, sort-of biased, sort-of corrupt, pointless and damaging. As companies have grown used to the fact that they aren't hauled over the coals for this sort of carpet-bagging, they have grown increasingly confident and blatant in their manipulation.
The counter-arguments to this sponsor-a-corrupt test approach are two-fold.
1. The tests wouldn't happen otherwise and
2. If the test hadn't come out in the sponsor's favour, the report would have been hidden
Both these arguments have become so flimsy in recent years that you can actually see the corporate marketing teams standing behind them, their ears cocked to hear if people soak this nonsense up one more time.
The tests wouldn't have happened otherwise. Rubbish. As this glob of surveys show, if the tests are actually worthwhile running, they will be run - and by an independent body that has no financial interest in the results. If you look at the sponsored surveys that do appear, companies are often right - these tests wouldn't be done by an independent body - and that's because *they're not worth running*.
Even if, by some freakish occurence there was a product comparison worth running and everyone refused - you have to question what the point is when the results will be biased anyway.
The second argument: if the test hadn't come out in the sponsor's favour, the report would have been hidden. This is also crap. But it is a cunning argument because it gives you an unknown - all those hundreds of tests that we never hear about and no one is allowed to talk about either. Whole rooms of reports that never see the light of day. Although of course there is no evidence that they exist. It is the IT Industry's Area 51.
If all these reports exist - how come none of them ever get leaked? You can get memos from the Board of a top company. You can get sales figures, and report of faults, and gossip, and whatever else, but somehow these reports never see the light of day. How odd.
But if you really want to be certain about how absolutely corrupt this phoney practice of sponsored surveys is, just start asking a few questions about it.
How much did the company pay? How many meetings were there? Did company executives meet with the testers at any point between the commission and the final report? Who chose the products? Who chose the criteria? What are the financial terms of the agreement?
Does the company get a bonus if their results are shown to be in the company's favour? Does the testing company do work for the sponsor's competitiors? Does the testing company have any other financial or business arrangements with the sponsoring company?
Likewise, the "independent" testers: how many reports have you done this year? How many have been shelved? How many clients do you deal with? What are the market sectors that you have done work for? What exactly was the input of the company? Were any changes made to your report and the one released?
You will find that you do not get one single answer to any of these questions, usually because they are "commercially sensitive" or that one or other party is "under a confidentiality contract".
The fact is however that if things were above board - in fact if things were exactly how companies and testers constantly paint them out to be - there would be no problem at all in answering some of these questions. But they are never answered and you can see the results for yourself.
There are easy solutions. One is to stop writing about sponsored surveys. The press coverage is the petrol that drives this juggernaut of nonsense. Any company not prepared to answer basic questions about the testing should be ignored, and continued to be ignored until they do so.
There are dozens of ways in which companies can sponsor tests and separate themselves from having any influence. Blind donations. Organised donations - where any company that wants its product tested alongside others can pay a set fee. And any company approached that turns down a request is named but entitled to give a reason as to why they said no.
The fact that these systems do not exist and that everyone remains so cagey about how exactly the systems work would be suspicious in itself. But it is the conclusions that give the game away.
Sponsored surveys are a scam. And a pretty weak scam at that.