Spam has become the curse of our time, with some experts estimating that more than half of all Internet email is now unsolicited adverts. This has created a great business for companies offering anti-spam tools and services - we even get spam from people advertising anti-spam software.
There is a certain twisted logic in this though - Ken Schneider, chief technology officer of anti-spam company Brightmail acknowledges that if anyone is making money out of spam, it is the anti-spam companies. He describes a spam ecosystem, where the people producing spamming toolkits will even bundle in anti-spam software such as SpamAssassin, so would-be spammers can check that the mailservers they hijack are not already blacklisted.
"We have raised the amount of spam out there by raising barriers, as it means the spammers need to send more messages in order to get some of them through," he admits. "The ones feeling the most pain are those email users without protection."
Raising barriers has also created an arms race, in which the spammers develop increasingly sophisticated techniques to beat the blockers, and the blockers hurry to catch up. One of the latest tricks is to embed chunks of ordinary text, perhaps taken from novels or news reports, as a comment invisible to an HTML mailreader, in order to beat Bayesian content analysis. Another is the seemingly innocuous 'I saw this website' message.
"The spammers have two goals, the first is social engineering to get you to read the message, for example with the subject header, and the other is to get past the filters," Schneider says. "It could be HTML with randomisation or added text, or a passage from the Hitchhikers Guide to the Galaxy with a link that really goes to an adult site."
The anti-spam companies compete to see who can develop new rules and blocking technologies and distribute them to their users fastest. For example, Brightmail operates two million addresses to bring in spam for analysis, and also gets around two million suspect emails from its clients every day, most it automatically analysed to create new filtering rules.
"We have a business intelligence team that buys every spam tool going - and half the time we get stiffed on the deal, so not all spam software vendors are genuine," he adds.
"The software comes from bulk mail software vendors - there is some legit software that needs similar techniques. Spam tools come with address lists and tools to randomise the message, plus there's lots of tricks to make URLs look unique, such as converting characters to Hex."
He adds that Brightmail even tests the opt-out links that spammers provide: "Most of the time they work, the trouble is some of the time - maybe 15 percent - you get taken off that list but you get even more spam from other associated lists."
His view is that no one spam technique can be 100 percent effective or reliable. For example, blacklisting of open relays can all too easily block legitimate email as has been seen more than once recently, where over-zealous blocking by AOL techs has prevented AOL users from receiving Freeserve email.
The answer, he says, is to use multiple overlaid techniques, with each one tuned to produce the minimum of false positives. For example, Brightmail uses six different methods, including content analysis, blacklists of URLs and open relays, and body hashes to detect previously-reported spam, and is updating its tools to ignore random text insertions. He adds though that once you are blocking 95 percent or more, you get into personal definitions of what is and isn't spam.
And he is sceptical of legal proposals such as the US opt-out list, pointing out that email is a completely different model to phone and fax. There is also the problem that the list itself would be a hugely valuable target for the unethical, and would be under cracker attack 24 hours a day.
Apart from a long list of known-good addresses, the other prize for any spammer is an open relay to send through. With some viruses and Trojans setting up hidden mailservers on infected PCs, this is a growing problem.
"We're doing a lot more with our customers on outbound filtering - either trial accounts, open relays or increasingly Trojans," Schneider says. "The nice thing as a spammer is you find open proxies and can blast away all day long. No logging, no added lines, no traceability - that's the real value, not the bandwidth.
"We're pushing with our customers for email authentication. You can do some tracking already but email is an open platform. For example I could list in my DNS all the IP addresses on my domain allowed to send outbound - it gives others a traceback, but it is only a building block."
And of course, a technical solution will take time to implement, while the spammers seek ways around it. The genie is out of the bottle and is not going to go back in without a fight.
Find your next job with techworld jobs