Subnets began dropping off the MortgageIT network one after another. Entire bank branches went off-line for days as Joe Bruner, network engineering manager there at the time, scrambled to purchase and install replacement parts.
At first, he figured some of the new WAN interface cards (WIC) he recently installed to upgrade 50 Cisco 2811 routers during expansion and reorganisation were faulty. But as more routers failed and dropped off the network, Bruner realised he was dealing with fakes.
Thirty cards turned out to be counterfeit, he says. Despite repeated calls and e-mails to his supplier, Atec Group, the issue was not resolved.
Nor did he get an answer to the most important question: How did a registered Cisco reseller (also a platinum Network Appliance partner and gold partner to Microsoft and Symantec) acquire the counterfeit WICs in the first place?
What he didn't know was that phoney network equipment had been quietly creeping into sales and distribution channels since early 2004, when manufacturers began seeing more returns, faster mean-time between failures and higher failure rates, says Nick Tidd, vice president of North American channels for 3Com and president of the Alliance for Gray Market and Counterfeit Abatement (AGMA).
Counterfeit gear has become a big problem that could put networks - and health and safety - at risk. "Nobody wants to say they've got counterfeit gear inside their enterprises that can all of a sudden stop working. But it's all over the place, just like pirated software is everywhere," says Sharon Mills, director of IT procurement organisation Caucus.
10 percent counterfeit suspected
There are no statistics specific to network hardware counterfeit rates. But according to a white paper by AGMA and consulting company KPMG, counterfeit products account for nearly 10 percent of the overall IT products market.
"That's $100 billion in fake memory sticks, drives, monitors, networking gear and other IT products floating around out there in black and grey market channels. This has huge implications for the enterprise," says Tidd, who became involved in his first counterfeit case in 2001. That case led him to a Canadian reseller who also was under investigation by HP. Out of that case, 3Com and HP, along with Cisco and Nortel, founded AGMA.
It was when the supply of second-hand equipment from the dot-com fallout dried up that vendors and resellers started seeing counterfeits in the grey-market channel, where used and refurbished products are sold, says Phillip Wright, director of world-wide brand protection for Cisco, which is the most counterfeited brand.
"Users got a taste for new used equipment at bargain prices. So counterfeiters moved in to meet the demand," Wright says. "It didn't help that some resellers turned a blind eye to possible counterfeits so they could keep their own revenue streams going."
The vast majority is still being purchased from grey market, uncertified resellers who unload their goods on eBay at extremely low prices, says Scott Augenbaum, supervisory special agent for the FBI Cybercrime Fraud unit.
These parts sometimes move sideways into the hands of legitimate resellers and integrators.
"Recently, I did some voice over IP integration for a client in Huntsville, and the engineer there asked if he could pay me with five extra VoIP network cards he had left over from the project," says Neal Rauhauser, founder of system integrator Layer 3 Arts. "I got four cards I could use, and one that was counterfeit."
Fortunately, Rauhauser never installs anything before checking it first. He's wise to counterfeits, having had his first run-in with such products in 2004, when two of six new Cisco 1721 routers started acting up at one of his client sites, a large auto manufacturer in Michigan. They turned out to be counterfeit, and he has since been campaigning against counterfeit products.
There were visible differences between the counterfeit and the real gear, he says, but only after close inspection. The counterfeit VoIP card had a brand-new box even though the card was four years old. He also noticed discrepancies in packaging and labelling.
"The printing on the bar-code label was fuzzy like it'd been printed off a low-quality printer instead of a laser. And its internal packaging was a plastic bag instead of a plastic box like the others," Rauhauser says.
He contacted the customer who gave him the product, and the customer admitted he bought the cards off eBay. The four good cards came from a reputable seller. The bad card came from TFS Systems, which claims to be a Cisco registered reseller that buys only from Cisco's top-tier distributors. Rauhauser took pictures of the differences in products and called TFS to find how they wound up selling counterfeit product to his client.
"They were ready to pull my leg and tell me I was wrong. So I told them I was going to the FBI," Rauhauser says. "Then they asked me to box it up again, keep it pristine and they'll get me my money. I'm sure they sold it again on eBay right after they got it."
In the MortgageIT case, Bruner figures his representative at Atec got burned when she went outside her normal supplier to purchase the cards in late 2004.
"We were notoriously cheap with our equipment purchases, so she might have bought from someone besides Ingram, her usual supplier, to get us a better bargain," says Bruner, who left MortgageIT in July, shortly after Deutsche Bank signed an agreement to buy the company.
How Atec came into possession of the counterfeit WAN interface cards can only be hypothesised because repeated calls and e-mail to Bruner's former representative at Atec, and to the company vice president, were not returned. The company's operations manager says MortgageIT was a big client, and sales representatives don't see the gear that's being shipped to their clients.
Gambling on quality
No matter how the counterfeits got into MortgateIT's authorised channel, such slippages highlight the complexities of dealing with this problem - not just in the sales and distribution channels, but also in the manufacturing supply chain, says Pete van de Gohm, director of IT security and quality at Bayer.
AGMA's Tidd acknowledges this, adding, "In some geographies, you've got resellers and distributors blending their inventories, which is why a single shipment might contain five good and five counterfeit parts."
It's difficult to control past the distributor layer, Tidd says, especially when Cisco has 28,000 registered resellers, 3Com has 3,000 and so on.
That means organisations face loss of equipment that vendors may or may not support (Cisco handles on a case-by-case basis). They also could experience critical network outages that, in the right circumstances, could affect human health and safety.
"What if it wasn't a bank subnet that went off-line because of a faulty card in the router? What if it were an air-traffic control network instead?" van de Gohm asks. "This is no different than counterfeit medicine in the pharmaceutical industry. And it's potentially just as life-threatening."
Such concerns also grip the network vendors whose reputations and brands are at stake if they can't stop the dumping of counterfeit parts into the channel. "We worry about things like wiring in the motherboard overheating and the potential for network outages that would impact personal health and safety," Wright says.
Efforts to fend off fakes
Manufacturers are working on ways to make their products harder to clone through use of packaging labels, logos and three-dimensional holograms. Vendors such as 3Com are working on RFID tagging systems, and cryptographic machine authentication is a viable option to help devices call home.
For the past few years, Cisco and 3Com have been building anti-counterfeit culture into every level of their product-to-market channels, educating suppliers and distributors about what they need to do to protect their own channels, while building international investigative teams to help law enforcement agencies shut counterfeiters down.
Cisco's 30 investigators stationed world-wide are dedicated to 200 active counterfeit cases at any given time. From the Mandarin characters on the back of his business card, it's clear that Wright spends a lot of time in China. And a whiteboard behind Wright's desk has a hand-drawn diagram titled "Stopping the counterfeit flow," which contains multiple loops back to Chinese distribution and law enforcement intervention points.
According to Wright and Tidd, China is the source for most counterfeit gear. Tidd toured multiple floors of counterfeit consumer electronics and network gear last year at a public shopping mall in Shenzhen, China.
"3Com has done raids in China, co-operating with local law enforcement who've shut down factories and seized counterfeit goods. Once they've done the seizure, we go in and try to figure out how many boxed products went out before they were shut down," Tidd says. "Unfortunately, as fast as you shut the factories down, other factories go back up."
According to the AGMA study, the United States is the second major point of origin for counterfeit goods - California in particular, say Rauhauser and Dana Andrews, owner of Digital Surplus in Boston. They point to the port of Los Angeles as a big dumping ground for Chinese counterfeit parts and to Silicon Valley as a place of production.
Since 1994, Andrews has made a pastime of helping federal agents catch criminals he says are polluting the reseller channel and costing him his business. In February, he helped the FBI catch a fraudulent buyer who had set up a phoney escrow company and tried to scam Andrews out of half a million dollars worth of Cisco gear.
While law enforcement agencies wouldn't talk about specific cases, a June raid on Sun Valley Technical Repair could turn out to be a big case of counterfeit in Silicon Valley.
Reports in the San Francisco Chronicle made it appear at first like an immigration raid, as 12 illegal immigrants were taken away. But that wouldn't explain the presence of so many agencies, including the FBI, the US Immigration and Customs Enforcement, the US Postal Service and the Rapid Enforcement Allied Computer Team, which investigates large-scale, high-tech piracy and counterfeit cases.
Law enforcement efforts are helping to shut these factories down, say Tidd and Wright, who is active with Business Action to Stop Counterfeit and Piracy, which is sponsored by the International Chamber of Commerce.
Industry leaders need to do more to keep counterfeit out of the distribution channel, resellers and users say, before it affects public safety.
"The networking industry should reach out to other industries that have problems with counterfeit parts," van de Gohm adds. "The industry should apply the best practices already learned in the auto, pharmaceutical, aviation and other industries where counterfeit parts could result in loss of life."
Recognising the real deal
Because clones and packaging are getting more realistic, many people don't realize they have counterfeit network equipment until it's installed and begins acting quirky. Outages and failures are often the tip-off that the gear is fake.
* Don't shop on eBay for deeply discounted gear, particularly from sellers in China.
* Don't go outside your trusted channel to buy critical network components.
* If you're in the market for refurbished gear, the safest bet is to purchase certified products through the manufacturer.
* Check serial numbers against the vendor database.
* Check the packaging carefully, inspecting for anything out of the ordinary in the logo, size and type of packaging materials by comparing them with others in the same shipment.
* Closely examine the gear and compare holograms and chip sets.