If 10Gigabit Ethernet has a problem, it's multiple personalities. 10G over fibre is well established as a network backbone technology, but in past generations it's been Ethernet over copper - and especially telephone-style twisted pair cabling - that has brought the big upsurge in usage and sales.
While it's true that 10G over twisted pair is getting cheaper and less power-hungry, will it ever be cheap enough? And even if it is, how long it will take before it makes sense for server manufacturers to build it in?
"Most [10G] shipments to date have been optical fibre," says Jag Bolaria, senior analyst at the Linley Group. "We think in 2007 the market was about 700,000 switch ports, and this year will be something in the order of 1.2 to 1.3 million. To put that in perspective though, Gigabit is over 100 million ports and Fast Ethernet is over 200 million."
The problem is that fibre is too pricey to replace Gigabit to the server or desk. 10G over copper could fill the latter role, but is still maturing - with the exception of CX4, which is a short range (up to 15m) technology using relatively bulky cables.
The great hope among networking suppliers has been 10GBase-T, the standard for 10G over shielded or unshielded twisted-pair (STP/UTP) copper. It allows 10G to run up to 100 metres - or further, in some cases - over Cat6 cabling.
Initial versions of 10GBase-T were costly and power-hungry, with server NICs costing over $500 and only just fitting within the 25W per slot specification for PCI Express (PCIe) expansion cards.
However, new chips on the way will make a big difference by halving its cost and power consumption, argues Kamal Dalmia, VP of marketing at Teranetics. His company recently began sampling a dual-port 10GBase-T PHY (or transceiver chip) that it claims needs just 6W per port.
Other developers have also announced lower-power PHYs for delivery later this year or next, including Aquantia, Broadcom and Solarflare. All will be backwards-compatible with slower versions of Ethernet, so 10G switches and NICs can be used within existing networks.
"The rule of thumb is the next Ethernet speed becomes attractive when it reaches three-times the cost, and Gigabit today is $100 a port. I expect 10G ports to be $250 to $300, where the first generation was $500," says Dalmia.
Switches will benefit most from the new PHYs now on the way, he says, adding: "The key factor is the ability to build 48-port 1U switches, because there's two main factors to switch density - power, and the physical size of the chip."
He explains that you can fit a maximum of 48 RJ45s in a 19-inch width, but when you put the PHYs on a circuit board behind that, they have to go in multiple rows. This PHY will allow 48 ports in just two rows - with more than two rows, the distance to the front panel becomes too long to meet the IEEE specs on the waveforms and the signal gets distorted.
In addition, the lower power consumption means that a 24-port 10G switch might need 500W all told, Dalmia argues. That sounds like a lot, but it would offer the same switching bandwidth as ten 24-port Gigabit Ethernet switches, needing perhaps 150W each - a two-thirds saving, he claims.
This new generation of PHYs will also include energy-saving modes, where the power is throttled back when the full bandwidth isn't needed, and wake-on-LAN support, adds rival chip developer SolarFlare.
Other players in the business are cautious, however. "Everybody's doing 6W solutions, but it remains to be seen if its really 6W and when they'll ship in switches," says Vik Karvat, marketing director at NetXen, which designs controller chips and intelligent network adapters. "The 12W generation took two to three years - it has to be in a switch from a vendor with a degree of credibility, too."
One possible solution to the heat issue is a short-range copper successor to CX4, called SFP+ Direct Attach. It's thinner, easier to install and lower power, and has the additional benefit of using the SFP+ form factor, so a switch can be built with SFP+ receptacles and fitted with either short-range copper or long-range optical transceivers as needed.
"10GBase-T PHYs are very expensive and extremely power-hungry," says Karvat. "As a result, there's a variety of PHYs, including direct-attach copper which comes from the SFP+ [fibre-optic] community. It's a single lane of copper, not four like CX4, so it is easier and cheaper to cable, and more dense."
It will make it easier to build 10Gig switches, he adds: "Teranetics and Solarflare are both 6W per port, but SFP+ copper is 1W. A 12W solution is impossible in a switch, 6W is extremely challenging, but 1W is fine. There is absolutely no reason for CX4 users not to migrate to direct-attach. Between them, CX4 and direct-attach address the 80 percent of the market that's in-room."
"SFP+ copper seems to be gaining momentum," agrees Jag Bolaria. "But they're still setting standards - should it be three metres or 10, for instance? 3m is the point where you can develop devices without any complex equalisation or EDC [electronic dispersion compensation] circuitry. I think it will probably come out at 10m, but even there you're really looking at connections within a rack or data centre."
The power struggle
Bolaria adds that 10GBase-T - which of course offers ten times the range of SFP+ Direct Attach - will eventually catch up on power.
"Gigabit was 7W, then sub-5W, and six to seven years later we're at less than 500mW," he explains. "First generation 10Gig PHYs were around 10W, the second generation - which is sampling now, with initial products shipping next year - is 5W to 6W, and the third will probably be around 2W."
Vik Karvat acknowledges that, like CX4, SFP+ copper is not a long term solution. "Eventually the market will migrate to 10GBase-T," he says. "6W will be enough of an advantage to drive it into switches, but they probably won't be available until late '09 or '10."
That's the big problem for NetXen and the other makers of Ethernet controller chips: with compatible switches still 18 months away, it's hard to persuade server manufacturers to start building in 10GBase-T PHYs now. Even if they're backwards-compatible with Gigabit or Fast Ethernet switches, they're a lot more expensive than a Gigabit PHY.
NetXen's solution is to persuade the server companies to use a new multi-speed controller chip with Gigabit PHYs for now, but with the addition of a relatively inexpensive connector on the motherboard so that 10G PHYs - whether SFP+, CX4 or Base-T - can be added later via a riser-card.
Called FlexLOM (LAN-on-motherboard), the new chip features four Gigabit and two 10G ports. "Using the same chip for both drives up manufacturing volumes," says Karvat, adding that in any case, putting bits on the wire at a certain speed isn't the controller's job.
"We were always agnostic to pipe speed," he explains. "Our IP has been in the programmability of the metacores - that's where we deliver functions such as bandwidth allocation and stateless offload. This chip is also the first to have full offload for Fibre Channel over Ethernet (FCoE)."
All this begs the question of whether a server will be able to fill a pair of 10G pipes. Not if it's a first-generation PCI-Express (PCIe) system, says Karvat - but an 8x (eight lane) second-generation PCIe 2.0 slot should be capable of 26Gbit/s (out of a theoretical 4GB/s), which leaves room to spare.
He warns though that, without offload capabilities in the NIC, a server could have difficulty doing enough application processing to generate that level of traffic.
For all these reasons, Jag Bolaria says he expects 10G Ethernet to be much less server and PC-oriented than previous generations - and much more fibre-oriented, as well.
"A lot is going to come from inter-switch links - it's not going to be like [10Mbit/s] Ethernet, where the volume came from NICs, or Gigabit, where they made the NIC and PHY costs very similar to Fast Ethernet," he says.
"Last year, Cisco started putting 10G uplinks on workgroup switches, for example to support 24 Gigabit ports. Two 10G uplinks instead of four Gigabit uplinks means a lot less blocking."
He's sceptical too about copper-based 10G being embedded in servers any time soon. Given that servers typically have multiple redundant NICs, and that Gigabit is still much cheaper than 10G, he argues that it makes more sense to aggregate the former in place of the latter.
"I think fibre will stay strong," he concludes. "The issue is heat - optical modules are under 2W now."