TOEs won't be needed on all servers talking iSCSI. Infiniband is dead too by the way.
The TOE theory says you need to have a TCP/IP Offload engine (TOE) on servers that talk TCP/IP. That's because the processing needed to put packets of data inside TCP/IP packets can consume a lot of CPU cycles. By putting specific hardware on network interface cards (NICs) that connect a network cable to a server the server's CPU recovers otherwise lost cycles. Applications that use that CPU are then unaffected by TCP/IP processing, which is now done on the NIC.
This being the case we might expect all application servers that use TCP/IP to access networked SCSI storage to need TOEs. Not so.
Kianoosh Naghshineh, CEO of Chelsio, a TOE NIC manufacturer, explained where the TOEs will and will not be needed.
Take a NAS gateway box linking to a set of application servers via an Ethernet switch. It aggregates 1gig Ethernet links from the servers and links itself to the NAS gateway across a 10gig link. The servers don't need TOE NICs. They can run Microsoft's software TCP/IP stack. Unless they have significant CPU cycle loss from this they can use TOE-less NICs.
The NAS gateway is in a different situation. It may talk to drive arrays via Fibre Channel and it has aggregated Ethernet links from the servers coming in over a single 10gig link. It has a significant amount of TCP/IP processing. We envisage it running Windows Storage Server 2003 with Microsoft's Chimney support for TOEs.
As the traffic through the gateway increases, because more connected servers are added and more drives are added at the back end, it can find its performance limit being reached. Instead of adding more CPU resource if you can, you add a TOE NIC as the interface to the 10gig line from the switch.
This recovers many of the lost CPU cycles. It is a good example of a TOE NIC being worthwhile and illustrates that instead of, say, 20 servers needing TOEs plus the NAS gateway, only the NAS gateway needs the TOE. This means that IP SAN cost calculations won't include as many TOE NICs as you might think.
In an IP SAN we start with several servers talking 1gigE to a switch as before. This talks to the IP SAN 'controller' over a 10gigE link, which aggregates the traffic from the 1 gigE links. This 'controller' is a SCSI target for the application servers.
As before the application servers only need a 1gigE link and their likely processing drain is relatively insignificant, even using Microsoft's software TCP/IP initiator. NO need for TOEs here.
The IP SAN front end (or controller) is in a different situation. It talks to SCSI targets in the IP SAN - drive arrays for example. It definitely is a candidate for having TOE NICs.
Naghshineh says Chelsio will cost-reduce its TOE NICS such that their price is the same as NICs without TOEs. There will be a zero cost impact from having a TOE. A current Chelsio TOE NIC costs around $795. It could be half that next year. The effect of this will be to make IP SANs even more cost-effective.
Intel NIC opposition
Naghshineh thinks it's not in Intel's interests to support TOEs. It wants to sell more CPUs and its interest in this regard is best served by TCP/IP-onload. Hence its IO-AT concept which says optimise hardware at all levels from server CPU to motherboard to NIC and you don't need a TOE.
Naghshineh thinks the situation is analogous to the time when graphics cards arrived. Intel's response was to add so-called graphics instruction primitives to its x86 instruction set. It wasn't good enough and ceated a window of opportunity for Nvidia and others. Today Intel manufactures its own graphics offload products.
Testing has, Naghshineh says, shown that Chelsio TOEs significantly relieve serer CPUs and TOEs deliver a good ROI.
The implication is that Intel will come back on board the good ship TOE - it did have a TOE product in its range but, Nagshineh thinks: "fired the guy that wrote the TOE code."
Naghshineh thinks iSCSI was first mooted in 2001. Interest rose but: "petered out. Now it's coming back up." The drivers are the arrival of 10 gigE, the lower cost of TOE NICs, and Microsoft's endorsement of iSCSI in 2003.
Infiniband will, he thinks, be pretty much stopped in its tracks by 10 gigE and TOE NICs used to link servers in a cluster: "Infiniband's window has closed. 10gigE has arrived and has better performance."
Combine this Chelsio TOE NIC product story and Naghshineh's perception of the iSCSI market with the Intransa story and we might allow ourselves to think that iSCSI will be of increasing attractiveness to business moving forward.