Fibre Channel over IP involves the insertion of FC frames inside IP packets and then transmitting them to a Fibre Channel device across an IP link; the Fibre Channel has a 'tunnel' through the IP network.
You start with two separated Fibre Channel SANs. Connecting them via Fibre Channel is uneconomic but you need to connect them. That means a switch port on one SAN has to be connected to a switch port on the other SAN. A device is needed that wraps Fibre Channel frames inside IP packets and sends them across an IP network to a twin device the other side which unwraps the frames and delivers them to the destination switch port.
The IP packets contain the TCP/IP addressing information to get the packet across the IP network to the destination FCIP device.
The Fibre Channel switches either side of the tunnel use E_ports to link to the FCIP device. What we have, in effect, is a single logical SAN.
The device will be a stand-alone box, a blade or a specific FCIP port in a Fibre Channel switch. The FCIP network forms a virtual inter-switch link. The IP network has to have sufficiently good performance and reliability characteristics such that Fibre Channel links spanning it perform correctly in Fibre Channel terms, such as keeping inside timeout limits.
Given a sufficiently high performance TCP/IP link then oceanic and intercontinental distances can be crossed. We can envisage remote mirroring taking place across such links.
As far as the individual SAN islands are concerned they could be co-located in the same data centre. The TCP/IP link doesn't know anything about Fibre Channel or SAN traffic protocols. Were on SAN island to generate mis-formed traffic then it would get passed across the link to the other SAN. There is no routing taking place, no filtering.
If one SAN needs a rebuild due to a port failure or zoning alteration then the other SAN has to rebuild its tables too. If the link breaks then the two SAN islands involved reconfigure themselves.
Protocol and flow differences
The TCP/IP network has different protocols from Fibre Channel for determining if it is functioning (keep-alive messages). These need to be understood so that situations are avoided where the FIbre Channel networks believe the link is broken but the TCP/IP network thinks it has recovered from a break. The Fibre Channel network needs telling that the link is back up so that it can be re-established at the Fibre Channel level.
The different flow control methods need aligning. A receiving fibre channel switch may have a full buffer. The sending FCIP device needs to buffer what it has and stop additional packets coming across the IP network. If it doesn't they will get lost and the transmission will need to be repeated. Time will be lost and, in the worst case, the link declared invalid and SAN reconfuration results.
If there is a network services supplier involved for the TCP/IP portion of the link then there is an organisatonal boundary at the same point as the protocol boundary and this can make fault detection and fixing, and overall network performance harder to ensure. Finger pointing is certainly not wanted in a SAN environment.
A future problem is that if and when SANs either side of the FCIP link grow then the traffic across the link could alter. It could grow in turn and overwhelm the link. Careful monitoring of the network traffic is needed to avoid this or to catch it in time and upgrade the FCIP link appropriately.
Another issue is that the TCP/IP network may well have other aspects to it than just the FCIP link. Yet its management tools cannot see into the link or control the sending and receiving FCIP devices.
There are management, performance and protocol tensions between the FC and IP parts of this overall network which will need continual attention to ensure good performance over time.
FCIP may be a good choice for a bounded remote SAN-linking application. But you might like to consider Fibre Channel routing as a way to add a little more control into the equation.
The Internet Fibre Channel Protocol (IFCP) is another way to join FC SANs together but it needs a separate and forthcoming feature