100G Ethernet has been implemented by Howard Hughes Medical Institute (HHMI), a nonprofit medical research organisation, to boost their network bandwidth for advanced data analysis.
The institute swapped out a Force10 network in favour of Brocade MLXe routers. Fifty-six ports in HHMI's two-router core run 100G Ethernet, making HHMI's Janelia Farm Research Campus the largest single site deployment of 100G Ethernet in a research institute, according to Brocade.
The network will be used by 250 scientists in Janelia Farm. Applications include data analysis for neuroscience and imaging research.
HHMI implemented the 100G network the first weekend in September. The 10G Force10 E1200 network had been deployed in March 2006, says Spartaco Cicerchia, director of network infrastructure systems for the Janelia Farm Research Campus in Ashburn, Va.
40G not large enough
"We have a five-year benchmark for life cycle of gear itself," Cicerchia says. "Looking at the challenges we could potentially face for the next three to five years is large generation and movement of data. There is a large amount of data that is constantly moving. In order for us to be as transparent as possible, we needed to deploy a network that was capable of producing no bottleneck and very transparent between data generation, storage and movement."
HHMI felt 40G Ethernet was not going to be a large enough bandwidth leap and would not give it enough capacity for the next three to five years, Cicerchia says.
"Going straight to 100G would prove to be the right move 36 months from now," he says.
The Janelia network has 200Gbps connections out to each of its wiring closets and 10G Ethernet directly to researchers' systems. From the core, Janelia Farm Research Campus aggregates, or bonds, two ports of 100G Ethernet to create a single 200-gigabit logical connection, allowing line rate bandwidth in both directions.
Previously, Janelia bonded together multiple 10G links to achieve 20G to 40G connections from core to closet, Cicerchia says.
At the aggregation layer, two fully populated Brocade MLXe-32 routers support more than 2,400 Gigabit Ethernet ports and a large subset of 10G Ethernet ports. The core and desktop connections are 10 times that of the previous network, which was 10G at the core and Gigabit Ethernet to the workstations.
The Brocade routers - MLXe32s and MLXe16s - are configured using Brocade's Multi-Chassis Trunk (MCT) capability, in which all paths between the routers are active for increased resiliency. This is designed to overcome the traditional active-passive redundancy of Spanning Tree. MCT lets two MLXe router chassis operate as a single fully redundant active/active logical router with no Spanning Tree overhead.
"We wanted to be Spanning Tree-free," Cicerchia says. "It can be challenging at times in large networks. In Spanning Tree you basically have 50% of your ports in non-forwarding mode. We wanted to have all of our links up."
MCT, though, produced the only glitch HHMI encountered during the 10G-to-100G cutover. There were inconsistencies in the way legacy Force10 switches at the top-of-rack performed IEEE 802.1w - rapid reconfiguration of Spanning Tree - and the way Brocade implemented it while simultaneously running MCT.
HHMI had to "scale back" the switches to 802.1s, in which VLAN bridges use multiple spanning trees to allow traffic belonging to different VLANs to flow over potentially different paths within the virtual bridged LAN. Brocade developed a patch to the 802.1w/MCT anomaly and HHMI will deploy it at its next maintenance window, Cicerchia says.
For additional performance - and security - Janelia Farm partitioned its IP communications and wireless LAN infrastructure using the Brocade FCX PoE+ stackable switches. This configuration aggregates the telephones and wireless access points on a different set of Brocade switches to segregate that traffic from the research data.
"That 200G is for data only," Cicerchia says, referring to the links from the core to the closets.
HHMI may increase its 100G ports beyond 56 if the additional capacity is needed, Cicerchia says.
HHMI ranks as one of the nation's largest philanthropies, investing $770 million (£490 million) in biomedical research per year with an endowment of $14 billion (£8.88 billion).