CERN’s Large Hadron Collider (LHC) in Switzerland is the focus of a massive international collaboration to explore the nature of matter and the origins of our universe, but it creates a data deluge beyond the processing power of traditional data centres. The data is therefore sent out to a worldwide grid of 350,000 servers, so that it can be processed and analysed.
The UK contributes the equivalent of over 40,000 cores to this worldwide effort, via a network of 20 academic and research institutions known as GridPP. Although LHC experiments make up 90 percent of GridPP's workload, it also supports other particle physics experiments as well as research from numerous other disciplines – including some proof-of-concept work for industry.
One of the Grid’s leading contributors is the PPE group at the Glasgow University School of Physics and Astronomy. The PPE group's network had grown organically with clusters of Cisco, Nortel and Dell switches, spread over two computer rooms and connected by a 20G backhaul link, but in 2012 it received funding from the Science and Technology Facilities Council (STFC) to upgrade the cluster’s entire network infrastructure.
A competitive tender was put out and an Extreme-based solution was selected, based on its performance, reliability and cost. The majority of tasks executed at the site run in parallel, without inter-process communication, so the need is not so much for high performance computing as high throughput with exceptional reliability and uptime.
“If you have a parallelisable problem that can go into a batch computing system, then the Grid's the place to do it. What the Grid isn't great at is non-parallelisable problems – so big weather systems and hurricane analysis, which is predominantly a niche HPC issue, is not something that we deal with,” said Mark Mitchell, EGEE/ScotGrid Technical Co-ordinator at the Glasgow University.
The Extreme system includes four Summit X670V switches interconnected with 4x40G to provide a 160G backbone within the building, and 10G links to the campus network and the JANET academic network. Two Summit X460-48t stacks also provide a resilient service platform for core services within the Cluster.
“Because of the density of 10G interfaces that were delivered, we can effectively move that up to a 320G backbone,” Mitchell told Techworld.
“We don't actually have a requirement for that, even though we move traffic at about 10G a second internally in the cluster, but it puts us in a position where, if there are major architectural changes in how CPU is delivered from suppliers Intel or AMD, we can obviously meet that criteria.”
Mitchell added that the PPE group is so bandwidth rich now that it could double the size of the site and still not have to upgrade the network infrastructure at all: “We have 2.5 million active files at any one time during some job runs, and we never see a problem on the network, so this shows that basically the network has future-proofed itself for us,” he said.
However, if any changes do need to be made to the network architecture, the organisation can modularly upgrade or remove switches from the core of the network without having to switch off the whole system, thanks to Extreme's OpenFlow approach.
The entire network is overseen and managed through Extreme’s network management tool, Ridgeline, which allows controlled access for nominated IT staff. The system also uses Multi-chassis Link Aggregation (MLAG), which eliminates spanning tree from the network and improves stability.
Mitchell said that adopting the Extreme solution has taken a lot of the onus off the IT team for maintaining the network.
“I'm a former telecoms engineer, and it was something I noticed that we weren't having to look at the network, because we weren't seeing any of the issues with bandwidth throttling between the two rooms,” said Mitchell.
“From there it allowed us to start identifying other areas that were potential bottlenecks within the system, in terms of how we had done hardware architectures, how some of the software had had to be deployed in the past. So basically it's improved the efficiency of the site.”
Mitchell described the level and quality of support from Extreme as “exemplary”. When installing the solution, there was a problem with the code on the X670V switches, and the bonding technology for bringing up the 40G did not work. The problem was reported on a Friday night and, by the following Monday, the main engineer for Extreme in the UK had come to the site and fixed it.
“That wasn't a service we had expected in terms of the thoroughness of what happened, and there was a full post mortem given to us,” said Mitchell.
The Rutherford Appleton Laboratory in Oxfordshire, which is the Tier 1 site for GridPP, (Glasgow is one of four Tier 2 sites), also uses Extreme switches, and the two sites have roughly similar code sets, which means that they can collaborate on trials of OpenFlow and other network protocols.
“We are in a good position to start looking at application intelligence into the network, where applications theoretically can reserve bandwidth and build their own QoS metrics. The Extreme environment's allowed us to look at that, take that slightly further, so moving into an Openflow environment where we can test those kind of things,” Mitchell concluded.