The Cisco Nexus 9000 series, the fruit of Cisco's Insieme spin-in, is more than another fast router -- it's a change in the way that high-end routers are designed and built. And, in the very near future, it will be a cornerstone of Cisco's application-centric infrastructure (ACI), a tighter melding of applications, servers, and network infrastructure than has ever existed.

We got a first look at Cisco's newest shipping hardware, the Nexus 9508 chassis and the 36-port 40Gbps line cards, in Cisco's own labs in San Jose, Calif. Although we weren't allowed to touch the hardware, we did supervise performance tests that confirmed the awesome throughput of the Nexus 9508.

With Internet-sized packets (1,500 octets), a fully populated Nexus 9508 delivered line speed (just shy of 40Gbps) on each of 288 ports, with zero packet loss, and average latency of 624 nanoseconds (that's .00006 milliseconds) port-to-port on the same card, or 2050 nanoseconds when crossing from one line card to another.

When we mixed up the ports a little bit so that every port sent traffic to every other port (meshed throughput testing), the per-port average speed was nearly identical, although latencies jumped by about 50% over the inter-card latency, with a range of 2,412 nanoseconds (for 64-octet frames) to 6,007 nanoseconds (for 1518-octet frames) and a high of 26,928 nanoseconds (for jumbogram frames of 9216 octets). Again, there was no packet loss.

In IP multicast testing, the Nexus 9508 kept up its line-rate, zero-loss performance, whether one port transmitted to 287 others (1 multicast group with 287 receivers) or broken up into 20 groups of 14 ports each.

These tests were done in Layer 3 mode -- each port was on a different subnet, and the device was routing, not switching the traffic. Cisco doesn't have the switching code ready quite yet, but claims that when the Nexus 9000 is switching instead of routing, performance will remain at line rate.

We also looked at power consumption, an important concern for data center managers. The fully-loaded Nexus 9508 chassis with 288 active 40 Gb fiber ports draws about 11 watts/port with no traffic at all. With ports connected to fiber and powered on, running a typical Internet traffic load (Ixia's IMIX), power load increases to about 16 watts/port. That's a very modest power budget for the speed.

Bottom line: The Nexus 9000 line is built for speed, and lots of it. This is line-rate, non-blocking, 40Gbps routing at large scale, all the way up to 576 ports of 40 Gbps in the not-yet-announced 16-slot Nexus 9500 chassis. Or, if you prefer it as 10Gbps, which Cisco supports using 40Gbps-to-10Gbps fiber breakout cables, there's potential for a mind-numbing 1152 10Gbps connections in the announced Nexus 9508 eight-slot chassis, or double that in the 16-slot chassis to come.

Network managers supporting virtualisation who are thinking about 10Gbps links to servers in data centers, with 40Gbps out of top-of-rack switches, can drop in the Nexus 9500 or 9300 as a cost-effective way of boosting bandwidth within the data center.  The Nexus 9000 series also connects directly to Cisco's Fabric Extender modules, first introduced with the Nexus 5000 and 7000 series, providing a way to deliver over-subscribed 1Gbps and 10Gbps connections in top-of-rack environments.

What makes Nexus 9500 different

Certainly the Insieme team that designed the Nexus 9500 brought a lot of routing and switching innovation to the table. Making use of commodity switch components where they could (in particular, Broadcom's Trident II switching chips) and minimising component count, the 9500 has an elegant design.

No backplane or midplane constrains the device. Instead, line cards inserted horisontally in the front of the chassis link directly to connectors on fabric modules inserted vertically in the back. The fabric modules handle inter-card communications, and provide scalability. This connection is very specific to the line cards and fabric modules. The junction between fabric module and line card is not a general purpose communications bus.

If your requirements stretch to the performance limits of the Nexus 9508, drop in up to six fabric modules to get full non-blocking performance across the entire switch. If you don't need that much throughput, save some money -- those fabric modules have a list price of $16,000 -- and start with just two.

The hardware is also designed for maintainability. Redundant half-wide supervisor modules to handle routing operations and management sit in dedicated slots in the front of the chassis, while redundant switch controller modules go in dedicated slots in the rear. Fans, power supplies, line cards, fabric modules, and almost every other component can be removed and replaced on-the-fly and the whole Nexus 9508 can be broken down into its component parts in just a few minutes.

The Nexus 9000 operating system, derived from the Nexus 7000 code base, has a number of differences that will appeal to data center managers. VxLAN support has been added, a major requirement for large-scale virtualisation. ~~

In addition, Cisco has cracked open the management interface to dramatically increase the network manager's ability to automate and extend the behavior of the switch, with multiple configuration APIs, direct access to the underlying BASH shell in the operating system, integration with orchestration tools such as OpenStack and Puppet/Chef, and the ability to add 64-bit Linux containers into the switch.  

All of these are valuable additions in large-scale environments where configuring networks using the CLI is impractical or too error-prone.

We spent a little time looking at the extended management tools, entirely as demonstrations scripted by Cisco technical team members. When things went right -- which they did more than half of the time -- we saw an impressive array of options to enable different techniques for configuration distribution, control and patching. However, we were reminded that the Nexus 9000 series was not quite ready for production deployment, as management interfaces crashed and we jumped from device to device to get the "right" software build for each of the scripted demos.