A rare large-scale, wireless LAN stress test using three vendors' equipment found many WLANs could run into a performance ceiling as they grow in size and traffic. The study's authors say the problem lies not with the vendors' gear but with the design of the 802.11 protocol.
The tests confirm two troubling issues for high-density nets, according to Novarum, a consulting firm that developed and ran the tests (PDF version). First, co-channel radio interference among the access points clobbers the aggregate throughput of the WLAN. Second, the conventional thin access point/controller architecture doesn't scale well as the number of access points increase in a given area.
According to Novarum, performance degrades because of radio interference in dense deployments, and because the conventional model for enterprise wireless LANs - thin access points linked with a controller - scales poorly.
These problems are related to the way the 802.11 media access control (MAC) layer is designed, how it handles acknowledgements and retries, and how these become problematic under high and sustained traffic loads. In many WLAN products, and the WLANs built with them, this protocol inefficiency hasn't been an issue, because there have been relatively few wireless clients and their traffic has consisted of relatively light, and bursty, data traffic.
High-throughput WLANs based on 802.11n, especially running in the 5GHz band, will partly mitigate these effects, according to the study's authors, but not entirely. Because 11n offers a larger 'data pipe,' it takes more traffic to reach network overload in a dense WLAN. But because of 11n's higher throughput, enterprises are looking to do more with it, such as voice and streaming video, which can push the net toward overload.
"If you don't deal with co-channel interference and [the need for] access point coordination, it will put more pressure on the protocols supporting [wireless] voice and video," says Phil Belanger, co-founder of Novarum.
Designing a real-world test
Novarum's tests, conducted in the fall of 2007 in a vacant second-floor office in Sunnyvale, California, are unusual because of the number of access points and clients actually deployed. As the report points out, WLAN tests typically involve a single access point and about a dozen clients deployed in a lab-like environment. (Though Network World in 2006 conducted a large-scale WLAN test of 25 access points.)
In this case, Novarum used 72 wireless laptops and up to 54 wireless VoIP handsets, linking to a typical office wireless LAN first with 15 and then with 10 access points. The net was recreated each time with access points from Aruba Networks, Cisco, and Meru Networks.
Novarum designed and ran the tests. But the project was paid for by Meru (the office space had been rented by Meru for employees who had not yet moved in).
Seven tests were run, in most cases using first 15 and then 10 access points. The access points had 802.11abg radios, but Novarum ran the test only for 11g in the 2.4GHz band. One test was data only, with the 72 laptops; others tested only voice traffic, with 24, 48 and 72 simulated VoIP conversations. Two tests mixed voice and data, and one tested the VoIP handsets to find out how many simultaneous calls the WLAN could support.
"There seems to be a threshold, and beyond that the [WLAN] system doesn't behave well," says Novarum's Belanger. "The bigger the load, the more errors [are introduced], leading to more retransmissions, and this [in turn] adds to the load. And that adds to still more errors."
The brands of WLAN gear handle this spiralling effect differently, depending on the kind of network architecture they use. Aruba and Cisco use what Belanger calls a microcell architecture: a stripped-down access point linked with a centralised controller. Adjacent microcells run on separate channels, with some coverage overlap to provide seamless coverage and roaming for mobile clients. This model is used by most of today's WLAN vendors.
Meru by contrast can set all the access points to run on a single channel, and exercises much more control over its "on-air" behaviour. The Meru controller can see the transmit queues on each access point for each associated client, and can allocate how much time the access point devotes to each, according to Belanger. Instead of trying on their own to get through the transmit "door," and jamming into it, the Meru access points patiently wait for their turn, so throughput is steady, predictable and optimal. That's the theory.
One other vendor, Extricom, takes a somewhat similar approach, though there are significant differences from Meru. Extricom packs four radios into a single device but runs the entire 802.11 MAC in its controller. The access point is nothing more than the radios and antennas; it even lacks a CPU. Extricom claims to eliminate co-channel interference and to optimally manage client wireless connectivity on a per-packet basis.
According to Belanger, in the data-only test, with 72 clients and 15 access points, Cisco and Aruba delivered less than 50 Mbit/ss of total system throughput. In other words, networkwide, each client was getting on average less than 1 Mbit/s throughput. One would expect something like 20 Mbit/s throughput per access point, or 300 Mbit/s aggregate for all 15, Belanger says.
But when the number of access points was decreased to 10, throughput jumped. In Aruba's case, it jumped nearly 40 percent, from 47 Mbit/s to 64 Mbit/s. "More [access points] allow more simultaneous transmissions, which caused more interference and lowered performance in these systems," the report says.
In Meru's case, the office was covered by five access points, all running on the same channel. Two more sets of five access points were added in the same locations, each set with a different one of the two remaining 2.4GHz channels. All ran at their maximum power setting. Meru delivered 100 Mbit/s of system throughput, or more than two times that of Cisco and Aruba. Interestingly, Meru's aggregate dropped to 60 Mbit/s when Novarum reduced the access points to 10 devices.
"It appears that co-channel interference from adjacent APs has a significant impact on the performance of the micro-cellular systems when they are loaded," the Novarum report concludes. Belanger notes the interference range of an 802.11 radio far exceeds its effective communications range. "Under heavy, constant loads, this interference becomes a factor," he says.
According to Belanger, at some points in these highly loaded networks, 30 percent to 40 percent of the packets in the air were retransmissions by the Aruba and Cisco access points. Meru had far fewer retransmissions.
Voice calls run into limits, too
In general, the Meru architecture also offered better voice support, being able to handle more simultaneous calls (Novarum didn't encounter an upper limit in the tests), and deliver toll quality, according to Belanger. With 10 Cisco access points, Novarum found the Cisco WLAN able to handle about 24 VoIP calls. With 48 or more simulated calls, Cisco was unable to deliver toll quality. When real handsets were tested, the Cisco infrastructure seemed to top out with 26 or 28 simultaneous calls.
In the report's appendix, Belanger writes, "I was surprised to see how easily we drove these systems to unstable behavior" with lots of access points and high, sustained loads, a problem he sees related to the 802.11 MAC protocol he championed. "There is nothing wrong with the Aruba and Cisco systems," he writes. "They simply chose not to address this [issue]."
Work by two IEEE 802.11 task groups, 11k and 11t, will address it, at least in part, by making it possible to exercise more control over access point radios and to some degree over client radios. That work is still in progress.
The question of bias
How much reliance one puts on the Novarum stress test depends on two issues: whether you agree with the testers' assumptions, and your assessment of Meru's influence on the test and the independence of the testers.
For example, in the "Conclusions" section of the test report, Belanger writes "The only unusual thing about our testing is the constant load during the data tests. That type of load is similar to 72 people on the same network downloading a movie from [the] iTunes store at the same time." Belanger himself admits this is not typical behavior for most enterprise networks but goes on to write "we would expect any of these systems to be able [to] handle it gracefully."
Was the test fair to Cisco and Aruba? Meru engineers configured and tuned the Meru equipment, a benefit not accorded to Aruba and Cisco.
Belanger acknowledges the appearance of bias but says the tests really were designed to test the optimal configurations for Aruba and Cisco, and were mainly concerned not with rating the three products but with the behaviour of the 802.11 protocol under load, and how different architectures dealt with that behavior.
"[Meru executives] wanted it to be about 'our product is better,'" he says. "But we were interested in the larger issue of the different architectures."
In an appendix to the complete test report, called "Full Disclosure," Belanger says his personal bias has been "in the micro-cellular, Cisco and Aruba camp." That's in part because he acknowledges he was a major proponent of this approach during development of the 802.11 standard. While at Aironet (later bought by Cisco), Belanger says he led the company's work on developing micro-cellular software for early wireless products, and much of that code is still part of the Cisco Compatible Extensions software.
Details of Novarum's test environment
Location: Second floor of typical Sunnyale, office building measuring about 20,000 square feet with a mix of hard-walled offices and cubicles. Testing was conducted from 9am to 9pm on normal workdays. Radio-frequency (RF) scans showed access points at neighbouring businesses, but at very low power levels.
Infrastructure equipment: Alcatel-Lucent OmniAccess 4308 (rebranded Aruba 800, according to Novarum) with software Version 18.104.22.168, and 15 Aruba AP 70; Cisco 4402 Wireless LAN Controller with Versions 4.0.217 and 4.1.185 and 15 Cisco AP1242; Meru Networks MC3000 controller with Version 3.4 and 15 Meru AP208.
All access points had two radios, but the test used only the 2.4GHz radio and 802.11g. Chariot Console 4.2 was used to generate traffic for the tests and collect performance statistics. Wildpacket's AiroPeekNX 2.0.3 was the 802.11 packet sniffer, Cognio Spectrum Expert 3.1.67 was used for RF scanning.
Client equipment: A mix of 72 Windows XP notebooks: Lenovo Thinkpad T43, and HP 510 and HP Compaq nc6200. All used the same wireless PC card, the Cisco Aironet 802.11a/b/g Cardbus Adapter, and all ran the Aironet Desktop Utility Version 22.214.171.124, which Novarum says is compatible with Version 5.0 of Cisco Compatible Extensions (CCX).
Notebooks were spread around the offices, cubicles and conference rooms; all stayed in the same locations throughout the tests. For voice tests, 50 Ascom i75 VoIP phones, with Version 1.2.19, and two Cisco 7921g VoIP phones running WLAN firmware ID 126.96.36.199 with CCX Version 4.