In a previous article, I discussed the installation of three open-source packages on a computer running Linux to capture useful network statistics.
Using Linux is economical simply because it can run on obsolete machines that can still work as well as network sensors. The more sensors that can be deployed, the more correlating data that may be collected and analyzed to solve network problems.
Recently, I purchased an HP Pavilion 6643 with 192MB of memory, a 500-MHz Intel Celeron processor and a 20GB hard drive at a garage sale for £6 to use as a network sniffer. I downloaded Fedora Core 5 and burned six CDs from the corresponding ISO images using my home computer and Internet service provider. I downloaded libpcap, tcpdump and iptraf and burned those compressed files onto another CD. I installed Fedora Core 5 using the Web server and development options and installed the three packages.
Factor in the installation of a 10/100 twisted-pair Ethernet card, and I had a functional 100Mbit/s Ethernet sniffer at a cost of less than £20 and a few hours for the software installation, which was mainly "select and wait" time.
Where to sniff?
Determining where and how to place a sniffer in the network is as important as the actual packet analysis. For temporary problems where on-site analysis is necessary, you should load the sniffing packages onto a laptop and go to the site to intercept the data stream close to one of the affected nodes. For more routine monitoring and baseline determination, you should consider permanent placement at critical junctures, such as the network egress to the Internet, the link to a server farm or a similar type of area.
In all cases, the sniffer must be plugged in to either a switch port that is mirroring the port to be analyzed or (in older networks) a hub. In a pinch, a hub can be placed inline between a switch and the network of interest, but by altering the environment before testing it, the data may be skewed.
Libpcap is not run directly but rather as a necessary component (dependency) of tcpdump and iptraf. Tcpdump essentially processes the information as collected by libpcap and presents it packet by packet.
Tcpdump is located in the directory /usr/local/sbin/. In the bash shell environment, the directory path is set in the PATH variable by the .bash_profile file in the user directory. Therefore, tcpdump may be called by simply typing tcpdump at the user prompt or the full file directory path may be used. Without using any other options, a sample output of a sniff on an SSH (Secure Shell) terminal emulation session from a client to the network analyzer is shown below.
21:10:33.307526 IP 192.168.2.3.1368 > 192.168.2.21.ssh: P 45:89(44) ack 232 win 5512
21:10:33.307547 IP 192.168.2.21.ssh > 192.168.2.3.1368: . ack 89 win 7504
The first two packets show communication from the client to the analyzer. In order, the default display is the time the packet was captured, the type of protocol, the source address and port, the destination address and port, and a short summary of the packet contents. Even in this basic form, many network problems can be diagnosed. Often, there is one component of the client/server communication pair that is, for machine-specific reasons, not responding either in a timely fashion or at all. What the end user may perceive as a "network problem" may in fact be an end node issue after all, and the proof is in the trace.
Look at the packet times in the above tcpdump output example. Subtract the first from the second to calculate the response latency of 0.000021 seconds. This latency specifically measures the throughput between nodes through a Microsoft MN-700 wireless 802.11 b/g router/switch. In this case, the latency that the MN-700 device adds between switched ports is not significant. An extended trace and subsequent intrapacket delta time analysis can be obtained to verify the initial latency calculation, and is suggested.
Tcpdump may be run with many qualifiers to modify the collection and presentation of data. First, the switch -n will prevent the sniffer from doing reverse lookups on the IP addresses. In the example above, the network is closed with no DNS services, yet the sniffer continues to send DNS queries as shown below.
21:10:33.308257 IP 192.168.2.21.32769 > 192.168.2.1.domain: 23316+ PTR? 220.127.116.11.in-addr.arpa. (42)
This querying with no response drastically slows down the data collection and display. By using the -n switch that lag may be eliminated.
If there are several nodes on the network and only traffic to and from the client is desired, then the host switch may be used to only display those packets. If further analysis is desired, the output may be redirected to a file. To determine the performance between two nodes on a network that has thousands of paired communications, it's often best to take a snapshot trace and perform post-trace filtering. The command tcpdump -n host 192.168.2.3 > trace.192.168.2.3.06050101 saves all traffic involving address 192.168.2.3. Keep in mind that using a descriptive file-naming convention, such as shown (file type, address, date and instance), aids in post-sniff analysis by providing an easy means to catalog trace files.
Further filtering the trace to the protocol level is possible with a simple command. To analyze all SSH communication between the client and server above, use the command more trace.192.168.2.3.06050101|grep 192.168.2.21.22. This will present only the packets involved with SSH communication during the time that the trace was taken.
To delve deeper into the packet, use the commands -vvv and -X. This will, among other results, produce the IP type of service and a hex and ASCII representation of the data contained within the packet. One useful application of deeper packet analysis is the ASCII data, as presented, will show whether or not the data is encrypted. A proof-of-concept exercise would be to capture two traces, one for a telnet (clear text) test terminal session and one for an SSH (encrypted text) test terminal session, and type the exact same information during both tests. The trace of the telnet session will reveal everything that was typed (including the username and password) whereas the data section of the SSH trace will resemble random gibberish.
Finally, it's often desirable to create a sniffer with at least two different interfaces, one for out-of-band analysis and one for the gathering of the data. The switch -i can tell tcpdump to pull data from the desired network interface card.
Catching the trend
Sometimes trending and historical analysis is more important than the actual packet data. For this, iptraf provides valuable information via a simple menu interface. The program is located in /usr/local/bin/. Starting iptraf gives the option menu.
By selecting the default options, the Linux sniffer starts capturing packets. Unlike tcpdump, iptraf will track every TCP connection in the top window and will scroll all UDP packets in the bottom window. The detailed interface statistics option on the main menu gives a summary of activity on a particular Ethernet interface.
These examples only scratch the surface of the capabilities of tcpdump and iptraf. The best way to learn them all is to use the packages in a real-time environment. Chances are you'll discover that with these tools you can become a network "MacGyver."
It's important to run tcpdump and iptraf periodically during periods of proper network operation to get a good idea of how a network operating efficiently looks from the packet level. Establishing baselines is vital for troubleshooting a network. In the next article, I will introduce another package for baseline establishment and bandwidth analysis called Multi Router Traffic Grapher.
Greg Schaffer is the Director of Network Services at Middle Tennessee State University. He has over 15 years of experience in networking, primarily in higher education.