The call forwarded to me seemed simple enough. A vendor's technician was at a client's site to install a card access system, and he was having issues connecting to the application running on a Windows Server 2003 over the corporate LAN. He requested that the network be checked to ensure there were no network issues. I proceeded to the site with my toolkit and a laptop with a sniffer program installed.
Physical connectivity didn't seem to be a problem, as both the server network interface and the port on the switch to which it was physically connected displayed a link light. To be sure, I used a cable analyser to ensure the integrity of the Cat 5E line. When troubleshooting possible network issues, it's often best to start from the physical layer and work up the OSI model.
The switch registered the server's media access control address on the correct port, and pings from switch to server were successful. The server was running two network services: Terminal Services and a proprietary card access application. I could Telnet to TCP Port 3389 from my laptop plugged into the switch and configured with the correct IP network information, but not to the access system's TCP port.
At this point, I was certain the issue was with the server and not network-related, but the card access system technician held a differing view. After a brief dialogue exchange, I realised he was quite serious when he said that my straight-through Cat 5E patch cable was incorrect because Ethernet relied on "twisted pair" and therefore needed a "turnaround" cable.
Keeping a straight face
Obviously, I had to keep my initial reaction to myself, even though this was one of the more humorous -- and incorrect -- networking statements I'd heard in quite a while. I realised I was dealing with a technician who was not very knowledgeable in Ethernet networking basics. Up until recently, the system that he had installed for years had relied on dedicated serial lines to the card reader controllers, and Ethernet was a relatively new concept for him. I explained that I was completely sure network connectivity was fine and that we should check the server.
Unfortunately, this sort of exchange happens more often nowadays. As more non-traditional equipment migrates to IP network connectivity, technicians used to dealing with older connectivity methods are forced to learn an entirely new transport mechanism.
As network administrators, it's our responsibility to recognise that the experts installing such systems may not share our networking skills and therefore may require a more basic approach to solving a connectivity problem than, say, a Unix system administrator. The following are a few tips I've learned to help smooth the process when the installation doesn't go as planned.
Respect the technician's abilities
In the above example, my initial reaction was more incredulous than anything else. I found it hard to believe that the card access company would send a technician so lacking in basic network knowledge to install a system that relied on TCP/IP networking to work.
It would have been easy to discount the technician by immediately requesting to speak with a high-level engineer at the company's support site. However, while that may have resulted in the solution, undermining the on-site technician would have been hurtful to him both personally and professionally. Here was someone who had worked on similar systems for many years and was most probably doing his first on-site installation of the new set-up.
Instead of escalating, I used this opportunity to help educate. I took a trace of both the successful connection with the server Terminal Services port and the failed connection to the proprietary application. I showed him the TCP three-way handshake and explained what it was. I detailed my troubleshooting methodology without getting too deep into "techno-babble," discussing what the link lights and successful pings meant.
Following this, the technician checked the server and discovered the card access service wasn't running. He contacted his engineering support group, and in quick order, they determined the problem and implemented a solution. Afterward, he asked if I could send him the trace so that he may refer to it in the future as a learning aid, and he thanked me for the explanation.
This is not to say that it's never appropriate to escalate the problem to the vendor's engineering level. The on-site technician often is not an expert system administrator for the system that he is installing. If you aren't getting the technical answers you need, escalation may be the only alternative.
While we all take pride in our extensive network knowledge, it's important to realise that others who may not share in the knowledge often have years of experience in other areas. I firmly believe that as network professionals we have an obligation to be ambassadors of the profession by being positive and helpful at all times.
Understand the application and its network requirements
Today's LANs are not the Wild West they were many years ago, where all that was needed for network connectivity was an IP address and an activated network port. Network access and admission controls, firewalls and quality of service are a few of the new issues that must be dealt with.
In the previous example, the application required access across the network infrastructure on a specific IP port. If the switch that provided connectivity between the main server and the card access controllers had access control lists or other similar firewall-type rules that prevented such communication, these rules would need modification to allow access.
Note if Layer 7-aware switches (such as traffic shapers) are installed between clients and servers, determining that there is no impediment to network access requires careful examination of policies on such devices. Not long ago I encountered a situation where just such a policy pushed my network troubleshooting skills to the limit before I discovered a flaw in the policy.
Understanding the application and its network requirements can go beyond asking the on-site technician. Often they are trained to install their products in a clean environment and may not know what ports the application needs. Looking at ports open on the server (netstat -an) and performing a trace of attempted connections will usually reveal enough to configure network devices accordingly.
Keep security at the forefront
A vendor's application should not dictate your LAN's security policy. If it's acceptable from a security policy standpoint to open a port in your LAN infrastructure, that's fine. However, if a vendor requests a "non-standard" port to be opened and it violates your company's security policy, it's reasonable to request an alternative solution.
A few years back I recommended that a client block TCP Port 81 inbound and outbound, as a variant of the Bagle worm was using that port to replicate. The problem was that Port 81 was sometimes used as an alternative Web server port. Not long after, an outside vendor requested that Port 81 be opened globally to provide for connectivity to a Web management application. Instead of reversing the security policy, I suggested the vendor use an alternative port for access. They readily complied and the problem was solved without compromising the local security policy.
I have found that vendors recognise that every LAN that they utilise has a distinct personality, so to speak, and have provisioned flexibility for such. It can be frustrating for a vendor at times because no real-world installation over a multi-use network mirrors a vanilla installation in a test lab.
Remember, the vendor's ultimate goal is to sell its product. In order to do so, the product needs to communicate on your network. While that usually requires some negotiation with the vendor's engineering staff, since it's in both their and your company's interests to ensure workability, this negotiation process is usually relatively painless.
Get involved early in the project
The last thing that you, as a network administrator, need to do is make network decisions under the gun. This will happen if the above issues show themselves close to project completion. The key to avoiding this is to get involved early.
In one case, a client of mine consulted with a vendor to provide credit card readers in several concession areas. The vendor and client performed a walk-through prior to installation and saw that there were network jacks installed at each location. The vendor provided a quote for the readers and had a technician arrive on-site to install.
The problem though was that while the cabling was installed, there was no network infrastructure to support the connections. Not only did that mean no Ethernet switches to make the network connections "hot," but no fibre optics to connect the communication rooms where the cables terminated to the company's backbone. Adding fibre-optics and network electronics to the installation drastically increased the project costs.
Since there were already significant sunk costs incurred, adding the infrastructure was approved. Involving the network team early in the project would have eliminated the surprise costs. This can be accomplished by implementing a corporate policy that any systems that require network access must have input from the corporate network team prior to purchase.
Remember, while you know your network like the back of your hand, vendors do not.
The variety of applications using your network may continue to grow. Being able to successfully work with vendors whose primary roots are not in data networking can be difficult, but following these few guidelines can ease the implementation process.