One of the biggest challenges I face in running my European operation is managing the networks at the remote sites, often without having network expertise on site. Keeping a long distance eye on all those flashing lights is not a particularly easy task, especially if you are limited by dwindling budgets. Certainly I know I would love to implement a full blown CiscoWorks or HP openview set up within my org, but I just cannot afford it. Fortunately, there are cheap and wonderful little tools that can help you out.
To monitor my bandwidth usage I use a lovely little app called MRTG (Multi Router Traffic Grapher). It’s free and for the basic stuff is very easy to use. You can have it up and running within 30 minutes. MRTG provides graphs of usage through any ports on the device. Its designed to provide bandwidth information but with some basic tweaking you can have it provide graphs on anything. I use it to keep an eye on CPU and memory utilization as well. For the more intensive monitoring, both on Network kit and my server farms, I use Sitescope, not the cheapest tool around, but not expensive either and again very powerful and extremely easy to use. Sitescope also has the advantage of being able to send reports and alerts to a mobile phone, therefore providing an instantaneous warning system when something is failing out of hours. Both of these combined with a shareware syslog server and you have a basic network management suite.
But these apps just front end the information available within the network kit itself. The basic information is there for any app, configured correctly, to be able to retrieve it. The secret is to choose the best SNMP utility you can afford; this is the protocol that is the real jewel in my battle to keep the network alive from a remote destination.
But SNMP is not perfect, certainly not v2, which suffers from its inherent lack of security. So, like most people, I continue to use insecure SNMP v2 for managing my kit and have to look at other ways for securing it. One obvious way is to ensure that such insecure information is not allowed out of your internal network. I shut all outside interfaces to any SNMP requests. This does complicate your management network a little, I have a server in each location that handles remote SNMP monitoring, but that extra cost is worth the piece of mind it gives me.
As well as just looking after and monitoring equipment in remote sites, I have the challenge of doing remote maintenance on remote kit. Now I know that the basic ability to do the maintenance is easy; it’s ensuring that if you make a mistake that the network will come back online. Our network has been designed so as to eliminate single points of failure. All remote sites have multiple circuits and multiple routers, but I still cannot just do even basic maintenance on remote kit without some form of back out mechanism. And if there is no-one on site to help, locking up a router (did I mean to type debug all?) can be a real problem. Fortunately, the Cisco equipment that I use offers a useful command: “Reload in hr:min”, allows you to tell the device to restart itself in a set time. Therefore if you have a 30 minute config window, and you know you will need 15 minutes to make the changes, setting the device to reboot automatically in 30 minutes means you can safely make your changes, knowing if it does go pear-shaped, the device will bring itself back. And if it goes well, then you simply remove the line and then save the config.
Documentation is vital to successful remote management, even more so than for localised equipment. What I find immensely useful is having a photograph of the various remote racks in the documentation. Obviously everything will be properly labelled, but having a photograph of the equipment in front of you when trying to talk someone through checking a loose cable, or even just visualising the setup when troubleshooting, is invaluable. If you can, having a webcam inside the remote comms Cabinet is also highly useful, not only to aide trouble shooting, but also as an added security device as well. Remote control and motion sensitive cameras are now relatively cheap and are worth every penny. These too come with SNMP management these days, which means that you configure your management tools to receive alerts every time the camera starts recording and you have a cheap and highly powerful security monitoring system.
Yet all these little tricks and tools are for just keeping tabs on the network. If you are going to have a complex, diverse network, nothing beats planning for it. If you know that a remote office needs full network connectivity, but does not have the resources for a dedicated network body then you need to plan for increased resiliency. You must have a back door into each bit of kit, but that back door must only be accessible by you. Do not use a modem-based remote access card, unless you fully comprehend the risks and have assessed each of them in turn. Personally, I do not allow modems on the network at all – although other IT managers would have their own opinions on that. Do have multiple circuits, even if one is only ISDN or ADSL backup. Have resilient router configurations and ensure that there is at least one server, or PC, on the remote network with access to a telnet session on each remote device.