I was in a meeting with one of my clients this afternoon when their techie knocked on the door. The CRM system had started to crawl, and he couldn't figure out what was going on.
Now, we'd moved it to a new database server the previous evening, but on the face of it there was no real reason to think that was the problem; after all, it'd worked in test form on that new box for a fortnight, and anyway, why should something that's worked like lightning since 9am suddenly start to crawl just after 3pm.
It took a good half-hour to figure out what was wrong; I had a hunch, but it took some proving. For some reason, the database server had decided to try to do a reverse DNS lookup on connecting clients before accepting connections. So it would pause for 45 seconds and then permit the connection to happen - which was fine, so long as you didn't mind waiting 45 seconds each time you clicked something. The solution was to tell the DBMS not to do DNS lookups - everything was restored to its previous liveliness.
Then the question came from the techie: "How do I spot weird stuff like that in future?". The answer: "Errr ... you probably don't". There was, after all, no reason I could think of that something that had been fine for a fortnight should suddenly stop behaving. I had the inkling about the DNS issue because I had something similar in the early 1990s on a Sun server that would pause for half a minute before giving the login prompt to Telnet clients. Once I'd had the inkling, it still took a good ten minutes of Googling with sensible-sounding terms to discover that the DBMS even bothered with DNS lookups, and how to tell it not to. You just can't sensibly protect yourself against stuff that you come across once every 15 years.
I'm fortunate in that I know this client has plans to keep me on in the medium term, so I don't have a point to prove - I was just happy that I happened to have come across this before and so we got them working within half an hour or so. But they're now a bit nervous. And wouldn't you be if the problem you had was solved not by anything you would learn on the average training course but instead by a hunch and a sad bastard who remembered he'd seen it 15 years previously?
Technology is a science. But, entertainingly, not entirely.