This week our customer's computers were all cracked.
Well, not all of them. In fact, not even most of them. Though it felt like it.
What did happen is that a high percentage of the problems we faced were website compromises. Two weeks earlier, it was RAID disk failures. A week or so before that, it was SAN mountpoint problems.
I've seen these fault patterns for years. They remind me of how - if you are a man - you go through vast deserts of affection, but when women turn on to you, they all do it at once, so that affection comes in lush, green waves.
Small lush green waves. Very small lush green waves.
I'm not talking about the noise of everyday hardware faults; the low disk space alerts or the processor load queues. Nor am I talking about the software misconfigurations that show up when traffic loads increase or the out-of-memory crashes on Java-cursed web servers.
This stuff is always there. It's bread and butter; the routine daily stuff that bonus-focused support techs jump on when it shows up in the ticket queue because they can close tickets on many more of these each day and make the month's highest closed-ticket count.
Instead, I'm talking about the stuff that gives a ticket queue its personality each day. The anything-but-routine tickets that represent problems that have broken the customer's business process, or whose remedy will involve breaking the customer's business process.
I've wondered if these problems are created by the implementation life-cycle, where the sales department's devotion to the latest Unique Value Proposition PowerPoint has them suddenly start selling - to take one example - slave MySQL servers as instant-on (more or less) backups for clustered MySQL master database servers. Or Plesk host-administration tool support to hosting resllers.
Support techs certainly see these come through in waves as successful sales lead to subsequent installation and customer education requests. Followed shortly after by the inevitable request for an analysis of their backup requirements.
It's the same with network problems. Sometimes the security guys get it into their heads that they like a particular PIX configuration and then that configuration spreads like a virus into every configuration - whether it's necessary or not - and suddenly shows up as the problem it realy is only when the customer asks to do some DNS or load balancer trick that used to be easy but which is impossible with their trendy new PIX config.
Similarly, installing a powerful, public web-available server configuration tool like Plesk is more likely to create compromised server tickets a few weeks later.
But the compromises we dealt with last week were not limited to Plesk installations, nor to the particular types of compromises that Plesk tends to suffer.
Instead, they were across the board... different types of compromises were represented throughout the array of problems we saw, and in their different ways, affected both Windows and Linux servers. They were folks having fun.
With the disk drive failures, we - naturally - checked that we weren't home to a bad batch of drives. Nope! Those drive failures were of all ages and appeared in several of our data centres around thw world.
Reviewing the earlier SAN mount failures we checked if it was one engineer (nope), or engineers following a particular wiki (apparently not) or a connectivity problem (nope). Nope, it just happened.
I've seen it so often that I used to speculate that the Earth occasionally passes through clouds of cosmic rays that affect different parts of electronic systems. Though to think that different cosmic rays are specialised enough to effect on different parts of computer hardware is, well, crazy.
So I'm abandoning rationalism and turning to whacko theories instead.
They do a better job of explaining the problem.
I'm wondering if the Waves of Attraction theory is correct. This theory states that you what you spend your time thinking about, or processing in your head, comes true. There's a whole self-help school that writes how-to books for the woo-woo classes on enabling this for positive effect.
Clearly, though it also has a counterpart, negative rule, which states that, as a support tech fixes a problem, their thoughts create Waves of Attraction that recreate the problem in another server elsewhere.
Of course, you and I know that this is bollocks. But computers - illogical beasts of burden that they are - don't.I just wonder what's round the corner - what's going to be the latest trend. Or am I alone in experiencing this and every other support honcho has completely random problems?