This is a true - and very recent - story. A fairly large UK-based financial institution with several international sites was adding an office in France to its existing MPLS WAN. This meant that the telco providing the UK part of the network had to work with its French counterpart to get the last mile connectivity from the nearest PoP into the office.
Nothing too difficult there. The customer was pleased to see that a termination box appeared on site, and seemed to be connected up. They were reassured that at the PoP similar work was ongoing to connect the circuit to the provider’s network.
Part of the commissioning process involved the telco running a 24 hour BERT (bit error rate test) across from the customer site to the PoP to make sure that the circuit was up to spec before connecting it to the network and carrying out end-to-end testing prior to handing it over to the customer for their own testing. This was done, and the BERT tester removed from the customer premises.
This was where it started to get interesting. The customer was in a hurry for the circuit - as usual the IT staff hadn’t been told until that last minute that this new office was appearing and they were down to the last day before the users moved in. So when their provider told them there was a problem, just hours after telling them that the BERT test had completed with excellent results, they weren’t best pleased.
The provider couldn’t get an end-to-end test to work at all. No connectivity from customer site to PoP. Which didn’t make sense. Until they did a bit more checking.
There was a circuit NTU on the customer site, yes. There was a connection at the telco’s PoP. The problem was with the four kilometre stretch of cable that should have been joining the two up. Which wasn’t there.
Which would have been spotted if it wasn’t for the fact that the UK arm of the telco had requested a local loop from its foreign colleagues. That should have meant the full connection end to end. For some reason (we can blame language difficulties, except that this could have easily happened in the UK too), that’s all they got - a loop. The provider installed an NTU on the customer site, put a loop on it, and went away.
So the BERT results were great - they should have been: the bits were only going about two inches, but didn’t in any way show that the circuit was fit for use.
We’ve seen the same thing happen with an E1 connection where the pairs get mixed up and you get basically a loop at the far end. You may think you’re getting a remote loop put on by your provider just for testing, and the circuit will look fine When they take the software loop off, though, you’ll still have the hardware one, and you won’t be able to ping the far end.
The danger here is that a BERT test success is taken by most providers to prove that there’s nothing wrong with the circuit. You’ll be arguing till you’re blue in the face that it’s not your kit at fault. So next time your provider tells you that your new circuit has passed its BERT test with flying colours, don’t relax too much - it could still be a long haul till you get a working connection!