A high-risk vulnerability has been discovered in systems running the IPsec VPN protocol, the UK’s National Infrastructure Security Co-ordination Centre (NISCC) has reported.

The hole affects any system using one of IPsec’s small family of security sub-protocols, Encapsulating Security Payload (ESP), while in tunnelling mode with "confidentiality" set, or with integrity being provided from a higher layer protocol.

It is now thought possible for any attacker able to intercept the flow of packets between two IPsec devices to make changes to this encapsulation packet, and in a roundabout way gain access to the plain-text form of the inner packet, breaking the security.

ESP is the part of IPsec used to actually encrypt data beyond the packet transport header, while the tunnelling mode is a way for IP header to be wrapped inside an ESP packet. Some configurations using the less common-used AH security mode are also said to be vulnerable.

The NISCC advisory states that the attack has been "demonstrated to work under realistic conditions", with a modest amount of effort, and should therefore be treated as more serious than a theoretical compromise.

"The attacks are probabilistic in nature and may need to be iterated many times in a first phase in order to be successful. Once this first phase is complete, the results can be reused to efficiently recover the contents of further inner packets," the NISCC advisory states.

The recommended fix is to configure ESP to use both confidentiality and integrity protection, or to use certain implementations of the AH protocol alongside ESP an extra layer of security. Alternatively, the generation of ICMP error packets, on which part of the compromise depends, can be limited or filtered out by a local firewall.