4IPSec protocol - description

IPSec protocol is a comprehensive set of protocols for encryption, authentication, data integrity, and tunneling. The security is implemented on the OSI reference network layer (Open System Interconnection), and therefore it provides transparent security for any transmission or network application.

Basic IPSec components include:

- security protocols - AH (Authentication Header), ESP (Encapsulating Security Payload),
- protocols for exchange of keys - ISAKMP (Internet Security Association and Key Management Protocol), IKE (Internet Key Exchange),
- Assistance Databases - SPD (Security Policy Database), SAD (Security Association Database), and
- DOI (Domain of Interpretation) – it contains different values, ​​such as identifiers and indicators for SA (Security Association)

IPSec offers two working modes:

1. Transport mode - for Host-to-Host connection

In a transport mode, only the content of a given IP packet is usually encrypted or authenticated. Routing information remains unchanged, unless the IP packet header is modified or encrypted. When an AH (Authentication Header) is used, IP addresses cannot be translated, because the hash value is always lost. Transport and application layers are always secured by hash function, so they cannot be modified (e.g. by changing the port number).

Structure of the IPSec packet in transport mode using the AH header

2. Tunneling mode - designed primarily for site-to-site connections

In the tunneling mode, the entire IP packet is encrypted or authenticated by ESP (Encapsulating Security Payload). It is then encapsulated in a new IP packet with a brand new header using the AH authentication header. This mode is used to create VPNs for communication between individual Site-to-Site networks (e.g., between routers linking different networks), Host-to-Site communications (e.g., remote user access) and Host-to-Host communications e.g. private chat).

Structure of the IPSec packet in tunneling mode using ESP

The tunneling mode supports NAT (Network Address Translation) and PAT (Port Address Translation).

IPSec does not contain in its header any field for the specification of an operating mode. The operating mode is set based on the value of the Next Header field ("IP" value specifies the tunneling mode; "TCP, UDP, ICMP" values (or other) identify the transport mode).

The benefits of IPSec include its transparency, there is no need to modify higher layer protocols, IPsec can secure any IP protocol, it secures “older” protocols that are unsecured and is widely supported by HW (Hardware) a SW (Software).

Disadvantages of IPSec include overhead, necessity of installing the client in case of remote access. It does not deal with user authentication; problematic NAT and PAT (possible to use only in the tunneling mode) and multicast and broadcast traffic.

IPSec protocol:

- provides network layer traffic,
- is universal for securing any TCP/IP traffic,
- protects from Packet Sniffing network layer traffic analysis,
- is suitable for fixed remote users,
- does not support multicast and broadcast transmission,
- exhibits address translation problems (NAT and PAT) - the address field protected by HMAC-SHA1 (Hash Message Authentication Code - Secure Hash Algorithm) is changed; the solution is to pack the IPSec packet into the UDP datagram (User Datagram Protocol) → the NAT-T method (NAT-Traversal), and:
- in case of remote access, client installation is required (but there may be compatibility issues with different implementations).