I need some help understanding the basics of IPSec. I don't seem to be setting things up correctly. We are trying to set up an IPSec connection from our Windows 2016 Server to an offsite Non-Windows device. Their IPSec configuration is looking for a handshake with Encryption Algorithm AES_CBC 256, Integrity SHA-256, and DH Group 24.
No matter how I set up a connection security rule within Windows Firewall and change the IPSec tab of the Advanced Firewall settings, those settings are not respected when also enabling an IP Security Policy in the Local Group Policy, it always transmits the IKE traffic at 3DES, SHA1, DH 2.
However, whenever we disable the IP Security Policy, thinking the Windows Firewall advanced setting and Connection Rule would apply, the outbound traffic is going un-encrypted as ICMP and not IKE.
Is there a way to use the IP Security Policy in Windows Server 2016 to send out IKE traffic, but add to the list of options to use security algorithms higher than 3DES in the Integrity and Encryption Security method of the Filter Action? Choices are limited. Or it not, how can the Windows Firewall and Connection Security Rule be leveraged to allow outbound ping and other traffic to transmit over IKE?