I seem to have found the cause of my issue.

http://bugs.centos.org/view.php?id=2853&nbn=1

>Is the ipsec-users list defunct?  The archives are full of SPAM and
>I've received no responses to my query.
>
>
>I have two computers running CentOS 5.1 with
>        kernel-xen-2.6.18-53.1.21.el5
>        ipsec-tools-0.6.5-8.el5
>I am trying to get these two computers communicating
>over an encrypted point-to-point link (with the eventual
>goal of migrating to a link between a CentOS and a Windows
>Server 2003 box.)
>
>There is a firewall which is set to allow icmp, IP proto
>50, 51 and UDP ports 500 and 4500 through.

Dean Takemori
Tech Support Supervisor
TD Food Group
dtakemori@thdfsg.com