So far, the 2.6 kernel still contains a bug which effectively prevents the use of IPSec with a secure netfilter configuration. For example, an incoming ipsec-encapsulated packet is not fed back through the input system after it is decapsulated. Thus, it never gets added to the conntrack table, so response packets will be dropped if the output chain has a drop or deny policy (see http://forums.gentoo.org/viewtopic-p-2436429.html for my original problem). patch-o-matic-ng contains patches for this issue, but they are outdated (I'm really surprised that "ipsec doesn't work" doesn't get more attention). Fortunately, the shorewall developers have updated patches available at http://shorewall.net/pub/shorewall/contrib/IPSEC/2.6.11/ . These apply cleanly to hardened-sources-2.6.11-r13, which I'm using, and they're working great for me. A few of the hunks fail on gentoo-sources-2.6.11-r9, but I expect anyone who knows C (read "not me") will probably find this easy to rectify. I'm asking that these patches please be included in the gentoo kernel patchset so that ipsec can actually be used in a real-world environment. Thanks!
Ideally this should be fixed upstream. Please file a bug for this at http://bugzilla.kernel.org and post the resultant bug URL here. You should also make sure the netfilter developers know of the problems by mailing netfilter-devel@lists.netfilter.org