Instead of doing its thing, `/etc/init.d/haproxy reload` creates new processes and leaves the old ones running. Additionally, at that point, the old process starts using between 130% and 200% CPU on a system with a dual processor. When further running `/etc/init.d/haproxy reload`, the new processes that had been created are indeed restarted, but the first process(es) that were started initially stay, still eating CPU for no apparent reason. Also, the CPU usage increases. Reproducible: Always Steps to Reproduce: 1. /etc/init.d/haproxy start 2. /etc/init.d/haproxy reload Actual Results: High CPU usage (looks like a livelock) and extra undesired processes (more each time if nbthread=1) Expected Results: Just haproxy reloading updated TLS certificates from disk My config contains: user haproxy group haproxy Setting `nbthread 1` in the config transforms the issue into starting new processes at each reload, instead of no more processes after the first reload, and 50% CPU usage by each of them.
I noticed that the issue is not present in 2.8.5, the high CPU usage starts at the first HTTPS request.
Excuse the mistake, I had meant to write: I noticed that the issue is not present in 2.8.5, _and that_ with 2.9.1 (which is the affected version), the high CPU usage starts at the first HTTPS request. After testing, it also appears that not using HTTPS (or not receiving any HTTPS requests on any front-end even though they may exist) completely nullifies the issue. It could therefore be a TLS-related issue.
Might be https://github.com/haproxy/haproxy/issues/2395?
Indeed, all symptoms are consistent with that bug report. Unless even more severe security issues exist, as far as gentoo is concerned, wouldn't it be wise to keyword 2.9.1 as (at least) testing instead of stable, given this impairs production horribly? To justify this, I should add that after 'a while' (facing the Internet) haproxy simply stops responding at all. I suspect that at the core of this bug is the fact that haproxy doesn't close the TLS connections, and eventually all of them are hogged up, causing haproxy to make new connections wait indefinitely and time out.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b24167aa561016cbe3dc6e8e23e45e0a016e17e4 commit b24167aa561016cbe3dc6e8e23e45e0a016e17e4 Author: Christian Ruppert <idl0r@gentoo.org> AuthorDate: 2024-01-15 18:25:26 +0000 Commit: Christian Ruppert <idl0r@gentoo.org> CommitDate: 2024-01-15 18:25:26 +0000 net-proxy/haproxy: Bump to 2.9.2 + Cleanup Bug: https://bugs.gentoo.org/921708 Signed-off-by: Christian Ruppert <idl0r@gentoo.org> net-proxy/haproxy/Manifest | 5 +- net-proxy/haproxy/haproxy-2.4.24.ebuild | 190 --------------------- net-proxy/haproxy/haproxy-2.7.10.ebuild | 190 --------------------- net-proxy/haproxy/haproxy-2.9.1.ebuild | 190 --------------------- ...{haproxy-2.6.15.ebuild => haproxy-2.9.2.ebuild} | 2 +- 5 files changed, 2 insertions(+), 575 deletions(-)
(In reply to Sam James from comment #3) > Might be https://github.com/haproxy/haproxy/issues/2395? Yeah, it most likely is. I just bumped to 2.9.2. Please try again.
(In reply to calimeroteknik from comment #4) > Unless even more severe security issues exist, as far as gentoo is > concerned, wouldn't it be wise to keyword 2.9.1 as (at least) testing > instead of stable, given this impairs production horribly? 2.9.1 is not marked as stable. It never was.
Indeed 2.9.2 has fixed the issue! Sorry for the confusion, I had indeed unmasked the testing version and forgotten about it.
