Summary: | mail-filter/rspamd-3.0 segmentation faults with USE=jemalloc | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Thomas Stein <himbeere> |
Component: | Current packages | Assignee: | Petr Vaněk <arkamar> |
Status: | IN_PROGRESS --- | ||
Severity: | normal | CC: | arkamar, bug, c, esigra, proxy-maint, sam, voyageur |
Priority: | Normal | Keywords: | PullRequest |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
URL: | https://github.com/rspamd/rspamd/issues/3871 | ||
See Also: |
https://github.com/gentoo/gentoo/pull/22274 https://github.com/gentoo/gentoo/pull/22860 https://github.com/gentoo/gentoo/pull/24945 https://github.com/gentoo/gentoo/pull/27613 https://github.com/gentoo/gentoo/pull/28150 |
||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 915000 |
Description
Thomas Stein
2021-08-25 14:52:52 UTC
Hi! I guess we need to do this: >2021-08-25 16:11:14 #24215(normal) rspamd_crash_sig_handler: caught fatal signal >11(Segmentation fault), pid: 24215, trace: >2021-08-25 16:11:14 #24215(normal) rspamd_crash_sig_handler: please see Rspamd >FAQ to learn how to dump core files and how to fill a bug > report https://rspamd.com/doc/faq.html#how-to-figure-out-why-rspamd-process-crashed I suspect you'll need this too: https://wiki.gentoo.org/wiki/Debugging. The same bug, segfault on every scan from cli and from exim. Core was generated by `rspamd: controller process (*:11334) '. Program terminated with signal SIGSEGV, Segmentation fault. #0 atomic_load_p (mo=atomic_memory_order_relaxed, a=0x6df68) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/atomic_gcc_atomic.h:31 31 /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/atomic_gcc_atomic.h: No such file or directory. (gdb) bt #0 atomic_load_p (mo=atomic_memory_order_relaxed, a=0x6df68) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/atomic_gcc_atomic.h:31 #1 rtree_leaf_elm_bits_read (dependent=true, elm=0x6df68, rtree=<optimized out>, tsdn=0x7fed4a0246b8) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/rtree.h:175 #2 rtree_leaf_elm_extent_read (dependent=true, elm=0x6df68, rtree=<optimized out>, tsdn=0x7fed4a0246b8) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/rtree.h:214 #3 rtree_extent_read (dependent=true, key=<optimized out>, rtree_ctx=<optimized out>, rtree=<optimized out>, tsdn=0x7fed4a0246b8) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/rtree.h:423 #4 iealloc (ptr=<optimized out>, tsdn=0x7fed4a0246b8) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/jemalloc_internal_inlines_b.h:83 #5 je_tcache_bin_flush_small (tsd=tsd@entry=0x7fed4a0246b8, tcache=tcache@entry=0x7fed4a0248a8, tbin=0x7fed4a024900, binind=<optimized out>, rem=<optimized out>) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/src/tcache.c:152 #6 0x00007fed4d540c19 in tcache_dalloc_small (slow_path=false, binind=<optimized out>, ptr=0x558ecdbedf20, tcache=<optimized out>, tsd=<optimized out>) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/tcache_inlines.h:179 #7 arena_sdalloc (slow_path=false, alloc_ctx=0x0, tcache=<optimized out>, size=<optimized out>, ptr=0x558ecdbedf20, tsdn=<optimized out>) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/arena_inlines_b.h:420 #8 isdalloct (slow_path=false, alloc_ctx=0x0, tcache=<optimized out>, size=<optimized out>, ptr=0x558ecdbedf20, tsdn=<optimized out>) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/include/jemalloc/internal/jemalloc_internal_inlines_c.h:131 #9 isfree (slow_path=false, tcache=<optimized out>, usize=<optimized out>, ptr=0x558ecdbedf20, tsd=<optimized out>) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/src/jemalloc.c:2644 #10 je_sdallocx_default (ptr=0x558ecdbedf20, size=48, flags=flags@entry=0) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/src/jemalloc.c:3583 #11 0x00007fed4d5416d5 in je_je_sdallocx_noflags (ptr=<optimized out>, size=<optimized out>) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/src/jemalloc.c:3611 #12 0x00007fed4d56fcae in operator delete (ptr=<optimized out>, size=<optimized out>) at /var/tmp/portage/dev-libs/jemalloc-5.2.1/work/jemalloc-5.2.1/src/jemalloc_cpp.cpp:131 #13 0x00007fed4e5aface in rspamd::css::css_tokeniser::next_token() () from /usr/lib/rspamd/librspamd-server.so #14 0x00007fed4e5b75be in rspamd::css::css_parser::component_value_consumer(std::unique_ptr<rspamd::css::css_consumed_block, std::default_delete<rspamd::css::css_consumed_block> >&) () from /usr/lib/rspamd/librspamd-server.so #15 0x00007fed4e5bad58 in rspamd::css::css_parser::consume_css_rule(std::basic_string_view<char, std::char_traits<char> > const&) () from /usr/lib/rspamd/librspamd-server.so #16 0x00007fed4e5bae77 in rspamd::css::get_rules_parser_functor(memory_pool_s*, std::basic_string_view<char, std::char_traits<char> > const&) () from /usr/lib/rspamd/librspamd-server.so #17 0x00007fed4e5bb17a in rspamd::css::parse_css_declaration(memory_pool_s*, std::basic_string_view<char, std::char_traits<char> > const&) () from /usr/lib/rspamd/librspamd-server.so #18 0x00007fed4e57a47d in rspamd::html::html_process_input(memory_pool_s*, _GByteArray*, _GList**, kh_rspamd_url_hash_s*, _GPtrArray*, bool)::{lambda()#2}::operator()() const () from /usr/lib/rspamd/librspamd-server.so #19 0x00007fed4e57b409 in rspamd::html::html_process_input(memory_pool_s*, _GByteArray*, _GList**, kh_rspamd_url_hash_s*, _GPtrArray*, bool) () from /usr/lib/rspamd/librspamd-server.so #20 0x00007fed4e5d7485 in rspamd_message_process () from /usr/lib/rspamd/librspamd-server.so #21 0x00007fed4e54e54e in rspamd_task_process () from /usr/lib/rspamd/librspamd-server.so #22 0x00007fed4e54e433 in rspamd_task_process () from /usr/lib/rspamd/librspamd-server.so #23 0x00007fed4e54e433 in rspamd_task_process () from /usr/lib/rspamd/librspamd-server.so #24 0x00007fed4e54e433 in rspamd_task_process () from /usr/lib/rspamd/librspamd-server.so #25 0x0000558ecc2a0dbd in rspamd_controller_handle_scan () #26 0x00007fed4e564b78 in rspamd_http_router_finish_handler () from /usr/lib/rspamd/librspamd-server.so #27 0x00007fed4e562d29 in rspamd_http_on_message_complete () from /usr/lib/rspamd/librspamd-server.so #28 0x00007fed4e657fcf in http_parser_execute () from /usr/lib/rspamd/librspamd-server.so #29 0x00007fed4e563974 in rspamd_http_event_handler () from /usr/lib/rspamd/librspamd-server.so #30 0x00007fed4df3ecb5 in ev_invoke_pending () from /usr/lib/rspamd/librspamd-ev.so #31 0x00007fed4df42a16 in ev_run () from /usr/lib/rspamd/librspamd-ev.so #32 0x0000558ecc29e6f0 in start_controller_worker () #33 0x00007fed4e5589b1 in rspamd_fork_worker () from /usr/lib/rspamd/librspamd-server.so #34 0x0000558ecc2a8caa in spawn_worker_type () #35 0x0000558ecc2a8d8b in spawn_workers () #36 0x0000558ecc29bd02 in main () (gdb) rspamd compiled with USE="-jemalloc" works. Ok. I'm trying with -jemalloc.... I opened https://github.com/rspamd/rspamd/issues/3871 because I figured rspamd itself has to be fixed too. mail ~ # emerge -pv rspamd These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ~] mail-filter/rspamd-3.0-r2::gentoo USE="jemalloc pcre2 -blas -jit -test" CPU_FLAGS_X86="ssse3" LUA_SINGLE_TARGET="lua5-1 -lua5-3 -luajit" 0 KiB Total: 1 package (1 reinstall), Size of downloads: 0 KiB Compiled with LDFLAGS="-Wl,-O1" ie. NOT -Wl,--as-needed seems to work. https://github.com/rspamd/rspamd/issues/3871#issuecomment-912308477 (no crashes here for 2 hours/20 mails). Not sure if it's a fix - or simply a temporary workaround around a bigger underlying issue. For now, what do we think about masking USE=jemalloc for 3.0? How about this https://wiki.gentoo.org/wiki/Project:Quality_Assurance/As-needed#Properly_filtering_--as-needed ? This works for me: pkg_setup() { use jemalloc && append-ldflags $(no-as-needed) } This is better for users, in my opinion, because they will not be forced to change their configuration if they have jemalloc support enabled. (In reply to Petr Vaněk from comment #7) > How about this > https://wiki.gentoo.org/wiki/Project:Quality_Assurance/As- > needed#Properly_filtering_--as-needed ? This works for me: > > pkg_setup() { > use jemalloc && append-ldflags $(no-as-needed) > } > > This is better for users, in my opinion, because they will not be forced to > change their configuration if they have jemalloc support enabled. I don't love it but that's not your fault -- it's just because the bug is a bit scary. Slightly concerned that the bug can be triggered by use of LTO(?) without as-needed though (and are there any other combinations)? If the configuration file at runtime mentions jemalloc, yeah, we can't really just mask it :| (In reply to Sam James from comment #8) > (In reply to Petr Vaněk from comment #7) > > How about this > > https://wiki.gentoo.org/wiki/Project:Quality_Assurance/As- > > needed#Properly_filtering_--as-needed ? This works for me: > > > > pkg_setup() { > > use jemalloc && append-ldflags $(no-as-needed) > > } > > > > This is better for users, in my opinion, because they will not be forced to > > change their configuration if they have jemalloc support enabled. > > I don't love it but that's not your fault -- it's just because the bug is a > bit scary. Slightly concerned that the bug can be triggered by use of LTO(?) > without as-needed though (and are there any other combinations)? > We could do what you said (in src_configure though; no need to do this for binpkgs) + add a warning on general use of jemalloc for now in pkg_postinst? "If you experience crashes, ..." I have changed my mind, I believe that suggested masking of USE=jemalloc for 3.0 is better solution. Just to clarify, users cannot configure jemalloc at runtime it is build time configuration. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=7d7a0aef8dc2b882d5d46b6cb06af6653c4433cc commit 7d7a0aef8dc2b882d5d46b6cb06af6653c4433cc Author: Petr Vaněk <arkamar@atlas.cz> AuthorDate: 2021-09-12 08:42:13 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2021-09-12 22:44:16 +0000 profiles/base: mask USE=jemalloc for ~mail-filter/rspamd-3.0 rspamd-3.0 segfaults a lot with jemalloc enabled. The issue is caused by -Wl,--as-needed linker flag. Bug: https://bugs.gentoo.org/810337 See-also: https://github.com/rspamd/rspamd/issues/3871 Signed-off-by: Petr Vaněk <arkamar@atlas.cz> Closes: https://github.com/gentoo/gentoo/pull/22274 Signed-off-by: Sam James <sam@gentoo.org> profiles/base/package.use.mask | 6 ++++++ 1 file changed, 6 insertions(+) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e6fe959ddc3183e70507510c2ee801902bc4f4d9 commit e6fe959ddc3183e70507510c2ee801902bc4f4d9 Author: Petr Vaněk <arkamar@atlas.cz> AuthorDate: 2021-11-07 21:38:06 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2021-11-07 23:28:39 +0000 profiles/base: mask USE=jemalloc for ~mail-filter/rspamd-3.1 The jemalloc issue affects rspamd-3.1 as well. Bug: https://bugs.gentoo.org/810337 See-also: https://github.com/rspamd/rspamd/issues/3871 Signed-off-by: Petr Vaněk <arkamar@atlas.cz> Closes: https://github.com/gentoo/gentoo/pull/22860 Signed-off-by: Sam James <sam@gentoo.org> profiles/base/package.use.mask | 1 + 1 file changed, 1 insertion(+) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=593968a8ab8ebd056c994d8e580845a4e2ab96f2 commit 593968a8ab8ebd056c994d8e580845a4e2ab96f2 Author: Petr Vaněk <arkamar@atlas.cz> AuthorDate: 2022-04-07 15:39:59 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-04-12 08:20:13 +0000 profiles/base: change USE=jemalloc to >=mail-filter/rspamd-3.0 The jemalloc issue affects rspamd-3.2 as well. This mask seems to be better than adding extra line for every new version. Bug: https://bugs.gentoo.org/810337 Upstream-issue: https://github.com/rspamd/rspamd/issues/3871 Signed-off-by: Petr Vaněk <arkamar@atlas.cz> Signed-off-by: Sam James <sam@gentoo.org> profiles/base/package.use.mask | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=4a424df747d23d5151e70495fce5e97521479e90 commit 4a424df747d23d5151e70495fce5e97521479e90 Author: Petr Vaněk <arkamar@atlas.cz> AuthorDate: 2022-10-04 08:55:10 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-10-07 11:28:35 +0000 mail-filter/rspamd: add 3.3 This version contains a fix for Clang-16 (error: implicit declaration of function ZSTD_resetDStream) reported in bug 870448. It still suffers with a bug 810337. Bug: https://bugs.gentoo.org/810337 Closes: https://bugs.gentoo.org/870448 Signed-off-by: Petr Vaněk <arkamar@atlas.cz> Signed-off-by: Sam James <sam@gentoo.org> mail-filter/rspamd/Manifest | 1 + mail-filter/rspamd/rspamd-3.3.ebuild | 135 +++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5272058e6d1b40f22550ce775c6cc75b7dfe9c2a commit 5272058e6d1b40f22550ce775c6cc75b7dfe9c2a Author: Petr Vaněk <arkamar@atlas.cz> AuthorDate: 2022-11-02 12:25:03 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-06 06:16:51 +0000 mail-filter/rspamd: add 3.4 This version still suffers with jemalloc issue, see bug 810337. Bug: https://bugs.gentoo.org/810337 Closes: https://bugs.gentoo.org/879575 Signed-off-by: Petr Vaněk <arkamar@atlas.cz> Closes: https://github.com/gentoo/gentoo/pull/28150 Signed-off-by: Sam James <sam@gentoo.org> mail-filter/rspamd/Manifest | 1 + mail-filter/rspamd/rspamd-3.4.ebuild | 136 +++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) |