nginx crash with segfault if compiled with rtmp useflag. rtmp-nginx-1.7.6.patch seems to be broken. Reproducible: Always Steps to Reproduce: 1. USE=rtmp emerge nginix 2. sudo nginx 3. dmesg Actual Results: [10772.694492] show_signal_msg: 25 callbacks suppressed [10772.694499] nginx[11101]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10772.835666] nginx[11102]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10772.992373] nginx[11103]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10773.124978] nginx[11104]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10773.274593] nginx[11105]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10773.306512] nginx[11106]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10773.499940] nginx[11107]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10773.646845] nginx[11108]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10773.805250] nginx[11109]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000] [10773.953843] nginx[11110]: segfault at 0 ip 0000000000420574 sp 00007fffb0b86730 error 4 in nginx[400000+d1000]
Created attachment 386928 [details] emerge --info
Thanks for the bug report. I'll try to reproduce tomorrow.
Core dump backtrace: t#0 ngx_event_process_posted (cycle=<optimized out>, posted=0xe027f0 <ngx_rtmp_init_queue>) at src/event/ngx_event_posted.c:31 31 ngx_delete_posted_event(ev); (gdb) bt #0 ngx_event_process_posted (cycle=<optimized out>, posted=0xe027f0 <ngx_rtmp_init_queue>) at src/event/ngx_event_posted.c:31 #1 0x000000000048e1c6 in ngx_rtmp_stat_init_process (cycle=<optimized out>) at /usr/src/debug/www-servers/nginx-1.7.6/nginx-rtmp-module-1.1.5/ngx_rtmp_stat_module.c:118 #2 0x0000000000425576 in ngx_worker_process_init (cycle=cycle@entry=0x1ee3400, worker=worker@entry=0) at src/os/unix/ngx_process_cycle.c:981 #3 0x0000000000425ab7 in ngx_worker_process_cycle (cycle=cycle@entry=0x1ee3400, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:746 #4 0x0000000000424694 in ngx_spawn_process (cycle=cycle@entry=0x1ee3400, proc=0x425aa0 <ngx_worker_process_cycle>, data=0x0, name=0x4a6276 "worker process", respawn=respawn@entry=0) at src/os/unix/ngx_process.c:198 #5 0x0000000000426be1 in ngx_reap_children (cycle=0x1ee3400) at src/os/unix/ngx_process_cycle.c:631 #6 ngx_master_process_cycle (cycle=cycle@entry=0x1ee3400) at src/os/unix/ngx_process_cycle.c:184 #7 0x0000000000408434 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:407
I confirm this. Portage 2.2.8-r2 (hardened/linux/amd64, gcc-4.7.3, glibc-2.19-r1, 3.14.17-hardened-r1 x86_64) ================================================================= System uname: Linux-3.14.17-hardened-r1-x86_64-Intel-R-_Core-TM-_i7-4770_CPU_@_3.40GHz-with-gentoo-2.2 KiB Mem: 4098288 total, 3768292 free KiB Swap: 4194300 total, 4194300 free Timestamp of tree: Mon, 20 Oct 2014 22:15:01 +0000 ld GNU ld (Gentoo 2.23.2 p1.0) 2.23.2 app-shells/bash: 4.2_p53 dev-lang/perl: 5.18.2-r2 dev-lang/python: 2.7.7, 3.3.5-r1, 3.4.1 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.13.4 sys-devel/binutils: 2.23.2 sys-devel/gcc: 4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2-r1 sys-devel/make: 4.0-r1 sys-kernel/linux-headers: 3.13 (virtual/os-headers) sys-libs/glibc: 2.19-r1 Repositories: gentoo x-mylay ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -mtune=native" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -mtune=native" DISTDIR="/var/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo" LANG="C" LDFLAGS="-Wl,--hash-style=gnu -Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j5" PKGDIR="/var/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/portage/mylay" SYNC="rsync://rsync.de.gentoo.org/gentoo-portage" USE="acl amd64 bash-completion berkdb bzip2 caps cli cracklib crypt cxx device-mapper dri gdbm gmp gnutls hardened iconv icu idn ipv6 justify mmx modules multilib ncurses nls nptl openmp pam pax_kernel pcre readline session sse sse2 ssl ssse3 unicode urandom vim-syntax xattr xtpax zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en ru" NGINX_MODULES_HTTP="access auth_basic browser empty_gif fastcgi geo gzip limit_req limit_conn map proxy rewrite spdy stub_status" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Tried to compile nginx with nginx-rtmp-module just cloned from git. Compiles and run successfully.
Yeah, it lacks one commit (I've only merged the build commit). I'm a bit reluctant to go on a merge fest since all other modules basically track releases. I'll give them a few more days and pull another patch otherwise. Thanks for your assistance with debugging!
I can confirm this. Updating rtmp module version to 1.1.6 and removing rtmp-nginx-1.7.6.patch fixes the issue. See https://github.com/arut/nginx-rtmp-module/issues/470 Tiziano, Johan: if this is okay with you, I can commit it with a rev bump.
I don't need this feature (it simply looked nice in useflags). Now i don't intend to activate it again anyway, so i fine with it as long as it don't break other things.
(In reply to Maxim Koltsov from comment #7) > I can confirm this. Updating rtmp module version to 1.1.6 and removing > rtmp-nginx-1.7.6.patch fixes the issue. See > https://github.com/arut/nginx-rtmp-module/issues/470 > Tiziano, Johan: if this is okay with you, I can commit it with a rev bump. I just pushed a branch to github. If you have an account there, could you please say review/OK (or tiziano, feel free to merge)? I prefer having a one-way from overlay to gentoo-x86. Thanks for helping out! https://github.com/gentoo/nginx-overlay/pull/13
..I've also bumped nginx to 1.7.7 with this branch: https://github.com/gentoo/nginx-overlay/pull/14. Feel free to review/merge.
Also believe this to be related to upstream issue here: https://github.com/arut/nginx-rtmp-module/pull/472 Basically this guy doesn't know how to write nginx extensions properly. It shouldn't do ANYTHING "crashable" until after it finds out if anyone enabled it in the config files. Also lots of bugs with his worker code, in general. Motion this should just be dropped, or moved to a "rtmp-server" flag, because "rtmp" turns on client/player support also. Negating a global "rtmp" for nginx specifically is dumb. Either upstream needs to learn how to code, or move this to another USE flag so nobody can turn it on unless they KNOW they are shooting themselves in the foot.
..problem is that above quote can probably apply to more than one module. My main priority (for what its worth) is to reinvigorate a stable nginx with a working third party extension offering. Based on that I'd suggest treading more lightly before calling anything stable again. Based on my previous replies (if you like essays), I will probably hold onto 1.7.x once 1.8 is out and lock down extensions. I could consider voting rtmp off if its so much worse than any other extension, but I feel like most of the users wouldn't know about this (and therefore punt a vote). Suggestions to improve the current situation? Not saying it's bad, but I'd like to be better than now.
I noticed lighttpd has its own USE flags unshared by other packages. It also doesn't use the added "NGINX_*" type vars to trigger packages, it's all USE flags. I think the separate variable is misleading, if the rtmp flag brings in the OPTION to have the rtmp plugin but then if you don't also put it in your NGINX_MODULES then it won't actually build. So either adopt some USE flags that don't clash with anything and eliminate the silly separate NGINX_MODULES stuff (which I think was adopted from Apache ebuilds?) OR let the USE flag be how it is and then require (and not by default in any stretch) the rtmp support to be explicitly included in the NGINX_MODULES var. Otherwise like I had done, you turn on "rtmp" to get PLAYER side supports, and "accidentally" end up being able to also SERVE the protocol, which I wouldn't use nginx to do in a million years anyway. Then you spend a day finally figuring out that rtmp is what is doing it, EVEN THOUGH it isn't in your config file and thus should be doing NADA, and then figure out the solution is to "-rtmp" in the package.use for nginx, which is turning off some crap I never actually turned on (in NGINX_MODULES) so... it's a very odd chain of logic fails that lead here, any one of which could be fixed reasonably easily.
...OR since apparently if you add the rtmp configuration to your config file and tell it to be disabled (which SHOULD be the default, but isn't), then rtmp doesn't load and nginx workers stop crashing... so then the other solution is when USE contains "rtmp", add the extra crap to shut it off by default into the nginx.conf. I'm not sure if this ebuild has a config-generator or just uses a canned one. I know I turned "gzip" off in NGINX_MODULES and it didn't take those out of the resulting config (I had purged all /etc/nginx before reinstalling, of course). So that's another case where bad handling leads to failed installation - you have to go modify the distro-supplied config and remove all gzip refs (including in the logmsg formatting string) to get it to fire up.
(In reply to Tony Butler from comment #14) > ...OR since apparently if you add the rtmp configuration to your config file > and tell it to be disabled (which SHOULD be the default, but isn't), then > rtmp doesn't load and nginx workers stop crashing... > > so then the other solution is when USE contains "rtmp", add the extra crap > to shut it off by default into the nginx.conf. I'm not sure if this ebuild > has a config-generator or just uses a canned one. I know I turned "gzip" > off in NGINX_MODULES and it didn't take those out of the resulting config (I > had purged all /etc/nginx before reinstalling, of course). So that's > another case where bad handling leads to failed installation - you have to > go modify the distro-supplied config and remove all gzip refs (including in > the logmsg formatting string) to get it to fire up. I don't think moving away from USE_EXPAND is a good idea (what you refer to as NGINX_MODULES). It gives us a clean separation of imap vs http configurations and is a central concept to gentoo/portage. rtmp falls out of http since its a different protocol. rtmp is disabled by default, so moving into making the default configuration more complicated is something I'd like to avoid (not sure why you say it's not. Do you perhaps use star notations in your package.use?). Also, this concept doesn't really solve the core of this issue which is avoiding specific scenarios in third party modules based on version bumping (read: how do we become more stable). Bugs will show up, so being much more careful for stablereq'ing is probably still the way to go. The reason 1.7.6 was stabereq'd was because it introduced fixes for security issues and wasn't tested enough. This on the other hand highlighted that we'd have to consider a "frozen" third party stable tree which is something I'm currently evaluating. The reason nginx's gzip module is enabled by default is because upstream does so. It's in the nginx "standard" modules list. If you read the ebuild, you can see that we've split them up pretty neatly.
I will check out the ebuild before yelling any more. Perhaps I also don't understand how the whole USE_EXPAND thing works or is supposed to work, as I have seen more packages NOT using it where they COULD preferring a billion USE flags instead. Apparently that is the "wrong" way? However I don't think I did anything out of the ordinary other than flip a USE flag (and for everything else ufed listed OTHER THAN nginx). And I didn't put it in the NGINX_MODULES so why would it get included by default just via the USE flag?
(In reply to Tony Butler from comment #16) > I will check out the ebuild before yelling any more. Perhaps I also don't > understand how the whole USE_EXPAND thing works or is supposed to work, as I > have seen more packages NOT using it where they COULD preferring a billion > USE flags instead. Apparently that is the "wrong" way? > > However I don't think I did anything out of the ordinary other than flip a > USE flag (and for everything else ufed listed OTHER THAN nginx). And I > didn't put it in the NGINX_MODULES so why would it get included by default > just via the USE flag? So, I feel as we're slightly derailing the topic in here. It's all good, but I prefer keeping it low noise for others in the thread. Feel free to ping me on IRC (jbergstroem@fnode) and I'll be happy to explain. Otherwise, you have some basics at this page: http://devmanual.gentoo.org/general-concepts/use-flags/ As for continuing this thread, lets focus on getting the fix into 1.7.6 as well as adding 1.7.7 to the tree as unstable. Keep your eyes on the nginx overlay at github to follow the overarching strategy of stable vs unstable vs ... : https://github.com/gentoo/nginx-overlay
(In reply to Johan Bergström from comment #10) > ..I've also bumped nginx to 1.7.7 with this branch: > https://github.com/gentoo/nginx-overlay/pull/14. > > Feel free to review/merge. Sadly I don't have write acces there, only gentoo-x86 CVS RW access, so I guess we should wait for Tiziano.
Fixed in stable 1.7.6 and in new 1.7.7, OK'd by Johan Bergström.
