Hello. redis-cli (2.6.7) crashes with segmentation fault. ask ~ # redis-cli redis 127.0.0.1:6379> set 1Segmentation fault ask ~ # ask ~ # emerge --info Portage 2.1.11.38 (default/linux/amd64/10.0, gcc-4.6.3, glibc-2.16.0, 3.4.24 x86_64) ================================================================= System uname: Linux-3.4.24-x86_64-QEMU_Virtual_CPU_version_1.1.1-with-gentoo-2.2 Timestamp of tree: Thu, 10 Jan 2013 11:00:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p39 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.7.2-r3, 3.1.4-r3 dev-util/cmake: 2.8.7-r2 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.5 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.69 sys-devel/automake: 1.11.2, 1.12.6 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 2.6.39 (virtual/os-headers) sys-libs/glibc: 2.16.0 Repositories: gentoo x-portage ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /opt/openfire/resources/security/ /usr/share/gnupg/qualified.txt /var/spool/munin-async/.ssh" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LDFLAGS="-Wl,-O1 -Wl,--as-needed" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="acl amd64 berkdb bzip2 cli cracklib crypt cups cxx dri fortran gdbm gpm iconv ipv6 mmx modules mudflap multilib ncurses nls nptl openmp pam pcre pppd readline session sse sse2 ssl tcpd unicode zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_form authn_dbd auth_digest auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias proxy proxy_ajp proxy_balancer proxy_connect proxy_ftp proxy_http proxy_scgi asis cern_meta charset_lite dbd dumpio ident imagemap log_forensic reqtimeout substitute version" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="dummy" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON ask ~ # best regards Reproducible: Always
Confirm. Also it fails only with jemalloc.
Sorry for my late response. I somehow missed this. Questions to both: 1. What version of jemalloc do you run? 2. Did you just update this? 3. Did you rebuild redis afterwards? Also, please try rebuilding redis. The jemalloc 3.2.0 ebuild did a change that would result in something similar to redis.
Hello. Just updated to 2.6.8. Problem remains. To your questions: > 1. What version of jemalloc do you run? [ebuild R ~] dev-libs/jemalloc-3.2.0 USE="-debug -static-libs -stats" 2. Did you just update this? No. I suppose a while back. But e redid it. 3. Did you rebuild redis afterwards? Yes. best regards t.
(In reply to comment #3) > Hello. > > Just updated to 2.6.8. Problem remains. > > To your questions: > > > 1. What version of jemalloc do you run? > > [ebuild R ~] dev-libs/jemalloc-3.2.0 USE="-debug -static-libs -stats" > > 2. Did you just update this? > > No. I suppose a while back. But e redid it. > > 3. Did you rebuild redis afterwards? > > Yes. > > best regards > t. thanks, I think I managed to reproduce this. Will look into it tomorrow.
Here's output from debug builds: Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7bbc8c9 in arena_mapbits_get () from /usr/lib64/libjemalloc.so.1 (gdb) bt #0 0x00007ffff7bbc8c9 in arena_mapbits_get () from /usr/lib64/libjemalloc.so.1 #1 0x00007ffff7bbcb09 in arena_mapbits_allocated_get () from /usr/lib64/libjemalloc.so.1 #2 0x00007ffff7bbf662 in arena_salloc () from /usr/lib64/libjemalloc.so.1 #3 0x00007ffff7bb85d7 in free () from /usr/lib64/libjemalloc.so.1 #4 0x000000000040cfbb in linenoise () #5 0x000000000040474b in main () The problem is essentially that it needs access to the internal jemalloc headers. Why these aren't installed, I'm not so sure about - but I'll keep investigating.
If I rebuild jemalloc with --with-jemalloc-prefix=je_ everything works as intended. This brings us back to the jemalloc ebuild removing this for 3.2.0.ebuild. I _think_ its a naming conflict, since the default prefix for jemalloc functions are je_malloc,free etc. A couple of options 1. Start bashing at the jemalloc ebuild again 2. Use a static jemalloc (as upstream does) 3. Dive deeper into the issue (what I'm doing now)
I don't want this bug open anymore, so I opted for #2. Please test ebuild from my overlay: https://github.com/jbergstroem/gentoo-overlay/compare/master...redis-jemalloc-static (note: this also version bumps redis to 2.6.9). I would really like to avoid using a static jemalloc, but lets solve that later.
Hello. Unfortunately it still does not work: [ebuild R ~] dev-db/redis-2.6.9::jbergstroem USE="jemalloc -tcmalloc {-test}" 0 kB Total: 1 package (1 reinstall), Size of downloads: 0 kB hostname /usr/local # ls -l /usr/bin/redis-cli -rwxr-xr-x 1 root root 94528 Jan 21 10:44 /usr/bin/redis-cli hostname /usr/local # redis-cli redis 127.0.0.1:6379> set 1Segmentation fault hostname /usr/local # But thanks for your work so far. t.
Uahh. I had to set -jemalloc. Now it works. redis 127.0.0.1:6379> help redis-cli 2.6.9 Type: "help @<group>" to get a list of commands in <group> "help <command>" for help on <command> "help <tab>" to get a list of possible help topics "quit" to exit redis 127.0.0.1:6379> set 1 (error) ERR wrong number of arguments for 'set' command redis 127.0.0.1:6379> cheers t.
Created attachment 336390 [details] build.log (In reply to comment #9) > Uahh. I had to set -jemalloc. Now it works. > > redis 127.0.0.1:6379> help > redis-cli 2.6.9 > Type: "help @<group>" to get a list of commands in <group> > "help <command>" for help on <command> > "help <tab>" to get a list of possible help topics > "quit" to exit > redis 127.0.0.1:6379> set 1 > (error) ERR wrong number of arguments for 'set' command > redis 127.0.0.1:6379> > > cheers > t. I can't reproduce this. If you use USE=-jemalloc, you're not using jemalloc, but the libc one. Did you perhaps forget to checkout the branch? Attached results are built with jemalloc 3.2.0 from gentoo-x86 (irrelevant, but anyway), using the redis-jemalloc-static branch from my repo (commit edb15599743542c2e385459c0ae687e52fb399b4): g2-x86_64 /home/jbergstroem # emerge -av redis These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ~] dev-db/redis-2.6.9::jbergstroem USE="jemalloc -tcmalloc {-test}" 0 kB Total: 1 package (1 reinstall), Size of downloads: 0 kB Would you like to merge these packages? [Yes/No] <see attached log> g2-x86_64 /home/jbergstroem # redis-cli info # Server redis_version:2.6.8 ...
>Did you perhaps forget to checkout the branch? I just used your ebuild. >redis_version:2.6.8 Why 2.6.8? Shouldn't this be 2.6.9? best regards t.
(In reply to comment #11) > >Did you perhaps forget to checkout the branch? > > I just used your ebuild. Yes, but as I showed earlier, I did all the commits for this change in a specific branch. Please check this out since you otherwise will be running the same ebuild as in tree. Exact steps: git clone/layman git://github.com/jbergstroem/gentoo-overlay.git cd gentoo-overlay && git checkout redis-jemalloc-static emerge redis > > >redis_version:2.6.8 > > Why 2.6.8? Shouldn't this be 2.6.9? I just upgraded without restarting server. Not really relevant since the issue here is -cli. > > best regards > t. Let me know how you go.
I just hit this and I figured out what's happening. So the backtrace: #3 0x00007ffff7bb85d7 in free () from /usr/lib64/libjemalloc.so.1 #4 0x000000000040cfbb in linenoise () #5 0x000000000040474b in main () linenoise is where the issue is. In particular: AM_CFLAGS="-std=c99 -pedantic -Wall -W -D__EXTENSIONS__ -D_XPG6" http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-db/redis/files/configure.ac-2.2?view=markup So that means linenoise gets compiled with -std=c99. Which noting this line as part of linenoiseHistoryAdd: linecopy = strdup(line); Which means this happens: "gcc enables the appropriate macro(s) by default, but using -ansi or -std=c99 will disable them" ... "In the absence of a visible declaration, the compiler assumes (under C90 rules) that strdup() returns int. This results in undefined behavior." http://stackoverflow.com/questions/8359966/strdup-returning-address-out-of-bounds Thus leading to the fun segfault we notice.
(In reply to comment #13) > I just hit this and I figured out what's happening. So the backtrace: > > #3 0x00007ffff7bb85d7 in free () from /usr/lib64/libjemalloc.so.1 > #4 0x000000000040cfbb in linenoise () > #5 0x000000000040474b in main () > > linenoise is where the issue is. In particular: > > AM_CFLAGS="-std=c99 -pedantic -Wall -W -D__EXTENSIONS__ -D_XPG6" > > http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-db/redis/files/ > configure.ac-2.2?view=markup > > So that means linenoise gets compiled with -std=c99. Which noting this line > as part of linenoiseHistoryAdd: > > linecopy = strdup(line); > > Which means this happens: > > "gcc enables the appropriate macro(s) by default, but using -ansi or > -std=c99 will disable them" > ... > "In the absence of a visible declaration, the compiler assumes (under C90 > rules) that strdup() returns int. This results in undefined behavior." > > http://stackoverflow.com/questions/8359966/strdup-returning-address-out-of- > bounds > > Thus leading to the fun segfault we notice. This is very interesting. Thanks for your work! What I don't quite understand is why this only happens when we compile jemalloc shared using JEMALLOC_NO_DEMANGLE and not with --with-jemalloc-prefix. If you've read through the thread, you've probably picked up that upstream chooses same prefixing as default names. I still don't get how that is tied to compiling linenoise with -std=c99.
Also there seems to be some parts of the redis codebase that need -std=c99 so taking it out completely won't be a good idea imho. One solution would be to remove it from the generated Makefile.in. Also addition of '-D_GNU_SOURCE' seems to help with that strdup issue at least according to here: http://gcc.gnu.org/ml/gcc-help/2011-10/msg00117.html Though I'm not sure what effect that would have if enabled globally.
> This is very interesting. Thanks for your work! What I don't quite > understand is why this only happens when we compile jemalloc shared using > JEMALLOC_NO_DEMANGLE and not with --with-jemalloc-prefix. > > If you've read through the thread, you've probably picked up that upstream > chooses same prefixing as default names. I still don't get how that is tied > to compiling linenoise with -std=c99. Either that jemalloc thing is another issue, or it's causing free() to be a lot more strict. Either way if you notice this linenoise code: case 13: /* enter */ history_len--; free(history[history_len]); return (int)len; case 3: /* ctrl-c */ When you hit ENTER (which is what most people are going to do) that free() happens which if you see here: linecopy = strdup(line); if (!linecopy) return 0; if (history_len == history_max_len) { free(history[0]); memmove(history,history+1,sizeof(char*)*(history_max_len-1)); history_len--; } history[history_len] = linecopy; That history array is going to get populated with unknown data because you're compiling with -std=c99 which means that: "gcc enables the appropriate macro(s) by default, but using -ansi or -std=c99 will disable them" strdup's macro will not be visible and: "In the absence of a visible declaration, the compiler assumes (under C90 rules) that strdup() returns int. This results in undefined behavior." strdup has undefined behavior so you're going to be trying to free garbage basically. Also note this doesn't happen in 2.4 series because the linenoise Makefile doesn't contain CFLAGS so this sed doesn't happen: for MKF in $(find -name 'Makefile' | cut -b 3-); do mv "${MKF}" "${MKF}.in" sed -i -e 's:$(CC):@CC@:g' \ -e 's:$(CFLAGS):@AM_CFLAGS@:g' \ -e 's: $(DEBUG)::g' \ -e 's:$(OBJARCH)::g' \ -e 's:ARCH:TARCH:g' \ -e '/^CCOPT=/s:$: $(LDFLAGS):g' \ "${MKF}.in" \ || die "Sed failed for ${MKF}" in 2.6 series it does contain CFLAGS so the -std=c99 replacement doesn't occur.
Without diving to deep into jemalloc (which I'm actually looking forward to, with my friend Jim Bean), here's an updated branch that removes -std=c99 for linenoise (and a cosmetic one for Lua): https://github.com/jbergstroem/gentoo-overlay/compare/redis-linenoise-c99 Please let me know how it goes! (big thanks to Chris for looking into this)
See "final" ebuild in bug 453606.
Fixed in 2.6.9.
