So, the final version of 2.6.0 arrived. This brings forth a lot of changes. I'm aiming to move some sed stuff to patches and pushed a fork to do so: https://github.com/jbergstroem/redis/compare/gentoo-fixes What's done: - Adapted shared jemalloc build (based on original pull req https://github.com/antirez/redis/pull/137) - Builds against shared lua Todo: - Create ebuilds for lua-stack, lua-cjson and lua-msgpack (which is currently bundled in the redis version of lua) - use .make-settings over our sed Makefile (?) - move config sed's to patches and so something similar for the sentinel config - init/conf for sentinel (http://redis.io/topics/sentinel) - Other minor ebuilds updates I won't attach an ebuild just now. Bare with me a few days. Also talking to djc/rafaelmartins to help out with the lua ebuilds since I'm not very comfortable pulling them in as deps just yet.
Created attachment 327728 [details] redis-2.6.2.ebuild 2.6.2 attached. Patches arrive shortly. Diff inlined: --- redis-2.4.17.ebuild 2012-10-08 16:31:06.000000000 +1100 +++ redis-2.6.2.ebuild 2012-10-30 14:03:34.902095679 +1100 @@ -1,8 +1,8 @@ # Copyright 1999-2012 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 -# $Header: /var/cvsroot/gentoo-x86/dev-db/redis/redis-2.4.17.ebuild,v 1.2 2012/10/08 05:15:34 robbat2 Exp $ +# $Header: $ -EAPI="4" +EAPI=4 inherit autotools eutils flag-o-matic user @@ -25,23 +25,13 @@ S="${WORKDIR}/${PN}-${PV/_/-}" -REDIS_PIDDIR=/var/run/redis/ -REDIS_PIDFILE=${REDIS_PIDDIR}/redis.pid -REDIS_DATAPATH=/var/lib/redis -REDIS_LOGPATH=/var/log/redis -REDIS_LOGFILE=${REDIS_LOGPATH}/redis.log - pkg_setup() { enewgroup redis 75 - enewuser redis 75 -1 ${REDIS_DATAPATH} redis + enewuser redis 75 -1 /var/lib/redis redis } src_prepare() { - epatch "${FILESDIR}/redis-2.4.3-shared.patch" - epatch "${FILESDIR}/redis-2.4.4-tcmalloc.patch" - if use jemalloc ; then - sed -i -e "s/je_/j/" src/zmalloc.c src/zmalloc.h - fi + epatch "${FILESDIR}/${P}"-{shared,config}.patch # now we will rewrite present Makefiles local makefiles="" for MKF in $(find -name 'Makefile' | cut -b 3-); do @@ -78,37 +68,25 @@ } src_install() { - # configuration file rewrites insinto /etc/ - sed -r \ - -e "/^pidfile\>/s,/var.*,${REDIS_PIDFILE}," \ - -e '/^daemonize\>/s,no,yes,' \ - -e '/^# bind/s,^# ,,' \ - -e '/^# maxmemory\>/s,^# ,,' \ - -e '/^maxmemory\>/s,<bytes>,67108864,' \ - -e "/^dbfilename\>/s,dump.rdb,${REDIS_DATAPATH}/dump.rdb," \ - -e "/^dir\>/s, .*, ${REDIS_DATAPATH}/," \ - -e '/^loglevel\>/s:(verbose|debug):notice:' \ - -e "/^logfile\>/s:stdout:${REDIS_LOGFILE}:" \ - <redis.conf \ - >redis.conf.gentoo - newins redis.conf.gentoo redis.conf - use prefix || fowners redis:redis /etc/redis.conf - fperms 0644 /etc/redis.conf + doins redis.conf sentinel.conf + use prefix || fowners redis:redis /etc/{redis,sentinel}.conf + fperms 0644 /etc/{redis,sentinel}.conf newconfd "${FILESDIR}/redis.confd" redis newinitd "${FILESDIR}/redis.initd" redis - nonfatal dodoc 00-RELEASENOTES BUGS CONTRIBUTING README + nonfatal dodoc 00-RELEASENOTES BUGS CONTRIBUTING MANIFESTO README dobin src/redis-cli dosbin src/redis-benchmark src/redis-server src/redis-check-aof src/redis-check-dump fperms 0750 /usr/sbin/redis-benchmark + dosym /usr/sbin/redis-server /usr/sbin/redis-sentinel if use prefix; then diropts -m0750 else diropts -m0750 -o redis -g redis fi - keepdir ${REDIS_DATAPATH} ${REDIS_LOGPATH} + keepdir /var/{log,lib}/redis } I never bothered to do a init script for sentinel since you'd probably want to use redis/a separate config for that anyway. If any users of Sentinel want this changed, please suggest by patch.
Created attachment 327730 [details, diff] redis-2.6.2-config.patch
Created attachment 327732 [details, diff] redis-2.6.2-shared.patch
So, here's the bump. Unfortunately, we couldn't split out lua since redis bakes its own with a couple of custom built extensions. I've discussed this with both djc and rafaelmartins (lua developers/maintainers) and the change to use shared lua libraries won't be the easiest of tasks. I also skipped .make-settings since there's no benefit other than cosmetics.
It doesn't compile with USE="-*": clang -o lua -Wl,-O1 -Wl,--as-needed lua.o liblua.a -lm clang -o luac -Wl,-O1 -Wl,--as-needed luac.o print.o liblua.a -lm make[3]: Leaving directory `/var/tmp/portage/dev-db/redis-2.6.2/work/redis-2.6.2/deps/lua/src' make[2]: Leaving directory `/var/tmp/portage/dev-db/redis-2.6.2/work/redis-2.6.2/deps' CC adlist.o CC ae.o In file included from In file included from ae.c:43: ./zmalloc.h:50:10: fatal error: 'jemalloc/jemalloc.h' file not found #include <jemalloc/jemalloc.h> ^ adlist.c:34: ./zmalloc.h:50:10: fatal error: 'jemalloc/jemalloc.h' file not found #include <jemalloc/jemalloc.h> ^ 1 error generated. 1 error generated. make[1]: *** [adlist.o] Error 1 make[1]: *** Waiting for unfinished jobs.... make[1]: *** [ae.o] Error 1 make[1]: Leaving directory `/var/tmp/portage/dev-db/redis-2.6.2/work/redis-2.6.2/src' make: *** [all] Error 2 Redis compiles fine when dev-libs/jemalloc is installed.
We should probably have some REQUIRED_USE shizzle?
(In reply to comment #6) > We should probably have some REQUIRED_USE shizzle? No, the problem is the fallback MALLOC build which for some reason still picks up jemalloc. I relied on upstream here - so something isn't working properly. I'll update the ebuild (and bump to 2.6.3) shortly.
Created attachment 328610 [details] redis-2.6.3.ebuild Ok, here's an updated ebuild. Changes from 2.6.2: - FORCE_LIBC_MALLOC was a mistake (heritage from 2.4). It's MALLOC now. This fixes Marcin's issue. - Simplify myconf Don't forget to rename patches (since we use ${P} in ebuild)
2.6.4 out, with this important addition: * [IMPROVED] BSD license and copyright notice added to every .c and .h file. Lets get this into tree now!
2.6.5. Renaming works. Pretty please with sugar on top? UPGRADE URGENCY: MODERATE Warning: this release of Redis introduces a different behavior in MULTI/EXEC handling of errors. This was done because the new behavior is safer compared to the old one, and should not break any code targeting Redis 2.6 in a critical way. For more information check http://redis.io/topics/transactions * [IMPROVED] RDB/AOF childern now log amount of additional memory used because of copy on write. * [BUGFIX] MIGRATE non critical fixes (see commits for details). * [BUGFIX] MULTI/EXEC: now EXEC aborts on errors before EXEC. * [BUGFIX] Fix integer overflow in zunionInterGenericCommand resulting into Z[INTER|UNION][STORE] commands to crash under extremely unlikely conditions (almost impossible in real world). * [BUGFIX] EVALSHA is now case insensitive (and will not crash).
Look at that, 2.6.6 already. UPGRADE URGENCY: CRITICAL if you experienced one more more crashes. MODERATE if Redis is running fine for you. Again, just rename all files and you're set. Unfortunately, changes in gentoo-x86 jemalloc behaviour is currently non-compatible with all running redis ebuilds (at least unstable). A path for resolving this needs to be in place before acting on any redis ebuilds. I would appreciate more feedback in bug 441834 (and the bug that made it visible, 444796).
Created attachment 331094 [details] redis-2.6.6.ebuild Since there's not been a very active discussions from maintainers in bug 441834 I'm going to assume that the current state is our working state. I've updated the ebuilds to correspond to the latest jemalloc version (as well as bumping version names). Good to go in my opinion, but looking forward to feedback.
Created attachment 331096 [details] redis-2.6.6-config.patch
Created attachment 331098 [details, diff] redis-2.6.6-shared.patch This patch forces redis to be built with JEMALLOC_NO_DEMANGLE. This is basically how redis does it (but with internal static library). It basically means that JEMALLOC shouldn't unset the prefixed names of its functions. It is future proof.
2.6.7. Rename the 2.6.6 files and you should be set. UPGRADE URGENCY: MODERATE (unless you BLPOP using the same key multiple times). * [BUGFIX] Don't crash if BLPOP & co are called with the same key repeated multiple times (Issue #801).
Hello Johan, thank you for your work on Redis :) I have a hardened machine with Redis 2.6.7 and it segfaults when running redis-cli: # redis-cli redis 127.0.0.1:6379> INFOSegmentation fault # dmesg [1281428.075071] redis-cli[23043]: segfault at ffffffffb9400000 ip 000003e2ba1cd363 sp 000003ffffffbe60 error 4 in libjemalloc.so.1[3e2ba1c8000+30000] [1281428.075093] grsec: From 85.248.1.14: Segmentation fault occurred at ffffffffb9400000 in /usr/bin/redis-cli[redis-cli:23043] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:16206] uid/euid:0/0 gid/egid:0/0 A build without shared jemalloc works fine: # make # ./src/redis-cli redis 127.0.0.1:6379> INFO # Server redis_version:2.6.7 redis_git_sha1:00000000 redis_git_dirty:0 redis_mode:standalone os:Linux 3.5.4-hardened-r1 x86_64 The server appears to work fine so far: uptime_in_seconds:71689 used_memory_human:7.13M used_memory_peak_human:12.05M total_connections_received:2465790 total_commands_processed:4972053 instantaneous_ops_per_sec:111 used_cpu_sys:365.92 used_cpu_user:301.67 used_cpu_sys_children:0.75 used_cpu_user_children:4.69 # emerge --info Portage 2.1.11.31 (hardened/linux/amd64, gcc-4.5.4, glibc-2.15-r3, 3.5.4-hardened-r1 x86_64) ================================================================= System uname: Linux-3.5.4-hardened-r1-x86_64-Intel-R-_Xeon-R-_CPU_E5520_@_2.27GHz-with-gentoo-2.1 Timestamp of tree: Fri, 07 Dec 2012 06:45:01 +0000 ld GNU ld (GNU Binutils) 2.22 app-shells/bash: 4.2_p37 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.7.3-r2, 3.2.3 dev-util/cmake: 2.8.9 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.11.7 sys-apps/sandbox: 2.5 sys-devel/autoconf: 2.68 sys-devel/automake: 1.9.6-r3, 1.11.6 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 4.5.4 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r3 sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo x-hydra ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA Oracle-BCLA-JavaSE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-mtune=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-mtune=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://tux.rainside.sk/gentoo/ http://gentoo.wheel.sk/" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j9" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/hydra" SYNC="rsync://rsync.sk.gentoo.org/gentoo-portage" USE="acl amd64 berkdb bzip2 cli cracklib crypt cups cxx dri gdbm gpm hardened iconv justify mmx modules mudflap multilib ncurses nls nptl openmp pam pax_kernel pcre pppd readline session sse sse2 ssl tcpd unicode urandom xattr zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="authn_core authz_core authz_host dir mime unixd socache_shmcb log_config rewrite vhost_alias log_forensic proxy proxy_fcgi" APACHE2_MPMS="event" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON (In reply to comment #15) > 2.6.7. Rename the 2.6.6 files and you should be set. > > > UPGRADE URGENCY: MODERATE (unless you BLPOP using the same key multiple > times). > > * [BUGFIX] Don't crash if BLPOP & co are called with the same key repeated > multiple times (Issue #801).
(In reply to comment #16) > Hello Johan, > thank you for your work on Redis :) > > I have a hardened machine with Redis 2.6.7 and it segfaults when running > redis-cli: > > # redis-cli > redis 127.0.0.1:6379> INFOSegmentation fault > This will unfortunately most likely not be fixed through upstream (if you use something else than their build its a no-go), but could you perhaps debug this? just download redis, apply patches and build it with debug (CFLAGS -g)? Would be interesting to follow. Do you use any other applications using a shared jemalloc? Does it occur with tcmalloc?
Also, could you try older redis ebuilds (for instance 2.4.16) with <jemalloc-3.2?
I would like to ask to add improvement to init script allowing run more than one intance of redis. (topic "Redis is single threaded, how can I exploit multiple CPU / cores?" on redis faq).
(In reply to comment #19) > I would like to ask to add improvement to init script allowing run more than > one intance of redis. (topic "Redis is single threaded, how can I exploit > multiple CPU / cores?" on redis faq). To be honest, this is premature optimisation. You'll consume your RAM way earlier than starving your CPU. With that said; we do have this for memcached. Feel free to send a patch.
Done. Thanks!
