Calling openrc leads to a crash with the following message: htpc ~ # openrc *** Error in `openrc': double free or corruption (fasttop): 0x00000000020f1080 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7394b)[0x7f699bfe694b] /lib64/libc.so.6(+0x78e2e)[0x7f699bfebe2e] /lib64/libc.so.6(+0x7961b)[0x7f699bfec61b] /lib64/librc.so.1(rc_stringlist_free+0x24)[0x7f699c92f774] /lib64/librc.so.1(rc_service_daemons_crashed+0x36b)[0x7f699c92bfdb] openrc[0x406a77] /lib64/libc.so.6(__libc_start_main+0xf0)[0x7f699bf92fa0] openrc[0x4074f5] ======= Memory map: ======== 00400000-0041d000 r-xp 00000000 00:0f 193567 /sbin/openrc 0061c000-0061d000 r--p 0001c000 00:0f 193567 /sbin/openrc 0061d000-0061e000 rw-p 0001d000 00:0f 193567 /sbin/openrc 020c9000-0210b000 rw-p 00000000 00:00 0 [heap] 7f699bb07000-7f699bb1d000 r-xp 00000000 00:0f 37991 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.2/libgcc_s.so.1 7f699bb1d000-7f699bd1c000 ---p 00016000 00:0f 37991 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.2/libgcc_s.so.1 7f699bd1c000-7f699bd1d000 r--p 00015000 00:0f 37991 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.2/libgcc_s.so.1 7f699bd1d000-7f699bd1e000 rw-p 00016000 00:0f 37991 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.9.2/libgcc_s.so.1 7f699bd1e000-7f699bd6e000 r-xp 00000000 00:0f 6921 /lib64/libncurses.so.5.9 7f699bd6e000-7f699bf6d000 ---p 00050000 00:0f 6921 /lib64/libncurses.so.5.9 7f699bf6d000-7f699bf71000 r--p 0004f000 00:0f 6921 /lib64/libncurses.so.5.9 7f699bf71000-7f699bf72000 rw-p 00053000 00:0f 6921 /lib64/libncurses.so.5.9 7f699bf72000-7f699bf73000 rw-p 00000000 00:00 0 7f699bf73000-7f699c104000 r-xp 00000000 00:0f 6894 /lib64/libc-2.20.so 7f699c104000-7f699c304000 ---p 00191000 00:0f 6894 /lib64/libc-2.20.so 7f699c304000-7f699c308000 r--p 00191000 00:0f 6894 /lib64/libc-2.20.so 7f699c308000-7f699c30a000 rw-p 00195000 00:0f 6894 /lib64/libc-2.20.so 7f699c30a000-7f699c30e000 rw-p 00000000 00:00 0 7f699c30e000-7f699c31b000 r-xp 00000000 00:0f 6932 /lib64/libpam.so.0.83.1 7f699c31b000-7f699c51a000 ---p 0000d000 00:0f 6932 /lib64/libpam.so.0.83.1 7f699c51a000-7f699c51b000 r--p 0000c000 00:0f 6932 /lib64/libpam.so.0.83.1 7f699c51b000-7f699c51c000 rw-p 0000d000 00:0f 6932 /lib64/libpam.so.0.83.1 7f699c51c000-7f699c51e000 r-xp 00000000 00:0f 6903 /lib64/libdl-2.20.so 7f699c51e000-7f699c71e000 ---p 00002000 00:0f 6903 /lib64/libdl-2.20.so 7f699c71e000-7f699c71f000 r--p 00002000 00:0f 6903 /lib64/libdl-2.20.so 7f699c71f000-7f699c720000 rw-p 00003000 00:0f 6903 /lib64/libdl-2.20.so 7f699c720000-7f699c725000 r-xp 00000000 00:0f 193387 /lib64/libeinfo.so.1 7f699c725000-7f699c924000 ---p 00005000 00:0f 193387 /lib64/libeinfo.so.1 7f699c924000-7f699c925000 r--p 00004000 00:0f 193387 /lib64/libeinfo.so.1 7f699c925000-7f699c926000 rw-p 00005000 00:0f 193387 /lib64/libeinfo.so.1 7f699c926000-7f699c932000 r-xp 00000000 00:0f 193386 /lib64/librc.so.1 7f699c932000-7f699cb31000 ---p 0000c000 00:0f 193386 /lib64/librc.so.1 7f699cb31000-7f699cb32000 r--p 0000b000 00:0f 193386 /lib64/librc.so.1 7f699cb32000-7f699cb33000 rw-p 0000c000 00:0f 193386 /lib64/librc.so.1 7f699cb33000-7f699cb35000 r-xp 00000000 00:0f 6950 /lib64/libutil-2.20.so 7f699cb35000-7f699cd34000 ---p 00002000 00:0f 6950 /lib64/libutil-2.20.so 7f699cd34000-7f699cd35000 r--p 00001000 00:0f 6950 /lib64/libutil-2.20.so 7f699cd35000-7f699cd36000 rw-p 00002000 00:0f 6950 /lib64/libutil-2.20.so 7f699cd36000-7f699cd58000 r-xp 00000000 00:0f 6885 /lib64/ld-2.20.so 7f699cf33000-7f699cf36000 rw-p 00000000 00:00 0 7f699cf53000-7f699cf57000 rw-p 00000000 00:00 0 7f699cf57000-7f699cf58000 r--p 00021000 00:0f 6885 /lib64/ld-2.20.so 7f699cf58000-7f699cf59000 rw-p 00022000 00:0f 6885 /lib64/ld-2.20.so 7f699cf59000-7f699cf5a000 rw-p 00000000 00:00 0 7fffd6498000-7fffd64b9000 rw-p 00000000 00:00 0 [stack] 7fffd64d3000-7fffd64d4000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] zsh: abort openrc gdb backtrace and possibly relevant init script attached. Reproducible: Always Steps to Reproduce: 1. run /sbin/openrc 2. 3. Portage 2.2.14 (python 2.7.8-final-0, default/linux/amd64/13.0, gcc-4.9.2, glibc-2.20, 3.14.25 x86_64) ================================================================= System uname: Linux-3.14.25-x86_64-Intel-R-_Celeron-R-_CPU_G530_@_2.40GHz-with-gentoo-2.2 KiB Mem: 8144528 total, 6918044 free KiB Swap: 0 total, 0 free Timestamp of tree: Tue, 02 Dec 2014 19:15:01 +0000 ld GNU ld (GNU Binutils) 2.24 app-shells/bash: 4.3_p30-r1 dev-java/java-config: 2.2.0 dev-lang/perl: 5.20.1-r3 dev-lang/python: 2.7.8, 3.4.2 dev-util/cmake: 3.0.2 dev-util/pkgconfig: 0.28-r2 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.13.6 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.11.6-r1, 1.14.1 sys-devel/binutils: 2.24-r3 sys-devel/gcc: 4.9.2 sys-devel/gcc-config: 1.8 sys-devel/libtool: 2.4.3-r2 sys-devel/make: 4.1-r1 sys-kernel/linux-headers: 3.17-r1 (virtual/os-headers) sys-libs/glibc: 2.20 Repositories: gentoo vdr-testing vdr-devel gen2ovl-googoo2 own ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--quiet-build=n" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="ftp://ftp-stud.fht-esslingen.de/Mirrors/gentoo http://distfiles.gentoo.org" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp/compile" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/vdr-testing /var/lib/layman/vdr-devel /var/lib/layman/gen2ovl-googoo2 /usr/local/portage/own" SYNC="rsync://rsync3.de.gentoo.org/gentoo-portage" USE="X a52 acl alsa amd64 berkdb bluetooth bluez bluray bzip2 cairo cdda cli consolekit corefonts cracklib crypt cxx dbus djvu dri dts dvd egl emacs encode exif fam ffmpeg flac fortran gdbm gif gimp git gles gmp gnutls gtk iconv idn inotify ipv6 ithreads jpeg lame libedit live lzma mad mainmenuhooks mmx modules mp3 mpeg multilib ncurses network nls nptl nsplugin ogg opengl openmp pam pch pcre pdf png qalculate readline rtsp samba session smp sound spell sse sse2 sse3 sse4 sse4_1 sse4_2 ssl ssse3 stream subversion svg tcpd theora threads tiff truetype udev unicode usb vdpau vlc vorbis vpx webkit x264 xcb xft xv xvid zlib zsh-completion" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="intel nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" USE_PYTHON="2.7" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Created attachment 390888 [details] gdb backtrace
Created attachment 390890 [details] vdr init script
I am unable to reproduce this. What about anyone else?
I've successfully reproduced it on my main machine. Just starting vdr and running openrc doesn't suffice. I had to add it to the default runlevenl first. The first call to openrc starts it successfully, the second one leads to the same crash. Calling openrc when vdr is started but does not belong to any runlevel does not lead to a crash. Instead, the service gets stopped successfully. In order to get a working VDR installation, emerging media-plugins/vdr-dummydevice and enabling it with eselect vdr-plugin enable should suffice.
Does this happen with all versions of OpenRC in the tree? If not, which is the first one it happens with?
=sys-apps/openrc-0.13.1 works, =sys-apps/openrc-0.13.2 crashes.
Could you paste all of the files in /run/openrc/daemons/vdr after replicating the issue into this bug? There is generally only one file named 001, but if there are multiple, paste them all, and mention which one is which.
001: exec=/usr/bin/vdr argv_0=/usr/bin/vdr argv_1=-u argv_2=vdr argv_3=--watchdog=60 argv_4=--epgfile=/var/vdr/epg.data argv_5=--cachedir=/dev/shm argv_6=--log=3 argv_7=--video=/var/vdr/video argv_8=--record=/usr/share/vdr/bin/vdrrecord-gate.sh argv_9=-D argv_10=1 argv_11=-D argv_12=2 argv_13=--shutdown=/usr/share/vdr/bin/vdrshutdown-gate.sh argv_14=--plugin=softhddevice -d :0.0 -v vdpau -a hw:0,0 -p hw:0,1 -f -s -w alsa-no-close-open argv_15=--plugin=femon argv_16=--plugin=skinnopacity --iconpath=/usr/share/vdr/plugins/skinnopacity/icons/ --logopath=/usr/share/channel-logos/dvbviewer/ --epgimages=/dev/shm/ argv_17=--plugin=tvguide --logodir=/usr/share/channel-logos/dvbviewer/ --epgimages=/dev/shm/ argv_18=--plugin=vnsiserver argv_19=--daemon pidfile= 002: exec=/usr/sbin/vdr-watchdogd argv_0=vdr-watchdogd pidfile=/var/run/vdrwatchdog.pid
Since there were so few commits, this is porbably due to netbsd's queue.h vs glibc's queue.h although I haven't looked carefully why. @reporter, let's settle this question. If its not too much trouble, rebuild openrc using /usr/include/sys/queue.h replacing openrc's ./src/includes/queue.h. Then test. If this doesn't double free, then its in the implementation details of TAILQ_* macros. My recommendation would be to fix up openrc's assumptions and keep the netbsd version since it is better maintained.
Created attachment 391796 [details] build.log with replaced queue.h Replacing queue.h leads to compile errors. See the attached build.log.
(In reply to n0t3p4d.opensource from comment #10) > Created attachment 391796 [details] > build.log with replaced queue.h > > Replacing queue.h leads to compile errors. See the attached build.log. Yep confirmed, because there were *two* changes. You also need to revert commit f9d1742a909f41d8a7994bb58be630eedfc0f574. It then does compile. Please try that and see if the double free goes away.
That wasn't the cause. I ran git bisect and can confirm that it was introduced by commit f9acd65497c6e561fbf5420386a99d681fede859.
This issue was solved in be952bebb3647069fb93b9791ee3439698f697ca and a new openrc was released just after that.. I wonder why broken version is still in tree :/
As far as I can tell, it's included in at least openrc-0.13.6. Nevertheless, I still get the crash with current git master so apparently there's still some work to do :)
(In reply to n0t3p4d.opensource from comment #12) > That wasn't the cause. I ran git bisect and can confirm that it was > introduced by commit f9acd65497c6e561fbf5420386a99d681fede859. Thanks, this makes sense now. We just missed this double free before.
Can we tell which variable is being double free'd?
Created attachment 393644 [details, diff] 0001-fix-double-freee.patch Can you please test with this patch applied? This makes the algorithm more similar to the way it was before the commit you cited; the spidfile variable is now only used as temporary storage. Thanks much, William
Created attachment 393656 [details, diff] 0001-fix-double-free.patch This is a cleaned up version of the previous patch that fixes indentations. I am putting it here because it may be easier to read.
I've applied the patch to openrc-0.13.6 and can confirm that it works as expected. Thanks a lot for fixing this!
After some rubber duck debugging, I found the cause. The first run through the loop reads the 002 file, and sets pidfile to /var/run/vdrwatchdog.pid. This reference is replicated to spidfile and then the chroot check is done. After that, the pidfile is read, and then spidfile is freed and NULLed. However, pidfile is *not* NULLed, and so in the second loop around, pidfile is still pointing to a freed memory location, and gets freed again. Because glibc doesn't do a heap check on every free (because it's expensive), the double free check doesn't trigger until much later, producing a sort of red herring making us think it's the stringlist breaking.
This is fixed in commit 7447883 and will be in OpenRC-0.14 and 0.13.7. Thanks for the report.
