Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 526796 - x11-misc/x11vnc-0.9.14-r1: stack smashing attack
Summary: x11-misc/x11vnc-0.9.14-r1: stack smashing attack
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Hardened (show other bugs)
Hardware: AMD64 Linux
: Normal normal (vote)
Assignee: Alex Xu (Hello71)
URL:
Whiteboard:
Keywords:
: 574060 (view as bug list)
Depends on: 591528
Blocks: 589164
  Show dependency tree
 
Reported: 2014-10-25 20:54 UTC by Ed Santiago
Modified: 2016-12-05 16:08 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
valgrind --log-file=/tmp/valgrind-x11vnc.log x11vnc (valgrind-x11vnc.log,3.14 KB, text/plain)
2015-07-13 12:01 UTC, Ed Santiago
Details
x11vnc backtrace (x11vnc-backtrace.txt,34.55 KB, text/plain)
2015-07-13 14:57 UTC, Ed Santiago
Details
x11vnc-backtrace.log (gdb) (x11vnc-backtrace.log,2.68 KB, text/plain)
2015-07-13 18:47 UTC, Ed Santiago
Details
x11vnc-backtrace.log (0.9.14) (x11vnc-backtrace.log,26.31 KB, text/plain)
2016-11-27 21:37 UTC, SpanKY
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Santiago 2014-10-25 20:54:01 UTC
ssh'ed in from host m (for "monitor") to host d (for "dead display"):

   m$ ssh d -L 5901:localhost:5900
   d-via-ssh$ x11vnc

Back to m:

   m$ vncviewer :1

...then, in an xterm on d, run:

   d-via-vnc$ Xdialog --combobox ttt 10 10 aaaaa bbbbb

...click the arrow with the pulldown box, and click a few times on the horizontal scrollbar element and on the right/left arrows. After a few clicks the vnc session dies, with the following output in d-via-ssh:

   *** stack smashing detected ***: x11vnc - terminated
   x11vnc: stack smashing attack in function <unknown> - terminated
   Report to http://bugs.gentoo.org/

d's profile is hardened/linux/amd64

Reproducible: Always




[I] x11-misc/x11vnc
     Available versions:  0.9.13 0.9.13-r1 {avahi crypt fbcon +jpeg ssl system-libvncserver threads tk xinerama +zlib}
     Installed versions:  0.9.13-r1(06:31:27 AM 10/21/2014)(crypt jpeg ssl zlib -avahi -fbcon -system-libvncserver -threads -tk -xinerama)
     Homepage:            http://www.karlrunge.com/x11vnc/
     Description:         A VNC server for real X displays



Portage 2.2.8-r2 (hardened/linux/amd64, gcc-4.8.3, glibc-2.19-r1, 3.16.6-gentoo x86_64)
=================================================================
System uname: Linux-3.16.6-gentoo-x86_64-Intel-R-_Core-TM-_i7-4810MQ_CPU_@_2.80GHz-with-gentoo-2.2
KiB Mem:     8121440 total,   4915348 free
KiB Swap:          0 total,         0 free
Timestamp of tree: Sat, 25 Oct 2014 06:45:01 +0000
ld GNU ld (Gentoo 2.23.2 p1.0) 2.23.2
app-shells/bash:          4.2_p53
dev-lang/perl:            5.18.2-r2
dev-lang/python:          2.7.7, 3.3.5-r1, 3.4.1
dev-util/cmake:           2.8.12.2-r1
dev-util/pkgconfig:       0.28-r1
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.12.4
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.11.6, 1.13.4
sys-devel/binutils:       2.23.2
sys-devel/gcc:            4.7.3-r1, 4.8.3
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4.2-r1
sys-devel/make:           4.0-r1
sys-kernel/linux-headers: 3.13 (virtual/os-headers)
sys-libs/glibc:           2.19-r1
Repositories: gentoo esm-g-cpan
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gentoo.edsantiago.com/ http://mirror.iawnet.sandia.gov/gentoo/ http://mirror.usu.edu/mirrors/gentoo/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j9"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/opt/gentoo/gcpan-overlay"
SYNC="rsync://gentoo.edsantiago.com/gentoo-portage"
USE="X acl alsa amd64 bash-completion berkdb bzip2 cli cracklib crypt cups curl cxx dbus dri emacs exif fontconfig gdbm hardened iconv icu ipv6 jpeg jpeg2k justify mmx modules mp3 multilib ncurses nls nptl opengl openmp pam pax_kernel pcre pdf perl png postscript pulseaudio readline session smp sse sse2 ssl systemd tcpd tiff truetype unicode urandom vorbis xattr xft xmp xtpax zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_US" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Comment 1 Johan Lidström 2015-02-15 12:23:06 UTC
The same thing happens for me on x86 hardened-selinux, from ultravnc win32 client.

Only happens when there are large changes happening (whether due to input or screen I do not know), otherwise it can run for days without problems.

[I] x11-misc/x11vnc
     Available versions:  0.9.13 0.9.13-r1 {avahi crypt fbcon +jpeg ssl system-libvncserver threads tk xinerama +zlib}
     Installed versions:  0.9.13-r1(17:11:36 2014-09-13)(avahi crypt fbcon jpeg ssl threads tk xinerama zlib -system-libvncserver)

Portage 2.2.14 (python 2.7.9-final-0, hardened/linux/amd64/selinux, gcc-4.8.4, glibc-2.19-r1, 3.15.5-hardened-r1 x86_64)
=================================================================
System uname: Linux-3.15.5-hardened-r1-x86_64-Intel-R-_Core-TM-2_Duo_CPU_T6670_@_2.20GHz-with-gentoo-2.2
KiB Mem:     3961576 total,    125328 free
KiB Swap:    2097148 total,   1787968 free
Timestamp of tree: Tue, 10 Feb 2015 23:30:02 +0000
ld GNU ld (Gentoo 2.24 p1.4) 2.24
app-shells/bash:          4.2_p53
dev-java/java-config:     2.2.0
dev-lang/perl:            5.20.1-r4
dev-lang/python:          2.7.9-r1, 3.3.5-r1, 3.4.2
dev-util/cmake:           2.8.12.2-r1
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.13.9
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.4_p6-r1, 1.11.6-r1, 1.12.6, 1.13.4
sys-devel/binutils:       2.24-r3
sys-devel/gcc:            4.7.4, 4.8.4
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4.4
sys-devel/make:           4.0-r1
sys-kernel/linux-headers: 3.16 (virtual/os-headers)
sys-libs/glibc:           2.19-r1
Repositories: gentoo gamerlay drdim gbin sugar vayerx pentoo
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA Oracle-BCLA-JavaSE PUEL AdobeFlash-11.x"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe -fPIC -fomit-frame-pointer -maccumulate-outgoing-args"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.5/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -march=native -pipe -fPIC -fomit-frame-pointer -maccumulate-outgoing-args -fpermissive"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox selinux sesandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://ftp.sunet.se/pub/Linux/distributions/gentoo/"
LANG="en_GB.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_COMPRESS="xz"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/gamerlay /var/lib/layman/drdim /var/lib/layman/gbin /var/lib/layman/sugar /var/lib/layman/vayerx /var/lib/layman/pentoo"
USE="X a52 aac aacplus acl adns afp alsa amd64 amr ao apache2 atm autotrace avahi berkdb bittorrent bluetooth bluray branding bsf bzip2 cairo caps cdda cdio cdparanoia cdr cjk cli cloudprint colord consolekit cracklib crypt cscope cups curl cvs cxx daap dbus device-mapper dga directfb djvu dri dts dv dvd dvdnav eds encode ewf exif expat faac fam fbcon ffmpeg fftw flac fontconfig foomaticdb fpx freetds ftp fuse gcj gconf gcrypt gdbm gif gimp git glamor gmp gnome gnome-keyring gnutls gphoto2 gpm gsm gssapi gstreamer gtk gtk3 gudev hardened hdri highlight http iconv icu idn ieee1394 image imagemagick imlib ipv6 irda java jbig jpeg jpeg2k json justify kerberos lame latex lcms ldap libass libav libedit libnotify libsamplerate libsecret lua luajit lzma lzo mad metalink mmap mms mmx mmxext mng modemmanager modplug modules mp3 mpeg mssql mtp multilib multitarget natspec ncurses netlink nettle networkmanager nls nptl nss nut offensive ogg open_perms openal openexr opengl openmp opus osmesa pam pango pax_kernel pcre pdf perl pic pkcs11 playlist plotutils png pnm policykit postgres postproc postscript ppds pulseaudio python quicktime radius rar raw readline resolvconf rle rtmp ruby samba scanner schroedinger scsi sdl selinux session sftp slp smartcard smp sndfile socks5 speex sqlite sse sse2 sse4_1 ssl ssse3 startup-notification subversion svg syslog system-libvpx szip taglib tcpd tga theora threads tiff tk truetype twolame udev udisks unconfined unicode upnp upnp-av upower urandom usb usbredir v4l vaapi vcd vdpau vim-syntax vnc vorbis vpx wavpack wayland webp wifi wmf wxwidgets x264 x265 xanim xattr xcb xcomposite xdg xft xinerama xinetd xml xmlrpc xmp xpm xscreensaver xtpax xv xvid xvmc yaml zeroconf zlib" ABI_X86="32 64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="*" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx sse sse2" CURL_SSL="nss" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="evdev keyboard mouse synaptics wacom" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_GB sv sv_SE" NETBEANS_MODULES="apisupport java javafx profiler websvccommon cnd dlight enterprise extide groovy mobility php webcommon" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3 python3_4" QEMU_SOFTMMU_TARGETS="x86_64 alpha arm i386 m68k ppc ppc64 sparc sparc64" QEMU_USER_TARGETS="alpha" RUBY_TARGETS="ruby19 ruby20 ruby21 ruby22" USERLAND="GNU" VIDEO_CARDS="intel i915 i965 sisusb" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="2.7 3.3"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC
Comment 2 Magnus Granberg gentoo-dev 2015-07-04 14:26:43 UTC
Do this still happens on newer versions?
Comment 3 Ed Santiago 2015-07-08 10:05:47 UTC
(In reply to Magnus Granberg from comment #2)
> Do this still happens on newer versions?

Latest I have is 0.9.13_p20150111 and yes, it still happens repeatedly.
Comment 4 Alex Xu (Hello71) 2015-07-08 11:46:13 UTC
(In reply to Ed Santiago from comment #3)
> (In reply to Magnus Granberg from comment #2)
> > Do this still happens on newer versions?
> 
> Latest I have is 0.9.13_p20150111 and yes, it still happens repeatedly.

Can you get a backtrace?
Comment 5 Ed Santiago 2015-07-08 13:26:20 UTC
(In reply to Alex Xu (Hello71) from comment #4)
> Can you get a backtrace?

It doesn't look like it. No core. Running under gdb, I see:

    Program terminated with signal SIGKILL, Killed.
    The program no longer exists.
    (gdb) bt
    No stack.

(None of this is surprising given SIGKILL). Nothing useful in systemd journal.

There might be something I've missed. Any suggestions? E.g. any way to temporarily bypass the stack-smashing check (without re-emerging glibc)? Any logs I haven't found?
Comment 6 Alex Xu (Hello71) 2015-07-08 13:41:46 UTC
(In reply to Ed Santiago from comment #5)
> (In reply to Alex Xu (Hello71) from comment #4)
> > Can you get a backtrace?
> 
> It doesn't look like it. No core. Running under gdb, I see:
> 
>     Program terminated with signal SIGKILL, Killed.
>     The program no longer exists.
>     (gdb) bt
>     No stack.
> 
> (None of this is surprising given SIGKILL). Nothing useful in systemd
> journal.
> 
> There might be something I've missed. Any suggestions? E.g. any way to
> temporarily bypass the stack-smashing check (without re-emerging glibc)? Any
> logs I haven't found?

use valgrind. and also make sure you actually compiled with debug symbols.
Comment 7 Ed Santiago 2015-07-08 17:03:32 UTC
Will do, but Sunday at the earliest: I'm out of town, and I'd prefer not to rebuild glibc with USE=debug over ssh.
Comment 8 Ed Santiago 2015-07-12 01:30:37 UTC
Is this helpful?

==31905== Syscall param write(buf) points to uninitialised byte(s)
==31905==    at 0x70FCDC0: __write_nocancel (in /lib64/libc-2.20.so)
==31905==    by 0x4E4CFD8: rfbWriteExact (in /usr/lib64/libvncserver.so.0.0.0)
==31905==    by 0x4E4A40F: rfbSendServerCutText (in /usr/lib64/libvncserver.so.0
.0.0)
==31905==    by 0x17A689: ??? (in /usr/bin/x11vnc)
==31905==    by 0x1C25F8: ??? (in /usr/bin/x11vnc)
==31905==    by 0x1782F5: ??? (in /usr/bin/x11vnc)
==31905==    by 0x11A177: ??? (in /usr/bin/x11vnc)
==31905==    by 0x7036EAA: (below main) (in /lib64/libc-2.20.so)
==31905==  Address 0xffefe6b41 is on thread 1's stack
==31905==  in frame #2, created by rfbSendServerCutText (???)
==31905== 
11/07/2015 19:23:05 XDAMAGE is not working well... misses: 57/228
11/07/2015 19:23:05 Maybe an OpenGL app like Beryl or Compiz is the problem?
11/07/2015 19:23:05 Use x11vnc -noxdamage or disable the Beryl/Compiz app.
11/07/2015 19:23:05 To disable this check and warning specify -xdamage twice.
11/07/2015 19:23:34 collect_xdamage: too many xdamage events 1000+2196
11/07/2015 19:23:58 added missing keysym to X display: 093 0x100001a "null"
^Zbg

==31905== Invalid read of size 4
==31905==    at 0x1C6547: ??? (in /usr/bin/x11vnc)
==31905==    by 0x1C73A2: ??? (in /usr/bin/x11vnc)
==31905==    by 0x593CBA8: ??? (in /usr/lib64/libXtst.so.6.1.0)
==31905==    by 0x593D13D: ??? (in /usr/lib64/libXtst.so.6.1.0)
==31905==    by 0x67C44E4: ??? (in /usr/lib64/libX11.so.6.3.0)
==31905==    by 0x67C4FEC: _XEventsQueued (in /usr/lib64/libX11.so.6.3.0)
==31905==    by 0x67B47D4: XPending (in /usr/lib64/libX11.so.6.3.0)
==31905==    by 0x1A33B9: ??? (in /usr/bin/x11vnc)
==31905==    by 0x1AECDC: ??? (in /usr/bin/x11vnc)
==31905==    by 0x178B7D: ??? (in /usr/bin/x11vnc)
==31905==    by 0x11A177: ??? (in /usr/bin/x11vnc)
==31905==    by 0x7036EAA: (below main) (in /lib64/libc-2.20.so)
==31905==  Address 0x10907a6c is 0 bytes after a block of size 28 alloc'd
==31905==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd
64-linux.so)
==31905==    by 0x593CF96: ??? (in /usr/lib64/libXtst.so.6.1.0)
==31905==    by 0x593D0BD: ??? (in /usr/lib64/libXtst.so.6.1.0)
==31905==    by 0x67C44E4: ??? (in /usr/lib64/libX11.so.6.3.0)
==31905==    by 0x67C4FEC: _XEventsQueued (in /usr/lib64/libX11.so.6.3.0)
==31905==    by 0x67B47D4: XPending (in /usr/lib64/libX11.so.6.3.0)
==31905==    by 0x1A3029: ??? (in /usr/bin/x11vnc)
==31905==    by 0x1AECDC: ??? (in /usr/bin/x11vnc)
==31905==    by 0x178B7D: ??? (in /usr/bin/x11vnc)
==31905==    by 0x11A177: ??? (in /usr/bin/x11vnc)
==31905==    by 0x7036EAA: (below main) (in /lib64/libc-2.20.so)
==31905== 
*** stack smashing detected ***: x11vnc terminated; report to <http://bugs.gento
o.org/>
==31905== 
==31905== HEAP SUMMARY:
==31905==     in use at exit: 29,445,127 bytes in 5,095 blocks
==31905==   total heap usage: 4,036,650 allocs, 4,031,555 frees, 406,670,409 byt
es allocated
==31905== 
==31905== LEAK SUMMARY:
==31905==    definitely lost: 7,870 bytes in 23 blocks
==31905==    indirectly lost: 72,762 bytes in 24 blocks
==31905==      possibly lost: 0 bytes in 0 blocks
==31905==    still reachable: 29,364,495 bytes in 5,048 blocks
==31905==         suppressed: 0 bytes in 0 blocks
==31905== Rerun with --leak-check=full to see details of leaked memory
==31905== 
==31905== For counts of detected and suppressed errors, rerun with: -v
==31905== Use --track-origins=yes to see where uninitialised values come from
==31905== ERROR SUMMARY: 15 errors from 2 contexts (suppressed: 1 from 1)
Comment 9 Magnus Granberg gentoo-dev 2015-07-13 08:58:54 UTC
rebuild x11vnc with debug symbols and glibc with debug on.
Comment 10 Ed Santiago 2015-07-13 12:01:08 UTC
Created attachment 406648 [details]
valgrind --log-file=/tmp/valgrind-x11vnc.log x11vnc
Comment 11 Magnus Granberg gentoo-dev 2015-07-13 12:38:42 UTC
check if you get the needed function in the stack smashing detected error
or a core file.
Comment 12 Ed Santiago 2015-07-13 12:52:39 UTC
(In reply to Magnus Granberg from comment #11)
> check if you get the needed function in the stack smashing detected error
> or a core file.

I'm really sorry, I don't know what that means. Could you please clarify?
Comment 13 Magnus Granberg gentoo-dev 2015-07-13 13:11:18 UTC
(In reply to Ed Santiago from comment #12)
> (In reply to Magnus Granberg from comment #11)
> > check if you get the needed function in the stack smashing detected error
> > or a core file.
> 
> I'm really sorry, I don't know what that means. Could you please clarify?
In the output of
*** stack smashing detected ***: x11vnc - terminated
   x11vnc: stack smashing attack in function <unknown> - terminated

You should get the function that the smashing ocure at
if you build glibc with debug on or edit the glibc ebuild and remove the
cp "${FILESDIR}"/2.18/glibc-2.18-gentoo-stack_chk_fail.c debug/stack_chk_fail.c || die
		cp "${FILESDIR}"/2.18/glibc-2.18-gentoo-chk_fail.c debug/chk_fail.c || die
lines and rebuild glibc
Comment 14 Ed Santiago 2015-07-13 14:57:12 UTC
Created attachment 406666 [details]
x11vnc backtrace

The stack-smashing message no longer includes "in function <anything>"; it's just:

  *** stack smashing detected ***: x11vnc terminated

I've edited the ebuild and rebuilt glibc, with FEATURES=nostrip, and now get a backtrace (attached). Still no core dump.

x11vnc is built with FEATURES=splitdebug; I don't know why the backtrace does not show symbols.
Comment 15 Magnus Granberg gentoo-dev 2015-07-13 15:46:15 UTC
just build it with nostrip instead(In reply to Ed Santiago from comment #14)
> Created attachment 406666 [details]
> x11vnc backtrace
> 
> The stack-smashing message no longer includes "in function <anything>"; it's
> just:
> 
>   *** stack smashing detected ***: x11vnc terminated
> 
> I've edited the ebuild and rebuilt glibc, with FEATURES=nostrip, and now get
> a backtrace (attached). Still no core dump.
> 
> x11vnc is built with FEATURES=splitdebug; I don't know why the backtrace
> does not show symbols.
just build it with nostrip instead
Comment 16 Ed Santiago 2015-07-13 16:17:51 UTC
(In reply to Magnus Granberg from comment #15)
> just build it with nostrip instead

No difference.

One factor I didn't mention in my initial report is that host d (the one on which I'm running x11vnc) has a 3200x1800 display. Could there be hardcoded limits somewhere, that this is exceeding? (Seems unlikely. I've run x11vnc with -scale and -clip, and still get the stack-smashing error).
Comment 18 Ed Santiago 2015-07-13 18:47:16 UTC
Created attachment 406698 [details]
x11vnc-backtrace.log (gdb)

Thanks. gdb backtrace attached. Fingers crossed that that will help...
Comment 19 kielhorn.martin 2015-08-04 10:07:37 UTC
According to your backtrace line:
#12 0x00005555555ef3ba in check_xrecord_mouse () at userinput.c:2930
the bug is related to the following code:

 2927                X_LOCK;
 2928#if HAVE_RECORD
 2929                SCR_LOCK;
 2930                XRecordProcessReplies(rdpy_data);
 2931                SCR_UNLOCK;
 2932#endif
 2933                X_UNLOCK;


I therefore tried to start x11vnc with this option:
x11vnc -noxrecord

This seems to prevent the stack smashing.

Regards, Martin
Comment 20 Ed Santiago 2015-08-04 15:41:08 UTC
> x11vnc -noxrecord [...] seems to prevent the stack smashing.

Thank you! Semi-confirmed: I can't reproduce the crash with Xdialog, but I did get a "*** buffer overflow detected ***" while trying other things:

   /usr/lib64/libvncserver.so.0(rfbCheckFds+0x57a)[0x7fdf2866c36a]

Haven't managed to reproduce it, but I may not try too hard: I've moved away from trying to use x11vnc. Thank you again for your perseverance on this issue.
Comment 21 Arackhaen 2015-09-20 17:10:49 UTC
This bug is still valid.
It has been for a while on one of my servers and now same thing on new server.
--- Clip! ---
20/09/2015 19:58:35 called initialize_xfixes()
*** stack smashing detected ***: x11vnc terminated; report to <http://bugs.gentoo.org/>
x11vnc-some-startup-script.sh line 41:  7424 Killed                  x11vnc -rfbauth file_somewhere -rfbport port -allow ${VNC_ALLOWED_LIST} -display :0 -auth /var/run/xauth/${XA_FILE} -forever -bg
--- Clip! ---

I found following links regarding this x11vnc stack smashing:
https://bugzilla.redhat.com/show_bug.cgi?id=972618
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735648
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746260
They state that using "non-system" libvncserver (or upgraded one) fixes this. I hope I can try this now.

I haven't done much debugging and with these servers I even cant do it after today, but probably I can test it in laptop environment if needed.
Comment 22 Alex Xu (Hello71) 2015-09-20 22:55:42 UTC
(In reply to Arackhaen from comment #21)
> This bug is still valid.
> It has been for a while on one of my servers and now same thing on new
> server.
> --- Clip! ---
> 20/09/2015 19:58:35 called initialize_xfixes()
> *** stack smashing detected ***: x11vnc terminated; report to
> <http://bugs.gentoo.org/>
> x11vnc-some-startup-script.sh line 41:  7424 Killed                  x11vnc
> -rfbauth file_somewhere -rfbport port -allow ${VNC_ALLOWED_LIST} -display :0
> -auth /var/run/xauth/${XA_FILE} -forever -bg
> --- Clip! ---
> 
> I found following links regarding this x11vnc stack smashing:
> https://bugzilla.redhat.com/show_bug.cgi?id=972618
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735648
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=746260
> They state that using "non-system" libvncserver (or upgraded one) fixes
> this. I hope I can try this now.
> 
> I haven't done much debugging and with these servers I even cant do it after
> today, but probably I can test it in laptop environment if needed.

libvncserver-0.9.10 is now stable. all reporters, please try again with newest ~libvncserver-0.9.10 and ~x11vnc-0.9.13_p20150627 and reopen if problem persists. please also test with and without -noxrecord. thanks.
Comment 23 Arackhaen 2015-09-22 14:23:36 UTC
Sorry about novice question, but why I am not getting version you mentioned (only getting upgrade to release1 of old date (x11vnc-0.9.13_p20150211-r1)?
--- Clip! ---
 x11-misc/x11vnc-0.9.13_p20150111-r1::gentoo [0.9.13_p20150111::gentoo]--- Clip! ---

I added x11vnc to accepted keywords and did emerge sync. Is there still something I need to do to get that version you are mentioning?
That r1 version failed to build with error.
Comment 24 Arackhaen 2015-10-02 13:15:22 UTC
(In reply to Arackhaen from comment #23)
> Sorry about novice question, but why I am not getting version you mentioned
> (only getting upgrade to release1 of old date (x11vnc-0.9.13_p20150211-r1)?
> --- Clip! ---
>  x11-misc/x11vnc-0.9.13_p20150111-r1::gentoo [0.9.13_p20150111::gentoo]---
> Clip! ---
> 
> I added x11vnc to accepted keywords and did emerge sync. Is there still
> something I need to do to get that version you are mentioning?
> That r1 version failed to build with error.

I couldn't install version you mentioned ("emerge: there are no ebuilds to satisfy "=x11-misc/x11vnc-0.9.13_p20150627"."), but I haven't seen any x11vnc crashes on the servers for a while.
For now it seems to work, but first rebuilds and restarts of service clearly didn't make fix even with correct libvncserver version.
Seems to be fixed - thanks!
Comment 25 Alex Xu (Hello71) 2016-07-19 20:51:20 UTC
*** Bug 574060 has been marked as a duplicate of this bug. ***
Comment 26 Alex Xu (Hello71) 2016-07-19 20:52:54 UTC
reopening temporarily; I may be able to repro.
Comment 27 SpanKY gentoo-dev 2016-11-27 20:53:12 UTC
it's not specific to hardened.  it crashes frequently on my non-hardened system.
Comment 28 SpanKY gentoo-dev 2016-11-27 21:36:47 UTC
0.9.14 crashes in the same way
Comment 29 SpanKY gentoo-dev 2016-11-27 21:37:10 UTC
Created attachment 454568 [details]
x11vnc-backtrace.log (0.9.14)
Comment 30 Alex Xu (Hello71) 2016-11-28 01:30:45 UTC
as I said in bug 591528, the issue was fixed upstream.
Comment 31 SpanKY gentoo-dev 2016-11-28 04:03:39 UTC
(In reply to Alex Xu (Hello71) from comment #30)

ok, but that isn't this bug ;)

if it's a simple fix, we can roll it into 0.9.14-r2 pretty easily
Comment 32 Pacho Ramos gentoo-dev 2016-12-05 16:08:25 UTC
[master a2dee49] x11-misc/x11vnc: Version bump by Alex Xu (Hello71) fixing bugs #526796, #584788 and others.
 2 files changed, 61 insertions(+)
 create mode 100644 x11-misc/x11vnc/x11vnc-0.9.14_p20161013.ebuild