Bug 435848 – x11-misc/x11vnc-0.9.13 with net-libs/libvncserver-0.9.9 - buffer overflow detected in /usr/lib/libvncserver.so.0

• Status: RESOLVED INVALID
• Product: Gentoo Linux

[Description Melendro 2012-09-22 12:24:48 UTC]

After updating libvncserver to version 0.9.9, x11vnc (0.9.13) dies when a client connects. The client I'm using is krdc version 4.8.5 (using libvncserver version 0.9.8.2) in a kubuntu box.

The backtrace in the x11vnc log file is:

*** buffer overflow detected ***: /usr/bin/x11vnc terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7ff4cb71ad37]
/lib64/libc.so.6(+0xfbba0)[0x7ff4cb718ba0]
/lib64/libc.so.6(+0xfdcee)[0x7ff4cb71acee]
/usr/lib/libvncserver.so.0(rfbProcessNewConnection+0xf9)[0x7ff4cd5fc099]
/usr/lib/libvncserver.so.0(rfbCheckFds+0x438)[0x7ff4cd5fc5e8]
/usr/lib/libvncserver.so.0(rfbProcessEvents+0x1e)[0x7ff4cd5f382e]
/usr/bin/x11vnc[0x4922ea]
/usr/bin/x11vnc[0x4565a1]
/usr/bin/x11vnc[0x49a961]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7ff4cb63f52d]
/usr/bin/x11vnc[0x40b639]
caught signal: 6

Attahed you can find the memory map in the x11vnc log file and my emerge --info.

[Comment 1 Melendro 2012-09-22 12:25:48 UTC]

Created attachment 324614 [details]
Memory map

[Comment 2 Melendro 2012-09-22 12:26:59 UTC]

Created attachment 324616 [details]
emerge --info

[Comment 3 Melendro 2012-09-22 12:30:38 UTC]

Soory, I forgot to say that the problem doesn't appear using embedded libvncserver (USE=-system-libvncserver).

[Comment 4 Wadlax 2012-12-10 06:51:55 UTC]

I want to confirm this issue :


x11-misc/x11vnc-0.9.13-r1   with USE +system-libvncserver
net-misc/tigervnc-1.2.0


starting x11vnc in first console , connecting from 2nd , using 
vncviewer 127.0.0.1



1) net-libs/libvncserver-0.9.8.2-r2
+++ screen shows up


2) net-libs/libvncserver-0.9.9-r1

--- Error (on x11vnc console)

*** buffer overflow detected ***: x11vnc terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f3224b69dd7]
/lib64/libc.so.6(+0xfdbe0)[0x7f3224b67be0]
/lib64/libc.so.6(+0xffd8e)[0x7f3224b69d8e]
/usr/lib/libvncserver.so.0(rfbProcessNewConnection+0xf7)[0x7f3226a43817]
/usr/lib/libvncserver.so.0(rfbCheckFds+0x3b8)[0x7f3226a43cd8]
/usr/lib/libvncserver.so.0(rfbProcessEvents+0x1c)[0x7f3226a3b2dc]
x11vnc[0x4a1cdf]
x11vnc[0x4648a7]
x11vnc[0x411caa]
/lib64/libc.so.6(__libc_start_main+0xed)[0x7f3224a8e60d]
x11vnc[0x41c66d]
--- Error (on x11vnc console)


--- On vncviewer console 
 CConn:       connected to host 127.0.0.1 port 5900
 CConn:       read: Соединение сброшено другой
              стороной (104)
--- On vncviewer console 


3) x11-misc/x11vnc-0.9.13-r1   with USE  -system-libvncserver
+++ screen shows up

[Comment 5 Wadlax 2012-12-10 06:59:43 UTC]

I think , that it is upstream issue.

I have opened a bug (that is closed now)
https://bugs.gentoo.org/show_bug.cgi?id=431728

Headless  app-emulation/virtualbox (with net-libs/libvncserver-0.9.9 )
crashes when connected via vnc


It was closed with 
"Pin libvncserver down to versions lower than 0.9.9 to prevent crashes in virtualbox"


Now virtualbox-4.2.4-r1.ebuild  depends on
  vnc? ( >=net-libs/libvncserver-0.9.9 )


I will also try to test bug with virtualbox later.

And sorry, i dont know how to report current bug to upstream.

[Comment 6 C. Wijtmans 2013-11-05 12:20:37 UTC]

i can confirm this issue and thus the system-libvnc useflag should be hard disabled

[Comment 7 Alex Xu (Hello71) 2015-04-04 20:53:05 UTC]

only latest stable version of packages is supported.