Bug 272361 – x11-base/xorg-server-1.5.3-r6 fails to fork while invoking xkbcomp

• Product: Gentoo Linux
• Status: RESOLVED OBSOLETE

[Description Attila Tóth 2009-06-03 05:16:44 UTC]

I've experienced recently, that my keyboard doesn't work as intended: the layout is incorrect and the compose key is not functional. I've found two relevant log entries possibly related to the problem:
Xorg.log:
(**) Option "xkb_rules" "evdev"
(**) Option "xkb_model" "evdev"
(**) Option "xkb_layout" "hu,us"
(**) Option "xkb_variant" "standard"
(**) Option "xkb_options" "grp:shift_toggle,compose:ralt"
(EE) XKB: Could not invoke xkbcomp
(EE) XKB: Couldn't compile keymap
(WW) Couldn't load XKB keymap, falling back to pre-XKB keymap

grsec.log:
grsec: (root:U:/usr/bin/Xorg) failed fork with errno -12 by /usr/bin/Xorg[X:pid] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/gdm-binary[gdm:pid] uid/euid:0/0 gid/egid:0/0

xorg-server-1.5.3-r6, xkbcomp-1.0.5, setxkbmap-1.0.4

I know, that I'm using a hardened experimental system, but the new evdev based keyboard interface was previously working well for the past few weeks before this issue popped up.

Any ideas would be greatly appreciated.

Portage 2.1.6.11 (hardened/x86, gcc-4.3.3, glibc-2.8_p20080602-r3, 2.6.29-hardened i686)
=================================================================
System uname: Linux-2.6.29-hardened-i686-AMD_Athlon-TM-_MP_2600+-with-glibc2.4
Timestamp of tree: Fri, 29 May 2009 11:00:01 +0000
ccache version 2.4 [disabled]
app-shells/bash:     3.2_p39
dev-java/java-config: 1.3.7-r1, 2.1.7
dev-lang/python:     2.5.4-r2
dev-python/pycrypto: 2.0.1-r8
dev-util/ccache:     2.4-r7
dev-util/cmake:      2.6.4
sys-apps/baselayout: 1.12.11.1
sys-apps/sandbox:    1.6-r2
sys-devel/autoconf:  2.13, 2.63
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2
sys-devel/binutils:  2.19.1-r1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   1.5.26
virtual/os-headers:  2.6.28-r1
ACCEPT_KEYWORDS="x86"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=i686 -mtune=athlon-mp -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib/mozilla/defaults/pref /var/bind /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /etc/udev/rules.d"
CXXFLAGS="-O2 -march=i686 -mtune=athlon-mp -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs autoconfig distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch"
GENTOO_MIRRORS="http://gentoo.inode.at/ http://gentoo.inf.elte.hu/"
LANG="hu_HU.utf8"
LC_ALL="hu_HU.utf8"
LDFLAGS=""
LINGUAS="hu en"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage /usr/portage/local/layman/xake-toolchain /home/atoth/public_html/overlay"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext 7zip GNU X X509 Xaw3d a52 aac aalib acl acpi aiglx alsa amr amrnb amrwb aotuv apache2 applet ares asf atmo audiofile bash-completion bcmath bdf berkdb bidi binfilter bitmap-fonts bittorrent blas bluetooth boost branding browserplugin bzip2 cairo caps cdda cddax cddb cdio cdparanoia cdr cdrom cgi chardet checkpath clamdtop cli consolekit contentcache context contrib cracklib crypt css ctype cups curl curlwrappers dba dbm dbus dcmtk deskbar detex devhelp device-mapper dga dhcp dia dicom dirac disassembler discard-path divx divx4linux djbfft djvu dlloader dmi dri dts dv dvd dvdnav dvdr dvdread dvi dvi2tty dvipdfm eds elf emerald enca encode enscript epiphany epoll evo exif expat extensions extra extrafilters extras faac faad fam fame ffmpeg fftw finger firefox flac flash flatfile follow-xff fontconfig foomaticdb force-cgi-redirect fortran fpx ftp gadu galago gd gdb gdl gedit gif gimp gimpprint ginac git glade glibc glitz gmedia gmp gnet gnome gnome-keyring gnome-print gnomecanvas gnomecd gopher gpac gphoto2 gpm graphics graphviz gs gsf gsl gsm gstreamer gtk gtk2 gtkhtml guile h323 hal hardened hub iconv icu id3 id3tag idea idn iksemel imagemagick imap imlib inherit-graph inifile inkjar inode iplsrc irc irda jabber jadetex java java-internal java6 javascript jingle jpeg jpeg2k jpgraph jrtplib json kate kpathsea ladspa lame lapack latex latex3 lcms libass libburn libcaca libnotify libplot libsamplerate libssh2 libv4l2 lm_sensors logitech-mouse loop-aes lyx lzma lzo lzw m17n-lib mad matroska mbox mcal md5sum memlimit mhash midi mikmod milter ming mjpeg mmap mmx mmxext mng mode-owner motif mozbranding mozcalendar mp2 mp3 mp4 mpeg mpeg1 mpeg2 mplayer mudflap musepack mysql mysqli nautilus ncurses nemesi network networking new-login nifti nls nopop3d nptl nptlonly nsplugin nss ntfs nuv oav odbc odk ofx ogg oggvorbis ogm oil onaccess openexr opengl optimisememory otr overload pam pam_chroot pam_timestamp pango paste64 pasteafter pccts pcmcia pcntl pcre pda pdf pear perl php pic plotutils png pnm posix postscript ppds pstricks publishers pvr python qhull quicktime quotas rar rc5 readline realmedia reflection reiserfs remoteosd replytolist rle rtc samba sasl scanner scenarios schroedinger science screen sdl sendmail sensord session sftplogging sid sieve silc skins slang smp sms sndfile soap sockets sound soundex soundtouch sourceview sox speex spell spf srt sse ssl ssp sspall startup-notification subtitles subversion suhosin svg svnserve syslog sysvipc t1lib taglib tagwriting tcl tcltk tcpd templates tetex tex4ht tga themes theora threads threadsafe tidy tiff tilepath tk tlen tokenizer toolbar tools totem tracker transcode truetype truetype-fonts twolame type1-fonts type3 udev underscores unicode urandom usb userlocales utils v4l v4l2 valgrind vcd vcdinfo vcdx vidix virus-scan vista visualization vlm volpack vorbis vtk wavplay wifi win32codecs wma wmf wmp wxwindows x264 x86 xattr xcb xetex xforms xine xmedcon xml xml2 xmlreader xmlrpc xmlwriter xorg xpm xsl xulrunner xv xvid xvmc yahoo zip zlib zvbi" ALSA_CARDS="cmipci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias auth_digest version filter ident charset_lite asis dbd authn_dbd proxy proxy_ajp proxy_balancer proxy_connect proxy_http imagemap" APACHE2_MPMS="worker" CAMERAS="ptp2" ELIBC="glibc" INPUT_DEVICES="keyboard mouse acecad evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="hu en" USERLAND="GNU" VIDEO_CARDS="radeon v4l"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Reproducible: Always

Steps to Reproduce:

[Comment 1 Attila Tóth 2009-06-05 14:10:31 UTC]

The issue might be related to PaX/Grsec. It popped only on the Athlon MP server, while no sigs of fork failures on my Pentium M laptop. The toolchain is the same. 

I'll get back upon any news on this one.

BTW hardened-sources-2.6.28-r8 failed to load on that Athlon MP, while the laptop was doing well. When I was about to take a screen capture of the remote console, hardened-sources-2.6.29 came out and the kernel loaded well. There were no problems except for the fork failure.

[Comment 2 Rémi Cardona (RETIRED) 2009-06-08 08:33:17 UTC]

I have no idea what this all means. I'm not sure there's anything we can do here. I'll gladly admit that X has a natural tendency of breaking in random and unpredictable ways but xkbcomp's usage in X hasn't changed for years...

Do let us know how things evolve.

Thanks

[Comment 3 Attila Tóth 2009-06-21 15:10:06 UTC]

(In reply to comment #2)
> I have no idea what this all means. I'm not sure there's anything we can do
> here. I'll gladly admit that X has a natural tendency of breaking in random and
> unpredictable ways but xkbcomp's usage in X hasn't changed for years...
> 
> Do let us know how things evolve.
> 
> Thanks
> 

It seems to be a wrong combination of grsecurity/PaX, architecture (Athlon MP) and kernel PAT support.
It turned out, that disabling PAT support workarounds the issue.

Regards,
Dw.