I'm using NVIDIA's proprietary drivers for my graphics card. I cannot run any executables in parallel that rely on hardware acceleration using mpirun. For example, glxgears works fine if I run it from the shell, but if I run it through mpirun: #$> mpirun -n 2 /usr/bin/glxgears libGL error: No matching fbConfigs or visuals found libGL error: failed to load driver: swrast X Error of failed request: BadValue (integer parameter out of range for operation) Major opcode of failed request: 153 (GLX) Minor opcode of failed request: 3 (X_GLXCreateContext) Value in failed request: 0x0 Serial number of failed request: 35 Current serial number in output stream: 37 When called through mpirun, the wrong OpenGL libraries are loaded as can be seen by using ldd: Without mpirun, NVIDIA's implementation is used: #$> ldd /usr/bin/glxgears | grep GL libGL.so.1 => /usr/lib64/opengl/nvidia/lib/libGL.so.1 (0x00007fe1c3721000) libGLX.so.0 => /usr/lib64/opengl/nvidia/lib/libGLX.so.0 (0x00007fe1c2920000) libGLdispatch.so.0 => /usr/lib64/opengl/nvidia/lib/libGLdispatch.so.0 (0x00007fe1c2637000) With mpirun, MESA's implementation is used: #$> mpirun -n 2 ldd /usr/bin/glxgears | grep GL libGL.so.1 => /usr/lib/libGL.so.1 (0x00007f0c69156000) Temporarily moving /usr/libGL.so* somewhere else will make it work correctly. So the problem might simply be the ordering of the search directories for the libraries? glxgears is simply an example, the same problem crashes ParaView's pvserver when I run it in parallel.
#$> emerge --info: Portage 2.2.28 (python 2.7.10-final-0, default/linux/amd64/13.0/desktop/plasma, gcc-4.9.3, glibc-2.22-r4, 4.6.4-gentoo x86_64) ================================================================= System uname: Linux-4.6.4-gentoo-x86_64-Intel-R-_Core-TM-_i7-3930K_CPU_@_3.20GHz-with-gentoo-2.2 KiB Mem: 16356712 total, 13725860 free KiB Swap: 15527340 total, 15527340 free Timestamp of repository gentoo: Sat, 23 Jul 2016 16:15:01 +0000 sh bash 4.3_p42-r1 ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1 app-shells/bash: 4.3_p42-r1::gentoo dev-java/java-config: 2.2.0-r3::gentoo dev-lang/perl: 5.20.2::gentoo dev-lang/python: 2.7.10-r1::gentoo, 3.3.5-r3::gentoo, 3.4.3-r1::gentoo dev-util/cmake: 3.3.1-r1::gentoo dev-util/pkgconfig: 0.28-r2::gentoo sys-apps/baselayout: 2.2::gentoo sys-apps/openrc: 0.19.1::gentoo sys-apps/sandbox: 2.10-r1::gentoo sys-devel/autoconf: 2.13::gentoo, 2.69::gentoo sys-devel/automake: 1.11.6-r1::gentoo, 1.14.1::gentoo, 1.15::gentoo sys-devel/binutils: 2.25.1-r1::gentoo sys-devel/gcc: 4.9.3::gentoo sys-devel/gcc-config: 1.7.3::gentoo sys-devel/libtool: 2.4.6::gentoo sys-devel/make: 4.1-r1::gentoo sys-kernel/linux-headers: 4.3::gentoo (virtual/os-headers) sys-libs/glibc: 2.22-r4::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.us.gentoo.org/gentoo-portage priority: -1000 x-portage location: /usr/local/portage masters: gentoo priority: 0 science location: /var/lib/layman/science sync-type: laymansync sync-uri: git://anongit.gentoo.org/proj/sci.git masters: gentoo priority: 50 steam-overlay location: /var/lib/layman/steam-overlay sync-type: laymansync sync-uri: git://github.com/anyc/steam-overlay.git masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-march=native -O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-march=native -O2 -pipe" GENTOO_MIRRORS="http://gentoo.mirrors.hoobly.com/ ftp://gentoo.mirrors.tds.net/gentoo http://gentoo.mirrors.tds.net/gentoo" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j6" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl acpi alsa amd64 avahi avx bash-completion berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt css cups cxx dbus declarative dri dts dvd dvdr emboss encode exif fam firefox flac fortran gdbm gif glamor gpm gstreamer gtk iconv ios ipod ipv6 jpeg kde kipi lcms ldap libnotify mad mmx mmxext mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pulseaudio qml qt3support qt4 qt5 readline sdl seccomp semantic-desktop session spell sse sse2 ssl startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vdpau vim-syntax vorbis widgets wifi wxwidgets x264 xattr xcb xcomposite xinerama xml xscreensaver xv xvid zeroconf zlib" ABI_X86="64" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" SANE_BACKENDS="epson epson2 net" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
I think this is related to openmpi's prefix behaviour. Same with other openmpi versions. You can either specify the library path with -x, run a bash script as wrapper which sets the library path before executing the binary or just disable the automatic prefix behaviour with --noprefix mpirun --noprefix -n 1 ldd /usr/bin/glxgears|grep GL libGL.so.1 => /usr/lib64/opengl/nvidia/lib/libGL.so.1 (0x00007fe2ac63f000) libGLX.so.0 => /usr/lib64/opengl/nvidia/lib/libGLX.so.0 (0x00007fe2ab94d000) libGLdispatch.so.0 => /usr/lib64/opengl/nvidia/lib/libGLdispatch.so.0 (0x00007fe2ab665000) I assume that this is the intended behaviour of openmpi and not a bug.
I agree on the principle. I submitted the bug report because the error is triggered by the way gentoo handles the OpenGL libraries. It didn't happen with the old approach where the libraries where links pointing to the proper version. From a user point of view, mpirun is picking up the wrong OpenGL version and I wouldn't expect to have to change anything to work with a "system" library. I was hoping there would be a solution that would work without changing openmpi's general behavior. (In reply to Marius Brehler from comment #2) > I think this is related to openmpi's prefix behaviour. Same with other > openmpi versions. You can either specify the library path with -x, run a > bash script as wrapper which sets the library path before executing the > binary or just disable the automatic prefix behaviour with --noprefix > > mpirun --noprefix -n 1 ldd /usr/bin/glxgears|grep GL > libGL.so.1 => /usr/lib64/opengl/nvidia/lib/libGL.so.1 > (0x00007fe2ac63f000) > libGLX.so.0 => /usr/lib64/opengl/nvidia/lib/libGLX.so.0 > (0x00007fe2ab94d000) > libGLdispatch.so.0 => > /usr/lib64/opengl/nvidia/lib/libGLdispatch.so.0 (0x00007fe2ab665000) > > I assume that this is the intended behaviour of openmpi and not a bug.
