| Summary: | media-libs/opencv-2.4.3 with USE=cuda : /opt/cuda/bin/nvcc compiles in /tmp which results in "No space left on device" | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Daniel Troeder <daniel> |
| Component: | Current packages | Assignee: | Andreas K. Hüttel <dilfridge> |
| Status: | RESOLVED OBSOLETE | ||
| Severity: | normal | CC: | amynka, juantxorena, kde, martin.dummer, tom |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | AMD64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: | media-libs:opencv-2.4.3:20121109-105730.log.bz2 | ||
Created attachment 329004 [details]
media-libs:opencv-2.4.3:20121109-105730.log.bz2
compile log (bzip2'd)
How is this a bug if your space is running out? Sounds like a job for the system admin to resize the partition. You should be using FEATURES='sandbox' to avoid compilations all over your running system. Or what Samuli said. The temporary files of nvcc *should* go to $TEMP, which is set by portage to the ebuild temp dir. Why this does not work is a mystery to me; unfortunately I cannot test it since I've got gcc-4.7. :( Do you actually have no space left on /tmp, or is this a "fake error" generated by the sandbox? I can confirm this bug. The space runs out in /tmp, which is 1GB on my system. My FEATURES are: FEATURES="assume-digests binpkg-logs compressdebug config-protect-if-modified distlocks ebuild-locks fakeroot fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch xattr" so using "sandbox" does not help When pausing the emerge just in the middle I can see a process like: /opt/cuda/bin/nvcc /var/tmp/portage/media-libs/opencv-2.4.3/work/OpenCV-2.4.3/modules/gpu/src/cuda/pyr_down.cu .... which starts a child process cudafe --m64 --gnu_version=40603 -tused --no_remove_unneeded_entities --gen_c_file_name /tmp/tmpxft_00007873_00000000-6_pyr_down.compute_20.cudafe1.c --stub_file_name /tmp/tmpxft_00007873_00000000-6_pyr_down.compute_20.cudafe1.stu and these files /tmp/tmpxft_<some-randomness> really exist in /tmp... So someone has to convince "nvcc" to use a different temp directory - but a quick "nvcc --help" gave no hint to me. Any ideas (except the one from Comment #2)??? My /tmp is on a 512MB tmpfs. I grew it, and could compile. The peek disk usage by the compilation was 2.4 GB in /tmp (plus 2 GB in PORTAGE_TMPDIR). There are dozens of files >30MB with names like /tmp/tmpxft_00000808_00000000-27_row_filter.compute_20.cpp3.i /tmp/tmpxft_00000808_00000000-30_row_filter.compute_12.cudafe2.gpu /tmp/tmpxft_00000f28_00000000-11_column_filter.compute_11.ptx /tmp/tmpxft_00001a54_00000000-3_element_operations.fatbin.c /tmp/tmpxft_00000808_00000000-11_row_filter.compute_11.ptx /tmp/tmpxft_00000f28_00000000-9_column_filter.compute_30.ptx /tmp/tmpxft_00001a54_00000000-49_element_operations.compute_11.ii that seem to be computer generated C and CUDA source files. PS: I enabled FEATURES="sandbox", but it didn't help, /tmp/tmpxft_* are still created. Not a regression. I confirm this bug. Also, I fail to see how this is not a regression, since I could compile opencv with the same useflags in the past and now I can't. The fact that this may not be an important bug because it affects only to people with a very specific configuration doesn't mean that is not a regression. I've finally compiled this. For that, I had to mount /tmp somewhere else with more space, so it didn't thrown me the error. Anyway, the compilation time was incredibly high. According to genlop, it lasted 3 hours and 46 minutes, whereas the older versions with the same useflags and same everything were about 10 minutes (13 minutes the longer one, the previous stable version, 2.3.1a-r1). Definitely there is something fishy going on there. I agree this is not a bug, at least not in Gentoo. Long compile time is a little wierd, but I suspect it is due to compiling for all CUDA versions, see http://answers.opencv.org/question/5090/why-opencv-building-is-so-slow-with-cuda/ for example. But this is really up to opencv developers to say if it is a bug or not. Creating files in /tmp is completely normal and OK. Maybe the ebuild should check if there is enough space, but it is hard to say what is "enough", as it depends mainly on your -jX setting in MAKEOPTS. I suggest compiling with MAKEOPTS="-j1" for a smaller footprint. If that is not enough, then - *sigh* - your /tmp just need to be larger. Are the devs on me with this one, or am I way out of line here? :) Cheers and pardon my English, Tom The compilation for every single GPU arch possible is quite stupid IMHO. If we're going to have this the gentoo way, we should only compile what we need. According to that single link that Tomáš give us, there's an easy workaround for this. Maybe we should have some kind of USE flag, or variable, or something, to compile only the version we need. As for the other thing, the tmp problem, I dunno. I doubt that the devs do something which isn't done upstream. I agree, some USE_EXPAND flag would be nice here. One have to know his HW, but that is why we love Gentoo :) (In reply to comment #11) > I agree this is not a bug, at least not in Gentoo. > ... > > Creating files in /tmp is completely normal and OK. Maybe the ebuild should > check if there is enough space, but it is hard to say what is "enough", as > it depends mainly on your -jX setting in MAKEOPTS. I do not agree, and if you scroll back and read comment #4 and my analyze in #5 again you also should be convinced that it IS a bug. Let me repeat in short: the nvcc compiler should use the content of the TEMP environment variable for his tempfiles which is set by portage to TEMP=/var/tmp/portage/media-libs/opencv-2.4.3/temp (in my setup). But he ifnores this and uses /tmp instead. > I suggest compiling with MAKEOPTS="-j1" for a smaller footprint. I will try this suggestion. Well, I see your point, but I was treating it as a compiler. gcc without -pipe option will IMHO do the same thing. @Andreas would help to add functionality (freespace check) of check-reqs.eclass here? Version is not currently in tree.. Amynka |
Compilation of media-libs/opencv-2.4.3 with USE=cuda fails with /opt/cuda/bin/nvcc /Media/ROOT/gentoo/tmp/build/working/portage/media-libs/opencv-2.4.3/work/OpenCV-2.4.3/modules/gpu/src/cuda/surf.cu [..] <built-in>:0:0: fatal error: when writing output to /tmp/tmpxft_00001037_00000000-46_pyr_up.cpp4.ii: No space left on device compilation terminated. Compiles cleanly with USE=-cuda. Reproducible: Always Steps to Reproduce: 1. USE=cuda emerge =media-libs/opencv-2.4.3 moja ~ # emerge -pqv '=media-libs/opencv-2.4.3' [ebuild U ] media-libs/opencv-2.4.3 [2.4.2] USE="cuda ffmpeg gstreamer gtk ieee1394 jpeg opencl%* opengl pch png threads%* tiff v4l -doc -eigen -examples (-ipp) -jpeg2k -openexr -qt4 -testprograms -xine (-tbb%)" moja ~ # emerge --info '=media-libs/opencv-2.4.3' Portage 2.1.11.31 (default/linux/amd64/10.0/desktop/gnome, gcc-4.6.3, glibc-2.16.0, 3.6.2-gentoo.moja x86_64) ================================================================= System Settings ================================================================= System uname: Linux-3.6.2-gentoo.moja-x86_64-Intel-R-_Core-TM-_i5_CPU_M_560_@_2.67GHz-with-gentoo-2.2 Timestamp of tree: Fri, 09 Nov 2012 07:45:01 +0000 ld GNU ld (GNU Binutils) 2.23 ccache version 3.1.8 [disabled] app-shells/bash: 4.2_p39 dev-java/java-config: 2.1.12 dev-lang/python: 2.7.3-r2, 3.2.3-r1 dev-util/ccache: 3.1.8 dev-util/cmake: 2.8.9-r1 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.4 sys-apps/sandbox: 2.6 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.9.6-r3, 1.11.6, 1.12.4 sys-devel/binutils: 2.23 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.6 (virtual/os-headers) sys-libs/glibc: 2.16.0 Repositories: gentoo AzP sunrise spring multimedia ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/gentoo/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="ftp://linux.rz.ruhr-uni-bochum.de/gentoo-mirror/ ftp://ftp.tu-clausthal.de/pub/linux/gentoo/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ http://distfiles.gentoo.org" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en" MAKEOPTS="-j5" PKGDIR="/gentoo/pkg" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/gentoo/tmp/build/working" PORTDIR="/gentoo/portage+edb+pkg/portage" PORTDIR_OVERLAY="/var/lib/layman/AzP /var/lib/layman/sunrise /var/lib/layman/Spring /var/lib/layman/multimedia" SYNC="rsync://rsync.de.gentoo.org/gentoo-portage" USE="10bit X a52 aac aalib acl acpi alsa amd64 apache2 apng applet asyncns autoipd avahi bash-completion berkdb bluetooth bonjour branding brasero bzip2 c++0x cairo calendar caps cdaudio cdda cddax cddb cdio cdr cdrkit cli clucene colord config-file connection-sharing consolekit cracklib crypt css cuda cups curl custom-optimization cxx daap dbus device-mapper dga dhclient dhcp divx dns dri dts dv dvb dvd dvdr dvdread dvi eds emboss enca encode evo exif expat expoblending faac faad fam fame fat ffmpeg fftw firefox flac fontconfig fts fts3 fuse galago gcj gconf gd gdbm gdu gedit gif gimp git glibc-omitfp gmedia gmp gnome gnome-keyring gnome-online-accounts gnutls gphoto2 gpm gps gs gtk gtk3 gudev gzip h323 hal hddtemp hpijs http http-forms http-server hwdb iconv icu id3tag ieee1394 imagemagick imlib inotify introspection ipv6 irc jabber java joystick jpeg kdeenablefinal kdrive kernel-patch kpathsea lame laptop lcms ldap libass libcaca libnotify lm_sensors lzma lzo mad maps matroska mdnsresponder-compat milter minizip mjpeg mmap mms mmx mmxext mng modules mp3 mp4 mpeg mpi mplayer mudflap multilib musicbrainz nautilus ncurses net network networkmanager nfs nfsv3 nfsv4 nls nntp nptl nsplugin nss ntfs nuv nvidia offensive ogg opengl openmp openssl openvg optimization opus pam pango pch pcre pdf pgo pidgin pipechan plotutils png policykit postproc postscript ppds pppd pth pulseaudio qemu qemu-ifup qos qpa qt-faststart quicktime quotas ramfs rar rchive readline realmedia realtime reiserfs remote romio rtmp rtsp ruby18 samba scanner sdl secure-delete semantic-desktop sendto session sftp shout sip smp socialweb spell sqlite sse sse2 sse3 sse4.1 ssl ssse3 startup-notification stream subtitles subversion svg system-sqlite taglib tagwriting tcpd theora threads thumbnail tiff tls truetype twolame udev udisks unicode upnp upower usb uuid v4l vaapi vcd vcdinfo vcdx vdpau video vorbis vpx wav wifi winbind wmf wmp wps wxwidgets x264 xattr xcb xfs xinerama xml xmlrpc xmp xnest xrandr xscreensaver xulrunner xv xvid xvmc zeroconf zip zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_core authn_dbd authn_dbm authn_default authn_file authz_core authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so socache_shmcb speling status unique_id unixd userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CURL_SSL="nss" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="evdev aiptek synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" PHP_TARGETS="php5-4" PYTHON_TARGETS="python3_2 python2_7" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby18 ruby19" SANE_BACKENDS="abaton agfafocus apple artec artec_eplus48u as6e avision bh canon canon630u canon_dr canon_pp cardscan coolscan coolscan2 coolscan3 dc210 dc240 dc25 dell1600n_net dmc epjitsu epson epson2 fujitsu genesys gt68xx hp hp3500 hp3900 hp4200 hp5400 hp5590 hpljm1005 hpsj5s hs2p ibm kodak leo lexmark ma1509 magicolor matsushita microtek microtek2 mustek mustek_pp mustek_usb nec net niash p5 pie pixma plustek plustek_pp qcam ricoh rts8891 s9036 sceptre sharp sm3600 sm3840 snapscan sp15c st400 stv680 tamarack teco1 teco2 teco3 test u12 umax umax1220u umax_pp xerox_mfp pnm" USERLAND="GNU" VIDEO_CARDS="vesa nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON