Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 239101 - Generated Makefile for ruby bindings is not parallel safe (breaks net-analyzer/rrdtool)
Summary: Generated Makefile for ruby bindings is not parallel safe (breaks net-analyze...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: New packages (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Gentoo Ruby Team
URL: http://redmine.ruby-lang.org/issues/s...
Whiteboard:
Keywords:
Depends on:
Blocks: 238568
  Show dependency tree
 
Reported: 2008-09-30 00:19 UTC by MATSUU Takuto (RETIRED)
Modified: 2009-04-03 12:42 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
rrdtool-1.3.3.ebuild-parallel.patch (rrdtool-1.3.3.ebuild-parallel.patch,580 bytes, patch)
2008-09-30 00:19 UTC, MATSUU Takuto (RETIRED)
Details | Diff
rrdtool-1.3.3-with-ruby.log (rrdtool-1.3.3-with-ruby.log,227.57 KB, text/plain)
2008-10-02 00:06 UTC, MATSUU Takuto (RETIRED)
Details
rrdtool emerge log with MAKEOPTS=-j1 (rrdtool-1.3.3-with-ruby-j1.log,227.43 KB, text/plain)
2008-10-02 00:07 UTC, MATSUU Takuto (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description MATSUU Takuto (RETIRED) gentoo-dev 2008-09-30 00:19:07 UTC
Sometimes net-analyzer/rrdtool parallel make install failed with ruby bindings, but emerge doesn't failed.

# USE=ruby emerge rrdtool
(snip)
make[4]: Entering directory `/var/tmp/portage/net-analyzer/rrdtool-1.3.3/work/rrdtool-1.3.3/bindings/ruby'
mkdir -p /var/tmp/portage/net-analyzer/rrdtool-1.3.3/image//usr/lib64/ruby/site_ruby/1.8/x86_64-linux
/usr/bin/install -c -m 0755 RRD.so /var/tmp/portage/net-analyzer/rrdtool-1.3.3/image//usr/lib64/ruby/site_ruby/1.8/x86_64-linux
/usr/bin/install: cannot create regular file `/var/tmp/portage/net-analyzer/rrdtool-1.3.3/image//usr/lib64/ruby/site_ruby/1.8/x86_64-linux': File exists
make[4]: *** [/var/tmp/portage/net-analyzer/rrdtool-1.3.3/image//usr/lib64/ruby/site_ruby/1.8/x86_64-linux/RRD.so] Error 1
make[4]: Leaving directory `/var/tmp/portage/net-analyzer/rrdtool-1.3.3/work/rrdtool-1.3.3/bindings/ruby'


I think that it should add "-j1" to "emake install".


Portage 2.2_rc11 (!../usr/portage/profiles/default-linux/amd64/2007.0/desktop, gcc-4.3.1, glibc-2.8_p20080602-r0, 2.6.26-gentoo-r1 x86_64)
=================================================================
System uname: Linux-2.6.26-gentoo-r1-x86_64-Intel-R-_Core-TM-2_Duo_CPU_L7300_@_1.40GHz-with-glibc2.2.5
Timestamp of tree: Unknown
distcc 2.18.3 x86_64-pc-linux-gnu (protocols 1 and 2) (default port 3632) [enabled]
ccache version 2.4 [enabled]
app-shells/bash:     3.2_p39
dev-java/java-config: 1.3.7, 2.1.6-r1
dev-lang/python:     2.5.2-r7
dev-python/pycrypto: 2.0.1-r6
dev-util/ccache:     2.4-r7
sys-apps/baselayout: 2.0.0
sys-apps/openrc:     0.2.5
sys-apps/sandbox:    1.2.18.1-r3
sys-devel/autoconf:  2.13, 2.63
sys-devel/automake:  1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1
sys-devel/binutils:  2.18-r3
sys-devel/gcc-config: 1.4.0-r4
sys-devel/libtool:   2.2.4
virtual/os-headers:  2.6.26
ACCEPT_KEYWORDS="amd64 ~amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=nocona -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d"
CXXFLAGS="-O2 -march=nocona -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs ccache collision-protect cvs distcc distlocks parallel-fetch preserve-libs protect-owned sandbox sfperms strict test unmerge-orphans userfetch"
GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo"
LANG="ja_JP.UTF-8"
LDFLAGS=""
LINGUAS="ja"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/home/matsuu/gentoo-x86"
PORTDIR_OVERLAY="/usr/local/portage /home/matsuu/work/matsuu_overlay/secondlife /home/matsuu/work/matsuu_overlay/misc /home/matsuu/work/matsuu_overlay/awesome /home/matsuu/work/matsuu_overlay/processing"
SYNC="cvs://matsuu@cvs.gentoo.org:/var/cvsroot"
USE="X a52 aac acl acpi alsa amd64 anthy apache2 arts avahi bash-completion berkdb bitmap-fonts cairo ccache cdr cjk cli cracklib crypt cups curl daap dbus dga dmx dri dvd dvdr dvdread eds emboss encode esd evo fam firefox fortran gdbm gif glitz gnome gpm gstreamer gtk hal iconv imap immqt-bc ipv6 ipw4965 isdnlog java jpeg kde kerberos logrotate mad mdnsresponder-compat midi mikmod mmx mozdevelop mp3 mpeg mudflap ncurses nls nptl nptlonly nsplugin ogg opengl openmp oss pam pcre pdf perl png pppd python qt3 qt3support qt4 quicktime readline reflection sdl session spell spl sse sse2 ssl ssse3 svg tcpd tiff truetype truetype-fonts type1-fonts unicode vorbis wireshark x264 xcb xcomposite xml xorg xosd xulrunner xv xvid zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="mouse keyboard synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="ja" USERLAND="GNU" VIDEO_CARDS="i810 vesa vga"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 MATSUU Takuto (RETIRED) gentoo-dev 2008-09-30 00:19:43 UTC
Created attachment 166787 [details, diff]
rrdtool-1.3.3.ebuild-parallel.patch
Comment 2 Peter Volkov (RETIRED) gentoo-dev 2008-09-30 05:41:42 UTC
Please, attach full build output. -j1 is only workaround which we should avoid as much as possible...
Comment 3 MATSUU Takuto (RETIRED) gentoo-dev 2008-10-02 00:06:11 UTC
Created attachment 166948 [details]
rrdtool-1.3.3-with-ruby.log

no ruby binding are installed.
it seems that 'mkmf' ruby module has something problem.
Comment 4 MATSUU Takuto (RETIRED) gentoo-dev 2008-10-02 00:07:21 UTC
Created attachment 166950 [details]
rrdtool emerge log with MAKEOPTS=-j1
Comment 5 Peter Volkov (RETIRED) gentoo-dev 2008-10-05 12:07:47 UTC
Please, provide full build output of broken build. I need to see how it fails...
Comment 6 MATSUU Takuto (RETIRED) gentoo-dev 2008-10-05 16:30:37 UTC
rrdtool-1.3.3-with-ruby.log is broken build. it has no emerge error, but /usr/lib64/python2.5/site-packages/rrdtoolmodule.so is not provided.

please check `diff -u rrdtool-1.3.3-with-ruby.log rrdtool-1.3.3-with-ruby-j1.log`.
Comment 7 Peter Volkov (RETIRED) gentoo-dev 2008-10-13 15:55:22 UTC
This is ruby build problem so this is impossible to fix in rrdtool.

Makefile for ruby bindings is created with 

ruby extconf.rb

where extconf.rb is really short file:
========================================
require 'mkmf'

if /linux/ =~ RUBY_PLATFORM
   $LDFLAGS += '-Wl,--rpath -Wl,$(EPREFIX)/lib'
elsif /solaris/ =~ RUBY_PLATFORM
   $LDFLAGS += '-R$(EPREFIX)/lib'
elsif /hpux/ =~ RUBY_PLATFORM
   $LDFLAGS += '+b$(EPREFIX)/lib'
elsif /aix/ =~ RUBY_PLATFORM
   $LDFLAGS += '-Wl,-blibpath:$(EPREFIX)/lib'
end

dir_config("rrd","../../src","../../src/.libs")
have_library("rrd", "rrd_create")
create_makefile("RRD")
========================================

Resulted Makefile is not parallel safe...
================================================================
install: install-so ## Not affects install-rb

install-so: $(RUBYARCHDIR)
install-so: $(RUBYARCHDIR)/$(DLLIB)
$(RUBYARCHDIR)/$(DLLIB): $(DLLIB)
        $(INSTALL_PROG) $(DLLIB) $(RUBYARCHDIR)

$(RUBYARCHDIR):
        $(MAKEDIRS) $@

================================================================

So it calls $(RUBYARCHDIR) and $(RUBYARCHDIR)/$(DLLIB) in parallel. From make file info:
"
One file can be the target of several rules.  All the prerequisites
mentioned in all the rules are merged into one list of prerequisites for
the target.
"

So reasons found but this is definitely not rrdtool issue. This is the issue with ruby 1.8.6_p287-r1.

Ruby is this know issue?
Comment 8 Hans de Graaff gentoo-dev Security 2008-10-26 07:27:53 UTC
It wasnt' a known issue to the ruby herd. It's not clear to me where the real problem is. Perhaps ruby's mkmf should write out parallel make safe Makefiles, but it can only do that if it understands the dependencies.

In any case I think some investigation into the ruby side of things is needed first.
Comment 9 Chris Gianelloni 2009-03-30 20:33:47 UTC
So, rather than forcing -j1 on everybody, can we "fix" rrdtool's ebuild to only use -j1 if USE=ruby, or, even better... simply patch the generated Makefile after it's been created?
Comment 10 MATSUU Takuto (RETIRED) gentoo-dev 2009-03-31 09:53:50 UTC
I found the issue and reported to upstream.
http://redmine.ruby-lang.org/issues/show/1337

I think it would be good idea to backport the patch to ruby-1.8.7_p72-r10.ebuild.
http://redmine.ruby-lang.org/attachments/download/338
Comment 11 Peter Volkov (RETIRED) gentoo-dev 2009-04-03 09:51:51 UTC
(In reply to comment #9)
> So, rather than forcing -j1 on everybody, can we "fix" rrdtool's ebuild to
> only use -j1 if USE=ruby

Since this issue affects only 'make install' do you think it's worth additional logic in ebuild?

(In reply to comment #10)
> http://redmine.ruby-lang.org/attachments/download/338

MATSUU, these are really great news. Thank you for localizing and fixing this issue.


@ruby: Could you backport patch, please? :)
Comment 12 Alex Legler (RETIRED) archtester gentoo-dev Security 2009-04-03 12:42:42 UTC
(In reply to comment #11)
> @ruby: Could you backport patch, please? :)
> 

Fixed in ruby-1.8.6_p287-r13.ebuild, ruby-1.8.7_p72-r11.ebuild and ruby-1.9.1-r1.ebuild. Built rrdtool successfully against it multiple times.

Thanks everyone!