Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 391257 - app-portage/smart-live-rebuild: Can't finish updating the repositories and waits forever
Summary: app-portage/smart-live-rebuild: Can't finish updating the repositories and wa...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Michał Górny
URL: https://github.com/mgorny/smart-live-...
Whiteboard:
Keywords: InVCS
Depends on:
Blocks:
 
Reported: 2011-11-21 15:57 UTC by keenblade
Modified: 2012-01-22 22:55 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description keenblade 2011-11-21 15:57:35 UTC
For a while, app-portage/smart-live-rebuild-1.2 can't finish updating the repositories and stucks on a repo. Then nothing happens. The same thin happens on my two laptop ~amd64 arch. Also using 1 job instead of 6 parallel does not change anything. Pressing ctrl+c makes it to emerge the live ebuilds it has checked before it stucks. Using one job, it can't proceed the first live ebuild, using 6 job it can check some ebuilds but finally stuck on. 

smart-live-rebuild -d -j 1
*** Updating the repositories...
->  [x11-libs/libfm:0] git://pcmanfm.git.sourceforge.net/gitroot/pcmanfm/libfm [master]
--> git ls-remote --heads git://pcmanfm.git.sourceforge.net/gitroot/pcmanfm/libfm master


smart-live-rebuild
*** Updating the repositories using 6 parallel jobs...
->  git://pcmanfm.git.sourceforge.net/gitroot/pcmanfm/libfm [master]
--> git ls-remote --heads git://pcmanfm.git.sourceforge.net/gitroot/pcmanfm/libfm master
->  http://svn.drobilla.net/lad/trunk
--> svn --config-dir /media/xs-2/usr/portage/distfiles/svn-src/.subversion info http://svn.drobilla.net/lad/trunk
->  git://pcmanfm.git.sourceforge.net/gitroot/pcmanfm/pcmanfm [master]
--> git ls-remote --heads git://pcmanfm.git.sourceforge.net/gitroot/pcmanfm/pcmanfm master
->  [x11-libs/libfm:0] git://pcmanfm.git.sourceforge.net/gitroot/pcmanfm/libfm [master]
--> update from 163aae3f1f9bb3e833cc29e7d0e8c144ffa5be04 to df46e3b4bc3d01cfdf676d67c8cc1a6bb6315ca8
->  git://github.com/Dieterbe/uzbl.git [master]
--> git ls-remote --heads git://github.com/Dieterbe/uzbl.git master

Also "emerge --info" says:
Error during set creation: Could not import 'smartliverebuild.sets.SmartLiveRebuildSet' for section 'smart-live-rebuild'
Comment 1 keenblade 2011-11-21 15:59:05 UTC
emerge --info
Error during set creation: Could not import 'smartliverebuild.sets.SmartLiveRebuildSet' for section 'smart-live-rebuild'
Portage 2.2.0_alpha75 (default/linux/amd64/10.0/desktop, gcc-4.6.2, glibc-2.13-r4, 3.1.0-gentoo-r1 x86_64)
=================================================================
System uname: Linux-3.1.0-gentoo-r1-x86_64-Intel-R-_Core-TM-2_CPU_T7200_@_2.00GHz-with-gentoo-2.1
Timestamp of tree: Fri, 18 Nov 2011 22:30:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [enabled]
ccache version 3.1.6 [enabled]
app-shells/bash:          4.2_p10
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.2-r3, 3.2.2
dev-util/ccache:          3.1.6
dev-util/cmake:           2.8.6-r3
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.1
sys-apps/openrc:          0.9.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13::<unknown repository>, 2.68
sys-devel/automake:       1.4_p6-r1, 1.5-r1, 1.7.9-r2, 1.8.5-r4, 1.9.6-r3, 1.10.3, 1.11.1-r1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.5.2, 4.6.2
sys-devel/gcc-config:     1.5-r2
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 2.6.39 (virtual/os-headers)
sys-libs/glibc:           2.13-r4
Repositories: gentoo mpd crossdev proaudio symix keenblade pcsx2 pok3d pd-overlay desktop-effects emacs java-overlay webapp-experimental rox enlightenment mozilla ladi gechi gamerlay-stable mysql sunrise gnome d-overlay
Installed sets: @compiz-fusion
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=core2 -O2 -pipe"
DISTDIR="/media/xs-2/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--keep-going --quiet-build=n"
FEATURES="assume-digests binpkg-logs ccache collision-protect distcc distlocks ebuild-locks fail-clean fixlafiles news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/ "
LANG="tr_TR.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="tr"
MAKEOPTS="-j8"
PKGDIR="/media/xs-2/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/media/xs-2/usr/local/overlays/mpd-gentoo-overlay /usr/local/overlays/crossdev /media/xs-2/usr/local/overlays/proaudio/trunk/overlays/proaudio /media/xs-2/usr/local/overlays/symix /media/xs-2/usr/local/overlays/keenblade /var/lib/layman/pcsx2 /var/lib/layman/pok3d /var/lib/layman/pd-overlay /var/lib/layman/desktop-effects /var/lib/layman/emacs /var/lib/layman/java-overlay /var/lib/layman/webapps-experimental /var/lib/layman/rox /var/lib/layman/enlightenment /var/lib/layman/mozilla /var/lib/layman/ladi /var/lib/layman/gechi /var/lib/layman/gamerlay /var/lib/layman/mysql /var/lib/layman/sunrise /var/lib/layman/gnome /var/lib/layman/d"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 aac acl acpi alsa amd64 apache2 berkdb bluetooth bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus dri dts dvd dvdr eds emboss encode evo exif fam ffmpeg firefox flac fortran gdbm gdu gif glitz gnome gpm gstreamer gtk iconv ipv6 jack java jpeg kde ladspa lash lcms ldap libnotify lm_sensors mad midi mmx mng modules mp3 mp4 mpeg mudflap multilib mysql mysqli ncurses nls nptl nptlonly nvidia ogg opengl openmp pam pango pcre pdf png policykit ppds pppd qt3support qt4 readline sdl session spell sqlite3 sse sse2 ssl startup-notification svg sysfs tcpd theora tiff truetype udev unicode usb vorbis x264 xcb xcomposite xinerama xml xorg xulrunner xv xvid xvmc zlib" ALSA_CARDS="hda-intel mpu401 usb-audio" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="tr" LIRC_DEVICES="alsa_usb" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nouveau nvidia vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 2 Jeroen Roovers (RETIRED) gentoo-dev 2011-11-22 20:37:24 UTC
so it should time-out at some point?
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-11-22 21:27:05 UTC
I'd rather prefer git to handle the timeouts itself but I guess such a feature could be feasible.
Comment 4 keenblade 2011-11-23 00:02:38 UTC
(In reply to comment #3)
> I'd rather prefer git to handle the timeouts itself but I guess such a feature
> could be feasible.

I agree, it would be great git or the related vcs handle timeouts, since it is happening with orther vcs's like subversion, too.

Also, some repos are very big and the updating takes time, so "a fixed time-out at some point" may break the updating.
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-11-23 08:14:29 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > I'd rather prefer git to handle the timeouts itself but I guess such a feature
> > could be feasible.
> 
> I agree, it would be great git or the related vcs handle timeouts, since it is
> happening with orther vcs's like subversion, too.
> 
> Also, some repos are very big and the updating takes time, so "a fixed time-out
> at some point" may break the updating.

Yes, that's what I'm afraid of. However, considering that nowadays only CVS and darcs require updating repos directly while other VCS-es just ping the server for newest rev, a timeout like 30 seconds seems reasonable.

Maybe it's time to finally drop support for CVS and darcs because they're a maintenance pain with their own branch of code? A quick look shows 8 and 1 user respectively but it's likely some of them don't use CVS anymore and the ebuild is outdated.
Comment 6 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-11-24 18:12:23 UTC
Committed to git HEAD. Will appreciate testing.
Comment 7 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-11-25 20:38:01 UTC
And it's in 1.2.1, as --timeout.
Comment 8 keenblade 2011-12-10 12:19:17 UTC
(In reply to comment #6)
> Committed to git HEAD. Will appreciate testing.
Sorry for being late. Problem still is here. I tried to reopen the bug, but there is no option for it. python3.2 is the active one. Anyway, Here is the first output:

smart-live-rebuild -j 1 -d -T 15
*** Updating the repositories...
->  [games-util/lakitu-qvm:0] svn://source.mercenariesguild.net/lakitu7-qvm/trunk
--> svn --config-dir /media/xs-2/usr/portage/distfiles/svn-src/.subversion info svn://source.mercenariesguild.net/lakitu7-qvm/trunk
--> at rev 181 (no changes)
->  [www-client/uzbl:0] git://github.com/Dieterbe/uzbl.git [master]
--> git ls-remote --heads git://github.com/Dieterbe/uzbl.git master
--> update from e2f3fe51149c8924b83899db87425ea1a462dc4b to e93bd72c2c253fd6d0f82b9606c23456869024ac
->  [net-wireless/brcm-firmware:0] git://git.kernel.org/pub/scm/linux/kernel/git/dwmw2/linux-firmware.git [master]
--> git ls-remote --heads git://git.kernel.org/pub/scm/linux/kernel/git/dwmw2/linux-firmware.git master
--> at rev 15888a2eab052ac3d3f49334e4f6f05f347a516e (no changes)
->  [app-portage/update-live-ebuilds:0] git://repo.or.cz/ule.git [master]
--> git ls-remote --heads git://repo.or.cz/ule.git master
fatal: The remote end hung up unexpectedly
Traceback (most recent call last):
  File "/usr/bin/smart-live-rebuild-2.7", line 26, in <module>
    sys.exit(main(sys.argv))
  File "/usr/lib64/python2.7/site-packages/smartliverebuild/cli.py", line 140, in main
    packages = SmartLiveRebuild(opts, pm, cliargs = args)
  File "/usr/lib64/python2.7/site-packages/smartliverebuild/core.py", line 107, in SmartLiveRebuild
    loop_iter()
  File "/usr/lib64/python2.7/site-packages/smartliverebuild/core.py", line 75, in loop_iter
    ret = vcs(blocking)
  File "/usr/lib64/python2.7/site-packages/smartliverebuild/vcs/__init__.py", line 144, in __call__
    return self._endupdate()
  File "/usr/lib64/python2.7/site-packages/smartliverebuild/vcs/__init__.py", line 212, in _endupdate
    raise Exception('update command returned non-zero result')
Exception: update command returned non-zero result


Here is the second output that waits forever, if it can't get response:
smart-live-rebuild -j 1 -d -T 15
*** Updating the repositories...
->  [games-util/lakitu-qvm:0] svn://source.mercenariesguild.net/lakitu7-qvm/trunk
--> svn --config-dir /media/xs-2/usr/portage/distfiles/svn-src/.subversion info svn://source.mercenariesguild.net/lakitu7-qvm/trunk
Comment 9 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-12-10 19:13:27 UTC
(In reply to comment #8)
>   File "/usr/lib64/python2.7/site-packages/smartliverebuild/vcs/__init__.py",
> line 144, in __call__
>     return self._endupdate()
>   File "/usr/lib64/python2.7/site-packages/smartliverebuild/vcs/__init__.py",
> line 212, in _endupdate
>     raise Exception('update command returned non-zero result')
> Exception: update command returned non-zero result

That's fine. --debug is not supposed to be used in other way than enforcing exceptions on errors. Rephrased its description in HEAD.

> Here is the second output that waits forever, if it can't get response:
> smart-live-rebuild -j 1 -d -T 15
> *** Updating the repositories...
> ->  [games-util/lakitu-qvm:0]
> svn://source.mercenariesguild.net/lakitu7-qvm/trunk
> --> svn --config-dir /media/xs-2/usr/portage/distfiles/svn-src/.subversion info
> svn://source.mercenariesguild.net/lakitu7-qvm/trunk

Hmm, '-j 1' may be the cause here. Will have to take a look.
Comment 10 keenblade 2011-12-11 01:53:54 UTC
(In reply to comment #9)
> 
> Hmm, '-j 1' may be the cause here. Will have to take a look.

Without '-j 1' , it passes lots of repo, but finally stuck at some random repo again., everytime. Like this:

smart-live-rebuild -T 15
... --- Lots of repo  correctly checked, but then again stuck at here: ----
--> git ls-remote --heads git://code.stapelberg.de/i3 tree
->  [x11-wm/sawfish:0] git://git.tuxfamily.org/gitroot/sawfish/main.git [master]
--> at rev eb75b4f51f4bfae36fc03a94c0139d24f5b98f37 (no changes)
Logging in to :pserver:anonymous@cvs.fvwm.org:2401/home/cvs/fvwm
Comment 11 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-12-11 17:59:01 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > 
> > Hmm, '-j 1' may be the cause here. Will have to take a look.
> 
> Without '-j 1' , it passes lots of repo, but finally stuck at some random repo
> again., everytime. Like this:

Yes, because '-j N' switches to '-j 1' when it reaches the end. Please try now.
Comment 12 keenblade 2011-12-11 19:04:31 UTC
(In reply to comment #11)
> 
> Yes, because '-j N' switches to '-j 1' when it reaches the end. Please try now.

How to try? Using your git repo? There does not seem a new version in portage. It still waits forever without '-j 1' switch, after passing some repos. As exactly in my previous post. Using only "-T 15" switch fails:
smart-live-rebuild -T 15
When it comes at a random repo it stops working and waits forever. Since everytime it stops working on a different repo, it does not reaches to the end, I guess.
Comment 13 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2012-01-22 22:55:19 UTC
Was fixed in 1.2.1.