Bugzilla Bug 140593

emerge-webrsync fails with a message like:

creating dir 'games-board/crafty/files'
removing node 'games-board/crafty/' crafty-20.1.ebuild'
removing node 'games-board/crafty/' crafty-20.11.ebuild'
removing node 'games-board/crafty/' crafty-20.13.ebuild'
removing node 'games-board/crafty/' files'
removing node 'games-board/crafty/' metadata.xml'
2 malloc failure on games-board/crafty/files/digest-crafty-20.11
games-board/crafty/files/, /usr/portage
failed enforcing file layout, bailing
tarsync failed; tarball is corrupt?

Points of failure differ run by run.
If I revert the version of portage to 2.0.54-r2, emerge-webrsync works.


emerge --info:
Gentoo Base System version 1.6.15
Portage 2.1-r1 (default-linux/x86/2006.0, gcc-3.3.6, glibc-2.3.6-r4,
2.6.16-gent
oo-r9 i686)
=================================================================
System uname: 2.6.16-gentoo-r9 i686 Intel(R) Pentium(R) III Mobile CPU     
1133
MHz
ccache version 2.3 [enabled]
app-admin/eselect-compiler: [Not Present]
dev-lang/python:     2.3.5-r2, 2.4.3-r1
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.3
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.59-r7
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2
sys-devel/binutils:  2.16.1-r3
sys-devel/gcc-config: 1.3.13-r3
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.11-r2
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CBUILD="i386-pc-linux-gnu"
CFLAGS="-O2 -mcpu=i686 -pipe -fomit-frame-pointer"
CHOST="i386-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.3/env /usr/kde/3.3/share/config
/usr/kde/3.3/shu
tdown /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shutdown
/usr/lib/
X11/xkb /usr/share/config /usr/share/texmf/dvipdfm/config/
/usr/share/texmf/dvip
s/config/ /usr/share/texmf/tex/generic/config/
/usr/share/texmf/tex/platex/confi
g/ /usr/share/texmf/xdvi/"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo"
CXXFLAGS="-O2 -mcpu=i686 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig ccache digest distlocks metadata-transfer sandbox sfperms
s
trict userpriv"
GENTOO_MIRRORS="http://mirror.gentoo.gr.jp http://distfiles.gentoo.org"
LANG="ja_JP.eucJP"
LINGUAS="ja it fr es de ar"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress
--force --whole-file --delete --delete-after --stats --timeout=180
--exclude='/d
istfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 X alsa apache2 apm arts avi berkdb bitmap-fonts bzip2 cjk cli crypt
cup
s dlloader dri eds emboss encode esd foomaticdb fortran freewnn gdbm gif gnome
g
pm gstreamer gtk gtk2 imlib ipv6 isdnlog jpeg kde libg++ libwww mad mikmod
motif
 mp3 mpeg ncurses nls nptl ogg opengl oss pam pcre pdflib perl png pppd python
q
t qt3 qt4 quicktime readline reflection sdl session spell spl sse ssl tcltk
tcpd
 truetype truetype-fonts type1-fonts udev vorbis xml xmms xorg xv zlib
elibc_gli
bc input_devices_keyboard input_devices_mouse input_devices_evdev kernel_linux
l
inguas_ja linguas_it linguas_fr linguas_es linguas_de linguas_ar userland_GNU"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LDFLAGS, MAKEOPTS,
P
ORTAGE_RSYNC_EXTRA_OPTS

emerge-webrsync in 2.1 has new support for tarsync and that seems to be where
the problem is.  What version of tarsync do you have?  If you're using
tarsync-0.1, could you upgrade to tarsync-0.2 to see if that helps?

I've just tried upgrading tarsync and diffball.
Unfortunately, tarsync-0.2 with diffball-1.0 produces the same error.

I uninstalled tarsync, and then emerge-webrsync seems to work as before.
Though, I don't think this is the resolution.

I've reproduced the "malloc failure" locally with tarsync-0.2.

need to mangle emerge-webrsync to report the tarball that failed...
cause without that info bit hard fishing what failed (I'm getting no failures).

finally, try a downgrade to 0.7.1 of diffball- doubt it has any affect, but the
failure is the return from scandir (and till I get enough info to debug this,
kind of stabbing in the dark).

I've reproduced the problem updating to portage-20060717.tar.bz2 from a normal
rsync tree that was last updated on 20060716.  Diffball patch to get from
20060717 to the initial state:

http://dev.gentoo.org/~zmedico/tmp/snapshot-20060717-20060716.patch.bz2

scanning tarball...
cleansed 1 dups out of 145045
scanning existing target directory...
removing node 'app-admin/skey/' skey-1.1.5-r3.ebuild'
removing node 'app-admin/skey/' skey-1.1.5-r4.ebuild'
removing node 'app-admin/skey/' skey-1.1.5-r5.ebuild'
removing node 'app-admin/skey/files/' digest-skey-1.1.5-r3'
removing node 'app-admin/skey/files/' digest-skey-1.1.5-r4'
removing node 'app-admin/skey/files/' digest-skey-1.1.5-r5'
removing node 'app-admin/skey/files/' skey-1.1.5-bind-now.patch'
removing node 'app-admin/skey/files/' skey-1.1.5-fPIC.patch'
removing node 'app-admin/skey/files/' skey-1.1.5-otp.diff'
removing node 'app-admin/skey/files/' skey-login_name_max.diff'
removing node 'app-cdr/qpxtool/' qpxtool-0.5.0.ebuild'
removing node 'app-cdr/qpxtool/' qpxtool-0.5.3.ebuild'
removing node 'app-cdr/qpxtool/files/' digest-qpxtool-0.5.0'
removing node 'app-cdr/qpxtool/files/' digest-qpxtool-0.5.3'
removing node 'app-laptop/thinkpad/' thinkpad-5.7.ebuild'
removing node 'app-laptop/thinkpad/' thinkpad-5.8.ebuild'
removing node 'app-laptop/thinkpad/' thinkpad-5.9-r1.ebuild'
removing node 'app-laptop/thinkpad/files/' digest-thinkpad-5.7'
removing node 'app-laptop/thinkpad/files/' digest-thinkpad-5.8'
removing node 'app-laptop/thinkpad/files/' digest-thinkpad-5.9-r1'
removing node 'app-laptop/thinkpad/files/' thinkpad'
removing node 'app-laptop/thinkpad/files/' thinkpad-5.8-cpu_ptr.patch'
removing node 'app-laptop/thinkpad/files/'
thinkpad-5.9-remove-inter_module.patch'
removing node 'app-laptop/thinkpad/files/'
thinkpad-5.9-remove-thinkpadapm-argument.patch'
creating dir 'app-text/build-docbook-catalog'
removing node 'app-text/' 'bow'
removing node 'app-text/' 'build-docbook-catalog'
2 malloc failure on app-text/build-docbook-catalog/ChangeLog
app-text/build-docbook-catalog/, /usr/portage
failed enforcing file layout, bailing

I take back what I said in Bug #141388 about this problem being a regression
between v0.1 and v0.2

Apparently I did not clean out the target directory between runs of the
different versions, thinking the failure took place during scanning so the
directory would not have been changed yet. Wrong! The first time it runs, it
fails AND deletes a bunch of files and directories. The second time it runs the
directory in which it failed is already *gone*, so it successfully gets through
the deleting phase and unpacks the proper versions of all the missing stuff
from the tar.

So...
the failure message about malloc is totally misleading, with the real error
code from scandir() being ENOENT because it was trying to sort a directory that
had mistakenly been deleted.

Created an attachment (id=92586) [edit]
strcmp fix

Try this please, ought to fix it.

The patch works for me.  A 0.2-r1 revbump with the patch is now in the tree.