receiving file list ... 82476 files to consider rsync: connection unexpectedly closed (1842857 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(342) Reproducible: Always Steps to Reproduce: 1. emerge net-misc/rsync-2.6.2-r2 2. emerge sync Actual Results: above described error Occured on P233, PIII 500, Athlon 800, AMD64 so it's not configuration related except for: -acl -build -static that show when doing emerge -vp net-misc/rsync I can post emerge info from all those systems, but I don't think it's that usefull I can work around it by emerging 2.6.0
I've the same here. Even with net-misc/rsync-2.6.2-r1
also occurs on alpha according to kloeri
It doesn't have to do with incompatibility between 2.6.2 and 2.6.0. I've tested with an 2.6.2-r2 rsync server (my own), and I got the same results
I get this too... We should pull 2.6.2 until this is fixed... Anyways the security bug that is fixed by 2.6.2 is for a pretty rare configuration.. While we break a very very very common use on Gentoo... I know solar is not going to like me here.. And please lets give a little more testing to stuff even if they are security patches..
This issue is being handled. At present time, 2.6.2 has been added to the package.mask file, so users should stay at 2.6.0 for the time being. -jeffrey
which ISP connection do you have ?
*** Bug 50072 has been marked as a duplicate of this bug. ***
taking bug since no one else is ... i run a local rsync server myself for my development LAN ... the server is a hppa box that was running rsync-2.6.2-r2 (yes i /etc/init.d/rsync restart to make sure) the clients are x86/arm ... both were running 2.6.2-r2 and had no problems syncing up ... for fun, i downgraded the x86 to 2.6.0-r1, but i still had no problems syncing against my server ...
Can anyone post a client/server combination, including arch, that they know craps out, including the accompanying error message from the client. So that we may better isolate what is going on..Thanks! -Jeffrey
I've had problems with: x86 - gentoo mirror ftp.snt.utwente.nl x86 - x86 amd64 - gentoo-mirror same as above amd64 - x86
this happens to me too, had to downgrade to 2.6.0 to be able to do a emerge sync again
which ISP connection do you have ? i've no problem in LAN with enough bandwith
x86 - x86 and x86-amd64 is on a 100mbit switched lan. the gentoo mirror is across a 8mbit down, 1 mbit up link
It doesn't have anything to do with bandwidth - I'm seeing this on one box when syncing against a local mirror across a switched 100mbit network with no other traffic.. I am however having a bit succes as I created a fresh chroot on that box with a stage3 tarball and the chroot doesn't exhibit this problem while the real root does show the problem. I'm using the exact same CFLAGS, CXXFLAGS and versions of rsync and portage.
uhh, i dont know about ftp.snt.utwente.nl ... it's running an old version of rysnc: root@vapier 0 root # nmap -sV -p 873 ftp.snt.utwente.nl PORT STATE SERVICE VERSION 873/tcp open rsync (protocol version 26)
I tried to monitor the net traffic during the sync with Ethereal, I didn't understand so much, the connection ends with row data.. Versions: client @RSYNCD: 28 server @RSYNCD: 27 _my net connection is an adsl 640/256 _the rsync server was rsync.gentoo.org _mine arch is x86 too
i've added -r3 with ipv6 use flag please try it
IPv6 might be the problem why it works with some machines, and doesnt with other. I have one x86 machine where -r2 works successfully. That machine rsyncs via ipv4, the rest via ipv6
-r3 doesn't fix the problem on my boxes - btw. I don't use ipv6 on any of those boxes. After a lot of fiddling around I discovered that the problem happens if I mount some disk at /usr/portage and goes away if I umount the disk again. Unfortunately I have no idea why this behaviour changed from 2.6.0 to 2.6.2.
ok, can others who've experienced this, please tell us which mirrors in particular are being shady? obviously, syncs against ftp.snt.utwente.nl are failing (and as Spanky said, that's running an ancient version of rsync).
Every mirror I managed to hit with my affected boxes including my own local mirror . It doesn't seem to change anything whether my local mirror runs 2.6.0 or 2.6.2.
kloeri: do you use hdparm tunings ? maybe your harddisc is running in PIO mode
It happens on a variety of boxes, some are scsi-only boxes, some are ide boxes with dma enabled, some have external idedisks connected via usb. I decided to spend some time testing different filesystems and I can only reproduce the error when using ext3. Emerge sync worked as expected when I used ext2, reiserfs, xfs or jfs. Ext3 works as expected as long as I don't mount an ext3 partition on /usr/portage - that it to say if / is one large ext3 partition I don't have any problems. I don't see any obvious permission problems as it works flawlessly with rsync-2.6.0 using the exact same setups. I should also state that I'm running different kernels - mostly 2.6 kernels but one box runs a 2.4.2x kernel.
I had problems with one huge ext3 partition for /... (I dont have my laptop with me right now so I can't do more tests.)
ftp.snt.utwente.nl is running debian stable with rsync 2.5.6cvs patched with all security fixes. Our server maintainer will examine this weekend.
Has anyone found the issue with this bug? I'm getting more admins who arent able to sync... @ERROR: auth failed on module gentoo rsync: connection unexpectedly closed (894 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(189) End: Thu May 20 18:50:15 CEST 2004
For reference, this bug blocks a security bug (vulnerability in rsync < 2.6.2), so I raise priority...
Mike -- what's the status on this bug?
this seems to look like http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=158170 anyways ... i'm adding 2.6.1 shortly; could you guys update to that and see if it falls apart for you too ?
2.6.1 breaks for me also. Tested from Home ADSL to rsync servers, and also inside a LAN at work.
Well, after deleting completely /usr/portage/ and running emerge sync again, it worked.
I'm still seeing this problem with 2.6.1 however it's unmasked for ~x86. Is this right?
can you test the idea that deleting /usr/portage/ and syncing again 'fixes' this on one of your machines ?
deleting /usr/portage/ and syncing again doesn't work for me
receiving file list ... 87880 files to consider rsync: connection unexpectedly closed (2123616 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(342) That's since I installed 2.6.1 PORTDIR=/tmp/portage emerge sync doesn't solve it at all.
ok, 2.6.2* didn't solve it, but 2.6.0-r1 works.
Same here. On my desktop home machine rsync doesn't work at all anymore. I don't consider IPv6 as a problem as both my machines -- server and desktop -- are IPv6-enabled but only the desktop fails. websync works, and I'll switch back to 2.6.0-r1 now. Portage 2.0.50-r8 (default-x86-1.4, gcc-3.3.3, glibc-2.3.3_pre20040420-r0, 2.6.5-gentoo-r1) ================================================================= System uname: 2.6.5-gentoo-r1 i686 AMD Athlon(tm) XP 2100+ Gentoo Base System version 1.4.16 ccache version 2.3 [enabled] Autoconf: sys-devel/autoconf-2.59-r4 Automake: sys-devel/automake-1.8.5-r1 ACCEPT_KEYWORDS="x86 ~x86" AUTOCLEAN="yes" CFLAGS="-O3 -march=athlon-xp -pipe -fomit-frame-pointer -frerun-loop-opt -falign-functions=4 -fforce-mem -funroll-loops -ffast-math -finline-functions -foptimize-sibling-calls -m3dnow -mmmx" CHOST="i686-pc-linux-gnu" COMPILER="gcc3" CONFIG_PROTECT="/etc /etc/tomcat /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.1/share/config /usr/kde/3.2/share/config /usr/kde/3/share/config /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O3 -march=athlon-xp -pipe -fomit-frame-pointer -frerun-loop-opt -falign-functions=4 -fforce-mem -funroll-loops -ffast-math -finline-functions -foptimize-sibling-calls -m3dnow -mmmx -Wno-deprecated" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs ccache sandbox" GENTOO_MIRRORS="http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://ftp.easynet.nl/mirror/gentoo/" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.de.gentoo.org/gentoo-portage" USE="3dnow X aalib acl acpi alsa apache2 apm avi berkdb bonobo cdr crypt cups dvd encode esd foomaticdb gd gdbm gif gimpprint gnome gpm gtk gtk2 gtkhtml guile imap imlib innodb ipv6 jack java jikes jpeg kde ldap libg++ libwww mad maildir mbox memlimit mikmod mmx mozilla moznocompose moznoirc mpeg mysql ncurses nls nowin oggvorbis opengl pam parse-clocks pdflib perl png postgres ppds python qt quicktime readline ruby sdl slang spell ssl svga tcltk tcpd tetex truetype unicode vim-with-x x86 xml2 xmms xv xvid zlib"
I have exacly the same proble as everyone is describing here but as opposed to everyone I am still using rsync-2.6.0 so the problem may probably be coming from the new servers. Here is my emerge info anyway: Gentoo Base System version 1.4.16 Portage 2.0.50-r8 (default-x86-1.4, gcc-3.3.3, glibc-2.3.3.20040420-r0, 2.6.7-gentoo-r6) ================================================================= System uname: 2.6.7-gentoo-r6 i686 Pentium II (Deschutes) Autoconf: sys-devel/autoconf-2.59-r3 Automake: sys-devel/automake-1.8.3 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-Os -march=i386" CHOST="i686-pc-linux-gnu" COMPILER="gcc3" CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-Os -march=i386" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs ccache sandbox" GENTOO_MIRRORS="http://gentoo.mirrors.pair.com/" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X apm avi berkdb crypt dillo directfb encode escreen fbcon fbdev foomaticdb freetype gdbm gif gphoto2 gpm gtk gtk2 imlib java jpeg libg++ libwww mad mmx motif moznocompose moznoirc moznomail mpeg ncurses nls offensive oggvorbis oss pam pcmcia pdflib perl png python quicktime readline sdl slang spell ssl stroke tcltk tcpd truetype vim-with-x x86 xml2 xmms xv zlib" I am starting an emerge-webrsync now, I will post back about this later
Everything is now fine after upgrading to 2.6.0-r2...
How does this affect 2.6.1?
http://www.samba.org/rsync/issues.html
2.6.3_pre2 is in portage (but package.mask-ed) so if you guys could give it a test run and see if it helps at all, that'd be awesome
released 2.6.3 final ... no one got back to me so i just unmasked it
no one seems to be complaining with 2.6.3 ;)
sudenly it doesnt work metadata/timestamp.x 40 100% 0.16kB/s 0:00:00 (9, 72.5% of 101903) net-misc/rsync/rsync-2.6.3.ebuild 2057 100% 7.76kB/s 0:00:00 (10, 81.1% of 101903) rsync: connection unexpectedly closed (2247880 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(359) I tried both available versions(2.6.0-r3,2.6.3), but dont work
After emerging rsync-2.6.3 a few days ago emerge sync fails every time after a timeout and with output similar to the following: metadata/cache/dev-games/hlsdk-2.3 241 100% 0.29kB/s 0:00:00 (25, 56.6% of 102698) rsync error: timeout in data send/receive (code 30) at io.c(153) rsync: connection unexpectedly closed (2388581 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(359) rsync: connection unexpectedly closed (2377929 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(359) >>> retry ... After going back to 2.6.0 emerge sync works again. I have a slow link (ISDN), reiserfs and udma.
you didnt say what you have RSYNC_TIMEOUT set to in make.conf if you increase that value, does rsync work ?
SpanKY, RSYNC_TIMEOUT normally is 180, but even increasing to 300 did not help. What does RSYNC_TIMEOUT have to do with the problem, anyway? AFAICS the connections are ok. It seems to be an rsync software problem, doesn't it?
your error message says it was a timeout problem: rsync error: timeout in data send/receive (code 30) at io.c(153)
SpanKY, I know that rsync says it's a timeout but there is no reason for rsync to time out: 1. It is not a connection problem. I can happily browse the internet while rsync is running. 2. With rsync-2.6.0 I NEVER have timeouts (as far as I can remember) whereas rsync-2.6.3 ALWAYS times out. I am going to try rsync-2.6.0-r3 today.
First emerge sync with rsync-2.6.0-r3 completed without errors.
I've been seeing some of these errors being reported on the -mirrors mailing list. I'll have them look at this bug. I'm not seeing a public archive, or I'd point you to that. This is was reported on the mailing list recently: rsync error: timeout in data send/receive (code 30) at io.c(153) rsync: connection unexpectedly closed (586418 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(359) rsync: connection unexpectedly closed (586418 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(359)
the first FAQ entry at http://rsync.samba.org/issues.html talks about this in a general sense (since the error is a general error)
I'm also getting this problem with rsync-2.6.0-r3 and rsync-2.6.3. There are a lot of people on the forums who have been seeing this as well. Some of the threads can be seen here: http://forums.gentoo.org/viewtopic.php?p=1994196#1994196 http://forums.gentoo.org/viewtopic.php?p=1989637#1989637 http://forums.gentoo.org/viewtopic.php?p=1994218#1994218 Perhaps rsync-2.6.3 should be masked again? Although with the security problem on lower versions of rsync I'm not sure what the best solution is until the newer versions of rsync are fixed.
*** Bug 78637 has been marked as a duplicate of this bug. ***
*** Bug 78756 has been marked as a duplicate of this bug. ***
I noticed 2.6.3 was just masked. Is rsync-2.6.0-r3 patched with all the latest security issues? And has there been any followup to whats causing all these problems?
rsync-2.6.0 wouldnt be in the tree if it had known security issues still :P besides, not everyone had 2.6.3 marked stable, so not everyone got downgraded in terms of this bug, i havent been able to reproduce with 2.6.3, and i'm really not looking into it either
:( that's a shame. Still reproduced on 2.6.3 on my laptop here. I believe (some evidence from this in the forums on this topic) that some people will move away from gentoo if they cannot update systems reliably. If this bug is caused by a combination of an upstream weakness and our syncing method then it may be difficult to pin down i guess... I hope that it will sort itself out or that someone can figure it out!
I don't think that deleting entries from the Changelog is the correct way to reflect what happened with this package. Between yesterday sync and today sync all entries newer than 21 Nov 2004 were deleted. But that isn't the same as an upgrade followed by a downgrade... Why isn't it possible to add an entry which tells about the downgrade and its reasons? And for the failures: I got these failures with rsync-2.6.0 when trying to sync two machines at the same time with my local mirror. The weaker machine failed almost everytime, while the stronger machine succeded. I solved this problem by syncing the machines one after the other, for then the machines succeeded reliably. I tried to track the problem using strace but then all machines succeeded. With 2.6.3 I was able to sync the machines at the same time from 28 Dec 2004 till today with only one failure yet.
I strongly object to messing with changelog this way, too. This is very confusing and bad practice, IMNSHO. :-(
I have this error ever since... But... I mostly see it when the memory of the receiving system is highly used. When I then close these memory eating applications rsync works again as expected. Here is my recent error message when doing an "emerge sync" to a local server. >>> Starting retry 3 of 3 with rsync://192.168.200.1/gentoo-portage >>> checking server timestamp ... receiving file list ... 1 file to consider Number of files: 1 Number of files transferred: 0 Total file size: 32 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 32 Total bytes written: 203 Total bytes read: 68 wrote 203 bytes read 68 bytes 542.00 bytes/sec total size is 32 speedup is 0.12 receiving file list ... 108387 files to consider rsync error: timeout in data send/receive (code 30) at io.c(109) rsync: connection unexpectedly closed (2551489 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(189) rsync: connection unexpectedly closed (2551489 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(189) !!! Rsync has not successfully finished. It is recommended that you keep !!! trying or that you use the 'emerge-webrsync' option if you are unable !!! to use rsync due to firewall or other restrictions. This should be a !!! temporary problem unless complications exist with your network !!! (and possibly your system's filesystem) configuration.
I thought I would update the bug with this info: http://forums.gentoo.org/viewtopic-p-2286988.html#2286988
*** Bug 90394 has been marked as a duplicate of this bug. ***
I've tried vanilla rsync-2.6.4 (i.e. compiled from source using ./configure; make; cp ./rsync /usr/bin), and it still has this problem.
Hey, interesting result! I tried a recent rsync snapshot, and "emerge sync" worked. The last approx. ten times I've tried with other rsync versions, including vanilla 2.6.4, have failed every time. Perhaps this means the bug in rsync is fixed now? (It's always possible that this time it was lucky, but it's been consistently failing for me on my portage tree for a while now, even after emerge-webrsync). The snapshot is: rsync-HEAD-20050424-2235GMT it reports itself as: rsync version 2.6.5cvs protocol version 29 Copyright (C) 1996-2005 by Andrew Tridgell and others <http://rsync.samba.org/> Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums
*** Bug 123544 has been marked as a duplicate of this bug. ***
So - anything new here? BTW, could someone finally add metadata.xml to this package?
this bug isnt about moving rsync to stable
I'm currently having this issue between a local rsync mirror and a Gentoo system running in a UML on the same host. It happens every time (for over a week now), so it's easy to reproduce. Both are running Gentoo x86 stable, updated daily (the client is lagging behind now, of course). Increasing timeout on both the client and server to 1800 didn't change anything. Neither did erasing and repopulating the portage tree on the server. On the server, rsyncd is running behind tcpserver and is configured to use chroot and drop privileges. The directory containing the rsync modules is on a tmpfs. On the client, /usr is on an ext3 filesystem. Client and server communicate via IPv4, but the kernels are IPv6 enabled. --- While writing this comment I noticed swap hadn't been activated on boot on the UML host due to a configuration error. After activating it, the sync worked fine! So some cases might be due to rsync having problems when running with few memory. But not in all cases, since it also happens sporadically on other systems with enough swap. On those other systems, it might have something to do with files left corrupted by a failed upstream sync. It might be useful to investigate why it's failing in the low memory case and how to make it print a more meaningful error message in that case. I'm not having the time to do that, though. The stats were obviously wrong, BTW: Number of files: 137670 >Number of files transferred: 0 Total file size: 116868894 bytes >Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 3041621 Total bytes written: 182 Total bytes read: 3041657 wrote 182 bytes read 3041657 bytes 17431.74 bytes/sec total size is 116868894 speedup is 38.42
