Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 414401

Summary: net-misc/openssh-6.0_p1[hpn] server triggers hangs in connections
Product: Gentoo Linux Reporter: Sean McGovern <gseanmcg>
Component: [OLD] Core systemAssignee: Gentoo's Team for Core System packages <base-system>
Status: RESOLVED FIXED    
Severity: normal CC: 96bd763529be62c7350d43e8ae67f9623c998ddc, alecm_88, alexanderyt, binki, bircoph, casta, creideiki+gentoo-bugzilla, david+gentoo.org, Dessa, doug.hunley, gem, gentoo, gmurray, hans, hoea, jon, jwbraun, kripton, Martin.vGagern, neil, nelchael, nholland, opensource, pageexec, polynomial-c, raimue, rei4dan, rob, rose, rsa4046, saintdev, shinydoofy, slashbeast, syscon780, tdalman, tgurr, tom, wolfpythonlondon, yamadharma
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=455524
Whiteboard:
Package list:
Runtime testing required: ---

Description Sean McGovern 2012-05-02 18:23:27 UTC
Not sure why this is happening, but ssh connections using -T (disable pseudo-tty allocation) as part of a batch process is hanging. I reverted back to 5.9 and masked it in portage as I need ssh to work on this machine but I thought I should definitely file a bug as something has changed between the 2 revisions that broke this for me.

Reproducible: Always

Steps to Reproduce:
1. emerge =net-misc/openssh-6.0
2. use an ssh connection with -T

Actual Results:  
ssh process appears to hang.

Expected Results:  
ssh process should complete and close down.
Comment 1 Nathan Caldwell 2012-05-03 07:49:33 UTC
I can confim 6.0 breaks scp for me. I was running 6.0 on both the client and server, scp would stall after only copying a few kilobytes. After downgrading back to 5.9 for both the client and server, files again transfer normally.

Before:
$ scp DVD.iso server:/srv/media/backups/
DVD.iso      0%  256KB   0.0KB/s  99.2KB/s - stalled -^CKilled by signal 2.

After:
$ scp DVD.iso inti:/srv/media/backups/
DVD.iso    100% 2858MB  45.4MB/s  51.3MB/s   01:03

Client emerge --info:
Portage 2.2.0_alpha101 (default/linux/amd64/10.0/desktop/kde, gcc-4.5.3, glibc-2.15-r1, 3.3.0-bfs420.0 x86_64)
=================================================================
System uname: Linux-3.3.0-bfs420.0-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q9300_@_2.50GHz-with-gentoo-2.1
Timestamp of tree: Wed, 02 May 2012 14:30:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [disabled]
ccache version 3.1.7 [enabled]
app-shells/bash:          4.2_p24-r1
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.3-r2, 3.2.3-r1
dev-util/ccache:          3.1.7
dev-util/cmake:           2.8.7-r5
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.1
sys-apps/openrc:          0.9.9.3
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.4_p6-r1, 1.9.6-r3, 1.11.5
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            4.5.3-r2
sys-devel/gcc-config:     1.7
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.3 (virtual/os-headers)
sys-libs/glibc:           2.15-r1
Repositories: gentoo x-portage shellbeach chrome-overlay x-kernels x-bfs-overlay msp430 x-xiph dstien x-moonlight forked-daapd x-dolphin-emu mythtv x-hdhomerun sage-on-gentoo
Installed sets: 
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA dlj-1.1 PUEL AdobeFlash-10.3 skype-eula google-chrome Oracle-BCLA-JavaSE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0 /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=core2 -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--nospinner"
FEATURES="assume-digests binpkg-logs ccache distlocks ebuild-locks fixlafiles news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox xattr"
FFLAGS=""
GENTOO_MIRRORS="http://mirror.usu.edu/mirrors/gentoo/ http://gentoo.mirrors.tds.net/gentoo"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_US"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage /usr/local/overlays/shellbeach /usr/local/overlays/chrome /usr/local/overlays/kernels /usr/local/overlays/bfs-overlay /usr/local/overlays/msp430-overlay /usr/local/overlays/xiph /usr/local/overlays/spotify /usr/local/overlays/moonlight /usr/local/overlays/forked-daapd /usr/local/overlays/dolphin-emu /usr/local/overlays/mythtv/Gentoo /usr/local/overlays/hdhomerun /usr/local/overlays/sage-on-gentoo"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 aac acl acpi alsa amd64 amr amrnb amrwb avahi bash-completion berkdb branding bzip2 cairo cdda cli consolekit cracklib crypt cxx dbus declarative dirac dri dts dvb dvd dvdr emboss encode exif fam fbcondecor ffmpeg fftw flac fltk fortran gdbm gdu gif gphoto2 gpm gstreamer gtk htmlhandbook iconv ieee1394 inotify ios ipv6 java jpeg kde kerberos kipi kvm lapack lcms libass libnotify libv4l2 lirc live lm_sensors lzma lzo mad matroska mdadm mmap mmx mmxext mng modules mp3 mp4 mpeg mudflap multilib musicbrainz mythtv ncurses nls nptl nptlonly ogg openexr opengl openmp openssl pam pango pcre pdf phonon plasma png policykit ppds pppd qt3support qt4 qwt rar readline redland rtmp samba schroedinger sdl semantic-desktop session spell sqlite sqlite3 srt sse sse2 sse3 sse4 sse4_1 ssl ssse3 startup-notification subversion svg sysfs tcpd theora threads thumbnail tiff truetype udev udisks unicode usb v4l2 vaapi vdpau vnc vorbis vpx webkit wxwidgets x264 xattr xcb xcomposite xinerama xml xorg xscreensaver xulrunner xv xvid xvmc zeroconf zip zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US" LIRC_DEVICES="inputlirc" PHP_TARGETS="php5-3" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="radeon r600" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

Server emerge --info:
Portage 2.1.10.57 (default/linux/amd64/10.0/server, gcc-4.5.3, glibc-2.15-r1, 3.2.5-bfs416.0 x86_64)
=================================================================
System uname: Linux-3.2.5-bfs416.0-x86_64-AMD_Athlon-tm-_II_X2_255_Processor-with-gentoo-2.1
Timestamp of tree: Wed, 02 May 2012 14:30:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [disabled]
ccache version 3.1.7 [enabled]
app-shells/bash:          4.2_p24-r1
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.3-r2, 3.2.3-r1
dev-util/ccache:          3.1.7
dev-util/cmake:           2.8.7-r5
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.1
sys-apps/openrc:          0.9.9.3
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.11.5
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            4.5.3-r2
sys-devel/gcc-config:     1.7
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.3 (virtual/os-headers)
sys-libs/glibc:           2.15-r1
Repositories: gentoo x-portage shellbeach x-vbox x-bfs-overlay x-xiph forked-daapd chrome-overlay mythtv x-hdhomerun
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA PUEL dlj-1.1"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=amdfam10 -pipe -O2"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/NX/etc /usr/NX/home /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo"
CXXFLAGS="-march=amdfam10 -pipe -O2"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--nospinner"
FEATURES="assume-digests binpkg-logs ccache distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox"
FFLAGS=""
GENTOO_MIRRORS="http://gentoo.mirrors.tds.net/gentoo http://mirror.mcs.anl.gov/pub/gentoo/ http://gentoo.cites.uiuc.edu/pub/gentoo/ http://gentoo.osuosl.org/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_US"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage /usr/local/overlays/shellbeach /usr/local/overlays/vbox /usr/local/overlays/bfs-overlay /usr/local/overlays/xiph /usr/local/overlays/forked-daapd /usr/local/overlays/chrome /usr/local/overlays/mythtv/Gentoo /usr/local/overlays/hdhomerun"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext X a52 aac acl alsa amd64 avahi avi berkdb bzip2 caps cdda cddb cdr cli consolekit cracklib crypt curl cxx dbus device-mapper dri dts dv dvb dvd dvdr dvdread encode exif expat fam fame fbcondecor ffmpeg fftw flac fortran gd gdbm gdu gif gnutls gpm gstreamer gtk gtk2 hal iconv ieee1394 imagemagick ipv6 java jpeg kde kdeenablefinal kdehiddenvisibility kerberos lame lcms lm_sensors logrotate lzma mad mdadm mmap mmx mmxext mng modules mozilla mp3 mpeg mpeg4 mudflap multilib mysql mythtv ncurses nls nptl nptlonly ogg openmp pam pcre php pic png policykit pppd qt qt3support quicktime readline redland samba sasl semantic-desktop server session snmp spell sql sqlite sse sse2 sse3 ssl subtitles svg sysfs tcpd threads tidy transcode truetype udev udisks unicode v4l v4l2 vorbis webkit x264 xfs xine xml xml2 xorg xulrunner xvid zeroconf zlib" ALSA_CARDS="intel8x0 bt87x" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers imagemap include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif so status unique_id userdir usertrack vhost_alias" APACHE2_MPMS="worker" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="canon logitech" COLLECTD_PLUGINS="cpu cpufreq df disk hddtemp interface load mbmon memory netlink network ntpd rrdcached rrdtool sensors swap thermal tail" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="nouveau radeon savage siliconmotion fbdev vesa vga" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Comment 2 Guillaume Castagnino 2012-05-03 08:06:41 UTC
rsync over ssh (rsync -e ssh) hangs to.

Just to notice: downgrading server side to 5.9 is sufficient to make it works again. It seems not to be a client issue.
Comment 3 Nils Holland 2012-05-04 19:59:17 UTC
Here on x86 I can confirm strange rsync behavior with openssh-6.0 as well. Particularly, it seems that when the machine that receives the files via rsync is running openssh-6.0, things will hang, regardless of what version the machine that "pushes" the files runs (this also happens when both source and target machine run openssh-6.0).

What seems to work for me under all circumstances is using rsync in a "fetch" mode of operation, i.e. running it on the target machine and "fetching" the files from the source, instead of running it on the source and "pushing" the files to the target. That seemed to work with any combination of 6.0 and 5.x versions I had around, so at least there seems to be an easy workaround (at least easy in my case. ;-))
Comment 4 SpanKY gentoo-dev 2012-05-04 22:16:25 UTC
post `emerge -pv openssh`.  if people are using hpn, try turning it off.
Comment 5 Robert Förster 2012-05-04 22:27:24 UTC
without hpn useflag it seems to be fine indeed.

[ebuild   R    ] net-misc/openssh-6.0_p1  USE="ldap pam tcpd -X -X509 -hpn -kerberos -libedit (-selinux) -skey -static" 0 kB
Comment 6 Sean McGovern 2012-05-04 22:38:01 UTC
hanyuu ~ # emerge -pv =openssh-6.0_p1

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U #] net-misc/openssh-6.0_p1 [5.9_p1-r4] USE="hpn ldap pam tcpd -X -X509 -kerberos -libedit (-selinux) -skey -static" 0 kB

Total: 1 package (1 upgrade), Size of downloads: 0 kB

The following mask changes are necessary to proceed:
#required by =openssh-6.0_p1 (argument)
# /etc/portage/package.mask:
=net-misc/openssh-6.0_p1

NOTE: The --autounmask-keep-masks option will prevent emerge
      from creating package.unmask or ** keyword changes.


... so yes it looks like I have hpn enabled.
Comment 7 Nathan Caldwell 2012-05-05 01:30:36 UTC
(In reply to comment #4)
> post `emerge -pv openssh`.  if people are using hpn, try turning it off.

It appears I also have it enabled.

# emerge -pv =openssh-6.0_p1

These are the packages that would be merged, in order:

Calculating dependencies  ..... ... ... ... done!
[ebuild     U #] net-misc/openssh-6.0_p1 [5.9_p1-r4] USE="X hpn kerberos pam tcpd -X509 -ldap -libedit (-selinux) -skey -static" 0 kB
Total: 1 package (1 upgrade), Size of downloads: 0 kB

The following mask changes are necessary to proceed:
#required by =openssh-6.0_p1 (argument)
# /etc/portage/package.mask:
=net-misc/openssh-6.0_p1

NOTE: The --autounmask-keep-masks option will prevent emerge
      from creating package.unmask or ** keyword changes.
Comment 8 Nils Holland 2012-05-05 06:50:58 UTC
Indeed, hpn was was enabled here, and re-emerging without hpn seems to fix this, according to the first few tests I've made.
Comment 9 Marios Andreopoulos 2012-05-06 02:09:39 UTC
I can confirm that this happens to me too on amd64.
My problem is with sftp file transfers.

Emerging without hpn indeed solves the problem.

Since so many people report that openssh 6.0_p1 breaks scp/sftp transfers, maybe the hpn flag shouldn't be set as default in the ebuild or at least a warning should be added.
Comment 10 August Hörandl 2012-05-08 17:39:33 UTC
I had a "hanging" unison over ssh (amd64). I fixed it by changing just the TcpRcvBufPoll Option

/etc/ssh/sshd_config
# the following are HPN related configuration options
# tcp receive buffer polling. disable in non autotuning kernels
#TcpRcvBufPoll yes
TcpRcvBufPoll  no

Is there a kernel config for this "autotuning"?
Comment 11 Sergey Kvachonok 2012-05-09 11:50:59 UTC
'TcpRcvBufPoll  no' fixed the hangs for me as well.
I'm running openssh 6.0p1 with hpn enabled.
Comment 12 Derk W te Bokkel 2012-05-11 13:58:45 UTC
I have tried the -hpn fix but this discussion also seems to effect files transfers
and I found it to work as well when (hpn was set) .. is there some interection between these two issues that may be relevent?

http://forums.gentoo.org/viewtopic-t-923080-highlight-.html

openssl version effecting openssh transfers as well ...
Comment 13 Matt Whitlock 2012-05-11 22:27:01 UTC
I have a cronjob that performs a nightly incremental backup over an SSH connection by piping output from xfsdump into "ssh <user>@<server-host> 'cat > <backup-path>'". That is, the ssh client executes 'cat' on the server with a redirection into a file. This has worked for years, but upon upgrading the server to openssh-6.0_p1 (from 5.9_p1-r4), the transfer hangs every time. The hang has occurred every night since I upgraded the server. I have to manually SIGTERM the 'ssh' client process every time.

The bizarre thing is that running the same backup command from a terminal during the daytime has worked every time; I have only seen the hang when the command runs from cron (i.e., without an attached terminal).

I tried setting ServerAliveInterval to 15 seconds in the client's SSH config, but that had no effect on the hang. I am going to try disabling TcpRcvBufPoll on the server to see if that resolves the problem.

 * Found these USE flags for net-misc/openssh-6.0_p1:
 U I
 + + X        : Adds support for X11
 - - X509     : Adds support for X.509 certificate authentication
 + + hpn      : Enable high performance ssh
 - - kerberos : Adds kerberos support
 - - ldap     : Add support for storing SSH public keys in LDAP
 - - libedit  : Use the libedit library (replacement for readline)
 - - pam      : Adds support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip
 - - skey     : Enable S/Key (Single use password) authentication support
 - - static   : !!do not set this during bootstrap!! Causes binaries to be statically linked instead of dynamically
 - - tcpd     : Adds support for TCP wrappers

 * Searching for openssl ...
[IP-] [  ] dev-libs/openssl-0.9.8w:0.9.8
[IP-] [  ] dev-libs/openssl-1.0.1b:0
Comment 14 Matt Whitlock 2012-05-11 22:50:37 UTC
(In reply to comment #13)
> The bizarre thing is that running the same backup command from a terminal
> during the daytime has worked every time; I have only seen the hang when the
> command runs from cron (i.e., without an attached terminal).

I think I spoke too soon. The SSH connection by my cronjob has been hanging at a bit over 100 KB into the transfer, whereas my manual backup has been completing, but today my manual backup got to 413 MB and is now hung. So I guess the presence or absence of a tty is not important to this bug.
Comment 15 Sean McGovern 2012-05-15 17:15:56 UTC
This bug is definitely not limited to PPC.
Comment 16 Michael Weber (RETIRED) gentoo-dev 2012-05-24 13:56:40 UTC
*** Bug 417373 has been marked as a duplicate of this bug. ***
Comment 17 Joseph 2012-05-24 15:45:12 UTC
It seems to me the best possible solution would be to mask "openssh-6.0" untill this issue is solved.
Comment 18 Juergen Rose 2012-05-25 08:16:43 UTC
Turning of hpn removes the issue for me.
Comment 19 Piotr Karbowski (RETIRED) gentoo-dev 2012-05-26 17:37:37 UTC
I think either 6.0 should be masked or just mask hpn useflag on 6.0. I am hitting it on x86_64 boxes, thats critical as the connections hangs.
Comment 20 Robin Bankhead 2012-06-05 11:42:08 UTC
Glad this has been pinpointed, I was having these issues with unison, rsync and (possibly) scp in recent weeks. Not done a comprehensive test yet but TcpRcvBufPoll = no has allowed consistent operation so far.

Some things I've observed while doing my own testing:

- When using unison and rsync, the hangs seemed to be caused by transferring certain files in particular. (Unfortunately I can't share these for business reasons but am trying to investigate further.)

- In these cases, invoking ssh with "-C" (compression) option seemed to help, though it did not prevent hanging in every case. This could be down to the apparent "bad file" (or rather some "bad byte" within it) being obfuscated, at least in part.

- The scp case was one I'd not considered as linked to the others until now, but I did experience stalling when transferring over wifi with poor signal. The same operation over 10/100 wired connection went fine, and the unison job fails less often over LAN than WAN. Perhaps link quality/bandwidth is also an issue?
Comment 21 Robin Bankhead 2012-06-05 15:25:46 UTC
Update: "TcpRcvBufPoll = no" did NOT fix the issue in the situation I originally encountered it (win7 native unison with cygwin ssh). Again, it hangs on the same file as before.  Furthermore, having to bail from the program leaves it unable to complete a subsequent run even if I re-add the -sshargs="-C" flag to the unison profile; I then have to delete the archive-files for the profile at both ends and run the profile "from scratch" (and with -C) in order to complete it.

(Sidenote: I assume here that /etc/init.d/sshd restart will have enacted the changed sshd_config; please advise if that's wrong.)

I'll try rebuilding openssh with -hpn to see if that makes a difference, but the TcpRcvBufPoll directive alone certainly hasn't nailed it in my case.
Comment 22 Robin Bankhead 2012-06-05 15:44:58 UTC
OK, USE="-hpn" has (after first trial) solved it here too.

Just to clarify my previous post, Win7 native unison and cygwin ssh is the client end; server is openssh-6.0_p1 on Gentoo amd64.  Cygwin rsync over ssh on the same client was puking on the same file too.
Comment 23 jon 2012-06-07 17:32:02 UTC
Although no real description is given here this sounds like what we are observing with rsync and sftp with openssh-5.9_p1-r4 and 6.0_p1. File transfers randomly hang until the client is killed. They hang more often then naught.
Comment 24 jon 2012-06-07 17:37:44 UTC
(In reply to comment #23)
> Although no real description is given here

Sorry about that I just didn't notice there was a whole lot more scrolling to do.

We have these issues going from Gentoo to Gentoo with same version client and server plus we have other clients running different versions of OpenSSH on different platforms which also have the problem.
Comment 25 Samuli Suominen (RETIRED) gentoo-dev 2012-06-07 17:49:14 UTC
(In reply to comment #23)

Why would you CC amd64@ here? Nothing for them to do here. Removing from CC.
Comment 26 Martin von Gagern 2012-06-07 19:17:41 UTC
(In reply to comment #23)
> Although no real description is given here this sounds like what we are
> observing with rsync and sftp with openssh-5.9_p1-r4 and 6.0_p1.

Also affects unison, even when that doesn't use rsync, so add that to the list.

With unison, at least for me it only affected one side of the communication: for a link established from Windows TO the Gentoo machine, only the bulk data transfer in the SENDING direction, i.e. TOWARDS the Gentoo machine, was affected. Even when that got stalled, the other direction seemed to work as intended. Perhaps this helps.
Comment 27 SpanKY gentoo-dev 2012-06-08 05:43:12 UTC
should be all set now in the tree; thanks for the report!

Commit message: Back hpn patch back down to v11 as v12 does not want to work for us
http://sources.gentoo.org/net-misc/openssh/openssh-6.0_p1-r1.ebuild?rev=1.1
Comment 28 Gary E. Miller 2012-06-08 16:54:23 UTC
(In reply to comment #27)

> Commit message: Back hpn patch back down to v11 as v12 

How old is patch v12?  This bug got much more obvious recently but I think I
have been seeing it intermittently for about a year.  

For a test last week I downgraded to openssh-5.9_p1-r4, which uses hpn v11,
and the problem persisted.

So I am reluctant to call this one fixed.
Comment 29 Nathan Phillip Brink (binki) (RETIRED) gentoo-dev 2013-02-20 16:06:02 UTC
(In reply to comment #28)
> (In reply to comment #27)
> 
> > Commit message: Back hpn patch back down to v11 as v12 
> 
> How old is patch v12?  This bug got much more obvious recently but I think I
> have been seeing it intermittently for about a year.  
> 
> For a test last week I downgraded to openssh-5.9_p1-r4, which uses hpn v11,
> and the problem persisted.

I am also noticing this in piping files to a program invoked on an openssh-5.9_p1-r4[hpn] server (from an outdated openssh-5.8_p1-r1[hpn] client). I can’t control the server so I’m using the “Compression yes” workaround for this host and that seems to reliably skirt the issue.