mosh fails to connect to a remote host running Gentoo, saying that it requires a UTF-8 locale. This occurs even when the remote host DOES have a UTF-8 locale. Reproducible: Always Steps to Reproduce: Emerge mosh, then attempt to ssh to a gentoo host whose locale has been set to, say, en_US.utf8: $ mosh <hostname> Actual Results: mosh quits with the following message: mosh requires a UTF-8 locale. Connection to <hostname> closed. /usr/bin/mosh: Did not find mosh server startup message. Expected Results: mosh should successfully start a session. mosh checks that the locale on the remote host is UTF-8-based by essentially running `ssh foohost locale`. When foohost is running Gentoo, this does not actually provide the same data as running `locale` from a bash login shell, whether the shell is invoked through ssh or run locally on foohost, seemingly because /etc/profile.env is not sourced by whatever shell is used to call `locale` when `ssh foohost locale` is run on the client machine. One can confirm this by simply running `ssh foohost locale` and noting the difference. A possible solution to this problem might be to wrap the mosh-server executable with a shell script: #!/bin/sh . /etc/profile.env /usr/bin/env mosh-server "$@" or something like that. (Sorry, my shell scripting is terrible.) Note: I've actually only tried this on one remote Gentoo host and one local Gentoo host (i.e. 127.0.0.1), so it could be something wrong with those particular machines, but this does seem to be very similar to a known problem which is mentioned on the mosh web page (in the FAQ) as well as in its github issues 74 and 98: http://mosh.mit.edu/#faq https://github.com/keithw/mosh/issues/74 https://github.com/keithw/mosh/issues/98 Marking this "major" because the entire server portion of mosh becomes unusable unless you're savvy enough to tack on '--server="env LC_ALL=en_US.UTF-8 mosh-server"' or similar to your invocation of mosh on the client side.
I cannot reproduce this on a lot of machines, I'll pass it to upstream. Can you please specify your locale setup? i.e. the output of `ssh remote-host env` and `env` (on the local machine).
And can you please supply your servers /etc/ssh/sshd_config and your clients /etc/ssh/ssh_config, thanks.
Uh oh, that sounds like it was my local problem after all. Well, here goes - just in case, I should mention that my local machine is actually running Funtoo, while the remote host is running actual Gentoo. ---- Commands: fs@boone ~ $ locale fs@boone ~ $ ssh fsvps locale fs@boone ~ $ ssh 0 locale fs@boone ~ $ ssh fsvps sh -lc "locale" fs@boone ~ $ ssh fsvps fs@vps ~ $ locale fs@vps ~ $ exit fs@boone ~ $ cat /etc/ssh/ssh_config fs@boone ~ $ ssh fsvps sudo cat /etc/ssh/sshd_config ---- Full transcript: fs@boone ~ $ locale LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE=POSIX LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= fs@boone ~ $ ssh fsvps locale LANG= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= fs@boone ~ $ ssh 0 locale LANG= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= fs@boone ~ $ ssh fsvps sh -lc "locale" LANG=en_US.utf8 LC_CTYPE="en_US.utf8" LC_NUMERIC="en_US.utf8" LC_TIME="en_US.utf8" LC_COLLATE="en_US.utf8" LC_MONETARY="en_US.utf8" LC_MESSAGES="en_US.utf8" LC_PAPER="en_US.utf8" LC_NAME="en_US.utf8" LC_ADDRESS="en_US.utf8" LC_TELEPHONE="en_US.utf8" LC_MEASUREMENT="en_US.utf8" LC_IDENTIFICATION="en_US.utf8" LC_ALL= fs@boone ~ $ ssh fsvps fs@vps ~ $ locale LANG=en_US.utf8 LC_CTYPE="en_US.utf8" LC_NUMERIC="en_US.utf8" LC_TIME="en_US.utf8" LC_COLLATE="en_US.utf8" LC_MONETARY="en_US.utf8" LC_MESSAGES="en_US.utf8" LC_PAPER="en_US.utf8" LC_NAME="en_US.utf8" LC_ADDRESS="en_US.utf8" LC_TELEPHONE="en_US.utf8" LC_MEASUREMENT="en_US.utf8" LC_IDENTIFICATION="en_US.utf8" LC_ALL= fs@vps ~ $ exit fs@boone ~ $ cat /etc/ssh/ssh_config # $OpenBSD: ssh_config,v 1.26 2010/01/11 01:39:46 dtucker Exp $ # This is the ssh client system-wide configuration file. See # ssh_config(5) for more information. This file provides defaults for # users, and the values can be changed in per-user configuration files # or on the command line. # Configuration data is parsed as follows: # 1. command line options # 2. user-specific file # 3. system-wide file # Any configuration value is only changed the first time it is set. # Thus, host-specific definitions should be at the beginning of the # configuration file, and defaults at the end. # Site-wide defaults for some commonly used options. For a comprehensive # list of available options, their meanings and defaults, please see the # ssh_config(5) man page. # Host * # ForwardAgent no # ForwardX11 no # RhostsRSAAuthentication no # RSAAuthentication yes # PasswordAuthentication yes # HostbasedAuthentication no # GSSAPIAuthentication no # GSSAPIDelegateCredentials no # BatchMode no # CheckHostIP yes # AddressFamily any # ConnectTimeout 0 # StrictHostKeyChecking ask # IdentityFile ~/.ssh/identity # IdentityFile ~/.ssh/id_rsa # IdentityFile ~/.ssh/id_dsa # Port 22 # Protocol 2,1 # Cipher 3des # Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc # MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160 # EscapeChar ~ # Tunnel no # TunnelDevice any:any # PermitLocalCommand no # VisualHostKey no # ProxyCommand ssh -q -W %h:%p gateway.example.com fs@boone ~ $ ssh fsvps sudo cat /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.82 2010/09/06 17:10:19 naddy Exp $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options change a # default value. #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # The default requires explicit activation of protocol 1 #Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key # Lifetime and size of ephemeral version 1 server key #KeyRegenerationInterval 1h #ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO # Authentication: #LoginGraceTime 2m #PermitRootLogin yes #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 #RSAAuthentication yes #PubkeyAuthentication yes #AuthorizedKeysFile .ssh/authorized_keys # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no # similar for protocol version 2 #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # RhostsRSAAuthentication and HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # To disable tunneled clear text passwords, change to no here! PasswordAuthentication no #PermitEmptyPasswords no # Change to no to disable s/key passwords #ChallengeResponseAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes # Set this to 'yes' to enable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. UsePAM yes #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no GatewayPorts yes #X11Forwarding no #X11DisplayOffset 10 #X11UseLocalhost yes PrintMotd no PrintLastLog no #TCPKeepAlive yes #UseLogin no #UsePrivilegeSeparation yes #PermitUserEnvironment no #Compression delayed #ClientAliveInterval 0 #ClientAliveCountMax 3 #UseDNS yes #PidFile /var/run/sshd.pid #MaxStartups 10 #PermitTunnel no #ChrootDirectory none # no default banner path #Banner none # override default of no subsystems Subsystem sftp /usr/lib/misc/sftp-server # the following are HPN related configuration options # tcp receive buffer polling. disable in non autotuning kernels #TcpRcvBufPoll yes # allow the use of the none cipher #NoneEnabled no # disable hpn performance boosts. #HPNDisabled no # buffer size for hpn to non-hpn connections #HPNBufferSize 2048 # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # ForceCommand cvs server
I am able to replicate this problem with mosh with an empty /etc/ssh/{sshd_config,ssh_config} and with ssh to localhost. ~» eix -e openssh [I] net-misc/openssh Available versions: 5.5_p1-r2 5.6_p1-r2 ~5.7_p1-r1 5.8_p1-r1 ~5.8_p2 ~5.8_p2-r1 ~5.9_p1-r3 5.9_p1-r4 {X X509 (+)hpn kerberos ldap libedit pam selinux skey static tcpd} Installed versions: 5.9_p1-r4(01:53:30 AM 04/13/2012)(X hpn ldap pam tcpd -X509 -kerberos -libedit -selinux -skey -static) Homepage: http://www.openssh.org/ Description: Port of OpenBSD's free SSH release ~» eix -e mosh [I] net-misc/mosh Available versions: (~)1.1.3 **9999 {+client examples +server +utempter} Installed versions: 1.1.3(07:10:29 PM 04/14/2012)(client server utempter -examples) Homepage: http://mosh.mit.edu Description: Mobile shell that supports roaming and intelligent local echo ~» sudo service sshd start * Starting sshd ... [ ok ] ~» ssh localhost locale <user>@localhost's password: X11 forwarding request failed on channel 0 LANG= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= # After adding "SendEnv LANG LC_*" to /etc/ssh_config and "AcceptEnv LANG LC_*" to /etc/sshd_config: ~» sudo service sshd reload * Reloading sshd ... [ ok ] ~» ssh localhost locale <user>@localhost's password: X11 forwarding request failed on channel 0 LANG=en_US.UTF8 LC_CTYPE="en_US.UTF8" LC_NUMERIC="en_US.UTF8" LC_TIME="en_US.UTF8" LC_COLLATE="en_US.UTF8" LC_MONETARY="en_US.UTF8" LC_MESSAGES="en_US.UTF8" LC_PAPER="en_US.UTF8" LC_NAME="en_US.UTF8" LC_ADDRESS="en_US.UTF8" LC_TELEPHONE="en_US.UTF8" LC_MEASUREMENT="en_US.UTF8" LC_IDENTIFICATION="en_US.UTF8" LC_ALL=
should mention that my ssh is aliased: ~» alias ssh ssh='TERM=xterm ssh -C -Y'
Solution 1 (assuming your shell is bash): edit the servers ~/.bashrc and add `export LANG=..` before the "if not interacti ve shell return" foo. Solution 2: on the server, edit /etc/ssh/sshd_config and add `AcceptEnv LANG LC_*` (off by default) on the client, edit /etc/ssh/ssh_config and add `SendEnv LANG LC_*` (off by default) Debian/derivates have Solution2 by default.
(In reply to comment #6) s/off by default/unset and thereby disabled by openssh-default/
Yes, I know of these solutions. Solution 2 at least is mentioned on mosh github issue #98, which I linked in the OP. Another solution, which I also mentioned in the OP, is to pass '--server="env LC_ALL=en_US.UTF-8" mosh-server' to the `mosh` command on the client. I reported this bug because I think both of those solutions are basically workarounds and not real fixes. Solution 1 is a workaround because it causes you to export the LANG variable twice when logging into a system normally. Solution 2 is a workaround because it forces the local system locale on the remote system, which is not necessarily desirable. Both solutions also require manual editing by the user. Solution 1 cannot be implemented automatically because it is user-specific and not controlled by root - or, if you want to edit /etc/bash/bashrc directly, that should be done by the bash ebuild, not the mosh ebuild. (Maybe this is in fact the correct solution?) Solution 2 cannot be implemented automatically because the Gentoo user installing mosh on the server might not have control or access to the intended client machine (which may not even be a Gentoo, or even a Linux machine) - in fact, that means that Solution 2 might not be possible to implement at all, for the person installing mosh on the server. Ideally mosh to a gentoo server should just work after installation of mosh on the server, i.e. without forcing the administrator of the server to do any extra manual editing. I think the way to do this is to create a wrapper for mosh-server to make it aware of the variables in /etc/profile.env, but maybe there's a better way. In particular I am just assuming that mosh calls mosh-server on the remote and not `locale` directly - otherwise, this is not really something that the mosh ebuild can handle by itself... If there is no way to avoid the server admin doing some manual editing of config files, the mosh ebuild should at least print a message noting this fact after installation.
either way, you have to declare your LANG on the server side. Just do it. Some `parse random files for settings the user might have intended to load` aka. parse /etc/profile* ~/.profile* does not make it any better - imho.
(In reply to comment #9) > either way, you have to declare your LANG on the server side. The question is whether one declares LANG in the usual way (i.e. using the localization guide on gentoo.org, which I have already done, and which it is reasonable to expect that any Gentoo user has done) or some contrived undocumented way which tries to populate LANG even for non-login shells. If you are saying that a user needs to do the latter if he wants to use mosh to connect to his account from somewhere else, then please document it somewhere, for example in a post-install message in the ebuild... but IMHO it would be nicer to just fix it: > Some `parse random files for settings the user might have intended to load` > aka. parse /etc/profile* ~/.profile* does not make it any better - imho. If you don't like that, then what about wrapping mosh-server with a login shell? Say, put the following in /usr/bin/mosh-server: #!/bin/sh exec sh -lc mosh-server-real and move the real mosh-server to /usr/bin/mosh-server-real (or whatever naming scheme). That way it's not arbitrary at all. Basically mosh-server is quitting because it thinks that the locale settings it sees are the same ones that a user in a login shell is going to see, but in fact on Gentoo there is a difference between what mosh-server sees and what a user would see in a login shell. So doing this wrapping would just be fixing up that discrepancy. What do you think? By the way, I tested that doing the above solves this problem completely without any need for configuration by users. Maybe others CC'd to this ticket can see if it works on their systems as well.
Replicated and confirmed that using a wrapper as in #10 works for me. Is there any particular reason why a wrapper would not be a good solution? (I'm writing myself an ebuild that incorporates said wrapper now)
This will be fixed by mosh-1.2 https://github.com/keithw/mosh/issues/98#issuecomment-5164851
I tried emerging mosh-9999, and this works for "mosh localhost" on my remote system. It fails however when connecting from an Ubuntu machine with mosh-1.1.3 installed. Employing the wrapper trick gets past the locale error, but it seems I have second (unrelated?) error between local and remote. (never gets past the connecting state)
Just wanted to clarify my previous comment: with 1.1.3 on local->remote and remote->remote, a wrapper is required to get past the UTF8 checks. With 1.1.3 on the local machine (Ubuntu 11.10) and latest-git on the remote (Gentoo), I still require the wrapper to connect local->remote, but remote->remote works with and without the wrapper. (And unrelated to any of this, I can't get local<->remote to work around my NAT ;-)
Resolved with 1.1.94 (pre-release for 1.2). Please update both server and clients to >=1.1.94, thanks.
(In reply to comment #15) > Resolved with 1.1.94 (pre-release for 1.2). > > Please update both server and clients to >=1.1.94, thanks. Great, an upstream solution is always better :) Though as Josh notes we still can't connect to Gentoo hosts unless the client has a new enough mosh version, I would still consider this bug fixed. Thanks for your help! (As the reporter of the bug, do I need to mark this bug "resolved", or do you do that? And I guess it probably needs to wait until we actually have a fixed non-9999 version of mosh in the portage tree anyway.)
(In reply to comment #16) > I guess it probably needs to wait until we actually have a > fixed non-9999 version of mosh in the portage tree anyway. ... which we now do. Thanks!
From #mosh on Freenode: <KeithW> kini: A wrapper is a fine solution. If you're moshing to an Ubuntu machine and you don't send variables, you _should_ get a working UTF-8 locale (many Mac users are in this situation). <KeithW> (Because a US-English installation of Ubuntu will give LANG=en_US.UTF-8 to any session started up over SSH.) <KeithW> Fundamentally, you do want the right locale even for non-login shells, because you presumably want "ssh remotehost command" also to be in the correct character set and locale. But that's not a battle we have to fight. :-) <kini> KeithW: The problem was that if I was moshing to a Gentoo machine, I wouldn't get a properly set LANG because `ssh gentoomachine mosh-server` doesn't execute mosh-server from a login shell, and gentoo doesn't set LANG except in login shells for some reason <kini> right, exactly :) <KeithW> kini: Yeah, fair enough. The mosh 1.2 client will send the locale information and the mosh 1.2 server will receive it, but you will need mosh 1.2 on both sides to take advantage of that extra layer of fail-safe. <kini> I guess we now lose the ability to preserve a different UTF-8 locale on either end of the connection <kini> but on the majority of linux distros this was already impossible anyway <KeithW> kini: No, you won't, because mosh only applies the client's locale _if_ all other means fail (including /etc/default locale and SSH env vars). <KeithW> We tried to take the least responsibility possible for this screwed-up situation while still fixing it. :-/ <kini> oh, OK. Then basically Gentoo should probably still adopt the wrapper in order to get the most "correct" operation, but I guess this minor lack of functionality is not nearly as serious as the original bug report, "mosh cannot connect to vanilla Gentoo server" :) So there is still a reason to use the wrapper, even with mosh 1.2. Though I won't complain if you decide this is not worth caring about.
+ 17 Apr 2012; Michael Weber <xmw@gentoo.org> mosh-1.1.94.ebuild: + Fix SRC_URI +
Follow up upstream bug report (what if LANG isn't locale-gen'd on the server side) https://github.com/keithw/mosh/issues/209
