Bug 470002 – sys-apps/portage: support PORTAGE_SSH_OPTS to provide arguments to ssh and sftp

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Jason A. Donenfeld 2013-05-15 20:54:16 UTC]

The summary says it all. Portage fetches sftp and ssh binary packages using a non-customizable command. This makes usage of private keys very cumbersome, not to mention proper placement of known-hosts files and other related ssh functionality. Adding a SSH_FETCH_ARGUMENTS which would be merged into the arguments for sftp and ssh for binary fetching would be quite nice.

Reproducible: Always

[Comment 1 Zac Medico 2013-05-15 22:46:52 UTC]

So both ssh and sftp commands support all of the same arguments? Maybe it's better to have a separate variable for each one?

[Comment 2 Jason A. Donenfeld 2013-05-16 00:45:12 UTC]

AFAIK, they support the same options.

[Comment 3 Paul Varner (RETIRED) 2013-05-16 15:25:57 UTC]

The port options may be different depending upon what command you are using

ssh use '-p' - lowercase 'p'
scp/sftp use '-P' - uppercase 'P'

[Comment 4 Zac Medico 2013-05-16 16:35:01 UTC]

Another thing to consider is that the binhost fetch code has 2 different parts which behave differently:

1) Fetching of the Packages file can close the connection early if the TIMESTAMP header has a value equal to the cached copy. The is optimized to avoid downloading the the whole (megabytes) of file when it has not been modified since it was previously downloaded. The relevant code parses the PORTAGE_BINHOST URI and generates ssh arguments as follows:

	if port is not None:
		port_args = ['-p', "%s" % (port,)]
	proc = subprocess.Popen(['ssh'] + port_args + \
		[user_passwd + host, '--', 'cat', path],
		stdout=subprocess.PIPE)

2) Meanwhile, fetching of the actual binary packages uses this command:

FETCHCOMMAND_SSH="bash -c \"x=\\\${2#ssh://} ; host=\\\${x%%/*} ; port=\\\${host##*:} ; host=\\\${host%:*} ; [[ \\\${host} = \\\${port} ]] && port=22 ; exec rsync --rsh=\\\"ssh -p\\\${port}\\\" -avP \\\"\\\${host}:/\\\${x#*/}\\\" \\\"\\\$1\\\"\" rsync \"\${DISTDIR}/\${FILE}\" \"\${URI}\""

It seems like we should be able to merge the proposed SSH_FETCH_ARGUMENTS value into the --rsh part of FETCHCOMMAND_SSH, and we can also merge it into the generated arguments in part (1). Does everyone think that will work?

[Comment 5 Jason A. Donenfeld 2013-05-16 17:33:48 UTC]

(In reply to comment #4)
> It seems like we should be able to merge the proposed SSH_FETCH_ARGUMENTS
> value into the --rsh part of FETCHCOMMAND_SSH, and we can also merge it into
> the generated arguments in part (1). Does everyone think that will work?

Perfect.

[Comment 6 Jason A. Donenfeld 2013-05-16 17:36:43 UTC]

(In reply to comment #3)
> The port options may be different depending upon what command you are using

Ahh, right, this. Shucks. At least there's "-oPort=2222", which works on all of them.

[Comment 7 Zac Medico 2013-05-17 01:43:22 UTC]

If sftp doesn't have a way to pipe the file like ssh has, then there's no point having a separate kind of fetch optimized for the Packages file. So for sftp, we can include the SSH_FETCH_ARGUMENTS variable in the default FETCHCOMMAND_SFTP setting, and use that command for both the Packages file and the binary packages.

[Comment 8 Zac Medico 2013-05-17 02:10:24 UTC]

Only use FETCHCOMMAND_SFTP for sftp support:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=75a524e9a53ef4dc941947cb99d34fdf1ccab6e4

Note that the current default FETCHCOMMAND_SFTP looks like this:

FETCHCOMMAND_SFTP="bash -c \"x=\\\${2#sftp://} ; host=\\\${x%%/*} ; port=\\\${host##*:} ; host=\\\${host%:*} ; [[ \\\${host} = \\\${port} ]] && port=22 ; exec sftp -P \\\${port} \\\"\\\${host}:/\\\${x#*/}\\\" \\\"\\\$1\\\"\" sftp \"\${DISTDIR}/\${FILE}\" \"\${URI}\""

We can update it to insert SSH_FETCH_ARGUMENTS after the port arguments.

[Comment 9 Zac Medico 2013-05-18 22:27:36 UTC]

Support for PORTAGE_SSH_OPTS is in git:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=1e984dab27458e3026163c5690aec5fd3298dd8b

[Comment 10 Zac Medico 2013-05-19 00:24:59 UTC]

This is fixed in 2.1.12.1 and 2.2.0_alpha176.