Bug 175891 – sftp not suported as transport for PORTAGE_BINHOST

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Will Holland 2007-04-24 17:33:22 UTC]

if I set
PORTAGE_BINHOST="sftp://user:passwd@mymachine"

portage fails saying sftp is not a valid transport.

We do not use ftp on our site, so it would be really useful to be able to use sftp as a transport

Reproducible: Always

[Comment 1 Zac Medico 2007-05-29 09:14:42 UTC]

Created attachment 120600 [details, diff]
use the paramiko library to add sftp support for PORTAGE_BINHOST

One thing missing is that RESUMECOMMAND is used to fetch the package, but the default RESUMECOMMAND (defined in /etc/make.globals) uses wget which unfortunately doesn't support sftp.  Do you know of a program that will support sftp in addition to http and ftp?  You can write a script which checks the protocol and calls a different program if necessary, but that's not quite ideal.  I suppose we could implement the fetch using python libraries and just not use RESUMECOMMAND at all, but that's not really ideal either.

[Comment 2 Marius Mauch (RETIRED) 2007-06-05 17:29:15 UTC]

(In reply to comment #1)
> Created an attachment (id=120600) [edit]
> use the paramiko library to add sftp support for PORTAGE_BINHOST
> 
> One thing missing is that RESUMECOMMAND is used to fetch the package, but the
> default RESUMECOMMAND (defined in /etc/make.globals) uses wget which
> unfortunately doesn't support sftp.  Do you know of a program that will support
> sftp in addition to http and ftp?  You can write a script which checks the
> protocol and calls a different program if necessary, but that's not quite
> ideal.  I suppose we could implement the fetch using python libraries and just
> not use RESUMECOMMAND at all, but that's not really ideal either.

Well, fetch() supports protocol specific variables like FETCHCOMMAND_SFTP and so on, so could use those to be consistent.

[Comment 3 Marius Mauch (RETIRED) 2007-06-23 16:41:29 UTC]

Zac: any reason not to commit that patch?

[Comment 4 Zac Medico 2007-06-24 06:17:03 UTC]

I think the patch is fine but it's incomplete until we fix binarytree.gettbz2() to share code with portage.fetch() to enable FETCHCOMMAND_SFTP support.

[Comment 5 Zac Medico 2007-06-25 03:30:17 UTC]

Created attachment 123003 [details, diff]
portage-2.1.3_rc5 patch for paramiko + {FETCHCOMMAND,RESUMECOMMAND}_SFTP

This patch should provide complete sftp support.

[Comment 6 Zac Medico 2007-06-25 06:32:17 UTC]

Created attachment 123020 [details, diff]
portage-2.1.3_rc5 patch for paramiko + {FETCHCOMMAND,RESUMECOMMAND}_SFTP

This version fixes a couple bugs.  It seems to work fine.

[Comment 7 Zac Medico 2007-06-25 06:39:20 UTC]

Created attachment 123021 [details]
python script for use in {FETCHCOMMAND,RESUMECOMMAND}_SFTP

I had trouble finding a good sftp client program so I wrote this little python script that uses the paramiko library.  The options are similar to wget:

$ sftp.py --help
usage: sftp.py [OPTION]... [URL]...

options:
  -h, --help            show this help message and exit
  -O FILE, --output-document=FILE
                        save uri to FILE
  -c, --continue        continue getting a partially-downloaded file
  -b BLOCKSIZE, --blocksize=BLOCKSIZE
                        block size used for read/write
  -q, --quiet           disable output

The following settings in make.conf will work:

FETCHCOMMAND_SFTP="sftp.py -O \${DISTDIR}/\${FILE} \${URI}"
RESUMECOMMAND_SFTP="sftp.py -c -O \${DISTDIR}/\${FILE} \${URI}"

[Comment 8 Zac Medico 2007-06-29 02:52:55 UTC]

This has been released in 2.1.3_rc6.

[Comment 9 Zac Medico 2010-09-29 05:28:24 UTC]

In portage-2.1.9.12 there's ssh and sftp support for the newer binhost protocol:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=d9c4a7dec291f3f57296a1bf249887fa950bbcc1
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=9065f86cf61ae69968e7d1155d7ecb89688c037c
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=a0c2fa6dd742a273d328238604594a78107581db