Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 437112 - no TIMESTAMP field when using ftps/https binhost
Summary: no TIMESTAMP field when using ftps/https binhost
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Unclassified (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
Depends on:
Blocks: 377365
  Show dependency tree
Reported: 2012-10-03 23:46 UTC by Kai
Modified: 2022-10-20 02:43 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---

emerge --info (info.txt,5.30 KB, text/plain)
2012-10-03 23:46 UTC, Kai
Apache configuration (03_binhost_vhost.conf,1.18 KB, text/plain)
2012-10-04 08:18 UTC, Kai

Note You need to log in before you can comment on or make changes to this bug.
Description Kai 2012-10-03 23:46:53 UTC
Created attachment 325630 [details]
emerge --info

I would like to use an ftps/https binhost for portage. From discussion on the forum, I was told the problem is that FETCHCOMMAND is not used for the Packages file? (

I believe I have the FETCHCOMMAND set correctly, as using it from the command line I can fetch files from my host.
Further info here:

From make.conf ======================
FETCHCOMMAND_FTP= "/usr/bin/curl -R \${URI} -o ${DISTDIR}/\${FILE} --ssl"
RESUMECOMMAND_FTP="/usr/bin/curl -R \${URI} -o ${DISTDIR}/\${FILE} --ssl -C -"

FETCHCOMMAND_HTTPS= "/usr/bin/curl -R \${URI} -o ${DISTDIR}/\${FILE} --digest --fail"
RESUMECOMMAND_HTTPS="/usr/bin/curl -R \${URI} -o ${DISTDIR}/\${FILE} --digest --fail -C -"
PORTAGE_BINHOST="ftp://user:password@domain.tld" (or https://)

output of "sudo emerge links" =====================

kai@kai-laptop ~ % sudo emerge -a links                                                    
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1136k  100 1136k    0     0   187k      0  0:00:06  0:00:06 --:--:--  250k

!!! Binhost package index  has no TIMESTAMP field.

Fetching bininfo from ftp://user:*password*@domain.tld
!!! 421 Sorry, cleartext sessions are not accepted on this server.

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R    ] www-client/links-2.7 

Would you like to merge these packages? [Yes/No] no

It seems that FETCHCOMMAND is being used, but then portage complains about the TIMESTAMP field, so I am not entirely sure what is happening. The error looks very similar for https when authentication is enabled, but the emerge works when it is disabled.
Comment 1 Zac Medico gentoo-dev 2012-10-04 00:38:15 UTC
What ftps/https server applications are you using? From personal testing, I know that it works with thttpd (with or without authentication, and also with or without encryption via stunnel).
Comment 2 Kai 2012-10-04 08:18:14 UTC
Created attachment 325634 [details]
Apache configuration
Comment 3 Kai 2012-10-04 08:20:35 UTC
I am using pure-ftpd-1.0.36 with -Y 3 set, and apache-2.2.22-r1 (config attached). The problem with HTTPS is that I need to use SNI, as I have several websites hosted on the computer for testing purposes.
Comment 4 Zac Medico gentoo-dev 2012-10-04 15:19:23 UTC
We've had issues reported with pure-ftpd before (see bug #350139), but none with apache.
Comment 5 Kai 2012-10-04 16:16:05 UTC
(In reply to comment #4)
> We've had issues reported with pure-ftpd before (see bug #350139), but none
> with apache.

I think this may be a different issue, it doesn't hang as described in that report. According to the progress printout shown the file transfers completely. It is only after this that portage complains about lack of a TIMESTAMP field (I wasn't able to dig out exactly where this comes from in the source). The output of "sudo emerge links" in the original report demonstrates this a bit.

Let me know if I should attach anything else (I can do an apache version, but it is very similar. The binhost is the second of several HTTPS vhosts).
Comment 6 Kai 2012-10-04 21:44:18 UTC
I think there may be a larger problem with either my setup or the FETCHCOMMAND - If I use HTTPS but turn off authentication it fetches the package index file fine, but then fails (after fetching is complete) as follows on the first binary:

>>> Emerging binary (1 of 43) dev-libs/expat-2.1.0-r2
 * Fetching '/usr/portage/packages/dev-libs/expat-2.1.0-r2.tbz2' in
 * the background. To view fetch progress, run `tail -f /var/log
 * /emerge-fetch.log` in another terminal.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  209k  100  209k    0     0  43272      0  0:00:04  0:00:04 --:--:-- 50894
!!! Fetching Binary failed for 'dev-libs/expat-2.1.0-r2'

>>> Failed to emerge dev-libs/expat-2.1.0-r2, Log file:

>>>  '/var/tmp/portage/dev-libs/expat-2.1.0-r2/temp/build.log'

I am not sure whether this was also the case before, or whether I had simply not tried further once seeing it did not fail on fetching Packages. The only thing that appears to be working is a plain HTTP server without authentication.

With normal FTP it hangs as in the linked bug (bug #350139) - whatever is being used to fetch files does not respect pure-ftpd's port flag (-p port1:port2) for passive FTP (I have only opened a few through my firewall, and they are not being used). I think the built-in python function is used rather than cURL, as the progress bar never appears, and I do not have this issue with cURL.

I'm not sure exactly what the scope of this is anymore, please let me know what you want me to attach or test.
Comment 7 Zac Medico gentoo-dev 2012-10-08 18:27:13 UTC
The ftp hang is supposed to be fixed in Python 3.3:

That fix is not included in Python 2.7.3 or 3.2.3, since their NEWS files do not mention it.

You can test it with Python 3.3 if you install sys-apps/portage with USE=python3 enabled.
Comment 8 Arfrever Frehtes Taifersar Arahesis 2012-10-08 21:24:38 UTC
Snapshots of Python 2.7.4_pre* and 3.2.4_pre* are available for testing in Progress Overlay.