Bug 625246

Summary:	sys-apps/portage: binhost https If-Modified-Since header support
Product:	Portage Development	Reporter:	ge1ger
Component:	Enhancement/Feature Requests	Assignee:	Portage team <dev-portage>
Status:	RESOLVED FIXED
Severity:	normal	Keywords:	InVCS
Priority:	Normal
Version:	unspecified
Hardware:	All
OS:	Linux
Whiteboard:
Package list:		Runtime testing required:	---
Bug Depends on:
Bug Blocks:	627566, 668302

Description ge1ger 2017-07-15 21:02:50 UTC

PORTAGE_BINHOST="http://example.com" emerge -G package

When the above is ran, http://example.com/Packages is redownloaded every time emerge is ran, even if it's the same exact file.

Is it possible to have emerge save this to a specific location and check timestamps to see if it needs to be redownloaded?

A workaround is possible by editing this file: 
/usr/lib/python3.4/site-packages/portage/dbapi/bintree.py

Comment 1 Zac Medico gentoo-dev

2017-07-15 21:54:52 UTC

For http protocol, we already support the If-Modified-Since header:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e06cb6d66db37ac7ab77acf65038b1f770c13c96

For https protocol, If-Modified-Since support has been disabled since bug 469888:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=b5969af9f575e4e4b669f44e76ad01f0dbc2dd27

Comment 2 ge1ger 2017-07-15 22:38:22 UTC

Will date-checking work for https binhosts in the future?



There's the following solution for having cache in https:

https://p.sicp.me/bCETD.bash

FETCHCOMMAND_HTTP="curlcache.sh \"\${URI}\" \"\${DISTDIR}/\${FILE}\""

Comment 3 ge1ger 2017-07-15 22:43:55 UTC

Correction:

FETCHCOMMAND_HTTPS="curlcache.sh \"\${URI}\" \"\${DISTDIR}/\${FILE}\""

Comment 4 Zac Medico gentoo-dev

2017-07-16 23:18:57 UTC

Since python 2.7.9/3.4.3, python libraries perform ssl certificate and hostname checks by default:

https://docs.python.org/2/library/httplib.html#httplib.HTTPSConnection
https://docs.python.org/3/library/http.client.html#http.client.HTTPSConnection

This means that we can apply our existing If-Modified-Since support to https, as long as the python version is new enough.

Comment 5 Zac Medico gentoo-dev

2017-07-19 07:28:00 UTC

I've got a patch here:

https://github.com/zmedico/portage/tree/bug_625246

Comment 6 Zac Medico gentoo-dev

2017-07-31 07:16:21 UTC

Patch posted for review:

https://archives.gentoo.org/gentoo-portage-dev/message/16a4da7f3eb2246a027e6b2c5c0a28da
https://github.com/gentoo/portage/pull/183

Comment 7 Zac Medico gentoo-dev

2017-07-31 16:33:08 UTC

This is in the master branch:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=1d821469d6b72ce051b02908f17302c500945788