Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 442874

Summary: net-proxy/http-replicator - local rsync server reports install flash player as wrong size
Product: Gentoo Linux Reporter: far jump <far.jmp>
Component: Current packagesAssignee: No maintainer - Look at https://wiki.gentoo.org/wiki/Project:Proxy_Maintainers if you want to take care of it <maintainer-needed>
Status: RESOLVED FIXED    
Severity: normal CC: bugs.gentoo.org-6, davidsparks, mmogilvi+gnto, proxy-maint
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 502574    
Attachments: emerge --info for rsync server
add x-unique-cache-name to http-replicator-3.0
modify portage's default FETCHCOMMAND to supply new header
add x-unique-cache-name to http-replicator-4.0.x
new 3.0-r5 ebuild applies patch and elogs instructions
wget wrapper script to be intelligent about adding extra header
wget wrapper script to be intelligent about adding extra header
add x-unique-cache-name to http-replicator-3.0
patch for 3.0: suggest repcacheman if cache doesn't exist
new ebuild 3.0-r5 applies patches and elogs instructions
new ebuild 3.0-r5 with patches, elog instructions, and portage updates
new ebuild 3.0-r5 with patches, elog instructions, and portage updates (2)
conf file
new ebuild 3.0-r5 with patches, elog instructions, and portage updates (3)

Description far jump 2012-11-12 14:25:41 UTC
I have a local rsync server for Gentoo portage. (to prevent the many systems from sync Gentoo mirror sites and repeatedly downloading the same files).  From time to time, a system will attempt to emerge adobe-flashplayer and will recieve a file size error for:
install_flash_player_11_linux.x86_64.tar.gz
or
install_flash_player_11_linux.i386.tar.gz

If I tell that linux box to ignore the local Gentoo mirror and rsync server, and allow it acces to the Internet, flash player will install just fine.

here is the fix:  I go to /var/cache/http-replicator/ on the rsync server and delete the offending install_flash_player_11_linux.*.tar.gz file and everything works -- ie the local systems will emerge the newest flash player perfectly.

Some how portage for local mirror sites does not update install_flash_player_11_linux.*.tar.gz when it changes.
Comment 1 far jump 2012-11-12 14:27:56 UTC
Created attachment 329342 [details]
emerge --info for rsync server
Comment 2 Jeroen Roovers gentoo-dev 2012-11-12 14:46:06 UTC
It shouldn't be caching install_flash_player_11_linux.x86_64.tar.gz
but adobe-flash-*.tar.gz instead.
Comment 3 far jump 2012-11-12 15:01:32 UTC
(In reply to comment #2)
> It shouldn't be caching install_flash_player_11_linux.x86_64.tar.gz
> but adobe-flash-*.tar.gz instead.

Do I have any control over what is caches and what it does not cache?  Does it not just download what the ebuild demands?
Comment 4 David Sparks 2012-11-13 23:57:00 UTC
(In reply to comment #2)
> It shouldn't be caching install_flash_player_11_linux.x86_64.tar.gz
> but adobe-flash-*.tar.gz instead.

install_flash_player_11_linux.x86_64.tar.gz is the filename of what is downloaded:

>>> Emerging (1 of 6) www-plugins/adobe-flash-11.2.202.251
 * Fetching files in the background. To view fetch progress, run
 * `tail -f /var/log/emerge-fetch.log` in another terminal.
>>> Downloading 'http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.251/install_flash_player_11_linux_x86_64.tar.gz'


The problem is Adobe is encoding the version # in a directory component and not the filename.  http_replicator flattens the files it caches into a single directory discarding directories in the process.  This causes the hash failure when a new version is released.

http_replicator log from my server shows that the version # is the directory name before the filename:

31 Jan 2012 12:31:49 INFO: HttpClient 12250 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.1.102.55/install_flash_player_11_linux.x86_64.tar.gz
19 Apr 2012 15:25:17 INFO: HttpClient 50676 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.228/install_flash_player_11_linux.x86_64.tar.gz
16 May 2012 15:38:07 INFO: HttpClient 59112 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.235/install_flash_player_11_linux.x86_64.tar.gz
09 Aug 2012 12:05:00 INFO: HttpClient 2527 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.236/install_flash_player_11_linux.x86_64.tar.gz
06 Sep 2012 09:12:16 INFO: HttpClient 10925 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.238/install_flash_player_11_linux.x86_64.tar.gz
24 Oct 2012 16:20:14 INFO: HttpClient 5880 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.243/install_flash_player_11_linux_x86_64.tar.gz
09 Nov 2012 16:43:42 INFO: HttpClient 11148 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.251/install_flash_player_11_linux_x86_64.tar.gz
Comment 5 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-06-05 13:27:20 UTC
Does this still happen with the latest version?

Could you report this upstream?
Comment 6 Holger Ackermann 2013-06-19 21:25:23 UTC
Same bug is annoying me too. Is upstream still supporting http-replicator anyway? Latest version (4.0alpha2) is dated January 2008! If not so, is there a chance for whoever to patch latest stable?
Comment 7 far jump 2013-07-13 18:46:38 UTC
(In reply to Tom Wijsman (TomWij) from comment #5)
> Does this still happen with the latest version?
> 
> Could you report this upstream?

Yes it is still happening.
Comment 8 far jump 2013-09-17 03:59:45 UTC
(In reply to Tom Wijsman (TomWij) from comment #5)
> Does this still happen with the latest version?
> 
> Could you report this upstream?

emerge -u world included adobe flash update
problem still happening
Comment 9 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2014-01-28 23:19:07 UTC
Which version is this? Has this since been reported upstream?
Comment 10 far jump 2014-01-28 23:31:16 UTC
www-plugins/adobe-flash Available versions:  11.2.202.33

who do you mean upstream?  why would adobe do anything?  do you think that adobe will accommodate gentoo emerge/resync protocols?

Any way the gentoo repositories that my main server sync from do not seem to have this problem?  what is gentoo doing that I my resync server is not doing?
Comment 11 Tom Poplawski 2014-02-04 20:54:14 UTC
When http-replicator was created I knew there were going to be files that were just not cacheable due to things beyond gentoo's control.

I created repcacheman to solve problems like this and included it with http-replicator.

My recommendation was to move all files from the cache dir to the distfile dir and run repcacheman on a regular basis (say weekly to monthly).  This info was in the original forum thread still listed in the ebuild but probably needs to go in the man page or similar.

repcacheman checksums all the files and moves verified files back to the cache.  The script then tells you to delete all files left in the distfile dir.

Adobe's install_flash_player_11_linux.x86_64.tar.gz will be one of the files left for deletion.  There will be other files you need to delete also.
Comment 12 far jump 2014-02-04 21:19:16 UTC
Hi Tom,
First of all - thanks for creating repcacheman!  I have been using it from the very very beginning of my Gentoo days.

unfortunately, that logic has holes it.  What if adobe_flash updates in between cycles of moving all cache'd files to ./distfiles/ and then scrubbing them? You might find that you must tighten the refresh cycle until your doing every day?  -- seems like using an atom bomb to kill a rabbit.

It was just easier to write a script using eix and ruby to detect an adobe update and then just preemptively delete the offending file.
Comment 13 David Sparks 2014-02-04 22:10:40 UTC
(In reply to Tom Poplawski from comment #11)
> When http-replicator was created I knew there were going to be files that
> were just not cacheable due to things beyond gentoo's control.
> 
> I created repcacheman to solve problems like this and included it with
> http-replicator.
...

I run http-replicator on a VM that does very little(*) so looking at the contents of /usr/portage/distfiles/ is somewhat pointless.  What about modifying repcacheman to manage the files in /var/cache/http-replicator/ and delete files that fail the xsum check, or are no longer referenced by an ebuild in the tree?

repcacheman could run as a post-sync step to minimize the delay in keeping the tree and cache in sync.  This would not catch the case where a dist does a slipstream update (change binary without changing version #) but would work well for regular updates.

(*) portage mirror, http-replicator cache, and local http overlay+distfiles for ~100 servers.
Comment 14 Tom Poplawski 2014-02-20 23:08:03 UTC
(In reply to far jump from comment #12)
> Hi Tom,
> First of all - thanks for creating repcacheman!  

Your welcome.

> What if adobe_flash updates in
> between cycles 

We both agree on your point, you made a fix for this particular problem with adobe only, that works for now.  I tried to make a more general fix, that works most of the time, for most problems, if used as recommended. 

By the way I couldn't find your script, have you shared it somewhere for others to use?

Did you ever run repcacheman and see the other files that are a problem too?  Are you going to write scripts for those also?  Don't forget to write a script for ftp files.  http-replicator can't cache those, but repcacheman will add those to the cache automatically. 

By the way, I looked at the actual adobe update release dates:

11.6 March 12th, 2013
11.7 June 11th, 2013
11.8 September 24th, 2013
11.9 December 10th, 2013
12.0 February 20th, 2014

I made the original recommendation to run repcacheman weekly or monthly back in 2004.  I didn't check adobe back that far but I hope all who followed my recommendation would have probably never known about this problem with adobe flash.
Comment 15 Tom Poplawski 2014-02-20 23:34:00 UTC
(In reply to David Sparks from comment #13)
> What about
> modifying repcacheman to manage the files in /var/cache/http-replicator/ and
> delete files that fail the xsum check, or are no longer referenced by an
> ebuild in the tree?


Good suggestion. I looked at this back in 2004 but not since.  Back then there were about a dozen distfile pruning scripts floating around the forums.  Nobody could agree on what to remove so you could pick a script that met your needs.  I still believe this is the case.  I didn't want to duplicate the efforts of others or impose my wishes for what gets pruned.  Isn't there an offical gentoo supported script now? eclean-dist is it?  Many scripts supported other than the offical distfile dir.  I think

DISTDIR=/var/cache/http-replicator/ eclean-dist

will work almost as good as repcacheman :-)





 
> (*) portage mirror, http-replicator cache, and local http overlay+distfiles
> for ~100 servers.

Wow, nice to know it's useful :-)
Comment 16 Matthew Ogilvie 2014-12-29 06:28:50 UTC
Created attachment 392616 [details, diff]
add x-unique-cache-name to http-replicator-3.0

Ebuilds and portage already know how to uniqueify download file names via the "URL -> cacheFile" syntax for the SRC_URI variable.  We just need to enhance http-replicator to use the cacheFile name for proxy caching purposes as well, which is fairly easy:

Here is a patch for http-replicator 3.0 to recognize and use an "X-unique-cache-name" HTTP header, if provided in the request and the replicator is running in "--flat" mode (which is default).

You also need to modify the FETCHCOMMAND to supply this header in make.conf unless the default is changed in portage:

FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O \"\${DISTDIR}/\${FILE}\" --header=\"X-unique-cache-name: \${FILE}\" \"\${URI}\""
Comment 17 Matthew Ogilvie 2014-12-29 06:38:01 UTC
Created attachment 392618 [details, diff]
modify portage's default FETCHCOMMAND to supply new header

This patch should change portage's default to supply the x-unique-cache-name header in the FETCHCOMMAND and RESUMECOMMAND, although I haven't actually tested this patch (I just override FETCHCOMMAND in make.conf).

But perhaps something smarter would be desired, where it only adds the header if http_proxy is defined and the last component of URI is different from FILE?  Perhaps install a customizable /etc/portage/fetchcommand wrapper script that has such smarts, and default FETCHCOMMAND to that?  Or some other way to make it conditional (more variables)?
Comment 18 Matthew Ogilvie 2014-12-29 06:46:15 UTC
Created attachment 392620 [details, diff]
add x-unique-cache-name to http-replicator-4.0.x

Here is a corresponding patch for the completely rewritten http-replicator 4.0.x.

I haven't tested this at all, partly because I reading through the code, I noticed what appears to be at least two pre-existing bugs in version 4:

      1. If it is not in "--flat" mode, then there does not appear to be
         anything to prevent a custom client from requesting
         "/../../../../../any/file/on/the/filesystem" (standard clients
         will simplify ".."s before sending the request, but see
         http://stackoverflow.com/questions/5951029/read-http-output-using-shell-bash-script )
         as long as the UID the process is running as can read it, including
         /etc/passwd or similar.  There may also be problems with
         writing to weird locations, although standard permissions
         renders this less likely.  This is a SECURITY bug, but
         fortunately (a) --flat mode is default on gentooo, (b)
         gentoo still defaults to http-replicator-3.0 on stable systems,
         which doesn't have this bug, and (c) 4.0 releases are still
         marked "alpha"...
      2. HTTP header names are not supposed to be case-sensitive, but
         version 4 does nothing to fold the case...

I'll let gentoo developers decide what if anything to do about these apparent bugs.
Comment 19 Pacho Ramos gentoo-dev 2015-01-07 14:16:45 UTC
Would you be willing to proxy maintain http-replicator? https://wiki.gentoo.org/wiki/Project:Proxy_Maintainers
Comment 20 Matthew Ogilvie 2015-01-09 07:24:44 UTC
I think I have time to do minor levels of proxy maintenance, especially for version 3.0.  3.0 seems fairly solid, and since I use it I have an incentive to fix problems.  I consider an http-replicator (along with an rsync mirror) to be a required/standard setup on any LAN that has more than one gentoo machine, regardless of the speed of its Internet connection.

I'm not sure when/if I can do more involved maintenance, such as digging into the version 4.0 apparent bugs described in comment 18, or actively searching for similar ones.  Especially since I don't really know python.  It took me years to decide to look into this "which name is it using in the cache" bug (instead of just working around it), although it was fairly easy after I actually got started...

(I wonder if version 4.0 should be package.mask'ed until me or someone actually looks into comment 18's bugs?  But the /etc/conf.d/http-replicator file changes dramatically, so extra transitions between 3 and 4 could be annoying to anyone currently using the testing (~) version.)
Comment 21 Matthew Ogilvie 2015-01-09 07:44:17 UTC
Created attachment 393508 [details]
new 3.0-r5 ebuild applies patch and elogs instructions

3.0-r5's changes compared to -r4:
  - Applies patch from comment 16.
  - Adds some more instructions to pkg_postinst(), and elog's them
    instead of einfo.  (Plus one ewarn for bug 502574.)
  - Keyworded for testing (~) only, so perhaps it could go in as-is.

Possible future enhancements not included:
  - I'm not sure the best way to handle client machine config.
    See next comment.  The wording of the elog messages may
    change depending on what is decided.
  - Perhaps instructions should be installed somewhere (man page
    or /usr/share/doc), not just elog'ed and referenced to external
    forums...
  - Perhaps the init and systemd scripts should check for the cache
    and suggest running repcacheman, rather than just letting
    http-replicator error out.  (bug 502574)
Comment 22 Matthew Ogilvie 2015-01-09 08:02:43 UTC
Created attachment 393512 [details]
wget wrapper script to be intelligent about adding extra header

I'm not sure the best way to reconfigure client machines to take advantage of the new header (comment 16 and comment 21) to avoid this caching bug.

This script only adds the new header when necessary.  FETCHCOMMAND and RESUMECOMMAND can refer to this script instead of calling wget directly.

Note that this probably shouldn't be packaged with http-replicator itself.

Options:
  - Maybe this script isn't necessary.  The new header probably
    doesn't hurt anything (except a minuscule amount of extra
    network traffic) if you just send it unconditionally.
  - Maybe it is OK to require that users manually override
    FETCHCOMMAND and RESUMECOMMAND on client machines?
  - Maybe the default FETCHCOMMAND and RESUMECOMMAND in make.globals
    (from sys-apps/portage) should already be setup to add the header, either
    unconditionally (comment 17), or via this "fetcher" script?
  - Perhaps there should be an "http-replicator-client" package that
    includes this script and makes appropriate changes to make.conf to
    be merged with something like etc-update?

Depending on the desired solution, the elog'ed instructions in comment 21's ebuild may need to change.
Comment 23 Matthew Ogilvie 2015-01-09 08:05:12 UTC
To use the script from comment 22, add these to make.conf:

FETCHCOMMAND="/etc/portage/fetcher -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
RESUMECOMMAND="/etc/portage/fetcher -c -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
Comment 24 Matthew Ogilvie 2015-01-09 09:07:11 UTC
Created attachment 393518 [details]
wget wrapper script to be intelligent about adding extra header

Newer version of wget wrapper script: fix f_args name and add comments.
Comment 25 Pacho Ramos gentoo-dev 2015-01-14 14:12:32 UTC
Well, http-replicator is completely orphan for some time and, then, even if you will do most for 3.x versions it's much better than current situation
Comment 26 Matthew Ogilvie 2015-01-17 19:53:05 UTC
Created attachment 394206 [details, diff]
add x-unique-cache-name to http-replicator-3.0

Patch: add x-unique-cache-name support to http-replicator.

Very minor tweaks to patch from comment 16: filename tweak, no email in patch [maybe avoid spam], and remove superfluous ';'.
Comment 27 Matthew Ogilvie 2015-01-17 19:57:19 UTC
Created attachment 394208 [details, diff]
patch for 3.0: suggest repcacheman if cache doesn't exist

Suggest running repcacheman if cache doesn't exist.

This really goes with bug 502574, but it might be handy to have everything for the new ebuild I'm about to post to all be collected under one bug.
Comment 28 Matthew Ogilvie 2015-01-17 20:03:05 UTC
Created attachment 394210 [details]
new ebuild 3.0-r5 applies patches and elogs instructions

Compared to comment 21, this ebuild also applies a patch to remind users to use repcacheman if the cache doesn't exist (for bug 502574).

It requires the patches from comments 26 and 27.

I'm still not sure how to deal with client-side configuration.  See earlier comments.
Comment 29 Pacho Ramos gentoo-dev 2015-11-08 12:56:10 UTC
@proxy-maint, could you please take a look to this? Current ebuild in stable has a lot of bugs and some of them have patches provided by Matthew that maybe could be accepted finally

Thanks
Comment 30 Ian Delaney (RETIRED) gentoo-dev 2015-11-10 02:18:12 UTC
Then I take it your are prompting  Matthew Ogilvie as to whether he'd consider being a proxy maintainer, is it a proxied maintainer. I you do, feel to peruse https://wiki.gentoo.org/wiki/Project:Proxy_Maintainers
and also to join in the channel listed there we offer first hand support to users and commit submitted patches and bumps to the portage tree.
Clearly pacho is keen to get this package back on the right rails.
Comment 31 Sam Jorna (wraeth) gentoo-dev 2016-03-22 01:02:08 UTC
Matthew, are you still interesting in maintaining this package? If not, or if there's no reply on this bug soon, I will commit fix and commit this to close this bug.

If you are, there are a couple of points with the ebuild in attachment 394210 [details] that should be addressed:

 - The header needs to be updated to current (see ${PORTDIR}/header.txt)
 - Current allowed EAPI's are 5 and 6 (6 being preferable)
 - python.eclass has been deprecated, please migrate to python-r1.eclass or similar

Otherwise the ebuild passes my build tests in it's current state, so good work! If you wish to proxy-maintain this package, then please address these last issues and re-attach.

Thanks!
Comment 32 Matthew Ogilvie 2016-03-22 05:16:22 UTC
Created attachment 428742 [details]
new ebuild 3.0-r5 with patches, elog instructions, and portage updates

New version of 3.0 ebuild that addresses EAPI, python eclass, newer header, and newer path to make.conf, as requested by Ian Delaney in comment 31.  These are the only changes compared to attachment 394210 [details] from comment 28.

Unchanged from attachment 394210 [details]:
   - Users still need to override FETCHCOMMAND to fully fix this issue, although it degrades to previous behavior without it.
   - Improved startup error message for bug 502574.
   - FYI: Bug 524208 is partially a duplicate of this issue, although not completely.  (If a source URI file is changed in-place, using a unique cache name doesn't really help.)
   - The ebuild now elog's important setup instructions after installation.
   - Version 4.0_alpha2's ebuild could probably use another update with similar changes for newer portage.  It could reasonably be argued that 4 has a cleaner internal design and it would eventually be worth switching, but currently 4.0 is still very "alpha" (incomplete/buggy).  (See comment 18.)

I don't really want to spend much time maintaining http-replicator, but I'm at least willing to try to get my version 3.0 patches working well.
Comment 33 Sam Jorna (wraeth) gentoo-dev 2016-03-22 07:03:37 UTC
There are a couple of minor last-minute tweaks to this that I picked up as I was giving it a final inspection:

 - IUSE, DEPEND and RDEPEND do not need to be defined if they are blank
 - patching should occur in src_prepare()[0]
 - added call to 'default' to src_prepare() to cater for eapply_user[0].

I can make these changes as I commit the ebuild since they are only minor; however if you wish to do it and re-attach, you can of course do so.

(In reply to Matthew Ogilvie from comment #32)
> I don't really want to spend much time maintaining http-replicator, but I'm
> at least willing to try to get my version 3.0 patches working well.

So just to be clear, do you wish to become the proxy maintainer of this package? This will mean that the existing net-proxy/http-replicator bugs will be assigned to you and it is expected that you will begin attempting to address these - from the sounds of it you have already familiarised yourself with them.

If you are only wishing to submit patches on some bugs but do not wish to proxy-maintain, that's fine, I just need to clarify in order to commit it!

0: https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-1000009.1.5

Thanks!
Comment 34 Sam Jorna (wraeth) gentoo-dev 2016-03-22 07:28:21 UTC
A quick note I forgot to mention in my last comment.

Since the only thing you're doing in src_prepare() is applying patches, you can list them in the PATCHES variable and leave src_prepare() undefined. The default definition of src_prepare() in EAPI6 applies any patches listed in the ${PATCHES} array, as well as runs eapply_user.

As an example, see ${PORTDIR}/dev-db/mysql-connector-c++/mysql-connector-c++-1.1.3.ebuild
Comment 35 Matthew Ogilvie 2016-03-23 06:00:33 UTC
Created attachment 428792 [details]
new ebuild 3.0-r5 with patches, elog instructions, and portage updates (2)

Essentially the same as comment 32.  Includes a couple of more portage best practices tweaks from comment 33 and comment 34.

After this makes it in, it will likely be years (or never) before I do anything to http-replicator again, unless some high-priority issue affects me personally.  So I guess that meets the definition of an occasional patch submitter rather than a proxy maintainer?

I have skimmed through "http-replicator" bugs; I think all the ones I consider high-priority are handled by this ebuild, or at least improved.  Besides bugs from comment 32:  I think my ebuild also effectively fixes bug 561674, although I'm not sure how -r4 became broken, because the last time I tried it (more than a year ago), it seemed to install and work as expected.  Maybe the 21 Mar 2015 ChangeLog-2015 entry broke it?
Comment 36 Sam Jorna (wraeth) gentoo-dev 2016-03-23 11:01:38 UTC
This has now been committed. If possible, can you please check and comment on the bugs you believe this should resolve?

Thanks for the contribution!


commit 91d6d053b55f3c230fa90dc257d0bc994c3bc5e5
Author: Sam Jorna <wraeth@gentoo.org>
Date:   Wed Mar 23 21:58:13 2016 +1100

    net-proxy/http-replicator: revbump, bug 442874

    Adds X-Unique-Cache-Name header to try and catch files based on their
    saved name in DISTFILES. Additionally bumps ebuild to EAPI6 and elogs
    information for required setup. Ebuild supplied by Matthew Ogilvie.
Comment 37 far jump 2016-04-01 15:56:22 UTC
emerged http-replicator-3.0-r5 - does not work.

Client side:
Resolving eye... 192.168.1.106
Connecting to eye|192.168.1.106|:8080... failed: Connection refused.

Host side:
01 Apr 2016 11:40:48 STAT: HttpClient 1 bound to 192.168.1.101
01 Apr 2016 11:40:48 ERROR: HttpClient 1 caught an exception in __getattr__: HttpClient instance has no attribute 'data'
0

I when back to 3.0-r4 - everything works perfectly (except of course =>

!!! Fetched file: adobe-flash-11.2.202.577.x86_64.tar.gz VERIFY FAILED!
!!! Reason: Filesize does not match recorded size
!!! Got:      7278446
!!! Expected: 7279100
Refetching... File renamed to '/usr/portage/distfiles/adobe-flash-11.2.202.577.x86_64.tar.gz._checksum_failure_.d2P1MV'
Comment 38 Matthew Ogilvie 2016-04-02 07:51:48 UTC
> 01 Apr 2016 11:40:48 STAT: HttpClient 1 bound to 192.168.1.101
> 01 Apr 2016 11:40:48 ERROR: HttpClient 1 caught an exception in __getattr__: HttpClient instance has no attribute 'data'
> 0

I haven't been able to duplicate such a problem.

I don't see where / how it could be missing the attribute 'data'.
It seems to be created during construction (with a os.tmpfile()),
immediately after logging the "bound to" message.  Maybe
it somehow tried to handle a connection while it was still
starting up initially, although I don't even see how that
could generate these symptoms.

On the remote chance that systemd's invocation was poorly interacting
with my patches, I tried manually/directly running http-replicator
from a root command prompt without --pid, --daemon, or --log in an
attempt to mimic what systemd would do, but it still worked fine.

Possible Ideas and Questions: (I don't know if any of these will help,
but they are all I can think of.)

   - Was "Connection refused" the first error?  Or was that after
     the service had died during a previous download attempt?
   - How repeatable is it?  All the time, or intermittent?  How
     many times did you try?
   - How did you stop/start/restart it after emerging the new version?
     Did you try confirming it was shut down before starting it again?
     Was something already trying to connect before you started it up?
   - Is there any weird available space or permission problem that
     might have prevented it from creating an "os.tmpfile()" correctly?
   - What FETCHCOMMAND are you using on the client side?  Maybe I
     typoed the instructions, or you didn't copy it correctly?
     I usually use the fancy "fetcher" wrapper script instead of
     the one-liner, but it shouldn't really matter.
   - Do you have any weird modfications in
     /etc/conf.d/http-replicator.conf or the systemd
     http-replicator.service.conf equivalent (wherever it actually
     lives)?
   - Can you confirm it is using python 2.7?  I don't think the ebuild
     will try to use anything else, but I'm not an expert on portage
     nor python.
   - Is that final line of "0" literal, or just a sloppy cut-N-paste?
     There isn't anything obvious that could write a plain digit like
     that.
Comment 39 far jump 2016-04-02 16:28:22 UTC
   - Was "Connection refused" the first error?  Or was that after
     the service had died during a previous download attempt?
The connection is refused, first time and every time from every linux box on the local network (192.168.1.*)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   - How repeatable is it?  All the time, or intermittent?  How
     many times did you try?
The error happens every time, from every workstation on every emerge that needs to fetch a file.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   - How did you stop/start/restart it after emerging the new version?
     Did you try confirming it was shut down before starting it again?
     Was something already trying to connect before you started it up?
/etc/init.d/http-replicator restart
/etc/init.d/http-replicator stop
/etc/init.d/http-replicator start

>ps ax | grep http-r
29562 pts/11   S+     0:00 grep --colour=auto http-r

>/etc/init.d/http-replicator start
 * Starting Http-Replicator                                    [ ok ]

>ps ax | grep http-r                                       
29591 ?        S      0:00 /usr/bin/python3.4 /usr/lib/python-exec/python2.7/http-replicator -s -f --pid /var/run/http-replicator.pid --daemon --dir /var/cache/http-replicator --user portage --alias /usr/portage/packages:packages --log /var/log/http-replicator.log --ip 192.168.*.* --ip 10.*.*.* --port 8080

No something was NOT already trying to connect before I started it up
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   - Is there any weird available space or permission problem that
     might have prevented it from creating an "os.tmpfile()" correctly?
No
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
  - What FETCHCOMMAND are you using on the client side?  Maybe I
     typoed the instructions, or you didn't copy it correctly?
     I usually use the fancy "fetcher" wrapper script instead of
     the one-liner, but it shouldn't really matter.
wget,  I don't mess with the default setup for Gentoo Portage, every linux box on every network has portage setup the exact same way.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   - Do you have any weird modfications in
     /etc/conf.d/http-replicator.conf or the systemd
     http-replicator.service.conf equivalent (wherever it actually
     lives)?
No, I don't mess with the standard defaults, I figure that you and the guys/girls at Gentoo are smarter than I am.  see attachted conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   - Can you confirm it is using python 2.7?  I don't think the ebuild
     will try to use anything else, but I'm not an expert on portage
     nor python.
Yes, see the output a few lines above
>eselect python list
Available Python interpreters:
  [1]   python2.7
  [2]   python3.4 *
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   - Is that final line of "0" literal, or just a sloppy cut-N-paste?
     There isn't anything obvious that could write a plain digit like
     that.
yes, sloppy cut-N-Past its the 0 of 01 Apr of the next line that followed
(that line was me downgrading to 3.0-r4 and starting it up - which works perfectly)
Comment 40 far jump 2016-04-02 16:29:17 UTC
Created attachment 429496 [details]
conf file
Comment 41 far jump 2016-04-02 16:36:57 UTC
Directly after I answered your questions I re-emerged 3.0-r4

>>> Installing (1 of 1) net-proxy/http-replicator-3.0-r4::gentoo
I did: /etc/init.d/http-replicator restart

 * Starting Http-Replicator ...                                    [ ok ]

02 Apr 2016 12:11:57 INFO: HttpReplicator started
02 Apr 2016 12:31:05 INFO: HttpReplicator terminated
02 Apr 2016 12:31:05 INFO: HttpReplicator started
02 Apr 2016 12:32:43 STAT: HttpClient 1 bound to 192.168.1.219
02 Apr 2016 12:32:43 INFO: HttpClient 1 proxy request for http://distfiles.gentoo.org/distfiles/babl-0.1.12.tar.bz2
02 Apr 2016 12:32:43 INFO: HttpServer 1 serving file from cache
02 Apr 2016 12:32:43 STAT: HttpClient 1 received 642234 bytes
02 Apr 2016 12:32:43 STAT: HttpClient 2 bound to 192.168.1.219
02 Apr 2016 12:32:43 INFO: HttpClient 2 proxy request for http://distfiles.gentoo.org/distfiles/gegl-0.2.0.tar.bz2
02 Apr 2016 12:32:43 INFO: HttpServer 2 serving file from cache
02 Apr 2016 12:32:48 STAT: HttpClient 2 received 7502040 bytes
02 Apr 2016 12:32:48 STAT: HttpClient 3 bound to 192.168.1.219
02 Apr 2016 12:32:48 INFO: HttpClient 3 proxy request for http://distfiles.gentoo.org/distfiles/gimp-2.8.14.tar.bz2
02 Apr 2016 12:32:48 INFO: HttpServer 3 serving file from cache
02 Apr 2016 12:33:00 STAT: HttpClient 3 received 20440077 bytes
02 Apr 2016 12:35:29 STAT: HttpClient 4 bound to 192.168.1.114
02 Apr 2016 12:35:29 INFO: HttpClient 4 proxy request for http://distfiles.gentoo.org/distfiles/libasyncns-0.8.tar.gz
02 Apr 2016 12:35:29 INFO: HttpServer 4 serving file from cache
02 Apr 2016 12:35:29 STAT: HttpClient 4 received 341591 bytes
02 Apr 2016 12:35:29 STAT: HttpClient 5 bound to 192.168.1.114
02 Apr 2016 12:35:29 INFO: HttpClient 5 proxy request for http://distfiles.gentoo.org/distfiles/webrtc-audio-processing-0.1.tar.xz
02 Apr 2016 12:35:29 INFO: HttpServer 5 serving file from cache
02 Apr 2016 12:35:29 STAT: HttpClient 5 received 392540 bytes
02 Apr 2016 12:35:30 STAT: HttpClient 6 bound to 192.168.1.114
02 Apr 2016 12:35:30 INFO: HttpClient 6 proxy request for http://distfiles.gentoo.org/distfiles/sbc-1.3.tar.xz
02 Apr 2016 12:35:30 INFO: HttpServer 6 serving file from cache
02 Apr 2016 12:35:30 STAT: HttpClient 6 received 246292 bytes
02 Apr 2016 12:35:30 STAT: HttpClient 7 bound to 192.168.1.114
02 Apr 2016 12:35:30 INFO: HttpClient 7 proxy request for http://distfiles.gentoo.org/distfiles/pulseaudio-8.0.tar.xz
02 Apr 2016 12:35:30 STAT: HttpServer 7 bound to distfiles.gentoo.org
02 Apr 2016 12:35:30 INFO: HttpServer 7 serving file from remote host
02 Apr 2016 12:35:32 STAT: HttpServer 7 sent 1517656 bytes
02 Apr 2016 12:35:32 INFO: HttpServer 7 cached pulseaudio-8.0.tar.xz
02 Apr 2016 12:35:32 STAT: HttpClient 7 received 1517656 bytes
02 Apr 2016 12:35:32 STAT: HttpClient 8 bound to 192.168.1.114
02 Apr 2016 12:35:32 INFO: HttpClient 8 proxy request for http://distfiles.gentoo.org/distfiles/alsa-plugins-1.0.29.tar.bz2
02 Apr 2016 12:35:32 INFO: HttpServer 8 serving file from cache
02 Apr 2016 12:35:32 STAT: HttpClient 8 received 366077 bytes
02 Apr 2016 12:35:53 STAT: HttpClient 9 bound to 192.168.1.101
02 Apr 2016 12:35:53 INFO: HttpClient 9 proxy request for http://distfiles.gentoo.org/distfiles/babl-0.1.12.tar.bz2
02 Apr 2016 12:35:53 INFO: HttpServer 9 serving file from cache
02 Apr 2016 12:35:53 STAT: HttpClient 9 received 642234 bytes
02 Apr 2016 12:35:53 STAT: HttpClient 10 bound to 192.168.1.101
02 Apr 2016 12:35:53 INFO: HttpClient 10 proxy request for http://distfiles.gentoo.org/distfiles/gegl-0.2.0.tar.bz2
02 Apr 2016 12:35:53 INFO: HttpServer 10 serving file from cache
02 Apr 2016 12:35:53 STAT: HttpClient 10 received 7502040 bytes
02 Apr 2016 12:35:53 STAT: HttpClient 11 bound to 192.168.1.101
02 Apr 2016 12:35:53 INFO: HttpClient 11 proxy request for http://distfiles.gentoo.org/distfiles/gimp-2.8.14.tar.bz2
02 Apr 2016 12:35:53 INFO: HttpServer 11 serving file from cache
02 Apr 2016 12:35:54 STAT: HttpClient 11 received 20440077 bytes

Everything works from every workstation, on every emerge request
Comment 42 Matthew Ogilvie 2016-04-02 22:22:41 UTC
Created attachment 429522 [details]
new ebuild 3.0-r5 with patches, elog instructions, and portage updates (3)

This new ebuild uses "python_*script" instead of "python_*exe" to install the python scripts, so that they use the correct version of the python interpreter, regardless of "eselect python list" setting.  It is just two lines different.

I'm not sure if it should bump to -r6 or not; I've left it -r5 as attached.  Should this bug be reopened somehow, and/or a new bug number?

-----

@far jump:

1. It was using python 3.4 on your machine ("/usr/bin/python3.4" tipped me off), not 2.7, and nobody has tried to update http-replicator to work with python 3.  The ebuild line "PYTHON_COMPAT=( python2_7 )" is insufficient by itself, even though /usr/bin/http-replicator is installed as a symlink to some kind of wrapper executable (apparently 2 in "python-exec2" has nothing to do with python version 2).  I was able to duplicate your problem by temporarily setting my machine to "eselect python set 2" (python 3 main default).

2. You could probably work around the problem by setting "eselect python set 1" (python 2), until someone pushes this new ebuild in.  (Unless you need your main default to be version 3 for some reason.  Be aware that both 2 and 3 also have sub-defaults, but they are awkward to see or modify and probably don't matter if you only have one of each installed.  See "eselect python --help".).

3. Minor FYI: The first error was actually (use -f to avoid the auto-background download thing):
> Resolving localhost... 127.0.0.1
> Connecting to localhost|127.0.0.1|:8082... connected.
> Proxy request sent, awaiting response... No data received.
Then http-replicator dies, and subsequent automatic retry attempts give the connection refused error.  (Note I use port 8082 instead of 8080 on my network.)

4. If you actually want to be able to update flash player without manually working around the cache collisions for the insufficiently versioned install_flash_player_11_linux.{ARCH}.tar.gz files (which only get renamed to include the full version on the client side), you'll need to modify the client machines' FETCHCOMMAND's to include the "X-unique-cache-name" header in the HTTP request, as suggested in the elogged output from "emerge http-replicator" (the rest of the FETCHCOMMAND is the same as the default embedded in sys-apps/portage).  Otherwise http-replicator will not have enough information to know that it should use a better/unique name for caching specific versions of the file.
    Potentially sys-apps/portage might eventually change the default to avoid needing to modify it manually, but given that http-replicator is not officially supported (no maintainer...), I wouldn't hold my breath.
Comment 43 far jump 2016-04-02 22:54:18 UTC
1. It was using python 3.4 on your machine ("/usr/bin/python3.4" tipped me off)

Ah yes, so it was... my eye when to python-exec/python2.7/, should have read the entire line closely.

hummm.... why didn't the previous version use 3.4?
Comment 44 Matthew Ogilvie 2016-04-03 00:05:33 UTC
> hummm.... why didn't the previous version use 3.4?

For the ebuild: I had to research how to update the deprecated "python.eclass" in the ebuild as requested in comment 31, and I made a mistake when I made the changes.  Newer techniques are supposed to improve how it deals with multiple python versions, but the details of how to write the ebuild are significantly different.

I think http-replicator itself substantially pre-dates python 3, and nobody has updated it to work with python 3, which has significant differences.
Comment 45 Sam Jorna (wraeth) gentoo-dev 2016-04-03 05:06:54 UTC
Yes, this should be a revision bump as it's resolving a specific runtime error, effectively introducing functionality to users since it's currently broken. I can fix the revision as I commit it if you would like, rather than attaching another copy of the ebuild.

However in an effort to avoid unnecessary continued revbumping in case attachment 429522 [details] does not resolve the issue, I would appreciate far jump testing this before I commit it - I would do it myself however I'm not familiar enough with the bug to runtest it effectively.

far jump, if you're willing to do so, you can test this by adding it to a local overlay [0] and emerging explicitly from that overlay with `emerge -1a net-proxy/http-replicator::<overlay_name>`. Alternatively, let me know and I can add it to my own overlay which you can clone and use.

Also, just an FYI: Matthew, given your continued persistence with this, if you decide you do wish to become the proxied maintainer for this, just let me know.

Thank you both for your continued work on this!

0: https://wiki.gentoo.org/wiki/Overlay/Local_overlay
Comment 46 far jump 2016-04-03 16:55:58 UTC
>if you're willing to do so, you can test this ...

Glad to!

I emerged this new ebuild from a local overlay.  http-replicator emerged w/o any problems and is functioning perfectly.  Now we need to wait for Adobe to upgrade flash.
Comment 47 Ian Delaney (RETIRED) gentoo-dev 2016-04-10 08:14:09 UTC
This package is also without any maintainer, proxy or otherwise.  If someone like far.jmp@gmail.com is not willing to be proxy maintainer, then proxy-maint@g.o ought be removed form CC
Comment 48 far jump 2016-04-10 16:26:59 UTC
Sure, I am willing to that.
Comment 49 far jump 2016-04-15 19:46:20 UTC
FYI - the orginal problem still remains

Proxy request sent, awaiting response... 200 OK
Length: 7279100 (6.9M)
Saving to: ‘/usr/portage/distfiles/adobe-flash-11.2.202.616.x86_64.tar.gz’

/usr/portage/distfi 100%[=====================>]   6.94M  --.-KB/s   in 0.06s  

2016-04-15 15:44:51 (112 MB/s) - ‘/usr/portage/distfiles/adobe-flash-11.2.202.616.x86_64.tar.gz’ saved [7279100/7279100]

!!! Fetched file: adobe-flash-11.2.202.616.x86_64.tar.gz VERIFY FAILED!
!!! Reason: Filesize does not match recorded size
!!! Got:      7279100
!!! Expected: 7280774
Refetching... File renamed to '/usr/portage/distfiles/adobe-flash-11.2.202.616.x86_64.tar.gz._checksum_failure_.0kv82X'

!!! Couldn't download 'adobe-flash-11.2.202.616.x86_64.tar.gz'. Aborting.
 * Fetch failed for 'www-plugins/adobe-flash-11.2.202.616', Log file:
 *  '/var/tmp/portage/www-plugins/adobe-flash-11.2.202.616/temp/build.log'

>>> Failed to emerge www-plugins/adobe-flash-11.2.202.616, Log file:
Comment 50 Matthew Ogilvie 2016-04-16 00:36:27 UTC
1. Did you remember to override FETCHCOMMAND on EVERY client machine's make.conf (or at least all of them that need to update flash), as mentioned in the elogged output, bullet 4 of comment 42, and other comments?  It won't work without it.

2. Do you have the correct version installed and running on the server, considering multiple versions of the ebuild, local overlays, etc?  You should be able to check the actually installed script:
> $ grep x-unique-cache-name /usr/lib/python-exec/python2.7/http-replicator
>                        uniqueCacheName = body.get('x-unique-cache-name')
> $
Comment 51 Ian Delaney (RETIRED) gentoo-dev 2016-04-17 03:11:49 UTC
Yes they rsyncing in gentoo has a recent history of deficits. Worth checking all those through.
Comment 52 Pacho Ramos gentoo-dev 2016-08-12 13:38:49 UTC
the original bug should be fixed in -r6