I have a local rsync server for Gentoo portage. (to prevent the many systems from sync Gentoo mirror sites and repeatedly downloading the same files). From time to time, a system will attempt to emerge adobe-flashplayer and will recieve a file size error for: install_flash_player_11_linux.x86_64.tar.gz or install_flash_player_11_linux.i386.tar.gz If I tell that linux box to ignore the local Gentoo mirror and rsync server, and allow it acces to the Internet, flash player will install just fine. here is the fix: I go to /var/cache/http-replicator/ on the rsync server and delete the offending install_flash_player_11_linux.*.tar.gz file and everything works -- ie the local systems will emerge the newest flash player perfectly. Some how portage for local mirror sites does not update install_flash_player_11_linux.*.tar.gz when it changes.
Created attachment 329342 [details] emerge --info for rsync server
It shouldn't be caching install_flash_player_11_linux.x86_64.tar.gz but adobe-flash-*.tar.gz instead.
(In reply to comment #2) > It shouldn't be caching install_flash_player_11_linux.x86_64.tar.gz > but adobe-flash-*.tar.gz instead. Do I have any control over what is caches and what it does not cache? Does it not just download what the ebuild demands?
(In reply to comment #2) > It shouldn't be caching install_flash_player_11_linux.x86_64.tar.gz > but adobe-flash-*.tar.gz instead. install_flash_player_11_linux.x86_64.tar.gz is the filename of what is downloaded: >>> Emerging (1 of 6) www-plugins/adobe-flash-11.2.202.251 * Fetching files in the background. To view fetch progress, run * `tail -f /var/log/emerge-fetch.log` in another terminal. >>> Downloading 'http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.251/install_flash_player_11_linux_x86_64.tar.gz' The problem is Adobe is encoding the version # in a directory component and not the filename. http_replicator flattens the files it caches into a single directory discarding directories in the process. This causes the hash failure when a new version is released. http_replicator log from my server shows that the version # is the directory name before the filename: 31 Jan 2012 12:31:49 INFO: HttpClient 12250 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.1.102.55/install_flash_player_11_linux.x86_64.tar.gz 19 Apr 2012 15:25:17 INFO: HttpClient 50676 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.228/install_flash_player_11_linux.x86_64.tar.gz 16 May 2012 15:38:07 INFO: HttpClient 59112 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.235/install_flash_player_11_linux.x86_64.tar.gz 09 Aug 2012 12:05:00 INFO: HttpClient 2527 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.236/install_flash_player_11_linux.x86_64.tar.gz 06 Sep 2012 09:12:16 INFO: HttpClient 10925 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.238/install_flash_player_11_linux.x86_64.tar.gz 24 Oct 2012 16:20:14 INFO: HttpClient 5880 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.243/install_flash_player_11_linux_x86_64.tar.gz 09 Nov 2012 16:43:42 INFO: HttpClient 11148 proxy request for http://fpdownload.macromedia.com/get/flashplayer/pdc/11.2.202.251/install_flash_player_11_linux_x86_64.tar.gz
Does this still happen with the latest version? Could you report this upstream?
Same bug is annoying me too. Is upstream still supporting http-replicator anyway? Latest version (4.0alpha2) is dated January 2008! If not so, is there a chance for whoever to patch latest stable?
(In reply to Tom Wijsman (TomWij) from comment #5) > Does this still happen with the latest version? > > Could you report this upstream? Yes it is still happening.
(In reply to Tom Wijsman (TomWij) from comment #5) > Does this still happen with the latest version? > > Could you report this upstream? emerge -u world included adobe flash update problem still happening
Which version is this? Has this since been reported upstream?
www-plugins/adobe-flash Available versions: 11.2.202.33 who do you mean upstream? why would adobe do anything? do you think that adobe will accommodate gentoo emerge/resync protocols? Any way the gentoo repositories that my main server sync from do not seem to have this problem? what is gentoo doing that I my resync server is not doing?
When http-replicator was created I knew there were going to be files that were just not cacheable due to things beyond gentoo's control. I created repcacheman to solve problems like this and included it with http-replicator. My recommendation was to move all files from the cache dir to the distfile dir and run repcacheman on a regular basis (say weekly to monthly). This info was in the original forum thread still listed in the ebuild but probably needs to go in the man page or similar. repcacheman checksums all the files and moves verified files back to the cache. The script then tells you to delete all files left in the distfile dir. Adobe's install_flash_player_11_linux.x86_64.tar.gz will be one of the files left for deletion. There will be other files you need to delete also.
Hi Tom, First of all - thanks for creating repcacheman! I have been using it from the very very beginning of my Gentoo days. unfortunately, that logic has holes it. What if adobe_flash updates in between cycles of moving all cache'd files to ./distfiles/ and then scrubbing them? You might find that you must tighten the refresh cycle until your doing every day? -- seems like using an atom bomb to kill a rabbit. It was just easier to write a script using eix and ruby to detect an adobe update and then just preemptively delete the offending file.
(In reply to Tom Poplawski from comment #11) > When http-replicator was created I knew there were going to be files that > were just not cacheable due to things beyond gentoo's control. > > I created repcacheman to solve problems like this and included it with > http-replicator. ... I run http-replicator on a VM that does very little(*) so looking at the contents of /usr/portage/distfiles/ is somewhat pointless. What about modifying repcacheman to manage the files in /var/cache/http-replicator/ and delete files that fail the xsum check, or are no longer referenced by an ebuild in the tree? repcacheman could run as a post-sync step to minimize the delay in keeping the tree and cache in sync. This would not catch the case where a dist does a slipstream update (change binary without changing version #) but would work well for regular updates. (*) portage mirror, http-replicator cache, and local http overlay+distfiles for ~100 servers.
(In reply to far jump from comment #12) > Hi Tom, > First of all - thanks for creating repcacheman! Your welcome. > What if adobe_flash updates in > between cycles We both agree on your point, you made a fix for this particular problem with adobe only, that works for now. I tried to make a more general fix, that works most of the time, for most problems, if used as recommended. By the way I couldn't find your script, have you shared it somewhere for others to use? Did you ever run repcacheman and see the other files that are a problem too? Are you going to write scripts for those also? Don't forget to write a script for ftp files. http-replicator can't cache those, but repcacheman will add those to the cache automatically. By the way, I looked at the actual adobe update release dates: 11.6 March 12th, 2013 11.7 June 11th, 2013 11.8 September 24th, 2013 11.9 December 10th, 2013 12.0 February 20th, 2014 I made the original recommendation to run repcacheman weekly or monthly back in 2004. I didn't check adobe back that far but I hope all who followed my recommendation would have probably never known about this problem with adobe flash.
(In reply to David Sparks from comment #13) > What about > modifying repcacheman to manage the files in /var/cache/http-replicator/ and > delete files that fail the xsum check, or are no longer referenced by an > ebuild in the tree? Good suggestion. I looked at this back in 2004 but not since. Back then there were about a dozen distfile pruning scripts floating around the forums. Nobody could agree on what to remove so you could pick a script that met your needs. I still believe this is the case. I didn't want to duplicate the efforts of others or impose my wishes for what gets pruned. Isn't there an offical gentoo supported script now? eclean-dist is it? Many scripts supported other than the offical distfile dir. I think DISTDIR=/var/cache/http-replicator/ eclean-dist will work almost as good as repcacheman :-) > (*) portage mirror, http-replicator cache, and local http overlay+distfiles > for ~100 servers. Wow, nice to know it's useful :-)
Created attachment 392616 [details, diff] add x-unique-cache-name to http-replicator-3.0 Ebuilds and portage already know how to uniqueify download file names via the "URL -> cacheFile" syntax for the SRC_URI variable. We just need to enhance http-replicator to use the cacheFile name for proxy caching purposes as well, which is fairly easy: Here is a patch for http-replicator 3.0 to recognize and use an "X-unique-cache-name" HTTP header, if provided in the request and the replicator is running in "--flat" mode (which is default). You also need to modify the FETCHCOMMAND to supply this header in make.conf unless the default is changed in portage: FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O \"\${DISTDIR}/\${FILE}\" --header=\"X-unique-cache-name: \${FILE}\" \"\${URI}\""
Created attachment 392618 [details, diff] modify portage's default FETCHCOMMAND to supply new header This patch should change portage's default to supply the x-unique-cache-name header in the FETCHCOMMAND and RESUMECOMMAND, although I haven't actually tested this patch (I just override FETCHCOMMAND in make.conf). But perhaps something smarter would be desired, where it only adds the header if http_proxy is defined and the last component of URI is different from FILE? Perhaps install a customizable /etc/portage/fetchcommand wrapper script that has such smarts, and default FETCHCOMMAND to that? Or some other way to make it conditional (more variables)?
Created attachment 392620 [details, diff] add x-unique-cache-name to http-replicator-4.0.x Here is a corresponding patch for the completely rewritten http-replicator 4.0.x. I haven't tested this at all, partly because I reading through the code, I noticed what appears to be at least two pre-existing bugs in version 4: 1. If it is not in "--flat" mode, then there does not appear to be anything to prevent a custom client from requesting "/../../../../../any/file/on/the/filesystem" (standard clients will simplify ".."s before sending the request, but see http://stackoverflow.com/questions/5951029/read-http-output-using-shell-bash-script ) as long as the UID the process is running as can read it, including /etc/passwd or similar. There may also be problems with writing to weird locations, although standard permissions renders this less likely. This is a SECURITY bug, but fortunately (a) --flat mode is default on gentooo, (b) gentoo still defaults to http-replicator-3.0 on stable systems, which doesn't have this bug, and (c) 4.0 releases are still marked "alpha"... 2. HTTP header names are not supposed to be case-sensitive, but version 4 does nothing to fold the case... I'll let gentoo developers decide what if anything to do about these apparent bugs.
Would you be willing to proxy maintain http-replicator? https://wiki.gentoo.org/wiki/Project:Proxy_Maintainers
I think I have time to do minor levels of proxy maintenance, especially for version 3.0. 3.0 seems fairly solid, and since I use it I have an incentive to fix problems. I consider an http-replicator (along with an rsync mirror) to be a required/standard setup on any LAN that has more than one gentoo machine, regardless of the speed of its Internet connection. I'm not sure when/if I can do more involved maintenance, such as digging into the version 4.0 apparent bugs described in comment 18, or actively searching for similar ones. Especially since I don't really know python. It took me years to decide to look into this "which name is it using in the cache" bug (instead of just working around it), although it was fairly easy after I actually got started... (I wonder if version 4.0 should be package.mask'ed until me or someone actually looks into comment 18's bugs? But the /etc/conf.d/http-replicator file changes dramatically, so extra transitions between 3 and 4 could be annoying to anyone currently using the testing (~) version.)
Created attachment 393508 [details] new 3.0-r5 ebuild applies patch and elogs instructions 3.0-r5's changes compared to -r4: - Applies patch from comment 16. - Adds some more instructions to pkg_postinst(), and elog's them instead of einfo. (Plus one ewarn for bug 502574.) - Keyworded for testing (~) only, so perhaps it could go in as-is. Possible future enhancements not included: - I'm not sure the best way to handle client machine config. See next comment. The wording of the elog messages may change depending on what is decided. - Perhaps instructions should be installed somewhere (man page or /usr/share/doc), not just elog'ed and referenced to external forums... - Perhaps the init and systemd scripts should check for the cache and suggest running repcacheman, rather than just letting http-replicator error out. (bug 502574)
Created attachment 393512 [details] wget wrapper script to be intelligent about adding extra header I'm not sure the best way to reconfigure client machines to take advantage of the new header (comment 16 and comment 21) to avoid this caching bug. This script only adds the new header when necessary. FETCHCOMMAND and RESUMECOMMAND can refer to this script instead of calling wget directly. Note that this probably shouldn't be packaged with http-replicator itself. Options: - Maybe this script isn't necessary. The new header probably doesn't hurt anything (except a minuscule amount of extra network traffic) if you just send it unconditionally. - Maybe it is OK to require that users manually override FETCHCOMMAND and RESUMECOMMAND on client machines? - Maybe the default FETCHCOMMAND and RESUMECOMMAND in make.globals (from sys-apps/portage) should already be setup to add the header, either unconditionally (comment 17), or via this "fetcher" script? - Perhaps there should be an "http-replicator-client" package that includes this script and makes appropriate changes to make.conf to be merged with something like etc-update? Depending on the desired solution, the elog'ed instructions in comment 21's ebuild may need to change.
To use the script from comment 22, add these to make.conf: FETCHCOMMAND="/etc/portage/fetcher -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\"" RESUMECOMMAND="/etc/portage/fetcher -c -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
Created attachment 393518 [details] wget wrapper script to be intelligent about adding extra header Newer version of wget wrapper script: fix f_args name and add comments.
Well, http-replicator is completely orphan for some time and, then, even if you will do most for 3.x versions it's much better than current situation
Created attachment 394206 [details, diff] add x-unique-cache-name to http-replicator-3.0 Patch: add x-unique-cache-name support to http-replicator. Very minor tweaks to patch from comment 16: filename tweak, no email in patch [maybe avoid spam], and remove superfluous ';'.
Created attachment 394208 [details, diff] patch for 3.0: suggest repcacheman if cache doesn't exist Suggest running repcacheman if cache doesn't exist. This really goes with bug 502574, but it might be handy to have everything for the new ebuild I'm about to post to all be collected under one bug.
Created attachment 394210 [details] new ebuild 3.0-r5 applies patches and elogs instructions Compared to comment 21, this ebuild also applies a patch to remind users to use repcacheman if the cache doesn't exist (for bug 502574). It requires the patches from comments 26 and 27. I'm still not sure how to deal with client-side configuration. See earlier comments.
@proxy-maint, could you please take a look to this? Current ebuild in stable has a lot of bugs and some of them have patches provided by Matthew that maybe could be accepted finally Thanks
Then I take it your are prompting Matthew Ogilvie as to whether he'd consider being a proxy maintainer, is it a proxied maintainer. I you do, feel to peruse https://wiki.gentoo.org/wiki/Project:Proxy_Maintainers and also to join in the channel listed there we offer first hand support to users and commit submitted patches and bumps to the portage tree. Clearly pacho is keen to get this package back on the right rails.
Matthew, are you still interesting in maintaining this package? If not, or if there's no reply on this bug soon, I will commit fix and commit this to close this bug. If you are, there are a couple of points with the ebuild in attachment 394210 [details] that should be addressed: - The header needs to be updated to current (see ${PORTDIR}/header.txt) - Current allowed EAPI's are 5 and 6 (6 being preferable) - python.eclass has been deprecated, please migrate to python-r1.eclass or similar Otherwise the ebuild passes my build tests in it's current state, so good work! If you wish to proxy-maintain this package, then please address these last issues and re-attach. Thanks!
Created attachment 428742 [details] new ebuild 3.0-r5 with patches, elog instructions, and portage updates New version of 3.0 ebuild that addresses EAPI, python eclass, newer header, and newer path to make.conf, as requested by Ian Delaney in comment 31. These are the only changes compared to attachment 394210 [details] from comment 28. Unchanged from attachment 394210 [details]: - Users still need to override FETCHCOMMAND to fully fix this issue, although it degrades to previous behavior without it. - Improved startup error message for bug 502574. - FYI: Bug 524208 is partially a duplicate of this issue, although not completely. (If a source URI file is changed in-place, using a unique cache name doesn't really help.) - The ebuild now elog's important setup instructions after installation. - Version 4.0_alpha2's ebuild could probably use another update with similar changes for newer portage. It could reasonably be argued that 4 has a cleaner internal design and it would eventually be worth switching, but currently 4.0 is still very "alpha" (incomplete/buggy). (See comment 18.) I don't really want to spend much time maintaining http-replicator, but I'm at least willing to try to get my version 3.0 patches working well.
There are a couple of minor last-minute tweaks to this that I picked up as I was giving it a final inspection: - IUSE, DEPEND and RDEPEND do not need to be defined if they are blank - patching should occur in src_prepare()[0] - added call to 'default' to src_prepare() to cater for eapply_user[0]. I can make these changes as I commit the ebuild since they are only minor; however if you wish to do it and re-attach, you can of course do so. (In reply to Matthew Ogilvie from comment #32) > I don't really want to spend much time maintaining http-replicator, but I'm > at least willing to try to get my version 3.0 patches working well. So just to be clear, do you wish to become the proxy maintainer of this package? This will mean that the existing net-proxy/http-replicator bugs will be assigned to you and it is expected that you will begin attempting to address these - from the sounds of it you have already familiarised yourself with them. If you are only wishing to submit patches on some bugs but do not wish to proxy-maintain, that's fine, I just need to clarify in order to commit it! 0: https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-1000009.1.5 Thanks!
A quick note I forgot to mention in my last comment. Since the only thing you're doing in src_prepare() is applying patches, you can list them in the PATCHES variable and leave src_prepare() undefined. The default definition of src_prepare() in EAPI6 applies any patches listed in the ${PATCHES} array, as well as runs eapply_user. As an example, see ${PORTDIR}/dev-db/mysql-connector-c++/mysql-connector-c++-1.1.3.ebuild
Created attachment 428792 [details] new ebuild 3.0-r5 with patches, elog instructions, and portage updates (2) Essentially the same as comment 32. Includes a couple of more portage best practices tweaks from comment 33 and comment 34. After this makes it in, it will likely be years (or never) before I do anything to http-replicator again, unless some high-priority issue affects me personally. So I guess that meets the definition of an occasional patch submitter rather than a proxy maintainer? I have skimmed through "http-replicator" bugs; I think all the ones I consider high-priority are handled by this ebuild, or at least improved. Besides bugs from comment 32: I think my ebuild also effectively fixes bug 561674, although I'm not sure how -r4 became broken, because the last time I tried it (more than a year ago), it seemed to install and work as expected. Maybe the 21 Mar 2015 ChangeLog-2015 entry broke it?
This has now been committed. If possible, can you please check and comment on the bugs you believe this should resolve? Thanks for the contribution! commit 91d6d053b55f3c230fa90dc257d0bc994c3bc5e5 Author: Sam Jorna <wraeth@gentoo.org> Date: Wed Mar 23 21:58:13 2016 +1100 net-proxy/http-replicator: revbump, bug 442874 Adds X-Unique-Cache-Name header to try and catch files based on their saved name in DISTFILES. Additionally bumps ebuild to EAPI6 and elogs information for required setup. Ebuild supplied by Matthew Ogilvie.
emerged http-replicator-3.0-r5 - does not work. Client side: Resolving eye... 192.168.1.106 Connecting to eye|192.168.1.106|:8080... failed: Connection refused. Host side: 01 Apr 2016 11:40:48 STAT: HttpClient 1 bound to 192.168.1.101 01 Apr 2016 11:40:48 ERROR: HttpClient 1 caught an exception in __getattr__: HttpClient instance has no attribute 'data' 0 I when back to 3.0-r4 - everything works perfectly (except of course => !!! Fetched file: adobe-flash-11.2.202.577.x86_64.tar.gz VERIFY FAILED! !!! Reason: Filesize does not match recorded size !!! Got: 7278446 !!! Expected: 7279100 Refetching... File renamed to '/usr/portage/distfiles/adobe-flash-11.2.202.577.x86_64.tar.gz._checksum_failure_.d2P1MV'
> 01 Apr 2016 11:40:48 STAT: HttpClient 1 bound to 192.168.1.101 > 01 Apr 2016 11:40:48 ERROR: HttpClient 1 caught an exception in __getattr__: HttpClient instance has no attribute 'data' > 0 I haven't been able to duplicate such a problem. I don't see where / how it could be missing the attribute 'data'. It seems to be created during construction (with a os.tmpfile()), immediately after logging the "bound to" message. Maybe it somehow tried to handle a connection while it was still starting up initially, although I don't even see how that could generate these symptoms. On the remote chance that systemd's invocation was poorly interacting with my patches, I tried manually/directly running http-replicator from a root command prompt without --pid, --daemon, or --log in an attempt to mimic what systemd would do, but it still worked fine. Possible Ideas and Questions: (I don't know if any of these will help, but they are all I can think of.) - Was "Connection refused" the first error? Or was that after the service had died during a previous download attempt? - How repeatable is it? All the time, or intermittent? How many times did you try? - How did you stop/start/restart it after emerging the new version? Did you try confirming it was shut down before starting it again? Was something already trying to connect before you started it up? - Is there any weird available space or permission problem that might have prevented it from creating an "os.tmpfile()" correctly? - What FETCHCOMMAND are you using on the client side? Maybe I typoed the instructions, or you didn't copy it correctly? I usually use the fancy "fetcher" wrapper script instead of the one-liner, but it shouldn't really matter. - Do you have any weird modfications in /etc/conf.d/http-replicator.conf or the systemd http-replicator.service.conf equivalent (wherever it actually lives)? - Can you confirm it is using python 2.7? I don't think the ebuild will try to use anything else, but I'm not an expert on portage nor python. - Is that final line of "0" literal, or just a sloppy cut-N-paste? There isn't anything obvious that could write a plain digit like that.
- Was "Connection refused" the first error? Or was that after the service had died during a previous download attempt? The connection is refused, first time and every time from every linux box on the local network (192.168.1.*) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - How repeatable is it? All the time, or intermittent? How many times did you try? The error happens every time, from every workstation on every emerge that needs to fetch a file. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - How did you stop/start/restart it after emerging the new version? Did you try confirming it was shut down before starting it again? Was something already trying to connect before you started it up? /etc/init.d/http-replicator restart /etc/init.d/http-replicator stop /etc/init.d/http-replicator start >ps ax | grep http-r 29562 pts/11 S+ 0:00 grep --colour=auto http-r >/etc/init.d/http-replicator start * Starting Http-Replicator [ ok ] >ps ax | grep http-r 29591 ? S 0:00 /usr/bin/python3.4 /usr/lib/python-exec/python2.7/http-replicator -s -f --pid /var/run/http-replicator.pid --daemon --dir /var/cache/http-replicator --user portage --alias /usr/portage/packages:packages --log /var/log/http-replicator.log --ip 192.168.*.* --ip 10.*.*.* --port 8080 No something was NOT already trying to connect before I started it up ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Is there any weird available space or permission problem that might have prevented it from creating an "os.tmpfile()" correctly? No ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - What FETCHCOMMAND are you using on the client side? Maybe I typoed the instructions, or you didn't copy it correctly? I usually use the fancy "fetcher" wrapper script instead of the one-liner, but it shouldn't really matter. wget, I don't mess with the default setup for Gentoo Portage, every linux box on every network has portage setup the exact same way. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Do you have any weird modfications in /etc/conf.d/http-replicator.conf or the systemd http-replicator.service.conf equivalent (wherever it actually lives)? No, I don't mess with the standard defaults, I figure that you and the guys/girls at Gentoo are smarter than I am. see attachted conf ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Can you confirm it is using python 2.7? I don't think the ebuild will try to use anything else, but I'm not an expert on portage nor python. Yes, see the output a few lines above >eselect python list Available Python interpreters: [1] python2.7 [2] python3.4 * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Is that final line of "0" literal, or just a sloppy cut-N-paste? There isn't anything obvious that could write a plain digit like that. yes, sloppy cut-N-Past its the 0 of 01 Apr of the next line that followed (that line was me downgrading to 3.0-r4 and starting it up - which works perfectly)
Created attachment 429496 [details] conf file
Directly after I answered your questions I re-emerged 3.0-r4 >>> Installing (1 of 1) net-proxy/http-replicator-3.0-r4::gentoo I did: /etc/init.d/http-replicator restart * Starting Http-Replicator ... [ ok ] 02 Apr 2016 12:11:57 INFO: HttpReplicator started 02 Apr 2016 12:31:05 INFO: HttpReplicator terminated 02 Apr 2016 12:31:05 INFO: HttpReplicator started 02 Apr 2016 12:32:43 STAT: HttpClient 1 bound to 192.168.1.219 02 Apr 2016 12:32:43 INFO: HttpClient 1 proxy request for http://distfiles.gentoo.org/distfiles/babl-0.1.12.tar.bz2 02 Apr 2016 12:32:43 INFO: HttpServer 1 serving file from cache 02 Apr 2016 12:32:43 STAT: HttpClient 1 received 642234 bytes 02 Apr 2016 12:32:43 STAT: HttpClient 2 bound to 192.168.1.219 02 Apr 2016 12:32:43 INFO: HttpClient 2 proxy request for http://distfiles.gentoo.org/distfiles/gegl-0.2.0.tar.bz2 02 Apr 2016 12:32:43 INFO: HttpServer 2 serving file from cache 02 Apr 2016 12:32:48 STAT: HttpClient 2 received 7502040 bytes 02 Apr 2016 12:32:48 STAT: HttpClient 3 bound to 192.168.1.219 02 Apr 2016 12:32:48 INFO: HttpClient 3 proxy request for http://distfiles.gentoo.org/distfiles/gimp-2.8.14.tar.bz2 02 Apr 2016 12:32:48 INFO: HttpServer 3 serving file from cache 02 Apr 2016 12:33:00 STAT: HttpClient 3 received 20440077 bytes 02 Apr 2016 12:35:29 STAT: HttpClient 4 bound to 192.168.1.114 02 Apr 2016 12:35:29 INFO: HttpClient 4 proxy request for http://distfiles.gentoo.org/distfiles/libasyncns-0.8.tar.gz 02 Apr 2016 12:35:29 INFO: HttpServer 4 serving file from cache 02 Apr 2016 12:35:29 STAT: HttpClient 4 received 341591 bytes 02 Apr 2016 12:35:29 STAT: HttpClient 5 bound to 192.168.1.114 02 Apr 2016 12:35:29 INFO: HttpClient 5 proxy request for http://distfiles.gentoo.org/distfiles/webrtc-audio-processing-0.1.tar.xz 02 Apr 2016 12:35:29 INFO: HttpServer 5 serving file from cache 02 Apr 2016 12:35:29 STAT: HttpClient 5 received 392540 bytes 02 Apr 2016 12:35:30 STAT: HttpClient 6 bound to 192.168.1.114 02 Apr 2016 12:35:30 INFO: HttpClient 6 proxy request for http://distfiles.gentoo.org/distfiles/sbc-1.3.tar.xz 02 Apr 2016 12:35:30 INFO: HttpServer 6 serving file from cache 02 Apr 2016 12:35:30 STAT: HttpClient 6 received 246292 bytes 02 Apr 2016 12:35:30 STAT: HttpClient 7 bound to 192.168.1.114 02 Apr 2016 12:35:30 INFO: HttpClient 7 proxy request for http://distfiles.gentoo.org/distfiles/pulseaudio-8.0.tar.xz 02 Apr 2016 12:35:30 STAT: HttpServer 7 bound to distfiles.gentoo.org 02 Apr 2016 12:35:30 INFO: HttpServer 7 serving file from remote host 02 Apr 2016 12:35:32 STAT: HttpServer 7 sent 1517656 bytes 02 Apr 2016 12:35:32 INFO: HttpServer 7 cached pulseaudio-8.0.tar.xz 02 Apr 2016 12:35:32 STAT: HttpClient 7 received 1517656 bytes 02 Apr 2016 12:35:32 STAT: HttpClient 8 bound to 192.168.1.114 02 Apr 2016 12:35:32 INFO: HttpClient 8 proxy request for http://distfiles.gentoo.org/distfiles/alsa-plugins-1.0.29.tar.bz2 02 Apr 2016 12:35:32 INFO: HttpServer 8 serving file from cache 02 Apr 2016 12:35:32 STAT: HttpClient 8 received 366077 bytes 02 Apr 2016 12:35:53 STAT: HttpClient 9 bound to 192.168.1.101 02 Apr 2016 12:35:53 INFO: HttpClient 9 proxy request for http://distfiles.gentoo.org/distfiles/babl-0.1.12.tar.bz2 02 Apr 2016 12:35:53 INFO: HttpServer 9 serving file from cache 02 Apr 2016 12:35:53 STAT: HttpClient 9 received 642234 bytes 02 Apr 2016 12:35:53 STAT: HttpClient 10 bound to 192.168.1.101 02 Apr 2016 12:35:53 INFO: HttpClient 10 proxy request for http://distfiles.gentoo.org/distfiles/gegl-0.2.0.tar.bz2 02 Apr 2016 12:35:53 INFO: HttpServer 10 serving file from cache 02 Apr 2016 12:35:53 STAT: HttpClient 10 received 7502040 bytes 02 Apr 2016 12:35:53 STAT: HttpClient 11 bound to 192.168.1.101 02 Apr 2016 12:35:53 INFO: HttpClient 11 proxy request for http://distfiles.gentoo.org/distfiles/gimp-2.8.14.tar.bz2 02 Apr 2016 12:35:53 INFO: HttpServer 11 serving file from cache 02 Apr 2016 12:35:54 STAT: HttpClient 11 received 20440077 bytes Everything works from every workstation, on every emerge request
Created attachment 429522 [details] new ebuild 3.0-r5 with patches, elog instructions, and portage updates (3) This new ebuild uses "python_*script" instead of "python_*exe" to install the python scripts, so that they use the correct version of the python interpreter, regardless of "eselect python list" setting. It is just two lines different. I'm not sure if it should bump to -r6 or not; I've left it -r5 as attached. Should this bug be reopened somehow, and/or a new bug number? ----- @far jump: 1. It was using python 3.4 on your machine ("/usr/bin/python3.4" tipped me off), not 2.7, and nobody has tried to update http-replicator to work with python 3. The ebuild line "PYTHON_COMPAT=( python2_7 )" is insufficient by itself, even though /usr/bin/http-replicator is installed as a symlink to some kind of wrapper executable (apparently 2 in "python-exec2" has nothing to do with python version 2). I was able to duplicate your problem by temporarily setting my machine to "eselect python set 2" (python 3 main default). 2. You could probably work around the problem by setting "eselect python set 1" (python 2), until someone pushes this new ebuild in. (Unless you need your main default to be version 3 for some reason. Be aware that both 2 and 3 also have sub-defaults, but they are awkward to see or modify and probably don't matter if you only have one of each installed. See "eselect python --help".). 3. Minor FYI: The first error was actually (use -f to avoid the auto-background download thing): > Resolving localhost... 127.0.0.1 > Connecting to localhost|127.0.0.1|:8082... connected. > Proxy request sent, awaiting response... No data received. Then http-replicator dies, and subsequent automatic retry attempts give the connection refused error. (Note I use port 8082 instead of 8080 on my network.) 4. If you actually want to be able to update flash player without manually working around the cache collisions for the insufficiently versioned install_flash_player_11_linux.{ARCH}.tar.gz files (which only get renamed to include the full version on the client side), you'll need to modify the client machines' FETCHCOMMAND's to include the "X-unique-cache-name" header in the HTTP request, as suggested in the elogged output from "emerge http-replicator" (the rest of the FETCHCOMMAND is the same as the default embedded in sys-apps/portage). Otherwise http-replicator will not have enough information to know that it should use a better/unique name for caching specific versions of the file. Potentially sys-apps/portage might eventually change the default to avoid needing to modify it manually, but given that http-replicator is not officially supported (no maintainer...), I wouldn't hold my breath.
1. It was using python 3.4 on your machine ("/usr/bin/python3.4" tipped me off) Ah yes, so it was... my eye when to python-exec/python2.7/, should have read the entire line closely. hummm.... why didn't the previous version use 3.4?
> hummm.... why didn't the previous version use 3.4? For the ebuild: I had to research how to update the deprecated "python.eclass" in the ebuild as requested in comment 31, and I made a mistake when I made the changes. Newer techniques are supposed to improve how it deals with multiple python versions, but the details of how to write the ebuild are significantly different. I think http-replicator itself substantially pre-dates python 3, and nobody has updated it to work with python 3, which has significant differences.
Yes, this should be a revision bump as it's resolving a specific runtime error, effectively introducing functionality to users since it's currently broken. I can fix the revision as I commit it if you would like, rather than attaching another copy of the ebuild. However in an effort to avoid unnecessary continued revbumping in case attachment 429522 [details] does not resolve the issue, I would appreciate far jump testing this before I commit it - I would do it myself however I'm not familiar enough with the bug to runtest it effectively. far jump, if you're willing to do so, you can test this by adding it to a local overlay [0] and emerging explicitly from that overlay with `emerge -1a net-proxy/http-replicator::<overlay_name>`. Alternatively, let me know and I can add it to my own overlay which you can clone and use. Also, just an FYI: Matthew, given your continued persistence with this, if you decide you do wish to become the proxied maintainer for this, just let me know. Thank you both for your continued work on this! 0: https://wiki.gentoo.org/wiki/Overlay/Local_overlay
>if you're willing to do so, you can test this ... Glad to! I emerged this new ebuild from a local overlay. http-replicator emerged w/o any problems and is functioning perfectly. Now we need to wait for Adobe to upgrade flash.
This package is also without any maintainer, proxy or otherwise. If someone like far.jmp@gmail.com is not willing to be proxy maintainer, then proxy-maint@g.o ought be removed form CC
Sure, I am willing to that.
FYI - the orginal problem still remains Proxy request sent, awaiting response... 200 OK Length: 7279100 (6.9M) Saving to: ‘/usr/portage/distfiles/adobe-flash-11.2.202.616.x86_64.tar.gz’ /usr/portage/distfi 100%[=====================>] 6.94M --.-KB/s in 0.06s 2016-04-15 15:44:51 (112 MB/s) - ‘/usr/portage/distfiles/adobe-flash-11.2.202.616.x86_64.tar.gz’ saved [7279100/7279100] !!! Fetched file: adobe-flash-11.2.202.616.x86_64.tar.gz VERIFY FAILED! !!! Reason: Filesize does not match recorded size !!! Got: 7279100 !!! Expected: 7280774 Refetching... File renamed to '/usr/portage/distfiles/adobe-flash-11.2.202.616.x86_64.tar.gz._checksum_failure_.0kv82X' !!! Couldn't download 'adobe-flash-11.2.202.616.x86_64.tar.gz'. Aborting. * Fetch failed for 'www-plugins/adobe-flash-11.2.202.616', Log file: * '/var/tmp/portage/www-plugins/adobe-flash-11.2.202.616/temp/build.log' >>> Failed to emerge www-plugins/adobe-flash-11.2.202.616, Log file:
1. Did you remember to override FETCHCOMMAND on EVERY client machine's make.conf (or at least all of them that need to update flash), as mentioned in the elogged output, bullet 4 of comment 42, and other comments? It won't work without it. 2. Do you have the correct version installed and running on the server, considering multiple versions of the ebuild, local overlays, etc? You should be able to check the actually installed script: > $ grep x-unique-cache-name /usr/lib/python-exec/python2.7/http-replicator > uniqueCacheName = body.get('x-unique-cache-name') > $
Yes they rsyncing in gentoo has a recent history of deficits. Worth checking all those through.
the original bug should be fixed in -r6
