I would like to run the script on a cluster but compute nodes have no direct network access, so thet the "wget htp://..." calls fail. Could one run the script with some argument to prefetch the file on some computer and then start bootstrap-prefix.sh again and let it do the actual compilation? Please document it at https://wiki.gentoo.org/wiki/Project:Prefix/Bootstrap
I also tried now: export GENTOO_MIRRORS="http://gentoo.mirror.dkm.cz/pub/gentoo/" ./bootstrap-prefix.sh at the very begginning I think there were some errors on the screen, but later bison sources were etched, compiled, and so on, from the GENTOO_MIRRORS. But, then, at some step something needs to be fetched from elsewhere: * python-2.7.3 bootstrapped * Fetching prefix-overlay-20160614.tar.bz2 --2016-06-17 12:09:29-- http://gentoo.mirror.dkm.cz/pub/gentoo//distfiles/prefix-overlay-20160614.tar.bz2 Connecting to 127.0.0.1:1234... connected. Proxy request sent, awaiting response... 301 Moved Permanently Location: http://mirror.dkm.cz/gentoo/distfiles/prefix-overlay-20160614.tar.bz2 [following] --2016-06-17 12:09:29-- http://mirror.dkm.cz/gentoo/distfiles/prefix-overlay-20160614.tar.bz2 Connecting to 127.0.0.1:1234... connected. Proxy request sent, awaiting response... 404 Not Found 2016-06-17 12:09:29 ERROR 404: Not Found. --2016-06-17 12:09:29-- http://dev.gentoo.org/~grobian/distfiles/prefix-overlay-20160614.tar.bz2 Connecting to 127.0.0.1:1234... connected. Proxy request sent, awaiting response... 404 Not Found 2016-06-17 12:09:29 ERROR 404: Not Found. !!! downloading http://dev.gentoo.org/~grobian/distfiles/prefix-overlay-20160614.tar.bz2 failed!
hmmm, I always use GENTOO_MIRRORS for this, which is just a local box that runs http server to serve a dir of distfiles. It should also pick the snapshot from the mirrors.
the only problem here, I didn't upload the latest snapshot to the mirrors, fixed that now.
OK, I will re-fetch the main script, with the current one I also had to fetch these: http://dev.gentoo.org/~vapier/dist/binutils-2.24-patches-1.3.tar.xz Not sure if that is to be fixed in your update script ..., well, will see.
the script has nothing to do with it, it's the mirrors
So I am puzzled. I re-fetched http://rsync.prefix.bitzolder.nl/scripts/bootstrap-prefix.sh but it is same which I fetched two hours ago. BTW, couldn't the script be hosted at some gentoo.org site? I do use GENTOO_MIRRORS="http://gentoo.mirror.dkm.cz/pub/gentoo/" as I showed and the script does respect the GENTOO_MIRRORS variable, but in the two cases it was fetching from elsewhere. OK, I infer that happened because the mirrors did not have the file, fine, but I will have to wait for the mirror to become in sync. I can use only one http site, as I have to work around the firewall here. ;-) What I was more asking about here was, if the bootstrap-prefix.sh script could be run in fetch-only mode, which would skip all the compile steps. Or, if you could give a list of http urls I could pre-fetch manually elsewhere? Copying my /usr/portage/distfiles from my desktop is too large, and with too many alternatives versions accumulated over time. Thank you.
Maybe of some help: The bootstrap-prefix.sh listens to OFFLINE_MODE=yes environment variable, where it does write something like "I need <FILE> from <URI> in <DISTDIR>" to stdout and then waits for keypress upon a missing file, instead of trying to download. Do this once manually, and then you have a list of files for the next run... Or use net-proxy/http-replicator [1] as proxy server on a connected machine - its cached files after bootstrap are the files you need to download... [1] https://github.com/gertjanvanzwieten/replicator
I've put binutils-2.24-patches-1.3.tar.xz on the mirrors (whitelisted), it's because our tree is behind on that one.
>>> Downloading 'http://gentoo.mirror.dkm.cz/pub/gentoo/distfiles/file-5.24.tar.gz' --2016-06-17 20:28:10-- http://gentoo.mirror.dkm.cz/pub/gentoo/distfiles/file-5.24.tar.gz Resolving gentoo.mirror.dkm.cz... 86.49.49.49, 2a02:8300:8000:3::49 Connecting to gentoo.mirror.dkm.cz|86.49.49.49|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: http://mirror.dkm.cz/gentoo/distfiles/file-5.24.tar.gz [following] --2016-06-17 20:28:10-- http://mirror.dkm.cz/gentoo/distfiles/file-5.24.tar.gz Resolving mirror.dkm.cz... 86.49.49.49, 2a02:8300:8000:3::49 Reusing existing connection to gentoo.mirror.dkm.cz:80. HTTP request sent, awaiting response... 404 Not Found 2016-06-17 20:28:10 ERROR 404: Not Found. >>> Downloading 'ftp://ftp.gw.com/mirrors/pub/unix/file/file-5.24.tar.gz' --2016-06-17 20:28:10-- ftp://ftp.gw.com/mirrors/pub/unix/file/file-5.24.tar.gz => `/scratch/mmokrejs/gentoo/usr/portage/distfiles/file-5.24.tar.gz' Resolving ftp.gw.com... failed: Name or service not known. wget: unable to resolve host address `ftp.gw.com' >>> Downloading 'ftp://ftp.astron.com/pub/file/file-5.24.tar.gz' --2016-06-17 20:28:10-- ftp://ftp.astron.com/pub/file/file-5.24.tar.gz => `/scratch/mmokrejs/gentoo/usr/portage/distfiles/file-5.24.tar.gz' Resolving ftp.astron.com... 38.117.134.18 Connecting to ftp.astron.com|38.117.134.18|:21... failed: Connection refused. !!! Couldn't download 'file-5.24.tar.gz'. Aborting. * Fetch failed for 'sys-apps/file-5.24', Log file: * '/scratch/mmokrejs/gentoo/tmp/var/tmp/portage/sys-apps/file-5.24/temp/build.log'
Doh, rsync is disabled here. >>> Syncing repository 'gentoo_prefix' into '/scratch/mmokrejs/gentoo/usr/portage'... rsync: failed to connect to 83.162.207.175 (83.162.207.175): Connection refused (111) rsync error: error in socket IO (code 10) at clientserver.c(125) [Receiver=3.1.2] >>> Starting rsync with rsync://83.162.207.175/gentoo-portage-prefix... >>> Checking server timestamp ... >>> Retrying... >>> Starting retry 1 of 1 with rsync://[2001:888:1022:0:20e:cff:fec1:df5]/gentoo-portage-prefix rsync: failed to connect to 2001:888:1022:0:20e:cff:fec1:df5 (2001:888:1022:0:20e:cff:fec1:df5): Network is unreachable (101) rsync error: error in socket IO (code 10) at clientserver.c(125) [Receiver=3.1.2] !!! Exhausted addresses for rsync.prefix.bitzolder.nl q: Updating ebuild cache in /scratch/mmokrejs/gentoo/usr/portage ... Maybe this popped up now because I fiddled with /scratch/mmokrejs/gentoo/etc/portage/repos.conf/gentoo.conf: [DEFAULT] main-repo = gentoo [gentoo] location = /usr/portage sync-type = rsync sync-uri = rsync://rsync.gentoo.org/gentoo-portage auto-sync = yes Seems this has precedence over the GENTOO_MIRRORS env variable from make.conf. It is confusing what the repos.conf/gentoo.conf is about but I suspect it is the reason why I get: * prefix-portage successfully bootstrapped * stage1 successfully finished !!! Section 'gentoo' in repos.conf has location attribute set to nonexistent directory: '/usr/portage' !!! Repository name or alias 'gentoo', defined for repository 'gentoo_prefix', overrides existing alias or repository. !!! Invalid Repository Location (not a dir): '/usr/portage' * Bootstrapping Gentoo prefixed portage installation using * host: x86_64-pc-linux-gnu * prefix: /scratch/mmokrejs/gentoo * ready to bootstrap stage3_log In another go, I get: >>> Starting retry 1 of 1 with rsync://[2001:888:1022:0:20e:cff:fec1:df5]/gentoo-portage-prefix rsync: failed to connect to 2001:888:1022:0:20e:cff:fec1:df5 (2001:888:1022:0:20e:cff:fec1:df5): Network is unreachable (101) rsync error: error in socket IO (code 10) at clientserver.c(125) [Receiver=3.1.2] !!! Exhausted addresses for rsync.prefix.bitzolder.nl q: Updating ebuild cache in /scratch/mmokrejs/gentoo/usr/portage ... q: Finished 40474 entries in 5.437950 seconds >>> Syncing repository 'gentoo' into '/usr/portage'... Traceback (most recent call last): File "/scratch/mmokrejs/gentoo/usr/lib/python2.7/site-packages/portage/util/_async/AsyncFunction.py", line 39, in _run result = self.target(*(self.args or []), **(self.kwargs or {})) File "/scratch/mmokrejs/gentoo/usr/lib/python2.7/site-packages/portage/sync/controller.py", line 140, in sync rval = self.pre_sync(repo) File "/scratch/mmokrejs/gentoo/usr/lib/python2.7/site-packages/portage/sync/controller.py", line 293, in pre_sync portage.util.ensure_dirs(repo.location, **perms) File "/scratch/mmokrejs/gentoo/usr/lib/python2.7/site-packages/portage/util/__init__.py", line 1435, in ensure_dirs raise PermissionDenied(func_call) PermissionDenied: makedirs('/usr/portage') !!! Section 'gentoo' in repos.conf has location attribute set to nonexistent directory: '/usr/portage' !!! Repository name or alias 'gentoo', defined for repository 'gentoo_prefix', overrides existing alias or repository. !!! Invalid Repository Location (not a dir): '/usr/portage' !!! Section 'gentoo' in repos.conf has location attribute set to nonexistent directory: '/usr/portage' !!! Repository name or alias 'gentoo', defined for repository 'gentoo_prefix', overrides existing alias or repository. !!! Invalid Repository Location (not a dir): '/usr/portage' WARNING: One or more repositories have missing repo_name entries: /usr/portage/profiles/repo_name NOTE: Each repo_name entry should be a plain text file containing a unique name for the repository on the first line. !!! Problem with sandbox binary. Disabling... Calculating dependencies ... done!
It tries to sync its tree, if raync fails, it should fall back to emerge-webrsync, which might indeed not use the mirror. This shouldn't be fatal, and I thought offline-mode actually skips this step.
Well, I did NOT try yet the offline mode as it seemed I will still need to parse the output and probably do more. So, I run it on a host where http is allowed but not ftp nor rsync.
One option: you can setup a rsync proxy with squid (open 873). http://www.sai.msu.su/~er/rsync_proxy.html Then export RSYNC_PROXY=<squid host>:<squid port> before running bootstrap.
This could also be made dependent on bug #358923
