Created attachment 593304 [details] Output of emerge --info Upon running emerge -avuND --with-bdeps=y @world last night, portage tried to install app-admin/sudo-1.8.28p1 and failed the fetch, showing the following error message: /usr/portage/distfiles/sudo-1.8.28p1.tar.gz.__download__: No such file or directory Running ls -l /usr/portage/distfiles/sudo-1.8.28p1.tar.gz.__download__ reveals that this is a symlink pointing to 48/sudo-1.8.28p1.tar.gz, which doesn't exist. I tried unlinking it and emerging sudo, but the same issue occurred. I was only able to fix the issue by manually downloading the sudo tarball from mirror that portage was downloading from, and placing it in /usr/portage/distfiles/sudo-1.8.28p1.tar.gz.
Portage 2.3.76 (python 3.6.9-final-0, default/linux/amd64/17.1/desktop, gcc-8.3.0, glibc-2.29-r2, 4.19.72-gentoo x86_64) ================================================================= System uname: Linux-4.19.72-gentoo-x86_64-Intel-R-_Core-TM-_i7-6700K_CPU_@_4.00GHz-with-gentoo-2.6 KiB Mem: 16355472 total, 11827092 free KiB Swap: 2097148 total, 2097148 free Timestamp of repository gentoo: Sat, 19 Oct 2019 14:30:01 +0000 Head commit of repository gentoo: c75406019f886d3405ffa521793369d692f1a36e sh bash 4.4_p23-r1 ld GNU ld (Gentoo 2.32 p2) 2.32.0 app-shells/bash: 4.4_p23-r1::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.28.2-r1::gentoo dev-lang/python: 2.7.16::gentoo, 3.6.9::gentoo dev-util/cmake: 3.14.6::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.41.2::gentoo sys-apps/sandbox: 2.13::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.13.4-r2::gentoo, 1.16.1-r1::gentoo sys-devel/binutils: 2.32-r1::gentoo sys-devel/gcc: 8.3.0-r1::gentoo sys-devel/gcc-config: 2.0::gentoo sys-devel/libtool: 2.4.6-r3::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 4.19::gentoo (virtual/os-headers) sys-libs/glibc: 2.29-r2::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 sync-rsync-extra-opts: sync-rsync-verify-max-age: 24 wiltz location: /usr/local/portage masters: gentoo betagarden location: /var/lib/layman/betagarden masters: gentoo priority: 50 eroen location: /var/lib/layman/eroen masters: gentoo priority: 50 octave location: /var/lib/layman/octave masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.6/conf" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage/distfiles" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="rsync://gentoo.gossamerhost.com/gentoo-distfiles/ http://gentoo.gossamerhost.com http://gentoo.mirrors.tera-byte.com/ ftp://mirrors.tera-byte.com/pub/gentoo rsync://mirrors.tera-byte.com/gentoo ftp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam flac fortran gdbm gif git glamor gpm gtk iconv icu ipv6 jpeg latex lcms ldap libnotify libtirpc mad mng mp3 mp4 mpeg multilib ncurses nls nptl offensive ogg opengl openmp pam pango pcre pdf png policykit ppds readline sdl seccomp spell split-usr ssl startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vim-syntax vorbis wxwidgets x264 xattr xcb xft xml xv xvid zlib zsh-completion" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="MMXEXT" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="nlpsolver" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu radeonsi radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
I'm not sure, but this problem may be due to the following change: https://archives.gentoo.org/gentoo-dev/message/fb8b2dd311816dae972f71ef264beea5 I also found the following thread on the Gentoo forums: https://forums.gentoo.org/viewtopic.php?p=8380908
Thanks Francesco -- it seems the OP in the thread you posted was having an identical issue. I'm curious if this is affecting everyone or if it's particular to his and my setup. It's still unclear to me if/how I should change anything to prevent something like this from happening again.
You should probably try upgrading portage to version 2.3.77.
See also the solution provided in the following post: https://forums.gentoo.org/viewtopic-p-8380946.html#8380946
Please attach the entire fetch log to this bug report.
Created attachment 593392 [details] Fetch log
(In reply to Harley Wiltzer from comment #7) > Created attachment 593392 [details] > Fetch log There's something wrong with your system. The following URLs are working fine: - ftp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/distfiles/sudo-1.8.28p1.tar.gz - http://gentoo.gossamerhost.com/distfiles/sudo-1.8.28p1.tar.gz - https://www.sudo.ws/sudo/dist/sudo-1.8.28p1.tar.gz I am also not sure if this is a valid fetch log, I would expect to see some connection attempts, DNS lookups and stuff like that. However, because you didn't show us complete `emerge --info` (we don't know your FETCH command), you are on your own.
(In reply to Thomas Deutschmann from comment #8) > (In reply to Harley Wiltzer from comment #7) > > Created attachment 593392 [details] > > Fetch log > > There's something wrong with your system. The following URLs are working > fine: > > - > ftp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/distfiles/sudo-1.8.28p1. > tar.gz > > - http://gentoo.gossamerhost.com/distfiles/sudo-1.8.28p1.tar.gz > > - https://www.sudo.ws/sudo/dist/sudo-1.8.28p1.tar.gz > > I am also not sure if this is a valid fetch log, I would expect to see some > connection attempts, DNS lookups and stuff like that. Those URLs work fine on my system as well, in fact I used one of them to download the tarball that I placed in /usr/portage/distfiles to resolve the issue. I don't think there's an issue downloading the tarballs. Not to mention, the OP in the thread that Francesco Turco posted was observing the exact same behavior [1]. > However, because you didn't show us complete `emerge --info` (we don't know > your FETCH command), you are on your own. I'm sorry about that, perhaps I'm doing something wrong. When I run `emerge --info | grep FETCH`, I don't get anything. What I posted is in fact the entire output of `emerge --info`, are there other parameters that I should be passing to it? I wasn't aware of the FETCHCOMMAND variable prior to your comment, but I don't think mine is set. The wiki says it defaults to wget, so maybe that's why it doesn't appear in `emerge --info`? [1] https://forums.gentoo.org/viewtopic.php?p=8380908
Your complete portage configuration is "interesting". GENTOO_MIRRORS is for distfiles mirror. Shouldn't contain anything with rsync://... Maybe backup /etc/portage and start over, i.e. delete directory and recreate. Don't forget to to set profile after that.
(In reply to Thomas Deutschmann from comment #10) > Your complete portage configuration is "interesting". GENTOO_MIRRORS is for > distfiles mirror. Shouldn't contain anything with rsync://... Interesting. I know how that got there -- when I originally installed Gentoo, I selected my mirrors via `mirrorselect -i -o`. Running this command again, I see that a bunch of options use rsync. Surely when I installed this I didn't realize that I shouldn't select the rsync ones, so that's why they're there. If we shouldn't be adding those mirrors, why does mirrorselect even present them? Having said that, this has never been an issue for me before, and when I first encountered this error, portage successfully built and installed a bunch of other packages during `emerge -avuND --with-bdeps=y @world`. Not to mention the fact that I've definitely merged app-admin/sudo many times before without any problems, in fact I think there was an update just a couple of weeks ago that caused no trouble. Do you really think the rsync mirrors are at fault here? > Maybe backup /etc/portage and start over, i.e. delete directory and > recreate. Don't forget to to set profile after that. Given what I wrote above in this comment, do you still think this is necessary? Can I not just remove the rsync mirrors?
(In reply to Harley Wiltzer from comment #9) > I'm sorry about that, perhaps I'm doing something wrong. When I run `emerge > --info | grep FETCH`, I don't get anything. What I posted is in fact the > entire output of `emerge --info`, are there other parameters that I should > be passing to it? I wasn't aware of the FETCHCOMMAND variable prior to your > comment, but I don't think mine is set. The wiki says it defaults to wget, > so maybe that's why it doesn't appear in `emerge --info`? You may try with: emerge --info --verbose | grep FETCHCOMMAND Or: portageq envvar FETCHCOMMAND
Well I solved this by setting GENTOO_MIRRORS="" sudo is also not an only package that has this problem. It is not a problem with sudo but rather with new mirrors layout and emerge. Problem is simple. Command run to fetch file is targeted to directory that is not created. The directory is simply missing and has to be created before fetch. Disabling mirrors is temporally fix until emerge/portage is fixed to correctly fetch distfiles to archive. Just so you know I have no modifications to fetching distrfiles and I was able to reproduce it with configured mirrors with stage3 as well so it is not some configuration exclusive problem. Just for completeness: $ emerge --info --verbose | grep FETCHCOMMAND FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"" FETCHCOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}"" FETCHCOMMAND_SFTP="bash -c "x=\${2#sftp://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port= ; eval \"declare -a ssh_opts=(\${3})\" ; exec sftp \${port:+-P \${port}} \"\${ssh_opts[@]}\" \"\${host}:/\${x#*/}\" \"\$1\"" sftp "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" FETCHCOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port= ; exec rsync --rsh=\"ssh \${port:+-p\${port}} \${3}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" Just to visualize problem here is example with expanded link (foo instead of real one) $ sudo wget -t 3 -T 60 --passive-ftp -O /usr/portage/distfiles/foo/sudo-1.8.28p1.tar.gz http://gentoo.gossamerhost.com/distfiles/sudo-1.8.28p1.tar.gz /usr/portage/distfiles/foo/sudo-1.8.28p1.tar.gz: No such file or directory
(In reply to Karel Kočí from comment #13) > Well I solved this by setting GENTOO_MIRRORS="" > > sudo is also not an only package that has this problem. It is not a problem > with sudo but rather with new mirrors layout and emerge. > > Problem is simple. Command run to fetch file is targeted to directory that > is not created. The directory is simply missing and has to be created before > fetch. > Please don't spread FUD. If you read the attached logs, you'd notice that the directory wget is targeting is correct. Your example is wholly made up.
The problem's in FETCHCOMMAND_RSYNC. It contains '-a' which enables copying symlinks verbatim. That is, any symlinks on the mirrors are copied *as symlinks*, without actually copying the target file. Therefore, a dangling symlink ends up in DISTDIR. The rest is most likely poor logic for checking file existence. stat() is used rather than lstat(), which fails for dangling symlinks and makes stuff believe the distfile does not exist. Then it is trying to write into it, and since the symlink target is in a subdirectory, you get ENOENT. I'm going to submit a patch fixing default FETCHCOMMAND_RSYNC.
PATCH: https://archives.gentoo.org/gentoo-portage-dev/message/44e66598376f77219aa60fa00fe8c6ac
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=03c54e340073620f489ca85bca94267a198174fe commit 03c54e340073620f489ca85bca94267a198174fe Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2019-10-21 14:59:43 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2019-10-21 18:07:40 +0000 make.globals: Change FETCHCOMMAND_RSYNC to --copy-links Change FETCHCOMMAND_RSYNC to use '-Lt' over '-a'. Notably, this replaces --links with --copy-links option, i.e. makes rsync copy underlying files when symlinks are met. This is important since we do not transfer symlink targets, therefore '-l' ends up creating dangling symlinks. This also removes most of the other options that are irrelevant or even undesirable to distfile fetching, that is: - '-r' since we always fetch a single file, so recursive operation is unnecessary - '-p', '-o', '-g' since we want to apply our permissions and ownership for distfiles rather than copying the one from mirrors, - '-D' since we do not expect any devices or specials in distfiles. Copying timestamps is preserved in case it's helpful in determining whether files need to be refetched. Bug: https://bugs.gentoo.org/698046 Reviewed-by: Zac Medico <zmedico@gentoo.org> Signed-off-by: Michał Górny <mgorny@gentoo.org> cnf/make.globals | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ecbfeae408f1bdcfa4241a3f4001d57cf30c3405 commit ecbfeae408f1bdcfa4241a3f4001d57cf30c3405 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-21 19:12:06 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-21 19:16:01 +0000 sys-apps/portage: Bump to version 2.3.78 #697566 fetch: Use FETCHCOMMAND to fetch mirror layout.conf #697890 emirrordist: Fix DeletionTask layout assumptions #697906 emirrordist: Delete potential symlinks for all layouts #698046 make.globals: Change FETCHCOMMAND_RSYNC to --copy-links Bug: https://bugs.gentoo.org/697734 Bug: https://bugs.gentoo.org/697566 Bug: https://bugs.gentoo.org/697890 Bug: https://bugs.gentoo.org/697906 Bug: https://bugs.gentoo.org/698046 Package-Manager: Portage-2.3.78, Repoman-2.3.17 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.78.ebuild | 261 +++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=0299aedef74e47c0a68acf7905d8714c9578f125 commit 0299aedef74e47c0a68acf7905d8714c9578f125 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-21 19:40:46 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-21 19:42:19 +0000 GetConfigTestCase: update FETCHCOMMAND_RSYNC Bug: https://bugs.gentoo.org/698046 Fixes: 03c54e340073 ("make.globals: Change FETCHCOMMAND_RSYNC to --copy-links") Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/util/test_getconfig.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b74125f7b34055f38bb546439d35e8b5c84dc1b1 commit b74125f7b34055f38bb546439d35e8b5c84dc1b1 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-21 19:48:16 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-21 19:49:39 +0000 sys-apps/portage: fixup 2.3.78 GetConfigTestCase Bug: https://bugs.gentoo.org/698046 Package-Manager: Portage-2.3.78, Repoman-2.3.17 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/portage-2.3.78.ebuild | 3 +++ 1 file changed, 3 insertions(+)
Created attachment 593584 [details] emerge-fetch log from sys-libs/timezone-data-2019c I encountered a similar issue when trying to merge sys-libs/timezone-data-2019c today. Unfortunately, manually downloading the tarball and placing it in /usr/portage/distfiles didn't work this time. I tried removing the rsync mirrors from GENTOO_MIRRORS and merging again, but the problem persists as shown in this emerge-fetch.log. Unless I'm misunderstanding something (which of course is very possible), I don't believe the issue is entirely due to the -a flag in FETCHCOMMAND_RSYNC.
(In reply to Harley Wiltzer from comment #21) > Created attachment 593584 [details] > emerge-fetch log from sys-libs/timezone-data-2019c > > I encountered a similar issue when trying to merge > sys-libs/timezone-data-2019c today. Unfortunately, manually downloading the > tarball and placing it in /usr/portage/distfiles didn't work this time. I > tried removing the rsync mirrors from GENTOO_MIRRORS and merging again, but > the problem persists as shown in this emerge-fetch.log. Unless I'm > misunderstanding something (which of course is very possible), I don't > believe the issue is entirely due to the -a flag in FETCHCOMMAND_RSYNC. Sorry, I made a slight mistake here -- I actually could again fix the issue by manually placing the tarball in /usr/portage/distfiles (I made a mistake previously). However, I was still experiencing the same failure without the rsync mirrors, which can be seen in the emerge-fetch.log (note that it's using http, not rsync, as far as I'm concerned). So I still believe FETCHCOMMAND_RSYNC isn't the sole cause of the error, with my current understanding of the issue.
(In reply to Harley Wiltzer from comment #21) > Created attachment 593584 [details] > emerge-fetch log from sys-libs/timezone-data-2019c > > I encountered a similar issue when trying to merge > sys-libs/timezone-data-2019c today. Unfortunately, manually downloading the > tarball and placing it in /usr/portage/distfiles didn't work this time. I > tried removing the rsync mirrors from GENTOO_MIRRORS and merging again, but > the problem persists as shown in this emerge-fetch.log. Unless I'm > misunderstanding something (which of course is very possible), I don't > believe the issue is entirely due to the -a flag in FETCHCOMMAND_RSYNC. It looks like you probably have a dangling symlink named /usr/portage/distfiles/tzcode2019c.tar.gz.__download__ that you need to remove.
(In reply to Zac Medico from comment #23) > It looks like you probably have a dangling symlink named > /usr/portage/distfiles/tzcode2019c.tar.gz.__download__ that you need to > remove. Yes, you are right. Just cleaned my distfiles and tested again without the rsync mirrors and everything went smoothly. Sorry about that.
*** Bug 698240 has been marked as a duplicate of this bug. ***
Noting comment https://bugs.gentoo.org/698046#c15 A work-around for successful rsync fetches (for myself at least) is to use: PORTAGE_RSYNC_OPTS="--recursive -L --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" emerge ... So... Where should PORTAGE_RSYNC_OPTS be updated/modified? Or should something else be done for a robust fix? Note, for my very old long running system, I have from long ago: # grep MIRRORS /etc/portage/make.conf GENTOO_MIRRORS="rsync://mirror.bytemark.co.uk/gentoo/ rsync://rsync.mirrorservice.org/distfiles.gentoo.org/ rsync://ftp.snt.utwente.nl/gentoo rsync://mirror.leaseweb.com/gentoo/ rsync://ftp.fau.de/gentoo" # emerge --version Portage 2.3.78 (python 3.5.7-final-0, !../../var/usr_portage/profiles/default/linux/amd64/17.0/desktop/plasma, gcc-9.2.0, glibc-2.29-r2, 5.3.6-gentoo-r1_m07d_54 x86_64) # emerge --info | grep -i rsync sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage sync-rsync-verify-max-age: 24 sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 sync-rsync-extra-opts: FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" GENTOO_MIRRORS="rsync://mirror.bytemark.co.uk/gentoo/ rsync://rsync.mirrorservice.org/distfiles.gentoo.org/ rsync://ftp.snt.utwente.nl/gentoo rsync://mirror.leaseweb.com/gentoo/ rsync://ftp.fau.de/gentoo" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS I hope that is of some help, Regards, Martin
(In reply to Martin from comment #26) > Noting comment https://bugs.gentoo.org/698046#c15 > > > A work-around for successful rsync fetches (for myself at least) is to use: > > PORTAGE_RSYNC_OPTS="--recursive -L --safe-links --perms --times > --omit-dir-times --compress --force --whole-file --delete --stats > --human-readable --timeout=180 --exclude=/distfiles --exclude=/local > --exclude=/packages --exclude=/.git" emerge ... > > > So... Where should PORTAGE_RSYNC_OPTS be updated/modified? Or should > something else be done for a robust fix? PORTAGE_RSYNC_OPTS is irrelevant because it applies to handling of rsync ebuild repositories (like rsync://rsync.gentoo.org/gentoo-portage) where symlink handling is not a problem. The problem is with FETCHCOMMAND* or RESUMECOMMAND* settings which are used to fetch distfiles and do not have this rsync option enabled: > -L, --copy-links transform symlink into referent file/dir
For my example, PORTAGE_RSYNC_OPTS has a singular effect to enable downloads to complete successfully. Before posting, I checked with running: # emerge -vNu sys-kernel/gentoo-sources The download failed with the spurious copied __download__ link for the tgz download file. Prefixing PORTAGE_RSYNC_OPTS to force the "-L" enabled a successful download and emerge. Or how do the PORTAGE_RSYNC_OPTS options propagate? (As tested this evening.) Regards, Martin
(In reply to Martin from comment #28) > For my example, PORTAGE_RSYNC_OPTS has a singular effect to enable downloads > to complete successfully. > > > Before posting, I checked with running: > > # emerge -vNu sys-kernel/gentoo-sources > > The download failed with the spurious copied __download__ link for the tgz > download file. > > Prefixing PORTAGE_RSYNC_OPTS to force the "-L" enabled a successful download > and emerge. > > Or how do the PORTAGE_RSYNC_OPTS options propagate? There is no relationship between PORTAGE_RSYNC_OPTS and FETCHCOMMAND unless you've created one yourself, perhaps by doing something like this in make.conf: FETCHCOMMAND="rsync ${PORTAGE_RSYNC_OPTS} ..." You should check the output of this command: > emerge --info -v | grep FETCHCOMMAND
OK... So testing just now, emerge now magically works with no need for prefixing "PORTAGE_RSYNC_OPTS". Running emerge for upgrading sys-apps/portage-2.3.78 to sys-apps/portage-2.3.78-r1 worked fine. Problem solved elsewhere? FYI, I have: # emerge --info -v | grep FETCHCOMMAND FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"" FETCHCOMMAND_RSYNC="rsync -LtvP "${URI}" "${DISTDIR}/${FILE}"" FETCHCOMMAND_SFTP="bash -c "x=\${2#sftp://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port= ; eval \"declare -a ssh_opts=(\${3})\" ; exec sftp \${port:+-P \${port}} \"\${ssh_opts[@]}\" \"\${host}:/\ ${x#*/}\" \"\$1\"" sftp "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" FETCHCOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port= ; exec rsync --rsh=\"ssh \${port:+-p\${port}} \${3}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR} /${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" # emerge --info -v | grep -i rsync sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage sync-rsync-extra-opts: sync-rsync-verify-max-age: 24 sync-rsync-verify-metamanifest: yes sync-rsync-verify-jobs: 1 FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-own ed sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FETCHCOMMAND_RSYNC="rsync -LtvP "${URI}" "${DISTDIR}/${FILE}"" FETCHCOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port= ; exec rsync --rsh=\"ssh \${port:+-p\${port}} \${3}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR} /${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" GENTOO_MIRRORS="rsync://mirror.bytemark.co.uk/gentoo/ rsync://rsync.mirrorservice.org/distfiles.gentoo.org/ rsync://ftp.snt.utwente.nl/gentoo rsync://mirror.leaseweb.com/gentoo/ rsync://ftp.fau.de/gentoo" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_RSYNC_RETRIES="-1" PWD="/home/martin/emerge_rsync_debug" RESUMECOMMAND_RSYNC="rsync -LtvP "${URI}" "${DISTDIR}/${FILE}"" RESUMECOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port= ; exec rsync --rsh=\"ssh \${port:+-p\${port}} \${3}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR }/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" In wonderment: Problem solved elsewhere? Thanks, Martin
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=1ca5b822133171b131cef3dc15dc43583893ad6b commit 1ca5b822133171b131cef3dc15dc43583893ad6b Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-29 00:56:47 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-29 01:07:45 +0000 fetch: remove symlink created by FETCHCOMMAND_RSYNC (bug 698046) This avoids confusing "No such file or directory" errors as demonstrated by the following test case: $ ln -s /foo/bar /tmp/sudo-1.8.29rc1.tar.gz $ wget http://distfiles.gentoo.org/distfiles/sudo-1.8.29rc1.tar.gz -O /tmp/sudo-1.8.29rc1.tar.gz /tmp/sudo-1.8.29rc1.tar.gz: No such file or directory Bug: https://bugs.gentoo.org/698046 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/package/ebuild/fetch.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f7b7f15761aaa20ad0b4eeddbb7b3637b2d356f0 commit f7b7f15761aaa20ad0b4eeddbb7b3637b2d356f0 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-29 01:31:41 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-29 01:42:55 +0000 sys-apps/portage: 2.3.76-r1 stable revbump for bug 698046 This avoids confusing "No such file or directory" errors as demonstrated by the following test case: $ ln -s /foo/bar /tmp/sudo-1.8.29rc1.tar.gz $ wget http://distfiles.gentoo.org/distfiles/sudo-1.8.29rc1.tar.gz -O /tmp/sudo-1.8.29rc1.tar.gz /tmp/sudo-1.8.29rc1.tar.gz: No such file or directory Bug: https://bugs.gentoo.org/691278 Bug: https://bugs.gentoo.org/698046 Package-Manager: Portage-2.3.78, Repoman-2.3.17 Signed-off-by: Zac Medico <zmedico@gentoo.org> .../portage/{portage-2.3.76.ebuild => portage-2.3.76-r1.ebuild} | 7 +++++++ 1 file changed, 7 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fb7c020a6779d020cc781eeb159ab9e60791c9a4 commit fb7c020a6779d020cc781eeb159ab9e60791c9a4 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-10-29 01:57:24 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-10-29 02:01:08 +0000 sys-apps/portage: 2.3.78-r2 revbump for bug 698046 This avoids confusing "No such file or directory" errors as demonstrated by the following test case: $ ln -s /foo/bar /tmp/sudo-1.8.29rc1.tar.gz $ wget http://distfiles.gentoo.org/distfiles/sudo-1.8.29rc1.tar.gz -O /tmp/sudo-1.8.29rc1.tar.gz /tmp/sudo-1.8.29rc1.tar.gz: No such file or directory Bug: https://bugs.gentoo.org/697734 Bug: https://bugs.gentoo.org/698046 Package-Manager: Portage-2.3.78, Repoman-2.3.17 Signed-off-by: Zac Medico <zmedico@gentoo.org> .../portage/{portage-2.3.78-r1.ebuild => portage-2.3.78-r2.ebuild} | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
