Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 175612 - sys-apps/portage: partial downloads trigger generation of incorrect digests in manifests
Summary: sys-apps/portage: partial downloads trigger generation of incorrect digests i...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
: 175609 477942 485458 (view as bug list)
Depends on:
Blocks: 377365 683434
  Show dependency tree
 
Reported: 2007-04-22 19:38 UTC by Pacho Ramos
Modified: 2019-09-13 07:06 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Pacho Ramos gentoo-dev 2007-04-22 19:38:27 UTC
For example, I run:
ebuild example-1.0.ebuild digest

Download starts, if network goes down, a wrong digest is done and later "emerge example" fails because it doesn't try to re-download source file again. For being able to redigest it, I have to remove digest-example-1.0, Metadata and downloaded part sources from distfiles dir.

Reproducible: Always

Steps to Reproduce:
1. ebuild example-1.0.ebuild digest
2. Switch down network
3. 

Actual Results:  
A wrong metadata and digest files are generated

Expected Results:  
When network is down and full source tarball couldn't be downloaded, ebuild should stop making the digest and doesn't generate any file

My emerge --info:
Portage 2.1.2.2 (default-linux/amd64/2006.1, gcc-4.1.1, glibc-2.5-r0, 2.6.20-ck1 x86_64)
=================================================================
System uname: 2.6.20-ck1 x86_64 AMD Athlon(tm) 64 Processor 3200+
Gentoo Base System release 1.12.9
Timestamp of tree: Sun, 22 Apr 2007 11:20:01 +0000
ccache version 2.4 [enabled]
dev-java/java-config: 1.3.7, 2.0.31-r5
dev-lang/python:     2.4.3-r4
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.4-r7
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.61
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.16.1-r3
sys-devel/gcc-config: 1.3.15-r1
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.19.2-r2
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=k8 -O2 -pipe -msse3"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/java-config/vms/ /etc/revdep-rebuild /etc/splash /etc/terminfo /etc/texmf/web2c"
CXXFLAGS="-march=k8 -O2 -pipe -msse3"
DISTDIR="/usr/distfiles"
FEATURES="autoaddcvs ccache collision-protect cvs distlocks metadata-transfer multilib-strict parallel-fetch sandbox sfperms strict"
GENTOO_MIRRORS="http://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/"
LANG="es_ES.UTF-8"
LC_ALL="es_ES.UTF-8"
LINGUAS="es en_US"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from=/etc/portage/rsync_excludes"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/portage/local/layman/sunrise /usr/portage/local/layman/musicbrainz /usr/local/portage"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
USE="X a52 aac acpi alsa amd64 arts asf audiofile bash-completion beagle bitmap-fonts bzip2 bzlib cairo caps cdb cdda cddb cdparanoia cdr cli cpdflib cpudetection cracklib cross crypt cups curl dbus dga divx4linux dlloader dri dts dvb dvd dvdr dvdread encode erandom escreen esd evo evolution exif fam fbcon flac foomaticdb fortran ftp galago gb gcj ggi gif gimp gimpprint glitz glut glx gnome gnome-print gphoto2 gpm gstreamer gtk gtk2 gtkhtml hal iconv idn imagemagick imlib isdnlog ithreads jabber java jpeg kde kdeenablefinal kdehiddenvisibility latex lcms libg++ libnotify logrotate mad madwifi midi mikmod mime mng mono motif mp3 mpeg mpi mplayer musepack musicbrainz nas nautilus ncurses nethack network nls nocardbus nptl nptlonly nvidia ogg oggvorbis openal opengl pam pcre pdf perl pic plotutils png posix ppds pppd python qt3 qt4 quicktime readline reflection rtc ruby scanner sdl seamonkey session slang slp sockets spell spl sse3 ssl startup-notification svg sysvipc tcltk tcpd tetex theora threads tiff timidity tk totem trayicon truetype truetype-fonts type1-fonts unicode usb v4l v4l2 vcd videos vim vorbis wma wmf xcomposite xine xml xml2 xorg xpm xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" DVB_CARDS="usb-dib0700" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="es en_US" USERLAND="GNU" VIDEO_CARDS="nvidia nv vesa fbdev vga"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS
Comment 1 Pacho Ramos gentoo-dev 2007-04-22 19:39:44 UTC
I want say "Manifest"
metadata -> manifest
Comment 2 Samuli Suominen gentoo-dev 2007-04-22 19:41:40 UTC
*** Bug 175609 has been marked as a duplicate of this bug. ***
Comment 3 Pacho Ramos gentoo-dev 2007-04-22 19:44:52 UTC
(In reply to comment #2)
> *** Bug 175609 has been marked as a duplicate of this bug. ***
> 

Sorry, my network is awful :-S
Comment 4 Alec Warner archtester Gentoo Infrastructure gentoo-dev Security 2007-04-22 23:26:23 UTC
Maybe fetchcommand should make a temporary file until the fetch completes and then move it into place when it's done?

That way if your download was incomplete the digest would complain about missing distfiles.

-Alec
Comment 5 Pacho Ramos gentoo-dev 2007-04-23 07:00:50 UTC
It is great idea :-D

Thanks
Comment 6 Zac Medico gentoo-dev 2013-02-15 21:37:39 UTC
This is much more feasible nowadays since FETCHCOMMAND has been updated to -O ${DISTDIR}/${FILE} instead of -P ${DISTDIR}:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=466cfb920412a47c1bf8e47628dd59f90a7220b2
Comment 7 Zac Medico gentoo-dev 2013-07-23 20:43:08 UTC
*** Bug 477942 has been marked as a duplicate of this bug. ***
Comment 8 Zac Medico gentoo-dev 2019-04-22 06:07:42 UTC
*** Bug 485458 has been marked as a duplicate of this bug. ***
Comment 10 Larry the Git Cow gentoo-dev 2019-04-30 02:47:43 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=ebbde237d33e783c562cc6c70987969ac7228b96

commit ebbde237d33e783c562cc6c70987969ac7228b96
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-04-27 21:59:57 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-04-30 02:28:21 +0000

    fetch: atomic downloads (bug 175612)
    
    Direct FETCHCOMMAND/RESUMECOMMAND output to a temporary file with
    a constant .__download__ suffix, and atomically rename the file
    to remove the suffix only after the download has completed
    successfully (includes digest verification when applicable).
    Also add unit tests to cover most fetch cases.
    
    Bug: https://bugs.gentoo.org/175612
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/BinpkgVerifier.py          |   4 +-
 lib/portage/package/ebuild/fetch.py    | 105 ++++++++++-----
 lib/portage/tests/ebuild/test_fetch.py | 230 +++++++++++++++++++++++++++++++++
 3 files changed, 303 insertions(+), 36 deletions(-)
Comment 11 Larry the Git Cow gentoo-dev 2019-04-30 02:57:07 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=227c1e8bcb5a6fc377be46dbac33ef9a3dcaa27d

commit 227c1e8bcb5a6fc377be46dbac33ef9a3dcaa27d
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-04-30 02:53:01 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-04-30 02:55:00 +0000

    sys-apps/portage: version bump to 2.3.66
    
     #175612 download distfiles to temp file and rename atomically
     #651678 enable FEATURES=strict-keepdir behavior for new EAPIs
    
    Bug: https://bugs.gentoo.org/683434
    Bug: https://bugs.gentoo.org/175612
    Bug: https://bugs.gentoo.org/651678
    Package-Manager: Portage-2.3.66, Repoman-2.3.12
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.66.ebuild | 259 +++++++++++++++++++++++++++++++++
 2 files changed, 260 insertions(+)
Comment 12 Matt Whitlock 2019-09-12 18:26:18 UTC
@Zac Medico:
How is this supposed to work when DISTDIR is a read-only network mount and FETCHCOMMAND is configured to fetch files onto the remote system? Such a setup was working for me until this change. Now what I see is that the file downloads successfully onto the DISTDIR host, but Portage then fails to rename it.

Read-only DISTDIR is supposedly a supported configuration, judging by the make.conf(5) man page, which reads:

skiprocheck
       Skip write access checks on DISTDIR when  fetching  files.  This  is
       useful when FETCHCOMMAND and RESUMECOMMAND are used to forward fetch
       requests to a server that exposes DISTDIR as a read-only NFS  share.
       A  read-only  DISTDIR is not compatible with the distlocks, so it is
       recommended to also add "-distlocks" to FEATURES in order  to  avoid
       warning messages that are triggered by this incompatibility.

This use case no longer seems possible since Portage tries to rename downloaded files itself rather than exposing any way to send those rename requests to the DISTDIR host.
Comment 13 Zac Medico gentoo-dev 2019-09-12 20:03:56 UTC
(In reply to Matt Whitlock from comment #12)
> @Zac Medico:
> How is this supposed to work when DISTDIR is a read-only network mount and
> FETCHCOMMAND is configured to fetch files onto the remote system? Such a
> setup was working for me until this change. Now what I see is that the file
> downloads successfully onto the DISTDIR host, but Portage then fails to
> rename it.
> 
> Read-only DISTDIR is supposedly a supported configuration, judging by the
> make.conf(5) man page, which reads:
> 
> skiprocheck
>        Skip write access checks on DISTDIR when  fetching  files.  This  is
>        useful when FETCHCOMMAND and RESUMECOMMAND are used to forward fetch
>        requests to a server that exposes DISTDIR as a read-only NFS  share.
>        A  read-only  DISTDIR is not compatible with the distlocks, so it is
>        recommended to also add "-distlocks" to FEATURES in order  to  avoid
>        warning messages that are triggered by this incompatibility.
> 
> This use case no longer seems possible since Portage tries to rename
> downloaded files itself rather than exposing any way to send those rename
> requests to the DISTDIR host.

Thanks for the report, I've re-opened bug 220533.