Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 451024 - running out of space during merge leaves unrecorded files in files system
Summary: running out of space during merge leaves unrecorded files in files system
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 345645 (view as bug list)
Depends on:
Blocks: 23851
  Show dependency tree
 
Reported: 2013-01-09 10:07 UTC by Justin Lecher (RETIRED)
Modified: 2016-02-11 15:33 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Justin Lecher (RETIRED) gentoo-dev 2013-01-09 10:07:35 UTC
When emerging with a full disk mounted on /usr/lib/debug with FEATURES=split-debug, portage copies files in live system but doesn't record them into the db.

emerge -1

-- /usr/lib/debug/
--- /usr/lib/debug/sbin/
!!! copy /var/tmp/portage/sys-fs/aufs-util-3.6/image/usr/lib/debug/sbin/auplink.debug -> /usr/lib/debug/sbin/auplink.debug failed.
!!! [Errno 28] No space left on device

>>> Failed to install sys-fs/aufs-util-3.6, Log file:

>>>  '/var/log/portage/build/sys-fs/aufs-util-3.6:20130109-095913.log'
 * 
 * The following package has failed to build or install:
 * 
 *  (sys-fs/aufs-util-3.6::gentoo, ebuild scheduled for merge), Log file:
 *   '/var/log/portage/build/sys-fs/aufs-util-3.6:20130109-095913.log'
 * 



After cleaning /usr/lib/debug

* Removing /usr/share/info
 * checking 18 files for package collisions
 * This package will overwrite one or more files that may belong to other
 * packages (see list below). You can use a command such as `portageq
 * owners / <filename>` to identify the installed package that owns a
 * file. If portageq reports that only one package owns a file then do
 * NOT file a bug report. A bug report is only useful if it identifies at
 * least two or more packages that are known to install the same file(s).
 * If a collision occurs and you can not explain where the file came from
 * then you should simply ignore the collision since there is not enough
 * information to determine if a real problem exists. Please do NOT file
 * a bug report at http://bugs.gentoo.org unless you report exactly which
 * two packages install the same file(s). Once again, please do NOT file
 * a bug report unless you have completely understood the above message.
 * 
 * package sys-fs/aufs-util-3.6 NOT merged
 * 
 * Detected file collision(s):
 * 
 * 	/usr/lib64/libau.so.2.6
 * 	/usr/bin/aubrsync
 * 	/usr/bin/auchk
 * 	/usr/bin/aubusy
 * 	/usr/share/doc/aufs-util-3.6/README.xz
 * 	/usr/share/man/man5/aufs.5.xz
 * 
 * Searching all installed packages for file collisions...
 * 
 * Press Ctrl-C to Stop
 * 
 * None of the installed packages claim the file(s).
 * 
 * Package 'sys-fs/aufs-util-3.6' NOT merged due to file collisions. If
 * necessary, refer to your elog messages for the whole content of the
 * above message.

>>> Failed to install sys-fs/aufs-util-3.6, Log file:







$ einfo aufs-util
Portage 2.2.0_alpha149 (default/linux/amd64/10.0, gcc-4.7.2-asneeded, glibc-2.16.0, 3.7.1-lh x86_64)
=================================================================
                        System Settings
=================================================================
System uname: Linux-3.7.1-lh-x86_64-Intel-R-_Core-TM-_i7-2600_CPU_@_3.40GHz-with-gentoo-2.2
Timestamp of tree: Wed, 09 Jan 2013 07:00:01 +0000
ld GNU gold (GNU Binutils 2.23.1) 1.11
ccache version 3.1.9 [enabled]
app-shells/bash:          4.2_p42
dev-java/java-config:     2.1.12-r1
dev-lang/python:          2.6.8-r1, 2.7.3-r3, 3.1.5-r1, 3.2.3-r2, 3.3.0-r1
dev-util/ccache:          3.1.9
dev-util/cmake:           2.8.10.2-r1
dev-util/pkgconfig:       0.27.1
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.6
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.9.6-r3, 1.10.3, 1.11.6, 1.12.6
sys-devel/binutils:       2.23.1
sys-devel/gcc:            4.1.2, 4.5.4, 4.6.3, 4.7.2
sys-devel/gcc-config:     1.8
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.7 (virtual/os-headers)
sys-libs/glibc:           2.16.0
Repositories: gentoo sunrise bicatali betagarden dummy neurogeek science last-hope g-ctan
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -ftracer -march=corei7-avx -mtune=corei7-avx -mcx16 -msahf -mno-movbe -maes -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=8192 -frecord-gcc-switches -g -Wimplicit-function-declaration"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/eselect/ /usr/share/gnupg/qualified.txt /usr/share/nano/ /usr/share/openvpn/easy-rsa /var/lib/hsqldb /var/spool/munin-async/.ssh /var/spool/torque"
CONFIG_PROTECT_MASK="${EPREFIX}/etc/gconf /etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -ftracer -march=corei7-avx -mtune=corei7-avx -mcx16 -msahf -mno-movbe -maes -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=8192 -frecord-gcc-switches -g -Wenum-compare"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=12 -t --keep-going --autounmask-write --autounmask --quiet-build=n"
FCFLAGS="-O2 -pipe -ftracer -march=corei7-avx -mtune=corei7-avx -mcx16 -msahf -mno-movbe -maes -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=8192 -frecord-gcc-switches -g"
FEATURES="assume-digests binpkg-logs buildsyspkg ccache collision-protect distlocks ebuild-locks fixlafiles merge-sync metadata-transfer multilib-strict news noinfo parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms sign split-log splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe -ftracer -march=corei7-avx -mtune=corei7-avx -mcx16 -msahf -mno-movbe -maes -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=8192 -frecord-gcc-switches -g"
GENTOO_MIRRORS=" /mnt/tmpfs/ http://gentoo.j-schmitz.net/mirror/ ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo ftp://ftp.tu-clausthal.de/pub/linux/gentoo/"
LANG="en_GB.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu -Wl,--sort-common"
MAKEOPTS="-j16 -l12"
PKGDIR="/usr/portage/packages"
PORTAGE_COMPRESS="xz"
PORTAGE_COMPRESS_FLAGS="-z -e -9 -v"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/local/overlays/sunrise-reviewed /local/overlays/bicatali /local/overlays/betagarden /local/overlays/dummy /local/overlays/neurogeek /local/overlays/sci /local/overlays/lh/ebuilds /local/overlays/g-ctan"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X acl alsa amd64 bash-completion berkdb branding bzip2 cairo caps cli cracklib crypt cups cxx dbus dri fortran gdbm gif gmp gnome gnome-keyring gpm gstreamer gtk iconv icu ipv6 jpeg jpeg2k ldap logrotate mmx mmxext modules mudflap multilib ncurses network-cron nls nptl nsplugin numa opengl openmp pam pcre png pppd pulseaudio qt3support readline session smp sse sse2 sse2_4way sse2check sse3 sse4 sse41 sse4_1 sse4a sse5 ssl ssse3 startup-notification tcpd threads tiff truetype unicode vaapi vdpau xinerama zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" DRACUT_MODULES="plymouth" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer nlpsolver" LINGUAS="en" NGINX_MODULES_HTTP="fastcgi rewrite dav stub_status" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python2_6 python3_1 python3_2 python3_3 pypy1_9 pypy2_0" QEMU_SOFTMMU_TARGETS="i386 x86_64 arm" QEMU_USER_TARGETS="i386 x86_64 arm" RUBY_TARGETS="ruby19" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="2.6 2.7 3.1 3.2 3.3 2.7-pypy-1.9 2.7-pypy-2.0"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Zac Medico gentoo-dev 2013-01-09 10:32:35 UTC
I guess we could try to reverse the merge process when we encounter ENOSPC.
Comment 2 abandoned account 2015-04-11 02:44:16 UTC
I thought this would work around it
# time ebuild /usr/portage/www-client/firefox/firefox-37.0.1.ebuild qmerge --force
but the --force is actually for manifest only according to `man ebuild`

But maybe ebuild could ignore the existing files if $I_KNOW_WHAT_I_AM_DOING  is set?

Or is there a way to tell it to ignore the fact that those files exist but only if no package owns them ? (because they're left over from the previous failed merge copy operation which failed with 'no space left on device' as OP states)
Comment 3 SpanKY gentoo-dev 2015-05-31 02:43:46 UTC
(In reply to EmanueL Czirai from comment #2)

no, we should not be eliding random files based on random env vars.  the merge step must fully succeed or it must be rejected.

the ENOSPC case is always hard to handle as calculating free space before merging does not guarantee that space continues to exist as you merge :/.  making hardlinks of all files in the original package and then removing those only once the merge has finished would allow for rollback, but it would also increase the amount of free space required than might otherwise be needed (since you're not reclaiming the space as you merge).

any other ideas on how to implement this safely ?
Comment 4 Alex Xu (Hello71) 2015-06-08 23:39:24 UTC
(In reply to SpanKY from comment #3)
> (In reply to EmanueL Czirai from comment #2)
> 
> no, we should not be eliding random files based on random env vars.  the
> merge step must fully succeed or it must be rejected.
> 
> the ENOSPC case is always hard to handle as calculating free space before
> merging does not guarantee that space continues to exist as you merge :/. 
> making hardlinks of all files in the original package and then removing
> those only once the merge has finished would allow for rollback, but it
> would also increase the amount of free space required than might otherwise
> be needed (since you're not reclaiming the space as you merge).
> 
> any other ideas on how to implement this safely ?

it all depends on how you want the system to end up when you're done failing and what compromises you're willing to make.

options:

0. delete and create
1. create hard links to ${D}
2. create side-by-side temp files and move into place
3. extend and overwrite

0 is the current solution, where the failure case is a mix of old and new files.

1 will not work if $PORTAGE_TMPDIR is on a tmpfs or any other different filesystem but is otherwise the best solution, since it requires no extra space. the failure case (e.g. if merge is terminated abnormally) is a mix of old and new files, but is unlikely because moving hard links is fast.

2 requires extra space equal to min(sum(old files), sum(new files)) and leaves temp files in case of failure that need to be cleaned later. the failure case is a mix of old and new files, but is unlikely because moving files is fast.

3 will not work with executable files. the worst-case failure is files that are mixes of old and new which hypothetically could lead to security problems in the worst case (e.g. imagine the file is /usr/bin/sshd and the old data was [JE ...] and the new data is [... JNE], then you will have [... JNE^WJE ...]).

there are probably other ideas, probably even worse than these. the best first step is to include a simple check that the current free space is greater than or equal to the sum of the file sizes of the package (assuming this is not yet done).
Comment 5 SpanKY gentoo-dev 2015-06-10 11:04:55 UTC
(In reply to Alex Xu (Hello71) from comment #4)

we do not do (0).  we first attempt to do the atomic rename from $D to $ROOT and if that fails, we fall back to (2).  but we do this on a per-file, not package, basis.

the fact that there's a mix of old and new files is not unique to any of the methods you describe.  the only way to avoid that would be to get FS support for transactions.  otherwise, we take the risk that the $D->$ROOT merge is fast or not critical enough to make things blow up.  this is the status quo and i don't think we need discuss that here.

your summary pretty much matches what i posted.  our options are:
- it works: we never leave the system in an inconsistent state
- it sometimes fails in which case we reserve the right to make it an implementation detail and we mark this bug was WONTFIX

if no one can think of a method that doesn't require free space matching `df $D`, then let's implement that and be done.
Comment 6 SpanKY gentoo-dev 2015-07-21 02:47:39 UTC
*** Bug 345645 has been marked as a duplicate of this bug. ***