Created attachment 366390 [details, diff] Patch for sys-apps/portage-2.2.7 When an already installed package is re-emerged (e.g. because compiler flags or use flags changed), some files usually don't change (e.g. documentation, icons, .., in case of unchanged compiler flags even binaries and libs). Unfortunately, emerge overwrites all files (except protected config files), even unchanged ones. This shortens the life of SSDs (solid state drives) because each page of flash memory can only be erased a limited number of times (only 1000 times in some contemporary consumer products with MLC or TLC flash memory). The attached patch makes vartree.py compare the MD5 digest of each file, which is computed anyway for /var/db/pkg/*-*/*/CONTENTS, with the MD5 digest of the already installed file that is going to be overwritten (if any), and when they match, leave the existing file alone and use the mtime of the existing file for CONTENTS.
[~] emerge --info Portage 2.2.7 (default/linux/amd64/13.0/no-multilib, gcc-4.7.3, glibc-2.15-r3, 3.12.5 x86_64) ================================================================= System uname: Linux-3.12.5-x86_64-Intel-R-_Core-TM-_i7-3537U_CPU_@_2.00GHz-with-gentoo-2.2 KiB Mem: 3660760 total, 2160612 free KiB Swap: 12582908 total, 12582908 free Timestamp of tree: Thu, 26 Dec 2013 15:45:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p45 dev-lang/python: 2.7.5-r3, 3.3.2-r2 dev-util/cmake: 2.8.11.2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.8 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.13.4 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage priority: -1000 ABI="amd64" ABI_X86="64" ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" ACCEPT_PROPERTIES="*" ACCEPT_RESTRICT="*" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ARCH="amd64" AUTOCLEAN="yes" BOOTSTRAP_USE="cxx unicode python_targets_python3_2 python_targets_python2_7 multilib -python_targets_python3_2 python_targets_python3_3 multilib -python_targets_python3_2 python_targets_python3_3" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -mtune=native -Os -fomit-frame-pointer -pipe" CFLAGS_amd64="-m64" CFLAGS_x32="-mx32" CFLAGS_x86="-m32" CHOST="x86_64-pc-linux-gnu" CHOST_amd64="x86_64-pc-linux-gnu" CHOST_x32="x86_64-pc-linux-gnux32" CHOST_x86="i686-pc-linux-gnu" CLEAN_DELAY="5" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" COLLISION_IGNORE="/lib/modules/* *.py[co] *$py.class */dropin.cache" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -mtune=native -Os -fomit-frame-pointer -pipe" DEFAULT_ABI="amd64" DISPLAY=":0.0" DISTDIR="/usr/portage/distfiles" EDITOR="/usr/bin/vi" ELIBC="glibc" EMERGE_DEFAULT_OPTS="-avt --fail-clean" EMERGE_WARNING_DELAY="10" ENLIGHTENMENT_MODULES="conf-comp conf-wallpaper2 contact music-control backlight battery clock conf-applications conf-dialogs conf-display conf-interaction conf-menus conf-paths conf-randr conf-shelves conf-theme conf-window-manipulation conf-window-remembers fileman fileman-opinfo ibar ibox mixer msgbus notification pager quickaccess start syscon systray tasks tiling winlist" EPREFIX="" EROOT="/" FCFLAGS="-march=native -mtune=native -Os -fomit-frame-pointer -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fail-clean fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"" FETCHCOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}"" FETCHCOMMAND_SFTP="bash -c "x=\${2#sftp://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; eval \"declare -a ssh_opts=(\${3})\" ; exec sftp -P \${port} \"\${ssh_opts[@]}\" \"\${host}:/\${x#*/}\" \"\$1\"" sftp "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" FETCHCOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec rsync --rsh=\"ssh -p\${port} \${3}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" FFLAGS="-march=native -mtune=native -Os -fomit-frame-pointer -pipe" FFTOOLS="" GCC_SPECS="" GENTOO_MIRRORS="rsync://de-mirror.org/gentoo/ ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo rsync://mirror.netcologne.de/gentoo/ http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ rsync://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.spline.inf.fu-berlin.de/mirrors/gentoo/ ftp://ftp.tu-clausthal.de/pub/linux/gentoo/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo rsync://ftp-stud.hs-esslingen.de/gentoo/ ftp://ftp6.uni-muenster.de/pub/linux/distributions/gentoo" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="" HOME="/root" INFOPATH="/usr/share/info:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.7.3/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.23.1/info" INPUT_DEVICES="keyboard mouse synaptics" IUSE_IMPLICIT="prefix" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LC_MESSAGES="C" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LDFLAGS_amd64="-m elf_x86_64" LDFLAGS_x32="-m elf32_x86_64" LDFLAGS_x86="-m elf_i386" LESS="-R -M --shift 5" LESSOPEN="|lesspipe %s" LIBDIR_amd64="lib64" LIBDIR_amd64_fbsd="lib64" LIBDIR_n32="lib32" LIBDIR_n64="lib64" LIBDIR_o32="lib" LIBDIR_ppc="lib32" LIBDIR_ppc64="lib64" LIBDIR_s390="lib32" LIBDIR_s390x="lib64" LIBDIR_sparc32="lib32" LIBDIR_sparc64="lib64" LIBDIR_x32="libx32" LIBDIR_x86="lib32" LIBDIR_x86_fbsd="lib32" LIBREOFFICE_EXTENSIONS="" LOGNAME="root" MAKEOPTS="-j4" MANPATH="/usr/local/share/man:/usr/share/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.7.3/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.23.1/man" MULTILIB_ABIS="amd64" MULTILIB_STRICT_DENY="64-bit.*shared object" MULTILIB_STRICT_DIRS="/lib32 /lib /usr/lib32 /usr/lib /usr/kde/*/lib32 /usr/kde/*/lib /usr/qt/*/lib32 /usr/qt/*/lib /usr/X11R6/lib32 /usr/X11R6/lib" MULTILIB_STRICT_EXEMPT="(perl5|gcc|gcc-lib|binutils|eclipse-3|debug|portage|udev|systemd|clang|python-exec)" MULTIOSDIRS="../lib64" NETBEANS="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" OPENGL_PROFILE="xorg-x11" PAGER="/usr/bin/less" PATH="~/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.7.3" PHP_TARGETS="php5-5" PKGDIR="/usr/portage/packages" PORTAGE_ARCHLIST="x86-freebsd x86-fbsd hppa x86-winnt m68k x86-linux sparc-solaris ppc64 x86-interix m68k-mint ppc-openbsd x86-macos x86-cygwin amd64-linux x86-solaris hppa-hpux sh ppc64-linux amd64 x64-macos x86 x64-openbsd sparc-fbsd arm-linux mips ia64-linux ppc-macos x86-netbsd ia64-hpux sparc x86-openbsd ia64 s390 x64-freebsd amd64-fbsd alpha ppc-aix sparc64-freebsd x64-solaris ppc arm sparc64-solaris" PORTAGE_BIN_PATH="/usr/lib64/portage/bin" PORTAGE_COMPRESS_EXCLUDE_SUFFIXES="css gif htm[l]? jp[e]?g js pdf png" PORTAGE_CONFIGROOT="/" PORTAGE_DEBUG="0" PORTAGE_DEPCACHEDIR="/var/cache/edb/dep" PORTAGE_ELOG_CLASSES="log warn error" PORTAGE_ELOG_MAILFROM="portage@localhost" PORTAGE_ELOG_MAILSUBJECT="[portage] ebuild log for ${PACKAGE} on ${HOST}" PORTAGE_ELOG_MAILURI="root" PORTAGE_ELOG_SYSTEM="save_summary:log,warn,error echo" PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS="5" PORTAGE_FETCH_RESUME_MIN_SIZE="350K" PORTAGE_GID="250" PORTAGE_GPG_SIGNING_COMMAND="gpg --sign --digest-algo SHA256 --clearsign --yes --default-key "${PORTAGE_GPG_KEY}" --homedir "${PORTAGE_GPG_DIR}" "${FILE}"" PORTAGE_INST_GID="0" PORTAGE_INST_UID="0" PORTAGE_INTERNAL_CALLER="1" PORTAGE_IONICE_COMMAND="schedtool -B ${PID}" PORTAGE_OVERRIDE_EPREFIX="" PORTAGE_PYM_PATH="/usr/lib64/portage/pym" PORTAGE_PYTHONPATH="/usr/lib64/portage/pym" PORTAGE_REPOSITORIES="[DEFAULT] main-repo = gentoo [gentoo] location = /usr/portage masters = priority = -1000 sync-type = rsync sync-uri = rsync://rsync.europe.gentoo.org/gentoo-portage "PORTAGE_RSYNC_EXTRA_OPTS="--inplace --compress-level=9" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --delete --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'" PORTAGE_RSYNC_RETRIES="-1" PORTAGE_SYNC_STALE="30" PORTAGE_TMPDIR="/var/tmp" PORTAGE_VERBOSE="1" PORTAGE_WORKDIR_MODE="0700" PORTAGE_XATTR_EXCLUDE="security.* system.nfs4_acl" PORTDIR="/usr/portage" PORT_LOGDIR_CLEAN="find "${PORT_LOGDIR}" -type f ! -name "summary.log*" -mtime +7 -delete" PRELINK_PATH_MASK="/usr/lib64/libfreebl3.so:/usr/lib64/libnssdbm3.so:/usr/lib64/libsoftokn3.so" PROFILE_ONLY_VARIABLES="ARCH ELIBC IUSE_IMPLICIT KERNEL USERLAND USE_EXPAND_IMPLICIT USE_EXPAND_UNPREFIXED USE_EXPAND_VALUES_ARCH USE_EXPAND_VALUES_ELIBC USE_EXPAND_VALUES_KERNEL USE_EXPAND_VALUES_USERLAND" PWD="/root" PYTHONDONTWRITEBYTECODE="1" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" QT_GRAPHICSSYSTEM="raster" RESUMECOMMAND="wget -c -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"" RESUMECOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}"" RESUMECOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec rsync --rsh=\"ssh -p\${port} \${3}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}" "${PORTAGE_SSH_OPTS}"" ROOT="/" ROOTPATH="/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.7.3" RPMDIR="/usr/portage/rpm" RUBY_TARGETS="ruby19 ruby18" SHELL="/bin/bash" SHLVL="1" SYMLINK_LIB="yes" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" TERM="xterm" UNINSTALL_IGNORE="/lib/modules/*" USE="X aac alsa amd64 apng avx bash-completion bzip2 cairo cli cracklib crypt cxx dri fontconfig gif gstreamer iconv icu imap inotify jpeg jpg lzma minimal mmx modules mp3 mudflap ncurses netlink nptl opengl openmp pdf png qt qt3support qt4 readline session smp sse sse2 sse3 sse4 sse4_1 sse4_2 ssh ssl ssse3 svg threads tiff truetype unicode usb vaapi vim-syntax wifi x264 xcb xcomposite xv xz zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" ENLIGHTENMENT_MODULES="conf-comp conf-wallpaper2 contact music-control backlight battery clock conf-applications conf-dialogs conf-display conf-interaction conf-menus conf-paths conf-randr conf-shelves conf-theme conf-window-manipulation conf-window-remembers fileman fileman-opinfo ibar ibox mixer msgbus notification pager quickaccess start syscon systray tasks tiling winlist" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" USER="root" USERLAND="GNU" USE_EXPAND="ABI_MIPS ABI_X86 ALSA_CARDS APACHE2_MODULES APACHE2_MPMS CALLIGRA_FEATURES CAMERAS COLLECTD_PLUGINS CROSSCOMPILE_OPTS CURL_SSL DRACUT_MODULES DVB_CARDS ELIBC ENLIGHTENMENT_MODULES FCDSL_CARDS FFTOOLS FOO2ZJS_DEVICES FRITZCAPI_CARDS GPSD_PROTOCOLS GRUB_PLATFORMS INPUT_DEVICES KERNEL LCD_DEVICES LIBREOFFICE_EXTENSIONS LINGUAS LIRC_DEVICES MONKEYD_PLUGINS NETBEANS_MODULES NGINX_MODULES_HTTP NGINX_MODULES_MAIL OFED_DRIVERS OFFICE_IMPLEMENTATION OPENMPI_FABRICS OPENMPI_OFED_FEATURES OPENMPI_RM PHP_TARGETS PYTHON_SINGLE_TARGET PYTHON_TARGETS QEMU_SOFTMMU_TARGETS QEMU_USER_TARGETS RUBY_TARGETS SANE_BACKENDS USERLAND VIDEO_CARDS VOICEMAIL_STORAGE XFCE_PLUGINS XTABLES_ADDONS" USE_EXPAND_HIDDEN="ABI_MIPS ABI_X86 CROSSCOMPILE_OPTS ELIBC KERNEL USERLAND" USE_EXPAND_IMPLICIT="ARCH ELIBC KERNEL USERLAND" USE_EXPAND_UNPREFIXED="ARCH" USE_EXPAND_VALUES_ARCH="alpha amd64 amd64-fbsd amd64-linux arm arm-linux hppa hppa-hpux ia64 ia64-hpux ia64-linux m68k m68k-mint mips ppc ppc64 ppc64-linux ppc-aix ppc-macos ppc-openbsd s390 sh sparc sparc64-freebsd sparc64-solaris sparc-fbsd sparc-solaris x64-freebsd x64-macos x64-openbsd x64-solaris x86 x86-cygwin x86-fbsd x86-freebsd x86-interix x86-linux x86-macos x86-netbsd x86-openbsd x86-solaris x86-winnt" USE_EXPAND_VALUES_ELIBC="AIX Cygwin Darwin FreeBSD glibc HPUX Interix mintlib musl NetBSD OpenBSD SunOS uclibc Winnt" USE_EXPAND_VALUES_KERNEL="AIX Cygwin Darwin FreeBSD freemint HPUX Interix linux NetBSD OpenBSD SunOS Winnt" USE_EXPAND_VALUES_USERLAND="BSD GNU" VIDEO_CARDS="intel i965" XDG_CONFIG_DIRS="/etc/xdg" XDG_DATA_DIRS="/usr/local/share:/usr/share" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" _="/usr/bin/emerge"
Script to see how `emerge rfkill` overwrites unchanged files (I choose net-wireless/rfkill because it installs only 2 files): ----- BEGIN SCRIPT ----- #!/bin/bash (md5sum `qlist rfkill`; stat `qlist rfkill`) >/tmp/emerge-stat-before.log inotifywatch -t15 `qlist rfkill|xargs -n1 dirname` >/tmp/emerge-inotifywatch.log & inotifywatch_pid="$!" emerge --ignore-default-opts rfkill (md5sum `qlist rfkill`; stat `qlist rfkill`) >/tmp/emerge-stat-after.log diff -u5 /tmp/emerge-stat-{before,after}.log wait $inotifywatch_pid cat /tmp/emerge-inotifywatch.log ----- END SCRIPT -----
seems like a good idea, however the implementation leaves a little to be desired. hashing every file is needlessly expensive. start off with a stat to verify time/sizes are the same before resorting to hashing. also, we'd like to have patches posted to the mailing list for review.
