Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 411555 - sys-devel/gcc failed to pack binary package due to tar "file changed as we read it" error for large files on ZFS
Summary: sys-devel/gcc failed to pack binary package due to tar "file changed as we re...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Unclassified (show other bugs)
Hardware: All All
: High normal with 1 vote (vote)
Assignee: Richard Yao (RETIRED)
URL: https://github.com/zfsonlinux/zfs/iss...
Whiteboard: Race condition in mmap code
Keywords: Bug, REGRESSION
Depends on:
Blocks:
 
Reported: 2012-04-11 05:56 UTC by Jonathan Vasquez (RETIRED)
Modified: 2012-12-11 19:45 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
sys-devel/gcc-4.5.3-r2/temp directory (gcc_temp_logsdir.tar.bz2,252.01 KB, application/x-bzip)
2012-04-11 05:56 UTC, Jonathan Vasquez (RETIRED)
Details
last part of gcc compile (lsof) (out,63.87 KB, text/plain)
2012-04-12 04:01 UTC, Jonathan Vasquez (RETIRED)
Details
lsof, output corrected (dddd,47.04 KB, text/plain)
2012-04-12 04:06 UTC, Jonathan Vasquez (RETIRED)
Details
Patch to workaround for bug in ZFSOnLinux (pax-utils-zfs-workaround.patch,678 bytes, patch)
2012-10-06 23:16 UTC, Richard Yao (RETIRED)
Details | Diff
Patch to solve mtime issue (zfs-kmod-0.6.0_rc12-mmap-mtime-update.patch,3.07 KB, patch)
2012-12-04 01:33 UTC, Richard Yao (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-11 05:56:31 UTC
Created attachment 308491 [details]
sys-devel/gcc-4.5.3-r2/temp directory

When compiling sys-devel/gcc-4.5.3-r2, portage dies after it successfully compiles gcc, when trying to make it into a package.

1. Make sure "buildpkg" is in your FEATURES variable in /etc/make.conf
2. emerge -av sys-devel/gcc
3. Wait for it to die after it spends a lovely time compiling.

Portage 2.1.10.49 (default/linux/amd64/10.0/desktop/kde, gcc-4.5.3, glibc-2.13-r4, 3.3.1-LAPTOP x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-3.3.1-LAPTOP-x86_64-Intel-R-_Core-TM-_i7-2640M_CPU_@_2.80GHz-with-gentoo-2.0.3
Timestamp of tree: Tue, 10 Apr 2012 13:15:01 +0000
app-shells/bash:          4.2_p20
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.2-r3, 3.2.2
dev-util/cmake:           2.8.6-r4
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.0.3
sys-apps/openrc:          0.9.9
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.11.1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.5.3-r2
sys-devel/gcc-config:     1.5-r2
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r1
sys-kernel/linux-headers: 3.1 (virtual/os-headers)
sys-libs/glibc:           2.13-r4
Repositories: gentoo mva bumblebee
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA google-chrome"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=core2 -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs buildpkg distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="ftp://mirrors.rit.edu/gentoo/ http://mirrors.rit.edu/gentoo/"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en_US"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/mva /var/lib/layman/bumblebee"
SYNC="rsync://rsync5.us.gentoo.org/gentoo-portage"
USE="X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt css cups cxx dbus declarative dri dts dvd dvdr emboss encode exif fam ffmpeg firefox flac fortran ftp gdbm gdu gif git gnutls gpm gstreamer hddtemp iconv ipv6 jpeg jpeg2k kde kipi lame lcms ldap libnotify lm_sensors mad mmx mng modules mp3 mp4 mpeg mudflap multilib ncurses nls nptl nptlonly ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pppd qt3support qt4 readline samba sdl session spell sse sse2 ssl startup-notification subversion svg sysfs syslog tcpd tiff truetype udev unicode usb v4l vorbis wavpack wifi x264 xcb xcomposite xinerama xml xorg xscreensaver xulrunner xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug 
ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses 
text" LINGUAS="en_US" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="intel nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
                        Package Settings
=================================================================
USE="cxx fortran mudflap (multilib) nls nptl openmp (-altivec) -bootstrap -build -doc (-fixed-point) -gcj -graphite -gtk (-hardened) (-libssp) -lto -multislot -nocxx -nopie -nossp -objc -objc++ -objc-gc -test -vanilla"

sys-devel/gcc-4.5.3-r2 was built with the following:
CFLAGS="-O2 -pipe"
CXXFLAGS="-O2 -pipe"
Comment 1 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-11 06:08:47 UTC
I just noticed that the attachment didn't upload as a .tar.bz2 file but rather as plain/text. 

Here is the build.log, let me know if you need any other logs or info.


./usr/bin/cpp-4.5.3
./usr/bin/x86_64-pc-linux-gnu-gfortran-4.5.3
./usr/bin/x86_64-pc-linux-gnu-c++-4.5.3
./usr/bin/x86_64-pc-linux-gnu-g++-4.5.3
./usr/bin/x86_64-pc-linux-gnu-cpp-4.5.3
./usr/bin/gcov-4.5.3
./usr/bin/g++-4.5.3
./usr/bin/c++-4.5.3
./usr/x86_64-pc-linux-gnu/
./usr/x86_64-pc-linux-gnu/gcc-bin/
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/gcov
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/x86_64-pc-linux-gnu-c++
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/c++
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/x86_64-pc-linux-gnu-gcov
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/x86_64-pc-linux-gnu-gcc-4.5.3
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/x86_64-pc-linux-gnu-gfortran
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/cpp
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/gfortran
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/gcc
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/g++
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/x86_64-pc-linux-gnu-g++
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/x86_64-pc-linux-gnu-gcc
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/x86_64-pc-linux-gnu-cpp
./usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/gccbug
 [31;01m*[0m ERROR: sys-devel/gcc-4.5.3-r2 failed (package phase):
 [31;01m*[0m   failed to pack binary package: '/usr/portage/packages/sys-devel/gcc-4.5.3-r2.tbz2.22130'
 [31;01m*[0m 
 [31;01m*[0m Call stack:
 [31;01m*[0m       misc-functions.sh, line 1249:  Called dyn_package
 [31;01m*[0m       misc-functions.sh, line 1132:  Called assert 'failed to pack binary package: '/usr/portage/packages/sys-devel/gcc-4.5.3-r2.tbz2.22130''
 [31;01m*[0m   isolated-functions.sh, line   14:  Called die
 [31;01m*[0m The specific snippet of code:
 [31;01m*[0m   		[[ $x -eq 0 ]] || die "$@"
 [31;01m*[0m 
 [31;01m*[0m If you need support, post the output of 'emerge --info =sys-devel/gcc-4.5.3-r2',
 [31;01m*[0m the complete build log and the output of 'emerge -pqv =sys-devel/gcc-4.5.3-r2'.
 [31;01m*[0m 
 [31;01m*[0m Please include /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc-build-logs.tar.bz2 in your bug report
 [31;01m*[0m 
 [31;01m*[0m The complete build log is located at '/var/tmp/portage/sys-devel/gcc-4.5.3-r2/temp/build.log'.
 [31;01m*[0m The ebuild environment file is located at '/var/tmp/portage/sys-devel/gcc-4.5.3-r2/temp/environment'.
 [31;01m*[0m S: '/var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build'
Comment 2 Jeroen Roovers (RETIRED) gentoo-dev 2012-04-11 14:31:57 UTC
Comment on attachment 308491 [details]
sys-devel/gcc-4.5.3-r2/temp directory

There's no magic in uploading - the MIME type simply wasn't properly set.
Comment 3 Jeroen Roovers (RETIRED) gentoo-dev 2012-04-11 14:38:19 UTC
./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/cc1
tar: ./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/cc1: file changed as we read it
./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/collect2
./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/cc1plus
tar: ./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/cc1plus: file changed as we read it
./usr/lib/                            
./usr/lib/gcc/

Now why would it do that? It causes tar to exit with a non-0 status, hence further on:
 * failed to pack binary package: '/usr/portage/packages/sys-devel/gcc-4.5.3-r2.tbz2.22130'

At a guess, a sub-process in src_install() hadn't finished yet?
Comment 4 Zac Medico gentoo-dev 2012-04-11 16:39:47 UTC
(In reply to comment #3)
> At a guess, a sub-process in src_install() hadn't finished yet?

Yeah, the gcc build system is known to leave processes running in the background, as shown by bug #278895.
Comment 5 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-11 18:45:20 UTC
Since #Bug 278895 is marked fixed (and it's from 09-10), this shouldn't be happening correct?
Comment 6 Zac Medico gentoo-dev 2012-04-11 18:56:37 UTC
The fix for bug #278895 simply makes execution continue to the next phase after the current phase function returns, regardless of any background processes that are left running after a phase function returns.
Comment 7 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-11 19:12:08 UTC
Oh ok. Is there anything else I can do to assist with debugging this problem?
Comment 8 Zac Medico gentoo-dev 2012-04-11 19:28:54 UTC
Maybe you can use a command like this to identify the background process:

   lsof | grep /var/tmp/portage
Comment 9 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-12 03:34:04 UTC
I used lsof about >10 before it stopped. I tried to use it before it crashed but when I did, the output of blank.

Here is what I saw:


emake      1104       root  cwd       DIR               0,15       22     554911 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build
make       1106       root  cwd       DIR               0,15       22     554911 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build
make       1106       root    6r      DIR               0,15       67    1580185 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/gcc-4.5.3
sh         1108       root  cwd       DIR               0,15       22     554911 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build
make       1121       root  cwd       DIR               0,15       22     554911 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build
make       1121       root    6r      DIR               0,15       67    1580185 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/gcc-4.5.3
sh         6339       root  cwd       DIR               0,15       22     554911 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build
make       6394       root  cwd       DIR               0,15       22     554911 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build
make       6394       root    6r      DIR               0,15       67    1580185 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/gcc-4.5.3
sh        20128       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
make      20170       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
xgcc      21112       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
xgcc      21112       root  txt       REG               0,15   350590     559990 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/xgcc
cc1       21113       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
cc1       21113       root  txt       REG               0,15 15212593     560017 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/cc1
as        21114       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
as        21114       root    3u      REG               0,15        0     559808 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc/insn-recog.o
xgcc      21485       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
xgcc      21485       root  txt       REG               0,15   350590     559990 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/xgcc
cc1       21486       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
cc1       21486       root  txt       REG               0,15 15212593     560017 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/cc1
as        21487       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
as        21487       root    3u      REG               0,15        0     559988 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc/lambda-code.o
xgcc      21518       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
xgcc      21518       root  txt       REG               0,15   350590     559990 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/xgcc
cc1       21519       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
cc1       21519       root  txt       REG               0,15 15212593     560017 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/cc1
as        21520       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
as        21520       root    3u      REG               0,15        0     560009 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc/loop-invariant.o
xgcc      21522       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
xgcc      21522       root  txt       REG               0,15   350590     559990 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/xgcc
cc1       21523       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
cc1       21523       root  txt       REG               0,15 15212593     560017 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/cc1
as        21524       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
as        21524       root    3u      REG               0,15        0     560011 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc/loop-iv.o
xgcc      21526       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
xgcc      21526       root  txt       REG               0,15   350590     559990 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/xgcc
cc1       21527       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
cc1       21527       root  txt       REG               0,15 15212593     560017 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/prev-gcc/cc1
as        21528       root  cwd       DIR               0,15      348     557098 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc
as        21528       root    3u      REG               0,15        0     560014 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build/gcc/loop-unroll.o
emerge    30052       root    4uW     REG               0,15        0    1580162 /var/tmp/portage/sys-devel/.gcc-4.5.3-r2.portage_lockfile
emerge    30052       root    6r     FIFO               0,15      0t0    1580178 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/.ipc_in
ebuild.sh 31430       root  cwd       DIR               0,15       12    1580163 /var/tmp/portage/sys-devel/gcc-4.5.3-r2
ebuild.sh 31447       root  cwd       DIR               0,15       22     554911 /var/tmp/portage/sys-devel/gcc-4.5.3-r2/work/build
Comment 10 Zac Medico gentoo-dev 2012-04-12 03:52:25 UTC
(In reply to comment #9)
> I tried to use it before it crashed
> but when I did, the output of blank.

I guess it exited before you could run lsof. That's unfortunate, since it leaves us without a clue about the program that interfered with tar.
Comment 11 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-12 03:54:10 UTC
Don't worry, I put lsof inside a "while :" script. It should die in a few more minutes of compiling. So far my lsof output file is 24M :).
Comment 12 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-12 04:01:14 UTC
Created attachment 308603 [details]
last part of gcc compile (lsof)
Comment 13 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-12 04:06:09 UTC
Created attachment 308605 [details]
lsof, output corrected

I uploaded this second file since I didn't know there was a 500 line limit. This is the last 47X+ lines of my output file.
Comment 14 Zac Medico gentoo-dev 2012-04-12 16:40:26 UTC
(In reply to comment #13)
> Created attachment 308605 [details]

PID 19148 appears to be the tar process that's spawned by emerge to create the binary package. The last file that it can be seen adding to the tar file is /var/tmp/portage/sys-devel/gcc-4.5.3-r2/image/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.3/info/gccint.info.bz2, and I don't see any other processes in the lsof output besides the expected subprocesses of emerge (misc-functions.sh and bzip2). Did the build log contain a "file changed as we read it" message for gccint.info.bz2?
Comment 15 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-13 03:41:40 UTC
There are "file changed as we read it" for the following:

./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/cc1plus
./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/cc1
Comment 16 Zac Medico gentoo-dev 2012-04-13 04:37:28 UTC
(In reply to comment #15)
> There are "file changed as we read it" for the following:
> 
> ./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/cc1plus
> ./usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/cc1

Does it happen only with the cc1plus and cc1 files every time it fails?

Does it fail like this every time you build gcc?
Comment 17 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-13 04:39:17 UTC
Yes. I had three build logs saved on my computer, and all of them display that same message when I search for "file changed as we read it".
Comment 18 Zac Medico gentoo-dev 2012-04-13 04:52:06 UTC
See if you can create the binary package manually with this command:

   ebuild /usr/portage/sys-devel/gcc/gcc-4.5.3-r2.ebuild package

If you have an existing failed build inside /var/tmp/portage/sys-devel/gcc-4.5.3-r2, then it will resume after the last successful ebuild phase.
Comment 19 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-13 05:30:53 UTC
Yup that worked. It created it successfully.
Comment 20 Zac Medico gentoo-dev 2012-04-13 05:34:57 UTC
What filesystem type is /var/tmp/portage on?
Comment 21 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-13 05:36:32 UTC
I did a ebuild <package> clean, cleaned out the gcc packages in /usr/portage/sys-devel/gcc since there were like 10 failed attempts (example: gcc..tbz2.####). Recompiling and going to wait till it fails, and rerun the command. I want to make sure there was no interference.
Comment 22 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-13 05:36:41 UTC
(In reply to comment #20)
> What filesystem type is /var/tmp/portage on?

ZFS
Comment 23 Zac Medico gentoo-dev 2012-04-13 05:41:47 UTC
(In reply to comment #22)
> (In reply to comment #20)
> > What filesystem type is /var/tmp/portage on?
> 
> ZFS

I suspect that we have an interaction between ZFS and tar that's causing this. The cc1plus and cc1 files are relatively large, and it seems as though ZFS is reporting different fstat results as it's flushing these large files to disk, which triggers the "file changed as we read it" error from tar.
Comment 24 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-13 05:57:29 UTC
Alright thanks.

I reran the test (took 20 min to compile gcc on core i5 4 cores 2.6 ghz), and it failed as usual. The /usr/portage/packages/sys-devel had the renmant of the attempted package make (gcc-4.5.3-r2.tbz2.3944, maybe portage can clean up if the build package step failed?). After running the ebuild package command, it created the .tbz2 successfully.
Comment 25 Zac Medico gentoo-dev 2012-04-13 19:38:00 UTC
(In reply to comment #24)
> The /usr/portage/packages/sys-devel had the renmant
> of the attempted package make (gcc-4.5.3-r2.tbz2.3944, maybe portage can
> clean up if the build package step failed?).

Okay, done:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=54597d1ce55bd3d67c22a1cc5c7e915a02406477

> After running the ebuild package command, it created the .tbz2 successfully.

This is consistent with my hypothesis from comment #23, since that stat results should stabilize after ZFS has finished flushing the files to disk. I guess that either ZFS needs to be fixed to return consistent stat results, or else tar needs to be fixed to interpret the stat results differently. I don't think that portage should have to call sync/syncfs for this to work, since it seems like things like this should "just work" regardless of disk sync state.
Comment 26 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-13 21:47:29 UTC
Thanks for the cleanup code.

I agree, portage shouldn't be responsible for filesystem stuff.
Comment 27 Richard Yao (RETIRED) gentoo-dev 2012-04-14 06:24:15 UTC
I am aware of this issue, but it is difficult to tell where to begin and this is really an upstream issue. I am marking this as confirmed, but fixing this is a low priority. I will bring this upstream as soon as I find a way for them to reproduce this issue that involves minimal effort on their part.
Comment 28 Richard Yao (RETIRED) gentoo-dev 2012-04-16 06:49:02 UTC
Jonathan, would you try the following and let me know if you can reproduce it:

1. Set `$EPREFIX` in your environment to point to an empty directory on a ZFS filesystem
2. Run `wget http://www.cs.stonybrook.edu/~ryao/prefix-install.sh && chmod u+x ./prefix-install.sh && env MAKEOPTS=-j5 ./prefix-install.sh`.  Note that this could take a few hours, but it does not require root privileges. Also, MAKEOPTS is optional. I use it to make this faster.
3.  After it is finished, you should a Gentoo userland installed in $EPREFIX, with a script at "${EPREFIX}/startprefix". Run `${EPREFIX}/startprefix` to enter a Gentoo shell.
4. Run `env FEATURES=buildpkg emerge -1v gcc`

I had planned to send these instructions upstream so that they could reproduce it, but unfortunately, I was unable to reproduce it by doing this myself. Knowing whether or not this issue occurs on your system when doing this would be helpful.
Comment 29 Jonathan Vasquez (RETIRED) gentoo-dev 2012-04-16 07:17:14 UTC
Alright. I will be testing this Tuesday evening since tomorrow I will be at school the whole day till about 8:10 PM.
Comment 30 Frido Ferdinand 2012-05-23 21:10:05 UTC
Oddly enough i hit the same problem while doing an emerge world in a chroot on a ZFS filesystem. It is only triggered with FEATURES=buildpkg. Strange behaviour so I searched a bit and the problem seems to be in two pax-mark functions in toolchain.class (line 1585):

# Disable RANDMMAP so PCH works. #301299
if tc_version_is_at_least 4.3 ; then
   pax-mark -r "${D}${PREFIX}/libexec/gcc/${CTARGET}/${GCC_CONFIG_VER}/cc1"
   pax-mark -r "${D}${PREFIX}/libexec/gcc/${CTARGET}/${GCC_CONFIG_VER}/cc1plus"
fi

since i don't have paxctl pax-utils.eclass will use scanelf to set the -r bit.

Simple reproduce looks like this, notice how the error dissapears the second
time we run tar.

enu ~ # scanelf -xX   /usr/bin/find
 TYPE    PAX   FILE 
ET_EXEC ---xe- /usr/bin/find 

enu ~ # scanelf -xXz -r    /usr/bin/find
 TYPE    PAX   FILE 
ET_EXEC ---xer /usr/bin/find 

enu ~ # tar -czvf find.tgz /usr/bin/find
tar: Removing leading `/' from member names
/usr/bin/find
tar: /usr/bin/find: file changed as we read it

enu ~ # scanelf -xX   /usr/bin/find
 TYPE    PAX   FILE 
ET_EXEC ---xer /usr/bin/find 

enu ~ # tar -czvf bla.tgz /usr/bin/find
tar: Removing leading `/' from member names
/usr/bin/find

enu ~ # scanelf -xXz -    /usr/bin/find
 TYPE    PAX   FILE 
ET_EXEC ---xe- /usr/bin/find 

enu ~ # tar -czvf bla.tgz /usr/bin/find
tar: Removing leading `/' from member names
/usr/bin/find
tar: /usr/bin/find: file changed as we read it

enu ~ # tar -czvf bla.tgz /usr/bin/find
tar: Removing leading `/' from member names
/usr/bin/find

Not exactly sure if this really a problem or what pax/scanelf exactly does but hope it helps somewhat. All this works fine on ext3/4.
Comment 31 Richard Yao (RETIRED) gentoo-dev 2012-05-24 07:09:43 UTC
(In reply to comment #30)
> Oddly enough i hit the same problem while doing an emerge world in a chroot
> on a ZFS filesystem. It is only triggered with FEATURES=buildpkg. Strange
> behaviour so I searched a bit and the problem seems to be in two pax-mark
> functions in toolchain.class (line 1585):

This is extremely helpful information. Thankyou for reporting this.
Comment 32 Richard Yao (RETIRED) gentoo-dev 2012-05-30 01:32:49 UTC
For the record, it is possible to reproduce this with a simple hello world file:

`gcc hello.c && scanelf -Xxz -r a.out && tar -cf hello.tgz a.out`

I suspect that this is a bug in the mmap code, which is what is used to modify the file.
Comment 33 Frido Ferdinand 2012-06-09 12:47:20 UTC
(In reply to comment #32)
> For the record, it is possible to reproduce this with a simple hello world
> file:
> 
> `gcc hello.c && scanelf -Xxz -r a.out && tar -cf hello.tgz a.out`
> 
> I suspect that this is a bug in the mmap code, which is what is used to
> modify the file.

Yes same with hello world:

enu data # gcc hello.c && scanelf -Xxz -r a.out && tar -cf hello.tgz a.out
 TYPE    PAX   FILE 
ET_EXEC ---xer a.out 
tar: a.out: file changed as we read it

Read that zfsonlinux has some pretty deep hooks in the memory management code different then what is used in most filesystems to emulate the solaris behaviour. Might not be what pax e.a. are expecting. (Added to CC)
Comment 34 Richard Yao (RETIRED) gentoo-dev 2012-10-06 22:05:51 UTC
I think I understand what is happening here now.

ZFSOnLinux will asynchronously perform mtime updates on mmapped files instead of updating mtime when changes are flushed. This is specific to the Linux port of the code and it appears that GNU Tar is the only software in the world that is sensitive to this.

I am testing a workaround that we could use to avoid triggering this during binary package generation, but the bug will remain open until it is fixed upstream.
Comment 35 Richard Yao (RETIRED) gentoo-dev 2012-10-06 23:16:12 UTC
Created attachment 325856 [details, diff]
Patch to workaround for bug in ZFSOnLinux

I have written a patch for pax-utils.eclass that will workaround this until a proper fix is done in ZFSOnLinux. Unfortunately, the Gentoo Hardened developers are unwilling to accept this change as a temporary measure. This will need to be applied manually by people affected by this after each sync until the actual upstream issue is solved.
Comment 36 Richard Yao (RETIRED) gentoo-dev 2012-10-22 00:45:07 UTC
This breaks catalyst, so I am increasing the priority.
Comment 37 Richard Yao (RETIRED) gentoo-dev 2012-12-04 01:33:53 UTC
Created attachment 331362 [details, diff]
Patch to solve mtime issue

I have written a patch for this issue. It has been submitted upstream:

https://github.com/zfsonlinux/zfs/pull/1127

I will commit it to portage within the next week.
Comment 38 Richard Yao (RETIRED) gentoo-dev 2012-12-11 19:45:25 UTC
This is fixed in sys-fs/zfs-kmod-0.6.0_rc12-r1.