Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 338509 - quickpkg turns hard linked files into empty files
Summary: quickpkg turns hard linked files into empty files
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
: 347778 354697 (view as bug list)
Depends on: 185305 359193
Blocks: 349307
  Show dependency tree
 
Reported: 2010-09-23 23:13 UTC by Hodur
Modified: 2011-03-16 21:15 UTC (History)
9 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Checks for zero byte files and attempts to lists or reinstalls as requested. (zerobyte-chk.sh,1.84 KB, text/plain)
2011-01-12 23:55 UTC, Roger
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Hodur 2010-09-23 23:13:58 UTC
"quickpkg gcc" creates empty /usr/x86_64-pc-linux-gnu/gcc-bin/4.4.4/x86_64-pc-linux-gnu-g++ file


Reproducible: Always

Steps to Reproduce:
1. Create gcc package
2. Unpack package into temp directory and check /usr/x86_64-pc-linux-gnu/gcc-bin/4.4.4/x86_64-pc-linux-gnu-g++ binary
3. Install package via emerge -g gcc


Actual Results:  
1. quickpkg gcc
 * Building package for sys-devel/gcc-4.4.4-r2 ...                       [ ok ]
 * Packages now in '/usr/portage/packages/amd64':
 * sys-devel/gcc-4.4.4-r2: 14.2M

> tar xjpf /usr/portage/packages/amd64/sys-devel/gcc-4.4.4-r2.tbz2 -C /tmp/
> file /tmp/usr/x86_64-pc-linux-gnu/gcc-bin/4.4.4/x86_64-pc-linux-gnu-g++
 /tmp/usr/x86_64-pc-linux-gnu/gcc-bin/4.4.4/x86_64-pc-linux-gnu-g++: empty

> emerge -g gcc
>>>> Emerging binary (1 of 1) sys-devel/gcc-4.4.4-r2
 * gcc-4.4.4-r2.tbz2 MD5 SHA1 size ;-) ...                               [ ok ]
>>> Extracting info
 * CPV:  sys-devel/gcc-4.4.4-r2
 * REPO: gentoo
 * USE:  amd64 elibc_glibc kernel_linux mudflap multilib nptl openmp userland_GNU
>>> Extracting sys-devel/gcc-4.4.4-r2

>>> Installing (1 of 1) sys-devel/gcc-4.4.4-r2
 * Removing /usr/share/man
 * Removing /usr/share/info
 * Switching native-compiler to x86_64-pc-linux-gnu-4.4.4 ...             [ ok ]

> gcc-config -l
 [1] x86_64-pc-linux-gnu-4.4.4 *

> g++
gcc-config: error: could not run/locate 'g++'


Expected Results:  
> g++
 g++: no input files

Portage 2.1.8.3 (default/linux/amd64/10.0, gcc-4.4.4, glibc-2.11.2-r0, 2.6.35-gentoo-r8 x86_64)
=================================================================
System uname: Linux-2.6.35-gentoo-r8-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_5000+-with-gentoo-2.0.1
Timestamp of tree: Thu, 23 Sep 2010 04:15:01 +0000
app-shells/bash:     4.0_p37
dev-java/java-config: 2.1.11
dev-lang/python:     2.6.5-r3, 3.1.2-r3
sys-apps/baselayout: 2.0.1
sys-apps/openrc:     0.6.3
sys-apps/sandbox:    1.6-r2
sys-devel/autoconf:  2.65-r1
sys-devel/automake:  1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.4-r2
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6b
sys-devel/make:      3.81-r2
virtual/os-headers:  2.6.34
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -fomit-frame-pointer -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/bacula /opt/openfire/resources/security/"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo"
CXXFLAGS="-O2 -fomit-frame-pointer -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests distlocks fixpackages news noinfo noman notitles parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch"
GENTOO_MIRRORS="http://ftp.ds.karen.hj.se/gentoo/ ftp://ftp.ing.umu.se/linux/gentoo/ ftp://ftp.ds.karen.hj.se/gentoo/ ftp://ftp.klid.dk/gentoo/ ftp://ftp.free.fr/mirrors/ftp.gentoo.org/"
LANG="ru_RU.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages/amd64"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from=/etc/portage/rsync_excludes"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage/layman/vmware /usr/local/portage"
SYNC="rsync://192.168.0.99/gentoo-portage"
USE="acl amd64 apache2 bash-completion berkdb bzip2 caps cli cxx dri gdbm gnutls gpm iconv ipv6 kerberos logrotate mmx modules mudflap multilib ncurses nptl nptlonly openmp pam pcre perl pppd python readline reflection sasl session sse sse2 ssl sysfs syslog tcpd threads truetype unicode vim-syntax xattr xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias cgi" ELIBC="glibc" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="radeon nv vesa vga fbdev" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS
Comment 1 Jeroen Roovers (RETIRED) gentoo-dev 2010-09-24 14:40:41 UTC
And what if you ran `gcc-config 1' after merging it?
Comment 2 Hodur 2010-09-24 14:59:14 UTC
(In reply to comment #1)
> And what if you ran `gcc-config 1' after merging it?
> 

> g++
gcc-config: error: could not run/locate 'g++'
> gcc-config 1
 * Switching native-compiler to x86_64-pc-linux-gnu-4.4.4 ...             [ ok ]
> g++
gcc-config: error: could not run/locate 'g++'
Comment 3 Jeroen Roovers (RETIRED) gentoo-dev 2010-09-24 15:11:12 UTC
You forgot to source /etc/profile.
Comment 4 Jeroen Roovers (RETIRED) gentoo-dev 2010-09-24 15:12:54 UTC
astrid ~ # gcc-config 2
 * Switching cross-compiler to hppa2.0-unknown-linux-gnu-4.4.3 ...
>>> Regenerating /etc/ld.so.cache...                                             [ ok ]

 * If you intend to use the gcc from the new profile in an already
 * running shell, please remember to do:

 *   # source /etc/profile

Using the same version of gcc-config, I get the above output. How come yours didn't show up?
Comment 5 Hodur 2010-09-24 16:01:43 UTC
This is pretty useless IMHO, because /usr/x86_64-pc-linux-gnu/gcc-bin/4.4.4/x86_64-pc-linux-gnu-g++ is just empty file in gcc package
 
> gcc-config 1
 * Switching native-compiler to x86_64-pc-linux-gnu-4.4.4 ...             [ ok ]
> source /etc/profile
> g++
gcc-config: error: could not run/locate 'g++'
> file /usr/x86_64-pc-linux-gnu/gcc-bin/4.4.4/x86_64-pc-linux-gnu-g++
/usr/x86_64-pc-linux-gnu/gcc-bin/4.4.4/x86_64-pc-linux-gnu-g++: empty
Comment 6 SpanKY gentoo-dev 2010-09-24 18:42:54 UTC
ive seen this on my systems before too.  but i dont see how it's a gcc problem.  maybe a tar issue, but probably a quickpkg.  probably related to the fact that the g++ binary is a hardlink to the c++ binary.

$ PKGDIR=$PWD/pkg quickpkg sys-devel/gcc
$ tar tvf pkg/sys-devel/gcc-4.4.3-r2.tbz2 | grep 'gnu-[gc]++$' | grep -v ^l
-rwxr-xr-x root/root    246944 2010-05-26 15:14 usr/x86_64-pc-linux-gnu/gcc-bin/4.4.3/x86_64-pc-linux-gnu-c++
-rwxr-xr-x root/root         0 2010-05-26 15:14 usr/x86_64-pc-linux-gnu/gcc-bin/4.4.3/x86_64-pc-linux-gnu-g++
Comment 7 Zac Medico gentoo-dev 2010-09-25 08:46:21 UTC
I wasn't able to reproduce this using either python-2.6.5-r2 or python-3.1.2-r3, and using tar-1.23-r2 for extraction. We do have a workaround for bug #185305 which breaks hardlinks, so hardlinks are supposed to be treated like any other files.
Comment 8 Guy 2010-10-30 20:48:26 UTC
I've run into this exact problem as well where quickpkg created the file "x86_64-pc-linux-gnu-g++" as zero bytes long.

I recently took the list of packages created by executing:

emerge -pet @world > world.text

and manipulated it in to a final list where I could execute:

cat final_list.text | xargs quickpkg --include-config=y

I create 1891 packages this way. I'm currently looking for other files which were created at zero bytes. If I find any, I will report them.

Not understanding that the zero length byte file was my problem in gcc, I requested some help here: http://forums.gentoo.org/viewtopic-t-850342-highlight-.html. With the number of things I tried conbined with google, I managed to finally find this reported bug.

Thanks all.
Comment 9 Guy 2010-10-30 21:11:12 UTC
This is just a guess based upon ancient behavioral experience with 'tar'.

Someone {definitely not me} may want to check the treatment of files which end in '++' when 'tar' is invoked through python in the quickpkg script. Depending on the shell and python interactions and the meta-character expansion of tar, the actual behavior may not be the expected behavior.

I _know_ this is way over my head.

Just saying.
Comment 10 Vasiliy Yeremeyev 2010-11-18 15:01:19 UTC
same with sys-apps/util-linux:
-rwxr-xr-x 1 root root     0 Oct 20 17:57 /sbin/fsck.ext2
-rwxr-xr-x 1 root root     0 Oct 20 17:57 /sbin/fsck.ext3
-rwxr-xr-x 1 root root     0 Oct 20 17:57 /sbin/fsck.ext4
-rwxr-xr-x 1 root root     0 Oct 20 17:57 /sbin/fsck.ext4dev
Comment 11 Jeroen Roovers (RETIRED) gentoo-dev 2010-12-05 21:05:21 UTC
*** Bug 347778 has been marked as a duplicate of this bug. ***
Comment 12 SpanKY gentoo-dev 2010-12-08 04:49:59 UTC
i doubt the file name is relevant.  the c++ binary works fine.

the e2fsprogs package is much nicer to test with considering gcc-4.5.1-r1 turns into a 150MiB bz2 file ;).

looking at `quickpkg`, it doesnt execute `tar` anywhere.  it uses the tarfile module (for both tar and bzip2).  so version of `tar` on host when creating doesnt matter.

as for extraction, just view the contents:
  tar tvvf e2fsprogs-1.41.12-r1.tbz2 | awk '$1 ~ /^-/ && $3 == "0"'
e2fsprogs shows a bunch of errors all centering around files with hardlinks.

reverting the workaround for Bug 185305 unbreaks quickpkg for me.  i suspect this workaround was merely masking an error that existing in older python's tarfile.py implementations, so i wonder why it wasnt reported to the python people instead of hacking up quickpkg.  the reporter at that time was using python-2.4.4 and the version of portage that he downgraded to was using the shell quickpkg which executed `tar | bzip2`.

if you add debugging to tar_contents(), you'll see that tar.gettarinfo is returning an object which says size == 0 (with linkname set to the file the thing is hardlinked to).  so i wouldnt be surprised if the following call to tar.addfile() with the type forced to REGTYPE causes the tarfile.py module to not output anything.

thus i suggest we revert the change added to Bug 185305 and if/when that issue comes up again, it can be researched properly and the actual bug located/fixed.
Comment 13 Zac Medico gentoo-dev 2010-12-08 05:29:47 UTC
(In reply to comment #12)
> thus i suggest we revert the change added to Bug 185305 and if/when that issue
> comes up again, it can be researched properly and the actual bug located/fixed.

Done:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=4c47e5857515ea69150fb1d7731e4a6c88b4476e
Comment 14 Zac Medico gentoo-dev 2010-12-13 07:15:58 UTC
This is fixed in 2.2.0_alpha8, but I'll leave this bug open until it's in an unmasked release.
Comment 15 Zac Medico gentoo-dev 2010-12-21 18:57:13 UTC
This is fixed in 2.1.9.6.
Comment 16 Zac Medico gentoo-dev 2010-12-21 18:58:28 UTC
(In reply to comment #15)
> This is fixed in 2.1.9.6.

I meant 2.1.9.26.
Comment 17 Roger 2011-01-12 23:55:20 UTC
Created attachment 259662 [details]
Checks for zero byte files and attempts to lists or reinstalls as requested.

Checks for zero byte files and attempts to lists or reinstalls as requested.

Don't know if anybody else has scripted anything as of yet for this bug, but feel free to suit this sh/bash script for your needs.
Comment 18 Zac Medico gentoo-dev 2011-02-13 08:22:53 UTC
*** Bug 354697 has been marked as a duplicate of this bug. ***
Comment 19 Zac Medico gentoo-dev 2011-02-13 08:24:17 UTC
*** Bug 354697 has been marked as a duplicate of this bug. ***