Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 491260 - git-r3.eclass has sandbox violations due to shallow clones of local repos
Summary: git-r3.eclass has sandbox violations due to shallow clones of local repos
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Eclasses (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Michał Górny
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-11-14 19:21 UTC by eroen
Modified: 2014-02-27 05:25 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description eroen 2013-11-14 19:21:47 UTC
When using git-r3 with a local repo (for example with the packagename_LIVE_REPO override), the clone fails with sandbox violations.

It fails with permission errors if using userpriv and portage user can't write to the local repo.

This does not happen if EGIT_NONSHALLOW is set or with git-2.eclass.


In the below example, userpriv is used, but the portage user has write access to the local repository. The ebuild has `inherit git-r3 distutils-r1` and does not define any phases.

falcon ga_wrapper # ga_wrapper_LIVE_REPO=~eroen/projects/git-annex-wrapper FEATURES="strict sandbox usersandbox" ebuild ga_wrapper-9999.ebuild install
Appending /home/eroen/repos/eroen-overlay to PORTDIR_OVERLAY...
>>> Existing ${T}/environment for 'ga_wrapper-9999' will be sourced. Run
>>> 'clean' to start with a fresh environment.
>>> Not marked as unpacked; recreating WORKDIR...
 * checking ebuild checksums ;-) ...                                                                          [ ok ]
 * checking miscfile checksums ;-) ...                                                                        [ ok ]
>>> Unpacking source...
 * Using ga_wrapper_LIVE_REPO, no support will be provided
 * Using ga_wrapper_LIVE_REPO, no support will be provided
 * Fetching HEAD from /home/eroen/projects/git-annex-wrapper ...
git fetch --no-tags /home/eroen/projects/git-annex-wrapper --depth 1 -f HEAD:dev-python/ga_wrapper/0/__main__
 * ACCESS DENIED:  mkstemp64:    /home/eroen/projects/git-annex-wrapper/.git/shallow_XXXXXX
fatal: Unable to create temporary file '/home/eroen/projects/git-annex-wrapper/.git/shallow_XXXXXX': Permission denied
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
 * ERROR: dev-python/ga_wrapper-9999::eroen failed (unpack phase):
 *   Unable to fetch from any of EGIT_REPO_URI
 * 
 * Call stack:
 *     ebuild.sh, line   93:  Called src_unpack
 *   environment, line 3984:  Called git-r3_src_unpack
 *   environment, line 2641:  Called git-r3_src_fetch
 *   environment, line 2635:  Called git-r3_fetch
 *   environment, line 2559:  Called die
 * The specific snippet of code:
 *       [[ -n ${success} ]] || die "Unable to fetch from any of EGIT_REPO_URI";
 * 
 * If you need support, post the output of `emerge --info '=dev-python/ga_wrapper-9999::eroen'`,
 * the complete build log and the output of `emerge -pqv '=dev-python/ga_wrapper-9999::eroen'`.
 * The complete build log is located at '/var/tmp/portage/dev-python/ga_wrapper-9999/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-python/ga_wrapper-9999/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-python/ga_wrapper-9999/work'
 * S: '/var/tmp/portage/dev-python/ga_wrapper-9999/work/ga_wrapper-9999'
 * --------------------------- ACCESS VIOLATION SUMMARY ---------------------------
 * LOG FILE: "/var/log/sandbox/sandbox-14066.log"
 * 
VERSION 1.0
FORMAT: F - Function called
FORMAT: S - Access Status
FORMAT: P - Path as passed to function
FORMAT: A - Absolute Path (not canonical)
FORMAT: R - Canonical Path
FORMAT: C - Command Line

F: mkstemp64
S: deny
P: shallow_XXXXXX
A: /home/eroen/projects/git-annex-wrapper/.git/shallow_XXXXXX
R: /home/eroen/projects/git-annex-wrapper/.git/shallow_XXXXXX
C: git-upload-pack /home/eroen/projects/git-annex-wrapper 
 * --------------------------------------------------------------------------------



falcon ga_wrapper # emerge --info
Portage 2.2.7 (default/linux/amd64/13.0, gcc-4.8.2, glibc-2.17, 3.10.7-e6-falcon0+ x86_64)
=================================================================
System uname: Linux-3.10.7-e6-falcon0+-x86_64-Intel-R-_Core-TM-2_Quad_CPU_@_2.40GHz-with-gentoo-2.2
KiB Mem:     4038272 total,   1763620 free
KiB Swap:    8388604 total,   8388604 free
Timestamp of tree: Thu, 14 Nov 2013 08:45:01 +0000
ld GNU ld (GNU Binutils) 2.23.2
distcc 3.1 x86_64-pc-linux-gnu [disabled]
ccache version 3.1.9 [disabled]
app-shells/bash:          4.2_p45
dev-java/java-config:     2.2.0
dev-lang/python:          2.7.5-r4, 3.2.5-r3, 3.3.2-r2
dev-util/ccache:          3.1.9-r3
dev-util/cmake:           2.8.12
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.12.4
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.9.6-r3, 1.11.6, 1.12.6, 1.13.4, 1.14
sys-devel/binutils:       2.23.2
sys-devel/gcc:            4.5.4, 4.6.4, 4.7.3-r1, 4.8.2
sys-devel/gcc-config:     1.8
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.11 (virtual/os-headers)
sys-libs/glibc:           2.17
Repositories: gentoo gamerlay eroen
ACCEPT_KEYWORDS="amd64 ~amd64 ~x86"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=core2 -O2 -pipe"
DISTDIR="/var/distfiles"
EMERGE_DEFAULT_OPTS="--jobs 3 --nospinner --quiet-build --with-bdeps=y --binpkg-respect-use y"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="/var/temdist/ /var/distfiles-occam/ http://mirrors.eu.kernel.org/gentoo/ http://mirrors.kernel.org/gentoo/ http://distfiles.gentoo.org http://www.ibiblio.org/pub/Linux/distributions/gentoo"
LANG="en_GB.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j4"
PKGDIR="/var/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/gamerlay /var/lib/layman/eroen"
SYNC="rsync://mirrors.eu.kernel.org/gentoo-portage/"
USE="X a52 aac aacs accessibility acl adns alsa amd64 ao atm berkdb bluray bs2b bzip2 cairo calendar canberra caps cdda cdio cdparanoia cdr cli corefonts cracklib crypt css cue cups curl cxx dbus dia dirac djvu dmapi dri dts dvd dvdnav dvi ebook eigen exif faac fam fbcondecor ffmpeg fftw fits flac fontconfig fortran fuse g3dvl gcrypt gd gdbm gif gimp git gmp gnutls gold gstreamer hdri highlight iconv icu id3tag idn imagemagick inotify iptc ipv6 jpeg jpeg2k kde kvm lcms lensfun libnotify loop-aes lqr lzma lzo mad man matroska midi mikmod mkv mms mmx mmxext mng mod modplug modules mp3 mp4 mpeg mpeg2 mpeg4 mpeg4pt2 mplayer mtp mudflap multilib musepack ncurses netlink nfs nls nouveau nptl ocr offensive ogg openal opencl openexr opengl opengtl openmp openvg optimization osmesa pam pango pcre pdf png pnm portaudio postscript pulseaudio qalculate qemu qt3support qt4 rar raw rdesktop readline real rtmp samba scanner schroedinger security session smp sndfile socks5 speex spice sse sse2 sse3 ssl ssse3 strong-optimization svg tbb tcpd theora threads tiff timidity tls truetype udev uml unicode usb usbredir v4l v4l2 vdpau virtualbox vnc vorbis wacom wavpack webp wmf x264 xattr xcb xcomposite xetex xfs xft xine xinerama xorg xps xrandr xv xvid xvmc zlib" ABI_X86="32 64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CURL_SSL="openssl" DRACUT_MODULES="biosdevname caps crypt crypt-gpg dash dmraid lvm mdraid syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" GRUB_PLATFORMS="efi-64 pc qemu" INPUT_DEVICES="evdev keyboard mouse synaptics wacom" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="nlpsolver presenter-console presenter-minimizer" LINGUAS="en en_US en_GB nb no nb_NO" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2 python3_3" RUBY_TARGETS="ruby20" SANE_BACKENDS="net plustek pnm" USERLAND="GNU" VIDEO_CARDS="nouveau nvidia vesa" XFCE_PLUGINS="menu" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="2.7 3.2 3.3"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-11-14 19:55:09 UTC
I will try to fix it this weekend. If you could try to provide a patch, it would make things faster. I think we can simply detect local repo URIs and disable shallow clones then (since git will hardlink stuff anyway).
Comment 2 Rick Farina (Zero_Chaos) gentoo-dev 2013-11-14 22:07:34 UTC
(In reply to Michał Górny from comment #1)
> I will try to fix it this weekend. If you could try to provide a patch, it
> would make things faster. I think we can simply detect local repo URIs and
> disable shallow clones then (since git will hardlink stuff anyway).

So if shallow clone doesn't work on googlecode, and doesn't work local, why on earth is it the default???  This is just silly honestly. Why is git trying to write to the source repo? Clearly it can't do this if the repo is not local so why is it doing it when it is?  I'm not sure if this is a git bug or a git-r3 bug but I can say either way these defaults suck, and if it doesn't get fixed to not break so much I'm going to undeprecate git-2 so we can have something that actually works by default.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-11-15 05:08:33 UTC
(In reply to Rick Farina (Zero_Chaos) from comment #2)
> (In reply to Michał Górny from comment #1)
> > I will try to fix it this weekend. If you could try to provide a patch, it
> > would make things faster. I think we can simply detect local repo URIs and
> > disable shallow clones then (since git will hardlink stuff anyway).
> 
> So if shallow clone doesn't work on googlecode, and doesn't work local, why
> on earth is it the default???  This is just silly honestly.

Please tell me, how many ebuilds in the tree use *local* repos? There are a few using Google Code but the majority *benefits* from it. This simply doesn't make sense.

> Why is git
> trying to write to the source repo? Clearly it can't do this if the repo is
> not local so why is it doing it when it is?  I'm not sure if this is a git
> bug or a git-r3 bug

I have no idea if it's a bug or feature. Git usually tries to hardlink local repos to the clones to save space, and it seems that '--depth 1' confuses it. I will look into it *when I have time*.

> but I can say either way these defaults suck, and if it
> doesn't get fixed to not break so much I'm going to undeprecate git-2 so we
> can have something that actually works by default.

Thanks for your professional opinion and a threat. Working for Gentoo is always a pleasure.
Comment 4 Rick Farina (Zero_Chaos) gentoo-dev 2013-11-15 18:38:16 UTC
(In reply to Michał Górny from comment #3)
> (In reply to Rick Farina (Zero_Chaos) from comment #2)
> > (In reply to Michał Górny from comment #1)
> > > I will try to fix it this weekend. If you could try to provide a patch, it
> > > would make things faster. I think we can simply detect local repo URIs and
> > > disable shallow clones then (since git will hardlink stuff anyway).
> > 
> > So if shallow clone doesn't work on googlecode, and doesn't work local, why
> > on earth is it the default???  This is just silly honestly.
> 
> Please tell me, how many ebuilds in the tree use *local* repos? There are a
> few using Google Code but the majority *benefits* from it. This simply
> doesn't make sense.

In the tree, almost none.  But things like chromeos and developers often use ebuilds with local repos.  git-r3 basically makes it impossible for a lot of developers right now, yet you were kind enough to deprecate git-r2.
> 
> > Why is git
> > trying to write to the source repo? Clearly it can't do this if the repo is
> > not local so why is it doing it when it is?  I'm not sure if this is a git
> > bug or a git-r3 bug
> 
> I have no idea if it's a bug or feature. Git usually tries to hardlink local
> repos to the clones to save space, and it seems that '--depth 1' confuses
> it. I will look into it *when I have time*.

If there are still bugs this significant, then git-r2 shouldn't be deprecated as CLEARLY git-r3 is unable to replicate it's feature set.
> 
> > but I can say either way these defaults suck, and if it
> > doesn't get fixed to not break so much I'm going to undeprecate git-2 so we
> > can have something that actually works by default.
> 
> Thanks for your professional opinion and a threat. Working for Gentoo is
> always a pleasure.

If you feel it is a threat to have a working gentoo then you are right, I'm directly threatening you personally.  Otherwise, like every other sane person, when you deprecate an old eclass with a new eclass please be sure it is actually better.  Thus far you have deprecated an eclass with almost no bugs with one that has caused nothing but problems.  Not only were you nice enough to deprecate a working eclass, you then proceeded to add in inherit for your broken eclass into the working eclass which in the process, obviously broke it as well.

This is not how gentoo works, the only reason you haven't been smacked down by QA is because there is no QA in gentoo.

In two weeks, as per gentoo policy, I'm going to undeprecate git-2 and remove the inherit for git-r3 unless the defaults in git-r3 work for local and googlecode.  This is not a threat, I'm not going to touch your shiny new eclass, I'm just going to unbreak the previously working eclass.  I would love it if git-r3 replaced git-2, but with your current attitude towards bug reports, I don't honestly think that is ever going to happen.
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-11-15 21:55:38 UTC
My attitude is not 'against bug reports' but against your attitude towards me. Believe me, there is a difference between reporting a bug and your monologues.
Comment 6 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-11-15 23:03:32 UTC
+  15 Nov 2013; Michał Górny <mgorny@gentoo.org> git-r3.eclass:
+  Use shallow clones for local repos. Bug #491260.

Please try now.
Comment 7 eroen 2013-11-16 00:21:12 UTC
Local repos in the configurations I could think off work well now. Thank you! :)
Comment 8 Nguyen Thai Ngoc Duy 2014-02-27 05:25:12 UTC
(In reply to Rick Farina (Zero_Chaos) from comment #2)
> Why is git
> trying to write to the source repo? Clearly it can't do this if the repo is
> not local so why is it doing it when it is?  I'm not sure if this is a git
> bug or a git-r3 bug (..snip..)

Git contributor here. I think I'm the one that made this change in Git. I believe we do not ever state that "for a fetch/clone request, git promises not to write anything in the repo". We could use $TMPDIR for these temporary files, but that would be the last resort, and I'm going to make that change for read-only repos. But that does not help this case. Fix your sandbox rules.