Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 489100

Summary: git-r3.eclass: support 'git rev-parse HEAD', 'git describe', and 'git log -1 HEAD'
Product: Gentoo Linux Reporter: Ulrich Müller <ulm>
Component: EclassesAssignee: Michał Górny <mgorny>
Status: RESOLVED FIXED    
Severity: normal CC: pesa, sci-chemistry
Priority: Normal Keywords: REGRESSION
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=912208
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: Patch 1.
Patch 2.
Patch 3.
Patch 4.

Description Ulrich Müller gentoo-dev 2013-10-23 07:03:35 UTC
Some packages test if they are being built from git, using commands like "git rev-parse HEAD" or "git describe". Examples are dev-vcs/git, app-admin/eselect, sys-apps/openrc, or Linux (the kernel). Presumably there are others.

Such tests used to work with git-2.eclass, but with git-r3.eclass they aren't any more.

Therefore, the eclass should offer a way to populate WORKDIR with "git clone" instead of "git checkout". IIUC, non-shallow fetches would also be required for this case.
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-10-23 08:40:32 UTC
(In reply to Ulrich Müller from comment #0)
> Some packages test if they are being built from git, using commands like
> "git rev-parse HEAD" or "git describe". Examples are dev-vcs/git,
> app-admin/eselect, sys-apps/openrc, or Linux (the kernel). Presumably there
> are others.

Just for the record, is Linux a valid example here? I don't think we're really interfacing it with ebuilds.

> Therefore, the eclass should offer a way to populate WORKDIR with "git
> clone" instead of "git checkout". IIUC, non-shallow fetches would also be
> required for this case.

Is this really worth non-shallow fetches? We're talking about increasing overhead seriously for the sake of pretty version string.
Comment 2 Ulrich Müller gentoo-dev 2013-10-24 07:59:54 UTC
(In reply to Michał Górny from comment #1)
> Just for the record, is Linux a valid example here? I don't think we're
> really interfacing it with ebuilds.

I've added Linux to show that detecting the git revision is not limited to marginal packages, but also used by very prominent ones. (It may also be worthwhile to note that for Linux the regular version number _isn't_ updated post release. From looking at a few other packages I'd conclude that this is the most common workflow.)

> > Therefore, the eclass should offer a way to populate WORKDIR with "git
> > clone" instead of "git checkout". IIUC, non-shallow fetches would also be
> > required for this case.
> 
> Is this really worth non-shallow fetches? We're talking about increasing
> overhead seriously for the sake of pretty version string.

How much overhead there is depends on the package, and AFAICS it will only occur when first fetching the package. Also this bug is not about a "pretty" version string, but about having accurate version information available. If this will save even a single enquiry in a (Gentoo or upstream) bug report, then it's worth quite some overhead, because developers' time is more valuable (by a large factor) than machine time.

However, I don't really care if fetches are shallow or not, as long as the end result is that EGIT_CHECKOUT_DIR contains a .git with a valid HEAD.
Comment 3 Ulrich Müller gentoo-dev 2013-10-24 08:08:11 UTC
(In reply to Ulrich Müller from comment #2)
> However, I don't really care if fetches are shallow or not, as long as the
> end result is that EGIT_CHECKOUT_DIR contains a .git with a valid HEAD.

Or, as an alternative, GIT_DIR could point to a valid repo.
Comment 4 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-10-24 14:33:15 UTC
(In reply to Ulrich Müller from comment #2)
> However, I don't really care if fetches are shallow or not, as long as the
> end result is that EGIT_CHECKOUT_DIR contains a .git with a valid HEAD.

Well, it'd be fairly easy to provide a suitable '.git' for getting a valid commit identifier. I guess I could work on that and re-consider if anyone needs something more from the repository.

(In reply to Ulrich Müller from comment #3)
> (In reply to Ulrich Müller from comment #2)
> > However, I don't really care if fetches are shallow or not, as long as the
> > end result is that EGIT_CHECKOUT_DIR contains a .git with a valid HEAD.
> 
> Or, as an alternative, GIT_DIR could point to a valid repo.

Many build systems assume '[ -d .git ]', so that wouldn't be a good idea.
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-10-26 20:30:44 UTC
Ok, tricking 'git rev-parse HEAD' is quite easy since it doesn't really verify that '.git/HEAD' is correct. Sadly, the same isn't true for 'git describe' since it really wants tags to be around.
Comment 6 Ulrich Müller gentoo-dev 2013-10-27 11:33:31 UTC
Something along the lines of the following should do the trick (I've used the variable names from git-r3_checkout here):

    cd "${out_dir}"
    cp -pPR "${GIT_DIR}" .git
    unset GIT_DIR
    git config core.bare false
    git checkout -f "${local_id}"/__main__

If a package really needs "git describe" then the ebuild could set EGIT_NONSHALLOW in addition.
Comment 7 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-10-27 12:44:20 UTC
Well, for 'rev-parse HEAD', I think I was able to get it as easily as:

  cd "${EGIT_CHECKOUT_DIR}"
  git init
  echo "${EGIT_VERSION}" > .git/HEAD

(git doesn't check the validity of whatever's in HEAD ATM)
Comment 8 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-10-27 13:44:59 UTC
   27 Oct 2013; Michał Górny <mgorny@gentoo.org> git-r3.eclass:
+  Create a fake ".git" directory inside the checkout to satisfy git rev-parse
+  uses in build systems. Bug #489100.

Please try now. Seems to work with eselect-9999 for me.
Comment 9 Ulrich Müller gentoo-dev 2013-10-27 14:42:04 UTC
(In reply to Michał Górny from comment #7)
> Well, for 'rev-parse HEAD', I think I was able to get it as easily as:
> 
>   cd "${EGIT_CHECKOUT_DIR}"
>   git init
>   echo "${EGIT_VERSION}" > .git/HEAD

Yeah, that hack will make "git rev-parse HEAD" work for the current git version, but presumably little else.


(In reply to Michał Górny from comment #8)
> Please try now. Seems to work with eselect-9999 for me.

Confirmed.
Comment 10 Ulrich Müller gentoo-dev 2014-01-08 17:58:32 UTC
Emacs upstream will switch from bzr to git, and they are going to use "git describe --tags" to obtain version information: http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00707.html
Comment 11 Ulrich Müller gentoo-dev 2014-02-10 20:56:30 UTC
Another problem: When trying to convert app-doc/pms-9999 to git-r3, I noticed that the vc bundle for TeX does "git log -1 HEAD". This also doesn't work with the current eclass.
Comment 12 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-02-10 21:11:57 UTC
I guess the eclass will need a major overhaul, starting with disabling shallow checkouts completely to satisfy all those mis-designs people love so much. I'll work on it when I have the time.
Comment 13 SpanKY gentoo-dev 2014-02-14 21:54:27 UTC
a faster example for testing is sys-devel/autoconf.  it needs tags to figure out its current version # (which is based on the last release).  other autotools use that to probe min requirements (like "i need autoconf-2.69 or newer").
Comment 14 Christoph Junghans (RETIRED) gentoo-dev 2014-02-18 17:22:48 UTC
I just want to add:
<https://github.com/gentoo-science/sci/commit/3870ba06f784b6f55712c08db6e7f75ba403426a#commitcomment-5393229>

sci-chemistry/gromacs runs: 
$ git rev-list -n1 "--pretty=format:%ci" HEAD
which fails with git-r3.eclass, too.
Comment 15 Ulrich Müller gentoo-dev 2014-02-18 17:48:55 UTC
(In reply to Christoph Junghans from comment #14)

Does the recipe from comment #6 work for you?
Comment 16 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-02-20 22:40:02 UTC
Can we assume that none of those weirdo tools will actually write (commit) to the repo?
Comment 17 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-02-20 23:32:27 UTC
Created attachment 370916 [details, diff]
Patch 1.
Comment 18 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-02-20 23:32:38 UTC
Created attachment 370918 [details, diff]
Patch 2.
Comment 19 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-02-20 23:32:51 UTC
Created attachment 370920 [details, diff]
Patch 3.
Comment 20 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-02-20 23:35:02 UTC
Could you test if after applying those three patches your issues are gone? I'm aware that it's still not the perfect solution but I'm working on it.

Now shallow clones are gone, everything is mirrored and cloned into the checkout dir. However, we checkout by commit rather than branch, so 'git branch' will not work yet.
Comment 21 Davide Pesavento gentoo-dev 2014-02-21 01:03:19 UTC
(In reply to Michał Górny from comment #16)
> Can we assume that none of those weirdo tools will actually write (commit)
> to the repo?

Yeah that would be a safe assumption I think.
Comment 22 Davide Pesavento gentoo-dev 2014-02-21 01:06:42 UTC
Comment on attachment 370916 [details, diff]
Patch 1.

>+	if [[ -e ${GIT_DIR}/shallow ]]; then
>+		einfo "${GIT_DIR} was a shallow clone, recreating..."
>+		rm -r "${GIT_DIR}" || die

I'd add -f too
Comment 23 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-02-21 16:12:32 UTC
Created attachment 370968 [details, diff]
Patch 4.

And this one also fixes 'git branch' output. I don't know if there's anything more to do here.
Comment 24 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-02-23 22:06:19 UTC
+  23 Feb 2014; Michał Górny <mgorny@gentoo.org> git-r3.eclass:
+  Use complete git clones, and clone them onto the checkout directory. This
+  makes it possible for build system to lookup all repository information as
+  requested in bug #489100. Remove shallow clone support since it would require
+  too much effort and make code hard to understand. Additionally obsoletes bug
+  #489100 and git-r3 part of bug #494934.