Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 117440 - 2.1_pre2 DISTDIR symlinking breaks cvs svn fetch
Summary: 2.1_pre2 DISTDIR symlinking breaks cvs svn fetch
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Configuration (show other bugs)
Hardware: All Linux
: High critical (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, REGRESSION
: 117526 117784 123644 (view as bug list)
Depends on:
Blocks: 115839
  Show dependency tree
 
Reported: 2006-01-02 05:27 UTC by Andrew Gaydenko
Modified: 2006-02-21 16:40 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
2.1_pre3-distdir-revert.patch (2.1_pre3-distdir-revert.patch,1.43 KB, patch)
2006-01-06 10:55 UTC, Zac Medico
Details | Diff
fixup.patch (patch,749 bytes, patch)
2006-01-07 16:15 UTC, Brian Harring (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Gaydenko 2006-01-02 05:27:12 UTC
Unfortunately, I can not specify which concrete syncing caused this problem. The problem is: all xxx-cvs.ebuild and xxx-svn.ebuild files fetch/update repositories to/in PORTAGE_TMPDIR instead of DISTDIR. You see, a fetching the whole repo tree discredits repo idea at all.

I have not found anything cvs/svn-src dirs specific in 'man make.conf'. portage-2.1_pre3-r1 is in use.
Comment 1 Ben Skeggs 2006-01-02 16:46:16 UTC
This appears to be an issue specific to portage-2.1_pre3-r1, if I reinstall portage-2.1_pre2 cvs ebuilds (specifically, x11-libs/evas) works correctly.  I've worked around the issue by adding these to my make.conf:
    ECVS_TOP_DIR="/usr/portage/distfiles/cvs-src"
    ESVN_STORE_DIR="/usr/portage/distfiles/svn-src"

Expected behaviour (<=portage-2.1_pre2):
    * Fetching CVS module e17/libs/evas into /usr/portage/distfiles/cvs-src ...

Behaviour with portage-2.1_pre3-r1:
    * Fetching CVS module e17/libs/evas into /var/tmp/portage/evas-9999/distdir/cvs-src ...
Comment 2 SpanKY gentoo-dev 2006-01-02 18:36:41 UTC
*** Bug 117526 has been marked as a duplicate of this bug. ***
Comment 3 Brian Harring (RETIRED) gentoo-dev 2006-01-04 13:37:33 UTC
*** Bug 117784 has been marked as a duplicate of this bug. ***
Comment 4 Brian Harring (RETIRED) gentoo-dev 2006-01-04 13:38:45 UTC
Suggested from ciaran is addition of another var that holds the actual distdir,
and modify eclasses in question to use that directory.  Would work, although
I'd like solar's input on any outstanding perm/security issues with our
existing cvs/svn handling- that and any suggestions of improvements we should
do since the opportunity exists now.
Comment 5 Zac Medico gentoo-dev 2006-01-04 16:30:41 UTC
Perhaps for an ugly interim hack, the eclasses can be modified to use `portageq envvar DISTDIR`. :)
Comment 6 solar (RETIRED) gentoo-dev 2006-01-05 09:54:57 UTC
(In reply to comment #4)
> Suggested from ciaran is addition of another var that holds the actual distdir,
> and modify eclasses in question to use that directory.  Would work, although



> I'd like solar's input on any outstanding perm/security issues with our
> existing cvs/svn handling- that and any suggestions of improvements we should
> do since the opportunity exists now.

Well even with the change to making cvs/svn/live ebuilds save files to another 
dir other than $DISTDIR wont solve the real problem of any user in the 
portage group can gain uid 0 pretty easy.

The default /var/tmp/portage itself is a big apart of the risk due to the ../ 
being 2777 (pick any of the RPATH bugs for a perfect example) and users & devs 
alike doing stuff like rm -rf /var/tmp/portage

Now for the DISTDIR handling in general. I'm kinda opposed to distdir mangling 
as it invalidates perfectly valid existing scripts that do stuff like post merge 
distdir cleanups. Another var would solve that but then poof we just added quite 
a bit of complexity to every existing script.

So whats the real gain of DISTDIR= symlinking? Does it need to be enabled for 
every EBUILD_PHASE or was it's goal to solve a specific problem with a specific 
phase?
Comment 7 Zac Medico gentoo-dev 2006-01-06 10:55:41 UTC
Created attachment 76363 [details, diff]
2.1_pre3-distdir-revert.patch

If you urgently need to use the cvs/svn eclasses, this well revert the "magic" distdir behavior.  It backs out the relevant hunk from here: http://sources.gentoo.org/viewcvs.py/portage/main/trunk/pym/portage.py?r1=2469&r2=2474
Comment 8 Brian Harring (RETIRED) gentoo-dev 2006-01-06 23:12:50 UTC
> Now for the DISTDIR handling in general. I'm kinda opposed to distdir mangling 
> as it invalidates perfectly valid existing scripts that do stuff like post
> merge distdir cleanups. Another var would solve that but then poof we just 
> added quite a bit of complexity to every existing script.
Not worried about breaking your script; design wise, your post distfile cleansing should occur in a different manner.
 
> So whats the real gain of DISTDIR= symlinking? Does it need to be enabled for 
> every EBUILD_PHASE or was it's goal to solve a specific problem with a
> specific phase?
Gain is there is no unstated access to DISTDIR by ebuilds.  Does a number on restrict="fetch" packages where devs forget to list a file in SRC_URI; does the same for normal packages, also.

It also lays the ground work for multiple distdirs, although that's minor.

Exempting cvs/svn reasons, not going to revert this change without further args against it- the QA benefit outweigh breaking some custom trickery (sorry :/ )
Comment 9 solar (RETIRED) gentoo-dev 2006-01-07 08:40:27 UTC
Brian,
Simply saying "it should occur in a different manner" is not enough nor  
a good way to work with others on the team. This should remain a
REGRESSION blocking any releases till such time as an alt variable is
provided when needing to access the real DISTDIR=
Comment 10 Brian Harring (RETIRED) gentoo-dev 2006-01-07 15:14:50 UTC
It *is* a regression, see the keywords.

No stable release till it's fixed, meanwhile the 2.1 releases are developmental releases- bugs do occur.

Not going to leave it broke, it will be fixed so just relax.  Right now the nasty var hack works, but I'd rather avoid that so I'm digging around for other stuff.
Comment 11 Brian Harring (RETIRED) gentoo-dev 2006-01-07 16:15:54 UTC
Created attachment 76492 [details, diff]
fixup.patch

Apply this manually for testing please, it adds PORTAGE_ACTUAL_DISTDIR var setting to portage.

already modified the eclasses in question to use this var if available, so it should revert the regression.
Comment 12 Andrew Gaydenko 2006-01-08 08:36:34 UTC
> Apply this manually for testing please, it adds PORTAGE_ACTUAL_DISTDIR var
> setting to portage.

I'm not portage developer. What must I test? OK, I can patch 'portage.py' file. And ... what? Must I modify 'make.conf'? xxx-svn/xxx-cvs ebuilds?

I'm ready, but I don't know what to do :-)
Comment 13 Robert Kruus 2006-01-09 08:42:03 UTC
seems to work here.  just have to mke sure those svn/cvs ebuilds/eclasses now use the PORTAGE_ACTUAL_DISTDIR where required
Comment 14 Harm Geerts 2006-01-09 18:36:34 UTC
works for me as well.

@Andrew: The cvs/svn ebuild you want to merge should contain a line similar to this:
ECVS_TOP_DIR="${DISTDIR}/cvs-src/${PN}"

If it doesn't then add it yourself, then replace the DISTDIR variable like this:
ECVS_TOP_DIR="${PORTAGE_ACTUAL_DISTDIR}/cvs-src/${PN}"
Comment 15 Brian Harring (RETIRED) gentoo-dev 2006-01-09 18:57:31 UTC
ECVS_TOP_DIR="${PORTAGE_ACTUAL_DISTDIR-${DISTDIR}}/cvs-src/"
^^^ use that instead.  ECVS_TOP_DIR doesn't specify the PN iirc, plus the var substitution I tagged in there is compatible both with older portages and with portages that have the symlinking in use.

All in all, only 4 mods in the tree for this- people shouldn't have to duplicate it themselves, but if they need to, raid from subversion.eclass and/or cvs.eclass for example of how to do it.

commited, meanwhile.
Comment 16 Andrew Gaydenko 2006-01-10 00:29:17 UTC
(In reply to comment #15)
> ECVS_TOP_DIR="${PORTAGE_ACTUAL_DISTDIR-${DISTDIR}}/cvs-src/"
> ^^^ use that instead.

I have tried to replace

ECVS_TOP_DIR="${DISTDIR}/cvs-src"

string with this one:

ECVS_TOP_DIR="${PORTAGE_ACTUAL_DISTDIR-${DISTDIR}}/cvs-src/"

Nevertheless, I have got:

>>> checksums files   ;-) files/digest-pcb-cvs-0.3
>>> Unpacking source...
 * Fetching CVS module pcb into /wrk/tmp/portage/portage/pcb-cvs-0.3/distdir/cvs-src/ ...


I have 

DISTDIR="/x/portage/distfiles"
PORTAGE_TMPDIR="/wrk/tmp/portage"

in my make.conf file. Have I missed something? Must I reopen the issue? Must I supply some additional info?
Comment 17 Brian Harring (RETIRED) gentoo-dev 2006-01-10 01:46:00 UTC
(In reply to comment #16)
> string with this one:
> ECVS_TOP_DIR="${PORTAGE_ACTUAL_DISTDIR-${DISTDIR}}/cvs-src/"
> 
> Nevertheless, I have got:
>  * Fetching CVS module pcb into
> /wrk/tmp/portage/portage/pcb-cvs-0.3/distdir/cvs-src/ ...
Assuming you're not running with the patch ;)
Comment 18 Andrew Gaydenko 2006-01-10 02:48:17 UTC
(In reply to comment #17)
> Assuming you're not running with the patch ;)

You are right. I was not waked up completely :-) With the patch I was able to do a work with one of my '-cvs' ebuild. 

Thanks!



Comment 19 Emil Beinroth 2006-01-12 10:43:52 UTC
${PORTAGE_ACTUAL_DISTDIR:-${DISTDIR}} would be the correct syntax.
Comment 20 Emil Beinroth 2006-01-12 14:10:07 UTC
I blame solar winds for my last comment.
Sorry about that ..
Comment 21 Zac Medico gentoo-dev 2006-02-21 16:40:53 UTC
*** Bug 123644 has been marked as a duplicate of this bug. ***