Bug 68787 – Distfiles overlay directory ($DISTDIR_OVERLAY)

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Alexander Asanov 2004-10-25 01:00:26 UTC]

Not all distfiles are hosted on gentoo mirrors. For example distfiles for Sun JDK and distfiles needed by ebuilds in my local ebuilds ($PORTDIR_OVERLAY). And that's ok. But...

Now I have to download sych distfiles manyally and place them in $DISTDIR. Everything goes fine until I synchronize my distfiles with a mirror (my local mirorr via a network or (more often) with a local rsync HDD to HDD). If I pass --delete option to rsync, that synchronization removes my manually downloaded distfiles . If I dont pass such an option my distfiles are not removed, but I got outdated versions of distfiles and my $DISTDIR grows.

Solution is to make a directory for manually managed distfiles (for example $DISTFILES_OVERLAY in make.conf) and include it in search path for distfiles (before $DISTDIR).

Reproducible: Always
Steps to Reproduce:
1. Download some distfiles that are not hosted by gentoo mirrors. For example distfiles for Sun JDK and distfiles needed by ebuilds in my local ebuilds ($PORTDIR_OVERLAY)
2. Plase them in $DISTDIR as explaind by gentoo documentation.
3. Synchronize your distfiles with a mirror (via a network or with a local rsync HDD to HDD)
Actual Results:  
If ypu pass --delete option to rsync, that synchronization removes your manually
downloaded distfiles . If you don't pass such an option your distfiles are not
removed, but you get outdated versions of distfiles and your $DISTDIR grows.

Expected Results:  
To have directory for manually managed distfiles (for example $DISTFILES_OVERLAY
in make.conf default to /usr/local/distfiles I think) and include it in search
path for distfiles (before $DISTDIR).

[Comment 1 Jason Stubbs (RETIRED) 2004-10-25 01:04:30 UTC]

man 5 portage

Find the part on /etc/portage/mirrors and try the "local" option. Although not yet documented, file system paths are also supported for "local" mirrors.

[Comment 2 Alexander Asanov 2004-10-25 01:40:52 UTC]

That's not a solution of my problem. It's not even close.

My mirror is not in my local network. I want to be able to synchronize with a mirror eventyally with my HDD and then go home with it (mirror in a pocket :). I want to give my HDD to my friends so they could synchronize their distfiles with mine. And I want distfiles to be clean of outdated versions of packages. But I dont want my manually managed distfiles to be affected.

That's what I want. To have distfiles from gentoo mirror and not only. And distfiles from mirror should not interfer with other.

[Comment 3 Alexander Asanov 2004-10-25 01:49:43 UTC]

OK. I reread your message more carefully and I agree, that's a solution of my problem. But I still think, that making $DISTDIR_OVERLAY option in make.conf would be much better way to do sych a thing. It would be a way for people who even don't have such a problem to separate gentoo hosted distfiles from others. And it would be more intuitive.

[Comment 4 Alexander Asanov 2004-10-25 02:27:05 UTC]

BTW. The feature you described doesn't work. I added "local /usr/local/distfiles" to my /etc/portage/mirrors but portage still can't find my distfiles.

[Comment 5 Carsten Lohrke (RETIRED) 2004-10-25 06:47:03 UTC]

fyi, Jason: I have seen a larger thread in the german section of forums.g.o regarding this issue. It was more or less about a workaround for users without a fast internet connection. To be able to moun a cd/dvd with tarballs, having both distfile locations acessible at the same time without the need to copy all of the tarballs to the hdd, would be appreciated as it seems.

[Comment 6 Jason Stubbs (RETIRED) 2004-10-25 06:50:25 UTC]

localhost portage # ls /usr/portage/distfiles/portage-2.0.51-r2.tar.bz2 /home/distfiles/portage-2.0.51-r2.tar.bz2
ls: /home/distfiles/portage-2.0.51-r2.tar.bz2: No such file or directory
/usr/portage/distfiles/portage-2.0.51-r2.tar.bz2
localhost portage # cat /etc/portage/mirrors
local   /usr/portage/distfiles/
localhost portage # emerge info | grep DISTDIR
DISTDIR="/home/distfiles"
localhost portage # emerge -f portage
Calculating dependencies ...done!
>>> emerge (1 of 1) sys-apps/portage-2.0.51-r2 to /
Local mirror has file: portage-2.0.51-r2.tar.bz2
>>> Previously fetched file: portage-2.0.51-r2.tar.bz2 size ;-)
>>> Previously fetched file: portage-2.0.51-r2.tar.bz2 MD5 ;-)
>>> md5 src_uri ;-) portage-2.0.51-r2.tar.bz2


Seems to work fine here...

[Comment 7 Jason Stubbs (RETIRED) 2004-10-25 06:53:52 UTC]

Will leave this open for the time being unless somebody else disagrees...

[Comment 8 Nicholas Jones (RETIRED) 2004-10-25 10:34:19 UTC]

DISTDIR=/dir
and
"local"

cover all permutations that we are aware of.
I move to punt.

[Comment 9 Carsten Lohrke (RETIRED) 2004-10-25 10:49:24 UTC]

Oh, right. I never needed it and wasn't aware that local paths are accepted in /etc/portage/mirrors, too.

[Comment 10 Alexander Asanov 2004-10-25 19:55:27 UTC]

Oh. No. You offer me to download files from my HDD to my HDD. I should have two copys of the same file. That's not a solution.

Dostfiles overlay dir and local distfiles mirror are different features. Both are needed. First to be done, second to be documented. I think this if not very hard.

I'll reopen this bug.

[Comment 11 Marius Mauch (RETIRED) 2004-10-26 15:53:28 UTC]

It's intelligent enough to not copy the files.

[Comment 12 cilly 2008-07-17 13:37:20 UTC]

distfiles overlay directory is needed, i.e. if you run a git-overlay with distfiles in it, i.e. unreleased patches.

Please reopen this bug and add the FEATURE.

[Comment 13 Zac Medico 2008-07-17 14:19:16 UTC]

In >=portage-2.1.5 there's a new variable called PORTAGE_RO_DISTDIRS. See `man make.conf`. Maybe that works for you?

[Comment 14 cilly 2008-07-17 14:41:58 UTC]

Seems to be fixed in portage >=portage-2.2_rc1 with PORTAGE_RO_DISTDIRS.

Please change state to FIXED.

[Comment 15 Zac Medico 2008-07-17 14:56:03 UTC]

Thanks, reopening to mark fixed...