Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 498648 - emirrordist: Does not add files to the mirror from the distfiles-local dir
Summary: emirrordist: Does not add files to the mirror from the distfiles-local dir
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Unclassified (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-01-20 04:52 UTC by Jeremy Olexa (darkside) (RETIRED)
Modified: 2014-02-02 04:35 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2014-01-20 04:52:10 UTC
Calling emirrordist as follows:

/usr/bin/emirrordist \
        --distfiles=/data/gmirror-distfiles/distfiles/ \
        --delete --jobs=10 --repo=gentoo \
        --failure-log=${DATADIR}/tmp/fail.log \
        --success-log=${DATADIR}/tmp/success.log \
        --scheduled-deletion-log=${DATADIR}/tmp/deletion.log \
        --deletion-db=${DATADIR}/tmp/deletion-db.bdb \
        --deletion-delay=1814400 \
        --temp-dir=${DATADIR}/tmp/ \
        --distfiles-local=${DATADIR}/distfiles-local \
        --distfiles-db=${DATADIR}/tmp/distfile-db.bdb \
        --whitelist-from=${DATADIR}/tmp/whitelist-master.txt \
        --mirror

However, when a new file is placed in ${DATADIR}/distfiles-local, it is not being placed in /data/gmirror-distfiles/distfiles/ - there is no log output present and -vv does not give any additional information. Can this be looked at?
Comment 1 Sebastian Luther (few) 2014-01-20 07:33:51 UTC
Is this a regression?

From reading the code it looks like $distfiles_local is just another place to get distfiles from. This means if the files in there are not needed by any ebuild then they wont be copied to $distfiles.

Or does "nothing happens" mean: "The file is not taken from $distfiles_local and is instead downloaded."?
Comment 2 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2014-01-20 14:05:01 UTC
(In reply to Sebastian Luther (few) from comment #1)
> Is this a regression?

First time using the tool. Not sure.

> 
> From reading the code it looks like $distfiles_local is just another place
> to get distfiles from. This means if the files in there are not needed by
> any ebuild then they wont be copied to $distfiles.

That is incorrect logic. distfiles-local is a place where files get unconditionally added to the mirror but immediately scheduled for removal if no ebuild references it.
Comment 3 Sebastian Luther (few) 2014-01-20 19:51:01 UTC
(In reply to Jeremy Olexa (darkside) from comment #2)
> That is incorrect logic. distfiles-local is a place where files get
> unconditionally added to the mirror but immediately scheduled for removal if
> no ebuild references it.

Where do you get these expectations from? 

From more code reading I get that files that are in $distfiles_local are never removed from $distfiles (once they somehow ended up in there).

Sorry, until yesterday I wasn't aware that this thing exists at all...
Comment 4 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2014-01-20 21:04:47 UTC
(In reply to Sebastian Luther (few) from comment #3)
> Where do you get these expectations from? 

emirrordist is modeled after "mirror-dist" - so the expectations are from existing behavior.

emirrordist has never been ran in production (Gentoo Infra), so I'm describing showstopper(s) (there may be others in the near future) to getting it used in production.

(In reply to Sebastian Luther (few) from comment #3)
> From more code reading I get that files that are in $distfiles_local are
> never removed from $distfiles (once they somehow ended up in there).

Correct, they are never removed, until the deletion logic kicks in. The deletion logic is to delete ("delete" OR "recycle") the distfile after $deletion-delay if nothing references it. Upon something referencing the distfile (which would be an ebuild OR a whitelist entry) then it is not scheduled for deletion and kept indefinitely.

So in other words, the lifecycle of a file (fileA) in distfiles-local is:
1) fileA is immediately added to $distfiles
2) fileA is immediately scheduled for deletion in $deletion-delay seconds
  2a) fileA is kept indefinitely if there exists an ebuild that references it.
  2b) fileA is kept indefinitely if there exists an whitelist entry that references it.
3) Once there are no references for fileA, goto step 2

I think this bug is just that 1) is not happening.
Comment 5 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2014-02-02 04:35:14 UTC
I kinda figured out what the code is doing.

emirrordist does NOT add files from distfiles-local, but once the files are in distfiles, then it will process them as normal (where the deletion logic kicks in, etc)

(In reply to Sebastian Luther (few) from comment #3)
> From more code reading I get that files that are in $distfiles_local are
> never removed from $distfiles (once they somehow ended up in there).

Correct, let's remove this "somehow" part and allow emirrordist to move the files there.