| Summary: | emirrordist: Does not add files to the mirror from the distfiles-local dir | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Jeremy Olexa (darkside) (RETIRED) <darkside> |
| Component: | Unclassified | Assignee: | Portage team <dev-portage> |
| Status: | CONFIRMED --- | ||
| Severity: | normal | ||
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
Is this a regression? From reading the code it looks like $distfiles_local is just another place to get distfiles from. This means if the files in there are not needed by any ebuild then they wont be copied to $distfiles. Or does "nothing happens" mean: "The file is not taken from $distfiles_local and is instead downloaded."? (In reply to Sebastian Luther (few) from comment #1) > Is this a regression? First time using the tool. Not sure. > > From reading the code it looks like $distfiles_local is just another place > to get distfiles from. This means if the files in there are not needed by > any ebuild then they wont be copied to $distfiles. That is incorrect logic. distfiles-local is a place where files get unconditionally added to the mirror but immediately scheduled for removal if no ebuild references it. (In reply to Jeremy Olexa (darkside) from comment #2) > That is incorrect logic. distfiles-local is a place where files get > unconditionally added to the mirror but immediately scheduled for removal if > no ebuild references it. Where do you get these expectations from? From more code reading I get that files that are in $distfiles_local are never removed from $distfiles (once they somehow ended up in there). Sorry, until yesterday I wasn't aware that this thing exists at all... (In reply to Sebastian Luther (few) from comment #3) > Where do you get these expectations from? emirrordist is modeled after "mirror-dist" - so the expectations are from existing behavior. emirrordist has never been ran in production (Gentoo Infra), so I'm describing showstopper(s) (there may be others in the near future) to getting it used in production. (In reply to Sebastian Luther (few) from comment #3) > From more code reading I get that files that are in $distfiles_local are > never removed from $distfiles (once they somehow ended up in there). Correct, they are never removed, until the deletion logic kicks in. The deletion logic is to delete ("delete" OR "recycle") the distfile after $deletion-delay if nothing references it. Upon something referencing the distfile (which would be an ebuild OR a whitelist entry) then it is not scheduled for deletion and kept indefinitely. So in other words, the lifecycle of a file (fileA) in distfiles-local is: 1) fileA is immediately added to $distfiles 2) fileA is immediately scheduled for deletion in $deletion-delay seconds 2a) fileA is kept indefinitely if there exists an ebuild that references it. 2b) fileA is kept indefinitely if there exists an whitelist entry that references it. 3) Once there are no references for fileA, goto step 2 I think this bug is just that 1) is not happening. I kinda figured out what the code is doing. emirrordist does NOT add files from distfiles-local, but once the files are in distfiles, then it will process them as normal (where the deletion logic kicks in, etc) (In reply to Sebastian Luther (few) from comment #3) > From more code reading I get that files that are in $distfiles_local are > never removed from $distfiles (once they somehow ended up in there). Correct, let's remove this "somehow" part and allow emirrordist to move the files there. |
Calling emirrordist as follows: /usr/bin/emirrordist \ --distfiles=/data/gmirror-distfiles/distfiles/ \ --delete --jobs=10 --repo=gentoo \ --failure-log=${DATADIR}/tmp/fail.log \ --success-log=${DATADIR}/tmp/success.log \ --scheduled-deletion-log=${DATADIR}/tmp/deletion.log \ --deletion-db=${DATADIR}/tmp/deletion-db.bdb \ --deletion-delay=1814400 \ --temp-dir=${DATADIR}/tmp/ \ --distfiles-local=${DATADIR}/distfiles-local \ --distfiles-db=${DATADIR}/tmp/distfile-db.bdb \ --whitelist-from=${DATADIR}/tmp/whitelist-master.txt \ --mirror However, when a new file is placed in ${DATADIR}/distfiles-local, it is not being placed in /data/gmirror-distfiles/distfiles/ - there is no log output present and -vv does not give any additional information. Can this be looked at?