Bug 300922 – Gentoo Prefix can't store .py[co] files in binpkgs.

• Product: Gentoo/Alt
• Status: RESOLVED WONTFIX

[Description Jeremy Olexa (darkside) (RETIRED) 2010-01-13 23:17:39 UTC]

We better start tracking this on on a bug tracker.

-pyc files store the EPREFIX of the host where it was built on.
-chpathtool does some "simple" heuristic to switch the build EPREFIX to the new target EPREFIX

-python (compileall.py) doesn't rebuild .pyc files if the mtime(?) doesn't change.
-python (seemingly) doesn't like when the pyc files are changed and *not* regenerated. You will see the message: "Bad marshalling data"

The solution is to not place .py[co] files in binpkgs
Commit http://overlays.gentoo.org/proj/alt/changeset/56120 "fixes" it for python. But introduces another problem (as pointed out by solar), if .pyc files are not in the image dir, then you will not be in CONTENTS/vdb and as such portage won't track them (and won't remove them).

This is a band-aid fix anyway because we were under that false impression that python was the only package that installed pyc files. That is simply not true.

%% find $EPREFIX/var/db/pkg -name CONTENTS | xargs grep \\.pyc|grep -v "dev-lang/python"|wc -l
25
%%

So, back to the drawing board. New Proposal:

If a binpkg is created, drop .pyc files in the tbz2 only. This can be achieved in portage in the function that builds the tbz2. But, I can't find it yet. Alternative, is to exlude it when it is extracted. Is that sane?

%% tar -tf test.tar 
test/
test/foo.s
test/foo.o
test/foo.c
%% tar --exclude=*.c -xvf test.tar 
test/
test/foo.s
test/foo.o
%%

[Comment 1 Jeremy Olexa (darkside) (RETIRED) 2010-01-13 23:18:50 UTC]

Created attachment 216429 [details]
--exclude=*.pyc --exclude=*.pyo

[Comment 2 Jeremy Olexa (darkside) (RETIRED) 2010-01-13 23:20:21 UTC]

Created attachment 216430 [details, diff]
--exclude=*.pyc --exclude=*.pyo

of course, a missing space.

[Comment 3 Fabian Groffen 2010-01-14 10:01:37 UTC]

I find it weird that python_mod_optimize is run /after/ installing in that case.  Because that means you don't get them in your VDB, while in fact they're still out there.

@python team: how is this supposed to work?  (the VDB contents and .py[co] files)

If this is all on purpose, I propose to either patch compileall and friends to have a force mode, used by python_mod_optimize, or if that is infeasible, to simply set mtime to Jan 1st 1970 or something, so make sure they will be regenerated.

[Comment 4 Jeremy Olexa (darkside) (RETIRED) 2010-01-14 15:51:05 UTC]

Comment on attachment 216430 [details, diff]
--exclude=*.pyc --exclude=*.pyo

It doesn't make sense to create this different install approach for binpkgs vs source packages. Patch is obsolete. Don't consider it.

[Comment 5 Arfrever Frehtes Taifersar Arahesis (RETIRED) 2010-01-14 17:40:20 UTC]

python_mod_optimize() already supports '-f' option.

  -f: force rebuild even if timestamps are up-to-date

[Comment 6 Fabian Groffen 2010-01-14 17:47:12 UTC]

ok, that's nice to know.

Arfrever how do you feel about some packages having .py[co] files in CONTENTS file/VDB, while others don't?  Should all have that, or should no package ever list it in VDB?

From another perspective, why doesn't python_mod_optimize generate it inside ${D} instead of on the live filesystem after installing?

[Comment 7 Michael Haubenwallner (RETIRED) 2010-01-14 21:19:47 UTC]

Maybe I'm wrong here, but IIRC python optimized binaries created in $D do not work in $ROOT. Iff this is really true, .py[co] in vdb are plain wrong.

[Comment 8 Fabian Groffen 2010-01-23 13:04:13 UTC]

prefix bits are irrelevant here, bug too polluted.