Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 396585

Summary: python.eclass should include a way to clean py-compile files (Was: sys-devel/automake-1.11.2 breaks ebuilds that use the py-compile symlink idiom)
Product: Gentoo Linux Reporter: Michał Górny <mgorny>
Component: Current packagesAssignee: Python Gentoo Team <python>
Status: RESOLVED FIXED    
Severity: normal CC: adrian, blakawk, eras, gnome, pkuegle
Priority: Normal Keywords: PATCH
Version: unspecified   
Hardware: All   
OS: Linux   
URL: http://code.google.com/p/gentoo-progress/source/detail?r=1535
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 397335    
Attachments: The build log
Fix py-compile calling error.

Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-12-31 10:49:36 UTC
The report-python submakefile states:

	am__py_compile = PYTHON=$(PYTHON) $(SHELL) $(py_compile)
	py_compile = $(top_srcdir)/py-compile

While py-compile is a symlink to /bin/true on my system. Thus, 'make install' fails with:

	../../py-compile: ../../py-compile: cannot execute binary file
	make[3]: *** [install-pyreportexecPYTHON] Error 126
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-12-31 10:54:50 UTC
Created attachment 297443 [details]
The build log
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-12-31 10:58:32 UTC
Created attachment 297445 [details, diff]
Fix py-compile calling error.

Replace the /bin/true symlink with an equivalent shell script which
would work fine when called through $(SHELL).

Fixes:
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2011-12-31 11:02:51 UTC
And same goes for app-admin/abrt.
Comment 4 Alexandre Rostovtsev (RETIRED) gentoo-dev 2011-12-31 14:34:14 UTC
(In reply to comment #0)
> The report-python submakefile states:
> 
>     am__py_compile = PYTHON=$(PYTHON) $(SHELL) $(py_compile)

I have never seen the "am__py_compile" definition. What version of automake are you using?

(In reply to comment #3)
> And same goes for app-admin/abrt.

No. The same goes for at least 71 packages in the tree, and countless others in overlays. 'ln -s "$(type -P true)" py-compile' is a standard idiom in python-related ebuilds :/
Comment 5 Daniel Moos 2011-12-31 14:47:50 UTC
I've hit the same problem for pygobject.  

(In reply to comment #4)
> I have never seen the "am__py_compile" definition. What version of automake are
> you using?

~ # automake --version
automake (GNU automake) 1.11.2
Comment 6 Pacho Ramos gentoo-dev 2011-12-31 15:21:14 UTC
Does it work again with 1.11.1?
Comment 7 Alexandre Rostovtsev (RETIRED) gentoo-dev 2011-12-31 15:27:58 UTC
(In reply to comment #6)
> Does it work again with 1.11.1?

Yes, this is caused by a change introduced in automake-1.11.2.
http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00059.html
Comment 8 Alexandre Rostovtsev (RETIRED) gentoo-dev 2011-12-31 16:27:03 UTC
(In reply to comment #2)
> Created attachment 297445 [details, diff] [details, diff]
> Fix py-compile calling error.
> 
> Replace the /bin/true symlink with an equivalent shell script which
> would work fine when called through $(SHELL).

Earlier automake versions execute py-compile instead of running it through $(SHELL). Relying on an executable text file that does not start with '#!' to be interpreted as a shell script works on normal linux setups, but may not be very portable, so I think that "echo -e '#!'$(type -P sh)'\n:' > py-compile" should be used for safety (of course echo -e is not portable either, but ebuilds are guaranteed to be run by a modern version of bash that understands echo -e).

Fixed libreport, abrt, pygobject, and pygtk. Four packages done, 67 left :/
Comment 9 Gilles Dartiguelongue gentoo-dev 2011-12-31 17:03:21 UTC
It's been a long time since this idiom should have been taken care of through eclass, gnome2 or python or distutils maybe eutils, we need to stick something like: cat >"${S}/py-compile" <<EOF
#!/bin/sh
$(type -P true)
EOF

somewhere easy to get for ebuilds in tree.
Comment 10 Pacho Ramos gentoo-dev 2011-12-31 17:55:29 UTC
(In reply to comment #9)
> It's been a long time since this idiom should have been taken care of through
> eclass, gnome2 or python or distutils maybe eutils, we need to stick something
> like: cat >"${S}/py-compile" <<EOF
> #!/bin/sh
> $(type -P true)
> EOF
> 
> somewhere easy to get for ebuilds in tree.

Will CC python team to know if that code could be included in one of that eclasses
Comment 11 Alexandre Rostovtsev (RETIRED) gentoo-dev 2011-12-31 18:28:01 UTC
Actually, "echo -e '#!/bin/sh' > py-compile" is sufficient. Worrying about potential prefix issues from /bin/sh is silly since autoconf anyway puts #!/bin/sh at the top of every configure script. And there is no need to add a no-op command to a no-op script (tested with both bash and dash as /bin/sh).
Comment 12 Alexandre Rostovtsev (RETIRED) gentoo-dev 2011-12-31 18:32:52 UTC
(In reply to comment #11)
Or rather, "echo '#!/bin/sh' > py-compile"
Comment 13 Arfrever Frehtes Taifersar Arahesis 2012-01-01 00:11:38 UTC
I have implemented python_clean_py-compile_files() function:
http://code.google.com/p/gentoo-progress/source/detail?r=1535
Comment 14 iGentoo 2012-01-01 17:19:43 UTC
The same problem with app-i18n/ibus.

find /var/db/pkg/ /usr/portage/ -maxdepth 3 -type f -name '*.ebuild' | xargs grep -Hn --color 'py-compile'
...app-i18n/ibus...
Comment 15 Ian Abbott 2012-01-04 11:08:30 UTC
(In reply to comment #12)
> (In reply to comment #11)
> Or rather, "echo '#!/bin/sh' > py-compile"

Don't you also need to chmod +x py-compile for the benefit of automake versions before 1.11.2?
Comment 16 Pacho Ramos gentoo-dev 2012-01-04 11:17:23 UTC
We should also try to know what way to handle this is better, I have also seen things like:
> py-compile

instead of this, and I am unsure witch one should I use :/

Even better would be to have patch prepared by Arfrever in our python.eclass ;)
Comment 17 Ian Abbott 2012-01-04 11:25:05 UTC
(In reply to comment #15)
> (In reply to comment #12)
> > (In reply to comment #11)
> > Or rather, "echo '#!/bin/sh' > py-compile"
> 
> Don't you also need to chmod +x py-compile for the benefit of automake versions
> before 1.11.2?

Ignore that.  If py-compile already exists, its permission bits will be preserved.  However, it might be in trouble if the original py-compile is read-only and FEATURES="userpriv" is used.
Comment 18 Alexandre Rostovtsev (RETIRED) gentoo-dev 2012-01-04 23:30:07 UTC
(In reply to comment #16)
> We should also try to know what way to handle this is better, I have also seen
> things like:
> > py-compile
> 
> instead of this, and I am unsure witch one should I use :/

In light of the gentoo-dev discussion about possibly moving things from /bin to /usr/bin, hardcoding "/bin/sh" may be a bad idea; "echo > py-compile" is probably more future-safe.

> Even better would be to have patch prepared by Arfrever in our python.eclass ;)

Yes, this *really* needs to be in python.eclass.
Comment 19 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2012-01-04 23:38:41 UTC
(In reply to comment #18)
> (In reply to comment #16)
> > We should also try to know what way to handle this is better, I have also seen
> > things like:
> > > py-compile
> > 
> > instead of this, and I am unsure witch one should I use :/
> 
> In light of the gentoo-dev discussion about possibly moving things from /bin to
> /usr/bin, hardcoding "/bin/sh" may be a bad idea; "echo > py-compile" is
> probably more future-safe.

/bin/sh symlink will probably be eternal; yet you may want to use '#!/usr/bin/env sh' to be cool.
Comment 20 Arfrever Frehtes Taifersar Arahesis 2012-01-04 23:45:19 UTC
If a function from python.eclass is used in ebuilds, then in case of some future changes only this function will have to be updated.
Comment 21 Adam Jiang 2012-01-05 05:38:59 UTC
I have this problem with gst-python-1.10.22.

Could you explain the reason why should be py-compile point to a dummy script but not py-compile file provided by automake?
Comment 22 Alexandre Rostovtsev (RETIRED) gentoo-dev 2012-01-05 06:21:20 UTC
(In reply to comment #21)
> Could you explain the reason why should be py-compile point to a dummy script
> but not py-compile file provided by automake?

Before EAPI3, Gentoo's package management spec did not guarantee that mtimes of files would be preserved when merging them from /var/tmp/portage to the live filesystem, so it was absolutely necessary to disable py-compile and generate .pyc/.pyo files in pkg_postinst(); otherwise, the .pyc file could end up with an earlier mtime than the corresponding .py file, rendering it invalid for the python interpreter.

EAPI3 and higher guarantee that the relative ordering of mtimes is preserved. However, even in EAPI3 and EAPI4 ebuilds, .pyc files are still generated in pkg_postinst() as a matter of policy. I am not sure of the reason - perhaps it's simply tradition, or perhaps there are tools that depend on the fact that packages' .pyc/.pyo files are not included in VDB?
Comment 23 Mart Raudsepp gentoo-dev 2012-01-05 06:29:52 UTC
I believe they are considered to be concrete machine dependent, .pyo for O probably standing for "optimized" and all; hence not belonging in binary packages that are taken before postinst.
Last I checked the reality is, that all those really do are compress away the indentation a lot and strip away comments or so, nothing machine dependent; but that may have changed by now, or could change, or be different in non-default python interpreters (jython, etc)
Comment 24 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2012-01-05 08:28:34 UTC
(In reply to comment #22)
> it's simply tradition, or perhaps there are tools that depend on the fact that
> packages' .pyc/.pyo files are not included in VDB?

Python can recompile packages during run-time. Then VDB would not match anymore, and files won't be unmerged correctly.
Comment 25 Pacho Ramos gentoo-dev 2012-01-05 12:01:31 UTC
python team, please try to implement:
http://code.google.com/p/gentoo-progress/source/detail?r=1535

in our eclass to allow us to stop spreading a lot of ways to clean py-compile around the tree

Thanks
Comment 26 Mike Gilbert gentoo-dev 2012-01-21 19:49:34 UTC
I just copied Arfrever's function verbatim. I'm also of the opinion that /bin/sh isn't going away any time soon.
Comment 27 Raymond Jennings 2012-09-11 01:58:18 UTC
Considering that /bin/sh is part of many posix standards the first thing I'd do if it was missing would be to report it as an ugly bugly.