Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 316937 - sys-cluster/mpich2: Verify and potentially improve Python-related code
Summary: sys-cluster/mpich2: Verify and potentially improve Python-related code
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All All
: High normal (vote)
Assignee: Justin Bronder (RETIRED)
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 308257
  Show dependency tree
 
Reported: 2010-04-24 18:20 UTC by Arfrever Frehtes Taifersar Arahesis (RETIRED)
Modified: 2010-07-08 12:59 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
mpich2-1.0.8.ebuild.patch (mpich2-1.0.8.ebuild.patch,1018 bytes, patch)
2010-06-14 20:02 UTC, Arfrever Frehtes Taifersar Arahesis (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-04-24 18:20:21 UTC
- If this package can be used as a library and installs Python modules
  (*.so or *.py) into site-packages directories, then consider supporting
  installation for multiple Python versions.
  Please read section "Types of packages" in documentation [1] to decide
  if this package can support installation for multiple Python versions.

- Ensure that the ebuilds do not use deprecated functions or variables.

- Please check if Python 3 is supported by this package. You can temporarily
  set Python 3 as main active version of Python to properly test if this
  package supports Python 3.

- If this package does not support Python 3:
  - Specify dependency on Python 2.
    You can use PYTHON_DEPEND helper variable, which should be set before
    inheriting of python eclass.
    Please read section "Specification of dependency on Python" in
    documentation [1].

  - If this package cannot support installation for multiple versions of
    Python, then set active version of Python using
    python_set_active_version().

  - Ensure that shebangs in installed scripts specify correct version of
    Python. If shebangs are too generic (e.g. '#!/usr/bin/python'), then you
    can use python_convert_shebangs() to convert shebangs.
    (Wrapper scripts generated by python_generate_wrapper_scripts() do not
    require any changes.)
    Please read section "Shebangs in installed scripts" in documentation [1].

  - To ensure that changes applied to the ebuilds are sufficient, please
    temporarily set Python 3 as main active version of Python and test if
    this package can be properly installed and if it works at run time.

Please see documentation [1] for more details.
[1] http://www.gentoo.org/proj/en/Python/developersguide.xml
Comment 1 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-06-14 20:02:03 UTC
Created attachment 235315 [details, diff]
mpich2-1.0.8.ebuild.patch

This patch only removes deprecation warning.

I will commit this patch, if you don't object in 1 week.
Comment 2 Justin Bronder (RETIRED) gentoo-dev 2010-06-14 20:39:05 UTC
(In reply to comment #1)
> Created an attachment (id=235315) [details]
> mpich2-1.0.8.ebuild.patch
> 
> This patch only removes deprecation warning.
> 
> I will commit this patch, if you don't object in 1 week.
> 

Looks fine to me, feel free to commit now.
Comment 3 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-06-14 21:44:09 UTC
Comment on attachment 235315 [details, diff]
mpich2-1.0.8.ebuild.patch

Committed.
Comment 4 Justin Bronder (RETIRED) gentoo-dev 2010-07-03 01:39:55 UTC
+  03 Jul 2010; Justin Bronder <jsbronder@gentoo.org> mpich2-1.2.1_p1.ebuild:
+  Update python handling (#316937). Cleanup deps. Automatically turn on
+  threads if mpi-threads are requested.

Please reopen if those changes were not sufficient.

Thanks again for patching the stable version.
Comment 5 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-03 02:03:06 UTC
PYTHON_DEPEND="*:2.4" means that each version >= 2.4, including 3.*, is accepted. You would need PYTHON_DEPEND="2:2.4", but 2.4 is the oldest slot currently present in the tree, so you can use PYTHON_DEPEND="2".

RESTRICT_PYTHON_ABIS is used only in ebuilds, which set SUPPORT_PYTHON_ABIS, so it can be removed from this ebuild.

python_mod_optimize() and python_mod_cleanup() should be used only for Python modules, not Python scripts. Pure-Python modules have .py extension. Python scripts don't need any extension. If Python scripts have .py extension, then corresponding .pyc / .pyo files are NOT used during execution of these scripts. .pyc / .pyo files are used only for imported modules.

$ echo 'print(1)' > test.py
$ touch -t 201007030000 test.py
$ python -m py_compile test.py
$ echo 'print(2)' > test.py
$ touch -t 201007030000 test.py
$ python test.py
2
$ python -c 'import test'
1
$

Please remove 'python_mod_optimize /usr/bin/' and 'python_mod_cleanup /usr/bin/'.
Comment 6 Justin Bronder (RETIRED) gentoo-dev 2010-07-03 02:13:07 UTC
(In reply to comment #5)
> python_mod_optimize() and python_mod_cleanup() should be used only for Python
> modules, not Python scripts. Pure-Python modules have .py extension. Python
> scripts don't need any extension. If Python scripts have .py extension, then
> corresponding .pyc / .pyo files are NOT used during execution of these scripts.
> .pyc / .pyo files are used only for imported modules.

You're assuming that upstream isn't importing from the files it stores in /usr/bin.  Sadly, they are for now, but the pm that does this is being dropped in 1.3.  Quick example:

$ ls /usr/lib/mpi/mpi-mpich2/usr/bin/mpdlib.py 
/usr/lib/mpi/mpi-mpich2/usr/bin/mpdlib.py
$ grep 'from mpdlib' /usr/lib/mpi/mpi-mpich2/usr/bin/*.py
/usr/lib/mpi/mpi-mpich2/usr/bin/mpdallexit.py:from mpdlib import mpd_set_my_id, mpd_get_my_username, mpd_uncaught_except_tb, \
/usr/lib/mpi/mpi-mpich2/usr/bin/mpdboot.py:from mpdlib   import mpd_set_my_id, mpd_get_my_username, mpd_same_ips, \
...etc.


> 
> Please remove 'python_mod_optimize /usr/bin/' and 'python_mod_cleanup
> /usr/bin/'.
> 

Given the above, does this still hold?
Comment 7 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-03 02:24:17 UTC
In such case, I suggest to call python_mod_optimize() / python_mod_cleanup() only for files belonging to sys-cluster/mpich2. Some other packages install scripts, not modules, into /usr/bin, and byte-compilation shouldn't concern these other packages.

If this package installs many .py files into /usr/bin and you don't want to explicitly specify all them in the ebuild, then you can use:

src_install() {
	...
	pushd "${D}" > /dev/null
	PYTHON_MODULES=(usr/bin/*.py)
	PYTHON_MODULES=("${PYTHON_MODULES[@]/#//}")
	popd > /dev/null
	...
}

pkg_postinst() {
	...
	python_mod_optimize "${PYTHON_MODULES[@]}"
}

pkg_postrm() {
	python_mod_cleanup "${PYTHON_MODULES[@]}"
}
Comment 8 Justin Bronder (RETIRED) gentoo-dev 2010-07-03 03:00:07 UTC
(In reply to comment #7)
> In such case, I suggest to call python_mod_optimize() / python_mod_cleanup()
> only for files belonging to sys-cluster/mpich2. Some other packages install
> scripts, not modules, into /usr/bin, and byte-compilation shouldn't concern
> these other packages.
> 
> If this package installs many .py files into /usr/bin and you don't want to
> explicitly specify all them in the ebuild, then you can use:
> 
> src_install() {
>         ...
>         pushd "${D}" > /dev/null
>         PYTHON_MODULES=(usr/bin/*.py)
>         PYTHON_MODULES=("${PYTHON_MODULES[@]/#//}")
>         popd > /dev/null
>         ...
> }
> 
> pkg_postinst() {
>         ...
>         python_mod_optimize "${PYTHON_MODULES[@]}"
> }
> 
> pkg_postrm() {
>         python_mod_cleanup "${PYTHON_MODULES[@]}"
> }
> 

That'll break binpkgs; I'll just list them :/

Thanks for helping again.
Comment 9 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-03 03:13:30 UTC
Binary packages contain saved environment.
Comment 10 Justin Bronder (RETIRED) gentoo-dev 2010-07-03 04:42:17 UTC
+  03 Jul 2010; Justin Bronder <jsbronder@gentoo.org> mpich2-1.2.1_p1.ebuild:
+  Isolate mpich2 specific python files in /usr/bin, simplify python deps.
+  Thanks to Arfrever, #316937.

Thanks.
Comment 11 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-03 04:49:53 UTC
python_mod_optimize() and python_mod_cleanup() accept multiple arguments.
Comment 12 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-08 12:59:27 UTC
(In reply to comment #11)
> python_mod_optimize() and python_mod_cleanup() accept multiple arguments.

I have simplified usage of python_mod_optimize() and python_mod_cleanup().