Created attachment 515290 [details] dev-python/ruamel_yaml-0.15.35.ebuild ruamel.yaml is a YAML 1.2 loader/dumper package for Python. It is a dependency for new version of app-misc/khard-0.12.1.
Thanks for this ebuild. I found there's a more elaborate ebuild in fkmclane overlay: https://data.gpo.zugaina.org/fkmclane/dev-python/ruamel-yaml/ruamel-yaml-0.15.35.ebuild I want to compare the merit of your ebuild vs fkmclane's before I decide what to put into the tree. Your opinion on this, Kristian, is appreciated. How much useful you find all the additional stuff in fkmclane's ebuild? Adding fkmclane to CC on this bug, I hope he doesn't mind.
I seem to have an eix misconfiguration, otherwise I should not have missed a preexisting ebuild. fkmclanes ebuild is obviously more elaborate. I have no idea why it does what it does. I myself missed a comment for the inclusion of the variable RUAMEL_NO_PIP_INSTALL_CHECK which is explained here https://bitbucket.org/ruamel/yaml/issues/114/setuppy-requirement-for-pip-prevents Assuming fkmclane knows what he is doing, I would suggest to take his ebuild. I try to test it later today.
The ordereddict dependency is only for python 2.7, which I did not need. I would say the preexiting ebuild shows much deeper insight into the package, and does not need the ugly variable hack for the install phase. So definitely go with fkmclanes ebuild(s).
Thanks for review Kristian. I have asked fkmclane to review his ebuild before we go with it, waiting for his reply on this now.
The ebuild should be ready to go assuming the test phase passes. I'll try this personally and double check repoman is happy when I'm at my Gentoo laptop later today.
I've updated my ebuild with (mostly) working tests and docs, but there is an issue I feel I need help with. Ryd, a documentation generation thing called from sphinx-build, does not find the `ruamel.std` package. Unsetting PYTHONPATH seems to make it work but I feel that doing `PYTHONPATH= emake -C _doc html` is not a correct solution. You can see the current ebuilds in my overlay branch ruamel-yaml. Any ideas on a proper solution? https://github.com/fkmclane/overlay/blob/ruamel-yaml/dev-python/ruamel-yaml/ruamel-yaml-0.15.35-r1.ebuild As an aside, would either of you know the proper way to change the SRC_URI location to an incompatible .tar.gz when bumping the revision? I just added -r1 to the distfile name with the plan to remove it whenever the next version of ruamel.yaml gets released.
Striving to get my hands on it, but haven't had a chance yet :) Is my understanding of doc building issue correct that it currently doesn't work unless the package is already installed? Then it seems a usual workaround is to set PYTHONPATH="${BUILD_DIR}/lib" or such. E.g. see https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/botocore/botocore-9999.ebuild#n52
Thanks to fkmclane, we have quite good ebuilds for its dependencies which are currently missing in the tree. What I don't like is amount of own "enhanced versions of standard packages" which Ruamel (author) uses in all his packages. Feels a bit like a cancer :) I was considering dropping "test", "doc" logic and python2_7 support to avoid pulling so many of Ruamel's packages into the tree. But now I see that having robust ruamel_yaml ebuild would be beneficial for other packages, e.g. mitmproxy (currently present in fkmclane and pentoo repositories).
There's a circular dependency - not just in ebuilds, but in upstream packages. ruamel-yaml: DEPEND: doc? (dev-python/ryd) ryd: RDEPEND on ruamel-yaml Due to this, I won't unclude doc building logics to central tree. Multi-step "bootstrapping" for such a minor package is not acceptable UX in my opinion.
Bear with me, one more unresolved issue in test phase. May be something stupid introduced by me: # USE=libyaml emerge -atv =app-misc/khard-0.12.2 --usepkg n --getbinpkg n * IMPORTANT: config file '/etc/portage/package.unmask/common' needs updating. * See the CONFIGURATION FILES and CONFIGURATION FILES UPDATE TOOLS * sections of the emerge man page to learn how to update config files. These are the packages that would be merged, in reverse order: Calculating dependencies... done! [ebuild UD ] app-misc/khard-0.12.2::gentoo [9999::gentoo] USE="-zsh-completion" PYTHON_TARGETS="python3_4 python3_6 -python3_5" 0 KiB [ebuild N ] dev-python/ruamel-yaml-0.15.35::gentoo USE="libyaml {test}" PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6" 0 KiB [ebuild N ] dev-python/ruamel-std-pathlib-0.6.3::gentoo USE="{test}" PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6" 0 KiB [ebuild N ] dev-python/pathlib2-2.3.0::gentoo PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6 -pypy" 34 KiB [ebuild N ] dev-python/scandir-1.6::gentoo USE="{-test}" PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6 -pypy" 29 KiB [ebuild N ] dev-python/pytest-3.2.2::gentoo USE="-doc {-test}" PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6 -pypy -pypy3" 768 KiB [ebuild N ] dev-python/py-1.4.34::gentoo USE="-doc {-test}" PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6 -pypy -pypy3" 190 KiB [ebuild N ] dev-python/unidecode-1.0.22::gentoo PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6 -pypy" 204 KiB Total: 8 packages (1 downgrade, 7 new), Size of downloads: 1,225 KiB Would you like to merge these packages? [Yes/No] >>> Verifying ebuild manifests >>> Emerging (1 of 8) dev-python/unidecode-1.0.22::gentoo >>> Emerging (2 of 8) dev-python/py-1.4.34::gentoo >>> Installing (2 of 8) dev-python/py-1.4.34::gentoo >>> Emerging (3 of 8) dev-python/scandir-1.6::gentoo >>> Installing (1 of 8) dev-python/unidecode-1.0.22::gentoo >>> Emerging (4 of 8) dev-python/pytest-3.2.2::gentoo >>> Installing (3 of 8) dev-python/scandir-1.6::gentoo >>> Emerging (5 of 8) dev-python/pathlib2-2.3.0::gentoo >>> Installing (4 of 8) dev-python/pytest-3.2.2::gentoo >>> Installing (5 of 8) dev-python/pathlib2-2.3.0::gentoo >>> Emerging (6 of 8) dev-python/ruamel-std-pathlib-0.6.3::gentoo >>> Failed to emerge dev-python/ruamel-std-pathlib-0.6.3, Log file: >>> '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/temp/build.log' >>> Jobs: 5 of 8 complete, 1 failed Load avg: 1.61, 1.21, 0.82 * Package: dev-python/ruamel-std-pathlib-0.6.3 * Repository: gentoo * Maintainer: fkmclane@gmail.com andrey_utkin@gentoo.org * USE: abi_x86_64 amd64 elibc_glibc kernel_linux python_targets_python2_7 python_targets_python3_4 python_targets_python3_5 python_targets_python3_6 test userland_GNU * FEATURES: preserve-libs sandbox selinux sesandbox test userpriv usersandbox >>> Unpacking source... >>> Unpacking ruamel-std-pathlib-0.6.3.tar.gz to /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work >>> Source unpacked in /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work >>> Preparing source in /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3 ... >>> Source prepared. >>> Configuring source in /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3 ... >>> Source configured. >>> Compiling source in /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3 ... * python2_7: running distutils-r1_run_phase distutils-r1_python_compile /usr/bin/python2.7 setup.py build sys.argv ['setup.py', 'build'] running build running build_py creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python2_7/lib/ruamel copying .ruamel/__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python2_7/lib/ruamel creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python2_7/lib/ruamel/std copying .ruamel/std/__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python2_7/lib/ruamel/std creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python2_7/lib/ruamel/std/pathlib copying ./tempdir.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python2_7/lib/ruamel/std/pathlib copying ./__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python2_7/lib/ruamel/std/pathlib copying ./LICENSE -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python2_7/lib/ruamel/std/pathlib warning: build_py: byte-compiling is disabled, skipping. * python3_4: running distutils-r1_run_phase distutils-r1_python_compile /usr/bin/python3.4 setup.py build sys.argv ['setup.py', 'build'] running build running build_py creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_4/lib/ruamel copying .ruamel/__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_4/lib/ruamel creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_4/lib/ruamel/std copying .ruamel/std/__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_4/lib/ruamel/std creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_4/lib/ruamel/std/pathlib copying ./tempdir.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_4/lib/ruamel/std/pathlib copying ./__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_4/lib/ruamel/std/pathlib copying ./LICENSE -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_4/lib/ruamel/std/pathlib warning: build_py: byte-compiling is disabled, skipping. * python3_5: running distutils-r1_run_phase distutils-r1_python_compile /usr/bin/python3.5 setup.py build sys.argv ['setup.py', 'build'] running build running build_py creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_5/lib/ruamel copying .ruamel/__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_5/lib/ruamel creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_5/lib/ruamel/std copying .ruamel/std/__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_5/lib/ruamel/std creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_5/lib/ruamel/std/pathlib copying ./tempdir.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_5/lib/ruamel/std/pathlib copying ./__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_5/lib/ruamel/std/pathlib copying ./LICENSE -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_5/lib/ruamel/std/pathlib warning: build_py: byte-compiling is disabled, skipping. * python3_6: running distutils-r1_run_phase distutils-r1_python_compile /usr/bin/python3.6 setup.py build sys.argv ['setup.py', 'build'] running build running build_py creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_6/lib/ruamel copying .ruamel/__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_6/lib/ruamel creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_6/lib/ruamel/std copying .ruamel/std/__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_6/lib/ruamel/std creating /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_6/lib/ruamel/std/pathlib copying ./tempdir.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_6/lib/ruamel/std/pathlib copying ./__init__.py -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_6/lib/ruamel/std/pathlib copying ./LICENSE -> /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3-python3_6/lib/ruamel/std/pathlib warning: build_py: byte-compiling is disabled, skipping. >>> Source compiled. >>> Test phase: dev-python/ruamel-std-pathlib-0.6.3 * python2_7: running distutils-r1_run_phase python_test =================================================== test session starts ==================================================== platform linux2 -- Python 2.7.14, pytest-3.2.2, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python2.7 cachedir: .cache rootdir: /var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3, inifile: collecting ... =============================================== no tests ran in 0.00 seconds =============================================== ERROR: file not found: _test/test_*.py * ERROR: dev-python/ruamel-std-pathlib-0.6.3::gentoo failed (test phase): * (no error message) * * Call stack: * ebuild.sh, line 124: Called src_test * environment, line 2639: Called distutils-r1_src_test * environment, line 901: Called _distutils-r1_run_foreach_impl 'python_test' * environment, line 363: Called python_foreach_impl 'distutils-r1_run_phase' 'python_test' * environment, line 2180: Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test' * environment, line 1555: Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test' * environment, line 1553: Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'python_test' * environment, line 575: Called distutils-r1_run_phase 'python_test' * environment, line 832: Called python_test * environment, line 2541: Called die * The specific snippet of code: * py.test -v _test/test_*.py || die * * If you need support, post the output of `emerge --info '=dev-python/ruamel-std-pathlib-0.6.3::gentoo'`, * the complete build log and the output of `emerge -pqv '=dev-python/ruamel-std-pathlib-0.6.3::gentoo'`. * The complete build log is located at '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/temp/environment'. * Working directory: '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3' * S: '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3' * Messages for package dev-python/ruamel-std-pathlib-0.6.3: * ERROR: dev-python/ruamel-std-pathlib-0.6.3::gentoo failed (test phase): * (no error message) * * Call stack: * ebuild.sh, line 124: Called src_test * environment, line 2639: Called distutils-r1_src_test * environment, line 901: Called _distutils-r1_run_foreach_impl 'python_test' * environment, line 363: Called python_foreach_impl 'distutils-r1_run_phase' 'python_test' * environment, line 2180: Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test' * environment, line 1555: Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test' * environment, line 1553: Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'python_test' * environment, line 575: Called distutils-r1_run_phase 'python_test' * environment, line 832: Called python_test * environment, line 2541: Called die * The specific snippet of code: * py.test -v _test/test_*.py || die * * If you need support, post the output of `emerge --info '=dev-python/ruamel-std-pathlib-0.6.3::gentoo'`, * the complete build log and the output of `emerge -pqv '=dev-python/ruamel-std-pathlib-0.6.3::gentoo'`. * The complete build log is located at '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/temp/environment'. * Working directory: '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3' * S: '/var/tmp/portage/dev-python/ruamel-std-pathlib-0.6.3/work/ruamel.std.pathlib-0.6.3'
Ugh, so pypi tarball doesn't include _test. Will switch back to bitbucket in SRC_URI.
Right now at failing tests of ruamel_yaml. Despite ruamel.std.pathlib is installed, test can't import it: >>> Test phase: dev-python/ruamel-yaml-0.15.35 * python2_7: running distutils-r1_run_phase python_test =================================================== test session starts ==================================================== platform linux2 -- Python 2.7.14, pytest-3.2.2, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python2.7 cachedir: .cache rootdir: /var/tmp/portage/dev-python/ruamel-yaml-0.15.35/work/ruamel-yaml-0.15.35, inifile: collecting ... collected 310 items / 1 errors ========================================================== ERRORS ========================================================== ________________________________________ ERROR collecting _test/test_api_change.py _________________________________________ ImportError while importing test module '/var/tmp/portage/dev-python/ruamel-yaml-0.15.35/work/ruamel-yaml-0.15.35/_test/test_api_change.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: _test/test_api_change.py:14: in <module> from ruamel.std.pathlib import Path E ImportError: No module named std.pathlib !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ================================================= 1 error in 1.74 seconds ================================================== * ERROR: dev-python/ruamel-yaml-0.15.35::gentoo failed (test phase):
The issue it because I removed the pth files without creating a namespace package. I'm fixing it now and pushing. See https://wiki.gentoo.org/wiki/Project:Python/Namespace_packages for reference.
Documentation generation works now too with the addition of dev-python/namespace-ruamel{,-std} to the overlay. I restricted tests on dev-python/ruamel-std-argparse and dev-python/ruamel-appconfig because they fail on python3_6 and python2_7 respectively. Repoman is also happy with the changes. Should be good to merge pending your review and testing.
Dropping python2_7 support (which is not needed for khard) allowed to reduce complexity (i.e. amount of stuff to add to the tree). See https://github.com/andrey-utkin/gentoo/commits/khard-0.12.2-v4 . Appears to work and passes tests. Going to commit already soon. Now I am thinking, do we need libyaml USE flag, or should we just put it to DEPEND unconditionally? with USE=-libyaml the package cannot be tested, which is bad. ruamel-yaml docs, as well as app-misc/khard-0.12.2, doesn't indicate how important libyaml support is for functionality of the package. Any insight here fkmclane?
Thanks to ruamel-yaml author who kindly elaborated on the actual situation (in a private mail to me and fkmclane), the confision was resolved. libyaml is not needed, compiled code has no special dependencies and can be built unconditionally. WIP change is https://github.com/andrey-utkin/gentoo/commit/5190232ab1be324439c3fba85a1ade404963aa0f Going to repomanize the commits and push soon.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=af34e161939acb123addba00dc65995944bfe006 commit af34e161939acb123addba00dc65995944bfe006 Author: Andrey Utkin <andrey_utkin@gentoo.org> AuthorDate: 2018-02-07 12:09:32 +0000 Commit: Andrey Utkin <andrey_utkin@gentoo.org> CommitDate: 2018-02-07 12:12:20 +0000 dev-python/ruamel-yaml: add new package A new dependency of app-misc/khard. Based on work by Foster McLane <fkmclane@gmail.com> Bug: https://bugs.gentoo.org/644994 Package-Manager: Portage-2.3.19, Repoman-2.3.6 dev-python/ruamel-yaml/Manifest | 1 + dev-python/ruamel-yaml/metadata.xml | 16 +++++++++ dev-python/ruamel-yaml/ruamel-yaml-0.15.35.ebuild | 44 +++++++++++++++++++++++ 3 files changed, 61 insertions(+) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a64be65cfe9aec2718c18d16a8185d5a5f5114e2 commit a64be65cfe9aec2718c18d16a8185d5a5f5114e2 Author: Andrey Utkin <andrey_utkin@gentoo.org> AuthorDate: 2018-01-27 02:03:19 +0000 Commit: Andrey Utkin <andrey_utkin@gentoo.org> CommitDate: 2018-02-07 12:12:19 +0000 dev-python/ruamel-std-pathlib: add new package Dependency of dev-python/ruamel-yaml, which is a new dependency of app-misc/khard and will be added to the tree in a following commit. Based on work by Foster McLane <fkmclane@gmail.com> Bug: https://bugs.gentoo.org/644994 Package-Manager: Portage-2.3.19, Repoman-2.3.6 dev-python/ruamel-std-pathlib/Manifest | 1 + dev-python/ruamel-std-pathlib/metadata.xml | 16 +++++++++ .../ruamel-std-pathlib-0.6.3.ebuild | 40 ++++++++++++++++++++++ 3 files changed, 57 insertions(+)}
Pushed! Thanks to Kristian, Foster and Anthon!
