Bug 759643 – =dev-python/sphinxcontrib-asyncio-0.2.0-r3[doc] fails to install: Could not import extension sphinx.builders.epub3 (exception: No module named 'sphinxcontrib.serializinghtml')

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description Anon Emuss 2020-12-12 21:39:24 UTC]

Attempting to install =dev-python/sphinxcontrib-asyncio-0.2.0-r3[doc] fails to install, with this error:
-------- BEGIN ERROR --------
...
make -j4 -C docs html
make: Entering directory '/var/tmp/portage/dev-python/sphinxcontrib-asyncio-0.2.0-r3/work/sphinxcontrib-asyncio-0.2.0/docs'
sphinx-build -b html -d _build/doctrees   . _build/html
Running Sphinx v3.2.1

Extension error:
Could not import extension sphinx.builders.epub3 (exception: No module named 'sphinxcontrib.serializinghtml')
-------- END ERROR --------

Reproducible: Always

Steps to Reproduce:
1. USE=doc emerge -a1 sphinxcontrib-asyncio
Actual Results:  
Install fails (cut of error message given above).

Expected Results:  
Install succeeds.

Easy workaround:  Install with USE=-doc.

This fail happened on a re-install, which means it used to work.  It is not clear what changed to break it now.

In lieu of full system information, I will give a few basic versions, and include a description of the problem and some patches that appear to fix the problem on my system.  I was able to produce a similar issue on Ubuntu, but searching around I cannot find any mention of other people having similar problems.  I apologize if the problem is specific to my system, but the fact I was able to get a similar problem on Ubuntu suggests it is not.  If other people have a similar problem, then I hope this helps them.  If this is specific to my system, feel free to close this bug.

The problem appears to occur with =dev-lang/python-3.8.6 and =dev-python/sphinx-3.2.1.  I did not record the exact version, but I had a similar problem when reverting to a python-3.7 version, too.

[Comment 1 Anon Emuss 2020-12-12 21:59:39 UTC]

Created attachment 678202 [details]
Tarball that extracts to a demonstration of the problem, with print statements showing the steps

The issue appears to be with how python deals with certain types of packages that are split into multiple directories.  The issue appears to happen with the sphinxcontrib package, but not, for example, the xml package.  Perhaps the difference is that the xml package has an__init__.py, while the sphinxcontrib package does not, using eggs instead.

Anyways, the attached tarball sets up a directory structure that demonstrates the problem.  It requires dev-python/sphinxcontrib-serializinghtml to be installed, although that could probably be swapped for a different sphinxcontrib-* package.  Unzip the tarball.  It creates a local sphinxcontrib package with a single module much like sphinxcontrib-asyncio does.  The package and module files only contain print statements to make it more obvious when they are imported.  Change to the demo/bin/ directory, and run python demo.py.

The demo.py is mostly print statements explaining what it is doing.  It imports the sphinxcontrib package, sets up the module search path to include the local sphinxcontrib package, attempts to import the local sphinxcontrib module, then a system sphinxcontrib module (sphinxcontrib.serializinghtml), and then the local module again.  As it is, on my system, the first attempt at loading the local module fails, and the second succeeds, which seems like inconsistent behavior.

If you change the True to False on line 32 of demo.py, it no longer imports the system sphinxcontrib package first.  Then, at least on my system, the script is unable to find the system sphinxcontrib.serializinghtml.  This appears to be what happens with sphinxcontrib-asyncio.  When compiling the documentation, the distutils ebuild class adds a reference to the build directory to the PYTHONPATH executable, which causes python to find the sphinxcontrib-asyncio version of the sphinxcontrib package first.  After that, it does not appear to recognize that there exists another sphinxcontrib package, and so cannot find the system modules like sphinxcontrib.serializinghtml, which sphinx attempts to load (perhaps via other extensions).

The inconsistent behavior exhibited when line 32 of demo.py has True instead of False demonstrates how having a split package like this is difficult for python.  It is inconsistent enough that it appears to be a python bug, although I could not find anybody else with a similar problem, so I wonder if it is specific to my system.  You can observe the problem by replacing importlib._bootstrap._find_spec() with a shim function that prints the inputs, calls the original function with those inputs, and then prints the return value before returning.  With sphinxcontrib imported before the module search path is modified, it remembers where the system package is.  After the module search path is modified, attempting to import a sphinxcontrib module that is not in that directory fails, because it only checks that system path.  However, when importing a module that does exist, for some reason it does go through the module search path, and records that there are two locations (but only if the local version is earlier in the path than the system version).  Then, the second attempt to import the local module succeeds.

Whether or not this is a bug in python, it appears that depending on a split sphinxcontrib package does not work well.  Attempts to remove the local version from the module search path usually result in being unable to find the local module (unless it is already installed on the system, which will be used in my proposed fix below).  If anybody else has a similar problem, I am open to other fixes, but I will describe my fix here, and supply patches for them.  Basically, I think sphinxcontrib-asyncio needs to be installed in order to compile its own documentation (if you remove it from the extensions in docs/conf.py, the documentation compiles, but is missing some asyncio-specific portions).  Having it attempt to reference its own asyncio.py module in its own local sphinxcontrib package appears to cause conflicts with the system-installed sphinxcontrib package, so it seems best to require it be installed first with USE=-doc before with USE=doc, and then have it use the system sphinxcontrib package entirely, excluding all references to the version being installed.

With that in mind, I will post a patch that removes the references to the local version in docs/conf.py, and then a second patch that modifies the ebuild to remove the references to the local version from PYTHONPATH, as well as implement the circular dependence that this implies:  sphinxcontrib-asyncio[doc] will require sphinxcontrib-asyncio.  While I do not particularly care for adding a circular dependency, I am aware this is not the only one (as an example, dev-python/sphinx[doc] typically requires dev-python/jinja, and dev-python/jinja[doc] requires dev-python/sphinx).  As with the others, this may be broken by first installing sphinxcontrib-asyncio[-doc], and then sphinxcontrib-asyncio[doc].

[Comment 2 Anon Emuss 2020-12-12 22:01:02 UTC]

Created attachment 678205 [details, diff]
Remove parent directory from module path in docs

This patch removes additions to the module search path to prevent python from finding the local sphinxcontrib package, which causes problems as described above.

[Comment 3 Anon Emuss 2020-12-12 22:06:16 UTC]

Created attachment 678208 [details, diff]
Use system-installed version to compile sphinxcontrib-asyncio docs

This patch removes the distutils-added PYTHONPATH modification to prevent python from finding the local sphinxcontrib package, which causes problems as described above.  An unfortunate side effect is that, in order for the documentation to compile, sphinxcontrib-asyncio needs to already be installed.  This is handled by introducing a circular dependency where sphinxcontrib-asyncio[doc] depends on itself.  As with other similar circular dependencies with python packages and USE=doc, this may be resolved by first installing the package with USE=-doc, and then installing with USE=doc.  The dependency requires the same version of sphinxcontrib-asyncio to be installed; if this is too onerous, then perhaps the ``:='' may be removed from the dependency atom.

[Comment 4 Anon Emuss 2021-08-06 17:11:51 UTC]

This issue is no longer a problem for me, likely due to various updates since reporting.  Closing.