Hello. I am cross compiling python and than using it for emerge on cross compiled system inside qemu. Let's imagine that current native container has python v3.6.9, we want to cross compile python v3.6.10. Build log is too big, I've uploaded it to github. https://gist.githubusercontent.com/andrew-aladev/f36e8e754278e4fad50ff13238326f7a/raw/245a72e8bd0fb3dc6e4185ec757b2d3ab59d5861/gistfile1.txt Please look: grep "3\.6\.9" gistfile1.txt Writing grammar tables to /usr/mips-unknown-linux-gnu/tmp/portage/dev-lang/python-3.6.10/image/usr/lib/python3.6/lib2to3/Grammar3.6.9.final.0.pickle >>> /usr/mips-unknown-linux-gnu/usr/lib/python3.6/lib2to3/Grammar3.6.9.final.0.pickle Than I've tried to use this container inside qemu, I've received the following error: > ACCESS DENIED open_wr: /usr/mips-unknown-linux-gnu/usr/lib/python3.6/lib2to3/Grammar3.6.10.final.0.pickle This bug is very simple: python v3.6.10 wants to write v3.6.10 pickle, because it doesn't exist. This issue is not related to sandbox. I saw issues on gentoo bugtracker about it. I want just to clarify - this is 100% cross compilation issue. I will try today to provide beautiful solution for it.
Issue looks easy, but it may require more work. I've created a patch for now. Use it if you are sure that there will be only one pickle per python.
Created attachment 602624 [details, diff] remove_python_version_from_pickle patch
Hello. I've received more problems with pickle generation, reviewed source code, invented better bike for solving it. So I can provide full description of this issue and reduce other developers time on debugging. Problem: 1. Martin v. Löwis <martin@v.loewis.de> introduced pickle generation Wed Mar 19 04:43:46 2008 commit https://github.com/python/cpython/commit/5e37baea8007cb64b65a180e4d6c80de292a8a4a#diff-c9bd4064884726c21716df7118dcaeb72a63cc8ddd49484c7f4676934b37d9bb 2. Martin provided code for generating of pickle file: head + tail + ".".join(map(str, sys.version_info)) + ".pickle". This code has never been changed and comes as is in 2021 year. Code means that current python interpreter uses its own version for creating a new pickle file for new cpython interpreter. It will work only when current python interpreter equals to new cpython interpreter. This code is broken by design. 3. We can see file "Lib/lib2to3/patcomp.py" in the commit above, today (2021) it is a part of "Lib/lib2to3/pygram.py". It comes with the following code: _GRAMMAR_FILE = ... "Grammar.txt" _PATTERN_GRAMMAR_FILE = ... "PatternGrammar.txt" 4. We can review step-by-step "load_grammar" real world usage from "Lib/lib2to3/pgen2/driver.py": driver.load_grammar(_GRAMMAR_FILE) driver.load_grammar(_PATTERN_GRAMMAR_FILE) def load_grammar(gt="Grammar.txt", gp=None gp = head + tail + ".".join(map(str, sys.version_info)) + ".pickle" if force or not _newer(gp, gt): logger.info("Generating grammar tables from %s", gt) def _newer(a, b): if not os.path.exists(a): return False This "code" comes as is today as a part of latest stable python 3.9.5. This workaround means that when pickle files doesn't exist - python will recreate it in runtime inside "lib/lib2to3" folder. This workaround will ruin your production if "lib/lib2to3" is readonly and pickle files are not inside. Everybody will try to use this workaround as a security hole. But it looks like nobody from cpython cares. Solution: 1. Apply provided remove_python_version_from_pickle.patch. 2. Cross compile new cpython. 3. Cross compile portage. 4. Chroot. 5. Run python -c "import lib2to3.pygram", it will create pickles without version postfix, stop python from creating junk in runtime, fix sandbox permission issues temporaly. 6. Reinstall portage. 7. Reinstall cpython without patch, it will recreate pickles during installation. 8. Reinstall portage. This is just bike on the top of Martin's workaround. The best variant is to ignore cpython (and maybe python) completely, ignore software written in python where possible.
Created attachment 724423 [details, diff] patch v2
This continues to be an issue even through python 3.9. Is there a possibility to get this added to the official python-gentoo-patches?
I wonder if this is actually still an issue now that we build a mini CBUILD Python.
