Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 704816 - dev-lang/python - Cross-compiled python installs wrong version of lib2to3/Grammar pickle
Summary: dev-lang/python - Cross-compiled python installs wrong version of lib2to3/Gra...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Python Gentoo Team
URL: https://bugs.python.org/issue39221
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-05 11:08 UTC by Andrew Aladjev
Modified: 2021-09-12 14:46 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
remove_python_version_from_pickle patch (remove_python_version_from_pickle.patch,468 bytes, patch)
2020-01-05 19:20 UTC, Andrew Aladjev
Details | Diff
patch v2 (remove_python_version_from_pickle.patch,591 bytes, patch)
2021-07-17 00:31 UTC, Andrew Aladjev
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Aladjev 2020-01-05 11:08:20 UTC
Hello. I am cross compiling python and than using it for emerge on cross compiled system inside qemu.

Let's imagine that current native container has python v3.6.9, we want to cross compile python v3.6.10. Build log is too big, I've uploaded it to github.

https://gist.githubusercontent.com/andrew-aladev/f36e8e754278e4fad50ff13238326f7a/raw/245a72e8bd0fb3dc6e4185ec757b2d3ab59d5861/gistfile1.txt

Please look:

grep "3\.6\.9" gistfile1.txt

Writing grammar tables to /usr/mips-unknown-linux-gnu/tmp/portage/dev-lang/python-3.6.10/image/usr/lib/python3.6/lib2to3/Grammar3.6.9.final.0.pickle
>>> /usr/mips-unknown-linux-gnu/usr/lib/python3.6/lib2to3/Grammar3.6.9.final.0.pickle

Than I've tried to use this container inside qemu, I've received the following error:

> ACCESS DENIED open_wr: /usr/mips-unknown-linux-gnu/usr/lib/python3.6/lib2to3/Grammar3.6.10.final.0.pickle

This bug is very simple: python v3.6.10 wants to write v3.6.10 pickle, because it doesn't exist. This issue is not related to sandbox.

I saw issues on gentoo bugtracker about it. I want just to clarify - this is 100% cross compilation issue. I will try today to provide beautiful solution for it.
Comment 1 Andrew Aladjev 2020-01-05 19:20:12 UTC
Issue looks easy, but it may require more work. I've created a patch for now. Use it if you are sure that there will be only one pickle per python.
Comment 2 Andrew Aladjev 2020-01-05 19:20:56 UTC
Created attachment 602624 [details, diff]
remove_python_version_from_pickle patch
Comment 3 Andrew Aladjev 2021-07-17 00:31:12 UTC
Hello. I've received more problems with pickle generation, reviewed source code, invented better bike for solving it. So I can provide full description of this issue and reduce other developers time on debugging.

Problem:

1. Martin v. Löwis <martin@v.loewis.de> introduced pickle generation Wed Mar 19 04:43:46 2008 commit https://github.com/python/cpython/commit/5e37baea8007cb64b65a180e4d6c80de292a8a4a#diff-c9bd4064884726c21716df7118dcaeb72a63cc8ddd49484c7f4676934b37d9bb

2. Martin provided code for generating of pickle file: head + tail + ".".join(map(str, sys.version_info)) + ".pickle". This code has never been changed and comes as is in 2021 year.

Code means that current python interpreter uses its own version for creating a new pickle file for new cpython interpreter. It will work only when current python interpreter equals to new cpython interpreter. This code is broken by design.

3. We can see file "Lib/lib2to3/patcomp.py" in the commit above, today (2021) it is a part of "Lib/lib2to3/pygram.py". It comes with the following code:

_GRAMMAR_FILE = ... "Grammar.txt"
_PATTERN_GRAMMAR_FILE = ... "PatternGrammar.txt"

4. We can review step-by-step "load_grammar" real world usage from "Lib/lib2to3/pgen2/driver.py":

driver.load_grammar(_GRAMMAR_FILE)
driver.load_grammar(_PATTERN_GRAMMAR_FILE)

def load_grammar(gt="Grammar.txt", gp=None
gp = head + tail + ".".join(map(str, sys.version_info)) + ".pickle"

if force or not _newer(gp, gt):
  logger.info("Generating grammar tables from %s", gt)

def _newer(a, b):
  if not os.path.exists(a):
    return False

This "code" comes as is today as a part of latest stable python 3.9.5. This workaround means that when pickle files doesn't exist - python will recreate it in runtime inside "lib/lib2to3" folder.

This workaround will ruin your production if "lib/lib2to3" is readonly and pickle files are not inside. Everybody will try to use this workaround as a security hole. But it looks like nobody from cpython cares.

Solution:

1. Apply provided remove_python_version_from_pickle.patch.
2. Cross compile new cpython.
3. Cross compile portage.
4. Chroot.
5. Run python -c "import lib2to3.pygram", it will create pickles without version postfix, stop python from creating junk in runtime, fix sandbox permission issues temporaly.
6. Reinstall portage.
7. Reinstall cpython without patch, it will recreate pickles during installation.
8. Reinstall portage.

This is just bike on the top of Martin's workaround. The best variant is to ignore cpython (and maybe python) completely, ignore software written in python where possible.
Comment 4 Andrew Aladjev 2021-07-17 00:31:34 UTC
Created attachment 724423 [details, diff]
patch v2
Comment 5 matoro 2021-09-12 14:46:22 UTC
This continues to be an issue even through python 3.9.  Is there a possibility to get this added to the official python-gentoo-patches?