Bug 903709 – unicode related test failures when running on sourcehut builds with pypy3

• Product: Portage Development
• Status: UNCONFIRMED ---

[Description Oskari Pirhonen 2023-04-02 23:14:04 UTC]

I get the following error when running the Portage test suite using sourcehut builds with pypy3 [1]:

> ERROR: test_gpkg_non_ascii_path (portage.tests.gpkg.test_gpkg_path.test_gpkg_path_case)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/build/portage/build/lib/portage/tests/__init__.py", line 274, in run
>     testMethod()
>   File "/home/build/portage/build/lib/portage/tests/gpkg/test_gpkg_path.py", line 149, in test_gpkg_non_ascii_path
>     with open(os.path.join(orig_full_path, "test"), "wb") as test_file:
> UnicodeEncodeError: 'ascii' codec can't encode characters in position 22-25: ordinal not in range(128)

If I replace the call with:

    open(os.path.join(orig_full_path, "test").encode(), "wb")

then I get:

> ERROR: test_gpkg_non_ascii_path (portage.tests.gpkg.test_gpkg_path.test_gpkg_path_case)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/build/portage/lib/portage/tests/__init__.py", line 274, in run
>     testMethod()
>   File "/home/build/portage/lib/portage/tests/gpkg/test_gpkg_path.py", line 181, in test_gpkg_non_ascii_path
>     test_gpkg.decompress(os.path.join(tmpdir, "test"))
>   File "/home/build/portage/lib/portage/gpkg.py", line 1082, in decompress
>     image_safe.extractall(decompress_dir)
>   File "/home/build/portage/lib/portage/gpkg.py", line 722, in extractall
>     os.path.join(".", member.name) in self.file_list
>   File "/home/build/portage/lib/portage/__init__.py", line 278, in __call__
>     wrapped_args, wrapped_kwargs = self._process_args(args, kwargs)
>   File "/home/build/portage/lib/portage/__init__.py", line 263, in _process_args
>     wrapped_args = [
>   File "/home/build/portage/lib/portage/__init__.py", line 264, in <listcomp>
>     _unicode_encode(x, encoding=encoding, errors="strict") for x in args
>   File "/home/build/portage/lib/portage/__init__.py", line 230, in _unicode_encode
>     s = s.encode(encoding, errors)
> UnicodeEncodeError: 'utf-8' codec can't encode character '\udce4' in position 6: surrogates not allowed
> 
> non_ascii tmpdir = '/tmp/portage.runtests.psbtxtt1/tmp35i437m7'
> orig_full_path = '/tmp/portage.runtests.psbtxtt1/tmp35i437m7/orig/中文测试/日本テスト/한국어시험/'
> os.path.join(orig_full_path, 'test') = '/tmp/portage.runtests.psbtxtt1/tmp35i437m7/orig/中文测试/日本テスト/한국어시험/test'

The second one was not from an automated run, but SSH'ing into the build env and running the test manually.

build image: ubuntu/jammy
pypy3: 7.3.11+dfsg-1~ppa1~ubuntu22.04 from PyPy PPA [2]

This only happens with the pypy3 from the PPA, not when it's installed from Ubuntu's repos. The one from Ubuntu has its own problems such as segfaulting during manifest tests (likely due to this issue [3] which was fixed upstream last year). I suspect we're not hitting this on GH actions because, from what I can tell based on the run logs, they are providing their own set of Pythons when you use actions/setup-python.

[1]: https://builds.sr.ht/~xxc3nsoredxx/job/966723
[2]: https://launchpad.net/~pypy/+archive/ubuntu/ppa?field.series_filter=jammy
[3]: https://foss.heptapod.net/pypy/pypy/-/issues/3741