Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 603946 - =dev-libs/libpcre2-10.22 fails src_test(): FAIL: RunTest when fr_FR defaults to UTF-8 instead of ISO-8859-1
Summary: =dev-libs/libpcre2-10.22 fails src_test(): FAIL: RunTest when fr_FR defaults ...
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Lars Wendler (Polynomial-C)
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-12-28 16:16 UTC by Jeroen Roovers
Modified: 2019-04-27 07:27 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
dev-libs:libpcre2-10.22:20161228-160234.log (dev-libs:libpcre2-10.22:20161228-160234.log,113.98 KB, text/plain)
2016-12-28 16:16 UTC, Jeroen Roovers
Details
test-suite.log (603946-test-suite.log,1.11 KB, text/plain)
2016-12-28 16:25 UTC, Jeroen Roovers
Details
603946-teststdout (603946-teststdout,2.74 KB, text/plain)
2016-12-28 16:30 UTC, Jeroen Roovers
Details
test-suite.log (test-suite.log,1.11 KB, text/plain)
2019-04-21 13:27 UTC, Jeroen Roovers
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jeroen Roovers gentoo-dev 2016-12-28 16:16:29 UTC
Created attachment 457630 [details]
dev-libs:libpcre2-10.22:20161228-160234.log

>>> Test phase: dev-libs/libpcre2-10.22
 * .hppa: running multilib-minimal_abi_src_test
make -j3   check
make  check-am
make[1]: Entering directory '/var/tmp/portage/dev-libs/libpcre2-10.22/work/pcre2-10.22-.hppa'
make
make[2]: Entering directory '/var/tmp/portage/dev-libs/libpcre2-10.22/work/pcre2-10.22-.hppa'
make  all-am
make[3]: Entering directory '/var/tmp/portage/dev-libs/libpcre2-10.22/work/pcre2-10.22-.hppa'
make[3]: Nothing to be done for 'all-am'.
make[3]: Leaving directory '/var/tmp/portage/dev-libs/libpcre2-10.22/work/pcre2-10.22-.hppa'
make[2]: Leaving directory '/var/tmp/portage/dev-libs/libpcre2-10.22/work/pcre2-10.22-.hppa'
make  check-TESTS
make[2]: Entering directory '/var/tmp/portage/dev-libs/libpcre2-10.22/work/pcre2-10.22-.hppa'
make[3]: Entering directory '/var/tmp/portage/dev-libs/libpcre2-10.22/work/pcre2-10.22-.hppa'
FAIL: RunTest
PASS: RunGrepTest
============================================================================
Testsuite summary for PCRE2 10.22
============================================================================
# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
============================================================================
Makefile:2534: recipe for target 'test-suite.log' failed
Comment 1 Jeroen Roovers gentoo-dev 2016-12-28 16:25:04 UTC
Created attachment 457634 [details]
test-suite.log
Comment 2 Jeroen Roovers gentoo-dev 2016-12-28 16:30:17 UTC
Created attachment 457636 [details]
603946-teststdout
Comment 3 Émeric Maschino 2017-03-28 19:44:03 UTC
Exactly same error here on ia64 with fr_FR locale.

     Émeric
Comment 4 Jeroen Roovers gentoo-dev 2017-03-29 03:19:29 UTC
So it's a stack direction issue?
Comment 5 Sergei Trofimovich gentoo-dev 2017-05-20 19:15:46 UTC
Does it still happen for you nowadays?

I tried to reproduce test failure on ia64 (guppy) and no tests failed:

$ FEATURES=test USE="bzip2 elibc_glibc hppa kernel_linux pcre16 readline recursion-limit unicode userland_GNU zlib" LANG=fr_FR emerge -v1 =dev-libs/libpcre2-10.22

PASS: RunTest
PASS: RunGrepTest
============================================================================
Testsuite summary for PCRE2 10.22
============================================================================
# TOTAL: 2
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
Comment 6 Émeric Maschino 2017-05-23 20:04:48 UTC
(In reply to Sergei Trofimovich from comment #5)
> Does it still happen for you nowadays?

Yes.

> I tried to reproduce test failure on ia64 (guppy) and no tests failed:
> 
> $ FEATURES=test USE="bzip2 elibc_glibc hppa kernel_linux pcre16 readline
> recursion-limit unicode userland_GNU zlib" LANG=fr_FR emerge -v1
> =dev-libs/libpcre2-10.22

Interesting, the USE flags on my system are quite different:

[ebuild   R   ~] dev-libs/libpcre2-10.22::gentoo  USE="bzip2 readline recursion-limit unicode zlib (-jit) -libedit -pcre16 -pcre32 -static-libs"

Naive question: what's the setting of ACCEPT_KEYWORDS on guppy? I'm running ia64, not ~ia64. Of course, some installed packages are ~ia64, but I'm sticking with ia64 globally.

     Émeric
Comment 7 Sergei Trofimovich gentoo-dev 2017-05-23 21:57:46 UTC
(In reply to Émeric Maschino from comment #6)
> (In reply to Sergei Trofimovich from comment #5)
> > Does it still happen for you nowadays?
> 
> Yes.
> 
> > I tried to reproduce test failure on ia64 (guppy) and no tests failed:
> > 
> > $ FEATURES=test USE="bzip2 elibc_glibc hppa kernel_linux pcre16 readline
> > recursion-limit unicode userland_GNU zlib" LANG=fr_FR emerge -v1
> > =dev-libs/libpcre2-10.22
> 
> Interesting, the USE flags on my system are quite different:
> 
> [ebuild   R   ~] dev-libs/libpcre2-10.22::gentoo  USE="bzip2 readline
> recursion-limit unicode zlib (-jit) -libedit -pcre16 -pcre32 -static-libs"
> 
> Naive question: what's the setting of ACCEPT_KEYWORDS on guppy? I'm running
> ia64, not ~ia64. Of course, some installed packages are ~ia64, but I'm
> sticking with ia64 globally.

I'm using all-stable chroot for this test. The only ~ia64 package is =dev-libs/libpcre2-10.22.

$ FEATURES=test USE="-* bzip2 readline recursion-limit unicode zlib" LANG=fr_FR emerge -v1 =dev-libs/libpcre2-10.22

The original test failure looks like 'fr_FR' locale
was treated like ASCII as if the locale would not be present at all.

Do you use locale filter through /etc/locale.gen?

Simplest locale test would be:
$ LANG=fr_FR date
mar. mai 23 21:56:34 UTC 2017
Comment 8 Émeric Maschino 2017-05-23 22:15:41 UTC
(In reply to Sergei Trofimovich from comment #7)
> 
> I'm using all-stable chroot for this test. The only ~ia64 package is
> =dev-libs/libpcre2-10.22.

OK, so we're "nearly" compatible, except that I'm not chrooted and that I have additional packages with ~ia64 keyword (basically, everything GNOME-related).

> $ FEATURES=test USE="-* bzip2 readline recursion-limit unicode zlib"
> LANG=fr_FR emerge -v1 =dev-libs/libpcre2-10.22
> 
> The original test failure looks like 'fr_FR' locale
> was treated like ASCII as if the locale would not be present at all.
> 
> Do you use locale filter through /etc/locale.gen?

I don't think so. I simply have:

fr_FR UTF-8

> Simplest locale test would be:
> $ LANG=fr_FR date
> mar. mai 23 21:56:34 UTC 2017

emeric@longspeak ~ $ LANG=fr_FR date
mer. mai 24 00:09:20 CEST 2017

And as a cross-test:

emeric@longspeak ~ $ date
mer. mai 24 00:09:34 CEST 2017

Seems fine to me.

     Émeric
Comment 9 Sergei Trofimovich gentoo-dev 2017-06-10 18:25:50 UTC
Yeah, looks fine.

Can you post your 'emerge --info =dev-libs/libpcre2-10.22' so i could diff it against guppy's system in hopes to get a clue where difference lies.
Comment 10 Sergei Trofimovich gentoo-dev 2017-06-10 18:43:13 UTC
> > Do you use locale filter through /etc/locale.gen?
> 
> I don't think so. I simply have:
> 
> fr_FR UTF-8

Aha! That's the trigger! Default 'fr_FR' locale is 1-byte: ISO-8859-1.
To trigger failure it's enough to have this file:

  $ cat /etc/locale.gen

  #fr_FR ISO-8859-1
  fr_FR UTF-8

Fails as bad on amd64 \o/
Comment 11 Sergei Trofimovich gentoo-dev 2017-06-10 19:27:22 UTC
(In reply to Sergei Trofimovich from comment #10)
> > > Do you use locale filter through /etc/locale.gen?
> > 
> > I don't think so. I simply have:
> > 
> > fr_FR UTF-8
> 
> Aha! That's the trigger! Default 'fr_FR' locale is 1-byte: ISO-8859-1.
> To trigger failure it's enough to have this file:
> 
>   $ cat /etc/locale.gen
> 
>   #fr_FR ISO-8859-1
>   fr_FR UTF-8
> 
> Fails as bad on amd64 \o/

Having chatted on #gentoo-dev this list looks like a canonical mapping
of locale without charset to a charset:

    https://github.com/lattera/glibc/blob/master/localedata/SUPPORTED#L206

    fr_FR.UTF-8/UTF-8
    fr_FR/ISO-8859-1
    fr_FR@euro/ISO-8859-15

I think it means glibc users can safely assume that 'fr_FR' is 'ISO-8859-1'-compatible. and you need to use 'fr_FR.UTF-8' to get UTF-8. IN local-gen syntax it would be:


   $ cat /etc/locale.gen
   fr_FR.UTF-8 UTF-8

and in LANG syntax it's LANG="fr_FR.UTF-8".
Comment 12 Émeric Maschino 2017-06-11 08:51:05 UTC
(In reply to Sergei Trofimovich from comment #11)
> > 
> > Aha! That's the trigger! Default 'fr_FR' locale is 1-byte: ISO-8859-1.
> > To trigger failure it's enough to have this file:
> > 
> >   $ cat /etc/locale.gen
> > 
> >   #fr_FR ISO-8859-1
> >   fr_FR UTF-8
> > 
> > Fails as bad on amd64 \o/

Well catched!

> Having chatted on #gentoo-dev this list looks like a canonical mapping
> of locale without charset to a charset:
> 
>     https://github.com/lattera/glibc/blob/master/localedata/SUPPORTED#L206
> 
>     fr_FR.UTF-8/UTF-8
>     fr_FR/ISO-8859-1
>     fr_FR@euro/ISO-8859-15

Fixed on my system right now.

> I think it means glibc users can safely assume that 'fr_FR' is
> 'ISO-8859-1'-compatible. and you need to use 'fr_FR.UTF-8' to get UTF-8. IN
> local-gen syntax it would be:
> 
> 
>    $ cat /etc/locale.gen
>    fr_FR.UTF-8 UTF-8

OK. I was mislead by the comment in /etc/locale.gen:

# Where <locale name> starts with a name as found in /usr/share/i18n/locales/.

as there are only fr_FR and fr_FR@euro, no UTF-8 reference there.

> and in LANG syntax it's LANG="fr_FR.UTF-8".

Actually, it seems it's rather LANG="fr_FR.utf8". This is at least how eselect locale {list|set} configured it in /etc/env.d/02locale.

Now that everything's correctly configured, I can confirm that dev-libs/libpcre2 emerges successfully with tests enabled.

So, in the end, was it really a problem with dev-libs/libpcre2 or simply because Jeroen and my systems were/are misconfigured? Jeroen, how are your locales configured?

     Émeric
Comment 13 Sergei Trofimovich gentoo-dev 2017-06-22 07:56:18 UTC
Reassigning to jer@ to clarify fr_FR charset on dev box.
Comment 14 Jeroen Roovers gentoo-dev 2019-04-21 13:27:15 UTC
Created attachment 573512 [details]
test-suite.log

# grep fr_ /etc/locale.gen
fr_FR ISO-8859-15
fr_FR UTF-8
fr_FR@euro ISO-8859-15

test-suite.log for dev-libs/libpcre2-10.33
Comment 15 Sergei Trofimovich gentoo-dev 2019-04-22 17:56:59 UTC
(In reply to Jeroen Roovers from comment #14)
> Created attachment 573512 [details]
> test-suite.log
> 
> # grep fr_ /etc/locale.gen
> fr_FR ISO-8859-15
> fr_FR UTF-8

Do you see it as a valid configuration?

I would expect a few problems:
- one locale overrides another mchanically in /usr/lib*/locale/
- 'fr_FR UTF-8' generates a 'fr_FR' locale in an unusual encoding (as the test suite shows)

> fr_FR@euro ISO-8859-15
> 
> test-suite.log for dev-libs/libpcre2-10.33
Comment 16 Jeroen Roovers gentoo-dev 2019-04-23 07:55:56 UTC
(In reply to Sergei Trofimovich from comment #15)
> (In reply to Jeroen Roovers from comment #14)
> > # grep fr_ /etc/locale.gen
> > fr_FR ISO-8859-15
> > fr_FR UTF-8
> 
> Do you see it as a valid configuration?

I am not sure what is valid any more. I would expect this to work because it has worked in the past. The comment in /etc/locale.gen does not say the first field should be unique, and it explains for both fields where to look them up. locale-gen finishes with no errors.
Comment 17 Sergei Trofimovich gentoo-dev 2019-04-27 07:27:05 UTC
Perhaps local-gen changed ordering (when parallel mode was added) of locale merging and now it exposed problem in this configuration.

toolchain@ will try to make locale-gen more clear about duplicate locales.