Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 106820 - >=ncurses-5.4-r6 with UTF8 locale and UTF8 console doesn't show national chars properly
Summary: >=ncurses-5.4-r6 with UTF8 locale and UTF8 console doesn't show national char...
Status: RESOLVED WONTFIX
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: x86 Linux
: High major (vote)
Assignee: Gentoo's Team for Core System packages
URL:
Whiteboard:
Keywords:
: 106821 133811 145080 (view as bug list)
Depends on:
Blocks:
 
Reported: 2005-09-21 12:23 UTC by Lukas Miczka
Modified: 2007-02-20 09:25 UTC (History)
10 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
a terrible mc (mc.png,26.61 KB, image/png)
2006-08-07 18:19 UTC, VirusCamp
Details
another terrible program, which uses ncurses this time (mp3blaster.png,28.73 KB, image/png)
2006-08-07 18:26 UTC, VirusCamp
Details
Ebuild for building only widechar ncurses, controlled by the USE flag "forceunicode" (ncurses-5.5-r9902.ebuild,5.71 KB, text/plain)
2006-08-22 05:03 UTC, Jozsef Daniel
Details
sys-libs/ncurses-5.5-r4.ebuild - ebuild with new USE flag forceunicode (ncurses-5.5-r4.ebuild,7.00 KB, text/plain)
2006-08-27 02:13 UTC, Jozsef Daniel
Details
ncurses-5.6-unicode.patch (ncurses-5.6-unicode.patch,1.45 KB, patch)
2007-02-18 13:31 UTC, Harald van Dijk (RETIRED)
Details | Diff
ncurses-5.6-unicode.patch (ncurses-5.6-unicode.patch,1.71 KB, patch)
2007-02-20 09:25 UTC, Harald van Dijk (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Lukas Miczka 2005-09-21 12:23:13 UTC
In my system (I'll attach emerge info later) this version of ncurses (5.4-r6 and
above) breaks national chars (I don't see polish hints) also english apps look
awful like kernel's menuconfig (I'll try to attach some screen shots when it
will be possible) so this is my config:
rc.conf - UNICODE="yes"
LC_ALL="pl_PL.UTF-8"
glibc configured with nptl and userlocales (I've tried without userlocales but
this won't work)
consolefont - lat2-16
problem: ncurses based apps look terrbile - no national hints (example: mc
kernel's menuconfig)
how I've solved my problem? - mask >=ncurses-5.4-r6 - version 5.4-r5 works great
Possible cause of this bug (I think):
23 Mar 2005; Mike Frysinger <vapier@gentoo.org> +ncurses-5.4-r6.ebuild:
Install both normal ncurses and wide ncurses with USE=unicode. Take a page
from RedHat to do this.
IMHO this brakes ncurses when you have USE="unicode" and want to use unicode in
console

Sorry for no emerge info and screen shots - I'll atach them as quick as it'll be
possible

Thanks Lukas

Reproducible: Always
Steps to Reproduce:
1.USE="unicode" emerge ncurses-5.4-r6 (or above)
2.set UNICODE in /etc/rc.conf to "yes"
3.set font "lat2-16" in /etc/conf.d/consolefont
4.set keymap "pl2" in /etc/conf.d/keymaps
5.set LC_ALL="pl_PL.UTF-8" in /etc/env.d/02locale
6.emerge mc with USE="ncurses"
7.run mc to see results

Actual Results:  
Downprgade to ncurses-5.4-r5 solves the problem

Expected Results:  
See polish hints and continous lines in ncurses apps with utf-8 enabled on
ncurses-5.4-r6 and above
Comment 1 Jakub Moc (RETIRED) gentoo-dev 2005-09-21 13:15:40 UTC
*** Bug 106821 has been marked as a duplicate of this bug. ***
Comment 2 SpanKY gentoo-dev 2005-09-21 13:42:22 UTC
sounds more like dialog wasnt emerged with USE=unicode

for the programs that dont display properly, run `ldd <blah>` and make sure they
link against libncursesw.so and not libncurses.so
Comment 3 Harald van Dijk (RETIRED) gentoo-dev 2005-09-21 15:11:15 UTC
That's kind of the point of this bug, if I'm understanding it right... Is there
a reason why a non-unicode version of ncurses is built and installed as the
default, regardless of any USE flags?
Comment 4 SpanKY gentoo-dev 2005-09-21 16:26:53 UTC
yes, so users can switch between unicode and non-unicode on the same system

this is pretty standard behavior, see Debian/Fedora
Comment 5 Harald van Dijk (RETIRED) gentoo-dev 2005-09-21 16:56:04 UTC
> yes, so users can switch between unicode and non-unicode on the same system

How can users do this? dialog is about the only program which lets you choose
which curses to link against; mostly everything else either forces
-lcurses/-lncurses, or autodetects -lncursesw without an override. The first two
examples from my own system: how can I (asking as a user) use a unicode-ncurses
version of alsamixer? How can I use a non-unicode-ncurses version of mutt after
installing libncursesw? Installing two versions hardly offers a choice for users.

> this is pretty standard behavior, see Debian/Fedora

I'm wondering whether it's a good choice, not whether it's a popular one.
Comment 6 SpanKY gentoo-dev 2005-09-21 19:37:51 UTC
(In reply to comment #5)
> > yes, so users can switch between unicode and non-unicode on the same system
> 
> How can users do this? dialog is about the only program which lets you choose
> which curses to link against; mostly everything else either forces
> -lcurses/-lncurses, or autodetects -lncursesw without an override.

then those packages are broken, fix them

i talked with nano upstream and now the configure script behavior is:
 --enable-unicode -> require libncursesw
 --disable-unicode -> require libncurses
 <no option> -> auto detect, defaulting to libncursesw but fallback to libncurses
Comment 7 Lukas Miczka 2005-09-22 04:52:47 UTC
I have bootstraped and then emerge -e system with USE="unicode" so dialog is
properly build but it's true that nano works OK seems that MC needs patch (there
is already 4.6.1 version of mc but not in the portage...) also kernel's
menuconfig seems to don't recognize unicode version of ncurses and maybe we'll
find more ncurses apps with this problem. Today I'll attach emerge info and
screenshots cause my system is almos build :)
Comment 8 Lukas Miczka 2005-09-22 13:20:53 UTC
So this is my emerge info

Portage 2.0.52-r1 (default-linux/x86/2005.1, gcc-3.4.4, glibc-2.3.5-r1,
2.6.12-love1 i686)
=================================================================
System uname: 2.6.12-love1 i686 AMD Athlon(tm)
Gentoo Base System version 1.12.0_pre8
dev-lang/python:     2.3.5, 2.4.1-r1
sys-apps/sandbox:    1.2.13
sys-devel/autoconf:  2.13, 2.59-r7
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6
sys-devel/binutils:  2.16.1
sys-devel/libtool:   1.5.20
virtual/os-headers:  2.6.11-r2
ACCEPT_KEYWORDS="x86 ~x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=athlon-xp -O3 -fomit-frame-pointer -ftracer
-fprefetch-loop-arrays -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.4/env
/usr/kde/3.4/share/config /usr/kde/3.4/shutdown /usr/kde/3/share/config
/usr/lib/X11/xkb /usr/share/config /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-march=athlon-xp -O3 -fomit-frame-pointer -ftracer
-fprefetch-loop-arrays -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig distlocks sandbox sfperms strict"
GENTOO_MIRRORS="ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/
ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo"
LC_ALL="pl_PL.UTF-8"
LINGUAS="pl"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 16bit 3dnow 3dnowext 7zip X aac acpi alsa apache2 apm arts avi
bash-completion berkdb bitmap-fonts browserplugin bzip2 cairo cgi cli
cpudetection crypt ctype cups curl dts dvd dvdread emboss encode esd exif fam
firefox flac foomaticdb fortran gd gdbm gif gimpprint gnome gphoto2 gpm
gstreamer gtk gtk2 gtkhtml imagemagick imlib ipv6 java javascript jce jpeg kde
libg++ libwww mad matroska mikmod mmx mmxext motif mozsvg mp3 mpeg musicbrainz
mysql ncurses nls nptl nsplugin ogg oggvorbis opengl pam pcre pdflib
pdo-external perl php png python qt quicktime readline real sdl session
simplexml soap spell sse ssl svg tcpd tiff truetype truetype-fonts type1-fonts
unicode usb utf8 vim-with-x vorbis win32codecs xine xml2 xsl xv xvid zlib
linguas_pl userland_GNU kernel_linux elibc_glibc"
Unset:  ASFLAGS, CTARGET, LANG, LDFLAGS
Comment 9 Harald van Dijk (RETIRED) gentoo-dev 2005-09-22 15:34:16 UTC
Thinking about it a bit, would it make sense to add an extra flag to ncurses to
allow both the old behaviour as well as the current? I doubt you (vapier) will
agree that one version is the right solution no matter what I say, but I hope
you can still agree that regardless of whether it's right, a lot of programs
would benefit from it, and making it optional should also keep you happy with
your system.
Comment 10 Lukas Miczka 2005-09-23 00:57:32 UTC
I think that is someone decides for UNICODE he want's to use it and doesn't need
non unicode version o ncurses as well - this is my opinion
It'll be easier to solve this problem modifing ncurses ebuild than makeing extra
patches for every broken ncurses app (like MC in my example) but maybe I'm wrong
In my system I've masked >=ncurses-5.4-r6 and I'm happy...
Comment 11 SpanKY gentoo-dev 2005-09-23 10:42:43 UTC
> It'll be easier to solve this problem modifing ncurses ebuild

you're not solving the problem, you're hacking around it

> than makeing extra
> patches for every broken ncurses app (like MC in my example)

if you have a package that reads input from the terminal and you forced it to
use unicode even though it's written only for traditional ASCII, wont you get
funky behavior ?  afterall, unicode takes up two bytes per character while ASCII
is just 1 ... so you'd have to worry about buffer sizes, parsing the characters
as unicode characters instead of two ASCII characters, etc...  it's been a while
since i wrote a ncurses app, but i know in nano it handles input differently
based upon whether it has UTF-8 support enabled
Comment 12 Harald van Dijk (RETIRED) gentoo-dev 2005-09-23 11:38:32 UTC
(In reply to comment #11)
> if you have a package that reads input from the terminal and you forced it to
> use unicode even though it's written only for traditional ASCII, wont you get
> funky behavior ?  afterall, unicode takes up two bytes per character while ASCII
> is just 1 ... so you'd have to worry about buffer sizes, parsing the characters
> as unicode characters instead of two ASCII characters, etc...  it's been a while
> since i wrote a ncurses app, but i know in nano it handles input differently
> based upon whether it has UTF-8 support enabled

In UTF-8 locales, input gets passed to programs as UTF-8, and output is
interpreted as UTF-8, with either version of ncurses. This means that with
either version, applications which ignore unrecognised keys and limit their
output to ASCII and standard curses special characters work, and applications
which make unreliable assumptions (see dev-util/rhide - except that I haven't
tried recent versions) break even with non-wide ncurses. The big difference from
a programmer's view is that wide curses makes available some extra functions
such as get_wch() to deal with wide characters internally, but it doesn't force
use of these.
Comment 13 Lukas Miczka 2005-09-24 04:28:25 UTC
(In reply to comment #11)
> > It'll be easier to solve this problem modifing ncurses ebuild
> 
> you're not solving the problem, you're hacking around it
> 
So you want to say that this problem is in my head? Cause I don't know what you
mean by talking that I'm hacking around it... sorry but I don't understand your
point of view
Comment 14 Thomas Dickey 2005-11-13 15:59:13 UTC
a screenshot would help, since I don't see enough information
here to guess what sort of problem.
Comment 15 Alexander Simonov 2005-11-21 11:15:44 UTC
I see we come back to old bug with unicode ncurses ....
It's bad ...

(In reply to comment #10)
> I think that is someone decides for UNICODE he want's to use it and doesn't 
need
> non unicode version o ncurses as well - this is my opinion
> It'll be easier to solve this problem modifing ncurses ebuild than makeing 
extra
> patches for every broken ncurses app (like MC in my example) but maybe I'm 
wrong

It's bad for updating non-unicode ncurses ...
But i think we must build unicode system by default... (IMHO)
or if user build ncurses with unicode check it and breake ebuild.

Comment 16 Thomas Dickey 2005-11-26 12:27:46 UTC
Upstream ncurses source is designed to allow ncurses (8-bit)
and ncursesw (wide-characters) to be installed without conflict.
Debian is doing that...
Comment 17 SpanKY gentoo-dev 2005-11-26 12:43:45 UTC
and that is what i changed the latest versions of ncurses to do ...

the people here though wish to install only ncursesw and just generate symlinks
for ncurses to the ncursesw versions

libncursesw.so
libncursesw.a
libncurses.so -> libncursesw.so
libncurses.a -> libncursesw.a
Comment 18 Jakub Moc (RETIRED) gentoo-dev 2006-05-19 14:35:26 UTC
*** Bug 133811 has been marked as a duplicate of this bug. ***
Comment 19 SpanKY gentoo-dev 2006-06-11 07:57:22 UTC
try ncurses-5.5-r2
Comment 20 Harald van Dijk (RETIRED) gentoo-dev 2006-06-11 12:42:25 UTC
Anything linking to lib(n)curses instead of lib(n)cursesw still gets a crippled ncurses with 5.5-r2, and this is still very visible when borders are drawn.
Comment 21 kavol 2006-06-20 07:49:10 UTC
hi,

I spent a day by recompiling glibc, playing with userlocales, localedef etc., to get MC working with UTF-8, finally resulting that it has to be some bug because on another system I have no problem

so turning to bugzilla, I have found this and consequently that the problem is that mc links against non-unicode version of ncurses

$ ldd /usr/bin/mc
...
        libncurses.so.5 => /lib/libncurses.so.5 (0xb7e46000)
...

oh great ...

from the user's point of view:

I want to use unicode only, there is no point in using anything else, and I do not want have installed two versions of the same without a reason

from the sysadmin's point of view:

* since UTF-8 can cope with ASCII transparently, there is no need for special version for the programs which do not support national characters

* "other distros do it that way" is not a reason - other distros do not have the USE flags, they have to support everything possible, but if I say USE="unicode", it means I want to build with unicode and not without ... if you really feel the need for a choice of both, then add a flag to build both (or allow a kind of unicode-only flag)

* as it was already pointed out, it is impossible (without heavy hacking) for the user to have a choice between unicode and non-unicode, so this is not an argument too
Comment 22 VirusCamp 2006-08-07 18:19:21 UTC
Created attachment 93712 [details]
a terrible mc

ncurses-5.5-r3 still has this problem.

localhost ~ # ldd /usr/bin/mc
        linux-gate.so.1 =>  (0xffffe000)
        libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0xb7f70000)
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb7ee9000)
        libext2fs.so.2 => /lib/libext2fs.so.2 (0xb7ecf000)
        libcom_err.so.2 => /lib/libcom_err.so.2 (0xb7ecc000)
        libgpm.so.1 => /lib/libgpm.so.1 (0xb7ec6000)
        libslang-utf8.so.1 => /usr/lib/libslang-utf8.so.1 (0xb7e52000)
        libc.so.6 => /lib/libc.so.6 (0xb7d35000)
        libdl.so.2 => /lib/libdl.so.2 (0xb7d31000)
        libm.so.6 => /lib/libm.so.6 (0xb7d0c000)
        /lib/ld-linux.so.2 (0xb7f8a000)

but we can see that mc doesn't ncurses at all, it use slang.
My slang-1.4.9-r2 is configured with unicode.
Comment 23 VirusCamp 2006-08-07 18:26:43 UTC
Created attachment 93713 [details]
another terrible program, which uses ncurses this time

media-sound/mp3blaster-3.2.2 is using ncurses, it has the same problem.
It is linked to /lib/libncurses.so.5 not libncursesw.so.5 .
but i have tried this:
 ln -s libncursesw.so.5 /lib/libncurses.so.5
It still doesn't work.

localhost ~ # ldd /usr/bin/mp3blaster
        linux-gate.so.1 =>  (0xffffe000)
        libvorbisfile.so.3 => /usr/lib/libvorbisfile.so.3 (0xb7ef2000)
        liblirc_client.so.0 => /usr/lib/liblirc_client.so.0 (0xb7eeb000)
        libncurses.so.5 => /lib/libncurses.so.5 (0xb7ea9000)
        libSDL-1.2.so.0 => /usr/lib/libSDL-1.2.so.0 (0xb7df8000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb7de5000)
        libstdc++.so.6 => /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libstdc++.so.6 (0xb7d06000)
Comment 24 SpanKY gentoo-dev 2006-08-07 18:59:31 UTC
uhh, completely unrelated ... this bug is about *ncurses*

slang-1.x has broken unicode handling, you need slang-2.x for proper unicode
Comment 25 Jozsef Daniel 2006-08-22 03:08:21 UTC
I think we don't need the narrowchar ncurses AT ALL. If we just use the widechar ncurses lib, and make a symlink of it, then all the apps will just link to that... And in 95% of all cases, this means the app will be unicode compatible.

(Too bad, but mc is in the remaining 5%)

Sure it's a hack. But it's a clean hack that works.

I propose the use of a new USE flag for this functionality. There are already USE flags around that you can't just omit in a running system - like pam -, so this can't be an argument.

The flag could be called "forceunicode", and would stop the narrowchar ncurses from being built, and create the symlinks ncurses.so, ncurses.5.so and ncurses.5.x.so.
I will soon upload a modified ebuild to do this. One question: what is the Gentoo way of creating arbitrary symlinks install-time in an ebuild?
Comment 26 Jozsef Daniel 2006-08-22 05:03:22 UTC
Created attachment 94849 [details]
Ebuild for building only widechar ncurses, controlled by the USE flag "forceunicode"

I think my experiment can be called a success. Would love to see this in Portage very soon.
It's a really nice hack for working around all those non-ncursesw-aware apps. After all, ncursesw is totally source compatible with ncurses!
Comment 27 SpanKY gentoo-dev 2006-08-22 19:57:08 UTC
source compatible means nothing, what matters is binary compatibility
Comment 28 Harald van Dijk (RETIRED) gentoo-dev 2006-08-22 22:58:06 UTC
Wide ncurses is not 100% binary compatible with narrow ncurses, but it is certainly safer to change lib(n)curses.so from narrow to wide than it is to change it from wide to narrow (which was done already).
Comment 29 Jozsef Daniel 2006-08-23 13:52:24 UTC
(In reply to comment #27)
> source compatible means nothing, what matters is binary compatibility

Well, I removed the narrowc ncurses, and just replaced ncurses.so and co with symlinks to ncursesw.so... And lo and behold, everything worked perfectly well. I did revdep-rebuild and re-emerged by hand all the revdeps of ncurses that revdep-rebuild didn't emerge.
I see no reason to be afraid of the widechar ncurses.
Comment 30 Jozsef Daniel 2006-08-25 08:01:44 UTC
I created an ebuild to solve this problem. I marked the previous ebuild obsolete, since it was put togeter in 5 minutes, and is obviously faulty.
I filed it as bug#145080

How about marking this one SOLVED?
Comment 31 SpanKY gentoo-dev 2006-08-25 22:17:49 UTC
i have no problem closing this bug but i think other people would re-open it
Comment 32 SpanKY gentoo-dev 2006-08-25 22:19:53 UTC
*** Bug 145080 has been marked as a duplicate of this bug. ***
Comment 33 Jozsef Daniel 2006-08-27 02:13:02 UTC
Created attachment 95176 [details]
sys-libs/ncurses-5.5-r4.ebuild - ebuild with new USE flag forceunicode

Since my ebuild bug was marked duplicate, here's the ebuild. I think the "component" field of this bug should be changed to "ebuilds".

This version adds a new USE flag, "forceunicode", which, if set along with "unicode", builds only the widechar lib (with default include path!), and creates symlinks to replace the narrowchar files.
Comment 34 Rafael Fernández López 2006-09-04 07:52:13 UTC
From spain, where we have special characters. My system is system-wide configured as UTF-8 (/etc/env.d/02locale says es_ES.UTF-8 for LC_ALL, LANGUAGE and LANG variables), and I have enabled by default unicode flag.

New ebuild didn't fix anything. I emerged ncurses-5.5-r4 with unicode and forceunicode USE flags enabled, and I still have the same problem with mp3blaster, tmsnc... they don't paint smoothly (as bad as 'make menuconfig').

ldd /usr/bin/mutt revealed that this is linked against wide ncurses, and I CAN WRITE SPANISH ACCENTS AND SYMBOLS without any problem (and euro symbol too). No strange behaviours in mutt.

What makes me mad is that emacs is linked against narrow ncurses and I CAN WRITE ACCENTS AND SPECIAL CHARACTERS without any problem, and no painting errors are shown.

In mp3blaster I cannot do that (it is linked against narrow ncurses).

If I start gentoo minimal livecd and I chroot to my root system and go to /usr/src/linux and run 'make menuconfig' I see that it is painted as it should, but if I boot from my root system (without live-cd) and do the same they are strange-painted. It happens to me in ~x86 and ~amd64. Using the last ncurses lib.

Here are some snapshots:

http://www.maestroprogramador.com/ncurses1.JPG
http://www.maestroprogramador.com/ncurses2.JPG

Bye !!
Comment 35 Rafael Fernández López 2006-09-05 15:08:01 UTC
I've found something interesting in ncurses sources. I'm pretty sure I'm mistaken, but I'd like you to tell me.

This is /usr/include/curses.h (Line 588)

extern NCURSES_EXPORT(int) hline (chtype, int);   /* generated */

This is /usr/include/cursesw.h (Line 1054)

int hline (int len, chtype ch = 0) { return ::whline(w, ch, len); }

My question is:
Is it possible that when a program forced to be linked against ncursesw calls hline with parameters in a mistaken order, or am I mistaken?
Comment 36 Harald van Dijk (RETIRED) gentoo-dev 2006-09-05 22:46:16 UTC
(In reply to comment #35)

The definition of hline() you listed is a member function, so no, that can't get called accidentally (thankfully).
Comment 37 Rafael Fernández López 2006-09-06 11:20:28 UTC
Here is an ~x86 system with ncurses-5.5-r3. Spanish, with UTF-8 enabled. I've been working a bit to know why is this happening. Well, I have some results.

(*) It does not matter whether an app is linked against libncurses or libncursesw. For example: emacs is linked against libncurses and can show national characters. mp3blaster is linked against libncurses and can not show national characters. mutt is linked against libncursesw and can show national characters. There is a counterexample for an app linked against libncursesw and cannot show national characters, but must be written by hand.

(*) The most important point is *HOW* data is read. I mean, some apps will use some kind of loop (for, while...) through getch() or friends (YES, INCLUDING wide ncurses) with noecho() called, so then the app will handle the way it is shown. But there is a problem, getch() and friends (YES, INCLUDING wide ncurses) cannot handle at all special characters. It *WON'T* work.

The way data should be read is:

(*) If we need to read only a character (we have a menu, a) Go to ... b) Exit), we should use getch, since here nobody should do something like 
Comment 38 Rafael Fernández López 2006-09-06 11:20:28 UTC
Here is an ~x86 system with ncurses-5.5-r3. Spanish, with UTF-8 enabled. I've been working a bit to know why is this happening. Well, I have some results.

(*) It does not matter whether an app is linked against libncurses or libncursesw. For example: emacs is linked against libncurses and can show national characters. mp3blaster is linked against libncurses and can not show national characters. mutt is linked against libncursesw and can show national characters. There is a counterexample for an app linked against libncursesw and cannot show national characters, but must be written by hand.

(*) The most important point is *HOW* data is read. I mean, some apps will use some kind of loop (for, while...) through getch() or friends (YES, INCLUDING wide ncurses) with noecho() called, so then the app will handle the way it is shown. But there is a problem, getch() and friends (YES, INCLUDING wide ncurses) cannot handle at all special characters. It *WON'T* work.

The way data should be read is:

(*) If we need to read only a character (we have a menu, a) Go to ... b) Exit), we should use getch, since here nobody should do something like ) Option1 é) Exit), that's nosense. So here, getch() usage is perfectly correct.

(*) If we want to read a string, we shouldn't do it by something like:

  for (int n = 0; n < hell_length; n++)
  {
     // manipulate with getch() or friends
  }

  - Here there are two possible behaviours:

     a) We would like to show characters while they become written (and special characters will be shown, doesn't matter if it is linked against libncurses or libncursesw !!)

        char str[BUFSIZ];
        echo(); // If we called noecho() before
        getstr (str);
        noecho(); // If we want to restore noecho() again
        // Manipulate with str
        // ...


     b) We wouldn't like to show characters while written.

        char str[BUFSIZ];
        noecho(); // If we called echo () before...
        getstr (str);
        // Manipulate with str
        // ...
        echo (); // If we want to restore echo () again...

    The last option (b) can solve lots of problems. For example, we can get a string while gets written, and we can show it through printw, and special characters will be shown, with no problem.

    The best way to see that it doesn't matter what to link against is writing the next app:

#include <ncursesw/ncurses.h>
#include <string.h>

int
main ()
{
  char mesg[] = "Write a string: ";
  char str[80];
  int row, col;

  initscr();
  noecho ();
  getmaxyx (stdscr, row, col);
  mvprintw (row / 2, (col - strlen (mesg)) / 2, "%s", mesg);

  echo ();
  getstr (str);
  noecho ();
  mvprintw (LINES - 2, 0, "You did write: %s", str);
  getch ();
  endwin ();

  return 0;
}

Compile with: 'g++ -o testit -lncursesw thisfile.cpp'

============= The other test =============

#include <ncurses.h>
#include <string.h>

int
main ()
{
  char mesg[] = "Write a string: ";
  char str[80];
  int row, col;

  initscr();
  noecho ();
  getmaxyx (stdscr, row, col);
  mvprintw (row / 2, (col - strlen (mesg)) / 2, "%s", mesg);

  echo ();
  getstr (str);
  noecho ();
  mvprintw (LINES - 2, 0, "You did write: %s", str);
  getch ();
  endwin ();

  return 0;
}

Compile with: 'g++ -o testit -lncurses thisfile.cpp'

I hope this helps, and sorry for the huge message.
Comment 39 Harald van Dijk (RETIRED) gentoo-dev 2006-09-06 12:19:13 UTC
(In reply to comment #37)

Your comment is wrong in more than one way, but to pick one, you obviously don't get the benefits of wide curses if you restrict yourself to the byte functions (rather than the character functions). More importantly though, this isn't the right place for a discussion on how to write ncurses applications.
Comment 40 Rafael Fernández López 2006-09-06 12:30:18 UTC
(In reply to comment #38)
> (In reply to comment #37)
> Your comment is wrong in more than one way, but to pick one, you obviously
> don't get the benefits of wide curses if you restrict yourself to the byte
> functions (rather than the character functions). More importantly though, this
> isn't the right place for a discussion on how to write ncurses applications.

But non wide ncurses apps will be able to print out special characters. Those apps don't get the explicit benefit from wide ncurses, but can print national characters with narrow ncurses. (euro symbol, and so on...)

Comment 41 Harald van Dijk (RETIRED) gentoo-dev 2006-09-06 12:52:28 UTC
(In reply to comment #39)
> (In reply to comment #38)
> > (In reply to comment #37)
> > Your comment is wrong in more than one way, but to pick one, you obviously
> > don't get the benefits of wide curses if you restrict yourself to the byte
> > functions (rather than the character functions). More importantly though, this
> > isn't the right place for a discussion on how to write ncurses applications.
> 
> But non wide ncurses apps will be able to print out special characters. Those
> apps don't get the explicit benefit from wide ncurses, but can print national
> characters with narrow ncurses. (euro symbol, and so on...)

No, using non-MB-aware apps in multibyte environments can very well, if you use any multibyte character, lead to memory and/or screen corruption. Seriously. Just try it with some less trivial example: when you read a character, print it. When you read a second character, erase the previous, and print the new one. (This, while different from your earlier comment, is also unrelated to this bug, by the way.)
Comment 42 Jozsef Daniel 2006-09-13 09:21:33 UTC
(In reply to comment #34)
> From spain, where we have special characters. My system is system-wide
> configured as UTF-8 (/etc/env.d/02locale says es_ES.UTF-8 for LC_ALL, LANGUAGE
> and LANG variables), and I have enabled by default unicode flag.
> 
> New ebuild didn't fix anything. I emerged ncurses-5.5-r4 with unicode and
> forceunicode USE flags enabled, and I still have the same problem with
> mp3blaster, tmsnc... they don't paint smoothly (as bad as 'make menuconfig').

Now, for one you might need to do revdep-rebuild, or manually re-emerge packages that depend on ncurses.
(HINT: to find such packages I like to use
cd /var/db/pkg
grep -e ncurses -f `find . -n *DEPEND`)

Forcing the widechar ncurses won't solve all problems. Some software are coded in stupid ways that make them source-incompatible with wide characters. However, it solves the problems that arise when a package is coded "the right way", but its makefile or autotool setup doesn't take advantage of a widechar ncurses present on the system.
(In fact, the standard gentoo setup seems to downright discourage packages from finding the installed ncursesw library, placing the include files in a highly nonstandard place, where autotools needs an additional command line flag to ./configure to find them!)
Comment 43 William Keaney 2007-02-06 21:39:42 UTC
I don´t know how relevant this is, but it Worked For Me(tm):
ncurses detects the type of terminal it is in using the TERM variable; it assumes that certain terminals, in a UTF8 environment, won´t be able to correctly interpret line-drawing characters and modifies its behavior accordingly.  In particular, if TERM=linux it will initiate this behavior.  
This doesn´t work well with the console (at least in Gentoo).  Exporting NCURSES_NO_UTF8_ACS=0 allowed my ncurses apps to draw correctly (specifically, alsamixer and ´make menuconfig´ from /usr/src/linux).  
Note that ncurses apps work fine from urxvt for me, and that my system was built completely from stage1 with unicode enabled.
More complete information on this angle can be found in the ncurses(3X) man page.
Comment 44 Ross Drummond 2007-02-07 22:58:16 UTC
After playing around with settings described in comment #43 from William Kearney, I appended the following line to my /etc/env.d/00basic file;

NCURSES_NO_UTF8_UCS="1"

After loggin out and logging back in, this fix resolved the problem that I was having with Midnight Commander and alsaconf diplaying properly.

uname -m
x86_64

echo $TERM
linux
Comment 45 SpanKY gentoo-dev 2007-02-08 02:48:14 UTC
sounds like there's nothing to be changed in current ncurses ebuilds
Comment 46 Ross Drummond 2007-02-08 05:52:17 UTC
Correction to comment #44

Should be;

NCURSES_NO_UTF8_ACS="1"
Comment 47 Jozsef Daniel 2007-02-08 08:36:34 UTC
(In reply to comment #45)
> sounds like there's nothing to be changed in current ncurses ebuilds
> 

It seems to me that we have lost the thread. Like hell there's nothing, the current Gentoo behavior breaks console i18n in about 90% of ncurses apps!

*** excerpt from the ncurses-5.6.ebuild

src_install() {
        # install unicode version first so that the non-unicode
        # files overwrite the unicode versions
        if use unicode ; then
                cd "${WORKDIR}"/widec

***

It installs the widec stuff in a hidden subdir, just to make sure that nothing links to it.
I understand the consideration that use flags shouldn't break the system, but totally obliterating unicode on the console is pretty bad too.

I think what would be okay is compiling ncurses with widec support, with no regard to use flags. I mean, having widec support doesn't hurt anyone, and I doubt it would slow the system down, even if it isn't needed.
Comment 48 Jozsef Daniel 2007-02-08 08:55:18 UTC
Something just occurred to me. Maybe Unicode should be a profile instead of a USE flag. If I get it right, profiles are the way to handle choices that cannot easily be altered in a running system.

On a unicode profile, only widec ncurses would be built, thus enabling unicode in all curses apps that support it otherwise.

(FYI, lynx does support unicode, yet in Gentoo is built without unicode. After switching to my own, widec-only ebuild, lynx displays national characters beautifully.)
Comment 49 kavol 2007-02-08 09:46:43 UTC
(In reply to comment #48)
> Maybe Unicode should be a profile instead of a USE flag.
> On a unicode profile, only widec ncurses would be built, thus enabling
> unicode in all curses apps that support it otherwise.

it seems to me like a good idea ... I am tired of the various workarounds ...
Comment 50 Jakub Moc (RETIRED) gentoo-dev 2007-02-17 11:29:54 UTC
(In reply to comment #47)
> *** excerpt from the ncurses-5.6.ebuild
> 
> src_install() {
>         # install unicode version first so that the non-unicode
>         # files overwrite the unicode versions
>         if use unicode ; then
>                 cd "${WORKDIR}"/widec
> 
> ***
> 
> It installs the widec stuff in a hidden subdir, just to make sure that nothing
> links to it.

Erm?

$ equery f ncurses | grep libncurses
/lib/libncurses.so
/lib/libncurses.so.5
/lib/libncurses.so.5.6
/lib/libncursesw.so
/lib/libncursesw.so.5
/lib/libncursesw.so.5.6
/usr/lib/libncurses++.a
/usr/lib/libncurses++w.a
/usr/lib/libncurses.a
/usr/lib/libncurses.so
/usr/lib/libncursesw.a
/usr/lib/libncursesw.so

$ equery f ncurses | grep libcurses
/lib/libcurses.so
/lib/libcursesw.so
/usr/lib/libcurses.a
/usr/lib/libcurses.so
/usr/lib/libcursesw.so

I wouldn't exactly call /lib a "hidden subdir" where "nothing links to it"?! You've apparently misread the ebuild.
Comment 51 Harald van Dijk (RETIRED) gentoo-dev 2007-02-18 13:31:23 UTC
Created attachment 110528 [details, diff]
ncurses-5.6-unicode.patch

Having thought about it a bit more again, how about a different approach? This patch causes ncurses to install both the narrow and the wide versions, but makes the narrow versions only available for binary compatibility, while everything new that's linked will link to the wide versions.
Comment 52 Jozsef Daniel 2007-02-20 07:46:10 UTC
Sorry, I wasn't specific enough. It does install the widec _libs_ where they belong, but it installs the widec _headers_ - which are the ones that really count in a source based distro - under /usr/include/ncursesw, while installing the narrowc headers in /usr/include.

(In reply to comment #50)
> (In reply to comment #47)
> > *** excerpt from the ncurses-5.6.ebuild
> > 
> > src_install() {
> >         # install unicode version first so that the non-unicode
> >         # files overwrite the unicode versions
> >         if use unicode ; then
> >                 cd "${WORKDIR}"/widec
> > 
> > ***
> > 
> > It installs the widec stuff in a hidden subdir, just to make sure that nothing
> > links to it.
> 
> Erm?
> 
> $ equery f ncurses | grep libncurses
> /lib/libncurses.so
> /lib/libncurses.so.5
> /lib/libncurses.so.5.6
> /lib/libncursesw.so
> /lib/libncursesw.so.5
> /lib/libncursesw.so.5.6
> /usr/lib/libncurses++.a
> /usr/lib/libncurses++w.a
> /usr/lib/libncurses.a
> /usr/lib/libncurses.so
> /usr/lib/libncursesw.a
> /usr/lib/libncursesw.so
> 
> $ equery f ncurses | grep libcurses
> /lib/libcurses.so
> /lib/libcursesw.so
> /usr/lib/libcurses.a
> /usr/lib/libcurses.so
> /usr/lib/libcursesw.so
> 
> I wouldn't exactly call /lib a "hidden subdir" where "nothing links to it"?!
> You've apparently misread the ebuild.
> 

Comment 53 Jozsef Daniel 2007-02-20 07:50:58 UTC
This sounds like a great idea. However, it doesn't solve the problem of things breaking when recompiling ncurses with -unicode.

I just thought of a different approach - create a separate ebuild sys-libs/ncurses-unicode, which blocks and provides sys-libs/ncurses. Thus switching between the two is only possible by unmerging ncurses - and no sane man would do that without booting from a livecd, and chrooting into his gentoo filesystem.

Problem solved?

(In reply to comment #51)
> Created an attachment (id=110528) [edit]
> ncurses-5.6-unicode.patch
> 
> Having thought about it a bit more again, how about a different approach? This
> patch causes ncurses to install both the narrow and the wide versions, but
> makes the narrow versions only available for binary compatibility, while
> everything new that's linked will link to the wide versions.
> 

Comment 54 Harald van Dijk (RETIRED) gentoo-dev 2007-02-20 08:16:12 UTC
(In reply to comment #53)
> This sounds like a great idea. However, it doesn't solve the problem of things
> breaking when recompiling ncurses with -unicode.

Good point. It's possible now to solve this with preserve_old_lib, and I'll try to get a patch working for that later.

I personally think the idea of separate ebuilds for wide and narrow ncurses would not work well, by the way, even if it's just because so many ebuilds would need to be updated (since they directly depend on sys-libs/ncurses).
Comment 55 Harald van Dijk (RETIRED) gentoo-dev 2007-02-20 09:25:31 UTC
Created attachment 110716 [details, diff]
ncurses-5.6-unicode.patch

Now successfully tested with bash linked against libncursesw.so and ncurses re-emerged with USE=-unicode