| Summary: | app-editors/nano tab all messed up when tab-size is not 8 (when tab size is not default) | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Rob Stoddard <rob> |
| Component: | Current packages | Assignee: | Lars Wendler (Polynomial-C) (RETIRED) <polynomial-c> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | bensberg |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: | https://savannah.gnu.org/bugs/?56372 | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
What's your TERM setting? Environment variable? TERM=xterm I don't believe the TERM setting has changed, should I have changed it? I am using PuTTY over SSH. I believe even the 8 space tab has some issues. If I have a string of slashes, all but one is hidden... For example: ///////////////////////////////// appears as: / You could try the various putty TERM settings:
> ls /usr/share/terminfo/p/putty*
/usr/share/terminfo/p/putty
/usr/share/terminfo/p/putty-256color
/usr/share/terminfo/p/putty+fnkeys
/usr/share/terminfo/p/putty+fnkeys+esc
/usr/share/terminfo/p/putty+fnkeys+linux
/usr/share/terminfo/p/putty+fnkeys+sco
/usr/share/terminfo/p/putty+fnkeys+vt100
/usr/share/terminfo/p/putty+fnkeys+vt400
/usr/share/terminfo/p/putty+fnkeys+xterm
/usr/share/terminfo/p/putty-m1
/usr/share/terminfo/p/putty-m1b
/usr/share/terminfo/p/putty-m2
/usr/share/terminfo/p/putty-noapp
/usr/share/terminfo/p/putty-sco
/usr/share/terminfo/p/putty-vt100
This appears to affect me too, but only in certain shells. I can't see any meaningful difference in the output of 'set' between the good shell and the bad shell. Furthermore, `source /etc/profile; reset` in the bad shell seems to sort it out. So apparently it's caused by some disconnect between shell settings and one of nano's dependencies - I've updated neither nano nor ncurses since the bad shell was opened which were my first suspects. So if this might affect you, try opening a fresh shell and see what happens. Fwiw, my TERM=xterm-256color Rob, Michael: Which version of nano? Which version of ncurses? What is the output of 'locale' on the affected machine? Rob, when you don't use PuTTY but ssh into the machine from an xterm (or Gnome Terminal or something similar), do you then experience the problem? Michael, you say: "in certain shells". Do you mean in certain instances? Or in certain types of shell, like bash or csh? And if the 'reset' command fixes it, probably a command before nano left the terminal in an abnormal state? If you can figure out which setting is off-kilter, nano can be changed to properly initialize that setting too. Rob, "for the past few weeks" -- was that since nano-2.9.5? Or did it start with an earlier version? Or did it start when you upgraded to a newer ncurses (which is what I suspect, because the behavior you are seeing is not something that nano could cause)? I see the strange behaviour when I have a non-standard tabs setting in bash, using SecureCRT (like PuTTY on steroids, Windows program) to connect to my Gentoo machine. I use "set tabsize 4" in nanorc and I got fed up with the output of e.g. cat using a tabsize of 8, so I added "tabs 4" to my .bashrc. I have had problems since that last change, and when reverting it the problems go away. It does not happen in nano on my Kubuntu VM, neither directly in the VM nor via ssh with SecureCRT. It seems that a difference might be in the bash version, but emerging bash 5 causes so many rebuilds on my system that I don't want to try that. Issuing a reset does nothing for me, only changing tabs back to 8 "fixes" the problem.
I am using:
app-shells/bash-4.4_p23-r1
app-editors/nano-3.2 and 4.2 (did not test 4.1)
sys-libs/ncurses-6.1-r2 (since march 2018, over a year ago)
Kubuntu 19.04:
bash 5.0-3ubuntu1.1
nano 3.2-2
ncurses 6.1+20181013-2ubuntu2
# locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=
Via Kubuntu konsole to Gentoo (ssh)
TERM=xterm-256color ok
TERM=xterm ok
TERM=linux problem
Via Kubuntu xterm to Gentoo (ssh)
TERM=xterm-256color problem
TERM=xterm problem
TERM=linux problem
Via SecureCRT to Gentoo (ssh)
TERM=xterm-256color problem
TERM=xterm problem
TERM=linux problem
Via SecureCRT to Kubuntu (ssh): all ok
Directly in Kubuntu on konsole: all ok
Directly in Gentoo: the machine is about 7000km away from me at the moment
Observed behaviour with different values for tabs and tabsize, tabs being the bash builtin and tabsize the setting in nanorc:
Tabs 8, tabsize 8: everything as it should be
Tabs 8, tabsize 4: everything as it should be
Tabs 4, tabsize 8 (cursor is _ so abc with the cursor on a becomes _bc)
abc[home][tab][tab][tab][tab]:
_bc
_ abc
_bc
_bc
_bc
Tabs 4, tabsize 4:
abc[home][tab][tab][tab][tab]:
_bc
_bc
_ abc
_bc
_bc
abc[home][tab][tab][end][left][left][left][tab]:
_bc
_bc
_ abc
abc_abc
abc _bc
If the line is sufficiently long (8 chars onwards), the problem does not happen anymore. The example line from Rob does not behave strangely for me.
abcdefgh[home][tab][tab][tab]
_bcdefgh
_bcdefgh
_bcdefgh
_bcdefgh
But when I start tabbing where the rest of the line is less than 8 characters, it's funky again and the cursor is all over the place:
abcdefghi[5 left][tab...]
abcd_fghi
abcd_ efghi
efgh_
_fgh efghi
efgh _fghi
efgh_ efghi
efgh _fghi
efgh _fghi
Tabs 4, tabsize 4/8, nano file.sh (for syntax highlighting), type elif [[ "${level}"
elif [[ "${level_
elif"${level}_el
elif"${l"${level}"_
Also, the commands in the bottom bar may be misaligned in the "problem" cases, but that is dependent on terminal width. Some widths give proper alignment.
I just unpacked a fresh hardened stage3 tarball in my Kubuntu VM, mounted /dev /proc and /sys, and ran in konsole: chroot /gentoo /bin/bash source /etc/profile tabs 4 reset TERM=linux nano and the problem is there already, this is 100% stock. I then went on to upgrade to sys-libs/readline-8.0 (bash 5 dependency) and the problem still persits. Same story after upgrading to app-shells/bash-5.0_p7. All tests with SecureCRT give identical results as before. Now I don't know where to look anymore. I don't think you should change TERM. If you are doing your things inside Konsole, then leave TERM set to what Konsole set it to. When you run 'tabs -d 4', do all the asteriskes pair up? Are there historical stage3 tarballs available? If yes, can you test in your VM a stage3 tarball from two years ago? Because the newest ncurses I have available is version 6.1 (patch 20180127), and I cannot reproduce. I was just trying changing around TERM to see if and when I could reproduce the problem. It is not something I normally do ;) So, not switching around TERM, the default in Konsole(xterm-256color) looks good but in SecureCRT(linux) is broken. I tried switching to xterm in SecureCRT settings and that starts my session with TERM=xterm (as expected) but that didn't help. The asterisks line up just fine for any n>1 when running tabs -d n. I already typed up everything below before I saw your last comment. I found an old stage3 but I'm on a sketchy connection at the moment, so it'll be a while before I can post results. I tried another ncurses-based editor, joe, and found that tabs 4, -tab 4 in joerc looks ok tabs 4, -tab 4 and -usetabs shows the same erroneous behaviour as nano, regardless of TERM tabs 4, -tab 8 and -usetabs is also wrong but in a different way than nano So that would point to ncurses. The -usetabs setting in joerc lead me to believe that it (and nano, or probably ncurses) wants to use tabs for screen rendering, but does not know (or makes a wrong assumption) about the tab width that gets output to the screen and in some cases tries to compensate (but still does not know what the screen is doing). Digging deeper into ncurses I found that "tabs" which I thought was a bash builtin, actually calls the ncurses-supplied /usr/bin/tabs. I grepped through the nano source but couldn't really find anything that hardcodes tabs to be 8 wide, except for nano --help, which assumes tabs are 8 wide and will not align when tabs != 8. Not that exciting, but since you built some logic to align everything, you'll probably want to look into that and replace it with spaces or so. So anyway, this means the problem almost has to be in ncurses or another library, and not in nano (or joe?) itself. Kubuntu is running ncurses 6.1.20181013 Gentoo is running ncurses 6.1.20180127 Copying libncursesw.so.6.1 and its dependency libtinfo.so.6 from Kubuntu to Gentoo fixes the problem in nano, so I guess I narrowed it down to something *in* the ncurses library, but potentially paired with the way nano or joe use it. Joe keeps acting up though, even after copying libncurses.so.6.1 (no w), so I probably missed something. I hacked up an ebuild that applies all ncurses dev patches until 20181013 and emerged that. That did not fix the problem. Neither did rebuilding nano or rebuilding ncurses with USE=tinfo (like the Kubuntu one). So I guess there is still something different between the Gentoo and the Kubuntu build options. I see the same behaviour in a fresh hardened stage3 from 20170105, using nano 2.5.3 and ncurses 6.0.20150808 (and bash 4.3.48). I did some digging to try and find how nano handles tabs and calls ncurses to update the screen, but the winio code is pretty tough to get through. Some code in text.c lead me to test --tabstospaces, and with that switch I see the same results as without, even though moving around with the arrows shows that there are spaces instead of tabs now. So that's kind of weird, the setting doesn't seem to affect what happens to the cursor so I guess nano doesn't really touch it or it gets overwritten, and lets ncurses or the terminal handle that all on its own. I did some printf debugging and it seems that the openfile->placewewant gets updated properly (according to the nano tabsize) but that the cursor doesn't always take that position. Pressing Ctrl+C after every keypress has the side effect of moving the cursor in the correct position and "fixes" some of the problematic behaviour, but not the case with the bash code that shifted left. Disclaimer: I have no idea how ncurses works, and today is also the first time I've seen the nano codebase. Okay, I can reproduce when ssh'ing to the server that runs nano-editor.org. I'm pretty sure that that is a Gentoo box, but how can I find out? Method of reproduction: ssh to box, tabs 4, nano --ignore, then type: abc <Home> <Tab>. Result: the "abc" is pushed 8 positions to the right but the cursor sits only halfway. A ^L (redraw) puts the cursor in the correct place. The cursor misplacement happens for me both on an Xfce Terminal and on a Linux console (VT). On the latter also some misalignment of the help lines occurs: for example, the items for ^O, ^W, and ^K are placed four characters to the left of where they should be. That it also happens with --tabstospaces is another indication that this is an ncurses problem, not a nano one. Maybe it's not even ncurses, because the 'tabs' command does something strange: even when it is run on a remote machine, it affects the local terminal, /but/ somehow also stores something on the remote machine. Because the first time that I ssh'ed to the server, I ran nano --ignore without having given a tabs command, and things were misaligned. After giving a 'tabs 8', things were right. After a 'tabs 4', things were wrong again. How did things go wrong for the first try? Because it remembered I had last done a 'tabs 4' from a VT on a different machine? Anyway... I only see this on the nano-editor.org machine, not on three other remote boxes that I have access to (which all run some kind of Debian). You can be pretty sure that it's a Gentoo box if you can run "emerge".
I also observed that it doesn't matter which machine the tabs command in run on. If I use tabs 4 on Kubuntu and then ssh to my Gentoo box, it'll be misaligned too, and when I log out, tabs 8 and ssh again, it's fine. I'm not sure if it actually does anything on any remote machine, but I assume it only affects the local terminal, and that'll persist across ssh sessions. Somehow the Gentoo ncurses isn't using the correct value whereas the Kubuntu (and I guess Debian as well) one does. Keep in mind that the tabs command is part of ncurses, so I'm still fairly sure that the problem is in there.
As far as I can tell looking at strace, the only interesting thing that tabs does is write to the terminal:
write(1, "\33[3g\r\33H \33H \33H \33H...")
\33[3g = clear tabs, \r, \33H = set tabstop here, and 4 spaces between them until the right edge of the screen. It looks like nothing is "written" to the remote machine.
I just don't get how the Kubuntu ncurses and my hacked-up Gentoo version differ. I read all the Gentoo patches and they don't look like they do anything special, and all the Kubuntu patches I found in their source package are uninteresting as well. That leaves only some ./configure options or maybe a different build-dep and that doesn't sound too logical, either. Yet, when I just copy the Kubuntu .so into Gentoo, the problem goes away.
The bottom bar is indeed misaligned in some cases, dependent on terminal width, and the in-nano help (^G) appears to be tab-aligned as well, just as the usage(). The latter is an easy fix, every \t should be replaced by an appropriate amount of spaces, and the body of print_opt() can be replaced by the one-liner
printf(" %-14s %-23s %s\n", shortflag, longflag, _(desc));
which fixes the alignment regardless of tab width. I don't know why the elaborate tab-aligning code is there, but they should be used for indentation only, with spaces for alignment. But that's not a big issue.
I've pushed =sys-libs/ncurses-6.1_p20181020 to the tree but it's p.masked for now. If you have a non-crucial system for testing I'd appreciate if you could give that ncurses version a try and report back your results. Found it! :)
The Gentoo ebuild uses --enable-hard-tabs in do_configure(), and Kubuntu does not. I commented out that switch in my ebuild and re-emerged ncurses. The problem is gone now and everything works as it should. Some more digging lead me to the env var NCURSES_NO_HARD_TABS, that disables it on runtime, so it's not necessary to modify the ebuild and rebuild the package. Running "NCURSES_NO_HARD_TABS=1 nano" shows correct behaviour on my main Gentoo box, so I'll go ahead and stick that in my .bashrc.
INSTALL:
--enable-hard-tabs
Compile-in cursor-optimization code that uses hard-tabs. We would make
this a standard feature except for the concern that the terminfo entry
may not be accurate, or that your stty settings have disabled the use
of tabs.
ncurses/tty/lib_mvcur.c:
* The USE_HARD_TABS switch controls whether it is reliable to use tab/backtabs
* for local motions. On many systems, it's not, due to uncertainties about
* tab delays and whether or not tabs will be expanded in raw mode. If you
* have parm_right_cursor, tab motions don't win you a lot anyhow.
It is actually also sort of mentioned in the manpage of tabs:
These are hardware tabs, which cannot be queried rapidly by applications running in the terminal, if at all. Curses and other full-screen applications may use hardware tabs in optimizing their output to the terminal. If the hardware tabstops differ from the information in the terminal database, the result is unpredictable.
So I'm beginning to wonder why this switch was explicitly enabled for Gentoo.
As I understand it, ncurses wants to use hardware tabs because they are implemented in hardware and don't require placeholder spaces. Of course, in order to do that it needs to query the terminfo to see what the tab width is (8 in these cases). When the terminal is adjusted with the tabs command, it is only written to the terminal and not to the terminfo, so ncurses does not know about it and can't position the cursor properly anymore. Makes complete sense. Only the documentation is slightly unclear IMO, and you'd expect a tool that came with the library to fully work in synergy. Makes me wonder if there's not a better way to change the tab width that does work with hardware tabs.
... some time later ...
And yes there is. I think I finally found THE way to do it. Using SecureCRT I get TERM=linux, so I did
infocmp -L > linuxcustom.term
in that file, changed init_tabs#8 to init_tabs#4, saved it and
mkdir -p ~/.terminfo
tic -o .terminfo/ linuxcustom.term
logged out and in again, and nano was botched the other way around, because my terminal didn't know about the tab width being 4 yet. So I issued a "tabs 4" and got nano to work without problems, with the unmodified Gentoo ncurses and no special env vars. So it is not enough to just issue tabs 4 (as most google results say), terminfo needs to be modified for each TERM you are using to let ncurses know what you're doing. Most people just get away with it because Debian/Ubuntu and probably other more popular distros don't use hardware tabs. Now I also see why Gentoo does have it enabled (and ncurses wants to have it as a default someday).
The modifications to the terminfo file are persistent in ~/.terminfo across logins. However, the tab width is not initialized after logging in (or it is ignored), so it is mandatory to run "tabs x" on every login.
The other thing is, after issuing a "reset", ncurses forces tabs to 8 (this is hardcoded), and also only if init_tabs is not 8 in the terminfo file, so that looks like a bug to me and likely breaks nano on terminals that have init_tabs!=8 without modifications. For me, it looks botched again after a reset. I'll see if I have to file a bugreport against ncurses for that. In the meantime, I also fixed that by using
alias reset="reset; tabs 4"
So the full thing I'm using in .bashrc now is:
function nctabs {
tabw=$1
mkdir -p ~/.terminfo
infocmp -L > ~/.terminfo/.cur.ti
perl -pi -e 's/init_tabs#[0-9]+/init_tabs#'${tabw}'/' ~/.terminfo/.cur.ti
tic -o ~/.terminfo/ ~/.terminfo/.cur.ti
tabs ${tabw}
alias reset="reset; tabs ${tabw}"
rm ~/.terminfo/.cur.ti
}
nctabs 4
Great detective work! I can confirm that running "NCURSES_NO_HARD_TABS=1 nano" on the Gentoo box makes nano behave correctly after a 'tabs 4'. You say that you now understand why Gentoo enables hard tabs by default, but... I don't see what advantage these hard tabs bring. I find it strange that the 'tabs' command, when issued on the target machine, doesn't tell the ncurses there to override the tab width that it got from the terminfo database and to remember it for as long as the login session lasts. Because: it issued a command sequence to the local terminal to set tabs at certain places, but then continues to use the value that is in the database? That doesn't make sense. About the help text misalignments in nano, do you want to post a bug report on nano's Savannah page, and maybe attach a signed-off patch? Or shall I report the bug on your behalf and take it from there? > You say that you now understand why Gentoo enables hard tabs by default, but... > I don't see what advantage these hard tabs bring. Well, it's not *that* exciting, but just sending tabs as tabs the way they were intended is of course better than emulating them with spaces which cost 4, 8 or n times the amount of bytes. If the terminfo tabstop width or user-set tabs match the tabstop width requested by the application (TABWIDTH), there is no need to do anything, a tab is a tab and the same width everywhere. It's only when these differ that code needs to kick in to attempt to place the cursor at the correct position. > I find it strange that the 'tabs' command, when issued on the target machine, > doesn't tell the ncurses there to override the tab width that it got from the > terminfo database and to remember it for as long as the login session lasts. > Because: it issued a command sequence to the local terminal to set tabs at > certain places, but then continues to use the value that is in the database? > That doesn't make sense. Because of the way terminals work and, subsequently, "tabs" works, after changing the tab width there is no way for ncurses to know what's going on. You could store the output from tabs in a textfile and cat that to whichever terminal later to change the tabstop width without informing ncurses. Also, all adjustments you do on a terminal persist until you close the terminal, not whatever is running in there. You could ssh to box a, have it change your local terminal for you, log out and ssh to box b, but your terminal will still be the way a set it without b knowing about it. In the end, ncurses is also "just another library" that's not used by all applications or might not even be present at all. You'd need some other standardised infrastructure to store these "settings". And it's all old stuff, when this was conceived there wasn't a lot of hardware that could report on those settings, if they were adjustable at all, so they went with a pre-compiled library of loads of different terminals, just set the active one in TERM and go from there, hoping both sides were in agreement. As far as I can tell/think/guess, if you press tab in your terminal, it gets sent to the application or remote machine that you ssh'd into, but the tab isn't echoed back (usually). Your local terminal has to draw it according to its own rules, which it doesn't tell the other side. You've probably seen those effects using e.g. telnet and had to turn local echo on or eevveenn ooffff. And for things like nano, that allow changing the tabsize to something different than the terminal tabsize, some adjustment needs to happen, but there is only limited information on the app side. There are a lot of different settings in a terminfo file, and the current setup doesn't seem to allow for pushing all of those settings (or any) to the running application. I'm assuming most of this but I think it's pretty close to what's going on. > About the help text misalignments in nano, do you want to post a bug report on > nano's Savannah page, and maybe attach a signed-off patch? Or shall I report > the bug on your behalf and take it from there? For me, that's quite a lot of work in a workflow that I'm not used to for two lines of code that are already here :P So if you'd like to take care of it, please. Thanks :) Okay, I understand now: when the cursor needs to travel relatively small distances, it is more efficient to issue a couple of actual tab characters than to issue the full sequence of a cursor-movement command or a whole series of spaces. (The help-text bug I've reported as https://savannah.gnu.org/bugs/?56372.) I'd say this one can be closed as not a bug, rather than linger in UNCO? The reason for the behaviour was incorrect use of terminal features, that's been addressed in this thread, and two somewhat related other bugs in nano and ncurses have been found and fixed. To recap, the terminal and ncurses need to be in agreement on the tabstop width, whichever width the uses chooses. The bash function in #15 should take care of that. Only when that's in sync, nano should work properly with any combination of tabstop width and --tabsize, but the latter option still defaults to 8 regardless of what the terminal is using. Probably the most logical thing to do from a user perspective is to explicitly use the same value for nano --tabsize and the terminal tabstop width. Marking resolved as the issues in ncurses and nano that has been fixed was most likely causing this problem. I have not seen this issue for a very long time. Thank you Vincent! |
I usually use a tab-size of 4 in nano. I also use tab to indent rather than spaces. I know, it's the logical thing to do and therefore not recommended. Anyway, for the past few weeks, nano has been giving me really wonky visuals when I tab-indent stuff. When I open a file everything looks right and proper; when I page up and page down, things are proper... but if I hit tab twice on a line, characters from that line appear in the most inappropriate of places. For example, I have this line. Indented once, I get: For example, I have this line. Indented twice, I get: For example, I have this line.ne. Note the sudden loss of indent. Three times: For exa For example, I have this line. And four times: For exa For example, I have this line. I don't know if this is an issue with ncurses or nano, however, I suspect it may be ncurses since I recall seeing it upgrade at about the time this problem appeared. Also note, with a tab size of 8, this problem doesn't seem to happen.