Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 666582 - media-sound/lilypond-2.19.80-r1 doesn't handle non-ASCII characters correctly anymore
Summary: media-sound/lilypond-2.19.80-r1 doesn't handle non-ASCII characters correctly...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Scheme Project
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-09-19 19:54 UTC by Tobias Leupold
Modified: 2020-09-17 14:05 UTC (History)
10 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Patch for lilypond-guil (umlaute.patch,4.75 KB, patch)
2019-05-01 13:20 UTC, Robert Spillner
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tobias Leupold 2018-09-19 19:54:14 UTC
I'm not perfectly sure if this is an upstream bug or a Gentoo specific problem, but:

Since I upgraded to Lilypond 2.19.80-r1, it doesn't handle non-ASCII characters correctly anymore. Lilypond uses UTF-8, and up to version 2.18.2-r3, everything was fine: If I wrote e. g. German Umlauts in the sources, they were displayed as such.

With version 2.19.80-r1, I get a "?" character in my output instead of the non-ASCII character.

Unfortunately, I can't downgrade Lilypond anymore, because the older version pulls <=dev-scheme/guile-2, which collides with my GnuCash installation, but as said: as long as I used the older version, everything was fine.

I also get "?" characters on the console when using a non-C locale, e. g.:

Interpretation der Musik...
Vorverarbeitung der grafischen Elemente...
Musik wird auf eine Seite angepasst...
Systeme erstellen...
Layout nach »/tmp/lilypond-25RULQ« ausgeben...
Konvertierung nach ??1. Alt-Saxophon in Es.pdf??...
L??schen von ??/tmp/lilypond-25RULQ??...
Kompilation erfolgreich beendet

Any idea how to track this down?
Comment 1 Tobias Leupold 2018-09-19 20:03:14 UTC
I just tried to install the binary version from http://lilypond.org/unix.de.html manually ("GNU/Linux 64: LilyPond 2.18.2-1"). I can run it (apparently, there's no guile conflict with this package) and I don't get output errors.

I also don't get those double ?? in the console output:

Analysieren...
Interpretation der Musik...
Vorverarbeitung der grafischen Elemente...
Musik wird auf eine Seite angepasst...
Systeme erstellen...
Layout nach »1. Alt-Saxophon in Es.ps« ausgeben...
Konvertierung nach »./1. Alt-Saxophon in Es.pdf«...
Kompilation erfolgreich beendet

So probably, something goes wrong compiling the newer version on Gentoo?!
Comment 2 Tobias Leupold 2018-09-21 13:41:23 UTC
Apparently, guile 2 is the problem.

This (German) forum thread https://lilypondforum.de/index.php/topic,355.0.html links to a GNU mailing list post: http://lists.gnu.org/archive/html/bug-lilypond/2018-06/msg00011.html which includes a patchset to make Lilypond somewhat work with guile 2.

I could apply all those patches, with the exception of patch #13. Building it with this patchset does at least solve the Locale problem, I get my special chars again.

But apparently, Lilypond isn't meant to be used with guile 2 at all yet! So I think it would be meaningful to remove the guile2 USE flag by default.

This would lead to a slot conflict problem, at least for people also using GnuCash (like me), which blocks guile <2. So perhaps, one should enable slots for guile, so that one can have an <2 version in parallel to v2?

After all, I don't want/need to use the guile2 USE flag, but I can't build lilypond without as long as I have GnuCash installed.

Would be nice if this could be solved! Thanks in advance for all efforts!
Comment 3 Jouni Kosonen 2018-09-21 14:21:11 UTC
(In reply to Tobias Leupold from comment #2)
> This (German) forum thread https://lilypondforum.de/index.php/topic,355.0.html links to a GNU mailing list post: http://lists.gnu.org/archive/html/bug-lilypond/2018-06/msg00011.html which includes a patchset to make Lilypond somewhat work with guile 2.

Those are the relevant patches originally from the "guile-v2-work" branch in the lilypond git repository.

> I could apply all those patches, with the exception of patch #13. 
That one doesn't apply because it does the same as the existing Gentoo patch 
lilypond-2.19.80-remove-ly-protect.patch

> Would be nice if this could be solved! Thanks in advance for all efforts!
Seconded, and thanks to you for finding that patchset. Extracting those in userpatches and removing #13 did result in a lilypond binary that processes at least the utf-8 lyrics correctly.
Comment 4 Tobias Leupold 2018-09-22 07:17:29 UTC
I just got additional upstream information  from the lilypond forum. I'll cite (translated):

> Jouni Kosonen is only partially right: The patches 13-15 from my link are not
> part of this branch. I already merged some reworked patches from the "guile-v2-
> work" branch to master. The "guile-v2-work" branch should be updated soonish
> [...]
>
> Apparently, Gentoo already addressed the problem from patch-13 themselves. 
> patch-14 only turns off unneeded warnings.
> But patch-15 ist important if one uses file names with special characters.

I can confirm lilypond also not to handle non-ASCII characters in filenames ("File not found" or whatever) without patch #15.
Comment 5 Tobias Leupold 2018-09-22 07:26:11 UTC
(In reply to Tobias Leupold from comment #2)
> But apparently, Lilypond isn't meant to be used with guile 2 at all yet! So
> I think it would be meaningful to remove the guile2 USE flag by default.

I just saw that actually, guile2 is disabled by default (it really should be!). But, as said, I could not update my system anymore without it due to GnuCash blocking depending on >=dev-scheme/guile-2.0.0:12 and dev-scheme/guile-2.0.14-r3 does not allow to be dev-scheme/guile-1.8.8-r3 to be installed at the same time.

So probably, we should split this to two bugs? One: "Essential fixes for Lilypond with USE=guile2 set" and "Please change slotting for dev-scheme/guile"?
Comment 6 Christopher Smith 2019-03-24 04:21:50 UTC
I see this bug is still marked as "unconfirmed"; I'm experiencing this exact problem some months later. (I probably was at the time, but I didn't have non-ASCII characters in my input.)
Comment 7 hangglider 2019-03-30 16:41:21 UTC
Current situation (at least at here) is, that either lilypond is compiled with USE="-guile2" or lilypond won't handle non-ASCII characters at all, both in text handling and it's own (diagnostic or progress) output. So it seems to at least partially be related to some change in guile.

Would it be possible to find (and probably fix) the occansins where that happens? Unfortunately I'm far from being a Scheme/Lisp expert...
Comment 8 Robert Spillner 2019-05-01 13:20:16 UTC
Created attachment 574910 [details, diff]
Patch for lilypond-guil

This is a patch I cherry picked from patches upstream last year when I encountered this bug. I really don't recount how it works specifically, it was more or less try and error but it does the job for me.
Also, it's reasonable short so if you're fluent you'll hopefully be able to grasp the essence.
Comment 9 Robert Spillner 2019-05-01 13:20:49 UTC
Was supposed to be "Patch for lilypond/guile2 with German umlauts"
Comment 10 hangglider 2019-05-02 10:03:46 UTC
Works for me (and applies without problems), thanks a lot!
Comment 11 Frank Steinmetzger 2019-10-01 21:24:45 UTC
There are comments from 2018 that some patches are already merged. Another says that guile2 is disabled by default, and another to compile lilypond with USE=-guile2. The latter is actually not possible (anymore):

USE=-guile2 emerge -pv lilypond
[ebuild   R    ] media-sound/lilypond-2.19.80-r1::gentoo USE="(guile2) ..."

If there is a working patch from at least May, could this please be put into portage at some point (or is that rather an upstream issue)?

Greetings.
Comment 12 Jouni Kosonen 2019-10-01 21:43:50 UTC
I've had these in userpatches when I build with USE="emacs guile2" since Sep -18:

0001-Initialize-locale-in-GUILEv2.patch
0002-Source_file-init_port-Keep-GUILEv2-from-redecoding-s.patch
0003-Update-changes-from-commit-122525f-Keep-GUILEv2-from.patch
0004-XXX-Avoid-the-lockup-in-ly_scm_write_string.patch
0005-Fix-the-encoding-of-the-postscript-output.patch
0006-Fix-wide-characters-support-in-embedded-SCM-with-gui.patch
0007-Use-the-UTF-8-encoding-for-SVG-files.patch
0008-Fix-formatting-floating-point-variables-with-guile-2.patch
0009-Fix-ly_scm2string-to-work-with-guile-2.0.patch
0010-XXX-add-support-for-itexi-files-to-the-vim-config.patch
0011-XXX-don-t-override-LANG-globally-in-the-build-proces.patch
0012-XXX-reset-the-locale-when-building-index.html.patch
0014-Suggestions-by-Arne.patch
0015-Initialize-locale-in-lily.scm.patch

They are exactly the 15 patches referred to in comment 2, less patch #13 that Gentoo already handles.
Comment 13 Miroslav Šulc gentoo-dev 2019-12-02 11:24:54 UTC
can you please test with 2.19.83?
Comment 14 Jean-Claude Repetto 2019-12-10 18:23:14 UTC
Same problems with 2.19.83
Comment 15 Miroslav Šulc gentoo-dev 2020-09-17 07:34:45 UTC
can you please test whether the issue still exists in 2.21.6? and if it is still there, whether the attached patch still applies and fixes the issue?
Comment 16 hangglider 2020-09-17 12:57:55 UTC
For me, -2.21.6 seems to work without the need for patching.
Comment 17 hangglider 2020-09-17 13:01:48 UTC
(In reply to hangglider from comment #16)
> For me, -2.21.6 doesn't work without the need for patching (First test case was wrong, so I incorrectly reported it to work before).
Comment 18 Jean-Claude Repetto 2020-09-17 13:55:12 UTC
Problem not fixed:

$ lilypond àéè.ly
GNU LilyPond 2.21.6
warning: cannot find file: `??????.ly'
fatal error: failed files: "??????.ly"
Comment 19 hangglider 2020-09-17 14:05:19 UTC
Same problem.

Patch didn't apply here to lilypond-2.21.6, trying to manually patch files didn't work well either and result is the same.