Created attachment 704115 [details] Modified ebuild My system ran out of memory while trying to build this package. Specifically, while compiling src/common/iso639_language_list.cpp, gcc start consuming an ever increasing amount of RAM, then swapping hell starts, and eventually the linux OOM killer terminates the compiled. See attached log for example. I filed this bug with upstream: https://gitlab.com/mbunkus/mkvtoolnix/-/issues/3105 Apparently the problem lies with using any optimization when compiling this particular module. In fact this one should be compiled with -O0 and all of the others should be with -O3. The attached ebuild is a slight modification of the existing one in that it replaces '--disable-optimization' with '--enable-optimization' during configuration and filters out any user-specified compiler optimization.
Created attachment 704118 [details] Build log
Please add the output of emerge --info mkvtoolnix as a comment to this bug.
Created attachment 704124 [details] emerge --info for mkvtoolnix-56.1.0-r1 This is information relative to my *modified* ebuild that installed the package today.
Created attachment 704127 [details] emerge --info for mkvtoolnix-51.0.0 I don't know if this is useful or relevant, but this is information pertaining to the previous installation of mkvtoolnix in same environment on 2021-03-16
Rakefile replaces " -O\d+ " by " -O0 ". Note the spaces on both sides, and also that \d is digits-only e.g. for iso639_language_list.cpp: CXXFLAGS="-march=native -O2 -pipe" is replaced by -O0 CXXFLAGS="-march=native -O2" is still -O2 CXXFLAGS="-march=native -Os -pipe" still -Os
Borderline horrifying so I’ll see what Soap thinks.
Same problem with 16 GB RAM… :sigh:
(In reply to Vincent Reher from comment #0) > I filed this bug with upstream: > https://gitlab.com/mbunkus/mkvtoolnix/-/issues/3105 > Apparently the problem lies with using any optimization when compiling this > particular module. In fact this one should be compiled with -O0 and all of > the others should be with -O3. Why upstream is not using __attribute__((optimize(0))? > The attached ebuild is a slight modification of the existing one in that it > replaces '--disable-optimization' with '--enable-optimization' during > configuration and filters out any user-specified compiler optimization. This will filter-out other very useful options like -march.
MKVToolNix author here. What changed between versions is that the language list has grown considerably, from a couple hundred entries to over 7000. There might be some kind of exponential growth in time needed for optimizations based on the number of entries. I'm not using __attribute__((optimize(0))) for two reasons: 1. I hadn't actually been aware that such an attribute exists until it was mentioned over on my issue tracker today. 2. It doesn't actually work that well. With -O0 and any attributes compilation of that one file takes ~2.5 minutes (clang++; g++ is similar). Sure that's long but acceptable. With the default -O3 and __attribute__((optnone)) for clang++ compilation took nearly 9 minutes (g++ with __attribute__((optimize(0))) took more than 20 minutes which was when I aborted the experiment). As I'm compiling this stuff all the time (and compilation of the whole application only takes 3 minutes or so due to high parallelization), such a slowdown isn't acceptable to me. Haven't looked where memory usage peaks. If anyone can come up with a way to use the attributes and get similar compilation times as -O0, please let me know. As for the regex replacing the optimization flags: sure, that one can be changed to work better with custom flags. How about: (^| )-O[^ ]* That should work with -O at the start or at least after a space and continue up but excluding to the next space (or the end of the string). It should take care of all the examples Ionen Wolkens posted and doesn't require you to run configure with "--disable-optimizations" or so. I'll commit that for the next release (not 57.0.0 which was released earlier today, though).
I've just committed the change to Rakefile's regular expression for filtering out optimization flags: https://gitlab.com/mbunkus/mkvtoolnix/-/commit/ba6db89e24aa84316372d59e6bba4ac88145b32a Ideally this change alone should allow compilation on Gentoo again without any other fixes.
(In reply to Moritz Bunkus from comment #10) > I've just committed the change to Rakefile's regular expression for > filtering out optimization flags: > https://gitlab.com/mbunkus/mkvtoolnix/-/commit/ > ba6db89e24aa84316372d59e6bba4ac88145b32a > > Ideally this change alone should allow compilation on Gentoo again without > any other fixes. Thank you! With this patch mkvtoolnix builds like a charm (I built current in-tree version 56.1.0 for now). @Soap, OK to apply?
(In reply to Moritz Bunkus from comment #9) > MKVToolNix author here. > > What changed between versions is that the language list has grown > considerably, from a couple hundred entries to over 7000. There might be > some kind of exponential growth in time needed for optimizations based on > the number of entries. > > I'm not using __attribute__((optimize(0))) for two reasons: > > 1. I hadn't actually been aware that such an attribute exists until it was > mentioned over on my issue tracker today. > > 2. It doesn't actually work that well. > > With -O0 and any attributes compilation of that one file takes ~2.5 minutes > (clang++; g++ is similar). Sure that's long but acceptable. > > With the default -O3 and __attribute__((optnone)) for clang++ compilation > took nearly 9 minutes (g++ with __attribute__((optimize(0))) took more than > 20 minutes which was when I aborted the experiment). As I'm compiling this > stuff all the time (and compilation of the whole application only takes 3 > minutes or so due to high parallelization), such a slowdown isn't acceptable > to me. Haven't looked where memory usage peaks. Very interesting information. Maybe it has something to do with implicit class members.
(In reply to Andrew Savchenko from comment #11) > (In reply to Moritz Bunkus from comment #10) > > I've just committed the change to Rakefile's regular expression for > > filtering out optimization flags: > > https://gitlab.com/mbunkus/mkvtoolnix/-/commit/ > > ba6db89e24aa84316372d59e6bba4ac88145b32a > > > > Ideally this change alone should allow compilation on Gentoo again without > > any other fixes. > > Thank you! With this patch mkvtoolnix builds like a charm (I built current > in-tree version 56.1.0 for now). > > @Soap, OK to apply? sure, apply it
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bae380256f345d9a79a0941b6259b8fb700fbf5e commit bae380256f345d9a79a0941b6259b8fb700fbf5e Author: Andrew Savchenko <bircoph@gentoo.org> AuthorDate: 2021-05-23 07:16:28 +0000 Commit: Andrew Savchenko <bircoph@gentoo.org> CommitDate: 2021-05-23 07:23:09 +0000 media-video/mkvtoolnix: fix OOM during build Apply upstream patch[1] to drop iso639_language_list.cpp optimization level to 0 due to special nature of this source file. [1] https://gitlab.com/mbunkus/mkvtoolnix/- /commit/ba6db89e24aa84316372d59e6bba4ac88145b32a Closes: https://bugs.gentoo.org/786978 Package-Manager: Portage-3.0.18, Repoman-3.0.3 Signed-off-by: Andrew Savchenko <bircoph@gentoo.org> .../mkvtoolnix/files/mkvtoolnix-56.1.0-optlevel.patch | 15 +++++++++++++++ media-video/mkvtoolnix/mkvtoolnix-56.1.0.ebuild | 3 ++- 2 files changed, 17 insertions(+), 1 deletion(-)
