Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 906368 - Support sys-devel/dwz for debuginfo optimization
Summary: Support sys-devel/dwz for debuginfo optimization
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, PullRequest
Depends on: 921380
Blocks: 933986
  Show dependency tree
 
Reported: 2023-05-15 05:13 UTC by Sam James
Modified: 2024-06-10 02:19 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
dedupdebug POC (dedupdebug.patch,2.63 KB, patch)
2024-01-24 06:28 UTC, YiFei Zhu
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-05-15 05:13:04 UTC
sys-devel/dwz can be used to optimize debugging information before compression which can improve disk space usage.

We should see about integrating it into Portage. Especially when combined with zstd compressdebug support (bug 906367), this might lead to considerable disk space savings.
Comment 1 YiFei Zhu 2024-01-24 06:27:22 UTC
Is dwz multifile mode desired? I'm asking because I'm not sure what would be the best place & name to store the common object for the -m argument to dwz.

If not, I did a quick patch, naming the feature FEATURE=dedupdebug, attached here (I haven't edited any of the man pages yet). This is a quick test result with =app-shells/bash-5.2_p26, FEATURES="splitdebug compressdebug", and the size of /usr/lib/debug/usr/bin/bash.debug:

* -ggdb FEATURES=-dedupdebug: 1237808
* -ggdb FEATURES=dedupdebug: 1141408 (7.8% decrease)

* -ggdb3 FEATURES=-dedupdebug: 1444872
* -ggdb3 FEATURES=dedupdebug: 1347472 (6.7% decrease)

I made it run before the splitdebug phase in estrip because dwz seems unable to optimize a compressdebug-ed binary.
Comment 2 YiFei Zhu 2024-01-24 06:28:44 UTC
Created attachment 883016 [details, diff]
dedupdebug POC
Comment 3 YiFei Zhu 2024-01-26 20:21:05 UTC
Did my periodic @world upgrade (last one was on the 20th), /usr/lib/debug/ went down from 4.6G to 4.2G, even though I changed global CFLAGS -ggdb to -ggdb3 (though I also have a lot of packages, such as nodejs, not have any -g* at all), so there's probably some significant space savings.

I'll send a PR when I'm free.
Comment 4 Arsen Arsenović gentoo-dev 2024-01-26 20:31:52 UTC
this patch seems reasonable to me at a glance.  re, multi-file mode: it sounds like it could be useful (esp. for c++ programs).  what would that require and what effect would it have?
Comment 5 YiFei Zhu 2024-01-26 20:48:58 UTC
> what would that require and what effect would it have?

I believe what I need mainly is a filename for the "combined debug info". i.e.

> OPTIONS
>       -m FILE --multifile FILE
>              Multifile mode.  After  processing  all  named  executables  and
>              shared  libraries, attempt to create ELF object FILE and put de‐
>              bugging information duplicated in more than  one  object  there,
>              afterwards optimize each named executable or shared library even
>              further if possible.

What the file name for FILE should be. My initial thought is probably /usr/lib/debug/dwz/<atom with version>.debug

But another issue comes with how it'll interact with splitdebug & compressdebug. The new files in /usr/lib/debug/dwz/ should not be split again, but it should be processed by compressdebug (I don't think dwz performs compression itself but I might be wrong). I think it'll take a bit of trial and error for me to do that correctly.

As for the effect, I would believe that the final sum of the sizes of all the debug files will be even smaller.
Comment 6 Larry the Git Cow gentoo-dev 2024-02-21 02:13:07 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=749d231d57fb64d3f2e909dfbba26ca7c1ab9d0e

commit 749d231d57fb64d3f2e909dfbba26ca7c1ab9d0e
Author:     YiFei Zhu <zhuyifei1999@gmail.com>
AuthorDate: 2024-01-29 08:50:28 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-02-21 02:13:04 +0000

    estrip: Support debug info deduplication with sys-devel/dwz
    
    This this gates behind FEATURES=dedupdebug, and packages can
    opt-out with RESTRICT=dedupdebug, though I'm not sure why a package
    would need to opt this out, unless dwz could break something I'm not
    aware of...
    
    Bug: https://bugs.gentoo.org/906368
    Signed-off-by: YiFei Zhu <zhuyifei1999@gmail.com>
    Closes: https://github.com/gentoo/portage/pull/1245
    Signed-off-by: Sam James <sam@gentoo.org>

 bin/estrip                 | 35 +++++++++++++++++++++++++++++++++--
 lib/_emerge/EbuildPhase.py |  1 +
 lib/portage/const.py       |  1 +
 man/ebuild.5               |  3 +++
 man/make.conf.5            |  8 +++++++-
 5 files changed, 45 insertions(+), 3 deletions(-)
Comment 7 Larry the Git Cow gentoo-dev 2024-02-22 07:24:13 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77c44c46194922509bc4f2b5cfc099412a560a69

commit 77c44c46194922509bc4f2b5cfc099412a560a69
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-02-22 07:23:40 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-02-22 07:23:50 +0000

    sys-apps/portage: add 3.0.62
    
    Closes: https://bugs.gentoo.org/663324
    Closes: https://bugs.gentoo.org/728046
    Closes: https://bugs.gentoo.org/891137
    Closes: https://bugs.gentoo.org/906368
    Closes: https://bugs.gentoo.org/916566
    Closes: https://bugs.gentoo.org/921170
    Closes: https://bugs.gentoo.org/921208
    Closes: https://bugs.gentoo.org/921400
    Closes: https://bugs.gentoo.org/922038
    Closes: https://bugs.gentoo.org/922142
    Closes: https://bugs.gentoo.org/923368
    Closes: https://bugs.gentoo.org/923750
    Closes: https://bugs.gentoo.org/923841
    Closes: https://bugs.gentoo.org/923852
    Closes: https://bugs.gentoo.org/923854
    Closes: https://bugs.gentoo.org/924192
    Closes: https://bugs.gentoo.org/924273
    Closes: https://bugs.gentoo.org/924585
    Closes: https://bugs.gentoo.org/921380
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.62.ebuild | 246 +++++++++++++++++++++++++++++++++
 2 files changed, 247 insertions(+)