Bug 947033 – zig.eclass: system and bundled libraries handling in packages

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Eric Joldasov 2024-12-27 17:16:43 UTC]

Hello everyone,

on request from https://github.com/gentoo/gentoo/pull/39857#issuecomment-2563810769 I'd like to discuss with you way to use system libraries in Zig packages. Some of them contain special toggles for their build scripts (called "system integration options" in Zig terminology), which is often used to choose between:
 1) Dynamically or statically linking to system library using found compiled objects, or
 2) Compiling static (possibly modified) version of same library using Zig build system.

In zig.eclass this is controlled by adding "-fsys=[name]" or "-fno-sys=[name]" flags during src_configure. zig-ebuilder currently emits USE-flags "+system-[lib_name]" for every such option found, and based on them force-enables or force-disables them in ebuild. It was mentioned that:

> We generally do unconditional system libraries unless there's a good reason
> not to, e.g. upstream sometimes modify the bundled copy.

Which suggests removing these USE-flags and always enabling "system integrations" in ebuild options. I have tried USE-flags approach due to these factors:
 * Projects which are converted to Zig Build system usually follow one of the 4 routes:
 * * (Likely no modifications) Creating repository with build system part and adding tarball with upstream source as dependency in build.zig.zon or as git submodule. In this case you can be sure that it was not modified and likely close to the vanilla variant.
 * * (Can be patched) Same but tarball is changed to forked version,
 * * (Can be patched) Full forking of upstream source
 * * (Likely no modification) upstreamed back to C project, like in `dev-libs/libpcre2`: https://github.com/PCRE2Project/pcre2/blob/pcre2-10.44/build.zig

Sometimes modifications are unavoidable, because vanilla variant does not build correctly in zig-static case: `zig build` uses `zig cc` under hood
when compiling C/C++ source files, which differs from regular `clang` compiler
by 1) enabling UBSAN in "Debug" mode, 2) using compiler_rt written in zig instead of LLVM one. One such case happened with GLFW and its fork for game engine in Zig: https://devlog.hexops.com/2021/perfecting-glfw-for-zig-and-finding-undefined-behavior/ (they have improved since then and nowadays
also supply "verify.sh" script to show what files (apart from build files) differ from upstream)

 * It allows more reproducibility and easier troubleshooting for both users and upstream developers. If they are asked to try with bundled dependencies and check if some bug persist, it is very easy to do by disabling them in package.use (and re-enabling after that). For ebuild authors it's also useful for checking ZBS_DEPENDENCIES completeness in case they filled it manually (idk why would they but anyway) or zig-ebuilder messed up somewhere.
 * Without USE-flags, it would be more difficult for users to disable system
dependency for previously mentioned troubleshootings, they would need to
handle this in /etc/portage/env like so:

ZBS_ARGS_EXTRA="-fno-sys=[some_library]"

And also ensure that it applies only to package which have this option, since Zig will not ignore non-existant option and would instead fail with:

> system library name not recognized by build script: 'some_library'
>   access the help menu with 'zig build -h'

What do you think?

[Comment 1 Sam James 2024-12-27 17:34:08 UTC]

(In reply to Eric Joldasov from comment #0)
> 
>  * It allows more reproducibility and easier troubleshooting for both users
> and upstream developers. If they are asked to try with bundled dependencies
> and check if some bug persist, it is very easy to do by disabling them in
> package.use (and re-enabling after that). For ebuild authors it's also
> useful for checking ZBS_DEPENDENCIES completeness in case they filled it
> manually (idk why would they but anyway) or zig-ebuilder messed up somewhere.
>  * Without USE-flags, it would be more difficult for users to disable system
> dependency for previously mentioned troubleshootings, they would need to
> handle this in /etc/portage/env like so:

The issue with this argument is that it applies in general to bundled libraries. But we don't add USE=system-XXX for all bundled libraries, we default to using system libraries, and we only add a USE (in general, not just for zig) if there's a compelling reason, e.g. sometimes upstream make modifications to the bundled copy, or they're perhaps very hard to work with and really really want you to test with their bundled copy.

I appreciate that last part is similar to what you're saying, but doing it for an entire ecosystem is something else.

[Comment 2 Eli Schwartz 2025-01-01 15:36:04 UTC]

I was linked here from https://github.com/gentoo/gentoo/pull/39924#discussion_r1900398547

To quote myself from there...

It strikes me as incredibly wrong to first flout the rules and then open a bug report asking whether the rules should be respected. Instead, you should respect the rules and then open a bug report asking whether the rules should be broken.

> zig-ebuilder currently emits USE-flags "+system-[lib_name]" for
> every such option found


IMO you need to immediately fix this to not do so, and *then* have a discussion about whether to start doing so again.

[Comment 3 Eric Joldasov 2025-01-15 19:06:45 UTC]

After some thinking I think you are right:
 * For reproducing part: user can always build it by themselves toggling flags as you need,
 * other parts don't scale well for "entire ecosystem", I expect number of system integration options to only grow in the future, and adding all these as a flags is too much.

I have checked ghostty dependencies and all of them seem to be identical to their "vanilla" counterparts, excluding difference in version pinned here vs Gentoo (in meantime I have also found some small omissions in ebuild dependencies, and will open PR soon). I'll disable "system-*" USE-flag generation in zig-ebuilder now.

[Comment 4 Larry the Git Cow 2025-01-17 13:36:05 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5c8b025773af642a12ad84f94931b604e488addd

commit 5c8b025773af642a12ad84f94931b604e488addd
Author:     Eric Joldasov <[e-mail]>
AuthorDate: 2025-01-15 19:13:43 +0000
Commit:     Joonas Niilola <[e-mail]>
CommitDate: 2025-01-17 13:35:59 +0000

    x11-terms/ghostty: add 1.0.1-r3
    
    * Remove `system-*` USE-flags due to linked bug
    
    Also noticed when reviewing upstream build.zig files:
    * Enable `truetype` USE-flag on harfbuzz; Ghostty needs
      freetype-related functions as can be seen in `createFont` function
      from "pkg/harfbuzz/freetype.zig"
    * Remove libpng dependency and enable `png` USE-flag on freetype instead,
      as a separate dependency it is used in upstream only in static build
      case as a transient dependency of freetype (mentioned in build.zig
      comment). PNG is decoded with wuffs version instead.
    * Same with zlib dependency, it's only used as dependency of
      freetype and libpng. Decompression is done with Zig standard module
      version instead (`std.compress.zlib`).
    * Same with libxml2 dependency, for fontconfig.
    
    Bug: https://bugs.gentoo.org/947033
    Signed-off-by: Eric Joldasov <[e-mail]>
    Signed-off-by: Joonas Niilola <[e-mail]>

 x11-terms/ghostty/ghostty-1.0.1-r3.ebuild | 125 ++++++++++++++++++++++++++++++
 1 file changed, 125 insertions(+)

[Comment 5 Eric Joldasov 2025-03-05 18:52:43 UTC]

zig-ebuilder enabled all system integrations for some time now, and no ghostty version now has `system-*`, so I'll mark it as resolved now