Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 931512 - app-emulation/dxvk-2.3.1-r1 causes wine to crash. (stack smashing)
Summary: app-emulation/dxvk-2.3.1-r1 causes wine to crash. (stack smashing)
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Ionen Wolkens
URL: https://github.com/doitsujin/dxvk/iss...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-05-07 23:51 UTC by Tohka
Modified: 2024-05-10 03:03 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
wine.log (stack smashing with system compiled dxvk) (wine.log,23.41 KB, text/x-log)
2024-05-07 23:51 UTC, Tohka
Details
wine-dxvk-bin.log (upstream provided binaries showing that it works with wine) (wine-dxvk-bin.log,51.86 KB, text/x-log)
2024-05-07 23:52 UTC, Tohka
Details
emerge_info.txt (emerge_info.txt,18.40 KB, text/plain)
2024-05-08 03:47 UTC, Tohka
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tohka 2024-05-07 23:51:37 UTC
Created attachment 892455 [details]
wine.log (stack smashing with system compiled dxvk)

When using system compiled dxvk

app-emulation/dxvk-2.3.1-r1::gentoo was built with the following:
USE="d3d10 d3d11 d3d9 dxgi strip -crossdev-mingw" ABI_X86="32 (64)"
CFLAGS="-march=skylake -O2 -pipe -mno-avx"
CXXFLAGS="-march=skylake -O2 -pipe -mno-avx"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"

And then using setup_dxvk.sh to symlink to the system binaries in a wine prefix wine gives us an error (stack smashing) the wine.log provided is when WINEDEBUG=+loaddll

When using dxvk-2.3.1-r1 binaries provided in the upstream releases this error does not happen and dxvk loads just fine. The log for wine is provided as wine-dxvk-bin.log this can be taken as a reference to what I "should" be seeing.
Comment 1 Tohka 2024-05-07 23:52:17 UTC
Created attachment 892456 [details]
wine-dxvk-bin.log (upstream provided binaries showing that it works with wine)
Comment 2 Ionen Wolkens gentoo-dev 2024-05-08 03:24:48 UTC
emerge --info?

Sounds kind of odd given I don't think mingw64-toolchain enables stack-protector by default (unlike the gentoo-wide setting).

Either way it'd still be an upstream issue if stack smashing is happening.
Comment 3 Ionen Wolkens gentoo-dev 2024-05-08 03:27:27 UTC
Also, is this something you can reproduce with about anything using dxvk?

Because I never ran into this using several games, I don't think I could help.

If interested, you could try to build it with mingw64-toolchain-11.0.1 too (currently unkeyworded so need to add it to package.accept_keywords), it uses gcc-14.1.0 -- could potentially help.
Comment 4 Ionen Wolkens gentoo-dev 2024-05-08 03:30:07 UTC
Another thing you may want to try is building it without -march=skylake, it could be specific to cpu optimizations. Albeit I have a skylake as well and don't have problems.
Comment 5 Ionen Wolkens gentoo-dev 2024-05-08 03:39:41 UTC
(In reply to Ionen Wolkens from comment #2)
> emerge --info?
> 
> Sounds kind of odd given I don't think mingw64-toolchain enables
> stack-protector by default (unlike the gentoo-wide setting).

Oh wait, are you even using mingw64-toolchain, or using your own with crossdev?
Comment 6 Tohka 2024-05-08 03:47:05 UTC
Created attachment 892457 [details]
emerge_info.txt
Comment 7 Tohka 2024-05-08 03:50:31 UTC
(In reply to Ionen Wolkens from comment #3)
> Also, is this something you can reproduce with about anything using dxvk?
> 
> Because I never ran into this using several games, I don't think I could
> help.
> 
> If interested, you could try to build it with mingw64-toolchain-11.0.1 too
> (currently unkeyworded so need to add it to package.accept_keywords), it
> uses gcc-14.1.0 -- could potentially help.

From my testing some games it worked some games it didn't for example fallout 4 works fallout new vegas doesn't. yakuza 0 yakuza kiwami doesn't either. The games work without dxvk and also work with dxvk (from upstream) but not using the gentoo compiled dxvk. I had this work in the past (maybe end of last year) but this time the only thing that is different other than updated packages is that I am using an nvidia gpu. I tried downgrading versions of wine and dxvk but the issue still persists. I could try upgrading the latest mingw64-toolchain. I am not using crossdev.
Comment 8 Tohka 2024-05-08 03:51:26 UTC
(In reply to Ionen Wolkens from comment #4)
> Another thing you may want to try is building it without -march=skylake, it
> could be specific to cpu optimizations. Albeit I have a skylake as well and
> don't have problems.

I was thinking that as well maybe -march=x86-64
Comment 9 Tohka 2024-05-08 03:52:55 UTC
(In reply to Ionen Wolkens from comment #2)
> emerge --info?
> 
> Sounds kind of odd given I don't think mingw64-toolchain enables
> stack-protector by default (unlike the gentoo-wide setting).
> 
> Either way it'd still be an upstream issue if stack smashing is happening.

could it be a wine issue or a dxvk issue I believe its already reported on dxvk.
Comment 10 Ionen Wolkens gentoo-dev 2024-05-08 03:58:43 UTC
CXXFLAGS="-mtune=skylake -march=skylake -O2 -pipe -fcf-protection=full -fstack-clash-protection -fstack-protector-strong -fwrapv -ftrivial-auto-var-init=zero"

Note that Gentoo enable -fstack-protector-strong by default *except* for mingw because of how fragile the stack is.

If you didn't use these CXXFLAGS *for* dxvk, it may be related to having used it on mingw64-toolchain.
Comment 11 Ionen Wolkens gentoo-dev 2024-05-08 03:59:29 UTC
(In reply to Ionen Wolkens from comment #10)
> CXXFLAGS="-mtune=skylake -march=skylake -O2 -pipe -fcf-protection=full
> -fstack-clash-protection -fstack-protector-strong -fwrapv
> -ftrivial-auto-var-init=zero"
> 
> Note that Gentoo enable -fstack-protector-strong by default *except* for
> mingw because of how fragile the stack is.
> 
> If you didn't use these CXXFLAGS *for* dxvk, it may be related to having
> used it on mingw64-toolchain.
(it's used for the static libraries that dxvk is linking with)
Comment 12 Ionen Wolkens gentoo-dev 2024-05-08 04:34:28 UTC
(In reply to Ionen Wolkens from comment #11)
> > If you didn't use these CXXFLAGS *for* dxvk, it may be related to having
> > used it on mingw64-toolchain.
> (it's used for the static libraries that dxvk is linking with)
If trying to rebuild that without -fstack-protector-strong (or with -fno-stack-protector), remember to also rebuild dxvk again after -- and ensure the right dll are used (if a symlink, then it should be the right one).
Comment 13 Ionen Wolkens gentoo-dev 2024-05-08 04:39:21 UTC
If it still didn't help, may want to rebuild wine too albeit that being an issue would sound more surprising if the upstream binaries were working.
Comment 14 Tohka 2024-05-08 04:52:14 UTC
After rebuilding mingw64-toolchain without the extra flags I was able to recompile dxvk and it all worked. Which makes me wonder that I do recall that there is a useflag called "custom-cflags" however I don't have this enabled. So I am confused as to why it was reading the extra flags that I have in make.conf when it ideally shouldn't
Comment 15 Tohka 2024-05-08 04:56:38 UTC
(In reply to Tohka from comment #14)
> After rebuilding mingw64-toolchain without the extra flags I was able to
> recompile dxvk and it all worked. Which makes me wonder that I do recall
> that there is a useflag called "custom-cflags" however I don't have this
> enabled. So I am confused as to why it was reading the extra flags that I
> have in make.conf when it ideally shouldn't

I believe mingw64-toolchain was taking the following flags from my make.conf 

-fcf-protection=full -fstack-clash-protection -fstack-protector-strong
Comment 16 Ionen Wolkens gentoo-dev 2024-05-08 05:00:13 UTC
-fstack-protector-strong is considered a safe flag, so it doesn't get stripped by "strip-flags" which is what USE=custom-cflags control.

Issues with that are typically something that should be fixed bare some special cases at most. Albeit I would recommend not passing it and letting the toolchain control it though (as I said, it's default for the system-wide gcc+clang, while mingw does not).

I *could* potentially add -fno-stack-protector to mingw64-runtime+toolchain but I'd rather wait to see what dxvk upstream has to say given it's really not a great "fix".
Comment 17 Ionen Wolkens gentoo-dev 2024-05-08 05:06:45 UTC
On another note, I did find a game that let me reproduce if I do -fstack-protector-strong w/ mingw runtime.
Comment 18 Tohka 2024-05-08 22:12:35 UTC
Ooh what game is it? And I don't think i have emerged the runtime I just did the toolchain or is the runtime also pulled? Also I agree I think its best to see what upstream does to resolve this issue if they were to fix whatever is causing the stack smashing.
Comment 19 Ionen Wolkens gentoo-dev 2024-05-09 03:39:37 UTC
(In reply to Tohka from comment #18)
> And I don't think i have emerged the runtime I just did
> the toolchain or is the runtime also pulled?
mingw64-toolchain is 3 packages in 1, gcc, binutils, and mingw64-runtime -- it allows the single ebuild to bootstrap the toolchain similarly to what crossdev would do but without needing users to follow instructions and setup invasive configurations in /etc/portage.

gcc+binutils don't overly matter in this context, and if I did -fno-stack-protector in the ebuild it'd only be to build the runtime bits (so only applications run in wine would be lacking this, and not the compiler building these)
Comment 20 Ionen Wolkens gentoo-dev 2024-05-10 02:25:33 UTC
Sounds like it'll be difficult for upstream to handle this with mingw.

However, I'm thinking to only do this in mingw64-toolchain and not also in mingw64-runtime (the package), and still allow it if USE=custom-cflags is set (crossdev is kind of hope-you-know-what-you're-doing territory, so no need to overly go there).

Given mingw64-toolchain is never default-ssp, can just filter rather than do -fno-* too.
Comment 21 Larry the Git Cow gentoo-dev 2024-05-10 03:03:18 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=4778d9cbaca208869cb80f606db24d857c75aa3f

commit 4778d9cbaca208869cb80f606db24d857c75aa3f
Author:     Ionen Wolkens <ionen@gentoo.org>
AuthorDate: 2024-05-10 02:36:23 +0000
Commit:     Ionen Wolkens <ionen@gentoo.org>
CommitDate: 2024-05-10 03:00:43 +0000

    dev-util/mingw64-toolchain: filter -fstack-protector* again
    
    Used to be done because it was straight up broken (couldn't
    build), but now it's successfully causing problems and it's
    not obvious for upstreams to fix these issues with mingw.
    
    There may be real issues in dxvk & others, but support for
    this is new in mingw (also only partial), and believe using
    this can be considered too experimental/early "here". Still
    allowing it with USE=custom-cflags for those that really want
    it and don't mind if it breaks some components they don't use.
    
    Skipping revbump given it's an unusual configuration given
    users normally don't pass this in *FLAGS but rather rely
    on the toolchain's defaults. mingw64-toolchain-11.0.1 will
    also be keyworded in not that long for rebuilds.
    
    Only needed for mingw runtime, so the toolchain itself is still
    using it through the system-wide defaults.
    
    May revisit eventually.
    
    Not doing this for crossdev+mingw64-runtime main package again,
    these are for more expert use in the first place.
    
    Bug: https://bugs.gentoo.org/870136
    Closes: https://bugs.gentoo.org/931512
    Signed-off-by: Ionen Wolkens <ionen@gentoo.org>

 dev-util/mingw64-toolchain/mingw64-toolchain-11.0.0_p2.ebuild | 6 ++++++
 dev-util/mingw64-toolchain/mingw64-toolchain-11.0.1.ebuild    | 6 ++++++
 2 files changed, 12 insertions(+)