| Summary: | sys-devel/crossdev: support winpthreads by default for mingw (<mutex> and friends does not work) | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Rafael Kitover <rkitover> |
| Component: | Current packages | Assignee: | Gentoo Crossdev team <crossdev> |
| Status: | CONFIRMED --- | ||
| Severity: | normal | CC: | bgilbert, embedded, ionen, jstein, sam, slyfox, sven.koehler, theli.ua, toolchain |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=732106 https://bugs.gentoo.org/show_bug.cgi?id=631460 |
||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
|
Description
Rafael Kitover
2018-09-08 16:49:43 UTC
The threading is decided around:
https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/toolchain.eclass#n1066
case ${CHOST} in
mingw*|*-mingw*)
confgcc+=( --enable-threads=win32 ) ;;
*)
confgcc+=( --enable-threads=posix ) ;;
esac
I don't understand what proposed change is supposed to do. Will it just provide one more -lpthread library with pthread_* symbols implemented?
Will it also change threading primitives used by gcc to use that library instead of windows.h primitives? (Like c++'s std::thread implementation).
Here is the documentation I found on this option: https://gcc.gnu.org/install/configure.html So, yes, it does affect C++, but in a way that is completely transparent to the programmer/user, using posix threads on windows does not hurt any of these features. According to this: https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Preprocessor-Options.html#Preprocessor-Options the '-pthread' option turns on preprocessor macros for using POSIX threads in the compile phase. In the link phase, it adds the pthread library: https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Link-Options.html#Link-Options The pthread library is built along with gcc on mingw targets and is a runtime dependency, but it can easily be linked statically. For example, this should work everywhere: gcc <objects> -o bin -Wl,-Bstatic -lpthread -Wl,-Bdynamic Now as to the WHY. Suppose you are building the windows port of your favorite open source project. This by the way is the most common use case for a mingw toolchain. Otherwise you're just using visual studio. It will either itself, or in some libraries it depends on, require POSIX threads. Because these are all linux libraries that have windows ports. Adjusting all of your build system related stuff to use some kind of crappy pthreads replacement instead of using the gcc one like everyone else is a huge and unnecessary pain in the ass. For this reason, the popular distribution msys2 which is a mingw toolchain for windows with a slightly tweaked cygwin layer for some tools and the shell: https://www.msys2.org/ builds gcc with POSIX threads. I was recently fighting with one of the mac homebrew devs to also default to this option for the mingw-w64 brew, and he eventually agreed seeing that this was also the default for msys2. (In reply to Rafael Kitover from comment #2) > Here is the documentation I found on this option: > > https://gcc.gnu.org/install/configure.html > > So, yes, it does affect C++, but in a way that is completely transparent to > the programmer/user, using posix threads on windows does not hurt any of > these features. My main interest here is what actually happens in gcc to assess the impact of change. For example: - how painful it will be for people who already have a mingw toolchain built to update gcc - what could regress as a result I don't see any mention of implementation changes WRT exposed interfaces. Only about 'single' vs. non-'single'. Looking at gcc source code it seems to only affect: - what 'gcc -v' reports in threading model - libatomic/libgomp/libitm implement a few primitives based on pthreads instead of win32. See lib{atomic,gomp,itm}/config/posix/. That is not much. Does mingw64-runtime actually change it's behaviour based on default threading model? Looking at mingw64-runtime's configure.ac --with-libraries=yes|all is supposed to provide winpthreads. That's what IUSE=libraries does. Something like $ USE=libraries emerge -v1 cross-x86_64-w64-mingw32/mingw64-runtime Does it not work as-is for you? I still fail to see material difference that --enable-threads=posix for gcc provides. > According to this: > > https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Preprocessor-Options. > html#Preprocessor-Options ... AFAIU this is how gcc behaves regardless of --enable-threads=posix. (In reply to Sergei Trofimovich from comment #3) > My main interest here is what actually happens in gcc to assess the impact > of change. For example: > - how painful it will be for people who already have a mingw toolchain built > to update gcc > - what could regress as a result That's a good thing to consider, and I don't really know, but I doubt the choice of gcc threading primitives would break much of anything. The only issue is having to link libpthread, which is not that big of a deal. > Looking at mingw64-runtime's configure.ac --with-libraries=yes|all is > supposed to provide winpthreads. That's what IUSE=libraries does. > > Something like > $ USE=libraries emerge -v1 cross-x86_64-w64-mingw32/mingw64-runtime > Does it not work as-is for you? So I tried this for my build system, and it seems to work more or less. All the pthread related options work for gcc. But I have a lot of issues with this setup and I haven't built my project yet. Some of my issues are probably caused by my cross gcc being 6.4.0 because of #665508, which I haven't figured out yet. I had to add -L/usr/x86_64-w64-mingw32/usr/lib64 to my LDFLAGS. Why is it even lib64 for a single arch sysroot? Is this because the old gcc doesn't support any kind of multilib stuff? Assuming this works properly with a newer gcc and the search paths issue is not present, or can be fixed, can this 'libraries' use flag be default or required so that pthreads is always available? Especially since users will be using crossdev to build the whole toolchain and not emerging packages individually. > So I tried this for my build system, and it seems to work more or less. All > the pthread related options work for gcc. But I have a lot of issues with > this setup and I haven't built my project yet. Some of my issues are > probably caused by my cross gcc being 6.4.0 because of #665508, which I > haven't figured out yet. > > I had to add -L/usr/x86_64-w64-mingw32/usr/lib64 to my LDFLAGS. Why is it > even lib64 for a single arch sysroot? Is this because the old gcc doesn't > support any kind of multilib stuff? This is just a bug in cross-*/mingw64-runtime packages. It uses $(get_libdir) from host profile (amd64-linux) and not target profile. It's a bug #653246 unrelated to gcc's multilib handling. > Assuming this works properly with a newer gcc and the search paths issue is > not present, or can be fixed, can this 'libraries' use flag be default or > required so that pthreads is always available? Especially since users will > be using crossdev to build the whole toolchain and not emerging packages > individually. Enabling libraries by default makes sense. We can try it. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=478d91086baa442fb578e0982063b93a19caf382 commit 478d91086baa442fb578e0982063b93a19caf382 Author: Sergei Trofimovich <slyfox@gentoo.org> AuthorDate: 2018-10-06 17:35:13 +0000 Commit: Sergei Trofimovich <slyfox@gentoo.org> CommitDate: 2018-10-06 17:35:21 +0000 dev-util/mingw64-runtime: enable USE=libraries by default Enable mangle, pseh, winpthreads libraries by default. Notably this exposes libpthread.a compatible library and makes building mode pthread-based software. Reported-by: Rafael Kitover Bug: https://bugs.gentoo.org/665512 Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> Package-Manager: Portage-2.3.50, Repoman-2.3.11 dev-util/mingw64-runtime/metadata.xml | 12 +++--------- dev-util/mingw64-runtime/mingw64-runtime-6.0.0-r1.ebuild | 2 +- 2 files changed, 4 insertions(+), 10 deletions(-) Just tried building a new toolchain on an updated system with all traces of previous builds removed. I am getting a failure building mingw64-runtime-6.0.0-r1, in libmangle: configure:3243: checking whether the C compiler works configure:3265: x86_64-w64-mingw32-gcc -march=native -O2 -pipe -I/var/tmp/portage/cross-x86_64-w64-mingw32/mingw64-runtime-6.0.0-r1/temp/tmproot/include -Wl,-O1 -Wl,--as-needed conftest.c >&5 /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find crt2.o: No such file or directory /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingw32 /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmoldname /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingwex /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmsvcrt /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -ladvapi32 /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lshell32 /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -luser32 /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lkernel32 /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingw32 /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmoldname /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingwex /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmsvcrt collect2: error: ld returned 1 exit status configure:3269: $? = 1 configure:3307: result: no configure: failed program was: | /* confdefs.h */ | #define PACKAGE_NAME "mingw-w64-libmangle" | #define PACKAGE_TARNAME "mingw-w64-libmangle" | #define PACKAGE_VERSION "1.0" | #define PACKAGE_STRING "mingw-w64-libmangle 1.0" | #define PACKAGE_BUGREPORT "mingw-w64-public@lists.sf.net" | #define PACKAGE_URL "" | #define PACKAGE "mingw-w64-libmangle" | #define VERSION "1.0" | /* end confdefs.h. */ | | int | main () | { | | ; | return 0; | } configure:3312: error: in `/var/tmp/portage/cross-x86_64-w64-mingw32/mingw64-runtime-6.0.0-r1/work/mingw-w64-v6.0.0/mingw-w64-libraries/libmangle': configure:3314: error: C compiler cannot create executables See `config.log' for more details Trying to see if I have more luck with 5.0.4. (In reply to Rafael Kitover from comment #7) > Just tried building a new toolchain on an updated system with all traces of > previous builds removed. > > I am getting a failure building mingw64-runtime-6.0.0-r1, in libmangle: > > configure:3243: checking whether the C compiler works > configure:3265: x86_64-w64-mingw32-gcc -march=native -O2 -pipe > -I/var/tmp/portage/cross-x86_64-w64-mingw32/mingw64-runtime-6.0.0-r1/temp/ > tmproot/include -Wl,-O1 -Wl,--as-needed conftest.c >&5 > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find crt2.o: No such file or > directory > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingw32 > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmoldname > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingwex > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmsvcrt > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -ladvapi32 > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lshell32 > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -luser32 > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lkernel32 > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingw32 > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmoldname > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingwex > /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmsvcrt > collect2: error: ld returned 1 exit status > configure:3269: $? = 1 > configure:3307: result: no > configure: failed program was: > | /* confdefs.h */ > | #define PACKAGE_NAME "mingw-w64-libmangle" > | #define PACKAGE_TARNAME "mingw-w64-libmangle" > | #define PACKAGE_VERSION "1.0" > | #define PACKAGE_STRING "mingw-w64-libmangle 1.0" > | #define PACKAGE_BUGREPORT "mingw-w64-public@lists.sf.net" > | #define PACKAGE_URL "" > | #define PACKAGE "mingw-w64-libmangle" > | #define VERSION "1.0" > | /* end confdefs.h. */ > | > | int > | main () > | { > | > | ; > | return 0; > | } > configure:3312: error: in > `/var/tmp/portage/cross-x86_64-w64-mingw32/mingw64-runtime-6.0.0-r1/work/ > mingw-w64-v6.0.0/mingw-w64-libraries/libmangle': > configure:3314: error: C compiler cannot create executables > See `config.log' for more details > > Trying to see if I have more luck with 5.0.4. To effectively debug it I need a command you ran and full build.log. For example was it headers-only install or not. Just tried crossdev --libc 5.0.4, then another emerge with USE=libraries for that version of mingw64-runtime, and everything works including pthreads. I will try again with 6.0.0-r1 and post all logs a bit later. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=7d94614beac8d7c390bc79da7be1a61b70c89a7b commit 7d94614beac8d7c390bc79da7be1a61b70c89a7b Author: Sergei Trofimovich <slyfox@gentoo.org> AuthorDate: 2018-10-06 22:53:51 +0000 Commit: Sergei Trofimovich <slyfox@gentoo.org> CommitDate: 2018-10-06 22:59:24 +0000 dev-util/mingw64-runtime: revert "enable USE=libraries by default" This reverts commit 478d91086baa442fb578e0982063b93a19caf382. Fresh bootstrap fails to build libraries as those need full stage2 compiler: ``` $ crossdev -t x86_64-w64-mingw32 -P -v * Emerging cross-mingw64-runtime ... ... configure: error: in `.../mingw-w64-v6.0.0/mingw-w64-libraries/libmangle': configure: error: C compiler cannot create executables See `config.log' for more details ``` Disable default USE=libraries. Reported-by: Rafael Kitover Bug: https://bugs.gentoo.org/665512 Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> dev-util/mingw64-runtime/mingw64-runtime-6.0.0-r1.ebuild | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) This still leaves the question about whether pthreads can be enabled by default in some way. If not, I'll just add a note to the mingw page on the wiki. I would suggest working with upstream to make --with-libraries work on a system without mingw installed. While at it it would be nice to fix use of installed headers. We wourkaround headers mismatch via tmproot/include install:
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-util/mingw64-runtime/mingw64-runtime-6.0.0-r1.ebuild#n63
Until then I avoid would enabling it by default in gentoo as it needs more fiddling.
Well thank you for trying! The wiki is a bit confusing, I'll see if I can make any improvements there and I'll see if anything can be done upstream for the libraries. Another problem with the USE=libraries option, since you can only use it the second time you build it, you have to put it in your use flags for updates to not break it, but then you have to remember to remove it from your use flags when you're building a fresh toolchain. Well I finally got my project to build with mingw crossdev, previously I got it working on msys2, cygwin, fedora mingw and mingw-w64 brew in mac homebrew. Another thing I wanted to mention, there is some kind of problem with the mingw headers and I needed to add -DMINGW_HAS_SECURE_API to my CPPFLAGS. Using crossdev's mingw-w64, it's not possible to compile very simple C++ code. Consider following file:
#include <mutex>
std::mutex x;
Then try to compile it with crossdev's mingw-w64:
$ LANG=C i686-w64-mingw32-g++ -c test.cpp
test.cpp:3:6: error: 'mutex' in namespace 'std' does not name a type
3 | std::mutex x;
| ^~~~~
test.cpp:2:1: note: 'std::mutex' is defined in header '<mutex>'; did you forget to '#include <mutex>'?
1 | #include <mutex>
+++ |+#include <mutex>
2 |
As you can see, the example does not even compile. Standard C++ mutexes don't seem to be available. On Arch, this works just fine, but then again, Arch AUR mingw-w64 provides winpthread.
In conclusion, I would consider Gentoo's mingw-w64 Toolchain broken as far as C++ multithreading is concerned.
Basically, due to issues of build dependencies and their required order, the maintainers of these ebuilds believe we should improve the situation upstream in the mingw64 packages. I have not yet had time to look at this, would you be willing to look at the situation? Libstdc++ basically requires winpthreads to provide proper C++ threading support. There are standard C++ libraries for Windows which don't have this requirement like LLVM's libc++ but building it requires more tinkering around build system. You can see example here: https://github.com/mstorsjo/llvm-mingw/blob/master/build-libcxx.sh media-libs/openal can't compile without std::mutex Any progress? Many distributions (e.g., ubuntu, arch) provide a mingw-w64 toolchain with winpthread. It is required for libstdc++ to work. It depends on posix threads, mutexes, conditions variables, etc. These APIs are emulated by winpthreads.
On Ubuntu 23.04: apt install g++-mingw-w64-x86-64-posix
On Arch: pacman -S mingw-w64-gcc
All of the above include winpthread and allow the following simple test program to compile and link.
#include <mutex>
int main() {
std::mutex m;
std::unique_lock lock{m};
}
I was hoping someone else would handle this given I know little about mingw, but ionen's explanation on IRC at least helped me get the gist. so, to get posix threads, we have to do a third build / stage3, but crossdev doesn't do that right now (because you need to build mingw winpthreads, then build another gcc actually using it) anyone fancy working on it? mingw64-toolchain ebuild could be a rough reference, aka currently it does this ordering:
mwt-build binutils
mwt-build mingw64 headers
mwt-build gcc stage1
mwt-build -x mingw64 runtime
mwt-build gcc stage2
mwt-build -x mingw64 libraries # cross-compiles winpthreads
mwt-build gcc stage3 # --enable-threads=posix
Perhaps there's way to optimize this and reduce to 2 stages, but have not really experimented.
In current state, just emerging cross-*/gcc after crossdev steps works, albeit have to rely on EXTRA_ECONF. Doing it before have winpthreads will just give headers not found errors.
|
