Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bugzilla DB migration completed. Please report issues to Infra team via email via infra@gentoo.org or IRC
Bug 665512 - sys-devel/crossdev: support winpthreads by default for mingw (<mutex> and friends does not work)
Summary: sys-devel/crossdev: support winpthreads by default for mingw (<mutex> and fri...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Gentoo Crossdev team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-09-08 16:49 UTC by Rafael Kitover
Modified: 2020-07-10 22:19 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Rafael Kitover 2018-09-08 16:49:43 UTC
Other mingw toolkits enable the gcc pthread support by default, such as MSys2.

This is simple to do and does no harm.

It's practically impossible to build any actual software for windows without pthread support, which makes the crossdev mingw support useless.

The workaround described in the wiki:

https://wiki.gentoo.org/wiki/Mingw#Porting_POSIX_threads_to_Windows

is very bad compared to the gcc support for pthreads with winpthreads.

All that is necessary to do this is to build winpthreads which is in the mingw-w64 distribution and compile gcc with --enable-threads=posix .

I could work on a patch for this if there is interest.
Comment 1 Sergei Trofimovich gentoo-dev 2018-09-22 18:45:08 UTC
The threading is decided around:
    https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/toolchain.eclass#n1066

	case ${CHOST} in
	mingw*|*-mingw*)
		confgcc+=( --enable-threads=win32 ) ;;
	*)
		confgcc+=( --enable-threads=posix ) ;;
	esac

I don't understand what proposed change is supposed to do. Will it just provide one more -lpthread library with pthread_* symbols implemented?

Will it also change threading primitives used by gcc to use that library instead of windows.h primitives? (Like c++'s std::thread implementation).
Comment 2 Rafael Kitover 2018-10-04 00:53:56 UTC
Here is the documentation I found on this option:

https://gcc.gnu.org/install/configure.html

So, yes, it does affect C++, but in a way that is completely transparent to the programmer/user, using posix threads on windows does not hurt any of these features.

According to this:

https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Preprocessor-Options.html#Preprocessor-Options

the '-pthread' option turns on preprocessor macros for using POSIX threads in the compile phase.

In the link phase, it adds the pthread library:

https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Link-Options.html#Link-Options

The pthread library is built along with gcc on mingw targets and is a runtime dependency, but it can easily be linked statically.

For example, this should work everywhere:

gcc <objects> -o bin -Wl,-Bstatic -lpthread -Wl,-Bdynamic

Now as to the WHY.

Suppose you are building the windows port of your favorite open source project. This by the way is the most common use case for a mingw toolchain. Otherwise you're just using visual studio.

It will either itself, or in some libraries it depends on, require POSIX threads.

Because these are all linux libraries that have windows ports.

Adjusting all of your build system related stuff to use some kind of crappy pthreads replacement instead of using the gcc one like everyone else is a huge and unnecessary pain in the ass.

For this reason, the popular distribution msys2 which is a mingw toolchain for windows with a slightly tweaked cygwin layer for some tools and the shell:

https://www.msys2.org/

builds gcc with POSIX threads.

I was recently fighting with one of the mac homebrew devs to also default to this option for the mingw-w64 brew, and he eventually agreed seeing that this was also the default for msys2.
Comment 3 Sergei Trofimovich gentoo-dev 2018-10-04 22:04:35 UTC
(In reply to Rafael Kitover from comment #2)
> Here is the documentation I found on this option:
> 
> https://gcc.gnu.org/install/configure.html
> 
> So, yes, it does affect C++, but in a way that is completely transparent to
> the programmer/user, using posix threads on windows does not hurt any of
> these features.

My main interest here is what actually happens in gcc to assess the impact of change. For example:
- how painful it will be for people who already have a mingw toolchain built to update gcc
- what could regress as a result

I don't see any mention of implementation changes WRT exposed interfaces.
Only about 'single' vs. non-'single'.

Looking at gcc source code it seems to only affect:
- what 'gcc -v' reports in threading model
- libatomic/libgomp/libitm implement a few primitives based on pthreads instead of win32. See lib{atomic,gomp,itm}/config/posix/.

That is not much. Does mingw64-runtime actually change it's behaviour based on default threading model?

Looking at mingw64-runtime's configure.ac --with-libraries=yes|all is supposed to provide winpthreads. That's what IUSE=libraries does.

Something like
    $ USE=libraries emerge -v1 cross-x86_64-w64-mingw32/mingw64-runtime
Does it not work as-is for you?

I still fail to see material difference that --enable-threads=posix for gcc provides.

> According to this:
> 
> https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Preprocessor-Options.
> html#Preprocessor-Options
...

AFAIU this is how gcc behaves regardless of --enable-threads=posix.
Comment 4 Rafael Kitover 2018-10-06 16:35:00 UTC
(In reply to Sergei Trofimovich from comment #3)
> My main interest here is what actually happens in gcc to assess the impact
> of change. For example:
> - how painful it will be for people who already have a mingw toolchain built
> to update gcc
> - what could regress as a result

That's a good thing to consider, and I don't really know, but I doubt the choice of gcc threading primitives would break much of anything. The only issue is having to link libpthread, which is not that big of a deal.

> Looking at mingw64-runtime's configure.ac --with-libraries=yes|all is
> supposed to provide winpthreads. That's what IUSE=libraries does.
> 
> Something like
>     $ USE=libraries emerge -v1 cross-x86_64-w64-mingw32/mingw64-runtime
> Does it not work as-is for you?

So I tried this for my build system, and it seems to work more or less. All the pthread related options work for gcc. But I have a lot of issues with this setup and I haven't built my project yet. Some of my issues are probably caused by my cross gcc being 6.4.0 because of #665508, which I haven't figured out yet.

I had to add -L/usr/x86_64-w64-mingw32/usr/lib64 to my LDFLAGS. Why is it even lib64 for a single arch sysroot? Is this because the old gcc doesn't support any kind of multilib stuff?

Assuming this works properly with a newer gcc and the search paths issue is not present, or can be fixed, can this 'libraries' use flag be default or required so that pthreads is always available? Especially since users will be using crossdev to build the whole toolchain and not emerging packages individually.
Comment 5 Sergei Trofimovich gentoo-dev 2018-10-06 17:18:29 UTC
> So I tried this for my build system, and it seems to work more or less. All
> the pthread related options work for gcc. But I have a lot of issues with
> this setup and I haven't built my project yet. Some of my issues are
> probably caused by my cross gcc being 6.4.0 because of #665508, which I
> haven't figured out yet.
> 
> I had to add -L/usr/x86_64-w64-mingw32/usr/lib64 to my LDFLAGS. Why is it
> even lib64 for a single arch sysroot? Is this because the old gcc doesn't
> support any kind of multilib stuff?

This is just a bug in cross-*/mingw64-runtime packages. It uses $(get_libdir) from host profile (amd64-linux) and not target profile. It's a bug #653246 unrelated to gcc's multilib handling.

> Assuming this works properly with a newer gcc and the search paths issue is
> not present, or can be fixed, can this 'libraries' use flag be default or
> required so that pthreads is always available? Especially since users will
> be using crossdev to build the whole toolchain and not emerging packages
> individually.

Enabling libraries by default makes sense. We can try it.
Comment 6 Larry the Git Cow gentoo-dev 2018-10-06 17:35:28 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=478d91086baa442fb578e0982063b93a19caf382

commit 478d91086baa442fb578e0982063b93a19caf382
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2018-10-06 17:35:13 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2018-10-06 17:35:21 +0000

    dev-util/mingw64-runtime: enable USE=libraries by default
    
    Enable mangle, pseh, winpthreads libraries by default.
    Notably this exposes libpthread.a compatible library and
    makes building mode pthread-based software.
    
    Reported-by: Rafael Kitover
    Bug: https://bugs.gentoo.org/665512
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
    Package-Manager: Portage-2.3.50, Repoman-2.3.11

 dev-util/mingw64-runtime/metadata.xml                    | 12 +++---------
 dev-util/mingw64-runtime/mingw64-runtime-6.0.0-r1.ebuild |  2 +-
 2 files changed, 4 insertions(+), 10 deletions(-)
Comment 7 Rafael Kitover 2018-10-06 21:44:00 UTC
Just tried building a new toolchain on an updated system with all traces of previous builds removed.

I am getting a failure building mingw64-runtime-6.0.0-r1, in libmangle:

configure:3243: checking whether the C compiler works
configure:3265: x86_64-w64-mingw32-gcc -march=native -O2 -pipe  -I/var/tmp/portage/cross-x86_64-w64-mingw32/mingw64-runtime-6.0.0-r1/temp/tmproot/include -Wl,-O1 -Wl,--as-needed conftest.c  >&5
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find crt2.o: No such file or directory
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingw32
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmoldname
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingwex
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmsvcrt
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -ladvapi32
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lshell32
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -luser32
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lkernel32
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingw32
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmoldname
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingwex
/usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmsvcrt
collect2: error: ld returned 1 exit status
configure:3269: $? = 1
configure:3307: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "mingw-w64-libmangle"
| #define PACKAGE_TARNAME "mingw-w64-libmangle"
| #define PACKAGE_VERSION "1.0"
| #define PACKAGE_STRING "mingw-w64-libmangle 1.0"
| #define PACKAGE_BUGREPORT "mingw-w64-public@lists.sf.net"
| #define PACKAGE_URL ""
| #define PACKAGE "mingw-w64-libmangle"
| #define VERSION "1.0"
| /* end confdefs.h.  */
| 
| int
| main ()
| {
| 
|   ;
|   return 0;
| }
configure:3312: error: in `/var/tmp/portage/cross-x86_64-w64-mingw32/mingw64-runtime-6.0.0-r1/work/mingw-w64-v6.0.0/mingw-w64-libraries/libmangle':
configure:3314: error: C compiler cannot create executables
See `config.log' for more details

Trying to see if I have more luck with 5.0.4.
Comment 8 Sergei Trofimovich gentoo-dev 2018-10-06 22:00:46 UTC
(In reply to Rafael Kitover from comment #7)
> Just tried building a new toolchain on an updated system with all traces of
> previous builds removed.
> 
> I am getting a failure building mingw64-runtime-6.0.0-r1, in libmangle:
> 
> configure:3243: checking whether the C compiler works
> configure:3265: x86_64-w64-mingw32-gcc -march=native -O2 -pipe 
> -I/var/tmp/portage/cross-x86_64-w64-mingw32/mingw64-runtime-6.0.0-r1/temp/
> tmproot/include -Wl,-O1 -Wl,--as-needed conftest.c  >&5
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find crt2.o: No such file or
> directory
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingw32
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmoldname
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingwex
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmsvcrt
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -ladvapi32
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lshell32
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -luser32
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lkernel32
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingw32
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmoldname
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmingwex
> /usr/libexec/gcc/x86_64-w64-mingw32/ld: cannot find -lmsvcrt
> collect2: error: ld returned 1 exit status
> configure:3269: $? = 1
> configure:3307: result: no
> configure: failed program was:
> | /* confdefs.h */
> | #define PACKAGE_NAME "mingw-w64-libmangle"
> | #define PACKAGE_TARNAME "mingw-w64-libmangle"
> | #define PACKAGE_VERSION "1.0"
> | #define PACKAGE_STRING "mingw-w64-libmangle 1.0"
> | #define PACKAGE_BUGREPORT "mingw-w64-public@lists.sf.net"
> | #define PACKAGE_URL ""
> | #define PACKAGE "mingw-w64-libmangle"
> | #define VERSION "1.0"
> | /* end confdefs.h.  */
> | 
> | int
> | main ()
> | {
> | 
> |   ;
> |   return 0;
> | }
> configure:3312: error: in
> `/var/tmp/portage/cross-x86_64-w64-mingw32/mingw64-runtime-6.0.0-r1/work/
> mingw-w64-v6.0.0/mingw-w64-libraries/libmangle':
> configure:3314: error: C compiler cannot create executables
> See `config.log' for more details
> 
> Trying to see if I have more luck with 5.0.4.

To effectively debug it I need a command you ran and full build.log. For example was it headers-only install or not.
Comment 9 Rafael Kitover 2018-10-06 22:36:33 UTC
Just tried crossdev --libc 5.0.4, then another emerge with USE=libraries for that version of mingw64-runtime, and everything works including pthreads.

I will try again with 6.0.0-r1 and post all logs a bit later.
Comment 10 Larry the Git Cow gentoo-dev 2018-10-06 22:59:40 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=7d94614beac8d7c390bc79da7be1a61b70c89a7b

commit 7d94614beac8d7c390bc79da7be1a61b70c89a7b
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2018-10-06 22:53:51 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2018-10-06 22:59:24 +0000

    dev-util/mingw64-runtime: revert "enable USE=libraries by default"
    
    This reverts commit 478d91086baa442fb578e0982063b93a19caf382.
    
    Fresh bootstrap fails to build libraries as those need full stage2
    compiler:
    
    ```
    $ crossdev -t x86_64-w64-mingw32 -P -v
    
     * Emerging cross-mingw64-runtime ...
    ...
    configure: error: in `.../mingw-w64-v6.0.0/mingw-w64-libraries/libmangle':
    configure: error: C compiler cannot create executables
    See `config.log' for more details
    ```
    
    Disable default USE=libraries.
    
    Reported-by: Rafael Kitover
    Bug: https://bugs.gentoo.org/665512
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 dev-util/mingw64-runtime/mingw64-runtime-6.0.0-r1.ebuild | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Comment 11 Rafael Kitover 2018-10-07 00:06:24 UTC
This still leaves the question about whether pthreads can be enabled by default in some way. 

If not, I'll just add a note to the mingw page on the wiki.
Comment 12 Sergei Trofimovich gentoo-dev 2018-10-07 07:05:32 UTC
I would suggest working with upstream to make --with-libraries work on a system without mingw installed. While at it it would be nice to fix use of installed headers. We wourkaround headers mismatch via tmproot/include install:
    https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-util/mingw64-runtime/mingw64-runtime-6.0.0-r1.ebuild#n63

Until then I avoid would enabling it by default in gentoo as it needs more fiddling.
Comment 13 Rafael Kitover 2018-10-07 08:41:38 UTC
Well thank you for trying!

The wiki is a bit confusing, I'll see if I can make any improvements there and I'll see if anything can be done upstream for the libraries.
Comment 14 Rafael Kitover 2018-10-10 20:16:34 UTC
Another problem with the USE=libraries option, since you can only use it the second time you build it, you have to put it in your use flags for updates to not break it, but then you have to remember to remove it from your use flags when you're building a fresh toolchain.
Comment 15 Rafael Kitover 2018-10-11 14:03:24 UTC
Well I finally got my project to build with mingw crossdev, previously I got it working on msys2, cygwin, fedora mingw and mingw-w64 brew in mac homebrew.

Another thing I wanted to mention, there is some kind of problem with the mingw headers and I needed to add -DMINGW_HAS_SECURE_API to my CPPFLAGS.
Comment 16 Sven 2019-11-07 13:49:12 UTC
Using crossdev's mingw-w64, it's not possible to compile very simple C++ code. Consider following file:

#include <mutex>
std::mutex x;

Then try to compile it with crossdev's mingw-w64:

$ LANG=C i686-w64-mingw32-g++ -c test.cpp
test.cpp:3:6: error: 'mutex' in namespace 'std' does not name a type
    3 | std::mutex x;
      |      ^~~~~
test.cpp:2:1: note: 'std::mutex' is defined in header '<mutex>'; did you forget to '#include <mutex>'?
    1 | #include <mutex>
  +++ |+#include <mutex>
    2 | 


As you can see, the example does not even compile. Standard C++ mutexes don't seem to be available. On Arch, this works just fine, but then again, Arch AUR mingw-w64 provides winpthread.

In conclusion, I would consider Gentoo's mingw-w64 Toolchain broken as far as C++ multithreading is concerned.
Comment 17 Rafael Kitover 2019-11-07 20:58:41 UTC
Basically, due to issues of build dependencies and their required order, the maintainers of these ebuilds believe we should improve the situation upstream in the mingw64 packages. I have not yet had time to look at this, would you be willing to look at the situation?
Comment 18 Mateusz Mikuła 2020-05-17 08:48:50 UTC
Libstdc++ basically requires winpthreads to provide proper C++ threading support.
There are standard C++ libraries for Windows which don't have this requirement like LLVM's libc++ but building it requires more tinkering around build system. You can see example here: https://github.com/mstorsjo/llvm-mingw/blob/master/build-libcxx.sh