Created attachment 412360 [details, diff] 0001-mysql-v2-multilib-.eclass-add-libressl-support-and-a.patch Since mysql eclass is tightly integrated in several packages, we have to adjust these in one blow, revbump the latest ones in/to unstable arch, fix the virtuals to propagate the libressl USE flag and mask libressl USE flag for stable arch of these packages, because dev-libs/libressl is not in stable arch yet. https://github.com/gentoo/gentoo/pull/109
I will apply this in two weeks.
(In reply to Julian Ospald (hasufell) from comment #1) > I will apply this in two weeks. If you insist on rushing this, you must runtime test all changes. OpenSSL is tied to more than just simple connections. There are SQL functions that depend on it as well. SHA2() for example. Please run the entire test suite in the official instructions in every package to know if anything breaks.
(In reply to Brian Evans from comment #2) > (In reply to Julian Ospald (hasufell) from comment #1) > > I will apply this in two weeks. > > If you insist on rushing this, you must runtime test all changes. I am fine with waiting, if I know that someone will actually take a look at this. Otherwise I follow the normal procedure of "maintainer timeout". > OpenSSL > is tied to more than just simple connections. There are SQL functions that > depend on it as well. SHA2() for example. > > Please run the entire test suite in the official instructions in every > package to know if anything breaks. Can you help me with that?
As I thought, the server needs to be patched for this: -- OPENSSL_INCLUDE_DIR = /usr/include -- OPENSSL_LIBRARY = /usr/lib64/libssl.so -- CRYPTO_LIBRARY = /usr/lib64/libcrypto.so -- OPENSSL_MAJOR_VERSION = 2 -- Looking for SHA512_DIGEST_LENGTH -- Looking for SHA512_DIGEST_LENGTH - found CMake Error at cmake/ssl.cmake:247 (MESSAGE): Cannot find appropriate system libraries for SSL. Make sure you've specified a supported SSL version. Consult the documentation for WITH_SSL alternatives Call Stack (most recent call first): CMakeLists.txt:407 (MYSQL_CHECK_SSL)
(In reply to Brian Evans from comment #4) > As I thought, the server needs to be patched for this: > > -- OPENSSL_INCLUDE_DIR = /usr/include > -- OPENSSL_LIBRARY = /usr/lib64/libssl.so > -- CRYPTO_LIBRARY = /usr/lib64/libcrypto.so > -- OPENSSL_MAJOR_VERSION = 2 > -- Looking for SHA512_DIGEST_LENGTH > -- Looking for SHA512_DIGEST_LENGTH - found > CMake Error at cmake/ssl.cmake:247 (MESSAGE): > Cannot find appropriate system libraries for SSL. Make sure you've > specified a supported SSL version. Consult the documentation for WITH_SSL > alternatives > Call Stack (most recent call first): > CMakeLists.txt:407 (MYSQL_CHECK_SSL) https://github.com/gentoo/libressl/blob/master/dev-db/mysql/files/mysql-5.6.22-libressl.patch
I have a concern that the MD5 hashing support might be dropped at some point. The MD5() SQL function relies on the ssl library to provide this function. It is not deprecated even in the latest RC of MySQL.
mysql works fine with dev-libs/libressl-2.2.3 which is in the tree We can't look into the future. It may require further patching, but libressl is not just used by OpenBSD and as such the pressure is high to get upstream compatible patches. Mysql is a popular consumer. I don't see why such general doubts should block this. You probably won't get a definitive answer from upstream.
I am getting segfaults in my test environments (both a VM and an LXC) with dev-db/mariadb-10.1.7_rc with the innodb encryption tests. These tests are not failing with OpenSSL linked. This needs further investigation before proceeding.
interesting, maybe we should be able to dig something up at OpenBSD/FreeBSD resources, since they definitely use mariadb http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/databases/mariadb/ https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-client https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-server Did you test dev-db/mariadb-10.0.21-r1 too?
(In reply to Julian Ospald (hasufell) from comment #9) > interesting, maybe we should be able to dig something up at OpenBSD/FreeBSD > resources, since they definitely use mariadb > > http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/databases/mariadb/ > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-client > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-server > > Did you test dev-db/mariadb-10.0.21-r1 too? This is a new feature to 10.1, so 10.0 won't show it.
(In reply to Brian Evans from comment #10) > (In reply to Julian Ospald (hasufell) from comment #9) > > interesting, maybe we should be able to dig something up at OpenBSD/FreeBSD > > resources, since they definitely use mariadb > > > > http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/databases/mariadb/ > > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-client > > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-server > > > > Did you test dev-db/mariadb-10.0.21-r1 too? > > This is a new feature to 10.1, so 10.0 won't show it. since it's just an rc anyway one possibility would be to hardmask it (either completely or just the libressl USE flag for this package), revbump 10.0.21-r1 for libressl support and file bugs upstream
(In reply to Julian Ospald (hasufell) from comment #11) > > since it's just an rc anyway one possibility would be to hardmask it (either > completely or just the libressl USE flag for this package), for clarification, I meant an entry in profiles/base/package.use.mask like so: >=dev-db/mariadb-10.1 libressl That's the good thing about our approach, it's a USE flag and no "|| ( )" depstring, so we have proper control over it.
(In reply to Julian Ospald (hasufell) from comment #9) > interesting, maybe we should be able to dig something up at OpenBSD/FreeBSD > resources, since they definitely use mariadb > > http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/databases/mariadb/ > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-client > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-server > > Did you test dev-db/mariadb-10.0.21-r1 too? Let us discuss FreeBSD, https://github.com/freebsd/freebsd-ports/blob/master/databases/mariadb100-server/Makefile seems like it wants OpenSSL or bundled. I cannot follow how LibreSSL would be pulled in. I'm of the opinion to not support libressl since upstream does not. To make it easier on users, s/ssl/openssl/ USE flag since ssl must always be on in >=5.6. This becomes very visible to switch between the supported methods on a critical system service.
(In reply to Brian Evans from comment #13) > (In reply to Julian Ospald (hasufell) from comment #9) > > interesting, maybe we should be able to dig something up at OpenBSD/FreeBSD > > resources, since they definitely use mariadb > > > > http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/databases/mariadb/ > > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-client > > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-server > > > > Did you test dev-db/mariadb-10.0.21-r1 too? > > Let us discuss FreeBSD, > https://github.com/freebsd/freebsd-ports/blob/master/databases/mariadb100- > server/Makefile seems like it wants OpenSSL or bundled. I cannot follow how > LibreSSL would be pulled in. > > I'm of the opinion to not support libressl since upstream does not. To make > it easier on users, s/ssl/openssl/ USE flag since ssl must always be on in > >=5.6. This becomes very visible to switch between the supported methods on > a critical system service. If not even critical server packages like mysql and mariadb in gentoo want to support libressl we can as well abort the transition and revert all commits. This will not get us very far. And that's probably what I will do then, since these packages are one of the reasons I even do this transition.
(In reply to Brian Evans from comment #13) > (In reply to Julian Ospald (hasufell) from comment #9) > > interesting, maybe we should be able to dig something up at OpenBSD/FreeBSD > > resources, since they definitely use mariadb > > > > http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/databases/mariadb/ > > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-client > > https://github.com/freebsd/freebsd-ports/tree/master/databases/mariadb-server > > > > Did you test dev-db/mariadb-10.0.21-r1 too? > > Let us discuss FreeBSD, > https://github.com/freebsd/freebsd-ports/blob/master/databases/mariadb100- > server/Makefile seems like it wants OpenSSL or bundled. I cannot follow how > LibreSSL would be pulled in. > I don't understand what you are trying to tell me. I don't know how FreeBSD Makefiles work. If you look at the OpenBSD link you can see that they do use mariadb 10.0.21 and you can be pretty damn sure that people use that in production. > I'm of the opinion to not support libressl since upstream does not. To make > it easier on users, s/ssl/openssl/ USE flag since ssl must always be on in > >=5.6. This becomes very visible to switch between the supported methods on > a critical system service. If gentoo only supported things that are strictly supported upstream, it would be a dead distro. Instead we should add support downstream and send patches upstream. So far, I've really not seen a good argument, except being lazy and just following upstream without improving the linux ecosystem in general. Yes, adding LibreSSL support will mean we all have to do a bit more work... and it will mean to improve overall security which is a proven fact (see the number of LibreSSL/OpenSSL vulnerabilities on wikipedia). This kind of behavior is really depressing after I have added LibreSSL support to ~100 packages _alone_ and now have to fight various maintainers on various bugs. I'm not really interested in that stress. Instead I will probably just abort this thing and keep everything in an overlay indefinitely.
(In reply to Julian Ospald (hasufell) from comment #14) <snip> > If not even critical server packages like mysql and mariadb in gentoo want > to support libressl we can as well abort the transition and revert all > commits. This will not get us very far. There are several issues here: 1. We (MySQL maintainers) are not the ones behind this move to support libressl. 2. It doesn't matter as much if Gentoo MySQL maintainers want to support libressl, as whether upstream is likely to support it and there's someone to do the work to keep MySQL / MariaDB working with libressl. 3. At least I, would even like to see libressl succeed, but that doesn't mean I have the extra time that I fear would be required to keep this working. Also, I definitely don't want to risk getting this thrown in my lap in 6 months / 1 year from now, when this breaks on a bump, and have no one around to support it. 4. As you noted above, for most, these are "critical server packages". This is definitely not the type of package you want to "experiment" with. Any issues with the SSL implementation might cause data loss, information disclosure or worse. 5. Because of all of the above and my impression that MariaDB is easier to approach, quicker to react and more open to patches, I feel we should consider enabling support for libressl for mariadb, but not for MySQL or Percona. 6. Finally, we should do a news item or make sure we make it clear and loud to everyone the impact the use of libressl can / will have in MariaDB. > And that's probably what I will do then, since these packages are one of the > reasons I even do this transition. Given the "critical" nature, you should expect some resistance and concerns. That doesn't mean you should throw out your work, but make sure you work closely with others.
(In reply to Julian Ospald (hasufell) from comment #15) > So far, I've really not seen a good argument, except being lazy and just > following upstream without improving the linux ecosystem in general. Yes, > adding LibreSSL support will mean we all have to do a bit more work... and > it will mean to improve overall security which is a proven fact (see the > number of LibreSSL/OpenSSL vulnerabilities on wikipedia). > > > This kind of behavior is really depressing after I have added LibreSSL > support to ~100 packages _alone_ and now have to fight various maintainers > on various bugs. I'm not really interested in that stress. Instead I will > probably just abort this thing and keep everything in an overlay > indefinitely. I'll give good reasons soon enough. The tests failed last night on MariaDB-10.0.21 specifically on SSL. It took all night and I have not looked in-depth just yet. Planning on that in a few hours. I don't see what the problem is. MariaDB/MySQL is not crippled by turning off OpenSSL and not having LibreSSL. It falls back to an internal implementation.
(In reply to Jorge Manuel B. S. Vicetto from comment #16) > (In reply to Julian Ospald (hasufell) from comment #14) > <snip> > > > If not even critical server packages like mysql and mariadb in gentoo want > > to support libressl we can as well abort the transition and revert all > > commits. This will not get us very far. > > There are several issues here: > 1. We (MySQL maintainers) are not the ones behind this move to support > libressl. > 2. It doesn't matter as much if Gentoo MySQL maintainers want to support > libressl, as whether upstream is likely to support it and there's someone to > do the work to keep MySQL / MariaDB working with libressl. OpenBSD uses mariadb, so they are clearly going to patch things if needed. I don't see that mysql currently requires any patching (except a _very_ trivial cmake patch) and I have it installed on production servers with libressl. What exactly is the issue here? > 3. At least I, would even like to see libressl succeed, but that doesn't > mean I have the extra time that I fear would be required to keep this > working. Also, I definitely don't want to risk getting this thrown in my lap > in 6 months / 1 year from now, when this breaks on a bump, and have no one > around to support it. > 4. As you noted above, for most, these are "critical server packages". This > is definitely not the type of package you want to "experiment" with. Any > issues with the SSL implementation might cause data loss, information > disclosure or worse. Again: People are already using mysql/mariadb in conjunction with libressl in production. I don't see how you can call that "experimenting", unless you call using OpenSSL "experimenting" too which would be a pretty good point.
(In reply to Brian Evans from comment #17) > > I don't see what the problem is. MariaDB/MySQL is not crippled by turning > off OpenSSL and not having LibreSSL. It falls back to an internal > implementation. The problem is that I now have to trust an internal implementation. How is that an improvement? It's even worse than using OpenSSL.
Created attachment 413940 [details] dev-db/mariadb-10.0.21-r1 test failures While most are trivial defaults differences, the main.openssl_1 test concerns me that proper support has been removed.
So, I will probably fork mysql and mariadb ebuilds which means we will have to introduce virtuals. That will allow us to develop them independently and hardmask the libressl versions with notes that there are test failures that are not properly investigated yet.
This comment discusses only the cipher issue, I'll post a separate one for other review. ssl.test, ssl_compress.test, ssl_timeout.test: These testcases are making an assumption about the cipher config. It's already been tweaked once to treat DHE-RSA-AES256-GCM-SHA384 the same as DHE-RSA-AES256-SHA. I think it should be changed to explicitly request a given cipher, or accept ALL of them. openssl_1: MySQL is requesting a SPECIFIC cipher, to test that requesting a specific cipher works. The cipher config in question is gone from LibreSSL. If Hasufell wants us to accept this, he needs to tweak the requested ciphers to something that is supported by OLD OpenSSL, Newer OpenSSL AND LibreSSL. Needs to have both a DHE and non-DHE variant, and must NOT be the default cipher in any of the three (one of the tests is to require a specific cipher on server, but the client connects with default).
Other comments about the patch: 1. Please DROP the changes to mysql-v2.eclass, and the ebuilds that use it. These are stable/legacy only, and we WILL be dropping support for them when upstream does (sometime in the next 6 months I expect). 2. Past that, I'm going to take the same approach as upstream: If a user reports a bug and they are using libressl, and it LOOKS likely that it's related, or we can't explain the crash any other way, we WILL require that they reproduce the bug with OpenSSL.
I don't think I'm going to invest time in this when there is so much FUD floating around about converting these ebuilds. So I think forking the ebuilds is a better approach. Users can still choose and you don't have to deal with anything libressl related. I might still fix the test cases and send patches upstream, but for my ebuild forks this will not be a blocker.
(In reply to Julian Ospald (hasufell) from comment #24) > I don't think I'm going to invest time in this when there is so much FUD > floating around about converting these ebuilds. How exactly are test failures FUD? It's clear and visible. Brian's concern about libressl entirely removing MD5 is very tangential, and NOT an issue, because the package will fail to compile and/or fail the testsuite. > So I think forking the > ebuilds is a better approach. Users can still choose and you don't have to > deal with anything libressl related. Since most of the SSL pieces are actually in the eclass, that means forking the eclass as well, which will be a giant mess; you'll make changes and diverge from the origin. The >=5.6/10.x series REQUIRE a SSL backend to compile. USE=-ssl => builds with yaSSL (upstream default) USE=+ssl => builds with OpenSSL If we want to be clear, we should DROP USE=ssl, and instead have: REQUIRED_USE="^^ ( yassl openssl libressl )" If you try to report an SSL-related failure to Oracle, they ALREADY require that you confirm it exists on yassl. MariaDB is more lenient, they'll try to confirm it themselves. > I might still fix the test cases and send patches upstream, but for my > ebuild forks this will not be a blocker. Wherever possible, the MySQL team only ships ebuilds that pass the upstream test suite. If you want to fork it, JUST to to avoid running the test suite, you're doing users a dis-service. Furthermore, I do agree with Brian that we need to arrange testsuite runs where the client and server NOT both running libressl, but that is a much larger project, and should be deferred for now.
Please update the patch so that the testcases pass, post the testsuite result, and then we'll merge it. Any bugs with USE=libressl will be assigned to you (this is already done for other things like OpenSSH, where USE=hpn/lpk get assigned to me). Forking the ebuilds is going to make a mess of the tree.
(In reply to Robin Johnson from comment #25) > (In reply to Julian Ospald (hasufell) from comment #24) > > I don't think I'm going to invest time in this when there is so much FUD > > floating around about converting these ebuilds. > How exactly are test failures FUD? It's clear and visible. > > Brian's concern about libressl entirely removing MD5 is very tangential, and > NOT an issue, because the package will fail to compile and/or fail the > testsuite. > The previous comments on this bug report (and on github) sounded different, like "no mysql and mariadb only if upstream supports it". That's not a sentiment I am interested in fighting. If everything that keeps this from being adopted is the failing test cases, then I guess we can add libressl to mysql right now, no? Otherwise I will have a hard time believing that people are actually interested in this. And then, I'm better of forking the packages. > > So I think forking the > > ebuilds is a better approach. Users can still choose and you don't have to > > deal with anything libressl related. > Since most of the SSL pieces are actually in the eclass, that means forking > the eclass as well, which will be a giant mess; you'll make changes and > diverge from the origin. > If I fork the packages, then there will be no such eclass.
(In reply to Julian Ospald (hasufell) from comment #27) > If everything that keeps this from being adopted is the failing test cases, > then I guess we can add libressl to mysql right now, no? Otherwise I will > have a hard time believing that people are actually interested in this. And > then, I'm better of forking the packages. Make the testcases pass on BOTH MariaDB and MySQL, then we can add this. The tests Brian identified will fail on BOTH. mysql-5.6.27 also has some further SSL tests that will fail, and should be patched out (like using RC4 for a testcase eww). A downside of them is that it seems those tests have diverged between MySQL & MariaDB, and will need different fixes :-(. > > > So I think forking the > > > ebuilds is a better approach. Users can still choose and you don't have to > > > deal with anything libressl related. > > Since most of the SSL pieces are actually in the eclass, that means forking > > the eclass as well, which will be a giant mess; you'll make changes and > > diverge from the origin. > If I fork the packages, then there will be no such eclass. So you're going to fork the eclasses as well?
LibreSSL support is in the tree
(In reply to Brian Evans from comment #29) > LibreSSL support is in the tree Thanks a lot. Were you able to fix the test suites?
> REQUIRED_USE="^^ ( yassl openssl libressl )" this is a problem. 'libressl' and 'openssl' USE flag must be set in a lot of ebuilds at once to achieve openssl/ssl features and linkage to libressl, so this will cause weird depgraph behavior when people try to enable libressl globally. You could have asked for review of the patches first. We need a different solution to this. E.g. > RESTRICT="libressl? ( test )"
would you mind to test mysql without ssl support..? or do you enforce mysql with ssl now? !!! The ebuild selected to satisfy "dev-db/mysql" has unmet requirements. - dev-db/mysql-5.6.27-r1::gentoo USE="client-libs perl server tools (-cluster) -community -debug -embedded -extraengine -jemalloc -latin1 -libressl -openssl -profiling (-selinux) -static -static-libs -systemtap -tcmalloc -test -yassl" The following REQUIRED_USE flag constraints are unsatisfied: exactly-one-of ( yassl openssl libressl )
(In reply to Hanno Meyer-Thurow (geki) from comment #32) > would you mind to test mysql without ssl support..? or do you enforce mysql > with ssl now? > > !!! The ebuild selected to satisfy "dev-db/mysql" has unmet requirements. > - dev-db/mysql-5.6.27-r1::gentoo USE="client-libs perl server tools > (-cluster) -community -debug -embedded -extraengine -jemalloc -latin1 > -libressl -openssl -profiling (-selinux) -static -static-libs -systemtap > -tcmalloc -test -yassl" > > The following REQUIRED_USE flag constraints are unsatisfied: > exactly-one-of ( yassl openssl libressl ) MySQL 5.6 does not support building without any form of SSL
having a look at [0] and [1] you are wrong. see my current mysql 5.6 install: mysql> SHOW VARIABLES LIKE 'have_ssl'; +---------------+----------+ | Variable_name | Value | +---------------+----------+ | have_ssl | DISABLED | +---------------+----------+ 1 row in set (0,01 sec) how is that possible? magic? [0] https://dev.mysql.com/doc/refman/5.6/en/source-configuration-options.html [1] https://dev.mysql.com/doc/refman/5.6/en/building-with-ssl-support.html
(In reply to Hanno Meyer-Thurow (geki) from comment #34) > | have_ssl | DISABLED | > how is that possible? magic? > > [0] https://dev.mysql.com/doc/refman/5.6/en/source-configuration-options.html > [1] https://dev.mysql.com/doc/refman/5.6/en/building-with-ssl-support.html Huh? The latter says: "If the value is DISABLED, the server is capable of supporting SSL connections but was not started with the appropriate --ssl-xxx options to enable SSL to be used" - Note the "capable" part. The former says, wrt -DWITH_SSL=no: "As of 5.6.6, this is no longer a permitted value" - which seems to be the end of it.
