See Joe's comment (https://bugs.gentoo.org/889326#c15): > Just use bundled libsecp256k1 (and leveldb) with a new ebuild bitcoin-core? > Currently bitcoin-qt, bitcoind, bitcoin-tx, and libbitcoinconsensus are all > part of the same suite. The only other stuff that needs libsecp256k1 are > some electron/electrum stuff (no relation to the better known electron). Can > they just rely on a new bitcoin-core ebuild with a minimal number of > useflags? > > Just thinking almost everything else in portage that has this situation had > a useflag for qt for the qt interface. If people want bitcoin-tx, make it a > useflag for bitcoin-core, same for the other items. > > It's nearly a decade old but blueness knows about this, maybe Sam should > know: > https://docs.google.com/document/d/ > 1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit?pli=1#heading=h. > i7tz3gqh65mi > > Note the signatures were removed around the time of Luke's compromise but > the web archive still has them: > http://web.archive.org/web/20221224205226/http://luke.dashjr.org/tmp/code/ > 20130723-linux-distribution-packaging-and-bitcoin.md.asc I think this makes sense (merging all Bitcoin packages into one and using USE flags for gui and such. Right now: - We unbundle dev-libs/leveldb (but have to pin to a specific version). There's no note in the leveldb ebuild about the care that needs to be taken with older versions of leveldb used by Bitcoin, despite it being known since 2013 that some care should be taken; - We unbundle dev-libs/libsecp256k1 despite the fact it, until 2 weeks ago, had unstable ABI and the Bitcoin repo upstream indeed vendors this as well at specific commits. - We unbundled univalue until very recently, but dev-libs/univalue upstream is inactive and Bitcoin has forked this in its main repo with possible behaviour changes. All of this is pretty much risk with no gain. Even if we're very careful, there's the risk of getting something wrong, and even if we don't get something wrong, we're at the risk of someone _suggesting_ we did something wrong, and I take our reputation of technical competence gravely seriously. In addition, all of this as-is acts as a substantial maintenance burden, with any changes only being able to be "safely" done by one person - who hasn't documented any of this within the ebuilds. (So, it's not just a bus-factor issue, but also an issue if someone makes a well-meaning change.)
(In reply to Sam James from comment #0) > See Joe's comment (https://bugs.gentoo.org/889326#c15): > > The only other stuff that needs libsecp256k1 are > > some electron/electrum stuff That's just plain wrong. On my system alone, I have the following installed packages that depend on dev-libs/libsecp256k1: dev-python/coincurve dev-python/python-bitcointx dev-util/bitcoin-tx net-libs/libwally-core net-p2p/bitcoind net-p2p/joinmarket Please don't assume that all reverse dependencies of a package live in the main repository. We have numerous consumers of libsecp256k1 in the bitcoin-gentoo repo.
(In reply to Matt Whitlock from comment #1) > (In reply to Sam James from comment #0) > > See Joe's comment (https://bugs.gentoo.org/889326#c15): > > > The only other stuff that needs libsecp256k1 are > > > some electron/electrum stuff > > That's just plain wrong. On my system alone, I have the following installed > packages that depend on dev-libs/libsecp256k1: > > dev-python/coincurve > dev-python/python-bitcointx > net-libs/libwally-core > net-p2p/joinmarket These would all be candidates to keep using dev-libs/libsecp256k1, even if Bitcoin doesn't. Although I suspect python-bitcointx may want to use the same version as Bitcoin. > dev-util/bitcoin-tx > net-p2p/bitcoind These two would be merged into one package, per above, and use the vendored copy for the reasons described. > > Please don't assume that all reverse dependencies of a package live in the > main repository. We have numerous consumers of libsecp256k1 in the > bitcoin-gentoo repo. (I was quoting there.) Note that we don't generally go out of our way to keep a library with no consumers in ::gentoo just for the benefit of overlays/other repositories, you would be free to then put it in ::bitcoin-gentoo, but I'm not suggesting we'd last-rite dev-libs/libsecp256k1 at this point in time anyway.
I meant to add in my original summary of issues wrt maintenance burden: the split we currently do isn't supported by upstream, it takes longer to package and test, and the test suite doesn't work when components are missing anyway.
Sorry, but I cringe at this pervasive, creeping trend of "vendoring" libraries and statically linking them into application executables. Not only is it bad for memory efficiency since the code then can't be shared across applications, but worse, it's bad for security because then there are multiple copies that all need to be rebuilt whenever a vulnerability is fixed, and not all projects are diligent at keeping up to date on their dependencies. I much prefer to be able to upgrade a single copy of a library and know that every application that uses it will immediately benefit from any fixes in the new version. Gentoo's policy of aggressive library unbundling is one of the primary reasons I use and love Gentoo.
(In reply to Matt Whitlock from comment #4) > Sorry, but I cringe at this pervasive, creeping trend of "vendoring" > libraries and statically linking them into application executables. Not only > is it bad for memory efficiency since the code then can't be shared across > applications, but worse, it's bad for security because then there are > multiple copies that all need to be rebuilt whenever a vulnerability is > fixed, and not all projects are diligent at keeping up to date on their > dependencies. I much prefer to be able to upgrade a single copy of a library > and know that every application that uses it will immediately benefit from > any fixes in the new version. Gentoo's policy of aggressive library > unbundling is one of the primary reasons I use and love Gentoo. I don't like it either, but it's difficult when upstream actively don't care and make (sometimes substantial) changes to libraries which never even get submitted anywhere (e.g. univalue). But given libsecp is maintained as a separate repository, unlike the other quasi-forks Bitcoin makes, and it's actually used by a fair amount of packages, I think at the very least we could continue to offer it unbundled optionally. I agree.
(In reply to Sam James from comment #5) > I think at the very least we could continue to offer it unbundled > optionally. I wouldn't mind a USE="system-secp256k1" with an associated ewarn about being an unsanctioned configuration.
> Please don't assume that all reverse dependencies of a package live in the main repository. I didn't assume, it's just not my focus or priority to recommend what every other repository does. You can maintain the libraries on your own and set a blocker on the gentoo repository copy. bitcoin-gentoo can do things the old way if they want to. > Sorry, but I cringe at this pervasive, creeping trend of "vendoring" libraries and statically linking them into application executables. You may cringe at this, use another repo, or don't use bitcoin. It's not worth putting users security at risk mixing incompatible libraries that might break consensus because you want to attain a more ideal system. As I already linked it is critical that coordination happens between Bitcoin Core devs and Gentoo in the unbundled scenario. Previously Gentoo relied solely that the devs knew to ping Luke-jr when library upgrades needed to happen that could risk breaking consensus. That is not currently tenable given recent events and has already been shown to be prone to errors and delays. I am still advocating that each of the components be controllable with a useflag on net-p2p/bitcoin (or net-p2p/bitcoin-core, whatever they name it). > I wouldn't mind a USE="system-secp256k1" with an associated ewarn about being an unsanctioned configuration. History has shown this didn't work out. The gentoo repo should get the safest, most boring configuration and leave the overlays expand upon it for power users.
This moves in the wrong direction. The goal has been for years to modularise and split things up more (eg, see recent work on libbitcoinkernel, the separate GUI github repo, etc). Univalue is an unfortunate case where upstream abandoned it, and there were significant benefits to forking. Note that the libsecp256k1 ABI issue was because the interface for Schnorr was originally considered experimental.
(In reply to Luke-Jr from comment #8) > This moves in the wrong direction. The goal has been for years to modularise > and split things up more (eg, see recent work on libbitcoinkernel, the > separate GUI github repo, etc). > I don't see that in the main bitcoin github org, but if it's going to happen pretty soon, then sure, that's a reason to wait. But the impression has never been to me that anyone was particularly interested in this at all. > Univalue is an unfortunate case where upstream abandoned it, and there were > significant benefits to forking. I don't blame them in this case, but it'd be nice if it was a separate repo and versioned properly, not just included in the monorepo.
FYI: I implemented this request in the Bitcoin Gentoo repository at https://gitlab.com/bitcoin/gentoo/-/commit/f230573190c77a2046451fc9a207a857d72aa824 net-p2p/bitcoin-core-25.0: introduce combined package This package combines: - dev-util/bitcoin-tx (always installed) - net-libs/libbitcoinconsensus (installed if USE="libs") - net-p2p/bitcoin-cli (installed if USE="bitcoin-cli") - net-p2p/bitcoind (installed if USE="daemon") - net-p2p/bitcoin-qt (installed if USE="qt5") To ease the transition, new ebuilds of the above packages at version 25.0 are added. These are empty packages that simply RDEPEND upon net-p2p/bitcoin-core with the necessary USE flags. Two new USE flags are introduced to control whether to embed internal copies of LevelDB and libsecp256k1 into Bitcoin Core or to dynamically link with the system-installed libraries. The "sqlite" USE flag is now enabled by default, as descriptor wallets are no longer experimental and are indeed now the default. The "wallet" USE flag has been dropped, as it was redundant. If "berkdb" and/or "sqlite" is enabled, then you get wallet support. If neither is enabled, then you don't. When a wallet is enabled, the 'bitcoin-wallet' utility is now installed. The OpenRC init script has been overhauled to use -daemonwait so that startup of other services needing bitcoind will be deferred until after bitcoind is ready to accept RPCs. The datadir that has long been located at the awkward /var/lib/bitcoin/.bitcoin/ is now migrated to /var/lib/bitcoind/. The ebuild installs a symlink to maintain backward compatibility on systems where the older path still exists. The init scripts now specify the location for the debug log file, defaulting to /var/log/bitcoind/debug.log, instead of allowing it to be written by default to /var/lib/bitcoin/.bitcoin/debug.log. It's an important distinction, as /var/log may be located on a different storage volume than /var/lib.
Moving this to https://github.com/gentoo/gentoo/pull/32978. Luke-Jr is alive(!) and reverted my addition of net-p2p/bitcoin-core to the Bitcoin Gentoo repo, so he will undoubtedly want to chime in here.
Keeping these separated is intentional, not a bug.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=380aad5fc649a00ae46644c130fb9a3b8970ee09 commit 380aad5fc649a00ae46644c130fb9a3b8970ee09 Author: Matt Whitlock <gentoo@mattwhitlock.name> AuthorDate: 2023-09-21 21:41:32 +0000 Commit: Florian Schmaus <flow@gentoo.org> CommitDate: 2023-10-09 10:54:05 +0000 net-p2p/bitcoin-core-25.0: introduce combined package This package combines: - dev-util/bitcoin-tx (always installed) - net-libs/libbitcoinconsensus (installed if USE="libs") - net-p2p/bitcoin-cli (installed if USE="bitcoin-cli") - net-p2p/bitcoind (installed if USE="daemon") - net-p2p/bitcoin-qt (installed if USE="qt5") To ease the transition, new ebuilds of the above packages at version 25.0 are added. These are empty packages that simply RDEPEND upon net-p2p/bitcoin-core with the necessary USE flags. Two new USE flags are introduced to control whether to embed internal copies of LevelDB and libsecp256k1 into Bitcoin Core or to dynamically link with the system-installed libraries. The "sqlite" USE flag is now enabled by default, as descriptor wallets are no longer experimental and are indeed now the default. The "wallet" USE flag has been dropped, as it was redundant. If "berkdb" and/or "sqlite" is enabled, then you get wallet support. If neither is enabled, then you don't. When a wallet is enabled, the 'bitcoin-wallet' utility is now installed. The OpenRC init script has been overhauled to use -daemonwait so that startup of other services needing bitcoind will be deferred until after bitcoind is ready to accept RPCs. The datadir that has long been located at the awkward /var/lib/bitcoin/.bitcoin/ is migrating to /var/lib/bitcoind/. On systems where the older path exists, the ebuild installs a symlink at the new path, so that the system service will continue to work, and emits an ewarn instructing the user to run the pkg_config() function to perform the migration on their system. The init scripts now specify the location for the debug log file, defaulting to /var/log/bitcoind/debug.log, instead of allowing it to be written by default to /var/lib/bitcoin/.bitcoin/debug.log. It's an important distinction, as /var/log may be located on a different storage volume than /var/lib. Closes: https://bugs.gentoo.org/890261 Signed-off-by: Matt Whitlock <gentoo@mattwhitlock.name> Closes: https://github.com/gentoo/gentoo/pull/32978 Signed-off-by: Florian Schmaus <flow@gentoo.org> dev-util/bitcoin-tx/bitcoin-tx-25.0.ebuild | 13 + .../libbitcoinconsensus-25.0.ebuild | 14 + net-p2p/bitcoin-cli/bitcoin-cli-25.0.ebuild | 13 + net-p2p/bitcoin-core/Manifest | 1 + net-p2p/bitcoin-core/bitcoin-core-25.0.ebuild | 340 +++++++++++++++++++++ net-p2p/bitcoin-core/files/25.0-syslibs.patch | 275 +++++++++++++++++ net-p2p/bitcoin-core/files/bitcoin-qt.protocol | 11 + net-p2p/bitcoin-core/files/bitcoind.logrotate-r1 | 8 + net-p2p/bitcoin-core/files/bitcoind.openrc | 89 ++++++ net-p2p/bitcoin-core/files/init.patch | 43 +++ .../files/org.bitcoin.bitcoin-qt.desktop | 15 + net-p2p/bitcoin-core/metadata.xml | 33 ++ net-p2p/bitcoin-qt/bitcoin-qt-25.0.ebuild | 18 ++ net-p2p/bitcoind/bitcoind-25.0.ebuild | 18 ++ 14 files changed, 891 insertions(+)
