Created attachment 569922 [details] Rust Crate Management in Chrome OS.pdf in CrOS, Chirantan put together a design for overhauling how crates get packaged as ebuilds. we find it a lot saner to track one-crate-per-ebuild, and we use subslots for the rebuild aspects. i'm attaching the design doc for reference. it's pretty similar to how go is packaged. here's the current eclass we're using in CrOS: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/release-R74-11895.B/eclass/cros-rust.eclass obviously there are CrOSisms in there that don't make sense in Gentoo. but we should be able to pull those out easily enough. thoughts ?
Is there any way you can attach a text copy of the design document? Thanks, William
this will require thousands of ebuilds for crates. for example a single ebuild for x11-terms/alacritty uses ~300 crates =D but since crates are only source is installed, ebuilds are quite simple. here's an example I've taken from cros-overlay https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/dev-rust/backtrace-sys/backtrace-sys-0.1.16.ebuild # Copyright 2019 The Chromium OS Authors. All rights reserved. # Distributed under the terms of the GNU General Public License v2 EAPI="6" inherit cros-rust DESCRIPTION="Bindings to the libbacktrace gcc library" HOMEPAGE="https://github.com/alexcrichton/backtrace-rs" SRC_URI="https://crates.io/api/v1/crates/${PN}/${PV}/download -> ${P}.crate" LICENSE="|| ( MIT Apache-2.0 )" SLOT="${PV}/${PR}" KEYWORDS="*" DEPEND=">=dev-rust/cc-1.0.0:= >=dev-rust/libc-0.2.0:= " to aid with creating those ebuilds we'll need some kind of tool to parse Cargo.toml and produce DEPEND (or should it be BDEPEND in EAPI=7?) I currently use https://github.com/cardoe/cargo-ebuild and https://github.com/sfackler/cargo-tree to track dependencies, but it's not exactly what we need. probably some basic python script to parse and transform toml to DEPEND should be enough. and looks like it'd work under crossdev-emerge transparently, nice.
We could write a tool to mostly automate the ebuild creation as well, since the ebuilds will be so simple it shouldn't be too much work. We would probably want to have automated version bumps as well on the cargo package ebuilds...
we've had wrapper tools for many other packaging environments to help integrate with our PM. gcpan for example. i'm not offering to write said tool, but automating the ebuild generation sounds like the only scalable way forward. i agree that 300+ crates is kind of rough, but i might argue so is the current alacritty ebuild :). it only sets LICENSE=Apache-2.0, but is that really the correct license for every single crate it depends on ?
Created attachment 570080 [details] pdf -> txt (In reply to William Hubbs from comment #1) the pdf isn't complicated, so i assume your normal pdf2txt type tools would work
any ideas how to solve circular dependencies in cargo? for example, proc-macro2 v0.4.27 (/tmp/portage/dev-rust/proc-macro2-0.4.27/work/proc-macro2-0.4.27) └── unicode-xid v0.1.0 [dev-dependencies] └── quote v0.6.11 └── proc-macro2 v0.4.27 └── unicode-xid v0.1.0 when cargo package is run it wants all dev-dependencies to be present in system registry. but proc-macro2 depends on quote which depends on proc-macro2 error: no matching package named `quote` found location searched: registry `https://github.com/rust-lang/crates.io-index` required by package `proc-macro2 v0.4.27 (/tmp/portage/dev-rust/proc-macro2-0.4.27/work/proc-macro2-0.4.27)` there is -Z avoid-dev-deps but cargo complains that error: the `-Z` flag is only accepted on the nightly channel of Cargo it's easy to patch out impl CliUnstable { pub fn parse(&mut self, flags: &[String]) -> CargoResult<()> { if !flags.is_empty() && !nightly_features_allowed() { failure::bail!("the `-Z` flag is only accepted on the nightly channel of Cargo") } for flag in flags { self.add(flag)?; } Ok(()) } i see no special treatment for dev-rust/quote and dev-rust/proc-macro2 in chromiumos-overlay, am I missing something? https://github.com/rust-lang/cargo/issues/5133
ah I see, good old sed. # Delete the serde dev dependency. Allowing the dependency introduces a # circular dependency, but serde has a legitimate dependency on # serde_derive. sed -i '/\[dev-dependencies.serde\]/{N;d;}' "${S}/Cargo.toml"
We also stub out some crates that are only needed as dev-dependencies. For example, https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/dev-rust/rayon/rayon-1.0.2.ebuild and the CROS_RUST_EMPTY_CRATE handling here: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/eclass/cros-rust.eclass#79 You also get fun things like the rand_chacha crate, which depends on rand_core >= 0.2, < 0.4 (https://crates.io/crates/rand_chacha) and rand_core 0.3.1 *depends on rand_core 0.4* (https://crates.io/crates/rand_core/0.3.1).
here's quickly hacked together implementation directly based on chromiumos-overlay cros-rust eclass. rust-crate eclass only handles crate installation within dev-rust category I'm thinking separate cargo-r1 eclass will be provided for actual consumers of crates and maybe cargo-utils.eclass with common utility functions, but not sure yet. my implementation also runs tests, which is kinda nice. eclass and couple ebuilds are here. https://github.com/gyakovlev/rust-dev-overlay with EAPI7 and BDEPEND we should be able to install crates on host root and emit cross-compiled binaries without even installing crates into cross-root I think, not sure how involved this would be.
> when cargo package is run it wants all dev-dependencies to be present in > system registry. > but proc-macro2 depends on quote which depends on proc-macro2 > > error: no matching package named `quote` found > location searched: registry `https://github.com/rust-lang/crates.io-index` > required by package `proc-macro2 v0.4.27 > (/tmp/portage/dev-rust/proc-macro2-0.4.27/work/proc-macro2-0.4.27)` That only happens when there's a binary target in the crate (which is anything installable by cargo install, which includes examples) , because: Binary targets existing in the crate requires a Cargo.lock file be shipped in the crate. Thus, `cargo package` must create Cargo.lock Thus, all dependency metadata must be available to compute Cargo.lock And due to having no local copy of a full index, this means we have to have local copies of the crate as well. I've found the sensible workaround is to nuke binary targets where-ever possible, which sometimes includes needing to patch [example] sections out of Cargo.toml This substantially reduces the number of circular dependency problems one might face. The residual ones are all either USE= gated, usually USE="test" gated, which can be easily worked around in portage with a 2-step install of: > FEATURES="-test" emerge --with-test-deps foo > FEATURES="test" emerge foo And recent portage's will handle circular deps in the above by weakening them to work more like PDEPEND. In older portages, one may need: > FEATURES="-test" emerge foo > FEATURES="-test" emerge --with-test-deps --only-deps foo > FEATURES="test" emerge foo Subsequently, I'm at: 425 Crates 635 Crate versions Most, if not all, have working tests. I've not yet needed to produce "fake" crates. :) Feel free to, uh, borrow stuff. But there's still a lot of room for improvement. WIP Repo is now: https://github.com/kentnl-gentoo-rust/rust-dev-overlay Dep Graphs: https://kentfredric.github.io/gentoo-rust-meta/graphs/index.html ( Though these graphs are now so large and complicated if one was to print them at a readable size, they would likely cover the floor of a large room )
Though I should mention, looking at the stuff on the chromiumos overlay, looks like you're doing deps rather wrong. Here's your curl: DEPEND=" >=dev-rust/curl-sys-0.4.18:= >=dev-rust/kernel32-sys-0.2.2:= >=dev-rust/libc-0.2.42:= >=dev-rust/schannel-0.1.13:= >=dev-rust/socket2-0.3.7:= >=dev-rust/winapi-0.2.7:= >=dev-rust/openssl-probe-0.1.2:= >=dev-rust/openssl-sys-0.9.43:= =dev-rust/mio-0.6*:= >=dev-rust/mio-extras-2.0.3:= " https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/dev-rust/curl/curl-0.4.22.ebuild?autodive=0%2F#16 That says that "curl-sys-0.5" statisfies this dependency, however, curl-sys's crate says: > curl-sys = { path = "curl-sys", version = "0.4.18", default-features = false } https://github.com/alexcrichton/curl-rust/blob/ed811d63d1934d3d167462d8b7e9f46152890928/Cargo.toml#L19 And this means: 0.4.0 ... 0.4.99999999999 0.5.0 will *not* satisfy this requirement, and having only curl-sys 0.5* available means your crate will not compile, nor will anything that depends on it. That's why I've opted to write that sort of constraint as: ( =dev-rust/curl-sys-0.4*:= >=dev-rust/curl-sys-0.4.18 ) Which more accurately maps the constraint. I suspect you'll have a lot of pain in your future without this. :/ It doesn't look wrong *everywhere*, one line here is OK.. https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/02d88755cadb3b876109c01e6ec335f679c4c8be/dev-rust/http/http-0.1.19.ebuild#20 But: >=dev-rust/fnv-1.0.5:= >=dev-rust/itoa-0.4.1:= These are both wrong, that allows dev-rust/fnv-10.999.999 and dev-rust/itoa-999.9999.999 , which would be a big, big difference from fnv-1* + itoa-0.4* https://wiki.gentoo.org/wiki/Project:Rust/rust-dev#Version_Mapping Here's my [incomplete] diagnosis with cited references to the cargo manual
https://bugs.chromium.org/p/chromium/issues/detail?id=1052069&q=rust&can=2 Guess I'm not the only one to notice :D
