Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 680984 - improve rust packaging
Summary: improve rust packaging
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Eclasses (show other bugs)
Hardware: All All
: Normal enhancement with 2 votes (vote)
Assignee: Gentoo Rust Project
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-03-19 23:56 UTC by SpanKY
Modified: 2020-05-22 00:23 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Rust Crate Management in Chrome OS.pdf (Rust Crate Management in Chrome OS.pdf,71.25 KB, application/pdf)
2019-03-19 23:56 UTC, SpanKY
Details
pdf -> txt (pdf2txt.txt,3.54 KB, text/plain)
2019-03-20 20:33 UTC, SpanKY
Details

Note You need to log in before you can comment on or make changes to this bug.
Description SpanKY gentoo-dev 2019-03-19 23:56:43 UTC
Created attachment 569922 [details]
Rust Crate Management in Chrome OS.pdf

in CrOS, Chirantan put together a design for overhauling how crates get packaged as ebuilds.  we find it a lot saner to track one-crate-per-ebuild, and we use subslots for the rebuild aspects.  i'm attaching the design doc for reference.  it's pretty similar to how go is packaged.

here's the current eclass we're using in CrOS:
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/release-R74-11895.B/eclass/cros-rust.eclass

obviously there are CrOSisms in there that don't make sense in Gentoo.  but we should be able to pull those out easily enough.

thoughts ?
Comment 1 William Hubbs gentoo-dev 2019-03-20 00:12:40 UTC
Is there any way you can attach a text copy of the design document?

Thanks,

William
Comment 2 Georgy Yakovlev archtester gentoo-dev 2019-03-20 01:03:44 UTC
this will require thousands of ebuilds for crates.

for example a single ebuild for x11-terms/alacritty uses ~300 crates =D

but since crates are only source is installed, ebuilds are quite simple.

here's an example I've taken  from cros-overlay 

https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/dev-rust/backtrace-sys/backtrace-sys-0.1.16.ebuild

# Copyright 2019 The Chromium OS Authors. All rights reserved.
# Distributed under the terms of the GNU General Public License v2
EAPI="6"
inherit cros-rust
DESCRIPTION="Bindings to the libbacktrace gcc library"
HOMEPAGE="https://github.com/alexcrichton/backtrace-rs"
SRC_URI="https://crates.io/api/v1/crates/${PN}/${PV}/download -> ${P}.crate"
LICENSE="|| ( MIT Apache-2.0 )"
SLOT="${PV}/${PR}"
KEYWORDS="*"
DEPEND=">=dev-rust/cc-1.0.0:=
	>=dev-rust/libc-0.2.0:=
"




to aid with creating those ebuilds we'll need some kind of tool to parse Cargo.toml and produce DEPEND (or should it be BDEPEND in EAPI=7?)
I currently use https://github.com/cardoe/cargo-ebuild and https://github.com/sfackler/cargo-tree to track dependencies, but it's not exactly what we need.
probably some  basic python script to parse and transform toml to DEPEND should be enough.



and looks like it'd work under crossdev-emerge transparently, nice.
Comment 3 Patrick McLean gentoo-dev 2019-03-20 01:10:29 UTC
We could write a tool to mostly automate the ebuild creation as well, since the ebuilds will be so simple it shouldn't be too much work.

We would probably want to have automated version bumps as well on the cargo package ebuilds...
Comment 4 SpanKY gentoo-dev 2019-03-20 20:31:02 UTC
we've had wrapper tools for many other packaging environments to help integrate with our PM.  gcpan for example.  i'm not offering to write said tool, but automating the ebuild generation sounds like the only scalable way forward.

i agree that 300+ crates is kind of rough, but i might argue so is the current alacritty ebuild :).  it only sets LICENSE=Apache-2.0, but is that really the correct license for every single crate it depends on ?
Comment 5 SpanKY gentoo-dev 2019-03-20 20:33:07 UTC
Created attachment 570080 [details]
pdf -> txt

(In reply to William Hubbs from comment #1)

the pdf isn't complicated, so i assume your normal pdf2txt type tools would work
Comment 6 Georgy Yakovlev archtester gentoo-dev 2019-03-21 05:25:54 UTC
any ideas how to solve circular dependencies in cargo?

for example, 

proc-macro2 v0.4.27 (/tmp/portage/dev-rust/proc-macro2-0.4.27/work/proc-macro2-0.4.27)
└── unicode-xid v0.1.0
[dev-dependencies]
└── quote v0.6.11
    └── proc-macro2 v0.4.27
        └── unicode-xid v0.1.0

when cargo package is run it wants all dev-dependencies to be present in system registry.
but proc-macro2 depends on quote which depends on proc-macro2

error: no matching package named `quote` found
location searched: registry `https://github.com/rust-lang/crates.io-index`
required by package `proc-macro2 v0.4.27 (/tmp/portage/dev-rust/proc-macro2-0.4.27/work/proc-macro2-0.4.27)`



there is
-Z avoid-dev-deps

but cargo complains that
error: the `-Z` flag is only accepted on the nightly channel of Cargo

it's easy to patch out
impl CliUnstable {
    pub fn parse(&mut self, flags: &[String]) -> CargoResult<()> {
        if !flags.is_empty() && !nightly_features_allowed() {
            failure::bail!("the `-Z` flag is only accepted on the nightly channel of Cargo")
        }
        for flag in flags {
            self.add(flag)?;
        }
        Ok(())
}


i see no special treatment for dev-rust/quote and dev-rust/proc-macro2 in chromiumos-overlay, am I missing something?


https://github.com/rust-lang/cargo/issues/5133
Comment 7 Georgy Yakovlev archtester gentoo-dev 2019-03-21 06:00:20 UTC
ah I see, good old sed.

# Delete the serde dev dependency. Allowing the dependency introduces a
        # circular dependency, but serde has a legitimate dependency on
        # serde_derive.
        sed -i '/\[dev-dependencies.serde\]/{N;d;}' "${S}/Cargo.toml"
Comment 8 chirantan 2019-03-21 06:20:52 UTC
We also stub out some crates that are only needed as dev-dependencies.  For example, https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/dev-rust/rayon/rayon-1.0.2.ebuild

and the CROS_RUST_EMPTY_CRATE handling here: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/eclass/cros-rust.eclass#79


You also get fun things like the rand_chacha crate, which depends on rand_core >= 0.2, < 0.4 (https://crates.io/crates/rand_chacha) and rand_core 0.3.1 *depends on rand_core 0.4* (https://crates.io/crates/rand_core/0.3.1).
Comment 9 Georgy Yakovlev archtester gentoo-dev 2019-03-21 06:36:14 UTC
here's quickly hacked together implementation directly based on chromiumos-overlay cros-rust eclass.

rust-crate eclass only handles crate installation within dev-rust category
I'm thinking separate cargo-r1 eclass will be provided for actual consumers of crates and maybe cargo-utils.eclass with common utility functions, but not sure yet.

my implementation also runs tests, which is kinda nice.


eclass and couple ebuilds are here.

https://github.com/gyakovlev/rust-dev-overlay


with EAPI7 and BDEPEND we should be able to install crates on host root and emit cross-compiled binaries without even installing crates into cross-root I think, not sure how involved this would be.
Comment 10 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2020-02-16 05:42:08 UTC
> when cargo package is run it wants all dev-dependencies to be present in
> system registry.
> but proc-macro2 depends on quote which depends on proc-macro2
> 
> error: no matching package named `quote` found
> location searched: registry `https://github.com/rust-lang/crates.io-index`
> required by package `proc-macro2 v0.4.27
> (/tmp/portage/dev-rust/proc-macro2-0.4.27/work/proc-macro2-0.4.27)`

That only happens when there's a binary target in the crate (which is anything installable by cargo install, which includes examples) , because:

Binary targets existing in the crate requires a Cargo.lock file be shipped in the crate.

Thus, `cargo package` must create Cargo.lock

Thus, all dependency metadata must be available to compute Cargo.lock

And due to having no local copy of a full index, this means we have to have local copies of the crate as well.

I've found the sensible workaround is to nuke binary targets where-ever possible, which sometimes includes needing to patch [example] sections out of Cargo.toml

This substantially reduces the number of circular dependency problems one might face. 

The residual ones are all either USE= gated, usually USE="test" gated, which can be easily worked around in portage with a 2-step install of:

> FEATURES="-test" emerge --with-test-deps foo
> FEATURES="test" emerge foo

And recent portage's will handle circular deps in the above by weakening them to work more like PDEPEND.

In older portages, one may need:

> FEATURES="-test" emerge foo
> FEATURES="-test" emerge --with-test-deps --only-deps foo
> FEATURES="test" emerge foo

Subsequently, I'm at:

425 Crates
635 Crate versions

Most, if not all, have working tests.

I've not yet needed to produce "fake" crates. :)

Feel free to, uh, borrow stuff.

But there's still a lot of room for improvement.

WIP Repo is now: https://github.com/kentnl-gentoo-rust/rust-dev-overlay

Dep Graphs: https://kentfredric.github.io/gentoo-rust-meta/graphs/index.html

( Though these graphs are now so large and complicated if one was to print them at a readable size, they would likely cover the floor of a large room )
Comment 11 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2020-02-16 08:02:29 UTC
Though I should mention, looking at the stuff on the chromiumos overlay, looks like you're doing deps rather wrong.

Here's your curl:

DEPEND="
	>=dev-rust/curl-sys-0.4.18:=
	>=dev-rust/kernel32-sys-0.2.2:=
	>=dev-rust/libc-0.2.42:=
	>=dev-rust/schannel-0.1.13:=
	>=dev-rust/socket2-0.3.7:=
	>=dev-rust/winapi-0.2.7:=
	>=dev-rust/openssl-probe-0.1.2:=
	>=dev-rust/openssl-sys-0.9.43:=
	=dev-rust/mio-0.6*:=
	>=dev-rust/mio-extras-2.0.3:=
"
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/refs/heads/master/dev-rust/curl/curl-0.4.22.ebuild?autodive=0%2F#16


That says that "curl-sys-0.5" statisfies this dependency, however, curl-sys's crate says:

> curl-sys = { path = "curl-sys", version = "0.4.18", default-features = false }
https://github.com/alexcrichton/curl-rust/blob/ed811d63d1934d3d167462d8b7e9f46152890928/Cargo.toml#L19

And this means:

  0.4.0 ... 0.4.99999999999

0.5.0 will *not* satisfy this requirement, and having only curl-sys 0.5* available means your crate will not compile, nor will anything that depends on it.

That's why I've opted to write that sort of constraint as:

    ( =dev-rust/curl-sys-0.4*:= >=dev-rust/curl-sys-0.4.18 )

Which more accurately maps the constraint.

I suspect you'll have a lot of pain in your future without this. :/

It doesn't look wrong *everywhere*, one line here is OK..

https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/02d88755cadb3b876109c01e6ec335f679c4c8be/dev-rust/http/http-0.1.19.ebuild#20

But:

	>=dev-rust/fnv-1.0.5:=
	>=dev-rust/itoa-0.4.1:=

These are both wrong, that allows dev-rust/fnv-10.999.999 and dev-rust/itoa-999.9999.999 , which would be a big, big difference from fnv-1* + itoa-0.4*

https://wiki.gentoo.org/wiki/Project:Rust/rust-dev#Version_Mapping 

Here's my [incomplete] diagnosis with cited references to the cargo manual
Comment 12 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2020-02-16 08:14:25 UTC
https://bugs.chromium.org/p/chromium/issues/detail?id=1052069&q=rust&can=2 

Guess I'm not the only one to notice :D