I wish to propose a new way to distribute distfiles : sharing them with bittorrent protocol. This can help to seriously decrease the load on the gentoo servers I think. Of course, this can be set as an option, as many people don't have a stable connexion (especially laptop users). But for desktop users, this can be a way to contribute to the community without effort (letting someone get what he need in a not-used part of my hard-disk which is wasted by definition...). Of course, this idea oblige to think of how to put it in ebuild and rethink a part of the download process of portage. I hope this idea would help you in some way. Best regards Reproducible: Always
(In reply to comment #0) > This can help to seriously decrease the load on the gentoo servers I think. Our mirror tier provides more bandwidth than p2p would. This is why we don't even distribute releases via p2p anymore.
(In reply to Jeremy Olexa (darkside) from comment #1) > (In reply to comment #0) > > > This can help to seriously decrease the load on the gentoo servers I think. > > Our mirror tier provides more bandwidth than p2p would. This is why we don't > even distribute releases via p2p anymore. Now, in 2015, many users have very fast internet (about 100 Mbit/s and more). I have 500 Mbit/s at home, and gentoo mirrors are so slow for me. Seriously, I can often build a kernel faster than download it. p2p can merge bandwidth of many peers.
I guess fetchcommandwrapper (which uses aria2) is the closest thing that is available: https://packages.gentoo.org/packages/app-portage/fetchcommandwrapper
(In reply to Zac Medico from comment #3) > I guess fetchcommandwrapper (which uses aria2) is the closest thing that is > available: > > https://packages.gentoo.org/packages/app-portage/fetchcommandwrapper Thanks for this. This is not yet what we really need but certainly a step ahead. Though would be nice if we finally were able to get all URIs for the package into one fetchcommand (or even better, for all packages in queue).
(In reply to Stéphane from comment #0) > I wish to propose a new way to distribute distfiles : sharing them with > bittorrent protocol. Here are a couple of things we might need: * Something like profiles/thirdpartymirrors which lists locations to fetch torrent files from. Shall we make it profiles/thirdpartytorrents? * A SRC_URI extension which allows torrent protocol hints, such as "torrent://gentoo/tar-1.0.gz" (which bears similarity to the existing format "mirror://gentoo/tar-1.0.gz").
I don't think this is really what we want. As I see it, SRC_URI should be limited to well-supported protocols such as http and https (I'd even avoid ftp there). Those two have the best chances of working everywhere, not being blocked, being proxied as necessary, etc. In fact, I'm pretty sure some of us don't want the Package Manager suddenly using any kind of P2P behind their backs. That said, I think we would like to use P2P as an optional extension to Gentoo mirrors. As in, user enables P2P fetching and gets all mirrored distfiles via P2P. Of course, this brings the problem of passing the necessary metadata.
(In reply to Michał Górny from comment #6) > I don't think this is really what we want. As I see it, SRC_URI should be > limited to well-supported protocols such as http and https (I'd even avoid > ftp there). Yes, of course. I didn't intend to suggest that it should be valid for an ebuild to specify a torrent uri as the sole means to fetch a file. Still, for cases where there might be licensing restrictions, it might be useful to be able to embed a reference to thirdpartytorrents in SRC_URI, much like we can do now with mirror:// references to thirdpartymirrors. > Those two have the best chances of working everywhere, not being > blocked, being proxied as necessary, etc. In fact, I'm pretty sure some of > us don't want the Package Manager suddenly using any kind of P2P behind > their backs. Yes, of course. It would certainly have to be enabled by the user, due to the nature of p2p protocols. > That said, I think we would like to use P2P as an optional extension to > Gentoo mirrors. As in, user enables P2P fetching and gets all mirrored > distfiles via P2P. Yeah, that's what I was thinking, except that I wanted to include the sort of flexibility that thirdpartymirrors gives us within the existing system. > Of course, this brings the problem of passing the > necessary metadata. Torrent mirrors can be represented much like the existing mirrors. You just need a base uri for the torrent mirror, and then you use the distfile name to generate the full uri of the torrent file. Then the client just fetches the torrent file from the mirror using http/https, and finally feeds the torrent file to a bittorrent client.
Well, the point kinda is to avoid having to fetch .torrent files explicitly as this adds unnecessary load to HTTP mirrors.
Since the .torrent files are small, they'll put negligible load on the mirrors (and of course that small load can be distributed across many mirrors).
Well, because they are small, the protocol overhead will be big. That's the problem I see. I wonder if we could abuse DHT somehow.
(In reply to Michał Górny from comment #8) > Well, the point kinda is to avoid having to fetch .torrent files explicitly > as this adds unnecessary load to HTTP mirrors. How about magnet-links?
Indeed, magnet links embedded in SRC_URI would be a very handy.
Why? This all strikes me as just "doing something for the sake of doing it and sounding cool".
Since p2p protocols are inherently more scalable, they make it possible to host files that might otherwise be impossible to host due to various kinds of resource constraints. It will increase the capacity of both gentoo and third-party ebuild developers to host ridiculously large files.
And the demonstration that a) scalability is a problem, b) developers actually want to do this, and c) that this replacement will be sufficiently scalable is where?
@games: Is there any interest in having bittorrent integration, in order to distribute game resources that might be too large to distribute via the existing http/https/ftp mirror system?
I didn't ask whether you could find someone who might say "yeah, this cool thing might be OK for us". If the games team has a demonstrated need for a "more scalable" solution, then we should evaluate their requirements and come up with a solution accordingly, not go "ok, bittorrent!".
Off the top of my head, I don't know any opensource game that exceeds 2GiB and even those that are somewhere around 1GiB are very rare (e.g. warsow or xonotic). Most of the big-sized games where that would make sense are proprietary (e.g. games-action/brutal-legend) and have fetch and/or mirror-restriction anyway. So I don't think this is really a pressing issue.
I can't think of a situation where it would be a big win for us today. Things that are too big for the mirrors are usually not allowed to be on the mirrors anyhow. The biggest thing I see is games-simulation/flightgear-data which is 1.1G. That used to feel big but today it doesn't, at least to me.
Not the games team here, but I can contribute one data point. I have two huge packages with chess endgame tablebases in my developer overlay. For example, distfiles for the following one have a total size of about 150 GB: https://gitweb.gentoo.org/repo/dev/ulm.git/tree/games-board/egtb-syzygy There is an even larger variant with a size of 1.2 TB. Problem is that you'll need disk space at least three times that size (DISTDIR, D, live system, and maybe WORKDIR), which means that above a certain size, using the package manager for installation of static data may become pointless. So, not sure if I would commit above ebuild to the gentoo tree, even if there was a bittorrent fetch mechanism available.
(In reply to Ulrich Müller from comment #20) > Not the games team here, but I can contribute one data point. I have two > huge packages with chess endgame tablebases in my developer overlay. For > example, distfiles for the following one have a total size of about 150 GB: > https://gitweb.gentoo.org/repo/dev/ulm.git/tree/games-board/egtb-syzygy > There is an even larger variant with a size of 1.2 TB. > Still sounds overkill to introduce such a feature for ~2 ebuilds? I don't see the problem of people using bittorrent manually for such a thing and then moving it to DISTDIR.
(In reply to Julian Ospald (hasufell) from comment #21) > Still sounds overkill to introduce such a feature for ~2 ebuilds? I don't > see the problem of people using bittorrent manually for such a thing and > then moving it to DISTDIR. That's what I was trying to say. :)
