When updating texlive suite during world update, the package fails to fetch and install with artus /etc # emerge -1v texlive-latexextra These are the packages that would be merged, in order: Calculating dependencies ... done! [ebuild U ] dev-texlive/texlive-latexextra-2020::gentoo [2019-r2::gentoo] USE="doc source" 509,613 KiB Total: 1 package (1 upgrade), Size of downloads: 509,613 KiB >>> Verifying ebuild manifests >>> Emerging (1 of 1) dev-texlive/texlive-latexextra-2020::gentoo [Errno 7] Argument list too long: b'/usr/bin/python3.7m': /bin/bash -c /usr/lib/portage/python3.7/ebuild.sh clean Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/portage/process.py", line 391, in spawn unshare_flags, cgroup) File "/usr/lib/python3.7/site-packages/portage/process.py", line 685, in _exec os.execve(binary, myargs, env) File "/usr/lib/python3.7/site-packages/portage/__init__.py", line 246, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) OSError: [Errno 7] Argument list too long: b'/usr/bin/python3.7m' [Errno 7] Argument list too long: b'/usr/bin/python3.7m': /bin/bash -c /usr/lib/portage/python3.7/ebuild.sh clean Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/portage/process.py", line 391, in spawn unshare_flags, cgroup) File "/usr/lib/python3.7/site-packages/portage/process.py", line 745, in _exec os.execve(binary, myargs, env) File "/usr/lib/python3.7/site-packages/portage/__init__.py", line 246, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) OSError: [Errno 7] Argument list too long: b'/usr/bin/python3.7m' * The ebuild phase 'die_hooks' has been aborted since PORTAGE_BUILDDIR * does not exist: '/var/tmp/portage/dev-texlive/texlive-latexextra-2020' >>> Failed to emerge dev-texlive/texlive-latexextra-2020 * Messages for package dev-texlive/texlive-latexextra-2020: * * The following package has failed to build, install, or execute postinst: * * (dev-texlive/texlive-latexextra-2020:0/0::gentoo, ebuild scheduled for merge) * When I try to manually fetch packages sources I get artus /etc # ebuild /var/db/repos/gentoo/dev-texlive/texlive-latexextra/texlive-latexextra-2020.ebuild fetch [Errno 7] Argument list too long: b'/bin/bash': /bin/bash -c >> /var/cache/distfiles/.__portage_test_write__ 2>/dev/null ; rval=$? ; rm -f /var/cache/distfiles/.__portage_test_write__ ; exit $rval Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/portage/process.py", line 391, in spawn unshare_flags, cgroup) File "/usr/lib/python3.7/site-packages/portage/process.py", line 745, in _exec os.execve(binary, myargs, env) File "/usr/lib/python3.7/site-packages/portage/__init__.py", line 246, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) OSError: [Errno 7] Argument list too long: b'/bin/bash' >>> Downloading 'https://ftp.halifax.rwth-aachen.de/gentoo/distfiles/layout.conf' [Errno 7] Argument list too long: b'/usr/bin/wget': wget -t 3 -T 60 --passive-ftp -O /var/cache/distfiles/.layout.conf.ftp.halifax.rwth-aachen.de.__download__ https://ftp.halifax.rwth-aachen.de/gentoo/distfiles/layout.conf Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/portage/process.py", line 391, in spawn unshare_flags, cgroup) File "/usr/lib/python3.7/site-packages/portage/process.py", line 745, in _exec os.execve(binary, myargs, env) File "/usr/lib/python3.7/site-packages/portage/__init__.py", line 246, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) OSError: [Errno 7] Argument list too long: b'/usr/bin/wget' No digest file available and download failed. !!! Couldn't download '.layout.conf.ftp.halifax.rwth-aachen.de'. Aborting. and similar errors for more fetch tries, until it aborts. Reproducible: Always emerge --info output attached as file.
Created attachment 634344 [details] emerge-info-texlive-latexextra.txt output of emerge --info texlive-latexextra
Not a texlive bug. It is somehow broken portage itself.
I was thinking about this possibility too. But it's strange, it's the only package out of about a thousand which have been emerged so far since this first popped up, where this happens. So how could I be of help with starting to solve this issue?
Try at least chown -RP portage:portage /var/tmp/portage looks like permission problems to me
The directory already had those permissions: waebbl@artus /var/db/repos/gentoo/media-gfx/gmic $ ls -la /var/tmp/portage total 4 drwxrwxrwt 3 portage portage 60 Apr 25 14:02 . drwxrwxrwt 13 root root 4096 Apr 24 22:07 .. drwxrwxr-x 2 portage portage 40 Apr 25 13:45 ._unmerge_
IMO the bug *is* related to the package and not to portage. I usually have USE="doc source" enabled for all my TeX related packages. When disabling the source USE flag, the package succeeds. I was playing around with some of the recent changes in texlive-module.eclass and checked the TEXLIVE_MODULE_SRC_CONTENTS variable in the ebuild, but haven't had any succes so far. As a temporary work-around I'm omitting the source USE flag for the moment for this package. Going to re-open the issue. Please consider investigating this further.
This is seen that portage dies, not even an texlive eclass. Also no installation logic changes, so it can not be texlive issue at all.
Same here: [ebuild U ] dev-texlive/texlive-latexextra-2020::gentoo [2019-r2::gentoo] USE="doc source" 495,205 KiB The important error seems to be: OSError: [Errno 7] Argument list too long: b'/usr/bin/python3.7m' USE='-doc -source' worked fine. So this sounds like a portage bug with extremely huge SRC_URIs.
I tried a few things on my test box. Downgrading to stable portage-2.3.89-r3 didn't help, as well as downgrading the default python on this box to 3.8.2-r1 doesn't solve it. But using either of USE="-doc -source", USE="doc -source" and USE="-doc source" builds the package successfully. So it looks like it's indeed getting a 'package overflow' when using both USE flags together.
*** Bug 270230 has been marked as a duplicate of this bug. ***
I am up the importance as this bug screwed up multiple systems now.
I guess SRC_URI probably leaked into the environment somehow. Our existing mitigation for issues like this is in these 2 commits that reference bug 262647: https://gitweb.gentoo.org/proj/portage.git/commit/?id=1ad3342b49e9d488a9beba86ea7096c68c6b1f44 https://gitweb.gentoo.org/proj/portage.git/commit/?id=7d2fa6927d3a60828f09a0a2cd569038c2c18482
I was able to reproduce the problem with USE="doc source" enabled, and it's at least the "A" variable triggering this (might need to do something about "AA" too even though it's only supposed to be available for older EAPIs).
This patch is enough to allow fetch to succeed: > diff --git a/lib/portage/package/ebuild/fetch.py b/lib/portage/package/ebuild/fetch.py > index 28e7caf53..126747bb7 100644 > --- a/lib/portage/package/ebuild/fetch.py > +++ b/lib/portage/package/ebuild/fetch.py > @@ -144,6 +144,10 @@ def _spawn_fetch(settings, args, **kwargs): > env = settings.environ() > if logname is not None: > env["LOGNAME"] = logname > + # The A and AA variables may be so large that they trigger > + # OSError: [Errno 7] Argument list too long (bug 719202). > + env.pop('A', None) > + env.pop('AA', None) > try: > rval = spawn_func(args, env=env, **kwargs) > finally: However, the "A" variable is required for ebuild phases (it's defined by PMS and tells the ebuild which files to unpack), and its size makes it impossible to spawn an ebuild phase without triggering a "OSError: [Errno 7] Argument list too long" error.
This ebuild has an insane number of distfiles. Perhaps it should be divided into separate packages.
Looks like A is longer than MAX_ARG_STRLEN of Linux, which is defined as PAGE_SIZE*32, i.e. 128 KiB.
If we really want to support this, I guess PMS would need to be changed to specify A as a bash variable instead of an environment variable.
Giving this back to the texlive maintainer. If you really need support for this, please request an enhancement in PMS.
PMS section 11.1 specifies that the variables listed in table 11.1 (which include A) must be defined as environment variables: https://projects.gentoo.org/pms/7/pms.html#x1-10900011.1 So, seems that there is nothing that Portage could do about it. (In reply to Mike Gilbert from comment #17) > If we really want to support this, I guess PMS would need to be changed to > specify A as a bash variable instead of an environment variable. That's unlikely to happen, because the package manager is free to implement most commands (including unpack) as shell functions or external commands.
(In reply to Ulrich Müller from comment #19) > That's unlikely to happen, because the package manager is free to implement > most commands (including unpack) as shell functions or external commands. unpack does not utilize the A variable directly; it's used by src_unpack, which is always a function. Assuming A could be defined as a bash variable, a src_unpack function like the one below would avoid any length limitations by passing the filenames to unpack one at a time: src_unpack() { for x in ${A}; do unpack ${x} done }
(In reply to Mike Gilbert from comment #20) > unpack does not utilize the A variable directly; it's used by src_unpack, > which is always a function. I stand corrected. > Assuming A could be defined as a bash variable, a src_unpack function like > the one below would avoid any length limitations by passing the filenames to > unpack one at a time: > > src_unpack() { > for x in ${A}; do > unpack ${x} > done > } That's not even needed, simple "unpack ${A}" should work fine (unless a single filename would be longer than 128 KiB). I wonder however why Linux has such a small limit for string length.
(In reply to Ulrich Müller from comment #21) > That's not even needed, simple "unpack ${A}" should work fine (unless a > single filename would be longer than 128 KiB). There's also a limit on the length of command lines in Linux. I just don't know the number off the top of my head. From execve(2): E2BIG The total number of bytes in the environment (envp) and argument list (argv) is too large.
(In reply to Mike Gilbert from comment #22) > There's also a limit on the length of command lines in Linux. I just don't > know the number off the top of my head. > > From execve(2): > > E2BIG The total number of bytes in the environment (envp) and argument list > (argv) is too large. sysconf(_SC_ARG_MAX) is returning 2097152 here.
Surely, we can debate why Portage doesn't handle stupid things as well as it could, and how to improve handling of stupid things. We could also call using 3368 distfiles a really stupid thing that's harmful to users and ask package maintainers to kindly repack this crap into a single tarball (or fewer tarballs) that would be much less problematic, faster and less space consuming. Last I check, they had to push all this nonsense into mirrors manually anyway because everything fell apart.
(In reply to Michał Górny from comment #24) > Surely, we can debate why Portage doesn't handle stupid things as well as it > could, and how to improve handling of stupid things. We could also call > using 3368 distfiles a really stupid thing that's harmful to users and ask > package maintainers to kindly repack this crap into a single tarball (or > fewer tarballs) that would be much less problematic, faster and less space > consuming. Last I check, they had to push all this nonsense into mirrors > manually anyway because everything fell apart. This isn't very helpful, especially since you've used this particular package as part of the rationale for the GLEP 75 algorithm. There are other packages with many distfiles as well, e.g., games-board/tablebase-syzygy, net-p2p/bisq (Java), and net-p2p/go-ipfs (Go). Certainly one can argue that upstreams are doing stupid things, but repacking distfiles cannot be the solution to overcome limitations of the package manager. (In reply to Mike Gilbert from comment #17) > If we really want to support this, I guess PMS would need to be changed to > specify A as a bash variable instead of an environment variable. We can consider it for EAPI 8. AFAICS, the only other candidate for unexporting in table 11.1 would be USE, but given that we are far below the limit there, I'd rather not touch it. (Largest size of IUSE in the metadata cache is 3699 bytes for app-metrics/collectd-5.10.0.) Of course, that won't help us with existing ebuilds in EAPI 7. For the record, ${A} had a length of 128337 bytes in texlive-latexextra-2019-r2 with USE="doc source", which is just barely below the 128 KiB limit.
(In reply to Ulrich Müller from comment #25) > This isn't very helpful, especially since you've used this particular > package as part of the rationale for the GLEP 75 algorithm. Yes, TeXLive was causing a major pain and required us to do a lot of extra work in the past. Are you arguing that since once we accepted it and found a workaround elsewhere, now we have to keep accepting this forever with more issues being caused by it? I've pushed for GLEP 75 since it seemed the best solution at the time. It resolved a few problems, making things more portable, faster. Now the best solution seems to be to fix TeXLive and not try hard to figure out a workaround for a design that didn't anticipate insanity. Maybe it's a bad design but it's not something that can be fixed overnight, and I'm not really convinced trying to hurry a workaround will make thing any better. On the other hand, stopping to use humongous number of distfiles (which have *ZERO* benefit for our users) seems a good solution. It solves this problem, it makes things faster, it stops cluttering up distdir. > There are other > packages with many distfiles as well, e.g., games-board/tablebase-syzygy, > net-p2p/bisq (Java), and net-p2p/go-ipfs (Go). Certainly one can argue that > upstreams are doing stupid things, but repacking distfiles cannot be the > solution to overcome limitations of the package manager. Is creating a precedent for allowing insanity the solution? We're not really talking about limitations of 'the package manager'. We've reached the point where ebuild design has met *system limitations*. Sure, we can return to drawing board, fix it in EAPI 8, have users wait 6-12 months for things to stop being broken. Then discover that other package managers are broken too because they didn't anticipate hidden assumptions made in EAPI 8 changes. Furthermore, FWIU the discussion sounds like we're going to move to relying on implicit bash limitations. Sure, things might work (I do not know if they do!) right now but is this really a good idea to do so in the first place? I dare say that if something turned out to be a bad idea once, you don't just workaround the problem and hope it won't happen again. Bash is not exactly the kind of upstream that doesn't break stuff randomly and claim it was never supported.
(In reply to Michał Górny from comment #26) > (In reply to Ulrich Müller from comment #25) > > This isn't very helpful, especially since you've used this particular > > package as part of the rationale for the GLEP 75 algorithm. > > Yes, TeXLive was causing a major pain and required us to do a lot of extra > work in the past. Are you arguing that since once we accepted it and found > a workaround elsewhere, now we have to keep accepting this forever with more > issues being caused by it? > > I've pushed for GLEP 75 since it seemed the best solution at the time. It > resolved a few problems, making things more portable, faster. This solved half problem not a problem. > > Now the best solution seems to be to fix TeXLive and not try hard to figure > out a workaround for a design that didn't anticipate insanity. Maybe it's a > bad design but it's not something that can be fixed overnight, and I'm not > really convinced trying to hurry a workaround will make thing any better. To be clear, texlive "fix" not gonna happen, what you do is an attempt of shifting responsibility, fix the package manager not to produce longer lines than the allowed limit? Now you seem to pretend that the problem does not exist, while the same situation may happen (and I am more than sure will happen) to rust and go packages. > > On the other hand, stopping to use humongous number of distfiles (which have > *ZERO* benefit for our users) seems a good solution. It solves this > problem, it makes things faster, it stops cluttering up distdir. > > > There are other > > packages with many distfiles as well, e.g., games-board/tablebase-syzygy, > > net-p2p/bisq (Java), and net-p2p/go-ipfs (Go). Certainly one can argue that > > upstreams are doing stupid things, but repacking distfiles cannot be the > > solution to overcome limitations of the package manager. > > Is creating a precedent for allowing insanity the solution? > > We're not really talking about limitations of 'the package manager'. That is what we really do, portage was designed to make one big A, how is it not a portage's design problem? > We've > reached the point where ebuild design has met *system limitations*. Sure, > we can return to drawing board, fix it in EAPI 8, have users wait 6-12 > months for things to stop being broken. Then discover that other package > managers are broken too because they didn't anticipate hidden assumptions > made in EAPI 8 changes. > > Furthermore, FWIU the discussion sounds like we're going to move to relying > on implicit bash limitations. Sure, things might work (I do not know if > they do!) right now but is this really a good idea to do so in the first > place? I dare say that if something turned out to be a bad idea once, you > don't just workaround the problem and hope it won't happen again. Bash is > not exactly the kind of upstream that doesn't break stuff randomly and claim > it was never supported.
> To be clear, texlive "fix" not gonna happen What I meant is the near future (a year or so).
(In reply to Mikle Kolyada from comment #27) > (In reply to Michał Górny from comment #26) > > Now the best solution seems to be to fix TeXLive and not try hard to figure > > out a workaround for a design that didn't anticipate insanity. Maybe it's a > > bad design but it's not something that can be fixed overnight, and I'm not > > really convinced trying to hurry a workaround will make thing any better. > > To be clear, texlive "fix" not gonna happen, what you do is an attempt of > shifting responsibility, No, what I do is an attempt to have a working fix today and stop the insanity. Laziness combined with lack of formal rule saying 'you must not use more than N files' is not an excuse to create broken stuff and then blame people for not foreseeing something that insane. > fix the package manager not to produce longer lines > than the allowed limit? As it has been already pointed out, it's not a 'package manager' problem but specification problem. This problem can not be trivially fixed without issuing a new version which will take at least a few months for full deployment. Without any real guarantee that the 'fix' will actually work reliably going forward, and that when EAPI 8 is released this won't hit another non-obvious limit. > Now you seem to pretend that the problem does not > exist, Nonsense. It is *you* who insist that creating huge distfile lists is not a problem, and demanding to make insanity-proof software (tip: you fix one tool, another will trip on it). > while the same situation may happen (and I am more than sure will > happen) to rust and go packages. And this is exactly why we need to make a clear statement that it must not happen, and if people are going to package this insanity, they need to figure out a *proper* way of doing so. This must not serve as a precedent to kill user systems with insane amounts of distfiles. > > We're not really talking about limitations of 'the package manager'. > > That is what we really do, portage was designed to make one big A, how is it > not a portage's design problem? It's required to make one big A by PMS. This is now set in stone, and won't change until the next EAPI.
In case it wasn't clear from my previous comments, I don't really think supporting ebuilds with thousands of distfiles is really worth changing the spec. I just had fun thinking about how it could possibly be fixed. Possible workarounds for texlive: 1. Repack the files as mgorny suggests. 2. Split the ebuild into multiple packages.
(In reply to Mike Gilbert from comment #30) > In case it wasn't clear from my previous comments, I don't really think > supporting ebuilds with thousands of distfiles is really worth changing the > spec. I just had fun thinking about how it could possibly be fixed. > > Possible workarounds for texlive: > > 1. Repack the files as mgorny suggests. > 2. Split the ebuild into multiple packages. The problem is that is not an ebuilb (not texlive) issue, this could happen to anything else in the near or distant future, thexlive was just the one which revealed the problem after very long time. We really must fix the reason (whatever time it will take), and do not have to deal with consequenses.
(In reply to Michał Górny from comment #29) > (In reply to Mikle Kolyada from comment #27) > > (In reply to Michał Górny from comment #26) (just one reply at once) You misunderstood my intention here. I am not asserting that it is not the problem, though I believe it is more deep and complicated than you want to believe. I also telling you that I do not have time to re-think the eclass' and buildscripts logic (real life is real). If you are so willing to participate you are more than welcome. I can even guide you (or anyone else) through the packaging logic if you want me to.
Well... ARG_MAX can be increased via ulimit so a workaround is easy. MAX_ARG_STRLEN is linux specific and fixed. Can't do much about it besides living with it. What I would suggest is to "compress" the filenames: Historically I had been using texlive-module-blabla as distfiles. By using a shorter prefix (which is arbitrary), like "tl", we could easily gain 30-40K on the length of $A. That should give enough space for now. On the other hand, MAX_ARG_STRLEN is execve specific, so a portage fix could be to just set 'environ' and use execv.
(In reply to Mike Gilbert from comment #30) > In case it wasn't clear from my previous comments, I don't really think > supporting ebuilds with thousands of distfiles is really worth changing the > spec. I just had fun thinking about how it could possibly be fixed. > > Possible workarounds for texlive: > > 1. Repack the files as mgorny suggests. Not great as this makes "hotfixes" (grabbing a new version of a package from ctan and upgrading it) harder. A couple of those are usually needed each year due to bugs found late. Or removing packages due to licensing issues found late. > 2. Split the ebuild into multiple packages. Sure we can do as upstream and have the current ebuilds as metapackages and have each package separately. That would mean one package for 3 distfiles. Not sure anyone will be happy about this. As everyone can see, the number of packages is tremendous, and the main problem in maintaining texlive is the inter-dependencies between them. Upstream does a great job at it (even if we can see bugs from time to time) but if we start to diverge this will fall entirely on us and I don't think this is reasonable with the current manpower.
(In reply to Alexis Ballier from comment #33) > On the other hand, MAX_ARG_STRLEN is execve specific, so a portage fix could > be to just set 'environ' and use execv. execv(3) is just a libc wrapper around the execve(2) syscall. The limitation is enforced on the syscall by the kernel. (In reply to Alexis Ballier from comment #34) > Sure we can do as upstream and have the current ebuilds as metapackages and > have each package separately. That would mean one package for 3 distfiles. > Not sure anyone will be happy about this. I don't thin you need to split it into a separate ebuild for each module. Just splitting the single ebuild into two ebuilds would suffice for this particular case. Or possibly you could split the docs and source into separate ebuilds that get pulled in by the USE flags.
(In reply to Mike Gilbert from comment #35) > (In reply to Alexis Ballier from comment #33) > > On the other hand, MAX_ARG_STRLEN is execve specific, so a portage fix could > > be to just set 'environ' and use execv. > > execv(3) is just a libc wrapper around the execve(2) syscall. The limitation > is enforced on the syscall by the kernel. Indeed, I had missed that "detail" that the libc calls execve with environ... ;) > > (In reply to Alexis Ballier from comment #34) > > Sure we can do as upstream and have the current ebuilds as metapackages and > > have each package separately. That would mean one package for 3 distfiles. > > Not sure anyone will be happy about this. > > I don't thin you need to split it into a separate ebuild for each module. > Just splitting the single ebuild into two ebuilds would suffice for this > particular case. Or possibly you could split the docs and source into > separate ebuilds that get pulled in by the USE flags. Splitting the package arbitrarily would need first to make a split that ensures there is no circular dep between the two and then check every revdep. Not impossible but very manual. Reworking the generators to undo the merge of -doc and -source metapackages done by upstream is easier. I still believe that for this particular issue, shortening distfiles names is far easier and saves enough for the forthcoming years.
(In reply to Mike Gilbert from comment #17) > If we really want to support this, I guess PMS would need to be changed to > specify A as a bash variable instead of an environment variable. I don't really understand why this needs a PMS change but here is a potential workaround to this execve limitation: I don't see the limitation at the setenv level, so portage could write ebuild environment variables to a file that get sourced before executing ebuild related functions.
(In reply to Mikle Kolyada from comment #31) > I don't really understand why this needs a PMS change but here is a > potential workaround to this execve limitation: I don't see the limitation > at the setenv level, so portage could write ebuild environment variables to > a file that get sourced before executing ebuild related functions. That would cause execve to fail for any command subsequently executed by said ebuild.
The general fix for this would be to patch the linux kernel to increase MAX_ARG_STRLEN or make it tunable at runtime.
(In reply to Mike Gilbert from comment #38) > (In reply to Mikle Kolyada from comment #31) > > I don't really understand why this needs a PMS change but here is a > > potential workaround to this execve limitation: I don't see the limitation > > at the setenv level, so portage could write ebuild environment variables to > > a file that get sourced before executing ebuild related functions. > > That would cause execve to fail for any command subsequently executed by > said ebuild. I've opened bug 720180 to make portage delay export of the "A" variable until the last moment. In theory, an ebuild could use "export -n A" to suppress "[Errno 7] Argument list too long" errors when spawning subprocesses.
(In reply to Alexis Ballier from comment #33) > What I would suggest is to "compress" the filenames: Historically I had been > using texlive-module-blabla as distfiles. By using a shorter prefix (which > is arbitrary), like "tl", we could easily gain 30-40K on the length of $A. > That should give enough space for now. I would like to ask everyone not to make any changes until we agree on a good solution. Changing distfile names only shoves the problem under the carpet. Furthermore, it means repopulating all the mirrors and forcing users to refetch everything. For the former, it also involves keeping twice as many distfiles during the retention period, for the latter -- until eclean-dist. This is a huge waste of bandwidth, disk space, inodes and everyone's patience.
In my opinion, reducing the number of distfiles is not only the proper solution but also the only reasonable way forward. I would really prefer if developers agreed on this but if necessary, I can call for a QA vote on setting a formal limit on number of distfiles per ebuild. I'm not saying about forcing one distfile. However, repacking the initial version and possibly adding updates in separate distfiles is still better than the status quo. I've finally been able to do a cheap benchmark. I've taken texlive-latexextra-2020, stubbed pkg_pretend() and pkg_setup(), and limited src_unpack() to default. Then I've timed 'ebuild ... unpack' with a warm cache, in two variants. The first variant is using original plethora of distfiles, the second using a single repacked distfile. I have only tested USE="-doc -source", I am pretty sure that the gain will grow nonlinearly, I can test more tomorrow. I also suspect that removing -2019 file would make Portage faster but too late today to test more. separate distfiles: real 4m3,236s user 2m51,853s sys 1m25,839s combined distfile: real 2m54,630s user 2m26,190s sys 0m28,442s It's a win-win situation. Small separate distfiles have the following implications: 1. more files = more inodes wasted, more HTTP overhead, more filesystem overhead = overall slower. 2. smaller files = worse compression = more bandwidth and disk space wasted. This has negative impact on woodpecker (where zlogene pushes all those tiny files), on the mirror network and on user systems. The only counterargument I've heard so far was more developer's effort. If you really insist, you can add some kind of hack to the eclass to restore old behavior via some environment variable and have it fetch the files for you and unpack them manually for repacking. Or you can just modify your generator to do that for you. It's not rocket science. Really, length of A variable is just the tip of the problem. Working around it doesn't make texlive any better for our users.
I've done a second test run, this time with -2019 ebuild removed and USE=sources. Old version: real 4m48,279s user 3m9,928s sys 2m3,041s Repacked: real 2m57,905s user 2m27,661s sys 0m30,621s Apparent sizes: 29M separate, 23M repacked. Not to mention you could also do the renaming voodoo before repacking, so users wouldn't have to suffer that. I mean, that part of src_unpack() that used to take 45 minutes in fontsextra.
(In reply to Michał Górny from comment #41) > Changing distfile names only shoves the problem under the carpet. It would give us the time necessary to fix it in a future EAPI, or in a future version of texlive. > Furthermore, it means repopulating all the mirrors and forcing users to > refetch everything. For the former, it also involves keeping twice as many > distfiles during the retention period, for the latter -- until eclean-dist. > This is a huge waste of bandwidth, disk space, inodes and everyone's > patience. You'd have the same problems with bandwidth and diskspace when combining distfiles. (Even worse, while users could rename their distfiles locally, that would not be possible after combining files.) I don't say that combining distfiles may not be a viable solution for the future, but I just wouldn't do anything so drastic for the existing 2020 version.
(In reply to Ulrich Müller from comment #44) > (In reply to Michał Górny from comment #41) > > Changing distfile names only shoves the problem under the carpet. > > It would give us the time necessary to fix it in a future EAPI, or in a > future version of texlive. It would give you an excuse not to fix it properly while a proper fix takes around 15 minutes. Most of that time due to inefficiency of the ebuild right now. > > Furthermore, it means repopulating all the mirrors and forcing users to > > refetch everything. For the former, it also involves keeping twice as many > > distfiles during the retention period, for the latter -- until eclean-dist. > > This is a huge waste of bandwidth, disk space, inodes and everyone's > > patience. > > You'd have the same problems with bandwidth and diskspace when combining > distfiles. There's a major difference between shoving 30000 tiny extra files on mirrors and a few large files that not only waste less inodes and disk space per se but also -- as proven above -- compress much better. > (Even worse, while users could rename their distfiles locally, > that would not be possible after combining files.) Do you seriously assume that Gentoo users have nothing better to do than track random bugs and go out of their way to rename downloaded distfiles? > I don't say that combining distfiles may not be a viable solution for the > future, but I just wouldn't do anything so drastic for the existing 2020 > version. Experience proves that if we don't do it for 2020, then 2021 will be just the same and then the argument will be that we shouldn't do anything drastic for 2021, and 2022... In the meantime, users would waste hundreds of hours dealing with bad ideas.
(In reply to Mike Gilbert from comment #38) > (In reply to Mikle Kolyada from comment #31) > > I don't really understand why this needs a PMS change but here is a > > potential workaround to this execve limitation: I don't see the limitation > > at the setenv level, so portage could write ebuild environment variables to > > a file that get sourced before executing ebuild related functions. > > That would cause execve to fail for any command subsequently executed by > said ebuild. Hmm. Right. The bash variable solution makes total sense now, thanks. I thought this settled it but then: (In reply to Zac Medico from comment #40) > I've opened bug 720180 to make portage delay export of the "A" variable > until the last moment. In theory, an ebuild could use "export -n A" to > suppress "[Errno 7] Argument list too long" errors when spawning > subprocesses. This might actually work. A little hacky as this would require changes to the eclass/ebuilds but seems to be the lesser evil to stop the bleeding for now. Thanks Zac. (In reply to Mike Gilbert from comment #39) > The general fix for this would be to patch the linux kernel to increase > MAX_ARG_STRLEN or make it tunable at runtime. Well... Considering it has been there and documented for a while now, that googling sent me to bug reports for much more widely used tools than a gentoo package with non default useflags (e.g. make and ninja), I fear this would take good convincing and several iterations to make it. Then at least one month for a release and then an undefined amount of time for all users to upgrade in order to be able to rely on this (not counting those stuck on older or vendor kernels because of HW). EAPI8 is likely to happen first.
(In reply to Michał Górny from comment #41) > (In reply to Alexis Ballier from comment #33) > > What I would suggest is to "compress" the filenames: Historically I had been > > using texlive-module-blabla as distfiles. By using a shorter prefix (which > > is arbitrary), like "tl", we could easily gain 30-40K on the length of $A. > > That should give enough space for now. > > I would like to ask everyone not to make any changes until we agree on a > good solution. Don't worry. zlogene has (thankfully) taken the lead on texlive and has already written this won't happen in 2020, so you should be safe there ;) (In reply to Michał Górny from comment #45) > (In reply to Ulrich Müller from comment #44) > > (In reply to Michał Górny from comment #41) > > > Changing distfile names only shoves the problem under the carpet. > > > > It would give us the time necessary to fix it in a future EAPI, or in a > > future version of texlive. > > It would give you an excuse not to fix it properly while a proper fix takes > around 15 minutes. Most of that time due to inefficiency of the ebuild > right now. I guess your argument would have much more weight if you talked to zlogene and submitted a tested patch. It definitely does not take 15 minutes. > > > Furthermore, it means repopulating all the mirrors and forcing users to > > > refetch everything. For the former, it also involves keeping twice as many > > > distfiles during the retention period, for the latter -- until eclean-dist. > > > This is a huge waste of bandwidth, disk space, inodes and everyone's > > > patience. > > > > You'd have the same problems with bandwidth and diskspace when combining > > distfiles. > > There's a major difference between shoving 30000 tiny extra files on mirrors > and a few large files that not only waste less inodes and disk space per se > but also -- as proven above -- compress much better. Are we seriously discussing a 6MB gain here ? > Experience proves that if we don't do it for 2020, then 2021 will be just > the same and then the argument will be that we shouldn't do anything drastic > for 2021, and 2022... > > In the meantime, users would waste hundreds of hours dealing with bad ideas. Experience proves that if nobody submits patches then the one doing the work ends up deciding whatever he wants for whatever reason, yes. So go ahead, submit a patch to the generators and this will make it to 2021. If you are so stressed about it, it is also easy to do a 2020.1 release by fetching up to date packages from texlive. But, as already said, I'm not the one needing convincing here. (In reply to Michał Górny from comment #42) > In my opinion, reducing the number of distfiles is not only the proper > solution but also the only reasonable way forward. I would really prefer if > developers agreed on this but if necessary, I can call for a QA vote on > setting a formal limit on number of distfiles per ebuild. Careful here, you may end up maintaining or last riting it ;) > I'm not saying about forcing one distfile. However, repacking the initial > version and possibly adding updates in separate distfiles is still better > than the status quo. A lot of things are better than the current way of packaging texlive, yes. It hasn't really changed for almost 15 years now. It is "simply" a matter of doing it. > I've finally been able to do a cheap benchmark. I've taken > texlive-latexextra-2020, stubbed pkg_pretend() and pkg_setup(), and limited > src_unpack() to default. Then I've timed 'ebuild ... unpack' with a warm > cache, in two variants. The first variant is using original plethora of > distfiles, the second using a single repacked distfile. I have only tested > USE="-doc -source", I am pretty sure that the gain will grow nonlinearly, I > can test more tomorrow. I also suspect that removing -2019 file would make > Portage faster but too late today to test more. > > separate distfiles: > > real 4m3,236s > user 2m51,853s > sys 1m25,839s > > combined distfile: > > real 2m54,630s > user 2m26,190s > sys 0m28,442s You should stop right here with those numbers: a 2mins gain less than 10 times a year is not worth anyone's time. Any 'emerge -uDN world' call takes much longer than this here. > This has negative impact on woodpecker (where zlogene pushes all those tiny > files), on the mirror network and on user systems. The only counterargument > I've heard so far was more developer's effort. As ulm already said, the same argument you made against renaming distfiles applies to repacking them. Maybe worse: Any single update to a package inside of them means duplicating it. So we are weighting a big overhead once a year and very small updates during the year vs. a smaller overhead once a year and bigger updates during the year. This is not as clear as you seem to believe. Also, unless you are maintaining it, you have no word against "more developer's effort" and no right to ignore it. Making it more complex to fix bugs implies that fixing those will be delayed, or grouped when enough have stacked, and thus users end up losing by having bugs persist for longer. A last note on repacking files that may need a closer look: licensing. I think this is a non issue but we should make sure all of them allows us to do so.
(In reply to Alexis Ballier from comment #46) > (In reply to Zac Medico from comment #40) > > I've opened bug 720180 to make portage delay export of the "A" variable > > until the last moment. In theory, an ebuild could use "export -n A" to > > suppress "[Errno 7] Argument list too long" errors when spawning > > subprocesses. > > This might actually work. A little hacky as this would require changes to > the eclass/ebuilds but seems to be the lesser evil to stop the bleeding for > now. Thanks Zac. Technically, "export -n A" would be a PMS violation. Also it would have to be done in every phase. IMHO it isn't a solution that could be used in EAPI 7.
(In reply to Ulrich Müller from comment #48) > (In reply to Alexis Ballier from comment #46) > > (In reply to Zac Medico from comment #40) > > > I've opened bug 720180 to make portage delay export of the "A" variable > > > until the last moment. In theory, an ebuild could use "export -n A" to > > > suppress "[Errno 7] Argument list too long" errors when spawning > > > subprocesses. > > > > This might actually work. A little hacky as this would require changes to > > the eclass/ebuilds but seems to be the lesser evil to stop the bleeding for > > now. Thanks Zac. > > Technically, "export -n A" would be a PMS violation. Also it would have to > be done in every phase. IMHO it isn't a solution that could be used in EAPI > 7. Why is it a PMS violation ? PMS defines what should be provided to ebuilds, not what they can or cannot do with them AFAIK. I'm not saying it's clean, it definitely isn't, but is there anything better atm ?
(In reply to Alexis Ballier from comment #49) > (In reply to Ulrich Müller from comment #48) > > (In reply to Alexis Ballier from comment #46) > > > (In reply to Zac Medico from comment #40) > > > > I've opened bug 720180 to make portage delay export of the "A" variable > > > > until the last moment. In theory, an ebuild could use "export -n A" to > > > > suppress "[Errno 7] Argument list too long" errors when spawning > > > > subprocesses. > > > > > > This might actually work. A little hacky as this would require changes to > > > the eclass/ebuilds but seems to be the lesser evil to stop the bleeding for > > > now. Thanks Zac. > > > > Technically, "export -n A" would be a PMS violation. Also it would have to > > be done in every phase. IMHO it isn't a solution that could be used in EAPI > > 7. > > Why is it a PMS violation ? PMS defines what should be provided to ebuilds, > not what they can or cannot do with them AFAIK. I'm not saying it's clean, > it definitely isn't, but is there anything better atm ? Distfiles renaming with '->' in SRC_URI might actually be simpler. Meh. I don't know what's best.
(In reply to Alexis Ballier from comment #49) > Why is it a PMS violation ? PMS defines what should be provided to ebuilds, > not what they can or cannot do with them AFAIK. I'm not saying it's clean, > it definitely isn't, but is there anything better atm ? PMS section 11.1 states: "The package manager must define the following environment variables. Not all variables are meaningful in all phases; variables that are not meaningful in a given phase may be unset or set to any value. ***Ebuilds must not attempt to modify any of these variables, unless otherwise specified.***" Asterisks added for emphasis. Removing the variables from the environment could be easily interpreted as modifying them.
That said, I think the intent behind the "must not attempt to modify" statement related more to the value of the variables than to their presence in the environment.
(In reply to Mike Gilbert from comment #52) > That said, I think the intent behind the "must not attempt to modify" > statement related more to the value of the variables than to their presence > in the environment. No, the intent is that you must not touch them or random things may break. For example, default_src_unpack will certainly fall apart if you modify *or unset* A.
(In reply to Mike Gilbert from comment #51) > (In reply to Alexis Ballier from comment #49) > > Why is it a PMS violation ? PMS defines what should be provided to ebuilds, > > not what they can or cannot do with them AFAIK. I'm not saying it's clean, > > it definitely isn't, but is there anything better atm ? > > PMS section 11.1 states: > > "The package manager must define the following environment variables. Not > all variables are meaningful in all phases; variables that are not > meaningful in a given phase may be unset or set to any value. ***Ebuilds > must not attempt to modify any of these variables, unless otherwise > specified.***" > > Asterisks added for emphasis. > > Removing the variables from the environment could be easily interpreted as > modifying them. Yeah, agreed. (In reply to Mike Gilbert from comment #52) > That said, I think the intent behind the "must not attempt to modify" > statement related more to the value of the variables than to their presence > in the environment. We could argue at length here. But it seems safer and simpler to go for the renaming of distfiles from the eclass, so that $A shrinks by itself.
(In reply to Michał Górny from comment #53) > (In reply to Mike Gilbert from comment #52) > > That said, I think the intent behind the "must not attempt to modify" > > statement related more to the value of the variables than to their presence > > in the environment. > > No, the intent is that you must not touch them or random things may break. > For example, default_src_unpack will certainly fall apart if you modify *or > unset* A. Not sure about that last part if it were moved to a bash variable instead of environment. That idea comes with the assumption that such a change will be restricted to the few cases needing it and well tested, so nothing will magically fall apart under a full moon.
(In reply to Alexis Ballier from comment #50) > (In reply to Alexis Ballier from comment #49) > > (In reply to Ulrich Müller from comment #48) > > > (In reply to Alexis Ballier from comment #46) > > > > (In reply to Zac Medico from comment #40) > > > > > I've opened bug 720180 to make portage delay export of the "A" variable > > > > > until the last moment. In theory, an ebuild could use "export -n A" to > > > > > suppress "[Errno 7] Argument list too long" errors when spawning > > > > > subprocesses. > > > > > > > > This might actually work. A little hacky as this would require changes to > > > > the eclass/ebuilds but seems to be the lesser evil to stop the bleeding for > > > > now. Thanks Zac. > > > > > > Technically, "export -n A" would be a PMS violation. Also it would have to > > > be done in every phase. IMHO it isn't a solution that could be used in EAPI > > > 7. > > > > Why is it a PMS violation ? PMS defines what should be provided to ebuilds, > > not what they can or cannot do with them AFAIK. I'm not saying it's clean, > > it definitely isn't, but is there anything better atm ? > > Distfiles renaming with '->' in SRC_URI might actually be simpler. > Meh. I don't know what's best. I would just strip the 'texlive-module-' part and go with the original upstream names + versioning. The fact that it is texlive module says nothing to anyone. Barely anyone even looks it up manually, so this would be suffucuent.
(In reply to Mikle Kolyada from comment #56) > > Distfiles renaming with '->' in SRC_URI might actually be simpler. > > Meh. I don't know what's best. > > I would just strip the 'texlive-module-' part and go with the original > upstream names + versioning. The fact that it is texlive module says nothing > to anyone. Barely anyone even looks it up manually, so this would be > suffucuent. yeah, it is only something I came up with to ensure no name collision happens tree-wide; IMHO better keep a small prefix like 'tl-' but 'texlive-module-' is useless: Those are all CTAN packages repackaged by texlive upstream in the end, so foo-2020.tar.xz is somewhat wrong if the version it represents is foo-1.2.3.4.tar.xz from CTAN.
This one is not relevan anymore, ulm said to take care of pms himself.
