It would be nice to have a configuration option to filter build logs through a command like ansifilter, in order to ensure that build logs do not contain ANSI escape codes.
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/threads/2020-02/ https://github.com/gentoo/portage/pull/509
I want to redo the patch to support separate filter commands for file and tty output. For example we can use PORTAGE_LOG_FILTER_FILE to filter non-tty output, and PORTAGE_LOG_FILTER_TTY to filter tty output.
Also, some discussion on IRC yielded the idea to move the compression feature into a separate section of a filter pseudo-pipeline, and remove it from the 'critical path' of execution potentially. The new logging features will be a useful upgrade to catalyst as well, as it currently only logs to either tty or file, and the facility to log to both with varying degrees of verbosity will be useful for both manual and automatic users.
I've found that there's some code in lib/_emerge/EbuildPhase.py and lib/portage/util/_async/SchedulerInterface.py which writes directly to the log, bypassing the filter. The affected code will have to spawn a filter process, and in order to do that it will have to be converted to coroutines (any callers using it to write to the log will also have to be converted to coroutines).
The simplest fix will probably be to have a long-running filter process, and when an ebuild process is not running we can consider writes from python code to its input pipe to be essentially non-blocking since the pipe's buffer will typically be empty or nearly empty in this case.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=a287c49f84ad3af7c8e20bebd116ea972f318e04 commit a287c49f84ad3af7c8e20bebd116ea972f318e04 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-02-17 21:39:21 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-02-17 22:49:32 +0000 AbstractEbuildProcess: add _async_start coroutine Convert the _start method to an _async_start coroutine, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 33 +++++++++------------------------ lib/_emerge/MiscFunctionsProcess.py | 8 +++++--- 2 files changed, 14 insertions(+), 27 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=d66e9ec0b10522528d62e18b83e012c1ec121787 commit d66e9ec0b10522528d62e18b83e012c1ec121787 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-02-17 18:32:06 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-02-17 22:29:09 +0000 AsynchronousTask: add coroutine async_start method Add coroutine async_start coroutine method which calls an _async_start template method. Eventually, subclasses having _start implementations that need to write to a build log will be required to implement an _async_start coroutine method to replace the _start method as discussed in bug 709746. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AsynchronousTask.py | 22 +++++++++++++++++----- lib/_emerge/CompositeTask.py | 17 ++++++++++++++--- lib/_emerge/TaskSequence.py | 7 ++++++- lib/portage/tests/ebuild/test_doebuild_fd_pipes.py | 6 ++++-- lib/portage/tests/ebuild/test_doebuild_spawn.py | 6 +++--- lib/portage/tests/ebuild/test_fetch.py | 9 ++++++--- lib/portage/tests/ebuild/test_ipc_daemon.py | 4 ++-- lib/portage/tests/ebuild/test_spawn.py | 4 ++-- .../test_lazy_import_portage_baseline.py | 6 +++--- lib/portage/tests/locks/test_asynchronous_lock.py | 18 +++++++++--------- lib/portage/tests/process/test_PopenProcess.py | 6 +++--- .../tests/process/test_PopenProcessBlockingIO.py | 4 ++-- lib/portage/tests/process/test_poll.py | 4 ++-- .../tests/util/futures/test_iter_completed.py | 16 +++++++++++----- lib/portage/tests/util/test_file_copier.py | 6 +++--- 15 files changed, 87 insertions(+), 48 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c7e52d0466211907d20cdbc04f1e90e7da626694 commit c7e52d0466211907d20cdbc04f1e90e7da626694 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-02-17 23:39:12 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-02-18 00:14:00 +0000 EbuildPhase: add _async_start coroutine Convert the _start method to an _async_start coroutine, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Also convert SequentialTaskQueue to use the async_start method, which is now required in order to start EbuildPhase instances. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/EbuildPhase.py | 21 +++++++++++---------- lib/_emerge/SequentialTaskQueue.py | 20 ++++++++++++++++---- 2 files changed, 27 insertions(+), 14 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=8f47d3fe1190d4476ae9eebfafcebdfb1794fc05 commit 8f47d3fe1190d4476ae9eebfafcebdfb1794fc05 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-02-18 07:43:12 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-02-20 09:15:52 +0000 AsyncScheduler: use async_start method Convert AsyncScheduler to use the async_start method, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Also fix async_iter_completed to be compatible with callback scheduling differences introduced by migration to the async_start method. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/ebuild/test_doebuild_fd_pipes.py | 8 ++--- .../tests/util/futures/test_iter_completed.py | 2 ++ lib/portage/util/_async/AsyncScheduler.py | 20 ++++++++++-- lib/portage/util/futures/iter_completed.py | 38 +++++++++++++++++----- 4 files changed, 53 insertions(+), 15 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=5c40c3e7ec180c9c7d1eea521d69487177c7f519 commit 5c40c3e7ec180c9c7d1eea521d69487177c7f519 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-02-23 23:17:16 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-02-24 02:35:15 +0000 SpawnProcess: use async_start method (bug 709746) Convert SpawnProcess to use the async_start method, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Also update AbstractEbuildProcess, AsynchronousLock, BinpkgExtractorAsync, BinpkgFetcher, EbuildFetcher, IterCompletedTestCase, AsyncFunction, and FileDigester for compatibility with SpawnProcess async_start. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 2 +- lib/_emerge/AsynchronousLock.py | 15 ++++++++++++--- lib/_emerge/BinpkgExtractorAsync.py | 9 +++++++-- lib/_emerge/BinpkgFetcher.py | 9 +++++++-- lib/_emerge/EbuildFetcher.py | 9 +++++++-- lib/_emerge/SpawnProcess.py | 8 ++++++-- lib/portage/dbapi/bintree.py | 4 ++-- lib/portage/tests/util/futures/test_iter_completed.py | 6 +++++- lib/portage/util/_async/AsyncFunction.py | 9 +++++++-- lib/portage/util/_async/FileDigester.py | 9 +++++++-- 10 files changed, 61 insertions(+), 19 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=27712651aa7014a960b012dc89457df09677edc1 commit 27712651aa7014a960b012dc89457df09677edc1 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-02-24 08:06:11 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-02-24 10:26:33 +0000 PipeLogger: non-blocking write to pipe (bug 709746) Add support to write to a non-blocking pipe instead of a log file. This is needed for the purposes of bug 709746, where PipeLogger will write to a pipe that is drained by anoher PipeLogger instance which is running in the same process. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/process/test_PopenProcess.py | 41 +++++++++++++++- lib/portage/util/_async/PipeLogger.py | 67 +++++++++++++++++++++----- 2 files changed, 94 insertions(+), 14 deletions(-)
PORTAGE_LOG_FILTER_FILE patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/c8c9ecc70d91fd9dc17e8fde892535c3 https://github.com/gentoo/portage/pull/514
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=73f72f526a66b9953a46868cc1390fde2820997f commit 73f72f526a66b9953a46868cc1390fde2820997f Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-03-01 02:17:52 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-03-01 04:29:41 +0000 Support PORTAGE_LOG_FILTER_FILE (bug 709746) This variable specifies a command that filters build log output to a log file. The plan is to extend this to support a separate filter for tty output in the future. In order to enable the EbuildPhase class to write elog messages to the build log with PORTAGE_LOG_FILTER_FILE support, convert its _elog method to a coroutine, and add a SchedulerInterface async_output method for it to use. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 2 + lib/_emerge/EbuildPhase.py | 36 +++++-- lib/_emerge/SpawnProcess.py | 32 ++++-- .../package/ebuild/_config/special_env_vars.py | 8 +- lib/portage/util/_async/BuildLogger.py | 116 +++++++++++++++++++++ lib/portage/util/_async/SchedulerInterface.py | 32 +++++- man/make.conf.5 | 7 +- 7 files changed, 213 insertions(+), 20 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=196e51a0010cf17a3733fe6bc2516cf9e01a4a8a commit 196e51a0010cf17a3733fe6bc2516cf9e01a4a8a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-03-01 06:51:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-03-01 06:58:10 +0000 sys-apps/portage: Bump to version 2.3.90 #601252 DISTDIR NFS root_squash support #709746 new PORTAGE_LOG_FILTER_FILE variable specifies a command that filters build log output to a log file #710076 einstalldocs: Fix test for DOCS being unset Bug: https://bugs.gentoo.org/711148 Bug: https://bugs.gentoo.org/601252 Bug: https://bugs.gentoo.org/709746 Bug: https://bugs.gentoo.org/710076 Package-Manager: Portage-2.3.90, Repoman-2.3.20 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.90.ebuild | 271 +++++++++++++++++++++++++++++++++ 2 files changed, 272 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=e24859eaa03ec86e10d842296f5570dd98bed4b7 commit e24859eaa03ec86e10d842296f5570dd98bed4b7 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-03-03 06:52:18 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-03-03 06:52:46 +0000 Rename PORTAGE_LOG_FILTER_FILE_CMD from PORTAGE_LOG_FILTER_FILE Suggested-by: Michael 'veremitz' Everitt <gentoo@veremit.xyz> Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> NEWS | 2 +- RELEASE-NOTES | 2 +- lib/_emerge/AbstractEbuildProcess.py | 2 +- lib/_emerge/EbuildPhase.py | 2 +- lib/portage/package/ebuild/_config/special_env_vars.py | 4 ++-- lib/portage/util/_async/BuildLogger.py | 2 +- man/make.conf.5 | 2 +- 7 files changed, 8 insertions(+), 8 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2fe510215d12499f0f4f955adfd0f22e761a015e commit 2fe510215d12499f0f4f955adfd0f22e761a015e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-03-04 10:16:59 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-03-04 10:25:03 +0000 sys-apps/portage: Bump to version 2.3.92 #601252 emerge --pretend --fetchonly event loop recursion #709334 socks5-server.py async and await coroutine syntax #709746 Rename PORTAGE_LOG_FILTER_FILE_CMD from PORTAGE_LOG_FILTER_FILE #711322 emerge hang after src_install #711362 egencache AttributeError: 'NoneType' object has no attribute 'ebuild' #711400 AttributeError: 'NoneType' object has no attribute 'depth' Bug: https://bugs.gentoo.org/711148 Bug: https://bugs.gentoo.org/709334 Bug: https://bugs.gentoo.org/709746 Closes: https://bugs.gentoo.org/711322 Bug: https://bugs.gentoo.org/711362 Bug: https://bugs.gentoo.org/711400 Package-Manager: Portage-2.3.92, Repoman-2.3.20 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 2 +- sys-apps/portage/{portage-2.3.91.ebuild => portage-2.3.92.ebuild} | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
For the time being, I'm planning to revert the changes related to this bug, since they appear to have triggered an elusive emerge hang issue reported as bug 716636.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b7ff1f6c1bf0e821e565c606d7df7d70bf575347 commit b7ff1f6c1bf0e821e565c606d7df7d70bf575347 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-08 06:52:08 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-04-08 06:55:33 +0000 sys-apps/portage: Bump to version 2.3.97 #709746 temporarily remove PORTAGE_LOG_FILTER_FILE_CMD support #715162 infer implicit soname from file basename, for musl #716636 emerge hangs in releases after 2.3.89-r1 Bug: https://bugs.gentoo.org/711148 Bug: https://bugs.gentoo.org/709746 Bug: https://bugs.gentoo.org/715162 Closes: https://bugs.gentoo.org/716636 Package-Manager: Portage-2.3.97, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.97.ebuild | 263 +++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=3e46825a047067a96ed997fe394f85e042e542a8 commit 3e46825a047067a96ed997fe394f85e042e542a8 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-02-24 08:06:11 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-13 06:30:01 +0000 PipeLogger: non-blocking write to pipe (bug 709746) Add support to write to a non-blocking pipe instead of a log file. This is needed for the purposes of bug 709746, where PipeLogger will write to a pipe that is drained by anoher PipeLogger instance which is running in the same process. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/process/test_PipeLogger.py | 54 +++++++++++++++++++++++ lib/portage/util/_async/PipeLogger.py | 66 ++++++++++++++++++++++------ 2 files changed, 107 insertions(+), 13 deletions(-)
(In reply to Larry the Git Cow from comment #18) > The bug has been referenced in the following commit(s): > > https://gitweb.gentoo.org/proj/portage.git/commit/ > ?id=3e46825a047067a96ed997fe394f85e042e542a8 > > commit 3e46825a047067a96ed997fe394f85e042e542a8 > Author: Zac Medico <zmedico@gentoo.org> > AuthorDate: 2020-02-24 08:06:11 +0000 > Commit: Zac Medico <zmedico@gentoo.org> > CommitDate: 2020-06-13 06:30:01 +0000 > > PipeLogger: non-blocking write to pipe (bug 709746) > > Add support to write to a non-blocking pipe instead of a > log file. This is needed for the purposes of bug 709746, > where PipeLogger will write to a pipe that is drained > by anoher PipeLogger instance which is running in the same > process. > > Bug: https://bugs.gentoo.org/709746 > Signed-off-by: Zac Medico <zmedico@gentoo.org> > > lib/portage/tests/process/test_PipeLogger.py | 54 +++++++++++++++++++++++ > lib/portage/util/_async/PipeLogger.py | 66 > ++++++++++++++++++++++------ > 2 files changed, 107 insertions(+), 13 deletions(-) This broke FEATURES=compress-build-logs, but it can be fixed by offloading compression to a gzip process (bug 712758), so I'll look into implementing that now.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=54cdf7e6cf9ec2030283ab23d659e10562267c6b commit 54cdf7e6cf9ec2030283ab23d659e10562267c6b Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-13 23:59:43 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-14 00:00:16 +0000 PipeLogger: fix FEATURES=compress-build-logs (bug 709746) For FEATURES=compress-build-logs, use log_file.write since data written directly to the file descriptor bypasses compression. Fixes: 3e46825a0470 ("PipeLogger: non-blocking write to pipe (bug 709746)") Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/util/_async/PipeLogger.py | 7 +++++++ 1 file changed, 7 insertions(+)
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/efa1665d961fda2ca921b9fea59224fc https://github.com/gentoo/portage/pull/558
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=ab7556a5c0df418d61148a05347f4c73115e5747 commit ab7556a5c0df418d61148a05347f4c73115e5747 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-03-01 02:17:52 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-14 22:42:09 +0000 Support PORTAGE_LOG_FILTER_FILE_CMD (bug 709746) This variable specifies a command that filters build log output to a log file. The plan is to extend this to support a separate filter for tty output in the future. In order to enable the EbuildPhase class to write elog messages to the build log with PORTAGE_LOG_FILTER_FILE_CMD support, convert its _elog method to a coroutine, and add a SchedulerInterface async_output method for it to use. Use a new BuildLogger class to manage log output (with or without a filter command), with compression support provided by PipeLogger. BuildLogger has a stdin property which provides access to a writable binary file stream (refers to a pipe) that log content is written to. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 3 +- lib/_emerge/BinpkgFetcher.py | 3 +- lib/_emerge/EbuildFetcher.py | 3 +- lib/_emerge/EbuildPhase.py | 47 +++++++-- lib/_emerge/SpawnProcess.py | 58 ++++++++--- lib/portage/dbapi/_MergeProcess.py | 3 +- .../package/ebuild/_config/special_env_vars.py | 8 +- lib/portage/util/_async/BuildLogger.py | 109 +++++++++++++++++++++ lib/portage/util/_async/SchedulerInterface.py | 32 +++++- man/make.conf.5 | 7 +- 10 files changed, 243 insertions(+), 30 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5330fc62703c0e2c535eb2ac8247d43faf501c8e commit 5330fc62703c0e2c535eb2ac8247d43faf501c8e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-14 23:33:47 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-14 23:47:09 +0000 sys-apps/portage: Bump to version 2.3.101 #661518 repos.conf: Add bool sync-openpgp-key-refresh option #709746 New PORTAGE_LOG_FILTER_FILE_CMD variable specifies a command that filters build log output to a log file #719810 Escape percent-signs in mirror url #725934 _better_cache._scan_cat: Avoid stat calls #728046 ecompress: Prefix eqawarn messages with QA Notice Bug: https://bugs.gentoo.org/721152 Bug: https://bugs.gentoo.org/661518 Bug: https://bugs.gentoo.org/709746 Bug: https://bugs.gentoo.org/719810 Bug: https://bugs.gentoo.org/725934 Bug: https://bugs.gentoo.org/728046 Package-Manager: Portage-2.3.101, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.101.ebuild | 263 ++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=2f657c642ff292fd44ed9cb253ab6d13da84303f commit 2f657c642ff292fd44ed9cb253ab6d13da84303f Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-15 19:46:03 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-15 19:47:05 +0000 Revert "Support PORTAGE_LOG_FILTER_FILE_CMD (bug 709746)" This reverts commit ab7556a5c0df418d61148a05347f4c73115e5747. We've had reports of emerge hangs, so reverting this for now. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 3 +- lib/_emerge/BinpkgFetcher.py | 3 +- lib/_emerge/EbuildFetcher.py | 3 +- lib/_emerge/EbuildPhase.py | 47 ++------- lib/_emerge/SpawnProcess.py | 58 +++-------- lib/portage/dbapi/_MergeProcess.py | 3 +- .../package/ebuild/_config/special_env_vars.py | 8 +- lib/portage/util/_async/BuildLogger.py | 109 --------------------- lib/portage/util/_async/SchedulerInterface.py | 32 +----- man/make.conf.5 | 7 +- 10 files changed, 30 insertions(+), 243 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=dd8ae90ca895c1dd17846dd9110894312c7074a6 commit dd8ae90ca895c1dd17846dd9110894312c7074a6 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-15 19:51:54 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-15 19:54:15 +0000 sys-apps/portage: Revbump to 2.3.101-r1 #709746 revert due to reports of emerge hangs Bug: https://bugs.gentoo.org/721152 Bug: https://bugs.gentoo.org/709746 Package-Manager: Portage-2.3.101, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + .../portage/{portage-2.3.101.ebuild => portage-2.3.101-r1.ebuild} | 7 +++++-- 2 files changed, 6 insertions(+), 2 deletions(-)
We need to review the code in https://github.com/gentoo/portage/pull/558 to find out what triggers the hang. For one thing we should audit the SpawnProcess subclasses to see if any need to override _async_waitpid like AsyncFunction does here: https://github.com/gentoo/portage/commit/d343bd2eebbb771cb6e43d4e3add85614db6b2a6
I think this shows all the SpawnProcess subclasses that need to be audited: > $ git grep '(\(SpawnProcess\|ForkProcess\|AbstractEbuildProcess\))' > lib/_emerge/AbstractEbuildProcess.py:class AbstractEbuildProcess(SpawnProcess): > lib/_emerge/BinpkgExtractorAsync.py:class BinpkgExtractorAsync(SpawnProcess): > lib/_emerge/BinpkgFetcher.py:class _BinpkgFetcherProcess(SpawnProcess): > lib/_emerge/EbuildFetcher.py:class _EbuildFetcherProcess(ForkProcess): > lib/_emerge/EbuildProcess.py:class EbuildProcess(AbstractEbuildProcess): > lib/_emerge/EbuildSpawnProcess.py:class EbuildSpawnProcess(AbstractEbuildProcess): > lib/_emerge/MiscFunctionsProcess.py:class MiscFunctionsProcess(AbstractEbuildProcess): > lib/portage/dbapi/_MergeProcess.py:class MergeProcess(ForkProcess): > lib/portage/dbapi/_SyncfsProcess.py:class SyncfsProcess(ForkProcess): > lib/portage/package/ebuild/_parallel_manifest/ManifestProcess.py:class ManifestProcess(ForkProcess): > lib/portage/tests/ebuild/test_doebuild_fd_pipes.py:class DoebuildProcess(ForkProcess): > lib/portage/tests/ebuild/test_ipc_daemon.py:class SleepProcess(ForkProcess): > lib/portage/tests/util/futures/test_iter_completed.py:class SleepProcess(ForkProcess): > lib/portage/util/_async/AsyncFunction.py:class AsyncFunction(ForkProcess): > lib/portage/util/_async/FileDigester.py:class FileDigester(ForkProcess): > lib/portage/util/_async/ForkProcess.py:class ForkProcess(SpawnProcess):
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=ca763549507d995e91a49753b13bcca8748fae6c commit ca763549507d995e91a49753b13bcca8748fae6c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-16 03:14:36 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-16 03:14:49 +0000 Revert "PipeLogger: non-blocking write to pipe (bug 709746)" This reverts commit 3e46825a047067a96ed997fe394f85e042e542a8. We've had reports of emerge hangs, so reverting this for now. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/process/test_PipeLogger.py | 54 -------------------- lib/portage/util/_async/PipeLogger.py | 73 +++++----------------------- 2 files changed, 13 insertions(+), 114 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e2f0155caafb1dde44bdf835554f9fbb932d7513 commit e2f0155caafb1dde44bdf835554f9fbb932d7513 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-16 03:20:34 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-16 03:20:59 +0000 sys-apps/portage: Revbump to 2.3.101-r2 #709746 revert due to reports of emerge hangs Bug: https://bugs.gentoo.org/721152 Bug: https://bugs.gentoo.org/709746 Package-Manager: Portage-2.3.101, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 2 +- .../portage/{portage-2.3.101-r1.ebuild => portage-2.3.101-r2.ebuild} | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
(In reply to Larry the Git Cow from comment #28) > The bug has been referenced in the following commit(s): > > https://gitweb.gentoo.org/proj/portage.git/commit/ > ?id=ca763549507d995e91a49753b13bcca8748fae6c > > commit ca763549507d995e91a49753b13bcca8748fae6c > Author: Zac Medico <zmedico@gentoo.org> > AuthorDate: 2020-06-16 03:14:36 +0000 > Commit: Zac Medico <zmedico@gentoo.org> > CommitDate: 2020-06-16 03:14:49 +0000 > > Revert "PipeLogger: non-blocking write to pipe (bug 709746)" > > This reverts commit 3e46825a047067a96ed997fe394f85e042e542a8. > We've had reports of emerge hangs, so reverting this for now. > > Bug: https://bugs.gentoo.org/709746 > Signed-off-by: Zac Medico <zmedico@gentoo.org> > > lib/portage/tests/process/test_PipeLogger.py | 54 -------------------- > lib/portage/util/_async/PipeLogger.py | 73 > +++++----------------------- > 2 files changed, 13 insertions(+), 114 deletions(-) I think the problem with this patch is that the remove_reader and remove_writer calls in the finally blocks were executed after the corresponding file descriptor had been closed and reused for another purpose. The exception shown in bug 716636 comment #10 proves that the finally blocks can easily execute after the file descriptor has been closed and reused.
Updated patches posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/e0e62c6cdbb942db4173f23bfb17510d https://github.com/gentoo/portage/pull/561
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=66da001e4a552b4794bd102d47a43a938583275a commit 66da001e4a552b4794bd102d47a43a938583275a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-23 02:33:05 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-23 02:34:45 +0000 sys-apps/portage: Bump to version 2.3.102 #709746 Support PORTAGE_LOG_FILTER_FILE_CMD once again, now that bug 716636 is fixed #716636 Fix emerge hang triggered by unsafe remove_reader and remove_writer calls related to bug 709746 Bug: https://bugs.gentoo.org/721152 Bug: https://bugs.gentoo.org/709746 Bug: https://bugs.gentoo.org/716636 Package-Manager: Portage-2.3.102, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.102.ebuild | 263 ++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=dd69ce742c62b9515cf7ae37e46bcf7f178777db commit dd69ce742c62b9515cf7ae37e46bcf7f178777db Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-03-01 02:17:52 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-23 02:13:06 +0000 Support PORTAGE_LOG_FILTER_FILE_CMD (bug 709746) This variable specifies a command that filters build log output to a log file. The plan is to extend this to support a separate filter for tty output in the future. In order to enable the EbuildPhase class to write elog messages to the build log with PORTAGE_LOG_FILTER_FILE_CMD support, convert its _elog method to a coroutine, and add a SchedulerInterface async_output method for it to use. Use a new BuildLogger class to manage log output (with or without a filter command), with compression support provided by PipeLogger. BuildLogger has a stdin property which provides access to a writable binary file stream (refers to a pipe) that log content is written to. Bug: https://bugs.gentoo.org/709746 Reviewed-by: Brian Dolbec <dolsen@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/AbstractEbuildProcess.py | 3 +- lib/_emerge/BinpkgFetcher.py | 3 +- lib/_emerge/EbuildFetcher.py | 3 +- lib/_emerge/EbuildPhase.py | 47 +++++++-- lib/_emerge/SpawnProcess.py | 58 ++++++++--- lib/portage/dbapi/_MergeProcess.py | 3 +- .../package/ebuild/_config/special_env_vars.py | 8 +- lib/portage/util/_async/BuildLogger.py | 109 +++++++++++++++++++++ lib/portage/util/_async/SchedulerInterface.py | 32 +++++- man/make.conf.5 | 7 +- 10 files changed, 243 insertions(+), 30 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=72ac22e722549833c1ee7e7ad1b585db55f7dafc commit 72ac22e722549833c1ee7e7ad1b585db55f7dafc Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-19 03:04:52 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-23 02:13:05 +0000 PipeLogger: non-blocking write to pipe (bug 709746) Add support to write to a non-blocking pipe instead of a log file. This is needed for the purposes of bug 709746, where PipeLogger will write to a pipe that is drained by anoher PipeLogger instance which is running in the same process. Bug: https://bugs.gentoo.org/709746 Reviewed-by: Brian Dolbec <dolsen@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/tests/process/test_PipeLogger.py | 58 +++++++++++++++++++++ lib/portage/util/_async/PipeLogger.py | 75 +++++++++++++++++++++------- 2 files changed, 116 insertions(+), 17 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6f495669aa852966ea87fd3ad2c72ede102ce86a commit 6f495669aa852966ea87fd3ad2c72ede102ce86a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-06-24 03:24:22 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-06-24 05:56:54 +0000 sys-apps/portage: Bump to version 2.3.103 #709746 set non-blocking for build_logger stdin in EbuildPhase _elog method #727522 ecompress: fix "Argument list too long" for sed Bug: https://bugs.gentoo.org/721152 Bug: https://bugs.gentoo.org/709746 Bug: https://bugs.gentoo.org/727522 Package-Manager: Portage-2.3.103, Repoman-2.3.23 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 2 +- sys-apps/portage/portage-2.3.103.ebuild | 263 ++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+), 1 deletion(-)
