Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 456296

Summary: sys-apps/portage-2.2.01.21688+ dies during emerge due to Errno 35
Product: Gentoo/Alt Reporter: Fabian Groffen <grobian>
Component: Prefix SupportAssignee: Gentoo Prefix <prefix>
Status: RESOLVED FIXED    
Severity: blocker CC: dev-portage, srcshelton, VED03370
Priority: Highest    
Version: unspecified   
Hardware: All   
OS: OS X   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=264435
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 462382    

Description Fabian Groffen gentoo-dev 2013-02-09 12:53:41 UTC
Completely reproducable on an existing system, same problem observed during bootstrapping.

% emerge --resume                       
These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-dicts/myspell-en-20081002  0 kB
[ebuild  N     ] dev-libs/boost-1.52.0-r5::gentoo-prefix-svn  USE="nls threads -debug -doc -icu -mpi -python -static-libs -tools" PYTHON_TARGETS="python2_7 python3_3 -python2_5 -python2_6 -python3_1 -python3_2" 0 kB
[ebuild  N     ] media-libs/netpbm-10.59.00::gentoo-prefix-svn  USE="zlib -X -doc -jbig -jpeg -jpeg2k -png -rle -svga -tiff -xml" 0 kB
[ebuild  N     ] app-text/noweb-2.11b-r3  USE="-emacs -examples" 0 kB
[ebuild  N     ] dev-tex/latex2html-2008-r1  USE="-gif -png" 0 kB
[ebuild  N     ] dev-tex/dvipost-1.1-r1  0 kB
[ebuild  N     ] app-text/hunspell-1.3.2-r3  USE="ncurses nls readline -static-libs" LINGUAS="-af -bg -ca -cs -cy -da -de -el -en -eo -es -et -fo -fr -ga -gl -he -hr -hu -ia -id -is -it -km -ku -lt -lv -mk -ms -nb -nl -nn -pl -pt -pt_BR -ro -ru -sk -sl -sq -sv -sw -tn -uk -zu" 0 kB
[ebuild  N     ] app-office/lyx-2.0.5.1  USE="hunspell latex nls -X -aspell (-cups) -debug -dia -docbook -dot -enchant -gnumeric -html -luatex -monolithic-build -rcs -rtf -subversion -svg -xetex" LINGUAS="-ar -ca -cs -da -de -el -en -es -eu -fi -fr -gl -he -hu -ia -id -it -ja -nb -nn -pl -pt -ro -ru -sk -sr -sv -tr -uk -zh_CN -zh_TW" 0 kB

Total: 8 packages (8 new), Size of downloads: 0 kB

*** Resuming merge...

>>> Verifying ebuild manifests

>>> Starting parallel fetch

>>> Emerging (1 of 8) app-dicts/myspell-en-20081002
 * myspell-en_AU-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ...       [ ok ]
 * myspell-en_GB-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ...       [ ok ]
 * myspell-en_US-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ...       [ ok ]
 * myspell-en_CA-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ...       [ ok ]
 * myspell-en_NZ-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ...       [ ok ]
 * myspell-en_ZA-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ...       [ ok ]
 * myspell-en_GB-oed-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ...   [ ok ]
 * myspell-hyph_en_GB-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ...  [ ok ]
 * myspell-thes_en_US_v2-20081002.zip SHA256 SHA512 WHIRLPOOL size ;-) ... [ ok ]
Traceback (most recent call last):
  File "/Library/Gentoo/usr/bin/emerge", line 51, in <module>
    retval = emerge_main()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/main.py", line 1045, in emerge_main
    gc_locals=locals().clear)
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/actions.py", line 3959, in run_action
    myopts, myaction, myfiles, spinner)
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/actions.py", line 473, in action_build
    retval = mergetask.merge()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/Scheduler.py", line 1011, in merge
    rval = self._merge()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/Scheduler.py", line 1396, in _merge
    self._main_loop()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/Scheduler.py", line 1367, in _main_loop
    self._event_loop.iteration()
  File "/Library/Gentoo/usr/lib/portage/pym/portage/util/_eventloop/EventLoop.py", line 260, in iteration
    if not x.callback(f, event, *x.args):
  File "/Library/Gentoo/usr/lib/portage/pym/portage/util/_async/PipeLogger.py", line 89, in _output_handler
    self.wait()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait
    self._wait_hook()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/SpawnProcess.py", line 165, in _pipe_logger_exit
    self.wait()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait
    self._wait_hook()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/EbuildPhase.py", line 264, in _ebuild_exit
    self.wait()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 57, in wait
    self._wait_hook()
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 175, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/EbuildBuild.py", line 184, in _pre_clean_exit
    already_fetched = fetcher.already_fetched(self.settings)
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/EbuildFetcher.py", line 101, in already_fetched
    self.scheduler.output(msg, log_path=self.logfile)
  File "/Library/Gentoo/usr/lib/portage/pym/portage/util/_async/SchedulerInterface.py", line 53, in output
    writemsg_level(msg, level=level, noiselevel=noiselevel)
  File "/Library/Gentoo/usr/lib/portage/pym/portage/util/__init__.py", line 101, in writemsg_level
    writemsg(msg, noiselevel=noiselevel, fd=fd)
  File "/Library/Gentoo/usr/lib/portage/pym/portage/util/__init__.py", line 77, in writemsg
    fd.flush()
BlockingIOError: [Errno 35] write could not complete without blocking
Traceback (most recent call last):

During handling of the above exception, another exception occurred:


====================================
Error in portage.process.run_exitfuncs

  File "/Library/Gentoo/usr/lib/portage/pym/portage/process.py", line 145, in run_exitfuncs
    func(*targs, **kargs)
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/actions.py", line 3845, in emergeexit
    xtermTitleReset()
  File "/Library/Gentoo/usr/lib/portage/pym/portage/output.py", line 301, in xte    f.flush()
[Errno 35] write could not complete without blocking
====================================

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Library/Gentoo/usr/lib/portage/pym/portage/process.py", line 145, in run_exitfuncs
    func(*targs, **kargs)
  File "/Library/Gentoo/usr/lib/portage/pym/_emerge/actions.py", line 3845, in emergeexit
    xtermTitleReset()
  File "/Library/Gentoo/usr/lib/portage/pym/portage/output.py", line 301, in xtermTitleReset
    xtermTitle(default_xterm_title, raw=True)
  File "/Library/Gentoo/usr/lib/portage/pym/portage/output.py", line 267, in xtermTitle
    f.flush()
BlockingIOError: [Errno 35] write could not complete without blocking


disabling openpty on Darwin doesn't make a difference
Comment 1 Zac Medico gentoo-dev 2013-02-09 15:43:23 UTC
You should check to see if the O_NONBLOCK flag is enabled by default for stdout (it shouldn't be):

#!/usr/bin/env python

import fcntl
import os
import sys

stdout_flags = fcntl.fcntl(sys.stdout.fileno(), fcntl.F_GETFL)

if stdout_flags & os.O_NONBLOCK:
	sys.stdout.write("O_NONBLOCK\n")
Comment 2 Stuart Shelton 2013-02-25 23:43:05 UTC
I have the same problem on OS X 10.8, and running the code from Comment 1 outputs nothing.

However, using 'emerge --quiet-build' does seem to fix the issue.
Comment 3 Fabian Groffen gentoo-dev 2013-02-26 07:06:11 UTC
(In reply to comment #2)
> I have the same problem on OS X 10.8, and running the code from Comment 1
> outputs nothing.
> 
> However, using 'emerge --quiet-build' does seem to fix the issue.

agreed, IMO it's some buffer that overflows (isn't read as immediate as necessary)
Comment 4 Fabian Groffen gentoo-dev 2013-02-28 20:31:23 UTC
Burcin Erocal <burcin@erocal.org> wrote:

It seems that the problem was introduced between these two merges.

cefe9ddc5e1b8a1ba727afddd11638723875949c 2013-01-05 19:12
754956bff8273df8ce2c3d9f6ff7fb22126ebd41 2013-01-10 22:01
Comment 5 Fabian Groffen gentoo-dev 2013-02-28 21:10:23 UTC
suspect:

e132ec4f753b9df4769f7e58dfc661617c7375b8  (PipeLogger)

(need to test)
Comment 6 Zac Medico gentoo-dev 2013-03-14 14:16:50 UTC
(In reply to comment #5)
> suspect:
> 
> e132ec4f753b9df4769f7e58dfc661617c7375b8  (PipeLogger)
> 
> (need to test)

I don't think that commit changes any behavior. Maybe this one, though:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=d916da77dfc94eb30c6f512f9d7d727a8f28600c
Comment 7 Fabian Groffen gentoo-dev 2013-03-23 21:26:04 UTC
checked both, not the problem.
Comment 8 Fabian Groffen gentoo-dev 2013-03-23 21:51:52 UTC
2.2.01.21476 works
2.2.01.21535 works
2.2.01.21567 broken
2.2.01.21580 broken

broken 21565:80a02c9243bf  Use EventLoop, no SchedulerInterface if possible
broken 21563:01e5fda3f1e1  SpawnProcess: handle fcntl ENOTTY for FreeBSD
broken 21548:762ffde02f9f  SpawnProcess: stdout_fd FD_CLOEXEC
works  21547:af05371e830f  ManifestTask: use PipeLogger for monitoring

victim: 762ffde02f9f  -> git: 2a05612d23561d4606e93e73a8e021dc91291ff6

backing out 762ffde02f9f on the latest (21580) results in working portage
Comment 9 Fabian Groffen gentoo-dev 2013-03-23 22:00:43 UTC
I don't quite understand why we need FD_CLOEXEC here, but I disabled it for Darwin in 80f9a5dff0605a5114bbed93464be68ffc0e668a.
Comment 10 Zac Medico gentoo-dev 2013-03-23 22:05:46 UTC
(In reply to comment #9)
> I don't quite understand why we need FD_CLOEXEC here, but I disabled it for
> Darwin in 80f9a5dff0605a5114bbed93464be68ffc0e668a.

It's just good practice for any file descriptor that doesn't explicitly need to be inherited through an exec call. It's good to skip it on platforms where it causes problems, so I've merged your fix.
Comment 11 Fabian Groffen gentoo-dev 2013-03-23 22:14:40 UTC
Fix released in 2.2.01.21858
Comment 12 MATSUI Tetsushi 2013-03-24 00:56:21 UTC
*** Bug 462706 has been marked as a duplicate of this bug. ***
Comment 13 Zac Medico gentoo-dev 2013-09-01 20:28:11 UTC
It turns out that the wrong fcntl commands were used (F_GETFD/F_SETFD should have been used instead of F_GETFL/F_SETFL):

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=30c652a9db1014fc720f7d6055520a07b731c984

If someone would like to test setting _disable_cloexec_stdout = False at the top of ${EPREFIX}/usr/lib/portage/pym/_emerge/SpawnProcess.py on Darwin, then that would be great.
Comment 14 Fabian Groffen gentoo-dev 2013-09-14 14:37:25 UTC
This commit is in Prefix tree now (2.2.4.*) it solves the disconnect problem on Solaris, which I'm very happy with!  Darwin bit needs testing.
Comment 15 Fabian Groffen gentoo-dev 2013-09-15 13:33:26 UTC
It appears as this has fixed the problem on Darwin.  I'll keep it in testing for a bit longer.
Comment 16 Fabian Groffen gentoo-dev 2013-09-18 18:17:37 UTC
I did quite some emerging with _disable_cloexec_stdout = False on Darwin, and didn't find any problem, so I feel it's good to go.  Zac, would you please remove this in master at your earliest convenience?