This is bug/limitation when deploying software that has over 1023 file descriptors open AND when using a multiprocessing Pipe or Queue. I am currently encountering this issue with software I am developing. The patches on the site for python 3 apply and compile fine after minor modification of the directories in the patch header. I will attach the patch I am currently using in my local repo. I will give further feedback after more thorough testing. Reproducible: Always Steps to Reproduce: 1. Open 1024 file descriptors 2. Create a multiprocessing Queue and Process 3. Attempt to read from and write to the Queue Actual Results: IOException: "handle out of range in select()" Expected Results: No exception, successful reading and writing to Queue http://bugs.python.org/issue10527 "I have a code that uses multiprocessing.Pipe to communicate with subprocesses. Spawning 500 subprocesses this way works like a charm, but when spawning about 600 of them the pipe ends raise the exception: "handle out of range in select()". I realized that this is because of the FD_SETSIZE limit. To address the situation I quickly hacked together a patch that uses poll() instead of select(), which solves the problem for me. I don't know the reason why select() was chosen for this task (maybe because of windows) but wouldn't it be better to use polling where possible?"
Created attachment 295261 [details, diff] based on multiproc2.patch with minor changes based on multiproc2.patch with minor changes in the header for the directory names so that it would apply with portage for python 2.7
This change is not yet accepted by upstream. Please create a patch for Python 3.3 and send it to upstream.
Marked as FIXED by upstream in https://bugs.python.org/issue10527. Closing.
