| Summary: | dev-python/pymongo-2.5.1 test suite segfaults | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Patrick Lauer <patrick> |
| Component: | New packages | Assignee: | Johan Bergström <bugs> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | bernie.hackett, idella4, julien, ultrabug |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: |
pymongo-2.5.1-r1.ebuild
pymongo-2.5.1-r1.ebuild pymongo-2.5.1-r1.ebuild |
||
|
Description
Patrick Lauer
2013-03-26 05:58:36 UTC
I actually get child process started successfully, parent exiting [ ok ] /mnt/gen2/TmpDir/portage/dev-python/pymongo-2.5/temp/environment: line 3533: 8011 Segmentation fault nosetests killing process with pid: 7710 >>> Source compiled.
* python2_7: running distutils-r1_run_phase python_test
* Trying to start mongod on port 27017 ...
about to fork child process, waiting until server is ready for connections.
forked process: 28446
all output going to: /var/tmp/portage/dev-python/pymongo-2.5/temp/python2.7/mongod.log
child process started successfully, parent exiting [ ok ]
/var/tmp/portage/dev-python/pymongo-2.5/temp/environment: line 3537: 28763 Segmentation fault nosetests
killing process with pid: 28446
* ERROR: dev-python/pymongo-2.5 failed (test phase):
* Tests fail with python2.7
I'm not sure what the segfault is (is that from PyMongo?) but the missing test files are included in pymongo-2.5.1. See this commit: https://github.com/mongodb/mongo-python-driver/commit/c0673df4eaeac3476cb8bf2bcc265319ae7ffa7f If you guys actually want to file upstream bugs: https://jira.mongodb.org/browse/PYTHON I'm ultimately responsible and use gentoo for everything. *** Bug 468226 has been marked as a duplicate of this bug. *** (In reply to comment #3) > I'm not sure what the segfault is (is that from PyMongo?) but the missing > test files are included in pymongo-2.5.1. See this commit: > > https://github.com/mongodb/mongo-python-driver/commit/ > c0673df4eaeac3476cb8bf2bcc265319ae7ffa7f > > If you guys actually want to file upstream bugs: > > https://jira.mongodb.org/browse/PYTHON > > I'm ultimately responsible and use gentoo for everything. Hey Bernie, great to hear that upstream also likes this ultra-cool linux dist 8-) So, a couple of things probably needs to be done here. 1) I also get segfaults (in libpython), needs more investigation 2) Our current src_test will try to launch a server from 27017 and forward, but the nose test suite assumes we use 27017. Would you consider passing port and/or socket to the test suite? It's probably good practise to avoid using the current mongodb instance for tests anyway? 3) As for Patrick's particular issue (certificate), I can at least say that it works for me in 2.5.1: Ran 478 tests in 70.651s OK (SKIP=129) 4) Lets get this in shape! python@ or ultrabug@, I would suggest starting off with renaming 2.5 to 2.5.1, and in parallel investigate segfaults and improve the test suite. For now, we could just sed port (and socket) to whatever we use. (In reply to comment #5) > So, a couple of things probably needs to be done here. > > 1) I also get segfaults (in libpython), needs more investigation Vital missing detail: the segfaults only occur while using emerge, not running them by invoking nosetests myself. I'm using FEATURES="userpriv user sandbox" > 2) Our current src_test will try to launch a server from 27017 and forward, > but the nose test suite assumes we use 27017. Would you consider passing > port and/or socket to the test suite? It's probably good practise to avoid > using the current mongodb instance for tests anyway? > 3) As for Patrick's particular issue (certificate), I can at least say that > it works for me in 2.5.1: > > Ran 478 tests in 70.651s > OK (SKIP=129) > > 4) Lets get this in shape! python@ or ultrabug@, I would suggest starting > off with renaming 2.5 to 2.5.1, and in parallel investigate segfaults and > improve the test suite. For now, we could just sed port (and socket) to > whatever we use. > Hey Bernie, great to hear that upstream also likes this ultra-cool linux > dist 8-) I've been using gentoo since you had to build from stage 1. PyMongo has been developed on it the entire time I've been maintainer. =) > 1) I also get segfaults (in libpython), needs more investigation No idea what that's about. I suspect it has something to do with running the tests through emerge. > 2) Our current src_test will try to launch a server from 27017 and forward, > but the nose test suite assumes we use 27017. Would you consider passing > port and/or socket to the test suite? It's probably good practise to avoid > using the current mongodb instance for tests anyway? Well, you can set env variables for it: https://github.com/mongodb/mongo-python-driver/blob/master/test/__init__.py#L26-L34 > 3) As for Patrick's particular issue (certificate), I can at least say that > it works for me in 2.5.1: > > Ran 478 tests in 70.651s > OK (SKIP=129) Great! > 4) Lets get this in shape! python@ or ultrabug@, I would suggest starting > off with renaming 2.5 to 2.5.1, and in parallel investigate segfaults and > improve the test suite. For now, we could just sed port (and socket) to > whatever we use. I noticed 432552 a while ago (trying to run PyMongo's tests under emerge with python 3.x). When I did py3k support the version of nose in portage didn't support running 2to3, so there were some hacks I added to setup.py to make things work right. Clearly the hacks don't play well with emerge. I'm planning to revisit all that now that the latest nose is available. Hopefully with version 2.6 you can make it available in 3.x. I think the segfault originates from test_socket_reclamation, but since I can no longer even get the tests started I'm going from memory when I ran it and the mongod did start, so I can't replicate it 2.5.1 in tree, fixes missing certificates (upstream). Lets focus on the segfault and improving test ports next. Created attachment 349112 [details]
pymongo-2.5.1-r1.ebuild
The segfault seemed to originate from nosetests using wrong python version. Here's the full error:
======================================================================
ERROR: Failure: AttributeError (PyCapsule_Import "datetime.datetime_CAPI" is not valid)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/nose/loader.py", line 413, in loadTestsFromName
addr.filename, addr.module)
File "/usr/lib64/python2.6/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/lib64/python2.6/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/test/__init__.py", line 20, in <module>
import pymongo
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/pymongo/__init__.py", line 80, in <module>
from pymongo.connection import Connection
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/pymongo/connection.py", line 39, in <module>
from pymongo.mongo_client import MongoClient
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/pymongo/mongo_client.py", line 44, in <module>
from bson.py3compat import b
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/bson/__init__.py", line 41, in <module>
from bson import _cbson
AttributeError: PyCapsule_Import "datetime.datetime_CAPI" is not valid
----------------------------------------------------------------------
Attached is an updated 2.5.1 ebuild:
1. support python 3.x
2. fix test segfaults
3. make sure we pass the DB_PORT we actually use our selves
I unfortunately get 4 (two distinct) new test errors. These only occur
when passing DB_PORT=27018. I initially thought it was related to differences
in mongodb startup flags, but after changing port in the mongodb init
script, I still got the same result:
======================================================================
ERROR: test_document_class (test.test_master_slave_connection.TestMasterSlaveConnection)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/test/test_master_slave_connection.py", line 472, in test_document_class
db.test.insert({"x": 1}, w=w)
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/pymongo/collection.py", line 357, in insert
continue_on_error, self.__uuid_subtype), safe)
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/pymongo/master_slave_connection.py", line 162, in _send_message
return self.__master._send_message(message, with_last_error)
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/pymongo/mongo_client.py", line 920, in _send_message
rv = self.__check_response_to_last_error(response)
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/pymongo/mongo_client.py", line 865, in __check_response_to_last_error
raise OperationFailure(details["err"])
OperationFailure: norepl
======================================================================
FAIL: test_insert_find_one_with_slaves (test.test_master_slave_connection.TestMasterSlaveConnection)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/test/test_master_slave_connection.py", line 344, in test_insert_find_one_with_slaves
self.assertTrue(count)
AssertionError: 0 is not true
----------------------------------------------------------------------
@Bernie, this might be relevant upstream. Want me to file a bug?
(In reply to comment #10) > I unfortunately get 4 (two distinct) new test errors. These only occur > when passing DB_PORT=27018. I initially thought it was related to differences > in mongodb startup flags, but after changing port in the mongodb init > script, I still got the same result Should probably note that I've verified this by calling nosetests myself with DB_PORT argument. Without reading any source (pymongo nor mongodb) the assumption that non-standard ports perhaps executes multi-server tests might be a reason? > > <snip> > > @Bernie, this might be relevant upstream. Want me to file a bug? (In reply to comment #10) > Created attachment 349112 [details] > pymongo-2.5.1-r1.ebuild > > The segfault seemed to originate from nosetests using wrong python version. > Here's the full error: > > ====================================================================== > ERROR: Failure: AttributeError (PyCapsule_Import "datetime.datetime_CAPI" is > not valid) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib64/python2.6/site-packages/nose/loader.py", line 413, in > loadTestsFromName > addr.filename, addr.module) > File "/usr/lib64/python2.6/site-packages/nose/importer.py", line 47, in > importFromPath > return self.importFromDir(dir_path, fqname) > File "/usr/lib64/python2.6/site-packages/nose/importer.py", line 94, in > importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File > "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/test/ > __init__.py", line 20, in <module> > import pymongo > File > "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/ > pymongo/__init__.py", line 80, in <module> > from pymongo.connection import Connection > File > "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/ > pymongo/connection.py", line 39, in <module> > from pymongo.mongo_client import MongoClient > File > "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/ > pymongo/mongo_client.py", line 44, in <module> > from bson.py3compat import b > File > "/var/tmp/portage/dev-python/pymongo-2.5.1/work/pymongo-2.5.1-python2_7/lib/ > bson/__init__.py", line 41, in <module> > from bson import _cbson > AttributeError: PyCapsule_Import "datetime.datetime_CAPI" is not valid > > ---------------------------------------------------------------------- > Upon closer inspection (and some help from mgorny), I've come to the conclusion that above is incorrect. The reason for the segfault is that python couldn't resolve all symbols in the c libraries that were newly built. This is remedied in the above ebuild since we now copy tests and cd into $BUILDDIR. Created attachment 349134 [details] pymongo-2.5.1-r1.ebuild (In reply to comment #11) > (In reply to comment #10) > > I unfortunately get 4 (two distinct) new test errors. These only occur > > when passing DB_PORT=27018. I initially thought it was related to differences > > in mongodb startup flags, but after changing port in the mongodb init > > script, I still got the same result > > Should probably note that I've verified this by calling nosetests myself > with DB_PORT argument. > > Without reading any source (pymongo nor mongodb) the assumption > that non-standard ports perhaps executes multi-server tests might be a > reason? Just had a look. __init__.py assumes second and third host gets 27018 and 27019. This essentially makes 27018 passed twice. @bernie: A suggested fix would perhaps be rewriting the config so it doesn't assume that {host,port}[23] needs to be set? As for now, we'll just pass "bogus" (DB_PORT+1, +2) values to the test runner. Will run test suite on all pythons I have next. Feel free to chip in and try yourself. > > > > > <snip> > > > > @Bernie, this might be relevant upstream. Want me to file a bug? Johan, I just made comments very similar to yours here: https://jira.mongodb.org/browse/PYTHON-521?focusedCommentId=345341&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-345341 Great minds think alike. :-) Sounds like you've sorted most of the final issues. There are a couple final comments I added here that should probably be addressed: https://jira.mongodb.org/browse/PYTHON-521?focusedCommentId=345348&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-345348 In the future we should probably keep the discussion in one bug tracker or the other. Just add me to the CC list on tickets here if you need my help. (In reply to Bernie Hackett from comment #14) > Johan, > > I just made comments very similar to yours here: > > https://jira.mongodb.org/browse/PYTHON-521?focusedCommentId=345341&page=com. > atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-345341 > > Great minds think alike. :-) > > Sounds like you've sorted most of the final issues. There are a couple final > comments I added here that should probably be addressed: > > https://jira.mongodb.org/browse/PYTHON-521?focusedCommentId=345348&page=com. > atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-345348 > > In the future we should probably keep the discussion in one bug tracker or > the other. Just add me to the CC list on tickets here if you need my help. I agree. Since this ticket was a bit abused - mainly by me - the line was a bit unclear. In general, I think its better to stay within Gentoo's grounds until we've confirmed that it's not our own bad. Portage is great, but adding logic to ebuilds sometimes introduces flaws from our end. Next time, we'll try to be less trigger happy upstream and make sure we only submit things that can be confirmed within and outside of portage. With that said, we much appreciate your participation as a gentoo user and upstream :-) (In reply to Johan Bergström from comment #15) > With that said, we much appreciate your participation as a gentoo user and > upstream :-) My pleasure. :-) Created attachment 349182 [details]
pymongo-2.5.1-r1.ebuild
Attached is my "final" ebuild. It passes tests on python 2.5,2.6,2.7 and 3.2. If someone could try 3.3 and all flavours of pypy, that would be much appreciated.
This version slightly refines paths in python_test() and removes pkg_postinst. It refers to important information which I cannot find in the 2.5.1 changelog. We should make it conditional if the original author intended it for transitioning between 2.4.x and 2.5.x.
Am I correct in saying we're having 2 parallel conversations here? From yesterday; (In reply to Johan Bergström from comment #17) > Created attachment 349182 [details] > pymongo-2.5.1-r1.ebuild > > Attached is my "final" ebuild. It passes tests on python 2.5,2.6,2.7 and > 3.2. If someone could try 3.3 and all flavours of pypy, that would be much > appreciated. > > This version slightly refines paths in python_test() and removes > pkg_postinst. It refers to important information which I cannot find in the > 2.5.1 changelog. What would that be? > We should make it conditional if the original author > intended it for transitioning between 2.4.x and 2.5.x. Frankly I prefer my committed form. Calling nosetests we know will trip over in py2. esetup.py test just works on testing, calling nosetests for py2 without tripping over. The changes I've made achieve the same as your intended changes but do so in a diff. way. IN_SOURCE build brings the tests into view without having to copy them, and so forth. Having booted into the alternate gentoo64 I finally have a 'working system' on running the testsuite. (In reply to Ian Delaney from comment #18) > Am I correct in saying we're having 2 parallel conversations here? > From yesterday; > > (In reply to Johan Bergström from comment #17) > > Created attachment 349182 [details] > > pymongo-2.5.1-r1.ebuild > > > > Attached is my "final" ebuild. It passes tests on python 2.5,2.6,2.7 and > > 3.2. If someone could try 3.3 and all flavours of pypy, that would be much > > appreciated. > > > > This version slightly refines paths in python_test() and removes > > pkg_postinst. It refers to important information which I cannot find in the > > 2.5.1 changelog. > > What would that be? Always prefixing with ${BUILD_DIR} (if you diff previous attached revisions you can see my trail of changes) > > > We should make it conditional if the original author > > intended it for transitioning between 2.4.x and 2.5.x. > > Frankly I prefer my committed form. Calling nosetests we know will trip over > in py2. esetup.py test just works on testing, calling nosetests for py2 > without tripping over. The changes I've made achieve the same as your > intended changes but do so in a diff. way. IN_SOURCE build brings the tests > into view without having to copy them, and so forth. Having booted into the > alternate gentoo64 I finally have a 'working system' on running the > testsuite. First of all, I would appreciate if you discussed what you wanted to change in this bug instead of simply implementing it and saying "I prefer my opinion". For me (not having a commit flag), contributing to Gentoo is about teamwork. Having my name in metadata.xml as proxy maintainer should at least give some hint of participation? I appreciate you taking some of the changes I made into regard but would still be more comfortable passing DB_PORT{,2,3} to nosetests so any other servers running won't be included. 27 May 2013; Ian Delaney <idella4@gentoo.org> pymongo-2.5.1.ebuild, pymongo-2.5.ebuild: Adjustment to test pahse from maintainer jbergstroem, fixes Bug #463310 by Patrick |