Go to:
Gentoo Home
Documentation
Forums
Lists
Bugs
Planet
Store
Wiki
Get Gentoo!
Gentoo's Bugzilla – Attachment 533248 Details for
Bug 656524
dev-python/s3fs-0.1.2: multiple test failures
Home
|
New
–
[Ex]
|
Browse
|
Search
|
Privacy Policy
|
[?]
|
Reports
|
Requests
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
build.log
build.log (text/plain), 300.18 KB, created by
Thomas Deutschmann (RETIRED)
on 2018-05-25 20:54:06 UTC
(
hide
)
Description:
build.log
Filename:
MIME Type:
Creator:
Thomas Deutschmann (RETIRED)
Created:
2018-05-25 20:54:06 UTC
Size:
300.18 KB
patch
obsolete
> * Package: dev-python/s3fs-0.1.2 > * Repository: gentoo > * Maintainer: python@gentoo.org > * USE: abi_x86_32 elibc_glibc kernel_linux python_targets_python2_7 python_targets_python3_5 test userland_GNU x86 > * FEATURES: preserve-libs sandbox test userpriv usersandbox >>>> Unpacking source... >>>> Unpacking s3fs-0.1.2.tar.gz to /var/tmp/portage/dev-python/s3fs-0.1.2/work >>>> Source unpacked in /var/tmp/portage/dev-python/s3fs-0.1.2/work >>>> Preparing source in /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2 ... >>>> Source prepared. >>>> Configuring source in /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2 ... >>>> Source configured. >>>> Compiling source in /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2 ... > * python2_7: running distutils-r1_run_phase distutils-r1_python_compile >python2.7 setup.py build >running build >running build_py >creating /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python2_7/lib/s3fs >copying s3fs/utils.py -> /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python2_7/lib/s3fs >copying s3fs/core.py -> /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python2_7/lib/s3fs >copying s3fs/__init__.py -> /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python2_7/lib/s3fs >copying s3fs/mapping.py -> /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python2_7/lib/s3fs >warning: build_py: byte-compiling is disabled, skipping. > > * python3_5: running distutils-r1_run_phase distutils-r1_python_compile >python3.5 setup.py build >running build >running build_py >creating /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python3_5/lib/s3fs >copying s3fs/utils.py -> /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python3_5/lib/s3fs >copying s3fs/core.py -> /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python3_5/lib/s3fs >copying s3fs/__init__.py -> /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python3_5/lib/s3fs >copying s3fs/mapping.py -> /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2-python3_5/lib/s3fs >warning: build_py: byte-compiling is disabled, skipping. > >>>> Source compiled. >>>> Test phase: dev-python/s3fs-0.1.2 > * python2_7: running distutils-r1_run_phase python_test >================================================= test session starts ================================================== >platform linux2 -- Python 2.7.14, pytest-3.2.2, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python2.7 >cachedir: .cache >rootdir: /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2, inifile: >plugins: hypothesis-3.6.0 >collecting ... collected 79 items > >s3fs/tests/test_mapping.py::test_simple PASSED >s3fs/tests/test_mapping.py::test_default_s3filesystem PASSED >s3fs/tests/test_mapping.py::test_errors PASSED >s3fs/tests/test_mapping.py::test_with_data PASSED >s3fs/tests/test_mapping.py::test_complex_keys PASSED >s3fs/tests/test_mapping.py::test_clear_empty PASSED >s3fs/tests/test_mapping.py::test_pickle PASSED >s3fs/tests/test_mapping.py::test_array PASSED >s3fs/tests/test_mapping.py::test_bytearray PASSED >s3fs/tests/test_mapping.py::test_new_bucket PASSED >s3fs/tests/test_s3fs.py::test_simple PASSED >s3fs/tests/test_s3fs.py::test_ssl_off PASSED >s3fs/tests/test_s3fs.py::test_client_kwargs PASSED >s3fs/tests/test_s3fs.py::test_config_kwargs PASSED >s3fs/tests/test_s3fs.py::test_tokenize PASSED >s3fs/tests/test_s3fs.py::test_idempotent_connect PASSED >s3fs/tests/test_s3fs.py::test_multiple_objects PASSED >s3fs/tests/test_s3fs.py::test_info PASSED >s3fs/tests/test_s3fs.py::test_xattr PASSED >s3fs/tests/test_s3fs.py::test_xattr_setxattr_in_write_mode PASSED >s3fs/tests/test_s3fs.py::test_delegate xfail >s3fs/tests/test_s3fs.py::test_not_delegate PASSED >s3fs/tests/test_s3fs.py::test_ls PASSED >s3fs/tests/test_s3fs.py::test_pickle PASSED >s3fs/tests/test_s3fs.py::test_ls_touch PASSED >s3fs/tests/test_s3fs.py::test_rm PASSED >s3fs/tests/test_s3fs.py::test_bulk_delete PASSED >s3fs/tests/test_s3fs.py::test_anonymous_access PASSED >s3fs/tests/test_s3fs.py::test_s3_file_access PASSED >s3fs/tests/test_s3fs.py::test_s3_file_info PASSED >s3fs/tests/test_s3fs.py::test_bucket_exists PASSED >s3fs/tests/test_s3fs.py::test_du PASSED >s3fs/tests/test_s3fs.py::test_s3_ls PASSED >s3fs/tests/test_s3fs.py::test_s3_big_ls PASSED >s3fs/tests/test_s3fs.py::test_s3_ls_detail PASSED >s3fs/tests/test_s3fs.py::test_s3_glob PASSED >s3fs/tests/test_s3fs.py::test_get_list_of_summary_objects PASSED >s3fs/tests/test_s3fs.py::test_read_keys_from_bucket PASSED >s3fs/tests/test_s3fs.py::test_url PASSED >s3fs/tests/test_s3fs.py::test_seek PASSED >s3fs/tests/test_s3fs.py::test_bad_open PASSED >s3fs/tests/test_s3fs.py::test_copy PASSED >s3fs/tests/test_s3fs.py::test_move PASSED >s3fs/tests/test_s3fs.py::test_get_put PASSED >s3fs/tests/test_s3fs.py::test_errors PASSED >s3fs/tests/test_s3fs.py::test_read_small PASSED >s3fs/tests/test_s3fs.py::test_seek_delimiter PASSED >s3fs/tests/test_s3fs.py::test_read_s3_block PASSED >s3fs/tests/test_s3fs.py::test_new_bucket PASSED >s3fs/tests/test_s3fs.py::test_dynamic_add_rm PASSED >s3fs/tests/test_s3fs.py::test_write_small PASSED >s3fs/tests/test_s3fs.py::test_write_small_secure PASSED >s3fs/tests/test_s3fs.py::test_write_large_secure PASSED >s3fs/tests/test_s3fs.py::test_write_fails PASSED >s3fs/tests/test_s3fs.py::test_write_blocks PASSED >s3fs/tests/test_s3fs.py::test_readline PASSED >s3fs/tests/test_s3fs.py::test_readline_from_cache PASSED >s3fs/tests/test_s3fs.py::test_readline_partial PASSED >s3fs/tests/test_s3fs.py::test_readline_empty PASSED >s3fs/tests/test_s3fs.py::test_readline_blocksize PASSED >s3fs/tests/test_s3fs.py::test_next PASSED >s3fs/tests/test_s3fs.py::test_iterable PASSED >s3fs/tests/test_s3fs.py::test_readable PASSED >s3fs/tests/test_s3fs.py::test_seekable PASSED >s3fs/tests/test_s3fs.py::test_writable PASSED >s3fs/tests/test_s3fs.py::test_merge PASSED >s3fs/tests/test_s3fs.py::test_append PASSED >s3fs/tests/test_s3fs.py::test_bigger_than_block_read PASSED >s3fs/tests/test_s3fs.py::test_current PASSED >s3fs/tests/test_s3fs.py::test_array PASSED >s3fs/tests/test_s3fs.py::test_no_connection_sharing_among_processes PASSED >s3fs/tests/test_s3fs.py::test_public_file xfail >s3fs/tests/test_s3fs.py::test_upload_with_s3fs_prefix PASSED >s3fs/tests/test_s3fs.py::test_multipart_upload_blocksize PASSED >s3fs/tests/test_s3fs.py::test_not_fill_cache PASSED >s3fs/tests/test_s3fs.py::test_default_pars PASSED >s3fs/tests/test_utils.py::test_tempfile PASSED >s3fs/tests/test_utils.py::test_read_block PASSED >s3fs/tests/test_utils.py::test_seek_delimiter_endline PASSED > >======================================== 77 passed, 2 xfailed in 535.04 seconds ======================================== > * python3_5: running distutils-r1_run_phase python_test >================================================= test session starts ================================================== >platform linux -- Python 3.5.5, pytest-3.2.2, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python3.5 >cachedir: .cache >rootdir: /var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2, inifile: >plugins: hypothesis-3.6.0 >collecting ... collected 79 items > >s3fs/tests/test_mapping.py::test_simple PASSED >s3fs/tests/test_mapping.py::test_default_s3filesystem PASSED >s3fs/tests/test_mapping.py::test_errors PASSED >s3fs/tests/test_mapping.py::test_with_data FAILED >s3fs/tests/test_mapping.py::test_complex_keys FAILED >s3fs/tests/test_mapping.py::test_clear_empty PASSED >s3fs/tests/test_mapping.py::test_pickle FAILED >s3fs/tests/test_mapping.py::test_array FAILED >s3fs/tests/test_mapping.py::test_bytearray FAILED >s3fs/tests/test_mapping.py::test_new_bucket PASSED >s3fs/tests/test_s3fs.py::test_simple FAILED >s3fs/tests/test_s3fs.py::test_ssl_off PASSED >s3fs/tests/test_s3fs.py::test_client_kwargs PASSED >s3fs/tests/test_s3fs.py::test_config_kwargs PASSED >s3fs/tests/test_s3fs.py::test_tokenize PASSED >s3fs/tests/test_s3fs.py::test_idempotent_connect PASSED >s3fs/tests/test_s3fs.py::test_multiple_objects PASSED >s3fs/tests/test_s3fs.py::test_info FAILED >s3fs/tests/test_s3fs.py::test_xattr FAILED >s3fs/tests/test_s3fs.py::test_xattr_setxattr_in_write_mode PASSED >s3fs/tests/test_s3fs.py::test_delegate xfail >s3fs/tests/test_s3fs.py::test_not_delegate PASSED >s3fs/tests/test_s3fs.py::test_ls PASSED >s3fs/tests/test_s3fs.py::test_pickle PASSED >s3fs/tests/test_s3fs.py::test_ls_touch PASSED >s3fs/tests/test_s3fs.py::test_rm PASSED >s3fs/tests/test_s3fs.py::test_bulk_delete PASSED >s3fs/tests/test_s3fs.py::test_anonymous_access PASSED >s3fs/tests/test_s3fs.py::test_s3_file_access FAILED >s3fs/tests/test_s3fs.py::test_s3_file_info PASSED >s3fs/tests/test_s3fs.py::test_bucket_exists PASSED >s3fs/tests/test_s3fs.py::test_du PASSED >s3fs/tests/test_s3fs.py::test_s3_ls PASSED >s3fs/tests/test_s3fs.py::test_s3_big_ls FAILED >s3fs/tests/test_s3fs.py::test_s3_ls_detail PASSED >s3fs/tests/test_s3fs.py::test_s3_glob PASSED >s3fs/tests/test_s3fs.py::test_get_list_of_summary_objects PASSED >s3fs/tests/test_s3fs.py::test_read_keys_from_bucket FAILED >s3fs/tests/test_s3fs.py::test_url PASSED >s3fs/tests/test_s3fs.py::test_seek FAILED >s3fs/tests/test_s3fs.py::test_bad_open PASSED >s3fs/tests/test_s3fs.py::test_copy FAILED >s3fs/tests/test_s3fs.py::test_move FAILED >s3fs/tests/test_s3fs.py::test_get_put FAILED >s3fs/tests/test_s3fs.py::test_errors PASSED >s3fs/tests/test_s3fs.py::test_read_small FAILED >s3fs/tests/test_s3fs.py::test_seek_delimiter FAILED >s3fs/tests/test_s3fs.py::test_read_s3_block FAILED >s3fs/tests/test_s3fs.py::test_new_bucket PASSED >s3fs/tests/test_s3fs.py::test_dynamic_add_rm PASSED >s3fs/tests/test_s3fs.py::test_write_small FAILED >s3fs/tests/test_s3fs.py::test_write_small_secure FAILED >s3fs/tests/test_s3fs.py::test_write_large_secure FAILED >s3fs/tests/test_s3fs.py::test_write_fails PASSED >s3fs/tests/test_s3fs.py::test_write_blocks PASSED >s3fs/tests/test_s3fs.py::test_readline FAILED >s3fs/tests/test_s3fs.py::test_readline_from_cache FAILED >s3fs/tests/test_s3fs.py::test_readline_partial FAILED >s3fs/tests/test_s3fs.py::test_readline_empty PASSED >s3fs/tests/test_s3fs.py::test_readline_blocksize FAILED >s3fs/tests/test_s3fs.py::test_next FAILED >s3fs/tests/test_s3fs.py::test_iterable FAILED >s3fs/tests/test_s3fs.py::test_readable PASSED >s3fs/tests/test_s3fs.py::test_seekable PASSED >s3fs/tests/test_s3fs.py::test_writable PASSED >s3fs/tests/test_s3fs.py::test_merge PASSED >s3fs/tests/test_s3fs.py::test_append FAILED >s3fs/tests/test_s3fs.py::test_bigger_than_block_read FAILED >s3fs/tests/test_s3fs.py::test_current PASSED >s3fs/tests/test_s3fs.py::test_array FAILED >s3fs/tests/test_s3fs.py::test_no_connection_sharing_among_processes PASSED >s3fs/tests/test_s3fs.py::test_public_file xfail >s3fs/tests/test_s3fs.py::test_upload_with_s3fs_prefix PASSED >s3fs/tests/test_s3fs.py::test_multipart_upload_blocksize PASSED >s3fs/tests/test_s3fs.py::test_not_fill_cache FAILED >s3fs/tests/test_s3fs.py::test_default_pars PASSED >s3fs/tests/test_utils.py::test_tempfile PASSED >s3fs/tests/test_utils.py::test_read_block PASSED >s3fs/tests/test_utils.py::test_seek_delimiter_endline PASSED > >======================================================= FAILURES ======================================================= >____________________________________________________ test_with_data ____________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb411d74c> > > def test_with_data(s3): > d = S3Map(root, s3) > d['x'] = b'123' > assert list(d) == list(d.keys()) == ['x'] >> assert list(d.values()) == [b'123'] > >s3fs/tests/test_mapping.py:38: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >/usr/lib/python3.5/_collections_abc.py:696: in __iter__ > yield self._mapping[key] >s3fs/mapping.py:68: in __getitem__ > result = f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb417c80c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >__________________________________________________ test_complex_keys ___________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb547122c> > > def test_complex_keys(s3): > d = S3Map(root, s3) > d[1] = b'hello' >> assert d[1] == b'hello' > >s3fs/tests/test_mapping.py:58: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/mapping.py:68: in __getitem__ > result = f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb4a12acc>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_____________________________________________________ test_pickle ______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb41836ac> > > def test_pickle(s3): > d = S3Map(root, s3) > d['x'] = b'1' > > import pickle > d2 = pickle.loads(pickle.dumps(d)) > >> assert d2['x'] == b'1' > >s3fs/tests/test_mapping.py:89: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/mapping.py:68: in __getitem__ > result = f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3faddcc>, shape = <Shape(ContentLength)>, text = "b'\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >______________________________________________________ test_array ______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3f88fcc> > > def test_array(s3): > from array import array > d = S3Map(root, s3) > d['x'] = array('B', [65] * 1000) > >> assert d['x'] == b'A' * 1000 > >s3fs/tests/test_mapping.py:97: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/mapping.py:68: in __getitem__ > result = f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3f5280c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >____________________________________________________ test_bytearray ____________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3f9a58c> > > def test_bytearray(s3): > from array import array > d = S3Map(root, s3) > d['x'] = bytearray(b'123') > >> assert d['x'] == b'123' > >s3fs/tests/test_mapping.py:105: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/mapping.py:68: in __getitem__ > result = f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3f9b84c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_____________________________________________________ test_simple ______________________________________________________ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb41bd9ec> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb4a1d84c> >method = 'GET', url = '/test/tmp/test/a' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1d14c> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-15728639', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo ...t-sha256;x-amz-date;x-amz-security-token, Signature=dd01d0c1f48bcca47c2566723702a7657c7be9c33c40115b563ee48b4d75b60c'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1dacc>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses >> httplib_response = conn.getresponse(buffering=True) >E TypeError: getresponse() got an unexpected keyword argument 'buffering' > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:372: TypeError > >During handling of the above exception, another exception occurred: > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb41bd9ec> >method = 'GET', url = '/test/tmp/test/a', body = None >headers = {'Range': b'bytes=0-15728639', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-C...nt-sha256;x-amz-date;x-amz-security-token, Signature=dd01d0c1f48bcca47c2566723702a7657c7be9c33c40115b563ee48b4d75b60c'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3f665ec>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1d14c> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, >> body=body, headers=headers) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:544: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb41bd9ec> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb4a1d84c> >method = 'GET', url = '/test/tmp/test/a' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1d14c> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-15728639', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo ...t-sha256;x-amz-date;x-amz-security-token, Signature=dd01d0c1f48bcca47c2566723702a7657c7be9c33c40115b563ee48b4d75b60c'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1dacc>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses > httplib_response = conn.getresponse(buffering=True) > except TypeError: # Python 2.6 and older >> httplib_response = conn.getresponse() > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:374: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb4a1d84c> > > def getresponse(self): > """Get the response from the server. > > If the HTTPConnection is in the correct state, returns an > instance of HTTPResponse or of whatever object is returned by > the response_class variable. > > If a request has not been sent or if a previous response has > not be handled, ResponseNotReady is raised. If the HTTP > response indicates that the connection should be closed, then > it will be closed before the response is returned. When the > connection is closed, the underlying socket is closed. > """ > > # if a prior response has been completed, then forget about it. > if self.__response and self.__response.isclosed(): > self.__response = None > > # if a prior response exists, then it must be completed (otherwise, we > # cannot read this response's header to determine the connection-close > # behavior) > # > # note: if a prior response existed, but was connection-close, then the > # socket and response were made independent of this HTTPConnection > # object since a new request requires that we open a whole new > # connection > # > # this means the prior response had one of two states: > # 1) will_close: this connection was reset and the prior socket and > # response operate independently > # 2) persistent: the response was retained and we await its > # isclosed() status to become true. > # > if self.__state != _CS_REQ_SENT or self.__response: > raise ResponseNotReady(self.__state) > > if self.debuglevel > 0: > response = self.response_class(self.sock, self.debuglevel, > method=self._method) > else: > response = self.response_class(self.sock, method=self._method) > > try: > try: >> response.begin() > >/usr/lib/python3.5/http/client.py:1198: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <http.client.HTTPResponse object at 0xb4a1d2cc> > > def begin(self): > if self.headers is not None: > # we've already started reading the response > return > > # read until we get a non-100 response > while True: > version, status, reason = self._read_status() > if status != CONTINUE: > break > # skip the header from the 100 response > while True: > skip = self.fp.readline(_MAXLINE + 1) > if len(skip) > _MAXLINE: > raise LineTooLong("header line") > skip = skip.strip() > if not skip: > break > if self.debuglevel > 0: > print("header:", skip) > > self.code = self.status = status > self.reason = reason.strip() > if version in ("HTTP/1.0", "HTTP/0.9"): > # Some servers might still return "0.9", treat it as 1.0 anyway > self.version = 10 > elif version.startswith("HTTP/1."): > self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 > else: > raise UnknownProtocol(version) > >> self.headers = self.msg = parse_headers(self.fp) > >/usr/lib/python3.5/http/client.py:321: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >fp = <httpretty.core.FakeSockFile object at 0xb40b05fc>, _class = <class 'http.client.HTTPMessage'> > > def parse_headers(fp, _class=HTTPMessage): > """Parses only RFC2822 headers from a file pointer. > > email Parser wants to see strings rather than bytes. > But a TextIOWrapper around self.rfile would buffer too many bytes > from the stream, bytes which we later need to read as bytes. > So we read the correct bytes here, as bytes, for email Parser > to parse. > > """ > headers = [] > while True: > line = fp.readline(_MAXLINE + 1) > if len(line) > _MAXLINE: >> raise LineTooLong("header line") >E http.client.LineTooLong: got more than 65536 bytes when reading header line > >/usr/lib/python3.5/http/client.py:208: LineTooLong > >During handling of the above exception, another exception occurred: > >self = <botocore.vendored.requests.adapters.HTTPAdapter object at 0xb41b8d4c>, request = <PreparedRequest [GET]> >stream = True, timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3f665ec> >verify = True, cert = None, proxies = {} > > def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): > """Sends PreparedRequest object. Returns Response object. > > :param request: The :class:`PreparedRequest <PreparedRequest>` being sent. > :param stream: (optional) Whether to stream the request content. > :param timeout: (optional) How long to wait for the server to send > data before giving up, as a float, or a (`connect timeout, read > timeout <user/advanced.html#timeouts>`_) tuple. > :type timeout: float or tuple > :param verify: (optional) Whether to verify SSL certificates. > :param cert: (optional) Any user-provided SSL certificate to be trusted. > :param proxies: (optional) The proxies dictionary to apply to the request. > """ > > conn = self.get_connection(request.url, proxies) > > self.cert_verify(conn, request.url, verify, cert) > url = self.request_url(request, proxies) > self.add_headers(request) > > chunked = not (request.body is None or 'Content-Length' in request.headers) > > if isinstance(timeout, tuple): > try: > connect, read = timeout > timeout = TimeoutSauce(connect=connect, read=read) > except ValueError as e: > # this may raise a string formatting error. > err = ("Invalid timeout {0}. Pass a (connect, read) " > "timeout tuple, or a single float to set " > "both timeouts to the same value".format(timeout)) > raise ValueError(err) > else: > timeout = TimeoutSauce(connect=timeout, read=timeout) > > try: > if not chunked: > resp = conn.urlopen( > method=request.method, > url=url, > body=request.body, > headers=request.headers, > redirect=False, > assert_same_host=False, > preload_content=False, > decode_content=False, > retries=self.max_retries, >> timeout=timeout > ) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py:370: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb41bd9ec> >method = 'GET', url = '/test/tmp/test/a', body = None >headers = {'Range': b'bytes=0-15728639', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-C...nt-sha256;x-amz-date;x-amz-security-token, Signature=dd01d0c1f48bcca47c2566723702a7657c7be9c33c40115b563ee48b4d75b60c'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3f665ec>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1d14c> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, > body=body, headers=headers) > > # If we're going to release the connection in ``finally:``, then > # the request doesn't need to know about the connection. Otherwise > # it will also try to release it and we'll have a double-release > # mess. > response_conn = not release_conn and conn > > # Import httplib's response into our own wrapper object > response = HTTPResponse.from_httplib(httplib_response, > pool=self, > connection=response_conn, > **response_kw) > > # else: > # The connection will be put back into the pool when > # ``response.release_conn()`` is called (implicitly by > # ``response.read()``) > > except Empty: > # Timed out by queue. > raise EmptyPoolError(self, "No pool connections are available.") > > except (BaseSSLError, CertificateError) as e: > # Close the connection. If a connection is reused on which there > # was a Certificate error, the next request will certainly raise > # another Certificate error. > if conn: > conn.close() > conn = None > raise SSLError(e) > > except SSLError: > # Treat SSLError separately from BaseSSLError to preserve > # traceback. > if conn: > conn.close() > conn = None > raise > > except (TimeoutError, HTTPException, SocketError, ConnectionError) as e: > if conn: > # Discard the connection for these exceptions. It will be > # be replaced during the next _get_conn() call. > conn.close() > conn = None > > if isinstance(e, SocketError) and self.proxy: > e = ProxyError('Cannot connect to proxy.', e) > elif isinstance(e, (SocketError, HTTPException)): > e = ProtocolError('Connection aborted.', e) > > retries = retries.increment(method, url, error=e, _pool=self, >> _stacktrace=sys.exc_info()[2]) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:597: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = Retry(total=0, connect=None, read=False, redirect=None), method = 'GET', url = '/test/tmp/test/a' >response = None >error = ProtocolError('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) >_pool = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb41bd9ec> >_stacktrace = <traceback object at 0xb409b75c> > > def increment(self, method=None, url=None, response=None, error=None, _pool=None, _stacktrace=None): > """ Return a new Retry object with incremented retry counters. > > :param response: A response object, or None, if the server did not > return a response. > :type response: :class:`~urllib3.response.HTTPResponse` > :param Exception error: An error encountered during the request, or > None if the response was received successfully. > > :return: A new ``Retry`` object. > """ > if self.total is False and error: > # Disabled, indicate to re-raise the error. > raise six.reraise(type(error), error, _stacktrace) > > total = self.total > if total is not None: > total -= 1 > > _observed_errors = self._observed_errors > connect = self.connect > read = self.read > redirect = self.redirect > cause = 'unknown' > > if error and self._is_connection_error(error): > # Connect retry? > if connect is False: > raise six.reraise(type(error), error, _stacktrace) > elif connect is not None: > connect -= 1 > _observed_errors += 1 > > elif error and self._is_read_error(error): > # Read retry? > if read is False: >> raise six.reraise(type(error), error, _stacktrace) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/util/retry.py:245: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >tp = <class 'botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError'> >value = ProtocolError('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) >tb = <traceback object at 0xb409b75c> > > def reraise(tp, value, tb=None): > if value.__traceback__ is not tb: >> raise value.with_traceback(tb) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/packages/six.py:309: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb41bd9ec> >method = 'GET', url = '/test/tmp/test/a', body = None >headers = {'Range': b'bytes=0-15728639', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-C...nt-sha256;x-amz-date;x-amz-security-token, Signature=dd01d0c1f48bcca47c2566723702a7657c7be9c33c40115b563ee48b4d75b60c'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3f665ec>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1d14c> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, >> body=body, headers=headers) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:544: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb41bd9ec> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb4a1d84c> >method = 'GET', url = '/test/tmp/test/a' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1d14c> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-15728639', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo ...t-sha256;x-amz-date;x-amz-security-token, Signature=dd01d0c1f48bcca47c2566723702a7657c7be9c33c40115b563ee48b4d75b60c'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb4a1dacc>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses > httplib_response = conn.getresponse(buffering=True) > except TypeError: # Python 2.6 and older >> httplib_response = conn.getresponse() > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:374: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb4a1d84c> > > def getresponse(self): > """Get the response from the server. > > If the HTTPConnection is in the correct state, returns an > instance of HTTPResponse or of whatever object is returned by > the response_class variable. > > If a request has not been sent or if a previous response has > not be handled, ResponseNotReady is raised. If the HTTP > response indicates that the connection should be closed, then > it will be closed before the response is returned. When the > connection is closed, the underlying socket is closed. > """ > > # if a prior response has been completed, then forget about it. > if self.__response and self.__response.isclosed(): > self.__response = None > > # if a prior response exists, then it must be completed (otherwise, we > # cannot read this response's header to determine the connection-close > # behavior) > # > # note: if a prior response existed, but was connection-close, then the > # socket and response were made independent of this HTTPConnection > # object since a new request requires that we open a whole new > # connection > # > # this means the prior response had one of two states: > # 1) will_close: this connection was reset and the prior socket and > # response operate independently > # 2) persistent: the response was retained and we await its > # isclosed() status to become true. > # > if self.__state != _CS_REQ_SENT or self.__response: > raise ResponseNotReady(self.__state) > > if self.debuglevel > 0: > response = self.response_class(self.sock, self.debuglevel, > method=self._method) > else: > response = self.response_class(self.sock, method=self._method) > > try: > try: >> response.begin() > >/usr/lib/python3.5/http/client.py:1198: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <http.client.HTTPResponse object at 0xb4a1d2cc> > > def begin(self): > if self.headers is not None: > # we've already started reading the response > return > > # read until we get a non-100 response > while True: > version, status, reason = self._read_status() > if status != CONTINUE: > break > # skip the header from the 100 response > while True: > skip = self.fp.readline(_MAXLINE + 1) > if len(skip) > _MAXLINE: > raise LineTooLong("header line") > skip = skip.strip() > if not skip: > break > if self.debuglevel > 0: > print("header:", skip) > > self.code = self.status = status > self.reason = reason.strip() > if version in ("HTTP/1.0", "HTTP/0.9"): > # Some servers might still return "0.9", treat it as 1.0 anyway > self.version = 10 > elif version.startswith("HTTP/1."): > self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 > else: > raise UnknownProtocol(version) > >> self.headers = self.msg = parse_headers(self.fp) > >/usr/lib/python3.5/http/client.py:321: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >fp = <httpretty.core.FakeSockFile object at 0xb40b05fc>, _class = <class 'http.client.HTTPMessage'> > > def parse_headers(fp, _class=HTTPMessage): > """Parses only RFC2822 headers from a file pointer. > > email Parser wants to see strings rather than bytes. > But a TextIOWrapper around self.rfile would buffer too many bytes > from the stream, bytes which we later need to read as bytes. > So we read the correct bytes here, as bytes, for email Parser > to parse. > > """ > headers = [] > while True: > line = fp.readline(_MAXLINE + 1) > if len(line) > _MAXLINE: >> raise LineTooLong("header line") >E botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) > >/usr/lib/python3.5/http/client.py:208: ProtocolError > >During handling of the above exception, another exception occurred: > >s3 = <s3fs.core.S3FileSystem object at 0xb4a1b80c> > > def test_simple(s3): > data = b'a' * (10 * 2**20) > > with s3.open(a, 'wb') as f: > f.write(data) > > with s3.open(a, 'rb') as f: >> out = f.read(len(data)) > >s3fs/tests/test_s3fs.py:108: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:170: in _send_request > success_response, exception): >/usr/lib/python3.5/site-packages/botocore/endpoint.py:249: in _needs_retry > caught_exception=caught_exception, request_dict=request_dict) >/usr/lib/python3.5/site-packages/botocore/hooks.py:227: in emit > return self._emit(event_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/hooks.py:210: in _emit > response = handler(**kwargs) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:183: in __call__ > if self._checker(attempts, response, caught_exception): >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:251: in __call__ > caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:277: in _should_retry > return self._checker(attempt_number, response, caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:317: in __call__ > caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:223: in __call__ > attempt_number, caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:359: in _check_caught_exception > raise caught_exception >/usr/lib/python3.5/site-packages/botocore/endpoint.py:204: in _get_response > proxies=self.proxies, timeout=self.timeout) >/usr/lib/python3.5/site-packages/botocore/vendored/requests/sessions.py:573: in send > r = adapter.send(request, **kwargs) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.adapters.HTTPAdapter object at 0xb41b8d4c>, request = <PreparedRequest [GET]> >stream = True, timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3f665ec> >verify = True, cert = None, proxies = {} > > def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): > """Sends PreparedRequest object. Returns Response object. > > :param request: The :class:`PreparedRequest <PreparedRequest>` being sent. > :param stream: (optional) Whether to stream the request content. > :param timeout: (optional) How long to wait for the server to send > data before giving up, as a float, or a (`connect timeout, read > timeout <user/advanced.html#timeouts>`_) tuple. > :type timeout: float or tuple > :param verify: (optional) Whether to verify SSL certificates. > :param cert: (optional) Any user-provided SSL certificate to be trusted. > :param proxies: (optional) The proxies dictionary to apply to the request. > """ > > conn = self.get_connection(request.url, proxies) > > self.cert_verify(conn, request.url, verify, cert) > url = self.request_url(request, proxies) > self.add_headers(request) > > chunked = not (request.body is None or 'Content-Length' in request.headers) > > if isinstance(timeout, tuple): > try: > connect, read = timeout > timeout = TimeoutSauce(connect=connect, read=read) > except ValueError as e: > # this may raise a string formatting error. > err = ("Invalid timeout {0}. Pass a (connect, read) " > "timeout tuple, or a single float to set " > "both timeouts to the same value".format(timeout)) > raise ValueError(err) > else: > timeout = TimeoutSauce(connect=timeout, read=timeout) > > try: > if not chunked: > resp = conn.urlopen( > method=request.method, > url=url, > body=request.body, > headers=request.headers, > redirect=False, > assert_same_host=False, > preload_content=False, > decode_content=False, > retries=self.max_retries, > timeout=timeout > ) > > # Send the request. > else: > if hasattr(conn, 'proxy_pool'): > conn = conn.proxy_pool > > low_conn = conn._get_conn(timeout=timeout) > > try: > low_conn.putrequest(request.method, > url, > skip_accept_encoding=True) > > for header, value in request.headers.items(): > low_conn.putheader(header, value) > > low_conn.endheaders() > > for i in request.body: > low_conn.send(hex(len(i))[2:].encode('utf-8')) > low_conn.send(b'\r\n') > low_conn.send(i) > low_conn.send(b'\r\n') > low_conn.send(b'0\r\n\r\n') > > r = low_conn.getresponse() > resp = HTTPResponse.from_httplib( > r, > pool=conn, > connection=low_conn, > preload_content=False, > decode_content=False > ) > except: > # If we hit any problems here, clean up the connection. > # Then, reraise so that we can handle the actual exception. > low_conn.close() > raise > else: > # All is well, return the connection to the pool. > conn._put_conn(low_conn) > > except (ProtocolError, socket.error) as err: >> raise ConnectionError(err, request=request) >E botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py:415: ConnectionError >______________________________________________________ test_info _______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb347d42c> > > def test_info(s3): > s3.touch(a) > s3.touch(b) > assert s3.info(a) == s3.ls(a, detail=True)[0] > parent = a.rsplit('/', 1)[0] > s3.dirs[parent].pop(0) # disappear our file! > assert a not in s3.ls(parent) >> assert s3.info(a) # now uses head_object > >s3fs/tests/test_s3fs.py:158: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:396: in info > kwargs, Bucket=bucket, Key=key, **self.req_kw) >s3fs/core.py:170: in _call_s3 > return method(**additional_kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb348194c>, shape = <Shape(ContentLength)>, text = "b''" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b''" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >______________________________________________________ test_xattr ______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb206e36c> > > def test_xattr(s3): > bucket, key = (test_bucket_name, 'tmp/test/xattr') > filename = bucket + '/' + key > body = b'aaaa' > public_read_acl = {'Permission': 'READ', 'Grantee': {'URI': 'http://acs.amazonaws.com/groups/global/AllUsers', 'Type': 'Group'}} > > s3.s3.put_object(Bucket=bucket, Key=key, > ACL='public-read', > Metadata=test_xattr_sample_metadata, > Body=body) > > # save etag for later > etag = s3.info(filename)['ETag'] > assert public_read_acl in s3.s3.get_object_acl(Bucket=bucket, Key=key)['Grants'] > >> assert s3.getxattr(filename, 'test_xattr') == test_xattr_sample_metadata['test_xattr'] > >s3fs/tests/test_s3fs.py:177: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:438: in getxattr > xattr = self.metadata(path, **kwargs) >s3fs/core.py:425: in metadata > **self.req_kw) >s3fs/core.py:170: in _call_s3 > return method(**additional_kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb2061e6c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_________________________________________________ test_s3_file_access __________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb4a2496c> > > def test_s3_file_access(s3): > fn = test_bucket_name+'/nested/file1' > data = b'hello\n' >> assert s3.cat(fn) == data > >s3fs/tests/test_s3fs.py:283: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:583: in cat > return f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb40f7f8c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >____________________________________________________ test_s3_big_ls ____________________________________________________ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3d82b0c> >method = 'GET', url = '/test?list-type=2&prefix=thousand%2F&delimiter=%2F' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d826ec> >httplib_request_kw = {'body': None, 'headers': {'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Conte...t-sha256;x-amz-date;x-amz-security-token, Signature=ed70e664c777172916f81c409cede2e39e6d3e452a298b0b3208d093b2f8a0f7'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d8264c>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses >> httplib_response = conn.getresponse(buffering=True) >E TypeError: getresponse() got an unexpected keyword argument 'buffering' > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:372: TypeError > >During handling of the above exception, another exception occurred: > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/test?list-type=2&prefix=thousand%2F&delimiter=%2F', body = None >headers = {'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Content-SHA256': b'e3b0c44298fc...nt-sha256;x-amz-date;x-amz-security-token, Signature=ed70e664c777172916f81c409cede2e39e6d3e452a298b0b3208d093b2f8a0f7'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb416af8c>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d826ec> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, >> body=body, headers=headers) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:544: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3d82b0c> >method = 'GET', url = '/test?list-type=2&prefix=thousand%2F&delimiter=%2F' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d826ec> >httplib_request_kw = {'body': None, 'headers': {'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Conte...t-sha256;x-amz-date;x-amz-security-token, Signature=ed70e664c777172916f81c409cede2e39e6d3e452a298b0b3208d093b2f8a0f7'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d8264c>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses > httplib_response = conn.getresponse(buffering=True) > except TypeError: # Python 2.6 and older >> httplib_response = conn.getresponse() > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:374: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3d82b0c> > > def getresponse(self): > """Get the response from the server. > > If the HTTPConnection is in the correct state, returns an > instance of HTTPResponse or of whatever object is returned by > the response_class variable. > > If a request has not been sent or if a previous response has > not be handled, ResponseNotReady is raised. If the HTTP > response indicates that the connection should be closed, then > it will be closed before the response is returned. When the > connection is closed, the underlying socket is closed. > """ > > # if a prior response has been completed, then forget about it. > if self.__response and self.__response.isclosed(): > self.__response = None > > # if a prior response exists, then it must be completed (otherwise, we > # cannot read this response's header to determine the connection-close > # behavior) > # > # note: if a prior response existed, but was connection-close, then the > # socket and response were made independent of this HTTPConnection > # object since a new request requires that we open a whole new > # connection > # > # this means the prior response had one of two states: > # 1) will_close: this connection was reset and the prior socket and > # response operate independently > # 2) persistent: the response was retained and we await its > # isclosed() status to become true. > # > if self.__state != _CS_REQ_SENT or self.__response: > raise ResponseNotReady(self.__state) > > if self.debuglevel > 0: > response = self.response_class(self.sock, self.debuglevel, > method=self._method) > else: > response = self.response_class(self.sock, method=self._method) > > try: > try: >> response.begin() > >/usr/lib/python3.5/http/client.py:1198: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <http.client.HTTPResponse object at 0xb3d82fac> > > def begin(self): > if self.headers is not None: > # we've already started reading the response > return > > # read until we get a non-100 response > while True: > version, status, reason = self._read_status() > if status != CONTINUE: > break > # skip the header from the 100 response > while True: > skip = self.fp.readline(_MAXLINE + 1) > if len(skip) > _MAXLINE: > raise LineTooLong("header line") > skip = skip.strip() > if not skip: > break > if self.debuglevel > 0: > print("header:", skip) > > self.code = self.status = status > self.reason = reason.strip() > if version in ("HTTP/1.0", "HTTP/0.9"): > # Some servers might still return "0.9", treat it as 1.0 anyway > self.version = 10 > elif version.startswith("HTTP/1."): > self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 > else: > raise UnknownProtocol(version) > >> self.headers = self.msg = parse_headers(self.fp) > >/usr/lib/python3.5/http/client.py:321: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >fp = <httpretty.core.FakeSockFile object at 0xb3d92b0c>, _class = <class 'http.client.HTTPMessage'> > > def parse_headers(fp, _class=HTTPMessage): > """Parses only RFC2822 headers from a file pointer. > > email Parser wants to see strings rather than bytes. > But a TextIOWrapper around self.rfile would buffer too many bytes > from the stream, bytes which we later need to read as bytes. > So we read the correct bytes here, as bytes, for email Parser > to parse. > > """ > headers = [] > while True: > line = fp.readline(_MAXLINE + 1) > if len(line) > _MAXLINE: >> raise LineTooLong("header line") >E http.client.LineTooLong: got more than 65536 bytes when reading header line > >/usr/lib/python3.5/http/client.py:208: LineTooLong > >During handling of the above exception, another exception occurred: > >self = <botocore.vendored.requests.adapters.HTTPAdapter object at 0xb345cd2c>, request = <PreparedRequest [GET]> >stream = False, timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb416af8c> >verify = True, cert = None, proxies = {} > > def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): > """Sends PreparedRequest object. Returns Response object. > > :param request: The :class:`PreparedRequest <PreparedRequest>` being sent. > :param stream: (optional) Whether to stream the request content. > :param timeout: (optional) How long to wait for the server to send > data before giving up, as a float, or a (`connect timeout, read > timeout <user/advanced.html#timeouts>`_) tuple. > :type timeout: float or tuple > :param verify: (optional) Whether to verify SSL certificates. > :param cert: (optional) Any user-provided SSL certificate to be trusted. > :param proxies: (optional) The proxies dictionary to apply to the request. > """ > > conn = self.get_connection(request.url, proxies) > > self.cert_verify(conn, request.url, verify, cert) > url = self.request_url(request, proxies) > self.add_headers(request) > > chunked = not (request.body is None or 'Content-Length' in request.headers) > > if isinstance(timeout, tuple): > try: > connect, read = timeout > timeout = TimeoutSauce(connect=connect, read=read) > except ValueError as e: > # this may raise a string formatting error. > err = ("Invalid timeout {0}. Pass a (connect, read) " > "timeout tuple, or a single float to set " > "both timeouts to the same value".format(timeout)) > raise ValueError(err) > else: > timeout = TimeoutSauce(connect=timeout, read=timeout) > > try: > if not chunked: > resp = conn.urlopen( > method=request.method, > url=url, > body=request.body, > headers=request.headers, > redirect=False, > assert_same_host=False, > preload_content=False, > decode_content=False, > retries=self.max_retries, >> timeout=timeout > ) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py:370: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/test?list-type=2&prefix=thousand%2F&delimiter=%2F', body = None >headers = {'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Content-SHA256': b'e3b0c44298fc...nt-sha256;x-amz-date;x-amz-security-token, Signature=ed70e664c777172916f81c409cede2e39e6d3e452a298b0b3208d093b2f8a0f7'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb416af8c>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d826ec> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, > body=body, headers=headers) > > # If we're going to release the connection in ``finally:``, then > # the request doesn't need to know about the connection. Otherwise > # it will also try to release it and we'll have a double-release > # mess. > response_conn = not release_conn and conn > > # Import httplib's response into our own wrapper object > response = HTTPResponse.from_httplib(httplib_response, > pool=self, > connection=response_conn, > **response_kw) > > # else: > # The connection will be put back into the pool when > # ``response.release_conn()`` is called (implicitly by > # ``response.read()``) > > except Empty: > # Timed out by queue. > raise EmptyPoolError(self, "No pool connections are available.") > > except (BaseSSLError, CertificateError) as e: > # Close the connection. If a connection is reused on which there > # was a Certificate error, the next request will certainly raise > # another Certificate error. > if conn: > conn.close() > conn = None > raise SSLError(e) > > except SSLError: > # Treat SSLError separately from BaseSSLError to preserve > # traceback. > if conn: > conn.close() > conn = None > raise > > except (TimeoutError, HTTPException, SocketError, ConnectionError) as e: > if conn: > # Discard the connection for these exceptions. It will be > # be replaced during the next _get_conn() call. > conn.close() > conn = None > > if isinstance(e, SocketError) and self.proxy: > e = ProxyError('Cannot connect to proxy.', e) > elif isinstance(e, (SocketError, HTTPException)): > e = ProtocolError('Connection aborted.', e) > > retries = retries.increment(method, url, error=e, _pool=self, >> _stacktrace=sys.exc_info()[2]) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:597: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = Retry(total=0, connect=None, read=False, redirect=None), method = 'GET' >url = '/test?list-type=2&prefix=thousand%2F&delimiter=%2F', response = None >error = ProtocolError('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) >_pool = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >_stacktrace = <traceback object at 0xb3d91b94> > > def increment(self, method=None, url=None, response=None, error=None, _pool=None, _stacktrace=None): > """ Return a new Retry object with incremented retry counters. > > :param response: A response object, or None, if the server did not > return a response. > :type response: :class:`~urllib3.response.HTTPResponse` > :param Exception error: An error encountered during the request, or > None if the response was received successfully. > > :return: A new ``Retry`` object. > """ > if self.total is False and error: > # Disabled, indicate to re-raise the error. > raise six.reraise(type(error), error, _stacktrace) > > total = self.total > if total is not None: > total -= 1 > > _observed_errors = self._observed_errors > connect = self.connect > read = self.read > redirect = self.redirect > cause = 'unknown' > > if error and self._is_connection_error(error): > # Connect retry? > if connect is False: > raise six.reraise(type(error), error, _stacktrace) > elif connect is not None: > connect -= 1 > _observed_errors += 1 > > elif error and self._is_read_error(error): > # Read retry? > if read is False: >> raise six.reraise(type(error), error, _stacktrace) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/util/retry.py:245: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >tp = <class 'botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError'> >value = ProtocolError('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) >tb = <traceback object at 0xb3d91b94> > > def reraise(tp, value, tb=None): > if value.__traceback__ is not tb: >> raise value.with_traceback(tb) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/packages/six.py:309: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/test?list-type=2&prefix=thousand%2F&delimiter=%2F', body = None >headers = {'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Content-SHA256': b'e3b0c44298fc...nt-sha256;x-amz-date;x-amz-security-token, Signature=ed70e664c777172916f81c409cede2e39e6d3e452a298b0b3208d093b2f8a0f7'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb416af8c>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d826ec> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, >> body=body, headers=headers) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:544: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3d82b0c> >method = 'GET', url = '/test?list-type=2&prefix=thousand%2F&delimiter=%2F' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d826ec> >httplib_request_kw = {'body': None, 'headers': {'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Conte...t-sha256;x-amz-date;x-amz-security-token, Signature=ed70e664c777172916f81c409cede2e39e6d3e452a298b0b3208d093b2f8a0f7'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3d8264c>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses > httplib_response = conn.getresponse(buffering=True) > except TypeError: # Python 2.6 and older >> httplib_response = conn.getresponse() > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:374: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3d82b0c> > > def getresponse(self): > """Get the response from the server. > > If the HTTPConnection is in the correct state, returns an > instance of HTTPResponse or of whatever object is returned by > the response_class variable. > > If a request has not been sent or if a previous response has > not be handled, ResponseNotReady is raised. If the HTTP > response indicates that the connection should be closed, then > it will be closed before the response is returned. When the > connection is closed, the underlying socket is closed. > """ > > # if a prior response has been completed, then forget about it. > if self.__response and self.__response.isclosed(): > self.__response = None > > # if a prior response exists, then it must be completed (otherwise, we > # cannot read this response's header to determine the connection-close > # behavior) > # > # note: if a prior response existed, but was connection-close, then the > # socket and response were made independent of this HTTPConnection > # object since a new request requires that we open a whole new > # connection > # > # this means the prior response had one of two states: > # 1) will_close: this connection was reset and the prior socket and > # response operate independently > # 2) persistent: the response was retained and we await its > # isclosed() status to become true. > # > if self.__state != _CS_REQ_SENT or self.__response: > raise ResponseNotReady(self.__state) > > if self.debuglevel > 0: > response = self.response_class(self.sock, self.debuglevel, > method=self._method) > else: > response = self.response_class(self.sock, method=self._method) > > try: > try: >> response.begin() > >/usr/lib/python3.5/http/client.py:1198: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <http.client.HTTPResponse object at 0xb3d82fac> > > def begin(self): > if self.headers is not None: > # we've already started reading the response > return > > # read until we get a non-100 response > while True: > version, status, reason = self._read_status() > if status != CONTINUE: > break > # skip the header from the 100 response > while True: > skip = self.fp.readline(_MAXLINE + 1) > if len(skip) > _MAXLINE: > raise LineTooLong("header line") > skip = skip.strip() > if not skip: > break > if self.debuglevel > 0: > print("header:", skip) > > self.code = self.status = status > self.reason = reason.strip() > if version in ("HTTP/1.0", "HTTP/0.9"): > # Some servers might still return "0.9", treat it as 1.0 anyway > self.version = 10 > elif version.startswith("HTTP/1."): > self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 > else: > raise UnknownProtocol(version) > >> self.headers = self.msg = parse_headers(self.fp) > >/usr/lib/python3.5/http/client.py:321: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >fp = <httpretty.core.FakeSockFile object at 0xb3d92b0c>, _class = <class 'http.client.HTTPMessage'> > > def parse_headers(fp, _class=HTTPMessage): > """Parses only RFC2822 headers from a file pointer. > > email Parser wants to see strings rather than bytes. > But a TextIOWrapper around self.rfile would buffer too many bytes > from the stream, bytes which we later need to read as bytes. > So we read the correct bytes here, as bytes, for email Parser > to parse. > > """ > headers = [] > while True: > line = fp.readline(_MAXLINE + 1) > if len(line) > _MAXLINE: >> raise LineTooLong("header line") >E botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) > >/usr/lib/python3.5/http/client.py:208: ProtocolError > >During handling of the above exception, another exception occurred: > >s3 = <s3fs.core.S3FileSystem object at 0xb41b804c> > > def test_s3_big_ls(s3): > for x in range(1200): > s3.touch(test_bucket_name+'/thousand/%i.part'%x) >> assert len(s3.walk(test_bucket_name)) > 1200 > >s3fs/tests/test_s3fs.py:329: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:503: in walk > return [f['Key'] for f in self._walk(path, refresh)] >s3fs/core.py:497: in _walk > filenames.extend(self._walk(f['Key'], refresh)) >s3fs/core.py:494: in _walk > filenames = self._ls(path, refresh=refresh)[:] >s3fs/core.py:362: in _ls > return self._lsdir(path, refresh) >s3fs/core.py:308: in _lsdir > for i in it: >/usr/lib/python3.5/site-packages/botocore/paginate.py:255: in __iter__ > response = self._make_request(current_kwargs) >/usr/lib/python3.5/site-packages/botocore/paginate.py:332: in _make_request > return self._method(**current_kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:170: in _send_request > success_response, exception): >/usr/lib/python3.5/site-packages/botocore/endpoint.py:249: in _needs_retry > caught_exception=caught_exception, request_dict=request_dict) >/usr/lib/python3.5/site-packages/botocore/hooks.py:227: in emit > return self._emit(event_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/hooks.py:210: in _emit > response = handler(**kwargs) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:183: in __call__ > if self._checker(attempts, response, caught_exception): >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:251: in __call__ > caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:277: in _should_retry > return self._checker(attempt_number, response, caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:317: in __call__ > caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:223: in __call__ > attempt_number, caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:359: in _check_caught_exception > raise caught_exception >/usr/lib/python3.5/site-packages/botocore/endpoint.py:204: in _get_response > proxies=self.proxies, timeout=self.timeout) >/usr/lib/python3.5/site-packages/botocore/vendored/requests/sessions.py:573: in send > r = adapter.send(request, **kwargs) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.adapters.HTTPAdapter object at 0xb345cd2c>, request = <PreparedRequest [GET]> >stream = False, timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb416af8c> >verify = True, cert = None, proxies = {} > > def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): > """Sends PreparedRequest object. Returns Response object. > > :param request: The :class:`PreparedRequest <PreparedRequest>` being sent. > :param stream: (optional) Whether to stream the request content. > :param timeout: (optional) How long to wait for the server to send > data before giving up, as a float, or a (`connect timeout, read > timeout <user/advanced.html#timeouts>`_) tuple. > :type timeout: float or tuple > :param verify: (optional) Whether to verify SSL certificates. > :param cert: (optional) Any user-provided SSL certificate to be trusted. > :param proxies: (optional) The proxies dictionary to apply to the request. > """ > > conn = self.get_connection(request.url, proxies) > > self.cert_verify(conn, request.url, verify, cert) > url = self.request_url(request, proxies) > self.add_headers(request) > > chunked = not (request.body is None or 'Content-Length' in request.headers) > > if isinstance(timeout, tuple): > try: > connect, read = timeout > timeout = TimeoutSauce(connect=connect, read=read) > except ValueError as e: > # this may raise a string formatting error. > err = ("Invalid timeout {0}. Pass a (connect, read) " > "timeout tuple, or a single float to set " > "both timeouts to the same value".format(timeout)) > raise ValueError(err) > else: > timeout = TimeoutSauce(connect=timeout, read=timeout) > > try: > if not chunked: > resp = conn.urlopen( > method=request.method, > url=url, > body=request.body, > headers=request.headers, > redirect=False, > assert_same_host=False, > preload_content=False, > decode_content=False, > retries=self.max_retries, > timeout=timeout > ) > > # Send the request. > else: > if hasattr(conn, 'proxy_pool'): > conn = conn.proxy_pool > > low_conn = conn._get_conn(timeout=timeout) > > try: > low_conn.putrequest(request.method, > url, > skip_accept_encoding=True) > > for header, value in request.headers.items(): > low_conn.putheader(header, value) > > low_conn.endheaders() > > for i in request.body: > low_conn.send(hex(len(i))[2:].encode('utf-8')) > low_conn.send(b'\r\n') > low_conn.send(i) > low_conn.send(b'\r\n') > low_conn.send(b'0\r\n\r\n') > > r = low_conn.getresponse() > resp = HTTPResponse.from_httplib( > r, > pool=conn, > connection=low_conn, > preload_content=False, > decode_content=False > ) > except: > # If we hit any problems here, clean up the connection. > # Then, reraise so that we can handle the actual exception. > low_conn.close() > raise > else: > # All is well, return the connection to the pool. > conn._put_conn(low_conn) > > except (ProtocolError, socket.error) as err: >> raise ConnectionError(err, request=request) >E botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py:415: ConnectionError >______________________________________________ test_read_keys_from_bucket ______________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3f1e48c> > > def test_read_keys_from_bucket(s3): > for k, data in files.items(): >> file_contents = s3.cat('/'.join([test_bucket_name, k])) > >s3fs/tests/test_s3fs.py:365: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:583: in cat > return f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb4a4a5ec>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >______________________________________________________ test_seek _______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb330812c> > > def test_seek(s3): > with s3.open(a, 'wb') as f: > f.write(b'123') > > with s3.open(a) as f: > f.seek(1000) > with pytest.raises(ValueError): > f.seek(-1) > with pytest.raises(ValueError): > f.seek(-5, 2) > with pytest.raises(ValueError): > f.seek(0, 10) > f.seek(0) >> assert f.read(1) == b'1' > >s3fs/tests/test_s3fs.py:396: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb32ff3ac>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >______________________________________________________ test_copy _______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3f2cf2c> > > def test_copy(s3): > fn = test_bucket_name+'/test/accounts.1.json' > s3.copy(fn, fn+'2') >> assert s3.cat(fn) == s3.cat(fn+'2') > >s3fs/tests/test_s3fs.py:418: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:583: in cat > return f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb40d6acc>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >______________________________________________________ test_move _______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3b9b2cc> > > def test_move(s3): > fn = test_bucket_name+'/test/accounts.1.json' >> data = s3.cat(fn) > >s3fs/tests/test_s3fs.py:423: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:583: in cat > return f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb41c008c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_____________________________________________________ test_get_put _____________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3e1f86c> > > def test_get_put(s3): > with tmpfile() as fn: >> s3.get(test_bucket_name+'/test/accounts.1.json', fn) > >s3fs/tests/test_s3fs.py:431: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:619: in get > data = f.read(f.blocksize) >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3e3220c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >___________________________________________________ test_read_small ____________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3e4584c> > > def test_read_small(s3): > fn = test_bucket_name+'/2014-01-01.csv' > with s3.open(fn, 'rb', block_size=10) as f: > out = [] > while True: >> data = f.read(3) > >s3fs/tests/test_s3fs.py:483: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb206a5cc>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_________________________________________________ test_seek_delimiter __________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3e3208c> > > def test_seek_delimiter(s3): > fn = 'test/accounts.1.json' > data = files[fn] > with s3.open('/'.join([test_bucket_name, fn])) as f: > seek_delimiter(f, b'}', 0) > assert f.tell() == 0 > f.seek(1) >> seek_delimiter(f, b'}', 5) > >s3fs/tests/test_s3fs.py:499: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/utils.py:59: in seek_delimiter > current = file.read(blocksize) >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3b1518c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >__________________________________________________ test_read_s3_block __________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3b0960c> > > def test_read_s3_block(s3): > data = files['test/accounts.1.json'] > lines = io.BytesIO(data).readlines() > path = test_bucket_name+'/test/accounts.1.json' >> assert s3.read_block(path, 1, 35, b'\n') == lines[1] > >s3fs/tests/test_s3fs.py:513: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:840: in read_block > bytes = read_block(f, offset, length, delimiter) >s3fs/utils.py:109: in read_block > seek_delimiter(f, delimiter, 2**16) >s3fs/utils.py:59: in seek_delimiter > current = file.read(blocksize) >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3f33d8c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >___________________________________________________ test_write_small ___________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3e16b2c> > > def test_write_small(s3): > with s3.open(test_bucket_name+'/test', 'wb') as f: > f.write(b'hello') >> assert s3.cat(test_bucket_name+'/test') == b'hello' > >s3fs/tests/test_s3fs.py:556: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:583: in cat > return f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3f1e6cc>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_______________________________________________ test_write_small_secure ________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3d7c76c> > > def test_write_small_secure(s3): > # Unfortunately moto does not yet support enforcing SSE policies. It also > # does not return the correct objects that can be used to test the results > # effectively. > # This test is left as a placeholder in case moto eventually supports this. > sse_params = SSEParams(server_side_encryption='aws:kms') > with s3.open(secure_bucket_name+'/test', 'wb', writer_kwargs=sse_params) as f: > f.write(b'hello') >> assert s3.cat(secure_bucket_name+'/test') == b'hello' > >s3fs/tests/test_s3fs.py:569: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:583: in cat > return f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3aaabac>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_______________________________________________ test_write_large_secure ________________________________________________ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3dcd6ac> >method = 'GET', url = '/mybucket/myfile' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcd2ec> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-16242879', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo ...t-sha256;x-amz-date;x-amz-security-token, Signature=ea0514d62f4468a0724f1e7cc66243e1174ab22ca6ac04e619fb4ad3e37dc41e'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcdccc>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses >> httplib_response = conn.getresponse(buffering=True) >E TypeError: getresponse() got an unexpected keyword argument 'buffering' > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:372: TypeError > >During handling of the above exception, another exception occurred: > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/mybucket/myfile', body = None >headers = {'Range': b'bytes=0-16242879', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-C...nt-sha256;x-amz-date;x-amz-security-token, Signature=ea0514d62f4468a0724f1e7cc66243e1174ab22ca6ac04e619fb4ad3e37dc41e'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb20c8cec>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcd2ec> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, >> body=body, headers=headers) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:544: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3dcd6ac> >method = 'GET', url = '/mybucket/myfile' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcd2ec> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-16242879', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo ...t-sha256;x-amz-date;x-amz-security-token, Signature=ea0514d62f4468a0724f1e7cc66243e1174ab22ca6ac04e619fb4ad3e37dc41e'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcdccc>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses > httplib_response = conn.getresponse(buffering=True) > except TypeError: # Python 2.6 and older >> httplib_response = conn.getresponse() > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:374: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3dcd6ac> > > def getresponse(self): > """Get the response from the server. > > If the HTTPConnection is in the correct state, returns an > instance of HTTPResponse or of whatever object is returned by > the response_class variable. > > If a request has not been sent or if a previous response has > not be handled, ResponseNotReady is raised. If the HTTP > response indicates that the connection should be closed, then > it will be closed before the response is returned. When the > connection is closed, the underlying socket is closed. > """ > > # if a prior response has been completed, then forget about it. > if self.__response and self.__response.isclosed(): > self.__response = None > > # if a prior response exists, then it must be completed (otherwise, we > # cannot read this response's header to determine the connection-close > # behavior) > # > # note: if a prior response existed, but was connection-close, then the > # socket and response were made independent of this HTTPConnection > # object since a new request requires that we open a whole new > # connection > # > # this means the prior response had one of two states: > # 1) will_close: this connection was reset and the prior socket and > # response operate independently > # 2) persistent: the response was retained and we await its > # isclosed() status to become true. > # > if self.__state != _CS_REQ_SENT or self.__response: > raise ResponseNotReady(self.__state) > > if self.debuglevel > 0: > response = self.response_class(self.sock, self.debuglevel, > method=self._method) > else: > response = self.response_class(self.sock, method=self._method) > > try: > try: >> response.begin() > >/usr/lib/python3.5/http/client.py:1198: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <http.client.HTTPResponse object at 0xb3dcd0ac> > > def begin(self): > if self.headers is not None: > # we've already started reading the response > return > > # read until we get a non-100 response > while True: > version, status, reason = self._read_status() > if status != CONTINUE: > break > # skip the header from the 100 response > while True: > skip = self.fp.readline(_MAXLINE + 1) > if len(skip) > _MAXLINE: > raise LineTooLong("header line") > skip = skip.strip() > if not skip: > break > if self.debuglevel > 0: > print("header:", skip) > > self.code = self.status = status > self.reason = reason.strip() > if version in ("HTTP/1.0", "HTTP/0.9"): > # Some servers might still return "0.9", treat it as 1.0 anyway > self.version = 10 > elif version.startswith("HTTP/1."): > self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 > else: > raise UnknownProtocol(version) > >> self.headers = self.msg = parse_headers(self.fp) > >/usr/lib/python3.5/http/client.py:321: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >fp = <httpretty.core.FakeSockFile object at 0xb40d247c>, _class = <class 'http.client.HTTPMessage'> > > def parse_headers(fp, _class=HTTPMessage): > """Parses only RFC2822 headers from a file pointer. > > email Parser wants to see strings rather than bytes. > But a TextIOWrapper around self.rfile would buffer too many bytes > from the stream, bytes which we later need to read as bytes. > So we read the correct bytes here, as bytes, for email Parser > to parse. > > """ > headers = [] > while True: > line = fp.readline(_MAXLINE + 1) > if len(line) > _MAXLINE: >> raise LineTooLong("header line") >E http.client.LineTooLong: got more than 65536 bytes when reading header line > >/usr/lib/python3.5/http/client.py:208: LineTooLong > >During handling of the above exception, another exception occurred: > >self = <botocore.vendored.requests.adapters.HTTPAdapter object at 0xb345cd2c>, request = <PreparedRequest [GET]> >stream = True, timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb20c8cec> >verify = True, cert = None, proxies = {} > > def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): > """Sends PreparedRequest object. Returns Response object. > > :param request: The :class:`PreparedRequest <PreparedRequest>` being sent. > :param stream: (optional) Whether to stream the request content. > :param timeout: (optional) How long to wait for the server to send > data before giving up, as a float, or a (`connect timeout, read > timeout <user/advanced.html#timeouts>`_) tuple. > :type timeout: float or tuple > :param verify: (optional) Whether to verify SSL certificates. > :param cert: (optional) Any user-provided SSL certificate to be trusted. > :param proxies: (optional) The proxies dictionary to apply to the request. > """ > > conn = self.get_connection(request.url, proxies) > > self.cert_verify(conn, request.url, verify, cert) > url = self.request_url(request, proxies) > self.add_headers(request) > > chunked = not (request.body is None or 'Content-Length' in request.headers) > > if isinstance(timeout, tuple): > try: > connect, read = timeout > timeout = TimeoutSauce(connect=connect, read=read) > except ValueError as e: > # this may raise a string formatting error. > err = ("Invalid timeout {0}. Pass a (connect, read) " > "timeout tuple, or a single float to set " > "both timeouts to the same value".format(timeout)) > raise ValueError(err) > else: > timeout = TimeoutSauce(connect=timeout, read=timeout) > > try: > if not chunked: > resp = conn.urlopen( > method=request.method, > url=url, > body=request.body, > headers=request.headers, > redirect=False, > assert_same_host=False, > preload_content=False, > decode_content=False, > retries=self.max_retries, >> timeout=timeout > ) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py:370: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/mybucket/myfile', body = None >headers = {'Range': b'bytes=0-16242879', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-C...nt-sha256;x-amz-date;x-amz-security-token, Signature=ea0514d62f4468a0724f1e7cc66243e1174ab22ca6ac04e619fb4ad3e37dc41e'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb20c8cec>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcd2ec> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, > body=body, headers=headers) > > # If we're going to release the connection in ``finally:``, then > # the request doesn't need to know about the connection. Otherwise > # it will also try to release it and we'll have a double-release > # mess. > response_conn = not release_conn and conn > > # Import httplib's response into our own wrapper object > response = HTTPResponse.from_httplib(httplib_response, > pool=self, > connection=response_conn, > **response_kw) > > # else: > # The connection will be put back into the pool when > # ``response.release_conn()`` is called (implicitly by > # ``response.read()``) > > except Empty: > # Timed out by queue. > raise EmptyPoolError(self, "No pool connections are available.") > > except (BaseSSLError, CertificateError) as e: > # Close the connection. If a connection is reused on which there > # was a Certificate error, the next request will certainly raise > # another Certificate error. > if conn: > conn.close() > conn = None > raise SSLError(e) > > except SSLError: > # Treat SSLError separately from BaseSSLError to preserve > # traceback. > if conn: > conn.close() > conn = None > raise > > except (TimeoutError, HTTPException, SocketError, ConnectionError) as e: > if conn: > # Discard the connection for these exceptions. It will be > # be replaced during the next _get_conn() call. > conn.close() > conn = None > > if isinstance(e, SocketError) and self.proxy: > e = ProxyError('Cannot connect to proxy.', e) > elif isinstance(e, (SocketError, HTTPException)): > e = ProtocolError('Connection aborted.', e) > > retries = retries.increment(method, url, error=e, _pool=self, >> _stacktrace=sys.exc_info()[2]) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:597: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = Retry(total=0, connect=None, read=False, redirect=None), method = 'GET', url = '/mybucket/myfile' >response = None >error = ProtocolError('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) >_pool = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >_stacktrace = <traceback object at 0xb32d37d4> > > def increment(self, method=None, url=None, response=None, error=None, _pool=None, _stacktrace=None): > """ Return a new Retry object with incremented retry counters. > > :param response: A response object, or None, if the server did not > return a response. > :type response: :class:`~urllib3.response.HTTPResponse` > :param Exception error: An error encountered during the request, or > None if the response was received successfully. > > :return: A new ``Retry`` object. > """ > if self.total is False and error: > # Disabled, indicate to re-raise the error. > raise six.reraise(type(error), error, _stacktrace) > > total = self.total > if total is not None: > total -= 1 > > _observed_errors = self._observed_errors > connect = self.connect > read = self.read > redirect = self.redirect > cause = 'unknown' > > if error and self._is_connection_error(error): > # Connect retry? > if connect is False: > raise six.reraise(type(error), error, _stacktrace) > elif connect is not None: > connect -= 1 > _observed_errors += 1 > > elif error and self._is_read_error(error): > # Read retry? > if read is False: >> raise six.reraise(type(error), error, _stacktrace) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/util/retry.py:245: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >tp = <class 'botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError'> >value = ProtocolError('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) >tb = <traceback object at 0xb32d37d4> > > def reraise(tp, value, tb=None): > if value.__traceback__ is not tb: >> raise value.with_traceback(tb) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/packages/six.py:309: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/mybucket/myfile', body = None >headers = {'Range': b'bytes=0-16242879', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-C...nt-sha256;x-amz-date;x-amz-security-token, Signature=ea0514d62f4468a0724f1e7cc66243e1174ab22ca6ac04e619fb4ad3e37dc41e'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb20c8cec>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcd2ec> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, >> body=body, headers=headers) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:544: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3dcd6ac> >method = 'GET', url = '/mybucket/myfile' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcd2ec> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-16242879', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo ...t-sha256;x-amz-date;x-amz-security-token, Signature=ea0514d62f4468a0724f1e7cc66243e1174ab22ca6ac04e619fb4ad3e37dc41e'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3dcdccc>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses > httplib_response = conn.getresponse(buffering=True) > except TypeError: # Python 2.6 and older >> httplib_response = conn.getresponse() > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:374: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3dcd6ac> > > def getresponse(self): > """Get the response from the server. > > If the HTTPConnection is in the correct state, returns an > instance of HTTPResponse or of whatever object is returned by > the response_class variable. > > If a request has not been sent or if a previous response has > not be handled, ResponseNotReady is raised. If the HTTP > response indicates that the connection should be closed, then > it will be closed before the response is returned. When the > connection is closed, the underlying socket is closed. > """ > > # if a prior response has been completed, then forget about it. > if self.__response and self.__response.isclosed(): > self.__response = None > > # if a prior response exists, then it must be completed (otherwise, we > # cannot read this response's header to determine the connection-close > # behavior) > # > # note: if a prior response existed, but was connection-close, then the > # socket and response were made independent of this HTTPConnection > # object since a new request requires that we open a whole new > # connection > # > # this means the prior response had one of two states: > # 1) will_close: this connection was reset and the prior socket and > # response operate independently > # 2) persistent: the response was retained and we await its > # isclosed() status to become true. > # > if self.__state != _CS_REQ_SENT or self.__response: > raise ResponseNotReady(self.__state) > > if self.debuglevel > 0: > response = self.response_class(self.sock, self.debuglevel, > method=self._method) > else: > response = self.response_class(self.sock, method=self._method) > > try: > try: >> response.begin() > >/usr/lib/python3.5/http/client.py:1198: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <http.client.HTTPResponse object at 0xb3dcd0ac> > > def begin(self): > if self.headers is not None: > # we've already started reading the response > return > > # read until we get a non-100 response > while True: > version, status, reason = self._read_status() > if status != CONTINUE: > break > # skip the header from the 100 response > while True: > skip = self.fp.readline(_MAXLINE + 1) > if len(skip) > _MAXLINE: > raise LineTooLong("header line") > skip = skip.strip() > if not skip: > break > if self.debuglevel > 0: > print("header:", skip) > > self.code = self.status = status > self.reason = reason.strip() > if version in ("HTTP/1.0", "HTTP/0.9"): > # Some servers might still return "0.9", treat it as 1.0 anyway > self.version = 10 > elif version.startswith("HTTP/1."): > self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 > else: > raise UnknownProtocol(version) > >> self.headers = self.msg = parse_headers(self.fp) > >/usr/lib/python3.5/http/client.py:321: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >fp = <httpretty.core.FakeSockFile object at 0xb40d247c>, _class = <class 'http.client.HTTPMessage'> > > def parse_headers(fp, _class=HTTPMessage): > """Parses only RFC2822 headers from a file pointer. > > email Parser wants to see strings rather than bytes. > But a TextIOWrapper around self.rfile would buffer too many bytes > from the stream, bytes which we later need to read as bytes. > So we read the correct bytes here, as bytes, for email Parser > to parse. > > """ > headers = [] > while True: > line = fp.readline(_MAXLINE + 1) > if len(line) > _MAXLINE: >> raise LineTooLong("header line") >E botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) > >/usr/lib/python3.5/http/client.py:208: ProtocolError > >During handling of the above exception, another exception occurred: > >s3 = <s3fs.core.S3FileSystem object at 0xb206946c> > > def test_write_large_secure(s3): > mock = moto.mock_s3() > mock.start() > > # build our own s3fs with the relevant additional kwarg > s3 = S3FileSystem(s3_additional_kwargs = {'ServerSideEncryption': 'AES256'}) > s3.mkdir('mybucket') > > with s3.open('mybucket/myfile', 'wb') as f: > f.write(b'hello hello' * 10**6) > >> assert s3.cat('mybucket/myfile') == b'hello hello' * 10**6 > >s3fs/tests/test_s3fs.py:583: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:583: in cat > return f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:170: in _send_request > success_response, exception): >/usr/lib/python3.5/site-packages/botocore/endpoint.py:249: in _needs_retry > caught_exception=caught_exception, request_dict=request_dict) >/usr/lib/python3.5/site-packages/botocore/hooks.py:227: in emit > return self._emit(event_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/hooks.py:210: in _emit > response = handler(**kwargs) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:183: in __call__ > if self._checker(attempts, response, caught_exception): >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:251: in __call__ > caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:277: in _should_retry > return self._checker(attempt_number, response, caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:317: in __call__ > caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:223: in __call__ > attempt_number, caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:359: in _check_caught_exception > raise caught_exception >/usr/lib/python3.5/site-packages/botocore/endpoint.py:204: in _get_response > proxies=self.proxies, timeout=self.timeout) >/usr/lib/python3.5/site-packages/botocore/vendored/requests/sessions.py:573: in send > r = adapter.send(request, **kwargs) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.adapters.HTTPAdapter object at 0xb345cd2c>, request = <PreparedRequest [GET]> >stream = True, timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb20c8cec> >verify = True, cert = None, proxies = {} > > def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): > """Sends PreparedRequest object. Returns Response object. > > :param request: The :class:`PreparedRequest <PreparedRequest>` being sent. > :param stream: (optional) Whether to stream the request content. > :param timeout: (optional) How long to wait for the server to send > data before giving up, as a float, or a (`connect timeout, read > timeout <user/advanced.html#timeouts>`_) tuple. > :type timeout: float or tuple > :param verify: (optional) Whether to verify SSL certificates. > :param cert: (optional) Any user-provided SSL certificate to be trusted. > :param proxies: (optional) The proxies dictionary to apply to the request. > """ > > conn = self.get_connection(request.url, proxies) > > self.cert_verify(conn, request.url, verify, cert) > url = self.request_url(request, proxies) > self.add_headers(request) > > chunked = not (request.body is None or 'Content-Length' in request.headers) > > if isinstance(timeout, tuple): > try: > connect, read = timeout > timeout = TimeoutSauce(connect=connect, read=read) > except ValueError as e: > # this may raise a string formatting error. > err = ("Invalid timeout {0}. Pass a (connect, read) " > "timeout tuple, or a single float to set " > "both timeouts to the same value".format(timeout)) > raise ValueError(err) > else: > timeout = TimeoutSauce(connect=timeout, read=timeout) > > try: > if not chunked: > resp = conn.urlopen( > method=request.method, > url=url, > body=request.body, > headers=request.headers, > redirect=False, > assert_same_host=False, > preload_content=False, > decode_content=False, > retries=self.max_retries, > timeout=timeout > ) > > # Send the request. > else: > if hasattr(conn, 'proxy_pool'): > conn = conn.proxy_pool > > low_conn = conn._get_conn(timeout=timeout) > > try: > low_conn.putrequest(request.method, > url, > skip_accept_encoding=True) > > for header, value in request.headers.items(): > low_conn.putheader(header, value) > > low_conn.endheaders() > > for i in request.body: > low_conn.send(hex(len(i))[2:].encode('utf-8')) > low_conn.send(b'\r\n') > low_conn.send(i) > low_conn.send(b'\r\n') > low_conn.send(b'0\r\n\r\n') > > r = low_conn.getresponse() > resp = HTTPResponse.from_httplib( > r, > pool=conn, > connection=low_conn, > preload_content=False, > decode_content=False > ) > except: > # If we hit any problems here, clean up the connection. > # Then, reraise so that we can handle the actual exception. > low_conn.close() > raise > else: > # All is well, return the connection to the pool. > conn._put_conn(low_conn) > > except (ProtocolError, socket.error) as err: >> raise ConnectionError(err, request=request) >E botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py:415: ConnectionError >____________________________________________________ test_readline _____________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3e7a72c> > > def test_readline(s3): > all_items = chain.from_iterable([ > files.items(), csv_files.items(), text_files.items() > ]) > for k, data in all_items: > with s3.open('/'.join([test_bucket_name, k]), 'rb') as f: >> result = f.readline() > >s3fs/tests/test_s3fs.py:627: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1014: in readline > self._fetch(self.loc, self.loc + 1) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3e2580c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_______________________________________________ test_readline_from_cache _______________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb20c2fec> > > def test_readline_from_cache(s3): > data = b'a,b\n11,22\n3,4' > with s3.open(a, 'wb') as f: > f.write(data) > > with s3.open(a, 'rb') as f: >> result = f.readline() > >s3fs/tests/test_s3fs.py:639: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1014: in readline > self._fetch(self.loc, self.loc + 1) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3f2e5cc>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >________________________________________________ test_readline_partial _________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb417fdcc> > > def test_readline_partial(s3): > data = b'aaaaa,bbbbb\n12345,6789\n' > with s3.open(a, 'wb') as f: > f.write(data) > with s3.open(a, 'rb') as f: >> result = f.readline(5) > >s3fs/tests/test_s3fs.py:660: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1014: in readline > self._fetch(self.loc, self.loc + 1) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb33a9a4c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_______________________________________________ test_readline_blocksize ________________________________________________ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3bbf20c> >method = 'GET', url = '/test/tmp/test/a' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbf4cc> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-5242880', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo B...t-sha256;x-amz-date;x-amz-security-token, Signature=0df35fb1a12dfb3cd9975778094bb1d1c8cba7f892ea7f7254648d1893e7087f'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbfe8c>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses >> httplib_response = conn.getresponse(buffering=True) >E TypeError: getresponse() got an unexpected keyword argument 'buffering' > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:372: TypeError > >During handling of the above exception, another exception occurred: > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/test/tmp/test/a', body = None >headers = {'Range': b'bytes=0-5242880', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Co...nt-sha256;x-amz-date;x-amz-security-token, Signature=0df35fb1a12dfb3cd9975778094bb1d1c8cba7f892ea7f7254648d1893e7087f'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3a825cc>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbf4cc> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, >> body=body, headers=headers) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:544: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3bbf20c> >method = 'GET', url = '/test/tmp/test/a' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbf4cc> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-5242880', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo B...t-sha256;x-amz-date;x-amz-security-token, Signature=0df35fb1a12dfb3cd9975778094bb1d1c8cba7f892ea7f7254648d1893e7087f'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbfe8c>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses > httplib_response = conn.getresponse(buffering=True) > except TypeError: # Python 2.6 and older >> httplib_response = conn.getresponse() > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:374: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3bbf20c> > > def getresponse(self): > """Get the response from the server. > > If the HTTPConnection is in the correct state, returns an > instance of HTTPResponse or of whatever object is returned by > the response_class variable. > > If a request has not been sent or if a previous response has > not be handled, ResponseNotReady is raised. If the HTTP > response indicates that the connection should be closed, then > it will be closed before the response is returned. When the > connection is closed, the underlying socket is closed. > """ > > # if a prior response has been completed, then forget about it. > if self.__response and self.__response.isclosed(): > self.__response = None > > # if a prior response exists, then it must be completed (otherwise, we > # cannot read this response's header to determine the connection-close > # behavior) > # > # note: if a prior response existed, but was connection-close, then the > # socket and response were made independent of this HTTPConnection > # object since a new request requires that we open a whole new > # connection > # > # this means the prior response had one of two states: > # 1) will_close: this connection was reset and the prior socket and > # response operate independently > # 2) persistent: the response was retained and we await its > # isclosed() status to become true. > # > if self.__state != _CS_REQ_SENT or self.__response: > raise ResponseNotReady(self.__state) > > if self.debuglevel > 0: > response = self.response_class(self.sock, self.debuglevel, > method=self._method) > else: > response = self.response_class(self.sock, method=self._method) > > try: > try: >> response.begin() > >/usr/lib/python3.5/http/client.py:1198: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <http.client.HTTPResponse object at 0xb3bbf5cc> > > def begin(self): > if self.headers is not None: > # we've already started reading the response > return > > # read until we get a non-100 response > while True: > version, status, reason = self._read_status() > if status != CONTINUE: > break > # skip the header from the 100 response > while True: > skip = self.fp.readline(_MAXLINE + 1) > if len(skip) > _MAXLINE: > raise LineTooLong("header line") > skip = skip.strip() > if not skip: > break > if self.debuglevel > 0: > print("header:", skip) > > self.code = self.status = status > self.reason = reason.strip() > if version in ("HTTP/1.0", "HTTP/0.9"): > # Some servers might still return "0.9", treat it as 1.0 anyway > self.version = 10 > elif version.startswith("HTTP/1."): > self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 > else: > raise UnknownProtocol(version) > >> self.headers = self.msg = parse_headers(self.fp) > >/usr/lib/python3.5/http/client.py:321: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >fp = <httpretty.core.FakeSockFile object at 0xb20c168c>, _class = <class 'http.client.HTTPMessage'> > > def parse_headers(fp, _class=HTTPMessage): > """Parses only RFC2822 headers from a file pointer. > > email Parser wants to see strings rather than bytes. > But a TextIOWrapper around self.rfile would buffer too many bytes > from the stream, bytes which we later need to read as bytes. > So we read the correct bytes here, as bytes, for email Parser > to parse. > > """ > headers = [] > while True: > line = fp.readline(_MAXLINE + 1) > if len(line) > _MAXLINE: >> raise LineTooLong("header line") >E http.client.LineTooLong: got more than 65536 bytes when reading header line > >/usr/lib/python3.5/http/client.py:208: LineTooLong > >During handling of the above exception, another exception occurred: > >self = <botocore.vendored.requests.adapters.HTTPAdapter object at 0xb345cd2c>, request = <PreparedRequest [GET]> >stream = True, timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3a825cc> >verify = True, cert = None, proxies = {} > > def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): > """Sends PreparedRequest object. Returns Response object. > > :param request: The :class:`PreparedRequest <PreparedRequest>` being sent. > :param stream: (optional) Whether to stream the request content. > :param timeout: (optional) How long to wait for the server to send > data before giving up, as a float, or a (`connect timeout, read > timeout <user/advanced.html#timeouts>`_) tuple. > :type timeout: float or tuple > :param verify: (optional) Whether to verify SSL certificates. > :param cert: (optional) Any user-provided SSL certificate to be trusted. > :param proxies: (optional) The proxies dictionary to apply to the request. > """ > > conn = self.get_connection(request.url, proxies) > > self.cert_verify(conn, request.url, verify, cert) > url = self.request_url(request, proxies) > self.add_headers(request) > > chunked = not (request.body is None or 'Content-Length' in request.headers) > > if isinstance(timeout, tuple): > try: > connect, read = timeout > timeout = TimeoutSauce(connect=connect, read=read) > except ValueError as e: > # this may raise a string formatting error. > err = ("Invalid timeout {0}. Pass a (connect, read) " > "timeout tuple, or a single float to set " > "both timeouts to the same value".format(timeout)) > raise ValueError(err) > else: > timeout = TimeoutSauce(connect=timeout, read=timeout) > > try: > if not chunked: > resp = conn.urlopen( > method=request.method, > url=url, > body=request.body, > headers=request.headers, > redirect=False, > assert_same_host=False, > preload_content=False, > decode_content=False, > retries=self.max_retries, >> timeout=timeout > ) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py:370: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/test/tmp/test/a', body = None >headers = {'Range': b'bytes=0-5242880', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Co...nt-sha256;x-amz-date;x-amz-security-token, Signature=0df35fb1a12dfb3cd9975778094bb1d1c8cba7f892ea7f7254648d1893e7087f'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3a825cc>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbf4cc> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, > body=body, headers=headers) > > # If we're going to release the connection in ``finally:``, then > # the request doesn't need to know about the connection. Otherwise > # it will also try to release it and we'll have a double-release > # mess. > response_conn = not release_conn and conn > > # Import httplib's response into our own wrapper object > response = HTTPResponse.from_httplib(httplib_response, > pool=self, > connection=response_conn, > **response_kw) > > # else: > # The connection will be put back into the pool when > # ``response.release_conn()`` is called (implicitly by > # ``response.read()``) > > except Empty: > # Timed out by queue. > raise EmptyPoolError(self, "No pool connections are available.") > > except (BaseSSLError, CertificateError) as e: > # Close the connection. If a connection is reused on which there > # was a Certificate error, the next request will certainly raise > # another Certificate error. > if conn: > conn.close() > conn = None > raise SSLError(e) > > except SSLError: > # Treat SSLError separately from BaseSSLError to preserve > # traceback. > if conn: > conn.close() > conn = None > raise > > except (TimeoutError, HTTPException, SocketError, ConnectionError) as e: > if conn: > # Discard the connection for these exceptions. It will be > # be replaced during the next _get_conn() call. > conn.close() > conn = None > > if isinstance(e, SocketError) and self.proxy: > e = ProxyError('Cannot connect to proxy.', e) > elif isinstance(e, (SocketError, HTTPException)): > e = ProtocolError('Connection aborted.', e) > > retries = retries.increment(method, url, error=e, _pool=self, >> _stacktrace=sys.exc_info()[2]) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:597: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = Retry(total=0, connect=None, read=False, redirect=None), method = 'GET', url = '/test/tmp/test/a' >response = None >error = ProtocolError('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) >_pool = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >_stacktrace = <traceback object at 0xb20a0f54> > > def increment(self, method=None, url=None, response=None, error=None, _pool=None, _stacktrace=None): > """ Return a new Retry object with incremented retry counters. > > :param response: A response object, or None, if the server did not > return a response. > :type response: :class:`~urllib3.response.HTTPResponse` > :param Exception error: An error encountered during the request, or > None if the response was received successfully. > > :return: A new ``Retry`` object. > """ > if self.total is False and error: > # Disabled, indicate to re-raise the error. > raise six.reraise(type(error), error, _stacktrace) > > total = self.total > if total is not None: > total -= 1 > > _observed_errors = self._observed_errors > connect = self.connect > read = self.read > redirect = self.redirect > cause = 'unknown' > > if error and self._is_connection_error(error): > # Connect retry? > if connect is False: > raise six.reraise(type(error), error, _stacktrace) > elif connect is not None: > connect -= 1 > _observed_errors += 1 > > elif error and self._is_read_error(error): > # Read retry? > if read is False: >> raise six.reraise(type(error), error, _stacktrace) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/util/retry.py:245: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >tp = <class 'botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError'> >value = ProtocolError('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) >tb = <traceback object at 0xb20a0f54> > > def reraise(tp, value, tb=None): > if value.__traceback__ is not tb: >> raise value.with_traceback(tb) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/packages/six.py:309: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >method = 'GET', url = '/test/tmp/test/a', body = None >headers = {'Range': b'bytes=0-5242880', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo Botocore/1.7.39', 'X-Amz-Co...nt-sha256;x-amz-date;x-amz-security-token, Signature=0df35fb1a12dfb3cd9975778094bb1d1c8cba7f892ea7f7254648d1893e7087f'} >retries = Retry(total=0, connect=None, read=False, redirect=None), redirect = False, assert_same_host = False >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3a825cc>, pool_timeout = None >release_conn = False, response_kw = {'decode_content': False, 'preload_content': False}, conn = None, err = None >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbf4cc> >is_new_proxy_conn = False > > def urlopen(self, method, url, body=None, headers=None, retries=None, > redirect=True, assert_same_host=True, timeout=_Default, > pool_timeout=None, release_conn=None, **response_kw): > """ > Get a connection from the pool and perform an HTTP request. This is the > lowest level call for making a request, so you'll need to specify all > the raw details. > > .. note:: > > More commonly, it's appropriate to use a convenience method provided > by :class:`.RequestMethods`, such as :meth:`request`. > > .. note:: > > `release_conn` will only behave as expected if > `preload_content=False` because we want to make > `preload_content=False` the default behaviour someday soon without > breaking backwards compatibility. > > :param method: > HTTP request method (such as GET, POST, PUT, etc.) > > :param body: > Data to send in the request body (useful for creating > POST requests, see HTTPConnectionPool.post_url for > more convenience). > > :param headers: > Dictionary of custom headers to send, such as User-Agent, > If-None-Match, etc. If None, pool headers are used. If provided, > these headers completely replace any pool-specific headers. > > :param retries: > Configure the number of retries to allow before raising a > :class:`~urllib3.exceptions.MaxRetryError` exception. > > Pass ``None`` to retry until you receive a response. Pass a > :class:`~urllib3.util.retry.Retry` object for fine-grained control > over different types of retries. > Pass an integer number to retry connection errors that many times, > but no other types of errors. Pass zero to never retry. > > If ``False``, then retries are disabled and any exception is raised > immediately. Also, instead of raising a MaxRetryError on redirects, > the redirect response will be returned. > > :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. > > :param redirect: > If True, automatically handle redirects (status codes 301, 302, > 303, 307, 308). Each redirect counts as a retry. Disabling retries > will disable redirect, too. > > :param assert_same_host: > If ``True``, will make sure that the host of the pool requests is > consistent else will raise HostChangedError. When False, you can > use the pool on an HTTP proxy and request foreign hosts. > > :param timeout: > If specified, overrides the default timeout for this one > request. It may be a float (in seconds) or an instance of > :class:`urllib3.util.Timeout`. > > :param pool_timeout: > If set and the pool is set to block=True, then this method will > block for ``pool_timeout`` seconds and raise EmptyPoolError if no > connection is available within the time period. > > :param release_conn: > If False, then the urlopen call will not release the connection > back into the pool once a response is received (but will release if > you read the entire contents of the response such as when > `preload_content=True`). This is useful if you're not preloading > the response's content immediately. You will need to call > ``r.release_conn()`` on the response ``r`` to return the connection > back into the pool. If None, it takes the value of > ``response_kw.get('preload_content', True)``. > > :param \**response_kw: > Additional parameters are passed to > :meth:`urllib3.response.HTTPResponse.from_httplib` > """ > if headers is None: > headers = self.headers > > if not isinstance(retries, Retry): > retries = Retry.from_int(retries, redirect=redirect, default=self.retries) > > if release_conn is None: > release_conn = response_kw.get('preload_content', True) > > # Check host > if assert_same_host and not self.is_same_host(url): > raise HostChangedError(self, url, retries) > > conn = None > > # Merge the proxy headers. Only do this in HTTP. We have to copy the > # headers dict so we can safely change it without those changes being > # reflected in anyone else's copy. > if self.scheme == 'http': > headers = headers.copy() > headers.update(self.proxy_headers) > > # Must keep the exception bound to a separate variable or else Python 3 > # complains about UnboundLocalError. > err = None > > try: > # Request a connection from the queue. > timeout_obj = self._get_timeout(timeout) > conn = self._get_conn(timeout=pool_timeout) > > conn.timeout = timeout_obj.connect_timeout > > is_new_proxy_conn = self.proxy is not None and not getattr(conn, 'sock', None) > if is_new_proxy_conn: > self._prepare_proxy(conn) > > # Make the request on the httplib connection object. > httplib_response = self._make_request(conn, method, url, > timeout=timeout_obj, >> body=body, headers=headers) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:544: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0xb2060c4c> >conn = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3bbf20c> >method = 'GET', url = '/test/tmp/test/a' >timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbf4cc> >httplib_request_kw = {'body': None, 'headers': {'Range': b'bytes=0-5242880', 'User-Agent': b'Boto3/1.4.7 Python/3.5.5 Linux/4.9.95-gentoo B...t-sha256;x-amz-date;x-amz-security-token, Signature=0df35fb1a12dfb3cd9975778094bb1d1c8cba7f892ea7f7254648d1893e7087f'}} >timeout_obj = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3bbfe8c>, read_timeout = 15 > > def _make_request(self, conn, method, url, timeout=_Default, > **httplib_request_kw): > """ > Perform a request on a given urllib connection object taken from our > pool. > > :param conn: > a connection from one of our connection pools > > :param timeout: > Socket timeout in seconds for the request. This can be a > float or integer, which will set the same timeout value for > the socket connect and the socket read, or an instance of > :class:`urllib3.util.Timeout`, which gives you more fine-grained > control over your timeouts. > """ > self.num_requests += 1 > > timeout_obj = self._get_timeout(timeout) > timeout_obj.start_connect() > conn.timeout = timeout_obj.connect_timeout > > # Trigger any extra validation we need to do. > try: > self._validate_conn(conn) > except (SocketTimeout, BaseSSLError) as e: > # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. > self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) > raise > > # conn.request() calls httplib.*.request, not the method in > # urllib3.request. It also calls makefile (recv) on the socket. > conn.request(method, url, **httplib_request_kw) > > # Reset the timeout for the recv() on the socket > read_timeout = timeout_obj.read_timeout > > # App Engine doesn't have a sock attr > if getattr(conn, 'sock', None): > # In Python 3 socket.py will catch EAGAIN and return None when you > # try and read into the file pointer created by http.client, which > # instead raises a BadStatusLine exception. Instead of catching > # the exception and assuming all BadStatusLine exceptions are read > # timeouts, check for a zero timeout before making the request. > if read_timeout == 0: > raise ReadTimeoutError( > self, url, "Read timed out. (read timeout=%s)" % read_timeout) > if read_timeout is Timeout.DEFAULT_TIMEOUT: > conn.sock.settimeout(socket.getdefaulttimeout()) > else: # None or a value > conn.sock.settimeout(read_timeout) > > # Receive the response from the server > try: > try: # Python 2.7, use buffering of HTTP responses > httplib_response = conn.getresponse(buffering=True) > except TypeError: # Python 2.6 and older >> httplib_response = conn.getresponse() > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:374: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb3bbf20c> > > def getresponse(self): > """Get the response from the server. > > If the HTTPConnection is in the correct state, returns an > instance of HTTPResponse or of whatever object is returned by > the response_class variable. > > If a request has not been sent or if a previous response has > not be handled, ResponseNotReady is raised. If the HTTP > response indicates that the connection should be closed, then > it will be closed before the response is returned. When the > connection is closed, the underlying socket is closed. > """ > > # if a prior response has been completed, then forget about it. > if self.__response and self.__response.isclosed(): > self.__response = None > > # if a prior response exists, then it must be completed (otherwise, we > # cannot read this response's header to determine the connection-close > # behavior) > # > # note: if a prior response existed, but was connection-close, then the > # socket and response were made independent of this HTTPConnection > # object since a new request requires that we open a whole new > # connection > # > # this means the prior response had one of two states: > # 1) will_close: this connection was reset and the prior socket and > # response operate independently > # 2) persistent: the response was retained and we await its > # isclosed() status to become true. > # > if self.__state != _CS_REQ_SENT or self.__response: > raise ResponseNotReady(self.__state) > > if self.debuglevel > 0: > response = self.response_class(self.sock, self.debuglevel, > method=self._method) > else: > response = self.response_class(self.sock, method=self._method) > > try: > try: >> response.begin() > >/usr/lib/python3.5/http/client.py:1198: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <http.client.HTTPResponse object at 0xb3bbf5cc> > > def begin(self): > if self.headers is not None: > # we've already started reading the response > return > > # read until we get a non-100 response > while True: > version, status, reason = self._read_status() > if status != CONTINUE: > break > # skip the header from the 100 response > while True: > skip = self.fp.readline(_MAXLINE + 1) > if len(skip) > _MAXLINE: > raise LineTooLong("header line") > skip = skip.strip() > if not skip: > break > if self.debuglevel > 0: > print("header:", skip) > > self.code = self.status = status > self.reason = reason.strip() > if version in ("HTTP/1.0", "HTTP/0.9"): > # Some servers might still return "0.9", treat it as 1.0 anyway > self.version = 10 > elif version.startswith("HTTP/1."): > self.version = 11 # use HTTP/1.1 code for HTTP/1.x where x>=1 > else: > raise UnknownProtocol(version) > >> self.headers = self.msg = parse_headers(self.fp) > >/usr/lib/python3.5/http/client.py:321: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >fp = <httpretty.core.FakeSockFile object at 0xb20c168c>, _class = <class 'http.client.HTTPMessage'> > > def parse_headers(fp, _class=HTTPMessage): > """Parses only RFC2822 headers from a file pointer. > > email Parser wants to see strings rather than bytes. > But a TextIOWrapper around self.rfile would buffer too many bytes > from the stream, bytes which we later need to read as bytes. > So we read the correct bytes here, as bytes, for email Parser > to parse. > > """ > headers = [] > while True: > line = fp.readline(_MAXLINE + 1) > if len(line) > _MAXLINE: >> raise LineTooLong("header line") >E botocore.vendored.requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) > >/usr/lib/python3.5/http/client.py:208: ProtocolError > >During handling of the above exception, another exception occurred: > >s3 = <s3fs.core.S3FileSystem object at 0xb204f08c> > > def test_readline_blocksize(s3): > data = b'ab\n' + b'a' * (10 * 2**20) + b'\nab' > with s3.open(a, 'wb') as f: > f.write(data) > with s3.open(a, 'rb') as f: >> result = f.readline() > >s3fs/tests/test_s3fs.py:684: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1014: in readline > self._fetch(self.loc, self.loc + 1) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:170: in _send_request > success_response, exception): >/usr/lib/python3.5/site-packages/botocore/endpoint.py:249: in _needs_retry > caught_exception=caught_exception, request_dict=request_dict) >/usr/lib/python3.5/site-packages/botocore/hooks.py:227: in emit > return self._emit(event_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/hooks.py:210: in _emit > response = handler(**kwargs) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:183: in __call__ > if self._checker(attempts, response, caught_exception): >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:251: in __call__ > caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:277: in _should_retry > return self._checker(attempt_number, response, caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:317: in __call__ > caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:223: in __call__ > attempt_number, caught_exception) >/usr/lib/python3.5/site-packages/botocore/retryhandler.py:359: in _check_caught_exception > raise caught_exception >/usr/lib/python3.5/site-packages/botocore/endpoint.py:204: in _get_response > proxies=self.proxies, timeout=self.timeout) >/usr/lib/python3.5/site-packages/botocore/vendored/requests/sessions.py:573: in send > r = adapter.send(request, **kwargs) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.vendored.requests.adapters.HTTPAdapter object at 0xb345cd2c>, request = <PreparedRequest [GET]> >stream = True, timeout = <botocore.vendored.requests.packages.urllib3.util.timeout.Timeout object at 0xb3a825cc> >verify = True, cert = None, proxies = {} > > def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): > """Sends PreparedRequest object. Returns Response object. > > :param request: The :class:`PreparedRequest <PreparedRequest>` being sent. > :param stream: (optional) Whether to stream the request content. > :param timeout: (optional) How long to wait for the server to send > data before giving up, as a float, or a (`connect timeout, read > timeout <user/advanced.html#timeouts>`_) tuple. > :type timeout: float or tuple > :param verify: (optional) Whether to verify SSL certificates. > :param cert: (optional) Any user-provided SSL certificate to be trusted. > :param proxies: (optional) The proxies dictionary to apply to the request. > """ > > conn = self.get_connection(request.url, proxies) > > self.cert_verify(conn, request.url, verify, cert) > url = self.request_url(request, proxies) > self.add_headers(request) > > chunked = not (request.body is None or 'Content-Length' in request.headers) > > if isinstance(timeout, tuple): > try: > connect, read = timeout > timeout = TimeoutSauce(connect=connect, read=read) > except ValueError as e: > # this may raise a string formatting error. > err = ("Invalid timeout {0}. Pass a (connect, read) " > "timeout tuple, or a single float to set " > "both timeouts to the same value".format(timeout)) > raise ValueError(err) > else: > timeout = TimeoutSauce(connect=timeout, read=timeout) > > try: > if not chunked: > resp = conn.urlopen( > method=request.method, > url=url, > body=request.body, > headers=request.headers, > redirect=False, > assert_same_host=False, > preload_content=False, > decode_content=False, > retries=self.max_retries, > timeout=timeout > ) > > # Send the request. > else: > if hasattr(conn, 'proxy_pool'): > conn = conn.proxy_pool > > low_conn = conn._get_conn(timeout=timeout) > > try: > low_conn.putrequest(request.method, > url, > skip_accept_encoding=True) > > for header, value in request.headers.items(): > low_conn.putheader(header, value) > > low_conn.endheaders() > > for i in request.body: > low_conn.send(hex(len(i))[2:].encode('utf-8')) > low_conn.send(b'\r\n') > low_conn.send(i) > low_conn.send(b'\r\n') > low_conn.send(b'0\r\n\r\n') > > r = low_conn.getresponse() > resp = HTTPResponse.from_httplib( > r, > pool=conn, > connection=low_conn, > preload_content=False, > decode_content=False > ) > except: > # If we hit any problems here, clean up the connection. > # Then, reraise so that we can handle the actual exception. > low_conn.close() > raise > else: > # All is well, return the connection to the pool. > conn._put_conn(low_conn) > > except (ProtocolError, socket.error) as err: >> raise ConnectionError(err, request=request) >E botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line',)) > >/usr/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py:415: ConnectionError >______________________________________________________ test_next _______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3f3c16c> > > def test_next(s3): > expected = csv_files['2014-01-01.csv'].split(b'\n')[0] + b'\n' > with s3.open(test_bucket_name + '/2014-01-01.csv') as f: >> result = next(f) > >s3fs/tests/test_s3fs.py:699: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1026: in __next__ > out = self.readline() >s3fs/core.py:1014: in readline > self._fetch(self.loc, self.loc + 1) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3f4b0ac>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >____________________________________________________ test_iterable _____________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb3e33d0c> > > def test_iterable(s3): > data = b'abc\n123' > with s3.open(a, 'wb') as f: > f.write(data) > with s3.open(a) as f, io.BytesIO(data) as g: >> for froms3, fromio in zip(f, g): > >s3fs/tests/test_s3fs.py:708: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1026: in __next__ > out = self.readline() >s3fs/core.py:1014: in readline > self._fetch(self.loc, self.loc + 1) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3aab36c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_____________________________________________________ test_append ______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb40d61ec> > > def test_append(s3): > data = text_files['nested/file1'] >> with s3.open(test_bucket_name+'/nested/file1', 'ab') as f: > >s3fs/tests/test_s3fs.py:762: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:293: in open > fill_cache=fill_cache, s3_additional_kwargs=kwargs) >s3fs/core.py:910: in __init__ > self.write(s3.cat(path)) >s3fs/core.py:583: in cat > return f.read() >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3df35ec>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_____________________________________________ test_bigger_than_block_read ______________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb4a85d2c> > > def test_bigger_than_block_read(s3): > with s3.open(test_bucket_name+'/2014-01-01.csv', 'rb', block_size=3) as f: > out = [] > while True: >> data = f.read(20) > >s3fs/tests/test_s3fs.py:792: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3d94e6c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >______________________________________________________ test_array ______________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb32f732c> > > def test_array(s3): > from array import array > data = array('B', [65] * 1000) > > with s3.open(a, 'wb') as f: > f.write(data) > > with s3.open(a, 'rb') as f: >> out = f.read() > >s3fs/tests/test_s3fs.py:811: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb209428c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\ > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >_________________________________________________ test_not_fill_cache __________________________________________________ > >s3 = <s3fs.core.S3FileSystem object at 0xb346fa2c> > > def test_not_fill_cache(s3): > fname = list(files)[0] > data = files[fname] > with s3.open(test_bucket_name+'/'+fname, 'rb', block_size=5, > fill_cache=False) as f: >> f.read(6) > >s3fs/tests/test_s3fs.py:896: >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >s3fs/core.py:1082: in read > self._fetch(self.loc, self.loc + length) >s3fs/core.py:1046: in _fetch > start, self.end, req_kw=self.s3.req_kw) >s3fs/core.py:1242: in _fetch_range > **req_kw) >/usr/lib/python3.5/site-packages/botocore/client.py:312: in _api_call > return self._make_api_call(operation_name, kwargs) >/usr/lib/python3.5/site-packages/botocore/client.py:592: in _make_api_call > operation_model, request_dict) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:141: in make_request > return self._send_request(request_dict, operation_model) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:168: in _send_request > request, operation_model, attempts) >/usr/lib/python3.5/site-packages/botocore/endpoint.py:233: in _get_response > response_dict, operation_model.output_shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:212: in parse > parsed = self._do_parse(response, shape) >/usr/lib/python3.5/site-packages/botocore/parsers.py:613: in _do_parse > member_shapes, final_parsed) >/usr/lib/python3.5/site-packages/botocore/parsers.py:669: in _parse_non_payload_attrs > member_shape, headers[header_name]) >/usr/lib/python3.5/site-packages/botocore/parsers.py:261: in _parse_shape > return handler(shape, node) >/usr/lib/python3.5/site-packages/botocore/parsers.py:150: in _get_text_content > return func(self, shape, text) >_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >self = <botocore.parsers.RestXMLParser object at 0xb3b0b86c>, shape = <Shape(ContentLength)> >text = "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > > @_text_content > def _handle_integer(self, shape, text): >> return int(text) >E ValueError: invalid literal for int() with base 10: "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" > >/usr/lib/python3.5/site-packages/botocore/parsers.py:419: ValueError >================================== 31 failed, 46 passed, 2 xfailed in 340.10 seconds =================================== > * ERROR: dev-python/s3fs-0.1.2::gentoo failed (test phase): > * (no error message) > * > * Call stack: > * ebuild.sh, line 124: Called src_test > * environment, line 2578: Called distutils-r1_src_test > * environment, line 834: Called _distutils-r1_run_foreach_impl 'python_test' > * environment, line 296: Called python_foreach_impl 'distutils-r1_run_phase' 'python_test' > * environment, line 2113: Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test' > * environment, line 1488: Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_test' > * environment, line 1486: Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'python_test' > * environment, line 508: Called distutils-r1_run_phase 'python_test' > * environment, line 765: Called python_test > * environment, line 2480: Called die > * The specific snippet of code: > * py.test -v || die > * > * If you need support, post the output of `emerge --info '=dev-python/s3fs-0.1.2::gentoo'`, > * the complete build log and the output of `emerge -pqv '=dev-python/s3fs-0.1.2::gentoo'`. > * The complete build log is located at '/var/tmp/portage/dev-python/s3fs-0.1.2/temp/build.log'. > * The ebuild environment file is located at '/var/tmp/portage/dev-python/s3fs-0.1.2/temp/environment'. > * Working directory: '/var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2' > * S: '/var/tmp/portage/dev-python/s3fs-0.1.2/work/s3fs-0.1.2'
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 656524
: 533248