Page MenuHomePhabricator

Fedora 20 ships python3 httplib2 0.7.7 with unpatched SSL cert file
Closed, ResolvedPublic

Description

IIRC, there is a problem with certificates in httplib2 pre-0.9 (bug 65189). FC20 ships 0.7.7 on python2 and python3

$ rpm -qi python-httplib2
Name : python-httplib2
Version : 0.7.7
Release : 3.fc20
...
$ rpm -qi python3-httplib2
Name : python3-httplib2
Version : 0.7.7
Release : 3.fc20
...

On py2, the 0.7.7 package works correctly.
On py3, the following occurs running site_tests

On py2, all tests pass using 0.7.7. On python3, tests fail badly (using en.wp as default site).

$ python3 pwb.py tests/site_tests.py
.../pywikibot/family.py:879: DeprecationWarning: imp.load_source() is deprecated; use importlib.machinery.SourceFileLoader(name, pathname).load_module() instead

myfamily = imp.load_source(fam, config.family_files[fam])

ERROR: Traceback (most recent call last):

File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 992, in _conn_request
  response = conn.getresponse()
File "/usr/lib64/python3.3/http/client.py", line 1135, in getresponse
  raise ResponseNotReady(self.__state)

http.client.ResponseNotReady: Idle

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File ".../pywikibot/data/api.py", line 517, in submit
  headers=headers, body=body)
File ".../pywikibot/tools.py", line 367, in wrapper
  return method(*__args, **__kw)
File ".../pywikibot/comms/http.py", line 258, in request
  raise request.data
File ".../pywikibot/comms/threadedhttp.py", line 219, in request
  max_redirects, connection_type
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1272, in request
  (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1026, in _request
  (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 999, in _conn_request
  conn.connect()
File "/usr/lib64/python3.3/http/client.py", line 1202, in connect
  server_hostname=server_hostname)
File "/usr/lib64/python3.3/ssl.py", line 245, in wrap_socket
  _context=self)
File "/usr/lib64/python3.3/ssl.py", line 345, in __init__
  raise x
File "/usr/lib64/python3.3/ssl.py", line 341, in __init__
  self.do_handshake()
File "/usr/lib64/python3.3/ssl.py", line 548, in do_handshake
  self._sslobj.do_handshake()

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:547)

WARNING: Waiting 5 seconds before retrying. ERROR: Traceback (most recent call last): File "/usr/lib/python3.3/site-packages/httplib2/__init.py", line 992, in _conn_request response = conn.getresponse() File "/usr/lib64/python3.3/http/client.py", line 1135, in getresponse raise ResponseNotReady(self.state) http.client.ResponseNotReady: Idle

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File ".../pywikibot/data/api.py", line 517, in submit
  headers=headers, body=body)
File ".../pywikibot/tools.py", line 367, in wrapper
  return method(*__args, **__kw)
File ".../pywikibot/comms/http.py", line 258, in request
  raise request.data
File ".../pywikibot/comms/threadedhttp.py", line 219, in request
  max_redirects, connection_type
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1272, in request
  (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1026, in _request
  (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 999, in _conn_request
  conn.connect()
File "/usr/lib64/python3.3/http/client.py", line 1202, in connect
  server_hostname=server_hostname)
File "/usr/lib64/python3.3/ssl.py", line 245, in wrap_socket
  _context=self)
File "/usr/lib64/python3.3/ssl.py", line 345, in __init__
  raise x
File "/usr/lib64/python3.3/ssl.py", line 341, in __init__
  self.do_handshake()
File "/usr/lib64/python3.3/ssl.py", line 548, in do_handshake
  self._sslobj.do_handshake()

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:547)

WARNING: Waiting 10 seconds before retrying. ^CTraceback (most recent call last): File "/usr/lib/python3.3/site-packages/httplib2/__init.py", line 992, in _conn_request response = conn.getresponse() File "/usr/lib64/python3.3/http/client.py", line 1135, in getresponse raise ResponseNotReady(self.state) http.client.ResponseNotReady: Idle

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File ".../pywikibot/data/api.py", line 517, in submit
  headers=headers, body=body)
File ".../pywikibot/tools.py", line 367, in wrapper
  return method(*__args, **__kw)
File ".../pywikibot/comms/http.py", line 258, in request
  raise request.data
File ".../pywikibot/comms/threadedhttp.py", line 219, in request
  max_redirects, connection_type
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1272, in request
  (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1026, in _request
  (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 999, in _conn_request
  conn.connect()
File "/usr/lib64/python3.3/http/client.py", line 1202, in connect
  server_hostname=server_hostname)
File "/usr/lib64/python3.3/ssl.py", line 245, in wrap_socket
  _context=self)
File "/usr/lib64/python3.3/ssl.py", line 345, in __init__
  raise x
File "/usr/lib64/python3.3/ssl.py", line 341, in __init__
  self.do_handshake()
File "/usr/lib64/python3.3/ssl.py", line 548, in do_handshake
  self._sslobj.do_handshake()

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:547)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "pwb.py", line 164, in <module>
  run_python_file(fn, argv, argvu)
File "pwb.py", line 69, in run_python_file
  exec(compile(source, filename, "exec"), main_mod.__dict__)
File "tests/site_tests.py", line 1481, in <module>
  unittest.main()
File "/usr/lib64/python3.3/unittest/main.py", line 125, in __init__
  self.runTests()
File "/usr/lib64/python3.3/unittest/main.py", line 265, in runTests
  self.result = testRunner.run(self.test)
File "/usr/lib64/python3.3/unittest/runner.py", line 168, in run
  test(result)
File "/usr/lib64/python3.3/unittest/suite.py", line 67, in __call__
  return self.run(*args, **kwds)
File "/usr/lib64/python3.3/unittest/suite.py", line 105, in run
  test(result)
File "/usr/lib64/python3.3/unittest/suite.py", line 67, in __call__
  return self.run(*args, **kwds)
File "/usr/lib64/python3.3/unittest/suite.py", line 105, in run
  test(result)
File "/usr/lib64/python3.3/unittest/case.py", line 529, in __call__
  return self.run(*args, **kwds)
File "/usr/lib64/python3.3/unittest/case.py", line 477, in run
  self._executeTestPart(testMethod, outcome, isTest=True)
File "/usr/lib64/python3.3/unittest/case.py", line 422, in _executeTestPart
  function()
File "tests/site_tests.py", line 1065, in testRandompages
  rn = list(mysite.randompages(total=10))
File ".../pywikibot/site.py", line 3574, in randompages
  g_content=content)
File ".../pywikibot/site.py", line 1357, in _generator
  gen = gen_class(type_arg, site=self, **args)
File ".../pywikibot/data/api.py", line 1100, in __init__
  QueryGenerator.__init__(self, generator=generator, **kwargs)
File ".../pywikibot/data/api.py", line 819, in __init__
  self.update_limit()  # sets self.prefix
File ".../pywikibot/data/api.py", line 914, in update_limit
  for param in self._modules[mod].get("parameters", []):
File ".../pywikibot/data/api.py", line 865, in _modules
  data = paramreq.submit()
File ".../pywikibot/data/api.py", line 764, in submit
  self._data = super(CachedRequest, self).submit()
File ".../pywikibot/data/api.py", line 535, in submit
  self.wait()
File ".../pywikibot/data/api.py", line 651, in wait
  time.sleep(self.retry_wait)

KeyboardInterrupt
<class 'KeyboardInterrupt'>
CRITICAL: Waiting for 1 network thread(s) to finish. Press ctrl-c to abort


Version: core-(2.0)
Severity: normal
OS: Linux
Platform: PC
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=72236
https://bugzilla.redhat.com/show_bug.cgi?id=1154641

Details

Reference
bz72009

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:51 AM
bzimport set Reference to bz72009.
bzimport added a subscriber: Unknown Object (????).

In addition, setting Family.ignore_certificate_error does not work in py3 (bug 72236 - upstream)

The reason for the problem with py3 is Fedora patches httplib2-0.7.7 to use the system certs, but only for py2. Therefore python3-httplib2-0.7.7 is using the certs included in the distributed 0.7.7 tarball, which is mostly junk data nowadays.

fwiw, it looks like FC21 is likely to be based on py34 and include httplib2-0.9

Our typical workarounds for httplib2 work well; either install 0.9 using pip, or recursively clone the pywikibot repo, so it can use 'externals/httplib2' - both tested and work correctly.

jayvdb claimed this task.

Switched to python-requests.

jayvdb changed the task status from Declined to Resolved.Jul 18 2015, 6:05 PM
jayvdb moved this task from Backlog to Reported Upstream on the Upstream board.