Page MenuHomePhabricator
Create Task
Maniphest T68102

use one library for all http requests
Closed, ResolvedPublic
Actions

Description

requests has been chosen as the http library for pywikibot v3.0 master.

There are a few cases of urllib.urlopen (and others) being used in the pywikibot library code, and a number of scripts which use other http request routines.
Multiple routines results in multiple configuration (e.g. proxy) and multiple sets of possible bugs/errors.

All http activity should be provided by utility methods in pywikibot.comms.http, so it is easy to test and support them, and possibly use a different http library in the future if necessary.

See Also: T71204

Details

Reference
bz66102
SubjectRepoBranchLines +/-
[bugfix] Re-enable script test for imageharvest.pypywikibot/coremaster+13 -9
[bugfix] bugfixes and improvements for checkimagespywikibot/coremaster+131 -150
[IMPROV] Replace openurl with http.fetchpywikibot/coremaster+16 -18
Customize query in gerrit

Related Objects
Search...

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:11 AM
• bzimport added a project: Pywikibot-network.
• bzimport set Reference to bz66102.
jayvdb created this task.Jun 4 2014, 12:00 AM
jayvdb added a comment.Jul 1 2014, 2:33 AM

The may be issues with using httplib2 for large downloads, like are possible in upload.py.

https://github.com/jcgregorio/httplib2/issues/224

A fork has been created for that, and distributed caching.

https://github.com/madlag/streaming_httplib2

jayvdb added a comment.Aug 6 2014, 9:07 PM

site.py & weblib.py use 'import urllib', but for urlencode

urllib:
pywikibot/page.py:1841: f = urllib.urlopen(self.fileUrl())
pywikibot/version.py:199: buf = urllib.urlopen(url).readlines()

scripts/upload.py
scripts/flickrripper.py
scripts/checkimages.py
scripts/weblinkchecker.py
scripts/imagerecat.py
scripts/maintenance/wikimedia_sites.py
scripts/data_ingestion.py

urllib2:
scripts/reflinks.py

httplib (not httplib2):
pywikibot/version.py:123: conn = httplib.HTTPSConnection('github.com')

scripts/weblinkchecker.py
scripts/reflinks.py

gerritbot added a comment.Aug 7 2014, 12:33 AM

Change 152200 had a related patch set uploaded by John Vandenberg:
HTTP requests with user-agent without version

https://gerrit.wikimedia.org/r/152200

gerritbot added a comment.Aug 10 2014, 12:19 PM

Change 153300 had a related patch set uploaded by John Vandenberg:
Replace httplib and urllib with httplib2

https://gerrit.wikimedia.org/r/153300

gerritbot added a comment.Sep 3 2014, 9:11 PM

Change 152200 merged by jenkins-bot:
User-agent graceful degradation

https://gerrit.wikimedia.org/r/152200

gerritbot added a comment.Sep 4 2014, 8:34 PM

Change 153300 merged by jenkins-bot:
Replace httplib and urllib with httplib2

https://gerrit.wikimedia.org/r/153300

jayvdb added a comment.Oct 6 2014, 11:21 AM

version.py now uses httplib2.

In addition to the list above, generate_family_file.py also uses urllib2

jayvdb added a comment.Nov 9 2014, 2:47 AM

https://github.com/ross/python-asynchttp might be a good solution, but it doesnt appear to be very active

Aklapper added a project: Pywikibot.Nov 27 2014, 4:14 PM
jayvdb added a comment.Apr 9 2015, 10:52 AM

Another httplib2 fork, which says it provides streaming: https://github.com/fffonion/httplib2-plus

Also, we have a patch to switch to python-requests: https://gerrit.wikimedia.org/r/#/c/189821/

jayvdb closed subtask T98439: Add support for using python-requests as Resolved.Jun 2 2015, 5:00 PM
gerritbot subscribed.Jun 2 2015, 6:01 PM

Change 208479 had a related patch set uploaded (by XZise):
[IMPROV] Replace openurl with http.fetch

https://gerrit.wikimedia.org/r/208479

gerritbot added a project: Patch-For-Review.Jun 2 2015, 6:01 PM
gerritbot added a comment.Jun 3 2015, 1:29 AM

Change 208479 merged by jenkins-bot:
[IMPROV] Replace openurl with http.fetch

https://gerrit.wikimedia.org/r/208479

XZise mentioned this in rPWBC690ca26ca3c1: [IMPROV] Replace openurl with http.fetch.Jun 3 2015, 1:29 AM
jayvdb mentioned this in T71204: weblinkchecker.py uses a fake user-agent.Jan 10 2016, 10:28 AM
jayvdb updated the task description. (Show Details)Jan 10 2016, 10:48 AM
jayvdb set Security to None.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 10 2016, 10:48 AM
jayvdb triaged this task as High priority.Jan 10 2016, 10:48 AM
jayvdb removed a project: Patch-For-Review.
jayvdb closed subtask T111300: Convert reflinks to requests as Resolved.Jan 19 2016, 5:10 AM
jayvdb closed subtask T113140: Convert weblinkchecker to requests as Resolved.Jan 21 2016, 3:19 PM
jayvdb created subtask T130522: Convert panoramiopicker.py to requests.Mar 21 2016, 11:37 AM
jayvdb created subtask T130523: Convert imagecopy.py to requests.
gerritbot added a comment.Apr 3 2016, 12:08 PM

Change 281131 had a related patch set uploaded (by Xqt):
[bugfix] bugfixes and improvements for checkimages

https://gerrit.wikimedia.org/r/281131

gerritbot added a project: Patch-For-Review.Apr 3 2016, 12:08 PM
gerritbot added a comment.Apr 5 2016, 3:32 PM

Change 281673 had a related patch set uploaded (by Xqt):
[bugfix] bugfixes and improvements for checkimages

https://gerrit.wikimedia.org/r/281673

Dvorapa closed subtask T130522: Convert panoramiopicker.py to requests as Declined.Jun 11 2017, 6:56 PM
Dvorapa mentioned this in T130523: Convert imagecopy.py to requests.Dec 24 2017, 11:58 AM
jayvdb removed a project: Patch-For-Review.Jan 7 2018, 3:35 AM
jayvdb removed subscribers: gerritbot, pywikibot-bugs-list.
jayvdb added a comment.Jan 7 2018, 4:10 AM

Two new possible GCI tasks T184360 & T184361 . Need more analysis done, and expanding the description to ensure that the work is of a high quality.

Framawiki closed subtask T130523: Convert imagecopy.py to requests as Resolved.Jan 7 2018, 6:04 PM
Xqt updated the task description. (Show Details)May 6 2018, 8:23 AM
xSavitar moved this task from Backlog to Needs Review on the Pywikibot board.Nov 5 2018, 11:38 AM
Xqt raised the priority of this task from High to Needs Triage.Feb 3 2019, 12:27 PM
Xqt moved this task from Needs Review to Backlog on the Pywikibot board.
Xqt closed subtask T184361: Convert specialbots to use requests as Invalid.Feb 3 2019, 12:38 PM
Xqt closed subtask T184360: Convert imageharvest to requests as Declined.Feb 3 2019, 12:40 PM
Xqt triaged this task as High priority.Feb 4 2019, 5:27 AM
Dvorapa subscribed.EditedJan 27 2020, 7:29 PM

I would focus on this task to avoid issues with urllib, urllib2, urllib3, httplib etc., which might be a cause for many other bugs we encounter

Xqt removed jayvdb as the assignee of this task.Jan 27 2020, 7:53 PM
gerritbot added a comment.Apr 12 2020, 9:51 AM

Change 588145 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] Re-enable script test for imageharvest.py

https://gerrit.wikimedia.org/r/588145

gerritbot added a project: Patch-For-Review.Apr 12 2020, 9:51 AM
gerritbot added a comment.Apr 12 2020, 11:21 AM

Change 588145 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] Re-enable script test for imageharvest.py

https://gerrit.wikimedia.org/r/588145

Xqt mentioned this in rPWBCf562036a8c22: [bugfix] Re-enable script test for imageharvest.py.Apr 12 2020, 11:28 AM
Xqt added a subtask: T255575: urllib.request.URLopener is deprecated since Python 3.3.Jun 16 2020, 3:35 PM
Xqt closed subtask T255575: urllib.request.URLopener is deprecated since Python 3.3 as Resolved.Oct 17 2020, 10:01 AM
Xqt closed this task as Resolved.Oct 17 2020, 1:01 PM
Xqt claimed this task.
Xqt mentioned this in T334053: Use only one entry point for all http requests.Apr 5 2023, 7:52 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL