Page MenuHomePhabricator

Cannot pickle Page objects
Closed, ResolvedPublic

Description

import pickle
import pywikibot

p = pywikibot.Page(pywikibot.Site('de'), 'user:Xqt/Test')
laFile = open('la.data', 'wb')
pickle.dump(p, laFile)
laFile.close()

causes the following error in core whereas it works in compat:

c:\Pywikipedia\core>pwb.py la2
Traceback (most recent call last):

File "C:\Pywikipedia\core\pwb.py", line 143, in <module>
  run_python_file(fn, argv, argvu)
File "C:\Pywikipedia\core\pwb.py", line 67, in run_python_file
  exec(compile(source, filename, "exec"), main_mod.__dict__)
File "C:\Pywikipedia\core\scripts\la2.py", line 9, in <module>
  pickle.dump(p, laFile)
File "C:\Python27\lib\pickle.py", line 1370, in dump
  Pickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 224, in dump
  self.save(obj)
File "C:\Python27\lib\pickle.py", line 331, in save
  self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
  self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 663, in _batch_setitems
  save(v)
File "C:\Python27\lib\pickle.py", line 331, in save
  self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
  self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 663, in _batch_setitems
  save(v)
File "C:\Python27\lib\pickle.py", line 331, in save
  self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
  self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 663, in _batch_setitems
  save(v)
File "C:\Python27\lib\pickle.py", line 306, in save
  rv = reduce(self.proto)
File "C:\Python27\lib\copy_reg.py", line 70, in _reduce_ex
  raise TypeError, "can't pickle %s objects" % base.__name__

TypeError: can't pickle lock objects
<type 'exceptions.TypeError'>
CRITICAL: Waiting for 1 network thread(s) to finish. Press ctrl-c to abort

c:\Pywikipedia\core>


Version: core-(2.0)
Severity: major

Details

Reference
bz62094

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:06 AM
bzimport set Reference to bz62094.
bzimport added a subscriber: Unknown Object (????).

The reason for this is because Site objects are not pickle-able.

import pywikibot, pickle
s=pywikibot.Site()
f=pickle.dumps(s)

Traceback (most recent call last):
[same traceback here]

File "/home/km/python/lib64/python2.7/copy_reg.py", line 70, in _reduce_ex
  raise TypeError, "can't pickle %s objects" % base.__name__

TypeError: can't pickle lock objects

Change 131032 had a related patch set uploaded by Legoktm:
Remove unused APISite.sitelock

https://gerrit.wikimedia.org/r/131032

Change 131033 had a related patch set uploaded by Legoktm:
Make Site objects pickle-able

https://gerrit.wikimedia.org/r/131033

Change 131032 merged by jenkins-bot:
Remove unused APISite.sitelock

https://gerrit.wikimedia.org/r/131032

That bug is still there after the patch.

Yes:

"There's still a lock in BaseSite, but this gets us closer to
being able to pickle Pages."

Change 131033 merged by jenkins-bot:
Make Site objects pickle-able

https://gerrit.wikimedia.org/r/131033

Change 144698 had a related patch set uploaded by Legoktm:
Page pickling test

https://gerrit.wikimedia.org/r/144698

Change 144698 merged by jenkins-bot:
Page pickling test

https://gerrit.wikimedia.org/r/144698

Still waiting for review of the unit test, but the bugs were all solved days ago.