Features to move from scripts to framework (tracking)
Open, LowPublic
Actions

Assigned To

None

Authored By

	valhallasw
	Dec 24 2013, 6:39 PM

Description

This bug is to track features that are now in specific scripts, but that should be moved to the framework.

Some examples:

functions in scripts that could be used more broadly (e.g. functions that work on page text could be moved to textlib)
parts of scripts that call the API, or some other low-level layer, directly
scripts that import eachother:

We have quite a few pywikibot scripts in core (https://git.wikimedia.org/tree/pywikibot%2Fcore.git/HEAD/scripts). Some of these scripts import other scripts. That shouldn't happen. These target scripts should probably be split up in a library part and a script part:

grep "from scripts import" *.py
data_ingestion.py:from scripts import upload
flickrripper.py:from scripts import upload
template.py:from scripts import replace

Most notable one is upload.py

Version: core-(2.0)
Severity: normal

Details

Reference: bz58942

	Subject	Repo	Branch	Lines +/-
	[bugfix] Use RangeMessage map xml file from International ISBN Agency	pywikibot/core	master	+6 K -1 K

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T60942 Features to move from scripts to framework (tracking)
Open		None	T61686 fixing_redirects.py: move treat() to pywikibot/textlib
Resolved		AbdealiJK	T67192 move imageuncat's uploadedYesterday to pagegenerators
Resolved		None	T76555 Create LogpagesPageGenerator page generator
Resolved		None	T60944 site.mediawiki_message: support multiple messages
Open		None	T85334 Migrate XML page generators to pagegenerators.py
Resolved		Xqt	T89499 Refactor upload.py / move re-usable parts to framework
Resolved		Xqt	T68561 Add 'upload' method to ImagePage
Resolved		Ladsgroup	T76324 Migrate cosmetic_changes to the core library
Resolved		jayvdb	T89993 remove isbn script dependency in cosmetic_changes
Resolved		jayvdb	T89996 Add isbn package dependency
Resolved		Xqt	T89998 cosmetic changes has no tests
Resolved		Xqt	T104265 Move UserEditFilterGenerator to pagegenerators
Resolved		Xqt	T168336 Refactor unlink.py / move re-usable parts to framework
Open		None	T57032 create "checkpage" option and global "-enable" argument
Declined		None	T194976 Move BaseRevertBot to specialbots.py
Open		None	T266997 Move solve_disambiguation.DisambiguationRobot to specialbots
Resolved	Feature	Xqt	T284388 Move add_text functionality to textlib

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:24 AM

• bzimport added projects: Pywikibot-General, good first task, Tracking-Neverending.

• bzimport set Reference to bz58942.

• bzimport added a subscriber: Unknown Object (????).

valhallasw created this task.Dec 24 2013, 6:39 PM

Aklapper added a project: Pywikibot.Nov 27 2014, 4:20 PM

jayvdb subscribed.Dec 25 2014, 2:58 PM

Liuxinyu970226 subscribed.Dec 26 2014, 6:43 AM

jayvdb added a subtask: T85334: Migrate XML page generators to pagegenerators.py.Dec 26 2014, 6:55 AM

valhallasw merged a task: T89498: Restructure pywikibot scripts to not import other pywikibot scripts.Feb 13 2015, 9:36 PM

valhallasw updated the task description. (Show Details)

valhallasw set Security to None.

valhallasw added subscribers: Aklapper, Multichill.

We have quite a few pywikibot scripts in core (https://git.wikimedia.org/tree/pywikibot%2Fcore.git/HEAD/scripts). Some of these scripts import other scripts. That shouldn't happen. These target scripts should probably be split up in a library part and a script part:

grep "from scripts import" *.py

data_ingestion.py:from scripts import upload
flickrripper.py:from scripts import upload
template.py:from scripts import replace

Most notable one is upload.py

valhallasw mentioned this in T89499: Refactor upload.py / move re-usable parts to framework.Feb 13 2015, 9:40 PM

That is not the full list. See https://gerrit.wikimedia.org/r/#/c/174594/

upload:

data_ingestion
flickrripper
imagetransfer

replace:

image
template

category:

casechecker
cfd

others:

commonscat imports add_text
cosmetic_changes imports isbn
misspelling imports solve_disambiguation
reflinks imports noreferences
nowcommons imports image (which imports replace) and imagetransfer (which imports upload)

And the best of all, ...

pywikibot.page imports scripts.cosmetic_changes (T76324/T76320)

That shouldn't happen.

nowcommons has always imported imagetransfer (since 2006), and has imported image since 2008.

https://mediawiki.org/wiki/Special:Code/pywikipedia/2505
https://mediawiki.org/wiki/Special:Code/pywikipedia/5343

Most of the other script inter-dependencies are of similar vintage.

jayvdb added a subtask: T76324: Migrate cosmetic_changes to the core library.Feb 14 2015, 3:35 AM

jayvdb removed a project: Pywikibot-General.Feb 14 2015, 4:41 AM

Ladsgroup closed subtask T76324: Migrate cosmetic_changes to the core library as Resolved.May 7 2015, 8:20 AM

jayvdb closed subtask T61686: fixing_redirects.py: move treat() to pywikibot/textlib as Resolved.Jun 12 2015, 12:36 PM

jayvdb closed subtask T104265: Move UserEditFilterGenerator to pagegenerators as Resolved.Jul 4 2015, 8:44 AM

Xqt subscribed.Jul 4 2015, 8:56 AM

Nemo_bis renamed this task from [tracking] features to move from scripts to framework to Features to move from scripts to framework (tracking).Jul 19 2015, 9:59 AM

jayvdb removed a project: good first task.Nov 11 2015, 9:50 PM

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 11 2015, 9:50 PM

jayvdb closed subtask T67192: move imageuncat's uploadedYesterday to pagegenerators as Resolved.Mar 7 2016, 10:04 PM

Remaining issues when T89499 would be merged

$ grep -r "from scripts" *
scripts/casechecker.py:from scripts.category import CategoryMoveRobot as CategoryMoveBot
scripts/cfd.py:from scripts.category import CategoryMoveRobot as CategoryMoveBot
scripts/commonscat.py:from scripts.add_text import add_text
scripts/image.py:from scripts.replace import ReplaceRobot as ReplaceBot
scripts/imagecopy.py:from scripts import image
scripts/imagecopy_self.py:from scripts import imagerecat, image
scripts/misspelling.py:from scripts.solve_disambiguation import DisambiguationRobot
scripts/nowcommons.py:from scripts.image import ImageRobot as ImageBot
scripts/panoramiopicker.py:from scripts import imagerecat
scripts/reflinks.py:from scripts import noreferences
scripts/selflink.py:from scripts.unlink import BaseUnlinkBot
scripts/template.py:from scripts.replace import ReplaceRobot as ReplaceBot
scripts/watchlist.py:from scripts.maintenance.cache import CacheEntry

Change 209176 had a related patch set uploaded (by Xqt):
[bugfix] Use RangeMessage map xml file from International ISBN Agency

https://gerrit.wikimedia.org/r/209176

gerritbot added a project: Patch-For-Review.Apr 17 2016, 3:50 PM

Danny_B removed a parent task: T60940: [DO NOT USE] Pywikibot tracking bugs (tracking) [superseded by #Pywikibot-Core].Jul 11 2016, 10:17 PM

Quiddity removed a parent task: T4007: [DO NOT USE] Tracking bug [superseded by #Tracking].Jul 14 2016, 4:19 AM

Xqt reopened subtask T61686: fixing_redirects.py: move treat() to pywikibot/textlib as Open.Aug 24 2016, 3:53 PM

Dalba subscribed.Aug 30 2016, 1:29 PM

Xqt mentioned this in T106121: RFC removing old code / code cleanup.Nov 24 2016, 9:56 AM

Magul closed subtask T89499: Refactor upload.py / move re-usable parts to framework as Resolved.Nov 28 2016, 6:36 PM

Xqt triaged this task as Low priority.Jun 19 2017, 8:04 PM

Xqt added a subtask: T168336: Refactor unlink.py / move re-usable parts to framework.Jun 19 2017, 8:09 PM

Dvorapa subscribed.Jun 25 2017, 2:50 PM

Dvorapa unsubscribed.

Dalba closed subtask T168336: Refactor unlink.py / move re-usable parts to framework as Resolved.Aug 21 2017, 1:55 AM

Change 209176 abandoned by Multichill:
[bugfix] Use RangeMessage map xml file from International ISBN Agency

Reason:
No response. This can always be re-opened if you plan to work on it again.

https://gerrit.wikimedia.org/r/209176

Dvorapa moved this task from Backlog to Tracking tasks on the Pywikibot board.Mar 18 2018, 3:40 PM

Dvorapa removed a project: Patch-For-Review.May 17 2018, 10:25 PM

$ grep -riP "from scripts\.?[^ ]* import" .
./scripts/template.py:from scripts.replace import ReplaceRobot as ReplaceBot
./scripts/nowcommons.py:from scripts.image import ImageRobot as ImageBot
./scripts/cfd.py:from scripts.category import CategoryMoveRobot as CategoryMoveBot
./scripts/casechecker.py:from scripts.category import CategoryMoveRobot as CategoryMoveBot
./scripts/imagecopy_self.py:from scripts import imagerecat, image
./scripts/commonscat.py:from scripts.add_text import add_text
./scripts/misspelling.py:from scripts.solve_disambiguation import DisambiguationRobot
./scripts/imagecopy.py:from scripts.image import ImageRobot
./scripts/image.py:from scripts.replace import ReplaceRobot as ReplaceBot
./scripts/reflinks.py:from scripts import noreferences
./scripts/watchlist.py:from scripts.maintenance.cache import CacheEntry