Page MenuHomePhabricator

Migrate "DrTrigon's tools" (pywikibot/bots/drtrigonbot) bugs from JIRA (TS) to Bugzilla (set up product etc)
Closed, DeclinedPublic

Assigned To
None
Authored By
bzimport
Oct 13 2013, 11:19 AM
Referenced Files
F12462: out
Nov 22 2014, 2:31 AM
F12460: test_python-lua.py
Nov 22 2014, 2:31 AM
F12459: wikipedia.patch
Nov 22 2014, 2:31 AM
F12461: opencv_classify.tar.gz
Nov 22 2014, 2:31 AM
F12456: clean_sandbox_v2.patch
Nov 22 2014, 2:31 AM
F12457: botlist.patch
Nov 22 2014, 2:31 AM
F12458: followup_patch.patch
Nov 22 2014, 2:31 AM
F12454: userlib_commenthidden.patch
Nov 22 2014, 2:31 AM

Description

Author: dr.trigon

Description:
See https://wikitech.wikimedia.org/wiki/DrTrigonBot
and https://jira.toolserver.org/browse/DRTRIGON

According to https://www.mediawiki.org/wiki/Bug_management/Project_Maintainers#To_add_a_project_or_component :


Version: wmf-deployment
Severity: enhancement
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=52692

Details

Reference
bz55673

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:31 AM
bzimport set Reference to bz55673.
bzimport added a subscriber: Unknown Object (MLST).

How does this depend on 52692?

Maybe it's better to put this bot under "Tool Labs tools:"?

dr.trigon wrote:

(In reply to comment #1)

How does this depend on 52692?

...in fact it more like it is "related to" but that does not exist in bugzilla, does it?

Maybe it's better to put this bot under "Tool Labs tools:"?

I don't know - but since DrTrigonBot is related to Pywikibot I think it should be at the same place...

I don't have any expertise in importing from JIRA to Bugzilla (both offer APIs if anybody feels like hacking). Do you have open tickets only in mind, or all tickets?

Above description of project does not describe the project so I can't judge what would be good in this case. :)

If I interpret https://wikitech.wikimedia.org/wiki/DrTrigonBot correctly, the bot will be / is / has been moved from toolserver.org to Tool Labs? In that case this indeed would be one of the tools on tool labs ("Tool Labs tools" product in Bugzilla).

dr.trigon wrote:

(In reply to comment #3)

I don't have any expertise in importing from JIRA to Bugzilla (both offer
APIs
if anybody feels like hacking). Do you have open tickets only in mind, or all
tickets?

I bother about MY ticket - but if this was done all other should not be a problem anymore. As mentioned at [1] there are already approaches for following scheme: JIRA→XML→(adopt data)→XML→bugzilla - look at [2] and [3]. The only thing needed is somebody having some experience with both tools and xml in order to setup the "adopt data middleware".

[1] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Bug_tracker
[2] http://stackoverflow.com/questions/7092559/export-jira-data
[3] http://www.bugzilla.org/docs/3.0/html/api/importxml.html

Above description of project does not describe the project so I can't judge
what would be good in this case. :)

Feel free to confer [4] and [5] - if you still miss something, please let me know?!

[4] https://en.wikipedia.org/wiki/User:DrTrigonBot
[5] https://de.wikipedia.org/wiki/User:DrTrigonBot

As this seems to run on Toollabs, I'm going to set up a new component under the Bugzilla product "Tool labs tools":
https://bugzilla.wikimedia.org/describecomponents.cgi?product=Tool%20labs%20tools

The task of migrating existing bug reports from JIRA to Bugzilla will be left to others.

Ah... So if you really want the components listed under https://jira.toolserver.org/browse/DRTRIGON#selectedTab=com.atlassian.jira.plugin.system.project%3Acomponents-panel
then this needs to be a product in Bugzilla.

As I wrote, I'd prefer DrTrigonBot as a component under the "Tool Labs tools" product, but that would mean that DrTrigonBot would have no subcomponents in Bugzilla. :-/

dr.trigon wrote:

What about creating several components in the product of your choice? E.g.:

  • "DrTrigonBot - General"
  • "DrTrigonBot - catimages"
  • "DrTrigonBot - subster"
  • "DrTrigonBot - web tool scripts" (was formely "toolserver scripts")
  • ("DrTrigonBot - pywikipedia" and "pywikipedia"; are already done as own product called Pywikibot with several components)

dr.trigon wrote:

Ping?! What is the current state here? Can we proceed?

dr.trigon wrote:

As I can see Merlijn has written the migration middle-ware that was missing (Thanks Merlijn!), so can we proceed now? The open parts are:
1.) create component
2.) migrate bugs
3.) (else?)
Thanks a lot and Greetings (enjoy the Holidays! ;)

Andre, the multiple-subcomponents-method looks sane to me. What do you think?

DrTrigon, could you provide:

  1. the names of the subcomponents you want. Am I correct these should be
    • DrTrigonBot - General
    • DrTrigonBot - subster
    • DrTrigonBot - web tool scripts (should this not be 'DrTrigon - web tool scripts'?)
  2. Check whether the following JQL queries map to these projects (via https://jira.toolserver.org/secure/IssueNavigator.jspa )

    General: project = DRTRIGON AND component = "DrTrigonBot - catimages" subster: project = DRTRIGON AND component = "DrTrigonBot - subster" web tool scripts: project = DRTRIGON AND component = "DrTrigonBot - toolserver scripts"
  3. project = DRTRIGON AND component = "DrTrigonBot - pywikipedia" should be imported into pywikibot/compat?
  4. For each of these, should closed bugs be imported?
  5. Under Administration, <your project>, People, add 'Bugzilla Bug Exporter (valhallasw)' to 'Administrators' and click 'Update'.

Please read the above steps carefully and check for any mistakes I may have made.

dr.trigon wrote:

Hello Merlijn!

Thanks for your efforts regarding migration!

(In reply to comment #10)

Andre, the multiple-subcomponents-method looks sane to me. What do you think?

DrTrigon, could you provide:

  1. the names of the subcomponents you want. Am I correct these should be
    • DrTrigonBot - General
    • DrTrigonBot - subster
    • DrTrigonBot - web tool scripts (should this not be 'DrTrigon - web

tool scripts'?)

  • jira: DrTrigonBot - pywikipedia
    • bugzilla: DrTrigonBot - General
    • description: DrTrigonBot code
  • jira: DrTrigonBot - catimages
    • bugzilla: DrTrigonBot - catimages
    • description: DrTrigonBot code; Categorize Images (by content) script
  • jira: DrTrigonBot - subster
    • bugzilla: DrTrigonBot - subster
    • description: DrTrigonBot code; SubsterBot script
  • jira: DrTrigonBot - toolserver scripts
    • bugzilla: DrTrigonBot - web tool scripts
    • description: DrTrigonBot web scripts

You forgot 'DrTrigonBot - catimages'. May be another component 'DrTrigon' under 'Tool Labs tools' should be created as well, could be a good idea...?!

  1. Check whether the following JQL queries map to these projects (via

https://jira.toolserver.org/secure/IssueNavigator.jspa )

General: project = DRTRIGON AND component = "DrTrigonBot - catimages"
subster: project = DRTRIGON AND component = "DrTrigonBot - subster"
web tool scripts: project = DRTRIGON AND component = "DrTrigonBot -
toolserver scripts"

General: project = DRTRIGON AND component = "DrTrigonBot - pywikipedia"
catimages: project = DRTRIGON AND component = "DrTrigonBot - catimages"
subster: project = DRTRIGON AND component = "DrTrigonBot - subster"
web tool scripts: project = DRTRIGON AND component = "DrTrigonBot - toolserver scripts"

You mixed 'Genral' and 'catimages' together, the rest is ok. What if a ticket is in multiple components? Will it then be added multiple times here?

  1. project = DRTRIGON AND component = "DrTrigonBot - pywikipedia" should be imported into pywikibot/compat?

This is wrong, should be: project = DRTRIGON AND component = "pywikipedia"
(The other one is 'General'.)
And it should be imported into 'pywikibot/unspecified' since it might also contain 'core' stuff.

  1. For each of these, should closed bugs be imported?

Depends on whether jira and TS will stay online in read-only mode or something like that... I don't know. Thus it might be more "safe" to import everything including the 'closed' and 'resolved' ones. (please ;) Btw.: They do get imported as "resolved" here, I assume?

  1. Under Administration, <your project>, People, add 'Bugzilla Bug Exporter

(valhallasw)' to 'Administrators' and click 'Update'.

Done.

Please read the above steps carefully and check for any mistakes I may have
made.

Please do the same again with my comment. 4 eyes are better than just 2... ;))

I've created the four components and set DrTrigon as the default assignee: https://bugzilla.wikimedia.org/describecomponents.cgi?product=Tool%20Labs%20tools - if you want any changes please tell me. Thanks!

(In reply to comment #11)

What if a ticket
is in multiple components? Will it then be added multiple times here?

A ticket is only processed once - and it will be in the component first converted (probably catimages - subster - web tool scripts - general)

This is wrong, should be: project = DRTRIGON AND component = "pywikipedia"
(The other one is 'General'.)
And it should be imported into 'pywikibot/unspecified' since it might also
contain 'core' stuff.

But you also specified you wanted those bugs in "DrTrigonBot - General". Shall I import them there, and let you manually move bugs to pywikibot/something if necessary?

Btw.: They do get
imported as "resolved" here, I assume?

Correct.

dr.trigon wrote:

(In reply to comment #13)

(In reply to comment #11)

What if a ticket
is in multiple components? Will it then be added multiple times here?

A ticket is only processed once - and it will be in the component first
converted (probably catimages - subster - web tool scripts - general)

Is is somehow bad... could lead to a lot of confusion later on - they should at least contain a hint about this.

This is wrong, should be: project = DRTRIGON AND component = "pywikipedia"
(The other one is 'General'.)
And it should be imported into 'pywikibot/unspecified' since it might also
contain 'core' stuff.

But you also specified you wanted those bugs in "DrTrigonBot - General".
Shall
I import them there, and let you manually move bugs to pywikibot/something if
necessary?

No, you have to watch out there are 2 different components having "pywikipedia" in their names:

  • project = DRTRIGON AND component = "DrTrigonBot - pywikipedia" goes to 'General'
  • project = DRTRIGON AND component = "pywikipedia" goes to 'pywikibot/unspecified'

so you have to watch out and distinguish between "DrTrigonBot - pywikipedia" and "pywikipedia" (without "DrTrigonBot") - these are 2 components.

Btw.: They do get
imported as "resolved" here, I assume?

Correct.

So... tickets marked as 'closed' do get imported as 'resolved' AND those marked as 'resolved' do get imported as 'resolved', right?

dr.trigon wrote:

...and I have just seen your imports - thanks for the work! - but it seams that the status/es was/were not set correct, see e.g. bug 59403 (jira: close - here: new).

No, that bug was closed in the process of migrating. I'm manually fixing the project = DRTRIGON AND component = "pywikipedia" bugs at the moment.

dr.trigon wrote:

Might it be possible that you missed some (a lot) tickets? Or did you ignore the closed ones? I asked to import them too... I am a little bit confused now... ;)

The closed ones are imported, but you need to tell Bugzilla to list closed bugs, too. Older (>3 months) closed bugs did not get an URL comment added in JIRA.

See https://bugzilla.wikimedia.org/buglist.cgi?list_id=264192&resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=LATER&resolution=DUPLICATE&resolution=WORKSFORME&query_format=advanced&bug_status=RESOLVED&bug_status=CLOSED&component=DrTrigonBot%20-%20catimages&component=DrTrigonBot%20-%20General&component=DrTrigonBot%20-%20subster&component=DrTrigonBot%20-%20web%20tool%20scripts&product=Tool%20Labs%20tools

I think it might actually be better to keep the 'project = DRTRIGON AND component = "pywikipedia"' in DrTrigon - general for now, as they are (at least partially) in German. In any case, all those bugs depend on https://bugzilla.wikimedia.org/59500, which /is/ in pywikibot now.

The full list of bugs that had multiple components is:

DRTRIGON-1 Migrate BUGS from wiki to JIRA
DRTRIGON-2 Migrate FEATURE REQUESTS from wiki to JIRA
DRTRIGON-3 Migrate TODO from wiki to JIRA
DRTRIGON-15 F53 (copied from wiki)
DRTRIGON-16 F54 (copied from wiki)
DRTRIGON-17 F55 (copied from wiki)
DRTRIGON-37 DRTRIGON-20 JIRA account für SVN
DRTRIGON-39 DRTRIGON-20 Cleanup ToDo's
DRTRIGON-42 MailerBot future
DRTRIGON-78 List all changed talk pages of articles under one category
DRTRIGON-81 Germanization 'disk' instead of 'disc' was used in several places...
DRTRIGON-85 Integrate sum_cat_disc as bot
DRTRIGON-87 Bot ERROR mail cannot be sent sometimes
DRTRIGON-96 DRTRIGON-15 'subster.py': Kann eigentlich öffentlich werden, da mit der [2] jeder den Bot jetzt auch konfigurieren kann
DRTRIGON-105 Support for multiple BS template configurations
DRTRIGON-118 Enable lua support to enable even better template programming inclusion (subster)
DRTRIGON-128 DRTRIGON-124 Future improvements as duplicate detection, classification and code clean-up (TD-018, TD-019, TD-020)
DRTRIGON-129 Check to usefulness and capabilities of Labs (vs. TS)
DRTRIGON-131 Reduce bot_control.py influence, try to unlink features

which, I think, is doable to fix by hand.

But it does seem some bugs were missed - strangely enough (eg DRTRIGON-133). I'll re-run the bot to make sure all bugs are there.

And, to finish that part, this is the list of corresponding components:

Key BZ ID Component/s
DRTRIGON-1 #59471 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-2 #59469 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-3 #59468 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-15 #59464 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-16 #59462 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-17 #59460 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-37 #59457 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-39 #59456 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-42 #59450 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-78 #59441 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-81 #59438 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-85 #59431 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-87 #59429 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-96 #59425 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-105 #59421 DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts
DRTRIGON-118 #59403 DrTrigonBot - pywikipedia, DrTrigonBot - subster
DRTRIGON-128 #59372 DrTrigonBot - catimages, DrTrigonBot - pywikipedia
DRTRIGON-129 #59366 DrTrigonBot - catimages, DrTrigonBot - pywikipedia, DrTrigonBot - subster, DrTrigonBot - toolserver scripts
DRTRIGON-131 #59393 DrTrigonBot - pywikipedia, DrTrigonBot - subster

As I said, I think the best option is if you either keep this bug as reference, or move the data to the bugs manually.

dr.trigon wrote:

The number of migrated bugs does still not match, JIRA gives me 133 [1] where as your link (of course the closed are not listed here - thanks! ;) to the bugzilla list [2] gives 119 only - so 14 are still missing...

[1] https://jira.toolserver.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+DRTRIGON
[2] https://bugzilla.wikimedia.org/buglist.cgi?resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=LATER&resolution=DUPLICATE&resolution=WORKSFORME&query_format=advanced&bug_status=RESOLVED&bug_status=CLOSED&component=DrTrigonBot%20-%20catimages&component=DrTrigonBot%20-%20General&component=DrTrigonBot%20-%20subster&component=DrTrigonBot%20-%20web%20tool%20scripts&product=Tool%20Labs%20tools&list_id=264355

Also if you look at this list [3], you can see that DRTRIGON-128 was not migrated (it's missing in [2]).

[3] https://jira.toolserver.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+DRTRIGON+AND+resolution+%3D+Unresolved+ORDER+BY+due+ASC%2C+priority+DESC%2C+created+ASC&mode=hide

So I checked all tickets manually:

  • "DrTrigonBot - pywikipedia", "pywikipedia" as well as the uncategorized DRTRIGON-133 are migrated to 'General' except: DRTRIGON-123 (is in pywikibot as bug 59500), DRTRIGON-18 (is in pywikibot as bug 59585) as well as the whole list given in comment #20 - OK!
  • "DrTrigonBot - catimages" in 'catimages' - PERFECT!
  • "DrTrigonBot - subster" in 'subster' - DRTRIGON-129 is missing (is on list given in comment #20) - OK!
  • "DrTrigonBot - toolserver scripts" in 'web tool scripts' - DRTRIGON-129 is missing (is on list given in comment #20) - OK!
  • DRTRIGON-128 is here as bug 59372 (but not on the list you linked - strange... ;) and was not closed in jira - why that?
  • list given in comment #20: moved all to "DrTrigonBot - General" since this is the most general component covering/containing all others as well (choose if unsure/undetermined) - OK!

Some minor ideas:

  • might be a good idea to migrate the jira original component info (list in comment #20) to the individual tickets as well, at least for future migrations... ;)
  • for the next time try to import first the most general component - that would guarantee that they are properly assigned here - since the most general fits always ;)

So again thanks work for work and effort! Greetings

128 is on the list:

DRTRIGON-128 #59372 DrTrigonBot - catimages, DrTrigonBot - pywikipedia
https://bugzilla.wikimedia.org/show_bug.cgi?id=59372

It's not in your search because it's not a closed bug ;-)

Adding component info is something I will consider if another project also uses components extensively.

dr.trigon wrote:

Merlijn sorry it's getting annoying I know - but I have to REOPEN it. Your import dropped ALL "Issue Links" from jira as well as parts of "Details" (as mentioned before).

I can work without the "Details" even though that is a serious drawback since they are important. But the "Issue Links" are ESSENTIAL, I need all those relations, dependecies, etc.!

Can you please import the missing (meta-)data?

What about "Dates", "History", etc.? As I see you imported primarily the comments, right?

The code used for the conversion is available at https://github.com/wikimedia/pywikibot-sf-export/blob/master/jira.py

At line 207, the existing converted bug is found.

bug.addcomment(....) could be added at line 210 to add a comment to the existing bugs, which could contain any information in issue['fields'] (see e.g. https://jira.toolserver.org/rest/api/2/issue/DRTRIGON-120 )

It also seems attachments were not uploaded -- apparently they don't always show up in the JSON response (see, again, https://jira.toolserver.org/rest/api/2/issue/DRTRIGON-120 vs https://jira.toolserver.org/browse/DRTRIGON-120 )

dr.trigon wrote:

(In reply to comment #24)

The code used for the conversion is available at
https://github.com/wikimedia/pywikibot-sf-export/blob/master/jira.py

At line 207, the existing converted bug is found.

bug.addcomment(....) could be added at line 210 to add a comment to the
existing bugs, which could contain any information in issue['fields'] (see
e.g.
https://jira.toolserver.org/rest/api/2/issue/DRTRIGON-120 )

Thanks for all the info provided! I do not have permission to look at jira API: Unauthorized (401).

It also seems attachments were not uploaded -- apparently they don't always
show up in the JSON response (see, again,
https://jira.toolserver.org/rest/api/2/issue/DRTRIGON-120 vs
https://jira.toolserver.org/browse/DRTRIGON-120 )

I not happy with this... We need to get a solution here. In past I put a lot of effort and time into ticket management on jira, as it was a crucial part of my work-flow/todo-list and docu... just to loose everything now because of a overhasty and non-mature migration process would be a mess!

dr.trigon wrote:

As can be read in [1] the JIRA data can be exported to XML with almost all content. An Example is [2] respective [3].

[1] http://wikimedia.7.x6.nabble.com/What-will-happen-with-the-Toolserver-domain-td5019092.html
[2] https://jira.toolserver.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+%22%5Bmigrated+to+bugzilla%2C+do+not+use%5D+DrTrigon%27s+tools%22+OR+summary+%7E+drtrigon+OR+description+%7E+drtrigon+OR+comment+%7E+drtrigon+OR+environment+%7E+drtrigon+ORDER+BY+key+DESC
[3] https://jira.toolserver.org/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml?jqlQuery=project+%3D+%22[migrated+to+bugzilla%2C+do+not+use]+DrTrigon%27s+tools%22+OR+summary+~+drtrigon+OR+description+~+drtrigon+OR+comment+~+drtrigon+OR+environment+~+drtrigon+ORDER+BY+key+DESC&tempMax=1000

I have now written an XSLT script - a viewer - that imitates [4], you can find it at [5]. An Example concerning this bug here is [6]. The entries can be linked directly as e.g. for DRTRIGON-68 [7] in order to be easily integrated with the bugs and bugzilla here.

[4] https://jira.toolserver.org/sr/jira.issueviews:searchrequest-fullcontent/temp/SearchRequest.html?jqlQuery=project+%3D+%22[migrated+to+bugzilla%2C+do+not+use]+DrTrigon%27s+tools%22+OR+summary+~+drtrigon+OR+description+~+drtrigon+OR+comment+~+drtrigon+OR+environment+~+drtrigon+ORDER+BY+key+DESC&tempMax=1000
[5] http://tools.wmflabs.org/drtrigonbot/cgi-bin/xsalt.py?xslt=jira2html.xslt
[6] http://tools.wmflabs.org/drtrigonbot/cgi-bin/xsalt.py?url=http%3A%2F%2Ftools.wmflabs.org%2Fdrtrigonbot%2FSearchRequest.xml&xslt=jira2html.xslt
[7] http://tools.wmflabs.org/drtrigonbot/cgi-bin/xsalt.py?url=http%3A%2F%2Ftools.wmflabs.org%2Fdrtrigonbot%2FSearchRequest.xml&xslt=jira2html.xslt#DRTRIGON-68

At the moment all links are still pointing to TS JIRA as this can easily be changed after its shutdown. Then it might also be needed to replace the used icons by other ones from commons e.g.

As mentioned in [1], attachments, patches and the like have to be downloaded manually if needed. They can then e.g. be added here as well.

(all this was done thanks to inspiration of http://cocoon.apache.org/2.0/howto/howto-html-pdf-publishing.html and https://de.wikipedia.org/wiki/Apache_Cocoon ;)

dr.trigon wrote:

[DRTRIGON-118] Enable lua support to enable even better template programming inclusion (subster)

http://tools.wmflabs.org/drtrigonbot/cgi-bin/xsalt.py?url=http%3A%2F%2Ftools.wmflabs.org%2Fdrtrigonbot%2FSearchRequest.xml&xslt=jira2html.xslt#DRTRIGON-118

Attached:

(In reply to Andre Klapper from comment #43)

What is left to do here?

--> no answer; closing.