#321 testing for MW compatibility

[Lee Worden]

If I want to be able to announce that WW is compatible with MW 1.19+ by the time I go to WikiSym, I need a testing strategy to make sure it's true so I don't embarrass us.

At the bare minimum, test a representative spectrum of WW functions in both 1.19 and 1.22alpha.

[Lee Worden]

I really need an at least semi-formalized testing protocol for all of WW/PE like, last year. Without it we're going to be playing whack-a-mole with bugs forever.

Fortunately, for what it's worth, for this I only need testing for WW, since the WW/PE split ensures that PE, its makefiles, etc., don't have any direct interaction with MediaWiki. So I don't need to test the makefiles, ability to pull from git repos, etc. against each version of MW.

What do I need to test?

• Importing files (see [#299]).
• Exporting (a cursory test ought to do it, since PE does the work).
• All the different ways to display project files: source highlighting, makefile highlighting with links, images, SVG images, html with/without the full-page treatment, wikitext including wikitext with project files embedded (probably wise to test wikitext files with wikitext files in them, actually), math, links and download links
  • including testing that iframe.js is doing whatever it's for <- it shrinks the iframe from its initial large height to the height of the contents.
• Check the different shortcuts for math: $$...$$, {$...$}, <latex>
  • make sure mathjax works
• Syncing source files from both text and File: pages
• Automatic adding, moving, deleting source/project files while editing
• all the API actions (this takes care of the git remote, getprojectfile, etc.)
• Cache-invalidating pages. Making sure project files get made and archived at the right time (for example, no archiving during preview; make targets after saving edits, not during).
• Preview: check that preview pages link to preview version of ManageProject, that background-make links are disabled, that preview session number is displayed.
• Test submitting and displaying background jobs, cursory ought to do it
• Test ManageProject (what tests?)
  • make sure all the parts display right; test that the actions work, including the ones that ask for confirmation
  • make sure the target of the "special page" tab is right when the page has been called with an action to perform.
  • make sure the javascript works: page suggestions, what else?
• Tests for GetProjectFile?
  • tests for file displaying take care of most of it. Make sure the tab is right when there's an action, that the delete tab is there (in the pulldown in newer MWs), that directory listing pages are correct and the buttons work
• Make sure the WW-specific preferences items are there
• Make sure project link is in the sidebar when it should be

(I just looked through the WW directory, all the functions in WWInterface.php, and all the bug reports in the workingwiki category of the bug tracker, and this list seems pretty good to me.)

[Lee Worden]

Note syncing files from File: pages into PE currently doesn't work in MW 1.21.

[Lee Worden]

So actually, the bare minimum should be 1.19 and 1.21, not 1.22alpha, unless I think 1.22 is about to be released soon. I should try to find out when it's likely to be released.

[Lee Worden]

https://www.mediawiki.org/wiki/MediaWiki_1.22:

The 1.22.0 stable release is expected to come out late in 2013.

So I should be developing on a test-mediawiki-1.21 wiki, not the test-mediawiki-core wiki that I've been using, which is running 1.22alpha.

[Lee Worden]

http://lalashan.mcmaster.ca/test-mediawiki-1.21 wiki is now up and running (in private).

[Lee Worden]

Tests in 1.21:

• importing seems okay
• exporting works
• displaying files works
  • discovered an infinite-recursion bug when a wikitext file includes itself. fixed.
  • wikitext file with math in it displays right embedded in its page, but not in GetProjectFiles: fails to compile the math.
• math/latex shortcuts work, mathjax works
• adding and removing files by editing the page seems okay in general.
  • things I added while previewing persist as "missing" after I save. aren't they supposed to be removed at some point?
    • Okay, they get removed when I sync. Presumably they would have gotten removed when I displayed the page, if it did any making.
• Syncing both kinds of files works.
• How to test all the API actions? Write a test-api script?
  • the git script fails uninformatively when it doesn't have good login/password
  • or maybe that's not why it's doing that, some api methods are failing in both new and 1.13 wikis
  • bin/test-ww-api script is written, soon to be added to WW repo
• preview is fine
• background jobs are fine, including the little box
• ManageProject is fine
  • except that delete action is missing
• GetProjectFile is fine
  • except that the Special Page tab takes you to the project's directory listing, which is kind of random. Better it should take you to the same thing you're looking at.
• Preferences are fine
• projects box is fine (though it could use a redesign in general)

Summary of issues to be resolved:

• API partly broken
• delete action on ManageProject
• tab on GetProjectFile
• wikitext math

API and maybe delete action should be fixed immediately, others are just garden-variety bugs. Make sure they're in the tracker, get to them when it's their time.

[Lee Worden]

• api broken: [#351]
• delete on MP: [#365]
• GPF tab: [#217]
• math in wikitext: [#364]

[Lee Worden]

Testing on 1.19:

• IPF seems okay
• export okay
• file displaying all good
  • except for "leave this page" bug ([#277])
• math ok, including mathjax
• sync is ok
• automatic adding and deleting is fine
• api known to be broken
• cache invalidation: to be tested
• background stuff is good
• MP seems fine
  • delete tab is there in this version
• GPF is same as in 1.21, with tab bug
  • 'upload' link when displaying a source file still goes to IPF, but no longer has the filename/page preloaded, though it does have the project. This is a bug, though not a show-stopper.
• preferences A-OK
• project link is ok
  • I notice it's missing from File: pages where it ought to be. I think it's never been there, but that could be fixed.

Summary of issues:

• "are you sure you want to leave this page" when clicking a link that just downloads a file without leaving edit page. [#277]
• API broken [#351]
• upload link doesn't preload [#366]
• project link missing from File: pages [#367]

I think #277 and #351 are blockers for 1.19 compatibility, the others not.

[Lee Worden]

So I've fixed the API [#351] and "leave this page" [#277]. I've also fixed the failure to make when the project is first created [#368] and made the 3 added buttons on the edit form load way quicker, both of which were annoying issues.

This leaves:

• delete tab on ManageProject [#365]
• upload link doesn't preload [#366]
• project link missing from File: pages [#367]
• GPF tab: [#217]
• math in wikitext: [#364]

All of which are respectable bugs, but not necessary for use of WW. So I'm now declaring WW compatible with MediaWiki 1.19 and 1.21, and closing this ticket, though it'll continue to be useful. In the future I'll probably move the list of tests to a wiki page and post the link here.