#294 ResourceLoader and Special:MultiUpload

[Lee Worden]

I'm having some trouble getting my javascript to get called the way it should as I'm working on a Special:MultiUpload page (see [#258]). It's complex enough that I want to create a ticket to organize my problem-solving process.

[Lee Worden]

Special:MultiUpload is for MW1.19+ only, and is not currently active even on theobio/worden (my 1.19 test wiki on theobio), only on my private within-laptop wiki. Anyway, what's happening there is that I refactored the javascript in MW's skins/common/upload.js - which is the 'mediawiki.legacy.upload' module in RL - to allow a sort of ad-hoc subclassing in the MultiUpload page, and then I created an ext.workingwiki.multiupload.js script (known to RL as ext.workingwiki.multiupload). It uses RL's dependency feature correctly as far as I can tell:

   $resourceModuleTemplate = array(
      'localBasePath' => dirname(__FILE__).'/resources',
   );

   $wgResourceModules['ext.workingwiki.multiupload'] = $resourceModuleTemplate + array(
      'scripts' => array( 'ext.workingwiki.multiupload.js' ),
      'dependencies' => array(
              'mediawiki.util',
              'mediawiki.special.upload',
              'mediawiki.legacy.upload',
      ),
   );

This doesn't work with ?debug=true, but that's okay - I've been hacking around that in various ways which are fine but the real problem isn't that. It's that ext.workingwiki.multiupload.js gets loaded before its dependencies and never gets evaluated.

Loading with debug=false, we get a RL-generated load script that starts with a call

mw.loader.implement("ext.workingwiki.multiupload",function($){(function(){ [...] }());;},{},{});

where that function is the contents of ext.workingwiki.multiupload.js. Using Firebug to step through the execution of mw.loader.implement() line-by-line, I find that it doesn't call execute(), because it only calls execute() if all the module's dependencies are loaded. In this case mediawiki.special.upload and mediawiki.legacy.upload have not yet been loaded, as each of those is represented in a call to mw.loader.implement() later in the same script.

Unfortunately, it seems that the code in ext.workingwiki.multiupload.js is never called later, either. So it doesn't run in the page. I've added a huge syntax error to my code, and it doesn't interfere with the page at all, and doesn't show up as an error message in the Firebug console.

Update: now that I'm walking through all this systematically, it is getting called! It is calling my function later, not when it does implement("ext.workingwiki.multiupload", ...), not when it does implement("mediawiki.legacy.upload", ...), but when it does implement("mediawiki.special.upload", ...), i.e. as soon as the dependencies are satisfied. Nothing like trying to write up a bug! Let me try it again with debug=true.

[Lee Worden]

OK so with debug=false, the ResourceLoader collects all the JS, CSS and some other stuff and compresses them down into a small number of files, to cut down on bandwidth and especially latency (which is tied to the number of requests). That's what it's for. Makes it hard to debug though, because all that code is kind of mangled and has very few line breaks. With debug=true, it falls back to leaving them unmangled and serving them separately, so that you have a chance of stepping around in the debugger and figuring out what's going on.

So the default assumption when you add a file (aka a 'module') is that it can be served directly, i.e. that it has a URL. This would be true if it were stored somewhere in http://whatever/extensions/WorkingWiki/. However, I try to support keeping the WorkingWiki code directory in other places as well as in extensions/, since it's a good practice to keep stuff outside the website directories, and it's convenient to keep it in the same place as the working files, which should be outside the website directories, and in a farm like ours you want to have it in a single shared location, not a copy in each wiki's extensions/.

To date I've had WW's JS, CSS, and image resources in the ProjectEngine resources directory, which allows them to have Special:GetProjectFile URLs, so they can be accessed that way. This is kind of wrong though, because they're properly part of WW, not PE, and we're supposed to be able to separate WW from PE and customize each independently. And anyway, with the ResourceLoader we don't need to make the resources directly accessible. So I'd like to leave my code where it is and get RL to do the right thing. I see at least two ways to do that:

1. Have them in ProjectEngine/resources/ and provide GetProjectFile URLs to load them while debugging.
2. Have them in WorkingWiki/resources/, and get RL to serve them directly in debug mode, as it does in non-debug mode, so that they don't need to have a URL.

I've tried both of these, and I guess I need to try them again more systematically.

The first method is done by changing the above definition of $resourceModuleTemplate to

    $resourceModuleTemplate = array(
      'localBasePath' => dirname(__FILE__).'/resources',
      'remoteBasePath' => 
        str_replace( '$1', 
          "Special:GetProjectFile?resources=1&display=raw&filename=js", 
          $wgArticlePath ),
    );

This 'remoteBasePath' option provides a partial URL to which the module's filename is appended with a joining slash, producing a full URL that retrieves the file, assuming it's in the proper location in ProjectEngine/resources. This requires the assignment statements for $resourceModuleTemplate and $wgResourceModules to be placed in a callback function in $wgExtensionFunctions[] so that it will be evaluated after $wgArticlePath is set up - executing them when WorkingWiki.php is evaluated is too early.

I tried the second method by creating a class that works just like ResourceLoaderFileModule except that it refuses to provide a URL. This seems like it ought to suffice to get RL to inline its code in debug mode.

class ResourceLoaderOfflineFileModule extends ResourceLoaderFileModule {
    public function supportsURLLoading() {
            return false;
    }
};

$resourceModuleTemplate = array(
    'localBasePath' => dirname(__FILE__).'/resources',
    'class' => 'ResourceLoaderOfflineFileModule',
);

When I did this, it worked insofar as using the custom class and putting the module's code in the load script in place of a URL, but the code didn't execute. I'll try again now, making sure to do all the steps carefully.

[Lee Worden]

OK, I'm retrying the ResourceLoaderOfflineFileModule technique. The assignment code is in a callback function, though the class definition isn't. That ought to be okay.

Indeed, in this case, calling via a url with debug=true, I have a load script that begins with

mw.loader.implement("ext.workingwiki.multiupload", function( $ ) {( function() {
 [...] 
}() );

;
}, {}, {});

as before (except in this case the code within the inner function is not 'minified') - and as far as I can tell the code is never called. Also it loads upload.js, but it never seems to load mediawiki.special.upload.js - which come to think of it might be the real problem, because it isn't expected to execute the multiupload code until both of those are loaded. Maybe I should look into that.

[Lee Worden]

Within the one load script are the three relevant lines:

mw.loader.implement("ext.workingwiki.multiupload", function( $ ) {( function() {
 [...]
}() );
;
}, {}, {}); 
[...]
mw.loader.implement("mediawiki.legacy.upload", ["http://localhost/wiki/skins/common/upload.js"], {}, {});
[...]
mw.loader.implement("mediawiki.special.upload", ["http://localhost/wiki/resources/mediawiki.special/mediawiki.special.upload.js"], {}, {"widthheight":"$1 \u00d7 $2","size-bytes":"$1 B","size-kilobytes":"$1 KB","size-megabytes":"$1 MB","size-gigabytes":"$1 GB","largefileserver":"This file is bigger than the server is configured to allow."});

So what happens between the upload.js and the mediawiki.special.upload.js? Or what?

[Lee Worden]

mw.loader.implement("mediawiki.special.upload", ...) is called, but doesn't call execute(), because one of mediawiki.special.upload's dependencies, "mediawiki.libs.jpegmeta", is not loaded. Where is it?

Well, it's the second-to-last implement call before this one. So are we waiting for it?

mediawiki.libs.jpegmeta was retrieved from the server. It has no dependencies, it's loaded asynchronously at the time of its implement() by sticking its URL in a <script> tag at the end of the DOM. The "net" panel in Firebug shows that mediawiki.libs.jpegmeta.js is loaded from the server simultaneously with upload.js. We go on to do an implement() call for mediawiki.page.ready, and then one for mediawiki.special.upload. In the call for mediawiki.special.upload, it has two dependencies - mediawiki.util and mediawiki.libs.jpegmeta; it finds that it has state 'ready' for mediawiki.util but not for mediawiki.libs.jpegmeta, and it requires both to be ready so it doesn't call execute() for mediawiki.special.upload.

So why isn't jpegmeta ready? Will have to come back to this a bit later.

[Lee Worden]

(Say, didn't I post another reply before this one?)

This is tiresome. I should at least try it with an extensions/WorkingWiki URL like a normal person and see what it does then. But as long as I'm in the middle of this thing, with all these breakpoints set...

Here is the breaks I'm getting:

• markModuleReady() on "user.options"
  • handlePending() on "user.options", which has nothing pending, i.e. nothing in "jobs".
• markModuleReady() and handlePending() on "user.tokens", "jquery.client", "jquery.cookie", "jquery.messageBox", some other stuff, "mediawiki.util".
  • in the mediawiki.util case, handlePending() calls execute() on "mediawiki.page.startup", which triggers a nested call to markModuleReady() and handlePending() for that, which does more nesting:
    • mediawiki.legacy.wikibits
      • mediawiki.legacy.ajax: in handlePending() for this, it finds something in jobs and takes it off the list. What was it?

Now I get out of that and the next stop is in the onload handler for DOM-appended scripts. Which seems to be the one that's called after loading upload.js. Maybe because my breakpoint only catches on that one?

That thing calls markModuleReady() on upload.js, or rather, on "mediawiki.legacy.upload". Maybe more than once? I thought I pressed run and it stopped in the same place again, but it might have been just a flicker or something. That passes to handlePending(), which doesn't call anything. From there, no more breakpoints.

I think I didn't set enough breakpoints. Apparently I need to set a breakpoint for each instance of that onload function.

[Lee Worden]

I guess the key thing is the one for the jpegmeta module.

[Lee Worden]

OK, now with

    $resourceModuleTemplate = array(
      'localBasePath' => dirname(__FILE__).'/resources',
      'remoteExtPath' => 'WorkingWiki/resources',
    );

    $wgResourceModules['ext.workingwiki.multiupload'] = $resourceModuleTemplate + array(
      'scripts' => array( 'ext.workingwiki.multiupload.js' ),
      'dependencies' => array(
              'mediawiki.util',
              'mediawiki.special.upload',
              'mediawiki.legacy.upload',
      ),
    );

and http://localhost/wiki/extensions/WorkingWiki/resources/ext.workingwiki.multiupload.js is now a symbolic link to the file.

Does it help? Nope. According to the Net panel in firebug, the last 3 thing loaded are upload.js, mediawiki.libs.jpegmeta.js, and jquery.byteLimit.js, which where did that one come from? The last 4 lines in the last load script are

mw.loader.implement("mediawiki.legacy.upload", ["http://localhost/wiki/skins/common/upload.js"], {}, {});
mw.loader.implement("mediawiki.libs.jpegmeta", ["http://localhost/wiki/resources/mediawiki.libs/mediawiki.libs.jpegmeta.js"], {}, {});
mw.loader.implement("mediawiki.page.ready", ["http://localhost/wiki/resources/mediawiki.page/mediawiki.page.ready.js"], {}, {});
mw.loader.implement("mediawiki.special.upload", ["http://localhost/wiki/resources/mediawiki.special/mediawiki.special.upload.js"], {}, {"widthheight":"$1 \u00d7 $2","size-bytes":"$1 B","size-kilobytes":"$1 KB","size-megabytes":"$1 MB","size-gigabytes":"$1 GB","largefileserver":"This file is bigger than the server is configured to allow."});

So it doesn't seem like my odd no-URL customization is really the source of this problem, it's something else. Back to tracing what happens when/if jpegmeta loads, I guess.

[Lee Worden]

OK, I have a breakpoint at the mw.loader.implement("mediawiki.libs.jpegmeta", ...) and one at the next line after that one. I plan to go into the former implement call and get a breakpoint on that onload function at the point in the code where it gets defined on the fly.

Module mediawiki.libs.jpegmeta has no dependencies. Therefore we go ahead into execute() and into addScript(), which does the DOM-script-onload thing. It says there's already a breakpoint in the onload script. I switched it off and back on to make sure.

The script gets loaded. I hit the breakpoint at mw.loader.implement("mediawiki.page.ready", ...). That module has missing dependencies, so no load. Likewise the next one, mediawiki.special.upload.

And then, no break.

In the DOM inspector, in window.document.body.children, I find the script element for the jpegmeta script. It has an onreadystatechange attribute, which is the function I expect it to be. Why is it not called?

Also, why doesn't the inspector show an onload attribute? The code sets both to the same function.

[Lee Worden]

It might be a bug in the ResourceLoader :(. I doubt it, though, because it seems like kind of a glaring one and serious hackers have been using that code for a while now. Even if it is a bug in there, it must be triggered by me doing something weird.

It does look like there has definitely been some action in RL's code since 1.19, so maybe I should try on the latest MW code and see if it works there.

I can also try removing the RL code from my special page so that it only does the calls that Special:Upload does, and make sure that works.

[Lee Worden]

Well, it didn't fix itself while I was sleeping :(

I think I'm going to make an MRE before I try it in MW 1.21 or 1.22.

[Lee Worden]

I've created a Special:Null page that does nothing, and I'm going to slowly add RL code until I get it to break or do something else useful.

Here (in the attachment) is what JS resources get loaded when it's really null, i.e. what SpecialPage loads by default.

[Lee Worden]

Now I add a simple module and verify that it works as intended.

I create a file resources/ext.workingwiki.null.js with a big syntax error in it, and give it an entry in $wgResourceModules, with no dependencies. I call for it in SpecialNull. I put a link in the wiki's extensions directory.

Now the module loads and reports an error message, as it should, both in debug mode and out.

So now I add the 3 dependencies that the other module uses.

In this case, it loads all these things in the proper order and reports the error.

So I guess I'll keep importing parts of SpecialMultiUpload until it breaks.

Making SpecialNull a subclass of SpecialUpload causes it to redirect to Special:Upload, but a little fix to __construct() takes care of that. It's still doing the error message, i.e. calling my js code.

And now putting the addModules() call into a stubbed-out version of an HTMLForm subclass, as it is in MultiUpload - this also works.

Hmm, now I switch it to call the ext.workingwiki.multiupload module instead of ext.workingwiki.null, and that works too... (it also has a syntax error in it)

Now I switch the names, so the new code is called SpecialMultiUpload. Still seems all right. I'll just keep bringing in bits of code from the non-working version until something breaks or doesn't...

OOh, there's a weird error where it's creating objects from the UploadFormRow class defined in SpecialCustomUpload (my old upload code) when it should be using the UploadFormRow class in the new code. Maybe that'll fix it. ... but actually I don't think so because I think it's only doing that when UploadFormRow isn't defined in SpecialMultiUpload, which only happened briefly there while I was assembling it bit by bit.

... well, I don't know - I brought everything over and it's working okay... guess I'll just go on with the development...

[Lee Worden]

Even going back to using that weird ResourceLoaderOfflineFileModule class it works, and I don't need to put the files in the extensions/ directory. What a mystery!

[Lee Worden]

Aarrrgh now it says "Browser does not allow XMLHttpRequest calls to localhost" so I guess I have to do all my testing on lalashan. It's a good thing there's nobody nearby for me to strangle. Just venting.

[Lee Worden]

Which means I have to transfer my modifications to mediawiki-1.19 to lalashan somehow - they're in a local git project. And I don't want to push them to the mediawiki server. Still just venting.

[Lee Worden]

the experimental code is on lalashan/theobio/worden now, and the AJAX file-duplication notification is working!