#472 Upload and unpack zip and tar files

[Lee Worden]

Special:ImportFiles used to do that, now nothing does. Time to reimplement that.

[Lee Worden]

http://stackoverflow.com/questions/166221/how-can-i-upload-files-asynchronously-with-jquery

[Lee Worden]

experimented with jquery.ajaxfileupload.js, linked from the above. Seems pretty good, but it seems to let me set up a file selector to upload directly using ajax but not let me switch it back to normal file selecting again afterward.

But anyway the code inside it's not very complex, and it's similar to what I did to download pdf files while previewing pages. So I think I'll just steal some of its code and handle it myself.

[Lee Worden]

Overview:

• User selects a .zip file or whatever; WW pops up a question asking whether you want to upload the file as is or unpack it. Assuming you want to unpack it:
• Immediately send the .zip file to a special API method on the server, which will unpack it, stash the files obtained, and send back info about all the files.
• Stuff the import form with all those files' info, so the user can say what to do with all of them. Replace each file selector by a placeholder for the stashed file.
• Add some kind of close button on the files, so you can get rid of the ones you don't want before importing. [#476]

Requires me to:

• implement asynchronous file uploading (what I've been discussing in earlier comments). Looks pretty doable. Reliable error handling might be a challenge.
• implement unpack API action. Crib from MW's upload API action, which hopefully exemplifies how to receive file contents in form data into an API action.
• extend the ImportProjectFiles to handle those stashed files, when they're submitted. Which should be easy.

[Anonymous]

Got the first step going, by helping myself to some code from UploadWizard: immediately uploading the .zip file to the server, storing it in a hidden "stash" location there, and keeping track of where it is. (Notes on this step are at http://lalashan.mcmaster.ca/theobio/projects/index.php/WorkingWiki/ImportWizard/Chunks.) Consequences of this choice are that you'll have to have UploadWizard installed in order to have this feature, and as a corollary that you have to have MW 1.20+, since that's what UploadWizard needs.

To do:

• unzip it and stash all the components
• finish import of each of them separately.

Also:

• switch from using a popup when a .zip file is selected, to providing an "unpack" control, in case the user decides to unpack it sometime after selecting it.

[Lee Worden]

And how to do step 2: unzip it and stash all the components?

First question: given the 'sessionkey' that indexes the stashed file, how to find its actual path, to give to unzip.

Second question: once it's unzipped, how to put all those files into stashed locations.

[Lee Worden]

given the 'sessionkey' that indexes the stashed file, how to find its actual path, to give to unzip.

Well, when ImportProjectFiles reads an uploaded file and inserts it into a text page, it uses $this->mUpload->getTempPath().

But here I don't have an mUpload... where does that come from and how do I get one?

SpecialUpload creates it using UploadBase::createFromRequest( $request ), which I can do using a fake request, I think. In the request I think I'll just need 'wpSourceType' => 'Stash' and 'wpSessionKey' => the key. Let's try that.

... ok, that doesn't seem to work. Looking into it.

[Lee Worden]

... ok, that does work - I had a typo in 'wpSourceType'.

But when I ask it for getTempPath() it gives me a name like /tmp/localcopy_44068b8478b9-1.zip - very plausible, but isn't actually there!

Where's the file's actual path?

ImportProjectFiles uses getTempPath(), and it does seem to work (I just tested it) - does it for sure work when a file is stashed?

...

... sure enough, it's actually pretty tricky to test that case, because my usual test for stashed files is to generate an error that a File: upload already exists on the wiki - so it doesn't generally happen with text page destinations. I got it to generate a warning and then inject the stashed file into a text page, and sure enough, getTempPath() doesn't give the right path for stashed files.

[Lee Worden]

So it works to do

    $upload = UploadBase::createFromRequest( $uploadRequest );
    $localfile = $upload->stashFile( $this->getUser() );
    $filename = $localfile->getLocalRefPath();

but the problem is calling stashFile() makes another copy of the zip file... that's not good.

But I don't see any way to get a path from UploadFromStash without stashFile(). Can I bypass use of UploadFromStash? What does it use?

• it creates an UploadStash object and calls its getMetadata( $key ). That's probably pretty helpful - let's explore that.

[Lee Worden]

... that gives me, among other things, a path beginning with "mwrepo://local/temp/", which has to be resolved to "/var/www/whatever/images/temp/".

OK, think I've got it with

    $repo = RepoGroup::singleton()->getLocalRepo();
    $stash = new UploadStash( $repo );
    $metadata = $stash->getMetadata( $sessionkey );
    $file = $repo->getLocalReference( $metadata['us_path'] );
    $filename = $file->getPath();

(and have confirmed that after doing this, we only have one copy of the zip file in the images/temp directory)

[Lee Worden]

Unzipping, stashing, and giving keys to the client is all working now.

On to step 3 - populating the import form and finishing the import. This means (to me):

• remove the package file from the form and replace it with all the package contents. Each unpacked file gets its own "row" in the form. There won't be a file selector, only fields to assign its destination.
• that's not hard, but to work well, I need:
  • on the fly renumbering of rows, so I can insert these things while there might be other rows afterward
  • a control to delete rows you don't want, since a .zip file may well contain extra junk
  • protection from overloading the server's submission size limit with a large combination of small files
    • though actually, maybe this can wait, since the files from a package will already be on the server and won't inflate the submission size...

[Lee Worden]

I think fixing the row numbering stuff is a good move for the future - it's pretty ramshackle right now, and piling on more hacks will make it a lot worse. So I'll make some notes, and I trust it'll quickly become clear how to fix it up.

How it is now:

• The HTML page arrives with a row 1 and a 'template' row.
  • in non-JS clients, the template row is hidden by CSS and remains empty, so is ignored.
  • in JS clients, the whole form is hidden and replaced by a spinner, while the JS code spirits away the template for future use, collapses row 1, and then makes the form visible with only the collapsed row 1.
• Form inputs need to have unique names, to provide unique names for the different submitted values. Each field's name is a combination of a basic name and a row number: for instance, the file selectors have names like 'wpUploadFile1' - 'wpUploadFile' is the name used in Special:Upload.
• It's unwieldy to find things by those compound names, so I also give them class names: for instance each file selector also has 'wpUploadFile' as a class name.
• Each row is an HTML fieldset, and I use those fieldsets to operate on the form. So for instance, I can find a given row's fieldset, and then operate on its file selector by doing a search on the file selector's class name: $fieldset.find( ':input.wpUploadFile' ).do_something().
• Each fieldset also knows what row number it is, stored as a row-index data value attached to the fieldset using jQuery's data storage features.
  • code in importprojectfiles.js seems to be looking for 'rowIndex' data. I should check whether that's causing buggy behavior or what. It actually seems to be working (it's in auto-updating the destination fields when the source changes), so maybe that data isn't so important.

What things I need to do:

• setupAnimation : for a given fieldset, figure out which fields to hide when collapsed, set up events to collapse and expand it
• setupRow : given only a number i, find the ith fieldset, assign its 'row-index' data, set up the auto-fill actions updating later fields from earlier ones, set it up to display thumbnail images.
• findLastFieldset : find the form, get its last row as a fieldset
• addRow : stick a new row either at the end or after a given fieldset. Give it the appropriate number, and maybe renumber any later rows.
• stuffRow : given an existing row, give it specific field values (such as no file selector, and a hidden stashed-file key)
  • this needs a way to figure out the row's index, to put it into the inputs' names.
  • or maybe not - maybe I can throw inputs around, get the names wrong, and then do a general "renumberRows" that will fix all the names and ids.
• need a way to find i'th row
• uploadSetupByIds is given id values of a bunch of elements in a single fieldset, and sets them up. It's inherited from SpecialUpload, so I don't want to reengineer it.
• need a way to make sure i'th row says "File i" in the fieldset legend.

How about:

• in general, when updating things within a row, you find inputs to work on using their class names.
• you find row i by finding the ith fieldset in the form.
• when needed, do a big renumber and make sure all ids and names in the ith row have 'i' in them.
  • to make this work, I'll give each input a 'data-name-base' attribute to record the name without any appended 'i'. Labels also have a 'data-htmlFor-base' so I can get the 'htmlFor' attribute right with appended 'i'.
  • at this point, also update the legend to "File i".
• I may not actually need data( 'row-index' ).
• Looks like I might want to call uploadSetupByIds more than once - better check if that'll be okay.

[Lee Worden]

uploadSetupByIds is all written in plain javascript, without jQuery, so it uses e.g. element.onchange = whatever, not $( element ).change( whatever ), and it looks like it'll be completely harmless to call more than once on the same elements.

[Lee Worden]

It's unpacking the zip file into form rows.

Not right yet:

• No thumbnail
• No delete-row button
• Row numbers are out of order
• Destination page isn't auto-filling.

Also, it's suggesting that .zip and .exe files should go on text pages for some reason, which is wrong. (It's looking in file contents for '\0' as a mark of a binary file, but the JS can't do that when the file's stashed on the server. So does it need to recognize the suffixes? Or is the server prepared to say if they're binary? Have to look into that.)

[Lee Worden]

Also, it's suggesting that .zip and .exe files should go on text pages for some reason, which is wrong. (It's looking in file contents for '\0' as a mark of a binary file, but the JS can't do that when the file's stashed on the server. So does it need to recognize the suffixes? Or is the server prepared to say if they're binary? Have to look into that.)

Looks like the server-side code is doing this check of '\0' in file contents, but the JS side isn't. I may or may not want to implement that right now, but if not I should at least make sure it does the right thing with .zip and the other file extensions.

[Lee Worden]

If the project name is filled in for the .zip file, I should make sure when I replace the package by its content files that they all get that project name as well.

[Lee Worden]

Problems with versions.

I'm using the UploadWizard extension for its smart uploading code. I'm not seeing a 1.21 version of UW in the git repo - only one for the latest MW and one for 1.20. I was testing on the latest version of UW, but I just found incompatibilities. So I might have to require a MW upgrade to 1.22 or 1.23 to use the .zip feature, or take a copy of UW's uploading code into WW, which may then gradually lose compatibility with MW.

[Lee Worden]

I have extracted the relevant file from UploadWizard into WW's directory, so we don't need to have UploadWizard installed. It should be easier to manage multiple MW versions that way. It may even allow enabling the zip file feature in MW 1.19, though we won't be able to use chunked uploading for big files in that case.

Currently:

• it seems to be unpacking the package into form rows correctly.
• Project name given for the .zip file is given to all its content files
• no close button yet
• importing files from the stash into source-file tags not working - I think because of the same getTempPath() issue that I had in other code.

[Lee Worden]

It does seem to be the same getTempPath() issue, but it needs a different fix.

By the time I want to paste the stashed text file's contents into a source-file tag, I am a subclass of UploadRow which has an mUpload member created by UploadBase::createFromRequest(). In this case it's a thing representing a stashed file, but in other cases it'll be a file that was just directly uploaded and is sitting in /tmp/ or something.

In the latter case, $this->mUpload->getTempFile() works, but in the stashed case it doesn't, and there doesn't seem to be a good way to get the full path from mUpload. For stashed files, I may have to go back to the POST parameters, get the session key, and go through the pathway with the Repo object that I used to find the zip file for unpacking.

...

update: seems to be an issue with cached filesystem data in php, actually. Using clearstatcache() makes it appear that the file from getTempFile() is actually usable.

So I don't have to work around it. I'll review the .zip code and see if I can use it there as well.

[Lee Worden]

... the .zip code would be a bit simpler written using UploadBase, but I don't think it's worth the trouble of rewriting it, so I'll leave it as is.

[Lee Worden]

Appears to work in 1.19, 1.21, 1.22, and MW master branch. Except that in 1.19 it doesn't delete the uploaded package file from the /tmp directory.

[Lee Worden]

close button is working.

I've merged all this code to trunk, and deployed it to lalashan.

Outstanding issues for the record, though I'm okay with leaving them as far as this ticket is concerned:

• no thumbnail for unpacked image.
  • thumbnail does show up if you submit and the image garners a warning
• It leaves files in /tmp on 1.19
• It may leave files in the stash when you don't upload them - I should remove them
• 'undefined' sometimes shows up when it's trying to retrieve image info. (There's a separate ticket for that.)