wekan / Tickets / #113 Support for import from Trello JSON export

Anonymous - 2015-01-23

Originally posted by: alireza-ahmadi

According to Libreboard's Roadmap, Importing data from Trello is already planned for the next release.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-02-24

Originally posted by: fnkr

+1 in particular the import of card attachments (files) is important to me

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-05-21

Originally posted by: mattab

Importing data from Trello would be super awesome :-)

FYI I've developed a little PHP script that exports all data from Trello.com boards (JSON). https://github.com/mattab/trello-backup - I look forward to the day where we can migrate our trello.com boards to this free/libre project 👍

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-12

Originally posted by: xavierpriour

I'll be happy to take a shot at it, if nobody's working on it yet.
@mquandalle is it the right place to discuss UI and implementation? or should this happen on the roadmap board?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-12

Originally posted by: mquandalle

Hi @xavierpriour, this is the right place to discuss the feature UI/UX and implementation. As far as a know (I being the lead dev) no one has started working on it. I would be happy to help you to get this merged in core if you decide to work on it :-)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-13

Originally posted by: xavierpriour

I've just had a deeper look, and here's my analysis:

Exported data

Trello allows to export a full board (at board level: Show menu > More > Print and Export) or a single card (on a card: Share and more > Export JSON).

Import mechanism

Importing self-contained data (card text and title, board labels) should not be an issue, but we face interesting questions when dealing with "shared" data - data that might also be referenced by elements of the wekan instance, like members, or even labels when importing a single card.

I see 3 possibilities:

create data: that's what Trello does for labels when you move a card to another board: its labels (if any) get created in the target board. In that case, we have to be cautious of collisions (if we already have a label with the same name/color, what should happen?).

map data: we find a corresponding, existing data. This can be sophisticated (like, try to map Trello card members to Wekan members based on mail address) or dumb (like, remove all members upon import).

merge: a combination of both: try to map, and if you fail, create on the fly. Of course, this is the most complex scenario

Your thoughts?

UI/UX

As a final target, I'd like to allow the user to pick and choose what to import where. For example, upon opening a "board" file, display data, then let user only select a few lists (or even cards).
But as a first goal, I'll aim to import whole files (card or board). This limits shared data problems
So that would give us two UI to discuss: importing a card, and importing a full board.

For the card, I plan on adding an item to the "List Actions" menu, just above "Archive this list", as this is a pretty infrequent action

For the board, I will add to "My Boards" view, but I'm hesitating: as an option in the "Add a new board" screen, as a button in the header by "Archives", or as a second line under "Add a new board"? What do you think?

Both items would then take you to a file select / text paste screen for the JSON, and perform their magic

Other points

Here's a brain dump for anything left:

Attachments: I put attachments apart, as they are actually exported as URLs to an S3 instance, so importing them means our server has to go and fetch potentially huge amounts of data. We need throttling, monitoring, retry, etc. So I want to keep this separate from the rest

Errors: I'll have minimal error handling for the moment.

Trello link: we have the Trello IDs in the export, so we could store them. I don't really see a use for the time being, and Wekan has no support for arbitrary data, so I'm dumping it.

Next action

Maxime, I'd like to proceed through multiple small PR's instead of a big bang.
If you're ok with it, I'll split it that way:

[x] import a card in a list through copy/pasting, no shared data (= remove members and labels), no attachments

[x] import labels and comments

[x] import a board, no shared data (= remove members, user that imports is set as board admin)

[x] import members

[x] import attachments

[ ] import a card through file upload

Then we can discuss where to go from there: map members, UI to select parts to import, or decide it's "good enough" and move to other features.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-13

Originally posted by: mquandalle

Here are some quick answers to some of the points you raised:

I think it’s best to map board members using their email address. If the email is unknown, we need to call something like inviteUserByEmail('aa@bb.cc'). This function isn’t currently implemented but we need it anyway and could probably re-use some of the code written for another Meteor application (I think Rocket.chat has this feature);

👍 for the UI to import a card

For the UI to import a board I would add a link below the “Create” button of the “Create a board” popup, something like (“or import a board from Trello”)

I think the JSON text copy/pasting will more used than downloading a file from Trello and upgrade it to a Wekan instance, so I think the UI should put the textarea field first.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-14

Originally posted by: xavierpriour

Just finished step 1 as [#358], @mquandalle can you review and tell me if some fixes are needed before I proceed further?

A few more questions that came along the way:

how does Wekan handle errors? Would love if you could point me to a place where you validate a user input and display error (I need this in case of invalid JSON).

I'm wondering if I should really import the whole task history: it might be a lot of work, as the history might reference objects outside the import (other boards, inactive users, etc.), and I don't see a strong use case to support that - I propose to postpone it for the time being. Thoughts?

linked to it, for user mapping I agree with the implicit invite for members (because if they're listed as member, they're probably active), but I think for the rest (users mentioned in comments or history), it would be better to not invite (they may have left). I propose to link them to a "technical" user "Old Trello user" - do you have this kind of users in place?

the code is starting to grow, it does not feel right to leave it stuck in a client/components/lists/ (except for the UI part). Especially as the import function should be IMHO more of a server function than a client one. In a traditional architecture I would have put it in a Controller, but you currently only have models + client components. What do you think we should do? Add a TrelloImport model even if we don't have schema or persistence?

also, history is currently handled implicitly by the models, and I'd like to add an 'import' event. Doesn't feel right to do it straight from the component...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-14

Originally posted by: fnkr

@xavierpriour Importing the whole history is very imporant to me. I use cards to store information and I may need to access that information at a later time.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-14

Originally posted by: xavierpriour

@fnkr I import comments and descriptions, that is all the info the user provides and Trello displays (just give me some time for the attachments).

The history I suggest we dump only include info like: "user A created card on m/d/y", "card was moved from board B on m/d/y", etc. Can you tell me what information this represents for you? Do your lists/boards represent task status (like "done") and you want to keep change dates?

If that's the case, I understand - what do you think we should do with missing pieces, like when mentioning a board we don't have? Or a user?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-14

Originally posted by: fnkr

@xavierpriour Oh, sorry, I thought you mean archived tasks. Loosing historical events would be ok for me.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-19

Originally posted by: xavierpriour

Hi guys, finished the board import - except attachments, members, and file upload.

I'm wondering about those:

I believe uploading a json file is a "nice to have" compared to copy/pasting, so I'm leaning toward not doing it. Anybody strongly wants file upload?

I understand attachments are important for @fnkr but I have no Meteor experience with files download (from Trello to wekan server), so @mquandalle could you point me in the right direction on how to do that?

regarding members invite, as a user I'm a bit worried about a data import potentially sending mail invitations to people without my (and their) permission. So I'd like to implement just a mapping with existing users (with default to current user): that way if you (as a Wekan host) want to get proper members transfer, you first manually add them as users to your Wekan instance, then you do the board import. Not too hard, very explicit, and you have very fine control. All Wwithout the need for complicated import flow and screens. Thoughts?

PS: @mquandalle could you please update the Wekan roadmap board to note the feature is being worked on - and maybe include a link to that Github page?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-19

Originally posted by: mquandalle

I for one don’t think uploading a JSON file is required, I believe copy pasting the JSON code should be enough. If someone has a strong need for it we could implement it later on.

We manage attachments using meteor/collectionFS which is pretty complete and has a great documentation. I haven’t verified but I think the wiki page insert one file from a remote URL is what we want here.

Are you saying that you would like to have an actual “user mapping” screen? What I had in mind was automatic mapping based on the user email address. If we are concerned about sending Wekan invites unexpectedly we could have a second screen saying

Users A, B, and C were not found in this server database. Do you want to invite them?

And I guess have a checkbox next to each user name and a Submit and Dismiss buttons. (BTW I don’t consider this as a requirement to merge [#362].)

I edited the roadmap, thanks for the reminder. Do you have an account on http://try.wekan.io so I could add you on the appropriate card?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-19

Originally posted by: xavierpriour

Hi, I just created a try.wekan.ion account: xavier.priour.

For the user mapping, I'm in favor of the mail mapping too. I'm just suggesting we don't do anything for non-existing accounts instead of doing: import screen > analysis > display list and ask for confirm > invite > manage "invited but not accepted" in the import process.

I'll do a PR with that part soon (hopefully), will probably be a better place to continue the discussion :)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-23

Originally posted by: xavierpriour

Ouch, I'm looking into member mapping now, and we don't have access to Trello members mail.
The only id we have to map is the username - I will match on it automatically, but I doubt success rate will be high. So we need a mapping screen after all :(

Thinking about attachments, the import will potentially trigger many downloads, possibly large ones. When you add possible failures, this screams for an upload manager - @mquandalle do you know if there is anything like that for collectionFS, like there is for meteor-uploads?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-23

Originally posted by: mquandalle

I’m not aware of any but I’ll search for a solution. By “upload manager” do you mean a way to schedule jobs, restart them if they fail, report progress and errors? Should it have any other specific feature?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-23

Originally posted by: mquandalle

Oh yes I remember that I already played with https://github.com/CollectionFS/Meteor-power-queue — but it doesn’t seem maintained anymore unfortunately.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-10-23

Originally posted by: xavierpriour

Just looking for something where the downloads would be queued, and you can view the list and their status, and cancel/retry them. To give users some feedback and control over what happens.

From the description it seems power-queue was ok, too bad it's not maintained. It seems development on collectionFS is slow too: https://github.com/CollectionFS/Meteor-CollectionFS/issues/840

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-11-10

Originally posted by: jadeqwang

I'd like to add a 👍 to importing CSV's as well, which many apps have the option of exporting as, including across to spreadsheets, forms, and Gmail-plugin-CRMs. (Actually, I was trying to figure out how to import a CSV to Wekan today, and realized that the feature didn't exist yet. 😹 )

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-11-10

Originally posted by: xavierpriour

I'm unclear - how would you map a CSV (that is, tabular data) to a Kanban
(list of lists)?

On Tue, Nov 10, 2015 at 10:35 PM, Jade Q Wang notifications@github.com
wrote:

I'd like to add a [image: 👍] to importing CSV's as well, which many
apps have the option of exporting as, including across to spreadsheets,
forms, and Gmail-plugin-CRMs. (Actually, I was trying to figure out how to
import a CSV to Wekan today, and realized that the feature didn't exist
yet. [image: 😹] )

—
Reply to this email directly or view it on GitHub
https://github.com/wekan/wekan/issues/113#issuecomment-155574076.

Xavier Priour

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-11-10

Originally posted by: mquandalle

Let's open another thread for CSV import and keep this one about Trello JSON (which is now implemented thanks to Xavier)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-11-10

Originally posted by: jadeqwang

Thanks @mquandalle, and great question @xavierpriour. I started https://github.com/wekan/wekan/issues/395 to talk about this. :)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-11-14

Originally posted by: xavierpriour

Members import is in (with mapping), only attachments left!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-11-16

Originally posted by: xavierpriour

That's it, attachment import is in [#401]
It's basic, but I believe if we want to do something more sophisticated, it would deserve its own issue.
So @mquandalle, I think you can close that issue :)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2015-12-02

Originally posted by: xavierpriour

Added a page of doc on the wiki: https://github.com/wekan/wekan/wiki/Migrating-from-Trello
Feedback welcome

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Support for import from Trello JSON export

Milestone

Searches

Help

#113 Support for import from Trello JSON export

Discussion

Exported data

Import mechanism

UI/UX

Other points

Next action