|
From: Samuel B. <sa...@mi...> - 2015-06-16 19:31:03
|
Ethan Grant wrote: > Hello, > > I was wondering if anyone had used the MAT tool in conjunction with a > mechanical turk and if so they could provide some guidance on how to make > that possible. Ideally I would want respondents to be presented with an > individual document in need of annotation for each task and have them > annotate and resubmit the annotated document. I am a very new user to > Amazon's mechanical turk so any help would be appreciated. > > Thanks for the help > > > > ------------------------------------------------------------------------------ > > > > _______________________________________________ > Mat-annotation-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/mat-annotation-users > My experience with AMT is that using custom Javascript in each item is very, very difficult. We've used the MAT display in presenting elements within an AMT item, but you should be aware that asking respondents to do span annotation, for instance, is unlikely to be a "good" AMT job, since it's very hard to price; it's essentially impossible to forecast how much work needs to be done in the annotation. Pretagging and asking users to make a decision (is this span a PERSON or not?) is more tractable, but it doesn't really require something like MAT; and you can't really find spans the pretagger missed. There's a fairly large literature involving AMT and corpus development. There was a workshop at NAACL 2010 on this topic, and there's been a lot since. LREC is probably also a good place to look for information. Good luck - there's a lot to learn. My personal opinion is that for gold standard corpora for evaluation, you don't want to use AMT - the reliability of Turkers isn't good enough. For model training, it's more tractable, and there are domains where it appears that you can Turk the training data with the same degree of accuracy, but you need to build reduncancy and quality checking into your AMT workflow, and it's very, very complicated. Sam |