Re: [Mat-annotation-users] Mechanical Turk

[Samuel B. <sa...@mi...>]

Ethan Grant wrote:
> Hello,
>
> I was wondering if anyone had used the MAT tool in conjunction with a
> mechanical turk and if so they could provide some guidance on how to make
> that possible. Ideally I would want respondents to be presented with an
> individual document in need of annotation for each task and have them
> annotate and resubmit the annotated document. I am a very new user to
> Amazon's mechanical turk so any help would be appreciated.
>
> Thanks for the help
>
>
>
> ------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Mat-annotation-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/mat-annotation-users
>

My experience with AMT is that using custom Javascript in each item is 
very, very difficult. We've used the MAT display in presenting elements 
within an AMT item, but you should be aware that asking respondents to 
do span annotation, for instance, is unlikely to be a "good" AMT job, 
since it's very hard to price; it's essentially impossible to forecast 
how much work needs to be done in the annotation. Pretagging and asking 
users to make a decision (is this span a PERSON or not?) is more 
tractable, but it doesn't really require something like MAT; and you 
can't really find spans the pretagger missed.

There's a fairly large literature involving AMT and corpus development. 
There was a workshop at NAACL 2010 on this topic, and there's been a lot 
since. LREC is probably also a good place to look for information.

Good luck - there's a lot to learn. My personal opinion is that for gold 
standard corpora for evaluation, you don't want to use AMT - the 
reliability of Turkers isn't good enough. For model training, it's more 
tractable, and there are domains where it appears that you can Turk the 
training data with the same degree of accuracy, but you need to build 
reduncancy and quality checking into your AMT workflow, and it's very, 
very complicated.

Sam