mat-annotation-users — For users of the MITRE Annotation Toolkit

[Mat-annotation-users] Announcing the release of version 3.3 of the MITRE Annotation Toolkit

[Samuel B. <sa...@mi...>]

All -

Version 3.3 of the MITRE Annotation Toolkit has been released. MAT 3.3 is a feature enhancement release, the first version fully compatible with Python 3. The release also includes full support for mixed-initiative sentence classification annotation using the jCarafe maximum entropy classifier.

The MAT 3.3 documentation is included in the download and available at

http://mat-annotation.sf.net/current_docs/html/index.html

(Firefox recommended)

Download available at

https://sf.net/projects/mat-annotation/files/

As usual, visit http://mat-annotation.sf.net for updates and more
information.

Cheers,
Sam Bayer
The MITRE Corporation

[Mat-annotation-users] Announcing the release of version 3.1.1 of the MITRE Annotation Toolkit

[Samuel B. <sa...@mi...>]

All -

Version 3.1.1 of the MITRE Annotation Toolkit has been released. This 
release is a bugfix release which addresses a host of tiny bugs in 
version 3.1. MAT 3.1 introduced crossvalidation in the experiment 
engine; a much more streamlined way of declaring annotation sets; the 
ability to infer tasks in the UI from annotated documents; and 
significant UI enhancements, including a guided mode for filling in 
annotation popups and much better management of overlapping annotations. 
For additional details, view the documentation and select "Upgrade and 
release notes" and "Version history" from the documentation sidebar.

The MAT 3.1.1 documentation is included in the download and available at

http://mat-annotation.sf.net/current_docs/html/index.html

(Firefox recommended)

Download available at

http://sf.net/projects/mat-annotation/files/

As usual, visit http://mat-annotation.sf.net for updates and more 
information.

Cheers,
Sam Bayer
The MITRE Corporation

Re: [Mat-annotation-users] 27.3994, Software: Computational Linguistics; Text/Corpus Linguistics: MITRE Annotation Toolkit 3.1

[Alexandre R. <ara...@gm...>]

Thank you so much for this detailed answer. I will let Claudia and Livy decide if it is worth to try.

Best,

--
Alexandre Rademaker
http://arademaker.github.io <http://arademaker.github.io/>


> On 6 Dec 2016, at 13:01, Samuel Bayer <sa...@mi...> wrote:
> 
> It's been a long time since someone begged me to use MAT :-).
> 
> There's no problem with the expressiveness of MAT's document model or annotation scheme - with the exception of discontinuous spans, which MAT doesn't do, it's at least as expressive as brat in every way. There's also no problem with viewing dependencies in the annotation table. My concerns are several:
> 
> - First, it's pretty clumsy to associate the document sentences with the relevant annotations in the table. I suppose you could get around this by having each sentence be a separate document.
> 
> - Second, I'm surprised that your annotators prefer to review dependencies in a table - when I've reviewed dependency annotations, I've found the arcs in the palette to be crucial to my figuring out what was going on. But to each his or her own, I suppose.
> 
> - Third, I think you'd find that editing the annotations would be extremely clumsy. If you don't care about doing that, this isn't an issue.
> 
> There's no on-line demo. But I can promise you that installing MAT is trivial, and if you want to check it out, by all means do so. However, you will need to spend some time to design your annotation task and convert your data into something MAT can read.
> 
> Cheers,
> Sam
> 

Re: [Mat-annotation-users] 27.3994, Software: Computational Linguistics; Text/Corpus Linguistics: MITRE Annotation Toolkit 3.1

[Samuel B. <sa...@mi...>]

It's been a long time since someone begged me to use MAT :-).

There's no problem with the expressiveness of MAT's document model or 
annotation scheme - with the exception of discontinuous spans, which MAT 
doesn't do, it's at least as expressive as brat in every way. There's 
also no problem with viewing dependencies in the annotation table. My 
concerns are several:

- First, it's pretty clumsy to associate the document sentences with the 
relevant annotations in the table. I suppose you could get around this 
by having each sentence be a separate document.

- Second, I'm surprised that your annotators prefer to review 
dependencies in a table - when I've reviewed dependency annotations, 
I've found the arcs in the palette to be crucial to my figuring out what 
was going on. But to each his or her own, I suppose.

- Third, I think you'd find that editing the annotations would be 
extremely clumsy. If you don't care about doing that, this isn't an issue.

There's no on-line demo. But I can promise you that installing MAT is 
trivial, and if you want to check it out, by all means do so. However, 
you will need to spend some time to design your annotation task and 
convert your data into something MAT can read.

Cheers,
Sam

Alexandre Rademaker wrote:
> Hi Sam,
>
>
> I am curious to understand why you described the annotation of dependencies in MAT as a likely ugly experience. If I understood right, MAT was designed for annotation of NE and relations between them. But we can take the POS as entity types and the relationships between then as the dependencies, right? The Brat tool takes this approach in making any span of text a possible entity that can receive key/values annotations and can be linked to other entities via relations; that is why Brat can be used for NE and relations but also for dependencies and POS annotations.
>
> Reading the documentation (http://mat-annotation.sourceforge.net/current_docs/html/index.html) I saw that you could import/export in a JSON format. Moreover, the relations are not drawn in the sentence with arrows, but displayed as a table in the bottom of the screen; the funny part is that our team of linguists actually prefer to read the dependencies in the table format. Moreover, we would not annotate the documents from scratch, our use case is to review and correct dependencies.
>
> Is there any online demo? I would like to give MAT a chance?! ;-)
>
> Cheers,
>
> --
> Alexandre Rademaker
> http://arademaker.github.io <http://arademaker.github.io/>
> http://researcher.ibm.com/person/br-alexrad <http://researcher.ibm.com/person/br-alexrad>
>
>
>
>> On 5 Dec 2016, at 20:39, Livy Real <liv...@gm...> wrote:
>>
>> Hi Livy -
>>
>> Thanks for writing, and thanks for your interest in the toolkit. Alas, doing dependency annotation in MAT would be an ugly, ugly thing. I've wanted for quite a while to have a custom annotation palette that would support this, but I've had neither the opportunity or the resources to do it.
>>
>> If you do find a tool that suits your needs, I'd love to hear about it. One tool you might look into is brat (http://brat.nlplab.org <http://brat.nlplab.org/>). It's more appropriate than MAT because it supports drawing arcs directly on its annotation palette.
>>
>> Cordially,
>> Sam Bayer
>> The MITRE Corporation
>>
>> On Sun, Oct 9, 2016 at 10:25 AM, Alexandre Rademaker <ara...@gm... <mailto:ara...@gm...>> wrote:
>>
>>
>> Sent from my iPhone
>>
>> Begin forwarded message:
>>
>>> *****************    LINGUIST List Support    *****************
>>>                        Fund Drive 2016
>>>                    25 years of LINGUIST List!
>>> Please support the LL editors and operation with a donation at:
>>>            http://funddrive.linguistlist.org/donate/ <http://funddrive.linguistlist.org/donate/>
>>>
>>> Editor for this issue: Amanda Foster <am...@li... <mailto:am...@li...>>
>>> ================================================================
>>>
>>>
>>> Date: Fri, 07 Oct 2016 11:05:06
>>> From: Samuel Bayer [sa...@mi... <mailto:sa...@mi...>]
>>> Subject: Computational Linguistics; Text/Corpus Linguistics: MITRE Annotation Toolkit 3.1
>>>
>>>
>>> All -
>>>
>>> My colleagues and I are pleased to announce the availability of version 3.1 of
>>> the MITRE Annotation Toolkit.
>>>
>>> - What is it?
>>>
>>> MAT is a suite of cooperating, loosely-coupled tools for supporting the human
>>> annotator in constructing models and corpora for computational linguistics and
>>> analysis, especially using the iterative 'tag a little, learn a little'
>>> workflow. MAT includes MITRE's open source Java Carafe CRF trainer/tagger; a
>>> Web-based annotation and process control UI; a flexible and powerful scorer;
>>> an XML-configurable experiment harness; a workflow manager; and more.
>>>
>>> MAT is platform-independent, open source, and compatible (with some effort)
>>> with multiple trainer/tagger engines and hand annotation tools.
>>>
>>> You can learn about MAT 3.1, and download it, at
>>>
>>> http://mat-annotation.sf.net <http://mat-annotation.sf.net/>
>>>
>>> - What's new in version 3.1?
>>>
>>> This release introduces crossvalidation in the experiment engine; a much more
>>> streamlined way of declaring annotation sets; the ability to infer tasks in
>>> the UI from annotated documents; and significant UI enhancements, including a
>>> guided mode for filling in annotation popups and much better management of
>>> overlapping annotations. For additional details, visit the documentation at
>>>
>>> http://mat-annotation.sf.net/current_docs/html/index.html <http://mat-annotation.sf.net/current_docs/html/index.html>
>>>
>>> and select ''Upgrade and release notes'' from the documentation sidebar.
>>>
>>> If you have MAT 3.0, MAT 3.1 is completely backward compatible. If you have
>>> MAT 2.0, there are some conversions you may have to perform. See the release
>>> notes for details.
>>>
>>> - How stable is it?
>>>
>>> For many years, MAT has served as the underpinning of MITRE's Identification
>>> Scrubber Toolkit (MIST), an open-source package for deidentification of
>>> free-text medical documents. As MIST, it has been used by a wide range of
>>> researchers in this area. MAT itself has been used extensively by MITRE in its
>>> internal research and development projects, and by a variety of MITRE's
>>> sponsors.
>>>
>>> - Is it supported?
>>>
>>> Unfortunately, no. MITRE doesn't (currently) have the resources to provide
>>> public open-source support. There is a public mailing list on Sourceforge,
>>> which the MITRE developers subscribe to, but we likely won't have the
>>> resources to answer.
>>>
>>> On the other hand, MAT has been under development for almost ten years, and
>>> has seen a good deal of use, and comes with a ton of documentation.
>>>
>>> If you download MAT, and you like it, please let us know.
>>>
>>> Cordially,
>>> Sam Bayer
>>> The MITRE Corporation
>>> sa...@mi... <mailto:sa...@mi...>
>>>
>
>

Re: [Mat-annotation-users] MITRE Annotation Toolkit

[Livy R. <liv...@gm...>]

hi Sam,

thank you for your quick answer!

We are currently using brat, but wanting to try other tools as well.

I hope my team has time to try MITRE tool kit and if so, I certainly write
you back.

Sincerely,
Livy

On Mon, Dec 5, 2016 at 8:07 PM, Samuel Bayer <sa...@mi...> wrote:

> Hi Livy -
>
> Thanks for writing, and thanks for your interest in the toolkit. Alas,
> doing dependency annotation in MAT would be an ugly, ugly thing. I've
> wanted for quite a while to have a custom annotation palette that would
> support this, but I've had neither the opportunity or the resources to do
> it.
>
> If you do find a tool that suits your needs, I'd love to hear about it.
> One tool you might look into is brat (http://brat.nlplab.org). It's more
> appropriate than MAT because it supports drawing arcs directly on its
> annotation palette.
>
> Cordially,
> Sam Bayer
> The MITRE Corporation
>
>
> Livy Real wrote:
>
>> Hi Sam,
>>
>> I hope this e-mail finds you well.
>>
>> I'm Livy Real, linguist, and I'm currently working on dependency
>> annotation
>> in Brazilian Portuguese.
>>
>> I'm searching for a tool with an easy interface for our team of
>> linguistics
>> and then I found Mitre announcement at Linguist List.
>>
>> In your online documentation, I found three examples of tasks, all of them
>> related to Named Entities. Is it possible to use Mitre Annotation Toolkit
>> for dependencies?
>>
>> Also you said that it is necessary to specify a language for the task.
>> What
>> would be the difficulties for working on Portuguese texts?
>>
>> Thank you in advance,
>>
>> Livy Real
>>
>>
>

Re: [Mat-annotation-users] MITRE Annotation Toolkit

[Samuel B. <sa...@mi...>]

Hi Livy -

Thanks for writing, and thanks for your interest in the toolkit. Alas, 
doing dependency annotation in MAT would be an ugly, ugly thing. I've 
wanted for quite a while to have a custom annotation palette that would 
support this, but I've had neither the opportunity or the resources to 
do it.

If you do find a tool that suits your needs, I'd love to hear about it. 
One tool you might look into is brat (http://brat.nlplab.org). It's more 
appropriate than MAT because it supports drawing arcs directly on its 
annotation palette.

Cordially,
Sam Bayer
The MITRE Corporation

Livy Real wrote:
> Hi Sam,
>
> I hope this e-mail finds you well.
>
> I'm Livy Real, linguist, and I'm currently working on dependency annotation
> in Brazilian Portuguese.
>
> I'm searching for a tool with an easy interface for our team of linguistics
> and then I found Mitre announcement at Linguist List.
>
> In your online documentation, I found three examples of tasks, all of them
> related to Named Entities. Is it possible to use Mitre Annotation Toolkit
> for dependencies?
>
> Also you said that it is necessary to specify a language for the task. What
> would be the difficulties for working on Portuguese texts?
>
> Thank you in advance,
>
> Livy Real
>

Re: [Mat-annotation-users] Mechanical Turk

[Samuel B. <sa...@mi...>]

Ethan Grant wrote:
> Hello,
>
> I was wondering if anyone had used the MAT tool in conjunction with a
> mechanical turk and if so they could provide some guidance on how to make
> that possible. Ideally I would want respondents to be presented with an
> individual document in need of annotation for each task and have them
> annotate and resubmit the annotated document. I am a very new user to
> Amazon's mechanical turk so any help would be appreciated.
>
> Thanks for the help
>
>
>
> ------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Mat-annotation-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/mat-annotation-users
>

My experience with AMT is that using custom Javascript in each item is 
very, very difficult. We've used the MAT display in presenting elements 
within an AMT item, but you should be aware that asking respondents to 
do span annotation, for instance, is unlikely to be a "good" AMT job, 
since it's very hard to price; it's essentially impossible to forecast 
how much work needs to be done in the annotation. Pretagging and asking 
users to make a decision (is this span a PERSON or not?) is more 
tractable, but it doesn't really require something like MAT; and you 
can't really find spans the pretagger missed.

There's a fairly large literature involving AMT and corpus development. 
There was a workshop at NAACL 2010 on this topic, and there's been a lot 
since. LREC is probably also a good place to look for information.

Good luck - there's a lot to learn. My personal opinion is that for gold 
standard corpora for evaluation, you don't want to use AMT - the 
reliability of Turkers isn't good enough. For model training, it's more 
tractable, and there are domains where it appears that you can Turk the 
training data with the same degree of accuracy, but you need to build 
reduncancy and quality checking into your AMT workflow, and it's very, 
very complicated.

Sam

[Mat-annotation-users] Mechanical Turk

[Ethan G. <et...@te...>]

Hello,

I was wondering if anyone had used the MAT tool in conjunction with a
mechanical turk and if so they could provide some guidance on how to make
that possible. Ideally I would want respondents to be presented with an
individual document in need of annotation for each task and have them
annotate and resubmit the annotated document. I am a very new user to
Amazon's mechanical turk so any help would be appreciated.

Thanks for the help

[Mat-annotation-users] Announcing the availability of the MITRE Annotation Toolkit, version 3.0.5

[Samuel B. <sa...@mi...>]

All -

My colleagues and I are pleased to announce the availability of version
3.0.5 of the MITRE Annotation Toolkit.

What is it?
-----------

MAT is a suite of cooperating, loosely-coupled tools for supporting the
human annotator in constructing models and corpora for computational
linguistics and analysis, especially using the iterative "tag a little,
learn a little" workflow. MAT includes MITRE's open source Java Carafe 
CRF trainer/tagger; a Web-based annotation and process control UI; a 
flexible and powerful scorer; an XML-configurable experiment harness; a 
workflow manager; and more.

MAT is platform-independent, open source, and compatible (with some
effort) with multiple trainer/tagger engines and hand annotation tools.

You can learn about MAT 3.0, and download it, at

http://mat-annotation.sf.net

What's new in version 3.0?
--------------------------

MAT 3.0 provides significant improvements for maintainers and annotators 
of complex multi-step annotation tasks, such as those requirement 
multiple mixed-initiative annotation steps. It also features a general 
and powerful hand reconciliation tool which works with relation and 
event annotations.

How stable is it?
-----------------

For many years, MAT has served as the underpinning of MITRE's
Identification Scrubber Toolkit (MIST), an open-source package for
deidentification of free-text medical documents. As MIST, it has been 
used by a wide range of researchers in this area. While the features of 
MAT 3.0 have not been exploited in MIST, and there is no 3.0 version of 
MIST, MAT itself continues to be used extensively by MITRE in its 
internal research and development projects, and by a variety of MITRE's 
sponsors.

Is it supported?
----------------

Unfortunately, no. MITRE doesn't (currently) have the resources to 
provide public open-source support. The MITRE developers subscribe to 
this mailing list, but we likely won't have the resources to answer.

On the other hand, MAT was under development for almost eight years, and 
has seen a good deal of use, and comes with a ton of documentation.

Future work
-----------

This is the terminal version of MAT. We may occasionally provide bug-fix 
releases, but no major additions are planned.

Cordially,
Sam Bayer
The MITRE Corporation
sa...@mi...

Re: [Mat-annotation-users] MAT UI

[Samuel B. <sa...@mi...>]

No one's done that.

Andrew Caines wrote:
> Hi,
> Did anyone ever try to change / succeed in changing the default action on
> click of an existing tag, from 'replace' to 'add', in the MAT UI? Thereby
> giving you the option to add a number of tags to a token just by clicks
> rather than swipes?
> thanks, Andrew
>
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
>
>
>
> _______________________________________________
> Mat-annotation-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/mat-annotation-users
>

[Mat-annotation-users] MAT UI

[Andrew C. <and...@gm...>]

Hi,
Did anyone ever try to change / succeed in changing the default action on
click of an existing tag, from 'replace' to 'add', in the MAT UI? Thereby
giving you the option to add a number of tags to a token just by clicks
rather than swipes?
thanks, Andrew