Recent changes to Holing_System

WikiPage Holing_System modified by Martin Riedl

Martin Riedl — Fri, 11 Jan 2013 09:40:28 -0000

--- v11
+++ v12
@@ -77,20 +77,5 @@
 relation: nsubj2 (denoting that the hole is at the second position)

 ##Example Code
-Within the package example code is given (src/test/java/jobimtext/holing/example/Pipeline) to extract ngrams, with a hole and output them in a format that can be directly used for the Hadoop calculations of the Distributional Thesaurus:
+see [jobimtext_programming]

-
-
-     String path = "src/test/resources";
-     String pattern = "test.txt";
-     CollectionReader reader = createCollectionReader(TextReader.class, TextReader.PARAM_PATH, path, TextReader.PARAM_PATTERNS, new String[] { "[+]" + pattern }, TextReader.PARAM_LANGUAGE, "en");
-     AnalysisEngine tokenizer = createPrimitive(OpenNlpSegmenter.class);
-     AnalysisEngine ngram = createPrimitive(NgramAnnotator.class, NgramAnnotator.PARAM_HOLE, 3, NgramAnnotator.PARAM_NGRAM, 4);
-     AnalysisEngine pseudoPosTagger = createPrimitive(PseudoPosTagger.class);
-     AnalysisEngine out = createPrimitive(OutputHadoopRelations.class);
-     AnalysisEngine out2 = createPrimitive(PosTaggedHadoopOutput.class);
-     SimplePipeline.runPipeline(reader, tokenizer, pseudoPosTagger, ngram, out, out2);
-
-
-Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP](http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims in this case using a 4-gram with a hole at position 3. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.
-

WikiPage Holing_System modified by Martin Riedl

Martin Riedl — Thu, 10 Jan 2013 11:43:56 -0000

--- v10
+++ v11
@@ -56,7 +56,7 @@
 * put several holes, like A(B,C,D) --> (B,D) - A(@@,C,@@)

-
+

 #Package
 A package with UIMA types, to model and extract relations of the @@operation is located at:
@@ -94,4 +94,3 @@

 Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP](http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims in this case using a 4-gram with a hole at position 3. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.

-

WikiPage Holing_System modified by Martin Riedl

Martin Riedl — Thu, 10 Jan 2013 11:43:00 -0000

--- v9
+++ v10
@@ -56,6 +56,7 @@
 * put several holes, like A(B,C,D) --> (B,D) - A(@@,C,@@)

+

 #Package
 A package with UIMA types, to model and extract relations of the @@operation is located at:
@@ -93,3 +94,4 @@

 Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP](http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims in this case using a 4-gram with a hole at position 3. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.

+

WikiPage Holing_System modified by Martin Riedl

Martin Riedl — Tue, 13 Nov 2012 11:23:08 -0000

--- v8
+++ v9
@@ -82,10 +82,7 @@
 
      String path = "src/test/resources";
      String pattern = "test.txt";
-     CollectionReader reader = createCollectionReader(TextReader.class,
-                        TextReader.PARAM_PATH, path, TextReader.PARAM_PATTERNS,
-			new String[] { "[+]" + pattern }, TextReader.PARAM_LANGUAGE,
-				"en");
+     CollectionReader reader = createCollectionReader(TextReader.class, TextReader.PARAM_PATH, path, TextReader.PARAM_PATTERNS, new String[] { "[+]" + pattern }, TextReader.PARAM_LANGUAGE, "en");
      AnalysisEngine tokenizer = createPrimitive(OpenNlpSegmenter.class);
      AnalysisEngine ngram = createPrimitive(NgramAnnotator.class, NgramAnnotator.PARAM_HOLE, 3, NgramAnnotator.PARAM_NGRAM, 4);
      AnalysisEngine pseudoPosTagger = createPrimitive(PseudoPosTagger.class);

WikiPage Holing_System modified by Martin Riedl

Martin Riedl — Tue, 13 Nov 2012 11:22:05 -0000

--- v7
+++ v8
@@ -77,23 +77,21 @@
 
 ##Example Code
 Within the package example code is given (src/test/java/jobimtext/holing/example/Pipeline) to extract ngrams, with a hole and output them in a format that can be directly used for the Hadoop calculations of the Distributional Thesaurus:
-:::java
-String path = "src/test/resources";
-		String pattern = "test.txt";
-		CollectionReader reader = createCollectionReader(TextReader.class,
-				TextReader.PARAM_PATH, path, TextReader.PARAM_PATTERNS,
-				new String[] { "[+]" + pattern }, TextReader.PARAM_LANGUAGE,
+
+
+
+     String path = "src/test/resources";
+     String pattern = "test.txt";
+     CollectionReader reader = createCollectionReader(TextReader.class,
+                        TextReader.PARAM_PATH, path, TextReader.PARAM_PATTERNS,
+			new String[] { "[+]" + pattern }, TextReader.PARAM_LANGUAGE,
 				"en");
-
-		
-		AnalysisEngine tokenizer = createPrimitive(OpenNlpSegmenter.class);
-		AnalysisEngine ngram = createPrimitive(NgramAnnotator.class,
-				NgramAnnotator.PARAM_HOLE, 3, NgramAnnotator.PARAM_NGRAM, 4);
-		AnalysisEngine pseudoPosTagger = createPrimitive(PseudoPosTagger.class);
-		AnalysisEngine out = createPrimitive(OutputHadoopRelations.class);
-		AnalysisEngine out2 = createPrimitive(PosTaggedHadoopOutput.class);
-		SimplePipeline.runPipeline(reader, tokenizer, pseudoPosTagger, ngram,
-				out, out2);
+     AnalysisEngine tokenizer = createPrimitive(OpenNlpSegmenter.class);
+     AnalysisEngine ngram = createPrimitive(NgramAnnotator.class, NgramAnnotator.PARAM_HOLE, 3, NgramAnnotator.PARAM_NGRAM, 4);
+     AnalysisEngine pseudoPosTagger = createPrimitive(PseudoPosTagger.class);
+     AnalysisEngine out = createPrimitive(OutputHadoopRelations.class);
+     AnalysisEngine out2 = createPrimitive(PosTaggedHadoopOutput.class);
+     SimplePipeline.runPipeline(reader, tokenizer, pseudoPosTagger, ngram, out, out2);
 
 
 Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP](http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims in this case using a 4-gram with a hole at position 3. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.

WikiPage Holing_System modified by Martin Riedl

Martin Riedl — Fri, 02 Nov 2012 10:19:43 -0000

--- v6
+++ v7
@@ -77,7 +77,7 @@
 
 ##Example Code
 Within the package example code is given (src/test/java/jobimtext/holing/example/Pipeline) to extract ngrams, with a hole and output them in a format that can be directly used for the Hadoop calculations of the Distributional Thesaurus:
-~~~~~~
+:::java
 String path = "src/test/resources";
 		String pattern = "test.txt";
 		CollectionReader reader = createCollectionReader(TextReader.class,
@@ -94,6 +94,7 @@
 		AnalysisEngine out2 = createPrimitive(PosTaggedHadoopOutput.class);
 		SimplePipeline.runPipeline(reader, tokenizer, pseudoPosTagger, ngram,
 				out, out2);
-~~~~~~
-Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP](http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.
 
+
+Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP](http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims in this case using a 4-gram with a hole at position 3. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.
+

WikiPage Holing_System modified by Martin Riedl

Martin Riedl — Fri, 02 Nov 2012 10:18:29 -0000

--- v5
+++ v6
@@ -1,4 +1,3 @@
-
 The @@ operation: From Text to Pairs
 =======
 This page describes the general idea of the @@ operation (pronounce: holing operation). This operation is used to split a structural observation on text into two parts, which could be thought of as word and context, or word and feature. These parts are called *Jo* and *Bim*, as they are not distinguishable in terms of type in the gereral case, but have to be distinguished in order to describe them.
@@ -96,5 +95,5 @@
 		SimplePipeline.runPipeline(reader, tokenizer, pseudoPosTagger, ngram,
 				out, out2);
 ~~~~~~
-Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP] (http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.
+Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP](http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.

WikiPage Holing_System modified by Martin Riedl

Martin Riedl — Fri, 02 Nov 2012 10:17:56 -0000

--- v4
+++ v5
@@ -1,10 +1,12 @@
+
 The @@ operation: From Text to Pairs
 =======
-
 This page describes the general idea of the @@ operation (pronounce: holing operation). This operation is used to split a structural observation on text into two parts, which could be thought of as word and context, or word and feature. These parts are called *Jo* and *Bim*, as they are not distinguishable in terms of type in the gereral case, but have to be distinguished in order to describe them.
 All the observations are subject to the @@ operation. These are written out, and serve as the input for the [Distributional Similarity with MapReduce](Distributional_Similarity_with_MapReduce) computation.
 
-Example:
+[TOC]
+
+#Example:
 -----
 
 Consider the following sentence:
@@ -41,7 +43,7 @@
 
 The count of 1 indicates that we see these pairs a single time in this sentence. Longer texts will produce higher pair counts. 
 
-Variations of the @@ operation
+#Variations of the @@ operation
 ------
 Context definition: Instead of dependency parses, we can use any kind of structure on text, including but not limited to:
 
@@ -49,9 +51,50 @@
 * positional co-occurrence
 * dependency chains
 
-@@ variants: It is possible to:
+#@@ variants: It is possible to:
 
 * subsume several parts of the observation into a single hole, e.g. to handle multiwords or to keep pairs together A(B,C) --> (B,C) - A(@@)
 * put several holes, like A(B,C,D) --> (B,D) - A(@@,C,@@)
 
 
+
+#Package
+A package with UIMA types, to model and extract relations of the @@operation is located at:
+Package: jobimtext.holing jobimtext.holing.type
+
+##The JoBim datastructure
+The relations have been structured using the JoBim UIMA type, which is a Annotation and has 3 attributes: 
+* Jo: an annotation which is named key
+* Bim: a list of annotations which are named values
+* relation: the name of the relation so we identify what relation we have
+
+
+
+Example:
+The relation I -- nsubj(suffered, @@) could be transformed into:
+Jo: I
+Bim: suffered
+relation: nsubj2 (denoting that the hole is at the second position)
+
+##Example Code
+Within the package example code is given (src/test/java/jobimtext/holing/example/Pipeline) to extract ngrams, with a hole and output them in a format that can be directly used for the Hadoop calculations of the Distributional Thesaurus:
+~~~~~~
+String path = "src/test/resources";
+		String pattern = "test.txt";
+		CollectionReader reader = createCollectionReader(TextReader.class,
+				TextReader.PARAM_PATH, path, TextReader.PARAM_PATTERNS,
+				new String[] { "[+]" + pattern }, TextReader.PARAM_LANGUAGE,
+				"en");
+
+		
+		AnalysisEngine tokenizer = createPrimitive(OpenNlpSegmenter.class);
+		AnalysisEngine ngram = createPrimitive(NgramAnnotator.class,
+				NgramAnnotator.PARAM_HOLE, 3, NgramAnnotator.PARAM_NGRAM, 4);
+		AnalysisEngine pseudoPosTagger = createPrimitive(PseudoPosTagger.class);
+		AnalysisEngine out = createPrimitive(OutputHadoopRelations.class);
+		AnalysisEngine out2 = createPrimitive(PosTaggedHadoopOutput.class);
+		SimplePipeline.runPipeline(reader, tokenizer, pseudoPosTagger, ngram,
+				out, out2);
+~~~~~~
+Here we use some components of [dkpro](http://code.google.com/p/dkpro-core-asl/), [uimafit](http://code.google.com/p/uimafit/) and from [OpenNLP] (http://opennlp.apache.org/) for reading the files, processing the pipeline and tokenizing. The NgramAnnotator annotates the Jo and Bims. After that step the relations can be transformed into a dataformat, that fits for the Hadoop processing (using the class OutputHadoopOutput). As this class has an extra method to extract the content of the Annotation (called getValue(Annotation)) the content can be easily changed, so the word might also include the Part-Of-Speech tag, as done with the extended class PosTaggedHadoopOutput which overwrites the getValue method.
+

WikiPage Holing_System modified by Chris Biemann

Chris Biemann — Wed, 17 Oct 2012 13:14:24 -0000

--- v3
+++ v4
@@ -1,7 +1,7 @@
 The @@ operation: From Text to Pairs
 =======
 
-This page describes the general idea of the @@ operation (pronounce: holing operation). This operation is used to split a structural observation on text into two parts, which could be thought of as word and context, or word and feature.
+This page describes the general idea of the @@ operation (pronounce: holing operation). This operation is used to split a structural observation on text into two parts, which could be thought of as word and context, or word and feature. These parts are called *Jo* and *Bim*, as they are not distinguishable in terms of type in the gereral case, but have to be distinguished in order to describe them.
 All the observations are subject to the @@ operation. These are written out, and serve as the input for the [Distributional Similarity with MapReduce](Distributional_Similarity_with_MapReduce) computation.
 
 Example:
@@ -19,32 +19,38 @@
 
 1. From A(B,C) ->  B -- A(@@,C)
 
-    suffered	  nsubj(@@, I)		1
-    took	  nsubj(@@, I)		1
-    cold	  det(@@, a)		1
-    suffered	  prep_from(@@, cold)	1
-    suffered	  conj_and(@@, took)	1
-    took	  dobj(@@, aspirin)	1
+Jo       |    Bim  | count
+--------- | ---------  | -------------
+suffered  |  nsubj(@@, I)		| 1
+took     |  nsubj(@@, I)		| 1
+cold     |	  det(@@, a)		| 1
+suffered  |	  prep_from(@@, cold)	| 1
+suffered  |	  conj_and(@@, took)	| 1
+took   |	  dobj(@@, aspirin)	| 1
 
 2. From A(B,C) ->  C -- A(B,@@)
 
-    I	nsubj(suffered, @@)	1
-    I	nsubj(took, @@)		1
-    a	det(cold, @@)		1
-    cold	prep_from(suffered, @@)	1
-    took	conj_and(suffered, @@)	1
-    aspirin	dobj(took, @@)		1
+Jo | Bim | count
+--------- | ---------  | -------------
+I |	nsubj(suffered, @@)	| 1
+I |	nsubj(took, @@)		| 1
+a |	det(cold, @@)		| 1
+cold |	prep_from(suffered, @@)	| 1
+took |	conj_and(suffered, @@)	| 1
+aspirin |	dobj(took, @@)		| 1
 
-The "1" indicates that we see these pairs a single time in this sentence. Longer texts will produce higher pair counts. 
+The count of 1 indicates that we see these pairs a single time in this sentence. Longer texts will produce higher pair counts. 
 
 Variations of the @@ operation
 ------
 Context definition: Instead of dependency parses, we can use any kind of structure on text, including but not limited to:
+
 * n-grams
 * positional co-occurrence
 * dependency chains
 
 @@ variants: It is possible to:
+
 * subsume several parts of the observation into a single hole, e.g. to handle multiwords or to keep pairs together A(B,C) --> (B,C) - A(@@)
 * put several holes, like A(B,C,D) --> (B,D) - A(@@,C,@@)

WikiPage Holing_System modified by Chris Biemann

Chris Biemann — Wed, 17 Oct 2012 13:07:48 -0000

--- v2
+++ v3
@@ -8,28 +8,32 @@
 -----
 
 Consider the following sentence:
+
     I suffered from a cold and took aspirin.
 
 Let's say we have observed the following structure (using a dependency parser in this case):
-      nsubj(suffered, I); nsubj(took, I); root(ROOT, suffered); det(cold, a); prep_from(suffered, cold); conj_and(suffered, took); dobj(took, aspirin)
+
+    nsubj(suffered, I); nsubj(took, I); root(ROOT, suffered); det(cold, a); prep_from(suffered, cold); conj_and(suffered, took); dobj(took, aspirin)
 
 If we define the @@ operation such that either the first or the second term in the dependency relation is used as the @@, we get the following pairs:
 
 1. From A(B,C) ->  B -- A(@@,C)
+
     suffered	  nsubj(@@, I)		1
     took	  nsubj(@@, I)		1
     cold	  det(@@, a)		1
     suffered	  prep_from(@@, cold)	1
     suffered	  conj_and(@@, took)	1
     took	  dobj(@@, aspirin)	1
+
 2. From A(B,C) ->  C -- A(B,@@)
+
     I	nsubj(suffered, @@)	1
     I	nsubj(took, @@)		1
     a	det(cold, @@)		1
     cold	prep_from(suffered, @@)	1
     took	conj_and(suffered, @@)	1
     aspirin	dobj(took, @@)		1
-
 
 The "1" indicates that we see these pairs a single time in this sentence. Longer texts will produce higher pair counts.