[DL-Learner SVN] SF.net SVN: dl-learner:[4186] trunk/components-core/src/main/java/org/ dllearner/algorithms/isle/index/TextDocument.java

[<dfl...@us...>]

Revision: 4186
          http://sourceforge.net/p/dl-learner/code/4186
Author:   dfleischhacker
Date:     2013-12-02 15:20:16 +0000 (Mon, 02 Dec 2013)
Log Message:
-----------
Re-enabled ignorePunctuation

Modified Paths:
--------------
    trunk/components-core/src/main/java/org/dllearner/algorithms/isle/index/TextDocument.java

Modified: trunk/components-core/src/main/java/org/dllearner/algorithms/isle/index/TextDocument.java
===================================================================
--- trunk/components-core/src/main/java/org/dllearner/algorithms/isle/index/TextDocument.java	2013-12-02 15:19:06 UTC (rev 4185)
+++ trunk/components-core/src/main/java/org/dllearner/algorithms/isle/index/TextDocument.java	2013-12-02 15:20:16 UTC (rev 4186)
@@ -112,13 +112,20 @@
     public List<Token> getTokensStartingAtToken(Token start, boolean ignorePunctuation) {
         ArrayList<Token> tokens = new ArrayList<Token>();
 
+        int relevantTokens = 0;
         boolean found = false;
 
-        for (int i = 0; i < this.size(); i++) {
-            Token t = this.get(i);
-            if (t == start) {
-                return this.subList(i, this.size());
+        for (Token t : this) {
+            if (found) {
+                tokens.add(t);
+                if (!ignorePunctuation || !t.isPunctuation()) {
+                    relevantTokens++;
+                }
             }
+            else if (t == start) {
+                found = true;
+                tokens.add(t);
+            }
         }
 
         return tokens;

This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.