The Lemur Project / Bugs / #235 Galago: TagTokenizer misses some single-character tokens

Search engine and data mining applications and ClueWeb datasets.

#235 Galago: TagTokenizer misses some single-character tokens

Milestone: v5.x

Status: closed

Owner: Weize Kong

Labels: galago (57)

Priority: 1

Updated: 2014-06-19

Created: 2014-04-28

Creator: Weize Kong

Private: No

org.lemurproject.galago.core.parse.TagTokenizer will tokenize "17.1" into {"17"}, not {"17", "1"}. The problem is in method tokenAcronymProcessing, line 544

if (token.length() - s > 1) { // should change to token.length() - s > 0
String subtoken = token.substring(s);
addToken(subtoken, start + s, end);
}

token.length() - s > 1 will miss single-character tokens

Discussion

Weize Kong - 2014-04-28

Group: v1.x --> v5.x
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Fisher - 2014-04-30

assigned_to: Weize Kong
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Fisher - 2014-06-10

status: open --> accepted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Fisher - 2014-06-10

to ship in 6/2014 release.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Fisher - 2014-06-19

Status: accepted --> closed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Galago: TagTokenizer misses some single-character tokens

Search engine and data mining applications and ClueWeb datasets.

Group

Searches

Help

#235 Galago: TagTokenizer misses some single-character tokens

Discussion