OmegaT - multiplatform CAT tool / Bugs / #869 Tokenizer language discovery at runtime not working

#869 Tokenizer language discovery at runtime not working

Milestone: 4.1

Status: closed-fixed

Owner: Aaron Madlon-Kay

Labels: None

Priority: 5

Updated: 2017-06-23

Created: 2017-05-24

Creator: Tomi Pieski

Private: No

Hi!
We are developing a fst based tokeniser for omegat to be able to use our fst tools.
We noticed a bug in the way BaseTokenizer detects the language. It might be elsewhere also.

getLanuages() has 'languages[0].equals(Tokenizer.DISCOVER_AT_RUNTIME)' which never returns true, because string equals() compares object reference.

All best,
Tomi

Discussion

Aaron Madlon-Kay - 2017-05-25

I believe you have it backwards: equals() compares content, while == compares the reference. Indeed BaseTokenizer.getLanguage() does use ==; I believe the assumption was that the string would be interned and thus a reference comparison would be better. However when inspecting at runtime, it appears that the value from the annotation is a different object.

Last edit: Aaron Madlon-Kay 2017-05-25

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Aaron Madlon-Kay - 2017-05-26

assigned_to: Aaron Madlon-Kay
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Aaron Madlon-Kay - 2017-06-20

status: open --> open-fixed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Aaron Madlon-Kay - 2017-06-20

This should now be fixed in trunk.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Aaron Madlon-Kay - 2017-06-20

summary: BaseTokenizer.getLanguage() change string.equals() to '==' --> Tokenizer language discovery at runtime not working
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Didier Briel - 2017-06-23

status: open-fixed --> closed-fixed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Didier Briel - 2017-06-23

Fixed in the released version 4.1.2 of OmegaT.

Didier

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Tokenizer language discovery at runtime not working

The free computer aided translation (CAT) tool for professionals

Group

Searches

Help

#869 Tokenizer language discovery at runtime not working

Discussion