#1219 OmegaT does not handle line breaks correctly on Windows

[Wynn Chen]

Translated segments that contains line breaks in their source text will become untranslated each time the project is reopened.

I used a text editorto view the project_save.tmx and noticed something might be helpful:

For examle, here is a source segment (en-US) containing a line break:

line1
line2

Translate it and move to another segment (this is important, the "current" segment will not be affected by this problem), then save the project, its corresponding entry in project_save.tmx file looks like:

    <tu>&#xd;
      <tuv lang="en-US">&#xd;
        <seg>line1&#xd;
line2</seg>&#xd;
      </tuv>&#xd;
      <tuv lang="[target lang]" changeid="..." changedate="..." creationid="..." creationdate="...">&#xd;
        <seg>[translation of "line1"]&#xd;
[translation of "line2"]</seg>&#xd;
      </tuv>&#xd;
    </tu>&#xd;

Close the project and reopen it, this segment became untranslated, translate it and save the project again, then there will be a new entry in project_save.tmx file looks like:

    <tu>&#xd;
      <tuv lang="en-US">&#xd;
        <seg>line1&#xd;#xd;
line2</seg>&#xd;
      </tuv>&#xd;
      <tuv lang="[target lang]" changeid="..." changedate="..." creationid="..." creationdate="...">&#xd;
        <seg>[translation of "line1"]&#xd;#xd;
[translation of "line2"]</seg>&#xd;
      </tuv>&#xd;
    </tu>&#xd;

Each time I re-open the projected and re-translate this segment and them save the project, there will be a new entry in tmx with an extra "
" at the end of lines in segment text. It seems that OmegaT does not handle \r in Windows style line breaks correctly.

I have tested it on source files with Windows and Unix line break style , both are the same. I have also tested it on a project containing plain text files only, so it is not caused by a plugin.

All tests is done with latest weekly build (2023-10-09) and on Windows 10 (pro, 22H2, 19045.3448).

[Wynn Chen]

I'm pretty sure this problem is caused by xml.writeCharacters(lineSeparator); in omegat/src/org/omegat/util/TMXWriter2.java. Since lineSeparator = System.lineSeparator() and it will get \r\n on Windows.
I found a similar question on stackoverflow: https://stackoverflow.com/questions/29675481/xmlstreamwriter-java-8-writecharacters-xd . Hope this helps.

[Hiroshi Miura]

Yes, project_save.tmx is internal format that we can unify line end to be UNIX style.

[Hiroshi Miura]

OmegaT 6.1 changes XML engine to Woodstox .
it has an input property P_NORMALIZE_LFS which default is true.

P_NORMALIZE_LFS (default: true): XML specification requires parsers to convert “alt linefeeds” (that is, \r\n and \r) into canonical linefeed (\n), but disabling this property allows exposing actual linefeed without normalization

it has output property

P_OUTPUT_ESCAPE_CR (default: true): setting that determines whether output of \r character will result in matching XML entity (when enabled) or simply character itself. Main distinction here is that embedded \r characters will be normalized during parsing (… unless prevented by P_NORMALIZE_LFS discussed earlier…), whereas results of entity expansion will be exposed as-is (never normalized).

this will change XML parser behavior with 6.0 and before.

[Hiroshi Miura]

https://github.com/omegat-org/omegat/pull/780 merged into master.