#1284 File structure corruption in *.tex files

[Dave W]

Confirmed to occur on Mac OS 15.3 (Intel) and Windows 11 23H2 using OmegaT version 6.0.1 with standard settings.

When creating target files for LaTex (.tex extension) newlines and tabs are either stripped or replaced by spaces, thus effectively corrupting the LaTex command/argument relationships in the files, causing the Typesetting process to fail.

This is particularly noticeable with the commands and arguments in the file that remain invisible to the translator.

I have attached two sanitised Source-Target example pairs and thei diffs

This may be a continuation of:
#1067 and #993

[Dave W]

*their diffs

[Jean-Christophe Helary]

Thank you for your report. It does look like a similar issue to the 2 tickets you reported.

I applied the provided patch to 6.0 in this branch:
https://github.com/omegat-org/omegat/tree/topic/jc/filters/latex

And here is a prototype build. Coulc you check how that works for you?
https://traductaire-libre.org/omegat/OmegaT_LaTeX_Filter_Fix.zip

[Dave W]

Thanks for your quick reply and for your work. Some changes, but largely the same. It still breaks LaTex Typestting.

I've uploaded the two new Target files and differences between them and (a) the other Target files and (b) them and the source files.

Obviously, it doesn't need to be super prettily indented, which would be nice, but the line breaks putting new commands on a new line and whitespace between commands and arguments must be maintained.

[Dave W]

Don't know if it's of any assistance, but I'm currently trying to get this to build to see if its of any help, but the gradle build keeps failing. :(

If I manage to get it working (it's apparently for 5.0 so, even if it builds, it might ot work) I'll let you know.

https://github.com/chelobaka/simple-latex-filter/blob/main/CONFIGURATION.md

[Jean-Christophe Helary]

The build requires either Java 8 or 11.

If you have more recent versions (17 or 21) use

export JAVA_HOME=/path/to/openjdk11/Contents/Home/; ./gradlew jar

to build it. It works here.

[Jean-Christophe Helary]

Here is the build with Java 11.

[Dave W]

Thanks ever so much for the help. As you may have guessed, I'm no Java expert.

I've installed it and set Local Filters to disable the default LaTex filter and use only simple-latex-filter and now the file strucure appears to be correctly maintined, so things look promising

However, I'll test it out properly on a full, Typesettable, document tomorrow (not just random test files) and let you know if it truly is a valid workaround.

[Jean-Christophe Helary]

Thank you for checking. If you find a significant improvement, I’m all for disabling the LaTeX filter and refering to that plugin instead, until we find a way to integrate it.

[Dave W]

OK. So it's good news and bad news:

The good news is that the simple-latex-filter (slf) plugin works well with un-extended 'clean' LaTex and makes much better sense of complicated documents than the standard LaTex plugin.

The bad news is that you can extend LaTex by defining your own commands and that has been done in the document I'm working on. I'm not certain, however, but my gut feeling is that may be causing some of the difficulties.

Thats said, there was much less debugging needed with slf purely from fact that the file structure is much closer to the original Source in the generated Target files.

What does seem to be a major issue is the way LaTex handles spaces in a Tex file.

I noticed, when editing the slf imported document in OmegaT, that there are a lot of leading and trailing spaces before and after segments and/or tags and, if they are omitted, it usually causes issues with the Typesetter in the form of concatenated commands and data.

I assume this is because of the way slf works, taking everything before and after commands including the spaces so that, for example, <i1/>Some text in OmegaT becomes \itemSome text instead of the correct \item Some text, causing the list \item command to break. Entering the missing space, <i1/> Some text fixes it.

If you're interested, the space issue is discussed in the comments to this Stack Exchange post:
https://tex.stackexchange.com/questions/31091/space-after-latex-commands

Bottom line: For now I am OK with continuing to work with the simple-latex-filter.

If it's any help and if you want, will add anything else of interest I discover here.

Thanks again for your much appreciated help. :)

[Jean-Christophe Helary]

Thank you very much for your report.

Now, that’s something that you could comment on in the slf issues.
https://github.com/chelobaka/simple-latex-filter/issues

I’m not sure whether @abashkinl is still actively developping the plugin, but that’s the kind of input that could be useful there for further developments.

[Lev Abashkin]

Hi Dave,

SLF didn't get much testing and its default configuration is not optimal. The \item command fits better in CONTROL section. You can change it in SimpleLatexFilter.json located in your Omegat config folder. After that segments with \item will change and won't match existent translation memory.

Update: I changed it in the repo https://github.com/chelobaka/simple-latex-filter/commit/972538a891430ebdb7decc95bed1b0d0e60c2d45

[Jean-Christophe Helary]

David, can you build and test on your side?

[Lev Abashkin]

There is no need to rebuild the plugin. Local settings override internal ones.

[Dave W]

@brandelune: I guess, but I'm using Java 21 and am not paticularly au fait with it or with building anything without detailed instructions.

@abashkinl: Good to know.. Thanks for your plugin, by the way. Good piece of work. It runs perfectly under OmegaT 6.0.1.
I'll look into SimpleLatexFilter.json when I have some time. For now I have an iterative workflow where I run a Typeset and then go in and fix the errors by adding linefeds and/or spaces in OmegaT so that the Tex code matches between the relevant Source and Target file. Then rinse and repeat. Clunky, but it works without altering or learning anything new while I'm busy translating and that's what I need right now. :)

@brandelune: In fact, once I've go this translation out of the way, I might try the same workflow with the default LaTex plugin to see if it works with that.