#933 Loss of linebreaks when translating LaTeX files

[Fabian Wehning]

On translating LaTeX documents the linebreaks are lost. This results in compilation errors in LaTeX in environments such as lstlisting.

The following code is the source paragraph

\lstset{caption={Program call on Windows}}
\begin{lstlisting}[basicstyle=\small]
jre\bin\javaw.exe -jar MyJar.jar "argument 1" 
\end{lstlisting} 

It will be replaced to

\lstset{caption={Program call on Windows}}\begin{lstlisting}[basicstyle=\small]jre\bin\javaw.exe -jar MyJar.jar "argument 1" \end{lstlisting} 

Without the linebreaks this results in compilation errors. The linebreaks are mandatory to compile this code.

I have attached a minimal example with a source document and the according generated target file (named appropriatly).

[Fabian Wehning]

Any progress on this?

[Didier Briel]

No.

For possible reasons for this, see Where's my feature or bugfix? .

Didier

[Fabian Wehning]

This should help. As far as I can tell all texts are found and the linebreaks are well, but the test file fails. But at the moment I have no more time to look into it.

[Aaron Madlon-Kay]

In addition to breaking the unit tests, you have reformatted the entire file, making it

• very difficult to see what you actually changed
• inconsistent in style with the rest of the project

I'm sorry, but we can't accept your contribution in this form. If you're interested in pursuing this further, please:

1. Fix the broken unit test
2. Provide either a .java file, or preferrably a diff, with the absolute minimum changes

[Fabian Wehning]

Here is a file with minimal changes. The filter will now find more strings than before (some which do not need to be translated) but retains the file format - which is way more important. The fixed test file will be in the next post.

[Fabian Wehning]

Test file

[Aaron Madlon-Kay]

(Starting new thread because the indentation was getting to be too much)

Thank you for the new files. However this is still very hard to look at:

• You have removed the try (why?) from processLatexFile() so the entire method body's indentation has changed
• Even diffing without whitespace changes, you have still changed a bunch of things that don't need changing, thus burying your substantive changes in (still) lots of noise
  • You have removed brackets from ifs (please don't)
  • You have reformatted comments that don't need reformatting

For the tests, you removed the @Test annotation, meaning the test won't even be run! When the test is made to run, it does work, but it's not clear to me that your changes are even desirable: the extracted text has changed, and a lot of clearly untranslatable material is being handled as if it was translatable.

I don't deny that the current behavior of the filter (shown by your example files) can be improved on. But I am cautious about introducing any fixes that change what text is extracted for translation. Can you fix the linebreak handling without changing the extracted text?

[Fabian Wehning]

Oh, I see the mistake. I took the 3.6.0 version of the source code. Which explains the bracket and annotations missing (which I did not removed) and comment thing . Basically I only changed in the last part of the processLatexFile() the block around

 /* at the end of the line */
            if (state.equals("N")) {

String endOfLine = lpin.getLinebreak();

            /* at the end of the line */
            if (state.equals("N")) {

                /* \par */
                if (par.length() > 0) {
                    out.write(processParagraph(commands, par.toString()));
                    out.write(endOfLine);
                    par.setLength(0);
                    commands.clear();
                } else if (comment.length() > 0) { // If there is a comment, write it
                    out.write(comment.toString());
                    out.write(endOfLine);
                    comment.setLength(0);
                } else {
                    out.write(endOfLine);
                }
            } else if (state.equals("M") || state.equals("S")) {
                out.write(processParagraph(commands, par.toString()));
                par.setLength(0);
                out.write(endOfLine);
                commands.clear();
            }
        }

The new behavior of the texts is in my opinion an improvement as sentences are now splitted correctly but you gain some texts more because they are within an argument of a function. And this is a place where you may always find texts. Just in the cases I observed it is only text stating formating information for the document. It might be possible to extend the init() list with further commands ike toc, but in a real test document I use this was the only really completely new command text (except of two instances of { and } which I can not explain.
The problem with this command pattern which is used in this filter is it focuses only on commands with no or only one pair of {}-Brackets. Ignoring totally commands with multiple or mixed bracket like \command[]{} or \command2{}{}.

[Aaron Madlon-Kay]

My apologies: I should have realized you were working from 3.6.0. Thanks for clearing that up.

Note that there wouldn't have been any confusion if you had supplied diffs rather than full files. Or, as I prefer, opened a PR on GitHub.

Here are my concerns going forward:

This is a big change in behavior. I don't have the knowledge to judge whether it is universally desirable, or whether it should be an option, and if so whether that option should be on by default, etc. I think this merits discussion on the userlist: a prototype build should be made availble, and a call should be made to solicit feedback from other LaTeX users.

Also I'm not willing to accept patches against anything other than trunk (4.1). 3.6 is (hopefully) going to be EOL soon, and I just don't have time to port back and forth.

[Jean-Christophe Helary]

I applied the patch to 6.0 in this branch:
https://github.com/omegat-org/omegat/tree/topic/jc/filters/latex