#574 Wrong syntax highlighting in dtx files

None
open
nobody
None
5*
2014-01-20
2013-10-08
mrpiggi
No

In .dtx files, matching braces are not recognized, if there's a source code segment like

%    \end{macrocode}
% Explanation
%    \begin{macrocode}

in between. It results in a wrong syntax highlighting, shown in the attached screenshot.

1 Attachments

Discussion

  • mrpiggi
    mrpiggi
    2013-10-08

    Same with tags...

     
    Attachments
  • Tim Hoffmann
    Tim Hoffmann
    2013-10-08

    This is because the bracing of \foo{} and \begin \end is interweaved in your example:

    %   \begin{macromode}
    \def\foo{
    %   \end{macromode}
    }
    

    In simple braces the equivalent would be ( { ) } which is also marked as an error.

    TXS assumes a strictly hirachial brace sequence: the first closing tag matches the last opening tag. It is considered an error if they are of different type. This assumption is correct in most languages. I'm not an expert in dtx, so does this not hold for dtx?

     
    Last edit: Tim Hoffmann 2013-10-08
  • mrpiggi
    mrpiggi
    2013-10-09

    In dtx files the concept of literate programming is used. This means, source code and it's documentation (maybe even the user documentation) is given in the same file. So when the dtx file is processed by docstrip, everything within the macrocode environment will be exported to a style or class file. Everything outside the macrocode environment is used for the source code documentation. So the segment

    % \begin{macro}{\foo}
    % Explanation of the macro
    %    \begin{macrocode}
    \def\foo{%
    %    \end{macrocode}
    % Explanation of a segment within the macro
    %    \begin{macrocode}
      foo%
    }
    %    \end{macrocode}
    % \end{macro}^^A \foo
    

    would result in this source code

    \def\foo{%
      foo%
    }
    

    and the attached documentation. You can find an introduction to dtx files here

     
    Attachments
  • Tim Hoffmann
    Tim Hoffmann
    2013-10-09

    Ok, we cannot fix this. One would need two parallel brace stacks, one for the dtx-related braces and one for tex-related braces. This is beyond the capabilities of the language model of the editor.

    There's one thing that I could offer, but I'm not sure if that's better. Since it's a speciality of \begin{macrocode} \end{macrocode} I can remove the parenthesis property from these commands in the language definition. Then, all brace-related functionality is deactivated for these commands (also including code folding, but that doesn't work correctly in your example code either). What do you think?

    Of course, you may alternatively switch off brace matching in the options completely.

     
    Last edit: Tim Hoffmann 2013-10-09
  • mrpiggi
    mrpiggi
    2013-10-09

    Removing the parenthesis property from macrocode would be fine. That would be enough to me (almost).

    But, since \begin{macrocode} and \end{macrocode} must be intended the same way, maybe there's another possibility to find \begin{macrocode} and matching \end{macrocode}. Both commands always have to be set with a leading percent sign followed by exactly four spaces. So maybe it is possible to use this characteristic, even for a different color markup?

    Silly code example

    % \begin{macro}{\foo}
    % Explanation of the macro
    %    \begin{macrocode}
    \def\foo{%
    %    \end{macrocode}
    % Explanation before table
    %    \begin{macrocode}
      \begin{table}
    %    \end{macrocode}
    % Explanation befor minipage
    %    \begin{macrocode}
        \begin{minipage}{3cm}
    %    \end{macrocode}
    % Explanation before tabular
    %    \begin{macrocode}
          \begin{tabular}{ccc}
    %    \end{macrocode}
    % Explanation within tabular
    %    \begin{macrocode}
            f & o & o
          \end{tabular}
        \end{minipage}
      \end{table}
    }
    %    \end{macrocode}
    % \end{macro}^^A \foo
    
     
  • Tim Hoffmann
    Tim Hoffmann
    2013-10-09

    Detecting the commands is not an issue. It's the non-stack-like sequence of opening and closing tags which poses the problem. Therefore, the %-and-space characteristic unfortunately doesn't help.

    On the bright side, different color markup is very easy. Should also other commands like \begin{macro} be highlighted differently or only macrocode? I could imagine a dtx-environment format for that.

     
  • mrpiggi
    mrpiggi
    2013-10-10

    Typically, there are only \DescribeMacro{\cmd} and \DescribeEnv{env} for the user documentation together with \begin{macro}{\cmd} and \begin{environment}{env} for the inline documentation. But with package dox, a class or package author can also create environments and the corresponding API for options, length and other user-defined items. So, if you really want to create a special dtx color markup, it should be customizable.

    Since theres are a documentation part and a code part in literate programming, a different background color for both parts would be fine. Here's an example. And maybe it should be possible to control the parenthesis property for macrocode and tags (<*tag>...</tag>) optionally.

     
    Last edit: mrpiggi 2013-10-10
  • Tim Hoffmann
    Tim Hoffmann
    2013-10-10

    I've added the four commands you specified to highlighting as dtx:commands. Customizable commands are not possible because the the language definitions are currently not user-editable. But I think basic highlighting is still better than no highlighting :). And of course, you can set the format back to the values of the original format if you don't need extra highlighting for these.

     
  • mrpiggi
    mrpiggi
    2013-10-11

    Where can I find a beta version for testing? And what about the different background colors for the both segments delimited by macrocode?

     
  • Tim Hoffmann
    Tim Hoffmann
    2013-10-11

    The code is in the version control system (mercurial) here on sf. If you want to have it right now, you'll have to compile it yourself. But we'll release a new version in a few days.

    The macrocode delimited highlighting suffers from the same problem: As long as the matching of these tags doesn't work correctly, we cannot highlight the lines in between (well at least not without the additional overhead of a separate dtx parser).

     
  • mrpiggi
    mrpiggi
    2013-10-22

    Thanks for the new release, everything works as expected and is very helpful.

    I would be pleased if you might still find a simple solution in the future, or the time for a separate dtx parser to check <*tag>...</tag> as well as macrocode for consistency, so they can be highlighted even better.

     
  • Tim Hoffmann
    Tim Hoffmann
    2013-10-22

    Ticket moved from /p/texstudio/bugs/818/

     
  • mrpiggi
    mrpiggi
    2014-01-20

    Hello,

    I got another issue. If you want to define a macro with following syntax:

    \mymacro[optional argument]{mandatory argument}[optional argument]
    

    than you have to use something like this:

    \newcommand\mymacro[2][]{%
        \@ifnextchar[{\@mymacro[#1]{#2}}{\@mymacro[#1]{#2}[]}
    }
    

    and within \@mymacro you have to do the argument handling.

    My problem is the opened square bracket after \@ifnextchar because there will be no closing square bracket at all. Is there any possibility to prevent the character after \@ifnextchar to be putted on the stack?

    BTW
    In AUCTeX you can do something like

    \@ifnextchar[%]
        {\@mymacro[#1]{#2}}{\@mymacro[#1]{#2}[]}
    
     
    Last edit: mrpiggi 2014-01-20
    • Tim Hoffmann
      Tim Hoffmann
      2014-01-20

      Actually, you can't do anything about this. Our parser is too smart for a workaround like the one in emacs (in general it is never correct for a language to match a bracket in a comment to one outside).

      But it doesn't really hurt too much. You only see that the bracket is not matched when you place the cursor next to it.

       
  • mrpiggi
    mrpiggi
    2014-01-20

    No offense. None taken.