Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

New UserDefinedLanguage module

Loreia2
2011-07-30
2014-06-11
1 2 3 .. 26 > >> (Page 1 of 26)
  • Loreia2
    Loreia2
    2011-07-30

    Hi everyone,

    I wrote a new UserDefineDialog colorization algorithm.
    Patch report is submitted to http://sourceforge.net/tracker/?func=detail&aid=3382886&group_id=95717&atid=612384 for review and possible inclusion in official release.

    New version removes old limitations and introduces some new features, namely:
    1. Operators and delimiters can now be longer that one character
    e.g.
    old operators: ! < >
    new operators: !! << >> """

    2. Added full support for three set of delimiters
    Old dialog allowed only two delimiters to be selected, even though xml file could save three delimiter sets. New dialog is extended to allow selection of third set of delimiters

    3. New algorithm works even when there is no whitespace separator of two elements.
    e.g.
    old algorithm could not colorize correctly:
    /*comment*/ (no spaces within comment)
    keyword!delimiter! (no spaces after the keyword)
    (This was the main thing for me, and the reason I started working on this)


    4. Export button works only if new definition is first saved with "Save As…" button, but old dialog would not warn user about it. (This left me scratching my head why Export button suddenly stopped functioning.) New version displays MessageBox that notifies user to save definition before attempting to export it.


    5. If you define the same characters as comments AND operators, colorization still works.
    e.g.
    comment start: /*
    comment end: */
    operators + / - *
    This is possible because comments are checked first.
    (That also means that if you define excatly the same character/string as both operator and comment, it will always be treated as comment.
    e.g.
    comment line: !
    operators + ! / - *
    In this case any occurence of ! character will be treated as start of line comment.)


    6. Last word is colorized even if file does not end with new line.
    7. No visible performance penalty (on my machine it takes roughly the same time to colorize few hundred thousand lines with old and new version)

    8. Also, some small bug fixes are included, like tab order in all four UDL dialogs or escape character box being initialized with garbage because of a missing _T() macro.

    9. New UDL has no use for "Treat keywords as symbols" option, so it was removed.

    I've done extensive testing of new algorithm, and I **think** I've covered everything. Now I need community support to test it more thoroughly.

    Basically, I need you to:
    1. download new version of Notepad++ (zip archive with two files: notepad2++.exe and SciLexer2.dll)
    2. save it to your installation folder (unzip it off course, nothing is overwritten because files have different names than defaults)
    3. close notepad++ if it is running
    4. double click new file notepad2++ to start it
    (this is a Unicode release build, if you need ANSI build let me know)
    5. go to Search/User-Defined Dialogue…

    Here it is:
    http://www.mediafire.com/?fx3dshychsg8v57
    (sorry about using media-fire, but Sourceforge is so slow, it is barely functoning for me, and I see no option to upload files)

    Try to import old definition (if you have one), to see if everything works correctly. Test everything in "Operators" tab. (Just remember that translation will not work here because old translation files
    don't have translation for new GUI elements.) Test export function too.

    Now try to see if colorization works. You can use your old code files, or quickly write something down, or you can use my "quick'n'dirty" python script to generate test cases (you'll need to install python version 2, e.g. 2.7 http://www.python.org/download/).

    Note:
    This script is a bit ugly (it was not intended for publishing), but it gets the job done. Feel free to extend it anyway you see fit.
    Also, It will generate some 20 or so *.loreia files, with thousands of test cases (but you'll scroll through them fairly quickly)
    Colorization for test cases can be imported from file "loreia.xml", import it, and it will give you another language named "Loreia".
    (And no, I am not an ego maniac, I just lack creativity at 23:00h :-))
    Here is the script:
    http://www.mediafire.com/?aryr7meyp1k1qph

    If you find anything wrong, just post it here and I'll fix it.

    Thanks a lot,
    Loreia

     
    • Chaparo
      Chaparo
      2012-11-22

      Hi,
      Your tool is great and awesome,
      I have a small question,I haven't readthe 19 page sorry,
      But after lot of research, I don't fine any information about the Regex.
      Do your tools take account of regular expression ?

       
    • yfital
      yfital
      2013-01-14

      UTF16 (UCS2 LE) coloring problems.

      Hey,
      I've been loving the new UDL(2), But I have a major issue of coloring text.
      I suspect the problem is the fact that the delimiter I am inserting is being treated as ANSI while my file itself is UTF16.

      You can see the problem in the attached document, all the white/blue colored xx are actually a single ¥ character, when a ¥ is being inputted as an operator list for styling, it screws up the display (this also happens for trademark signs such as ® and 'c' ).

      I would really appreciate some help with this as it is causing us a nightmare of reading our files ;]

      thank you very much !

      Alon

       
      Last edit: yfital 2013-01-14
      Attachments
  • This is great! I hope it gets into the main sources.
    Can you also add support for nested comments?
    e.g. { this is a comment { and so is this } }

     
  • Loreia2
    Loreia2
    2011-07-30

    Hi Alex,

    in your case an escape character is needed, say '\\'.
    Then you can write: { this is a comment { and so is this \} }

    I am not aware of any language that supports nested comments, but if there are such languages, and if it is a popular request, I guess I could add that feature in.Off course, one more option in GUI would be needed. Something like: "Allow nested comments".

    But, I don't think this feature is part of any language.
    If I am wrong, please put some links here.

    BR
    Loreia

     
  • Loreia2
    Loreia2
    2011-07-30

    I just googled "nested comments", and it seems some languages do allow them.
    I guess, I'll just have to add that option too :-)

    I can update Algorithm in LexUser.cxx fairly easily (just need to count open and close comment strings), but I'll need some time to adapt UDL dialog.

    I post here, when I finish it.

    BR,
    Loreia

     
  • Loreia2
    Loreia2
    2011-07-30

    Ok, I'm convinced. Next revision will have support for nested comments. I'll fix LexUser.cxx latter today (and post it on patch tracker). But it might take me few days to patch UDL dialog (due to my inexperience with NPP code, and some obligations I need to finish this weekend), even though it is just a single check box to be added.

    BR
    Loreia

     
  • If you've got changes in SciLexer, you ought to consider submitting them to Scintilla for inclusion in that source.

     
  • This has nothing to do with Scintilla sources.

     
  • Loreia2
    Loreia2
    2011-07-30


    Is this it?

     
  • Loreia2
    Loreia2
    2011-07-30

    cowagain,

    UDL is Notepad++'s extension of Scintilla. Source code should be reported here.

    BR
    Loreia

     
  • loreia2,

    Looks perfect. Good work!

     
  • Loreia2
    Loreia2
    2011-07-30

    Ok, I am glad I got it.
    I will test it a bit more, and post update to patch tracker latter today or maybe tomorrow.

    If anyone has additional requests, just post them here.

    BR,
    Loreia

     
  • If there's a version of SciLexer.dll that you're including in the package, it seems reasonable to assume changes to Scintilla source.

     
  • Vera
    Vera
    2011-07-31

    Hello loreia2 ,

    I really appreciate your effort and wish you success with it :-)

    But unfortunately your new UDLversion breaks two major highlings for me

    1a. the commenting: because you removed "Treat keywords as symbols", my commentsymbol ';' now MUST have a 'space' behind it

    1b. for the same reason, that commenting symbol may now corrupt following code, when it is used inside a dedicated "STRING"

    2. the one and only delimiter I need: " (to assign "STRINGS") cannot be made to work again.
    In my udl-setting the " is as well assigned as operator plus used as delimiter and works fine.
    Now whatever I tested as combination: being operator or not, doubling the operatorsymbols, taking first/middle/last or mixed delimiterfields … regarding blank or not … nothing will return the "String" highlighting.

    Additionally I also miss the specific verification of what operators are available/enabled at all, (which is easily identifiable from the facing boxes) - or is that meant to be totally free in your edition?

    sorry for not having splendid replies, but maybe others might cross the same issues with their own language

    regards ~ Vera

     
  • Loreia2
    Loreia2
    2011-07-31

    Hi Vera,

    thanks for taking time to test it.
    What you describe is a basic test case new UDL was designed to solve (no spaces are needed as separators). If something as simple as keyword;comment;keyword is not working, that means I've screwed my last build. I can assure you that something as basic as that was working when I tested it.

    I guess working without version control is a bad thing (and working late at night is even worse). I will review code changes, and post here what went wrong.

    In the mean time, is this the test case you are talking about:
    keyword;comment;keyword
    where ; is comment start and end

    And I don't even know what to say about delimiters not working at all.
    I will check what went wrong, and put here version that does work.

    One more time, thanks for warning me about it.

    BR
    Loreia

     
  • Loreia2
    Loreia2
    2011-07-31

    Hi Vera,

    I just recompiled a new project (source code of latest npp 5.9.3 with files posted on patch tracker), and it works just fine:



    You can see that setting " and ; as operators has no effect on their function as delimiter and comment marks, and no spaces are needed.

    Can you show me what you get?

    BR,
    Loreia

    P.S.
    I forgot to answer, the old drop-down menus are useless in new UDL because you can use whatever you want as operator (or delimiter for that matter). So, any of this can be an operator:
    !
    %
    &
    a
    b
    and
    or
    not
    or_any_long_sentence_without_spaces

     
  • Loreia2
    Loreia2
    2011-07-31

    Great, there are no preview or edit buttons on forum.
    Here is another try to post pictures from previous post: 

    first

    second

    I hope it works this time

     
  • Loreia2
    Loreia2
    2011-07-31

    OMG, am I really this tired?

    Ok, just the second pic this time

     
  • Loreia2
    Loreia2
    2011-07-31

    I get it, damn "www" string is confusing forum software.

     
  • Loreia2
    Loreia2
    2011-07-31

    Hi Vera.

    Here is the new recompiled binaries for testing (just in case something was wrong with the old ones)

    http://www.mediafire.com/?gd36i3mirolw8qv

    You must have installed latest 5.9.3 version of Notepad++ (forgot to mention this the last time)

    BR
    Loreia

     
  • cchris
    cchris
    2011-07-31

    If you wish to merge your change with my Improved UDL handling and corrections patch, please feel free to do so. Some of the features are the same as yours.

    CChris

     
1 2 3 .. 26 > >> (Page 1 of 26)