Create User-Defined Language

flashkid
2010-05-30
2012-11-13
  • flashkid

    flashkid - 2010-05-30

    Hey guys.
    Today I've tried to set up an User-Defined Language for AutoHotkey. It  works pretty well, but I've got two problems left:

    1. I've set the Folder open and close keywords to { and }. They should only be monitored when nothing stands before them. But they are also recognized when a number stands in front of them. Example:

    If (a = b)
    {
    Send, {Left 3}
    Send, {Down}
    }

    This should collapse from { to } but it collapses from { to 3} :( How can I fix this?
    It works with many predefined languages, like C, C++, C#, Java and so on.

    2. The next problem is, that within a block of

    If (a = b)
    {
    Send, xyz
    }

    the two correspondending braces gets not highlighted, but in one line, like

    Send, {Left}

    they get. How to solve this? This also works well in predefined languages :(
    When I set the { and } as operators the folding doesn't work anymore.

    Why can't user-defined languages have the same features as built-in? I have  tried to add my language directly into the langs.xml file, but then it doesn't get recognized :(

    I hope somebody can help me to built a nearly perfect language file :)

     
  • cchris

    cchris - 2010-05-31

    You are hitting limitations of the user defined languages framework, and they are not trivial to fix.

    Predefined languages have one "lexer" each, ie a piece of code that performs the colourig and folding for that language. All UDLs share the same, parametrised lexer. Finding a lexer that would match all possible languages is hopeless, and the interface would be a nightmare if it were possible to design it.

    You are left with writing an extrnal lexer plugin to support AHK Instruction to this effect are ti be found in
    http://sourceforge.net/apps/mediawiki/notepad-plus/index.php?title=Plugin_Development. and related plugin resources.

    CChris

     
  • flashkid

    flashkid - 2010-05-31

    Thanks for your fast reply, CChris. Unfortunately I can't write a lexer, because I can only write AHK programs and a little bit of batch and lazarus (free pascal).
    Can't the Folding options be done with a RegEx, like it is done in the function list plugin? That would be the same for all User-Defined languages.

    So the lexer for the predefined languages are hardcoded in the notepad++.exe and not in some xmls or such? Can't I change a standard lexer to support AHK in a better way and learn to understand writing a lexer?

    It would be so great if I could find a solution :(

     
  • cchris

    cchris - 2010-05-31

    I have tried implementng regex based keywords, and hit a technical issue with Scintilla reentrancy. I'll have to look back into it.
    Also, if Notepad++ starts using an external dll (pcre) for regular epxpressions, that should work fine.

    The lexers are hardcoded, but Notepad++ is open source, and you can download the sources from the official page - or
    check it out from SVN for bleeding edge code.
    Then you'll see plenty of Lex(langname).cxx files, each of which hosts a lexer. That will help you understand how it works.
    Free pascal is ok to write an external lexer. If AHK can produce a dll, that's fine too.

    CChris

     
  • flashkid

    flashkid - 2010-05-31

    I would much appreciate it, if you get this working :)
    I'll look at the code and try some things. Don't really know if I can produce a DLL file with AHK, I'll see ;)

    Many thanks for your replies.
    flashkid

     
  • flashkid

    flashkid - 2010-05-31

    I'm back :) I've tried to recompile np++ to test around with a lexer, but all I get is an error of Visual Studio Express 2008:

    Kompilieren…
    AboutDlg.cpp
    ..\src\WinControls\AboutDlg\AboutDlg.cpp(18) : error C2859: c:\dokumente und einstellungen\ncs\eigene dateien\visual studio 2008\projects\powereditor\visual.net\unicode debug\vc90.pdb ist nicht die pdb-Datei, die beim Erstellen dieses vorkompilierten Headers verwendet wurde. Erstellen Sie den vorkompilierten Header neu.
    ..\src\WinControls\AboutDlg\AboutDlg.cpp(18) : error C2859: c:\dokumente und einstellungen\ncs\eigene dateien\visual studio 2008\projects\powereditor\visual.net\unicode debug\vc90.idb ist nicht die idb-Datei, die beim Erstellen dieses vorkompilierten Headers verwendet wurde. Erstellen Sie den vorkompilierten Header neu.

    In english ~ :

    Compiling..
    AboutDlg.cpp
    ..\src\WinControls\AboutDlg\AboutDlg.cpp(18) : error C2859: c:\dokumente und einstellungen\ncs\eigene dateien\visual studio 2008\projects\powereditor\visual.net\unicode debug\vc90.pdb is not the pdb-file, that was used for the creation of the precompiled header. Create the precompiled header again.
    ..\src\WinControls\AboutDlg\AboutDlg.cpp(18) : error C2859: c:\dokumente und einstellungen\ncs\eigene dateien\visual studio 2008\projects\powereditor\visual.net\unicode debug\vc90.pdb is not the pdb-file, that was used for the creation of the precompiled header. Create the precompiled header again.

    How can I fix this?

     
  • RaptorX

    RaptorX - 2010-05-31

    Nice to see other people in the same mission as I am…

    I created a modded version of the scilexer.dll including the lexAHK1.cxx from PhiLho (http://www.autohotkey.com/forum/topic9656.html) which is a great lexer for AHK.

    For compiling that you just need to edit the ahk script to specify the location of the scintilla sources, and at the end you have to modify sciLexer.h to correct the definition of SCLEX_AHK, because the script uses a regex to add that line and for some reason it doesnt add the number at the end (should be #define SCLEX_AHK 101, or something on those lines).

    The only issue is that even though I finally compiled the .dll I would have to manually modify the notepad++ code to include that language in the language list and I am not good at C++ (yet) so I think that is well beyond what i can do.

    I tried with the External Lexer plugin and Lexer Template, and I havent been able to compile neither of them, maybe because im missing some libraries and as there is virtually no documentation or help on that  I am stuck there…

    It would be great if there was a little documentation on how the lexers are added in to the N++ interface… I dived through the code and even though I "almost" made it (from like 200 errors while compiling to like 3) I still cannot make N++ compile, well at least I got the scilexer.dll which I can use with SciTe with the help of a properties file, but I really love N++ and would like to see an AutoHotkey Lexer implemented. :D

     
  • flashkid

    flashkid - 2010-05-31

    That's exactly what I want to do as well. Downloaded the external lexer too, but I can't compile anything :(
    It would be great if someone can provide a step by step manual on how to add a custom lexer to np++.

     
  • RaptorX

    RaptorX - 2010-05-31

    actually the txt file that comes with NppExtLexer plugin is very good at explaining how to create it, the problem comes when the author doesnt mention the environment that he setup for compiling it, who knows if he used some non standard headers… but hey, AHK is growing, I think we can think of our own lexer being included in N++.

    note to CChris:
    The Lexer code is already developed, it is just matter of how to make N++ add it to the list of current languages. Any help on that would be appreciated! Already coded Lexer can be found here: http://www.autohotkey.com/forum/topic9656.html

     
  • cchris

    cchris - 2010-06-01

    About the .pdb file errors:
    - These files are out of sync, deleting them, or regenerating the whole solution, just works - that's my experience at least.

    @nikronius
    You just add the external lexer to Notepad++ as a normal plugin. It is just that the plugin does special things, which you'll notice in the language menu, as the new language will be added there (at least if using NppExternalLexers).

    What are the errors you get? If an include is missing, the compiler should be reporting that.

    Adding an internal lexer to SciLexer.dll requires amending Notepad++, and specially defineDocType() in ScintillaEditView.cpp. Looking at the code for some already supported languages should be self-explanatory.

    CChris

     
  • flashkid

    flashkid - 2010-06-01

    Got it with a test file. Try it now with my User-Defined Language and the lexer by philho.

     
  • RaptorX

    RaptorX - 2010-06-01

    @flashkid

    Im not sure what you want me to try. :D

    @CChris

    I download the sources for NppExternalLexer and MyLexerTemplate (the previous stages of NppExtLexer) and tried to compile without making any changes except adding the additional include folders:

    \NppExternalLexer_Plugin\src
    \PowerEditor\src\MISC\PluginsManager
    \scintilla\src
    \scintilla\include
    \PowerEditor\src\WinControls\ColourPicker

    \MyLexerTemplate
    \scintilla\include
    \scintilla\src
    \PowerEditor\src\MISC\PluginsManager"

    these are the results:

    Initial Try with MylexerTemplate:
    http://pastebin.ca/1875404

    Initial Try With NppExternalLexer:
    http://pastebin.ca/1875406

    I added #include <windows.h> to one of my header files in both projects and the errors were less.
    I had to #include <iostream> to MylexerTemplate as well.

    Second Try MyLexerTemplate:
    http://pastebin.ca/1875407

    Second Try NppExtLexer:
    http://pastebin.ca/1875408

    And here is where I am stuck, I am not sure where PropSet is defined, I googled around and I read something about scilexer.lib but I am not sure how to point at it or how to add it to my projects, so I am not able to compile none of those plugins which would certainly help me adding the LexAHK.cxx because as you pointed out the steps are really simple for someone who knows C++
    but as I mentioned before my level of C++ is not very high. So any help on that would be appreciated.

     
  • cchris

    cchris - 2010-06-01

    The fact that you have #include statements does not cause VS to include the files in its gneration process - DUMB. You have to go to the Project menu and manually add all the .cpp and .h that NppExtLexer specifically uses. At this point, you'll need to remove your inclusion of Winfow.h as it will have become redundant.

    The warnings you get are suppressed by a directive in win32\Scintilla.mak. This suggests that VS is noy using it, and hence that your project settings may need a review.

    Normally, you should be left with errors of your own code.

    CChris

     
  • RaptorX

    RaptorX - 2010-06-01

    you know what?

    The problem is with the lastest vb projects of N++.
    I downloaded the source for version 5.0, and both plug-ins compiled without errors, I didnt had to add anything else either.

    It is as you said there is a configuration problem, but it is from the new sources. Thanks anyways for the help.

     
  • flashkid

    flashkid - 2010-06-02

    That is what I found out in my last post too. The problem with the new sources is, that the "notepadPlus.vc.7.0.vcproj" file isn't included anymore (I think because it isn't used for a couple of releases, see last change date).
    With old sources it works very well and I got a simple lexer for AHK. But I have no skills in C++, so I can't make a real big lexer and the lexer by philho is not like the syntax for external lexers.
    Maybe I will succeed while trying around, but I have no real hope for that :D

     
  • flashkid

    flashkid - 2010-06-02

    Thanks nikronius, thats pretty cool :)
    I've got a few questions:
    1. Have you compiled the lexer by philho or build a language with the external lexer plugin?
    2. Did you change philhos code?
    3. Could you please upload your projects folder of visual c++?
    4. The folding does not work correctly, thats kind of bad :( But it's great, that it supports so much things, even labels are recognized :)

    Thanks for your work, would never have done it by myself^^

     
  • RaptorX

    RaptorX - 2010-06-02

    I built a language plugin with the External Lexer Plugin using PhiLo's Lexer.
    I did really minor changes to his code due to conflicts on Variables names, but no changes to the actual folding and coloring codes.

    my project files can be found here: http://www.4shared.com/file/bdzqOfHv/NppAHKLexer.html
    It is actually the default projects for Npp 5.4.3 and NppExtLexer 1.0 but when you open the NppExtLexer project you have to remove the additional projects and re-add them make sure you select the normal ones, not the "vc.7.0.vcproj" ones.

    Also you MUST build the SciLexer project before the NppExtLexer because some things are linked to the SciLexer libraries, so they must be built first (in the file I sent you they are already built).

    About the folding bugs, it is not that it bugs all the time, it works fine but some times it has this issue that it doesnt know where the folding ends. I have noticed it several times when there is nested folding but I am not sure if it is PhiLo's code that has issues or if it is because I imported it to N++ via the NppExtLexer. I will take a look on to it later.

    I am planning on making AHK tutorials and if you still have issues setting up the project I can make a quick tutorial showing how I compiled the Lexer. :D

     
  • flashkid

    flashkid - 2010-06-02

    Thanks for the files.
    I don't need a tutorial, I had compiled it myself before, but haven't got it complete with philho's lexer :)
    That will bring the ahk community a bit further, but I would much appreciate it, if AHK gets a native lexer in np++ :)
    What AHK tutorials do you plan?

     
  • RaptorX

    RaptorX - 2010-06-02

    It is true, AHK should be supported by N++ but I think one of the requirements is that it should be officially added to the Scintilla and then the N++ guys will be working on adding it to their application, so we should try to make it happen over the Scintilla forums first.

    about the AHK tutorials, I have been noticing a lot of people in the IRC channel asking the same things that are in the manual everywhere and I figured that a lot of people (me included) learn more when they see how to do it than reading some confusing paper (not that the manual is confusing but it has some tricky parts). So I will start with the easy part but then I will go more in to the advanced stuff as time and family allows.