New version removes old limitations and introduces some new features, namely:
1. Operators and delimiters can now be longer that one character
e.g.
old operators: ! < >
new operators: !! << >> """
2. Added full support for three set of delimiters
Old dialog allowed only two delimiters to be selected, even though xml file could save three delimiter sets. New dialog is extended to allow selection of third set of delimiters
3. New algorithm works even when there is no whitespace separator of two elements.
e.g.
old algorithm could not colorize correctly:
/*comment*/ (no spaces within comment)
keyword!delimiter! (no spaces after the keyword)
(This was the main thing for me, and the reason I started working on this)
4. Export button works only if new definition is first saved with "Save As…" button, but old dialog would not warn user about it. (This left me scratching my head why Export button suddenly stopped functioning.) New version displays MessageBox that notifies user to save definition before attempting to export it.
5. If you define the same characters as comments AND operators, colorization still works.
e.g.
comment start: /*
comment end: */
operators + / - *
This is possible because comments are checked first.
(That also means that if you define excatly the same character/string as both operator and comment, it will always be treated as comment.
e.g.
comment line: !
operators + ! / - *
In this case any occurence of ! character will be treated as start of line comment.)
6. Last word is colorized even if file does not end with new line.
7. No visible performance penalty (on my machine it takes roughly the same time to colorize few hundred thousand lines with old and new version)
8. Also, some small bug fixes are included, like tab order in all four UDL dialogs or escape character box being initialized with garbage because of a missing _T() macro.
9. New UDL has no use for "Treat keywords as symbols" option, so it was removed.
I've done extensive testing of new algorithm, and I **think** I've covered everything. Now I need community support to test it more thoroughly.
Basically, I need you to:
1. download new version of Notepad++ (zip archive with two files: notepad2++.exe and SciLexer2.dll)
2. save it to your installation folder (unzip it off course, nothing is overwritten because files have different names than defaults)
3. close notepad++ if it is running
4. double click new file notepad2++ to start it
(this is a Unicode release build, if you need ANSI build let me know)
5. go to Search/User-Defined Dialogue…
Here it is: http://www.mediafire.com/?fx3dshychsg8v57
(sorry about using media-fire, but Sourceforge is so slow, it is barely functoning for me, and I see no option to upload files)
Try to import old definition (if you have one), to see if everything works correctly. Test everything in "Operators" tab. (Just remember that translation will not work here because old translation files
don't have translation for new GUI elements.) Test export function too.
Now try to see if colorization works. You can use your old code files, or quickly write something down, or you can use my "quick'n'dirty" python script to generate test cases (you'll need to install python version 2, e.g. 2.7 http://www.python.org/download/).
Note:
This script is a bit ugly (it was not intended for publishing), but it gets the job done. Feel free to extend it anyway you see fit.
Also, It will generate some 20 or so *.loreia files, with thousands of test cases (but you'll scroll through them fairly quickly)
Colorization for test cases can be imported from file "loreia.xml", import it, and it will give you another language named "Loreia".
(And no, I am not an ego maniac, I just lack creativity at 23:00h :-))
Here is the script: http://www.mediafire.com/?aryr7meyp1k1qph
If you find anything wrong, just post it here and I'll fix it.
Thanks a lot,
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
Your tool is great and awesome,
I have a small question,I haven't readthe 19 page sorry,
But after lot of research, I don't fine any information about the Regex.
Do your tools take account of regular expression ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hey,
I've been loving the new UDL(2), But I have a major issue of coloring text.
I suspect the problem is the fact that the delimiter I am inserting is being treated as ANSI while my file itself is UTF16.
You can see the problem in the attached document, all the white/blue colored xx are actually a single ¥ character, when a ¥ is being inputted as an operator list for styling, it screws up the display (this also happens for trademark signs such as ® and 'c' ).
I would really appreciate some help with this as it is causing us a nightmare of reading our files ;]
in your case an escape character is needed, say '\\'.
Then you can write: { this is a comment { and so is this \} }
I am not aware of any language that supports nested comments, but if there are such languages, and if it is a popular request, I guess I could add that feature in.Off course, one more option in GUI would be needed. Something like: "Allow nested comments".
But, I don't think this feature is part of any language.
If I am wrong, please put some links here.
BR
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ok, I'm convinced. Next revision will have support for nested comments. I'll fix LexUser.cxx latter today (and post it on patch tracker). But it might take me few days to patch UDL dialog (due to my inexperience with NPP code, and some obligations I need to finish this weekend), even though it is just a single check box to be added.
BR
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I really appreciate your effort and wish you success with it :-)
But unfortunately your new UDLversion breaks two major highlings for me
1a. the commenting: because you removed "Treat keywords as symbols", my commentsymbol ';' now MUST have a 'space' behind it
1b. for the same reason, that commenting symbol may now corrupt following code, when it is used inside a dedicated "STRING"
2. the one and only delimiter I need: " (to assign "STRINGS") cannot be made to work again.
In my udl-setting the " is as well assigned as operator plus used as delimiter and works fine.
Now whatever I tested as combination: being operator or not, doubling the operatorsymbols, taking first/middle/last or mixed delimiterfields … regarding blank or not … nothing will return the "String" highlighting.
Additionally I also miss the specific verification of what operators are available/enabled at all, (which is easily identifiable from the facing boxes) - or is that meant to be totally free in your edition?
sorry for not having splendid replies, but maybe others might cross the same issues with their own language
regards ~ Vera
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thanks for taking time to test it.
What you describe is a basic test case new UDL was designed to solve (no spaces are needed as separators). If something as simple as keyword;comment;keyword is not working, that means I've screwed my last build. I can assure you that something as basic as that was working when I tested it.
I guess working without version control is a bad thing (and working late at night is even worse). I will review code changes, and post here what went wrong.
In the mean time, is this the test case you are talking about:
keyword;comment;keyword
where ; is comment start and end
And I don't even know what to say about delimiters not working at all.
I will check what went wrong, and put here version that does work.
One more time, thanks for warning me about it.
BR
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I just recompiled a new project (source code of latest npp 5.9.3 with files posted on patch tracker), and it works just fine:
You can see that setting " and ; as operators has no effect on their function as delimiter and comment marks, and no spaces are needed.
Can you show me what you get?
BR,
Loreia
P.S.
I forgot to answer, the old drop-down menus are useless in new UDL because you can use whatever you want as operator (or delimiter for that matter). So, any of this can be an operator:
!
%
&
a
b
and
or
not
or_any_long_sentence_without_spaces
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you wish to merge your change with my Improved UDL handling and corrections patch, please feel free to do so. Some of the features are the same as yours.
CChris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ups - I didn't expect you to be that quick ;-) I had an oldfashioned sunday afternoon with coffe & car-racing
To give you a first quick reply: it works_ (!)_
BUT I found another issue.
When running your latest version it even looked less colorfull, because now the whole commenting failed. I couldn't believe it and was a bit paralized while staring at the UDL-Gui.
Then I saw my marked option : 'IGNORE CASE'
- then I deselected it and all highlightings appeared as wanted. :-)
Also that: keyword_;comment_ works
likewise an 'embraced' comment-sign ";" doesn't corrupt following code anymore
Ok - test 'IGNORE CASE' and see what it does to you
and I'll have a look if I can find another hidden itch and test a priv_at_long_delimiter ;-)
~~~~~~~~~~~~~~~~~
Btw: I also cross-checked your two versions and found out
a - unselecting 'IGNORE CASE' in the previous version wouldn't help
B-ut - the previous EXE combined with the newest DLL would work like both NEW do
And yes - I used the current 5.9.3 uni for a start.
And on the long - I would be very interested in an favoured ANSI version as well ;-)
Hint @ALL: if you download the current UDL-version rename Notepad++.exe -> to Notepad2++.exe, for not to override the original by accident.
have a nice sunny night and a summerly week ~ Vera
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thanks a lot for your last post. It was VERY helpful. It seems I was so concerned testing everything in operators tab, that I forgot to test Ignore case option. It was indeed causing errors, and after a quick debugging session today, I discovered a bug in Scintilla. (I just went to http://www.scintilla.org to report it, but I see it is fixed in latest 2.28 version, version 2.27 that NPP is currently using still has it.)
Anyway, in StyleContext class (StyleContext.h file), function
bool MatchIgnoreCase(const char *s) {
if (MakeLowerCase(ch) != static_cast<unsigned char>(*s))
return false;
s++;
if (MakeLowerCase(chNext) != static_cast<unsigned char>(*s))
return false;
Does not work for a single character strings.
It should instead be:
bool MatchIgnoreCase(const char *s) {
if (ch != static_cast<unsigned char>(*s))
return false;
s++;
if (!*s)
return true;
if (chNext != static_cast<unsigned char>(*s))
return false;
I will make a bug report to NPP bug tracker a bit latter, so this could be fixed regardless of would Don and NPP team accept my changes to UDL.
So, I have added support for nested comments and fixed Ignore case bug in latest NPP as a result of this thread.
Note that I hard-coded nested commenting feature in, so you cannot disable it (I will update dialog with this feature, but for the time being, l'll keep it hard-coded in. For testing with nested comments disabled, just use dll file in subdirectory called "nested_comments_deselected" (you need to copy it to the same directory where notepad2++.exe is.)
My next task is to study Chris' code better, so I could incorporate his changes into new UDL.
Speaking of that, Chris, can you help me with these options:
* Fix replication of prefix options on creating new language
* Fix cross styling that would take plce when changing an UDL and the active document uses a different UDL
Do you remember what was this about? I am not sure I understand it fully.
Also this:
* Surrounding a keywords with double quotes will allow it to have multiple parts. One space is equivalent to any number of spaces and tabs.
* In prefix mode, if a multiple part keyword ends with a trailing space, the next word will be part of highlighted text. Such a trailing space is ignored if prefix mode is off.
This is the thing I will "attack" first. But why is this needed? Maybe for writing pseudo-code?
BR
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
A/
1/ The 4 prefix options (one per kw group) were not always carried over properly when creating a new language from existing.
2/ If you had a document opened using UDL A, open the UDL panel and change style parameters of UDL B, that would wrongly affect active documùent as it justs reports L_USER language.
B/
1/ Defining the "key word" keyword will highlight
key word
key <tab> word
and so on
2/ If prefix mode is checked for the group, "key word " will match
key word begin
key word <tab> fin
and so on. The trailing space(s) inside double quotes is ignored when prefix mode is off.
CChris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
just a quick update so this would not seem like a dead thread.
I am working on code folding right now. Progress is a bit slow due to my lack of free time these days.
So far folding forks more or less as expected, and my next step is to clean the code a bit (but optimize it and shrink it), and to add support for Folder Open and Close keywords in comments. That would allow users to fold any block of code just by creating comments like:
// foldingOpen
line 1
line 2
…
line n
// foldingClose
even in languages that don't have code folding built in. Basically, I am trying to mimic fantastic #region statement from C#.
Anyway, this is what I have done so far.
Picture should be self explanatory, but if anyone has any questions, just post them here.
Best regards,
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am yet to analyze all the stuff CChris wrote for UDL. There is a lot of great stuff in there, and I intend to merge all that stuff with my code. When finished, UDL will be really powerful stuff (not that it isn't powerful already, but this will be the next step)
Best regards,
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
@loreia2:
I'm keenly aware that my code is just as densely commented as Notepad++'s proper, which you may feel lacking. If in doubt, just don't hesitate to ask me about the code at quasar850 ta rediffmail tod com
CChris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2011-08-23
I've recently begun expanding my coding horizons with a number of languages and notations that benefit from syntax highlighting (like Backus Normal Form), and I just wanted you guys to know that I am absurdly excited to see that people are working on improving the language-definition features. There are just so many things to highlight out there (BNF! Redcode! LOLCODE! MediaWiki! Brainf***!) and most of us aren't quite skilled enough to code up a NP++ plugin.
So seriously: Thank you. Your hard work is 100% appreciated, and it will be used.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
excellent, depending on my schedule next few weeks, I will send you an e-mail or two. I've only skimmed through your code (basically, I just read .patch files directly in Notepad2), and code looks just fine. I guess I won't have much problems reading it, but it can save me few hours if some details can be explained within context of NPP.
LexUser.cxx is what I am familiar with at this point, so my questions will probably be more about UseDefineDialog.cpp and saving stuff to .xml file.
Anyway, thanks for your help.
@Poster after CChris
Thank you for your support and appreciation. I guess I only wanted to add support for comments without spaces (which was the only thing that was breaking for me), and then somehow I got hooked :-)
It took complete rewrite of coloring algorithm to work in comments without spaces, and now I just want to expand UDL with other stuff. It is fun, but it will have to go relatively slowly because I need to balance this work with my regular obligations (which are numerous).
But I must admit it is fun to see UDL with new features added in. In fact everywhere I look now, there is always somewhere in my mind something like: "Can I add this to UDL?"
E.g. yesterday I read a blog post http://www.elpauer.org/?p=971 (it popped up on planetKDE), and I already have a new task for this weekend: adding support for line comment continuation. Funny thing is, I believe line comments continuation to next line should be outlawed, but I forgot that we have some stupid in house scripting tool that allows this "feature" and people use it extensively. It will be great if we can finally highlight that stuff properly.
Also, if you have an idea (based on one of those languages that you mention) what could be added to UDL, just post it here. Someone suggested to add support for nested comments, and when I added that in and mixed it with comment folding, I created something that gave me an idea how to significantly speed thing up when folding code. When I finish optimizations I believe speed increase (when there are many folds) will be noticeable in FoldUserDoc function.
So, don't be afraid to discuss possible new features, good ideas are always welcome.
Best regards,
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2011-08-24
(Same user here; I finally bothered to fix my name.)
Well, the search that brought me to this thread was "nested delimiters." Specifically, I was interested in taking something like:
…and (for instance) coloring anything between <> delimiters while also bolding anything between {} delimiters, without completely overriding the formatting of the nested delimiters.
I present this more as a brainstorming spark than a "feature request," since the mainstream utility is dubious and the implementation would be complex - after all, string literals must override formatting, and you would have to figure out how to know which part of the formatting to selectively apply, and probably some other stuff I'm forgetting. On the other hand, it would synergize nicely with the vastly expanded options you've allowed for delimiter definitions.
Other than that, though, I think you may have already addressed most of the frustrations I've come across in the last few weeks. To keep things brief, I'm just going to point to everything you've done with operators and delimiters.
On a general note, I'm curious - I'm sure everyone would love to define an arbitrary number of keyword sets, delimiter sets, and so on, but we obviously can't. Is that limitation in the UDL dialogue, in the Notepad++ code, or somewhere else?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Nested delimiters are interesting idea, I can see how that would be useful.
It would be relatively easy to implement it for coloring of entire document, but it is a challenge to do it when re-coloring document while user is editing. You see, in case of { <addop> <term> }, if user is trying to change word "addop", I need to know that this is a say Delimiter2, but after next ">" I need to continue with Delimiter1. And if "addop" contains another subdelimiter like <add !delim3! op>, I need to carry that information too. I would need to use some STL container here, because I cannot know how deep it goes.
I'll think about it, but if code gets too complex, I'll probably skip it for time being.
Regarding arbitrary number of keywords and delimiters, I would have to dig deeper in Scintilla to see if that could be expanded, but that would be a big (really big) change. Dialog would have to be redesigned, and Coloring function would have to significantly chance its logic (keywords and delimiters would have to be checked in a while loop). This would be really unpractical. Besides, I'm surprised that you have exhausted even this amount keyword and delimiter sets (typically, I use just one or two sets)
So, I'll see about nested delimiters, but you are stuck with 4 keyword sets, and 3 delimiter sets for now.
Best regards,
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2011-08-26
Haha, no worries. If it seems like I'm talking about stuff that's completely out there, I assure you it's only because I can't think of many practical features you haven't already addressed. Keep up the good work!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2011-08-27
Oh, I've thought of something potentially useful, though I suspect it's utterly out of your hands: It would be lovely to be able to control the tab settings (TAB character vs. spaces, TAB width) for user-defined languages.
If I'm correct in my assumptions, then this would require completely new functionality (much like adding keyword sets), but I thought I'd throw it out there anyway.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Adding a tab settings: you need to persist the setting, display it on the dialog and maintain consistency, and then… get it applied. Notepad++ treats all user defined languages as same because they have the same lexer witth ID L_USER. And it will treat tab/spaces issues accordingly. So now we'd need to change this behaviour so that L_USER files are inquired for their actua language and then proper tab settings be applied. A better, but entailing more radical changes, idea is to have a range of individual language IDs associated to the single L_USER lexer.
Aout nested delimiters, there could be a simple way to implement them using Scintilla's indicators. Now the trouble is to include indicator details in both the definition file and the UI.
CChris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2011-08-28
Interesting. It may be helpful to explain the actual use-case that's been spawning a lot of these questions; it may spark some ideas on your end.
As a simple example, I've been working with the MIPS assembly language (which is quite different from Notepad++'s default Assembly language). A very simple program might look like this (with 8-space tab characters):
## Program to add two plus three
.text
.globl main
main:
ori $8,$0,0x2 # put two's comp. two into register 8
ori $9,$0,0x3 # put two's comp. three into register 9
addu $10,$8,$9 # add register 8 and 9, put result in 10
## End of file
As it stands, there are basically two options I know of to get the highlighting and formatting one would want for this language: Modify/extend the Assembly preset, or create a UDL from scratch. The downsides I have found thus far for each approach are:
Assembly Preset
No way to change line comment character from ';' to '#' (making comments incredibly distracting)
No way to set '$' as a symbol ("$8" is highlighted as an identifier, rather than a symbol and a number)
UDL
No support for highlighting hexadecimal numbers
No ability to set unique tab settings (8-space width TAB character is hardly appropriate for every UDL)
Most of this is out of scope for the UDL project, but I think that hexadecimal number support would be quite welcome and (hopefully?) easy to implement.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
They somehow are. See my patch about Extra Characters for Numbers. It has a broader scopen, as it also handles cureency/format signs and readability separators between digits.
Don didn't like it originally because it would add 4 text fields on the number tab. The fieds re now on on a separate Characters tab.
With this enhancement, and after filling the relevant properties, hex support would be in UDLs.
The fields are:
* Allowed refixes for prefixed numbers. For MIPS that would be $
* Additional chars in numbers. Not used there, or perhaps : for segment:offset addresses, if that makes sense.
* Additional characters in prefixed numbers. That would be abcdefABCDEF
* Optional suffix characters: probably not used here. Targets things like 27€ or 3.6f .
Consider reconsidering, DOn?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I would just like to add that I loved Chis' patch for Extra Characters for Numbers. I LOVE the flexibility of it. And as soon as I finish my work on comments and subsequent code optimizations, I plan to further study that patch and include it in my version of UDL. I am not sure if I would just straight copy/paste it, or modify it a bit (e-mail consultations on that will follow), but that part of code is the next one on my radar.
Best regards,
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Loreia, could you please say if you have some plans to introduce multipart keywords into UDL? :)
There were some older patch from Chris, but unfortunately I'm not really good to patch and use it.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
yes, I plan to review all of Chris' UDL patches, and then integrate them with my work. There are several of them, and so far I've just skimmed through the list, so it is hard to discuss the details at this point, but my plan is to go through entire list.
Obviously, some changes will be required in GUI, as it might be a bit "messy" to just put new elements on already crowded tabs.
Add to that my busy schedule, and you'll see that it will take some time to do it all. But the plan is to do it. I'll post intermediate binaries here for testing (and so impatient ones can use it if they can't wait for official release)
I hope, I'll have some news on it this weekend.
Best regards,
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Loreia, I'm very thankful for your answer and also for your great things in the field of UDL improving! :)
It is a really great job!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2011-08-30
Agreed, top marks all around.
Re: GUI
Yeah, I definitely sympathize with the pain of trying to fit yet more features into the UDL dialogue. In the current NP++ release, it is literally impossible to set the background color for Delimiter 2 without hiding the Windows taskbar (Vista, 1440x900). That's not a bug fix request; just an illustration of how vertically stuffed the dialogue is.
Re: Extra Number Characters
That's straight-up awesome.
Re: CChris
I'd like to say thanks for providing just about every single useful response concerning NP++'s syntax highlighting I've ever found with Google. That's not directly related to the UDL stuff, but it's true.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
is there anything about UDl that you didn't improve? :-)
I can't believe nothing made its way to main build (of all the UDL patches you've submitted).
Anyway, you made an interesting suggestion with your UI redesign, but for the time being, I'll just slightly redesign UDL GUI by repositioning and resizing existing elements (and adding few of my own in the process). The idea is to reduce vertical clutter, add few more "boxes", and avoid changing much in UserDefineDialog.cpp.
Basically, I don't want to spend too much time on GUI, as I am fully focused on UDL algorithm. I just need something that works for now. Latter, when algorithm is fully developed, any UI can be "soldered" on top of it.
Regarding UDL algorithm, optimizations start to show some progress, last revision is measurably faster than official build. I did just one test, on a fake 300.000 lines long file (with lots of folding), and official build took 2min 5sec to do the task, and new version took 1min 15 sec.
Code is still messy, but this test brought smile to my face, because results are promising.
Until next update,
Thank you and best regards,
Loreia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I am excited seeing this, I was working on this https://github.com/blakmatrix/CoffeeScript_notepad_UDL for coffeescript and thought I had gotten it as good as possible within the current limits. Any plans to address issues I cane across?(i.e. regex detection, embedded statements, and if # is a comment and #{@var} is a container, and also different number format detection hex octal(0 as a prefix) decimal ect)
Keep up the good work!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi everyone,
I wrote a new UserDefineDialog colorization algorithm.
Patch report is submitted to http://sourceforge.net/tracker/?func=detail&aid=3382886&group_id=95717&atid=612384 for review and possible inclusion in official release.
New version removes old limitations and introduces some new features, namely:
1. Operators and delimiters can now be longer that one character
e.g.
old operators: ! < >
new operators: !! << >> """
2. Added full support for three set of delimiters
Old dialog allowed only two delimiters to be selected, even though xml file could save three delimiter sets. New dialog is extended to allow selection of third set of delimiters
3. New algorithm works even when there is no whitespace separator of two elements.
e.g.
old algorithm could not colorize correctly:
/*comment*/ (no spaces within comment)
keyword!delimiter! (no spaces after the keyword)
(This was the main thing for me, and the reason I started working on this)
4. Export button works only if new definition is first saved with "Save As…" button, but old dialog would not warn user about it. (This left me scratching my head why Export button suddenly stopped functioning.) New version displays MessageBox that notifies user to save definition before attempting to export it.
5. If you define the same characters as comments AND operators, colorization still works.
e.g.
comment start: /*
comment end: */
operators + / - *
This is possible because comments are checked first.
(That also means that if you define excatly the same character/string as both operator and comment, it will always be treated as comment.
e.g.
comment line: !
operators + ! / - *
In this case any occurence of ! character will be treated as start of line comment.)
6. Last word is colorized even if file does not end with new line.
7. No visible performance penalty (on my machine it takes roughly the same time to colorize few hundred thousand lines with old and new version)
8. Also, some small bug fixes are included, like tab order in all four UDL dialogs or escape character box being initialized with garbage because of a missing _T() macro.
9. New UDL has no use for "Treat keywords as symbols" option, so it was removed.
I've done extensive testing of new algorithm, and I **think** I've covered everything. Now I need community support to test it more thoroughly.
Basically, I need you to:
1. download new version of Notepad++ (zip archive with two files: notepad2++.exe and SciLexer2.dll)
2. save it to your installation folder (unzip it off course, nothing is overwritten because files have different names than defaults)
3. close notepad++ if it is running
4. double click new file notepad2++ to start it
(this is a Unicode release build, if you need ANSI build let me know)
5. go to Search/User-Defined Dialogue…
Here it is:
http://www.mediafire.com/?fx3dshychsg8v57
(sorry about using media-fire, but Sourceforge is so slow, it is barely functoning for me, and I see no option to upload files)
Try to import old definition (if you have one), to see if everything works correctly. Test everything in "Operators" tab. (Just remember that translation will not work here because old translation files
don't have translation for new GUI elements.) Test export function too.
Now try to see if colorization works. You can use your old code files, or quickly write something down, or you can use my "quick'n'dirty" python script to generate test cases (you'll need to install python version 2, e.g. 2.7 http://www.python.org/download/).
Note:
This script is a bit ugly (it was not intended for publishing), but it gets the job done. Feel free to extend it anyway you see fit.
Also, It will generate some 20 or so *.loreia files, with thousands of test cases (but you'll scroll through them fairly quickly)
Colorization for test cases can be imported from file "loreia.xml", import it, and it will give you another language named "Loreia".
(And no, I am not an ego maniac, I just lack creativity at 23:00h :-))
Here is the script:
http://www.mediafire.com/?aryr7meyp1k1qph
If you find anything wrong, just post it here and I'll fix it.
Thanks a lot,
Loreia
Hi,
Your tool is great and awesome,
I have a small question,I haven't readthe 19 page sorry,
But after lot of research, I don't fine any information about the Regex.
Do your tools take account of regular expression ?
UTF16 (UCS2 LE) coloring problems.
Hey,
I've been loving the new UDL(2), But I have a major issue of coloring text.
I suspect the problem is the fact that the delimiter I am inserting is being treated as ANSI while my file itself is UTF16.
You can see the problem in the attached document, all the white/blue colored xx are actually a single ¥ character, when a ¥ is being inputted as an operator list for styling, it screws up the display (this also happens for trademark signs such as ® and 'c' ).
I would really appreciate some help with this as it is causing us a nightmare of reading our files ;]
thank you very much !
Alon
Last edit: yfital 2013-01-14
This is great! I hope it gets into the main sources.
Can you also add support for nested comments?
e.g. { this is a comment { and so is this } }
Hi Alex,
in your case an escape character is needed, say '\\'.
Then you can write: { this is a comment { and so is this \} }
I am not aware of any language that supports nested comments, but if there are such languages, and if it is a popular request, I guess I could add that feature in.Off course, one more option in GUI would be needed. Something like: "Allow nested comments".
But, I don't think this feature is part of any language.
If I am wrong, please put some links here.
BR
Loreia
I just googled "nested comments", and it seems some languages do allow them.
I guess, I'll just have to add that option too :-)
I can update Algorithm in LexUser.cxx fairly easily (just need to count open and close comment strings), but I'll need some time to adapt UDL dialog.
I post here, when I finish it.
BR,
Loreia
AFAIK there are a number of programming languages that allow for nested comments, including Scala /**/, D /++/, Cobra /##/, OCaml (**) and the Oberon family of programming languages, also (**).
See here the "Block comments" table: http://en.wikipedia.org/wiki/Comparison_of_programming_languages_(syntax)
If you look at this page: http://statlab.uni-hd.de/projects/oberon/ItO/www/Oberon2.EBNF.html
you may find the text "Comments may be nested" where comments are explained.
I doubt it would be a popular demand, but it can make Notepad++ a tiny bit closer to perfect : )
Ok, I'm convinced. Next revision will have support for nested comments. I'll fix LexUser.cxx latter today (and post it on patch tracker). But it might take me few days to patch UDL dialog (due to my inexperience with NPP code, and some obligations I need to finish this weekend), even though it is just a single check box to be added.
BR
Loreia
If you've got changes in SciLexer, you ought to consider submitting them to Scintilla for inclusion in that source.
This has nothing to do with Scintilla sources.
Is this it?
cowagain,
UDL is Notepad++'s extension of Scintilla. Source code should be reported here.
BR
Loreia
loreia2,
Looks perfect. Good work!
Ok, I am glad I got it.
I will test it a bit more, and post update to patch tracker latter today or maybe tomorrow.
If anyone has additional requests, just post them here.
BR,
Loreia
If there's a version of SciLexer.dll that you're including in the package, it seems reasonable to assume changes to Scintilla source.
Hello loreia2 ,
I really appreciate your effort and wish you success with it :-)
But unfortunately your new UDLversion breaks two major highlings for me
1a. the commenting: because you removed "Treat keywords as symbols", my commentsymbol ';' now MUST have a 'space' behind it
1b. for the same reason, that commenting symbol may now corrupt following code, when it is used inside a dedicated "STRING"
2. the one and only delimiter I need: " (to assign "STRINGS") cannot be made to work again.
In my udl-setting the " is as well assigned as operator plus used as delimiter and works fine.
Now whatever I tested as combination: being operator or not, doubling the operatorsymbols, taking first/middle/last or mixed delimiterfields … regarding blank or not … nothing will return the
"String"highlighting.Additionally I also miss the specific verification of what operators are available/enabled at all, (which is easily identifiable from the facing boxes) - or is that meant to be totally free in your edition?
sorry for not having splendid replies, but maybe others might cross the same issues with their own language
regards ~ Vera
Hi Vera,
thanks for taking time to test it.
What you describe is a basic test case new UDL was designed to solve (no spaces are needed as separators). If something as simple as keyword;comment;keyword is not working, that means I've screwed my last build. I can assure you that something as basic as that was working when I tested it.
I guess working without version control is a bad thing (and working late at night is even worse). I will review code changes, and post here what went wrong.
In the mean time, is this the test case you are talking about:
keyword;comment;keyword
where ; is comment start and end
And I don't even know what to say about delimiters not working at all.
I will check what went wrong, and put here version that does work.
One more time, thanks for warning me about it.
BR
Loreia
Hi Vera,
I just recompiled a new project (source code of latest npp 5.9.3 with files posted on patch tracker), and it works just fine:
You can see that setting " and ; as operators has no effect on their function as delimiter and comment marks, and no spaces are needed.
Can you show me what you get?
BR,
Loreia
P.S.
I forgot to answer, the old drop-down menus are useless in new UDL because you can use whatever you want as operator (or delimiter for that matter). So, any of this can be an operator:
!
%
&
a
b
and
or
not
or_any_long_sentence_without_spaces
Great, there are no preview or edit buttons on forum.
Here is another try to post pictures from previous post:
first
second
I hope it works this time
OMG, am I really this tired?
Ok, just the second pic this time
I get it, damn "www" string is confusing forum software.
Hi Vera.
Here is the new recompiled binaries for testing (just in case something was wrong with the old ones)
http://www.mediafire.com/?gd36i3mirolw8qv
You must have installed latest 5.9.3 version of Notepad++ (forgot to mention this the last time)
BR
Loreia
If you wish to merge your change with my Improved UDL handling and corrections patch, please feel free to do so. Some of the features are the same as yours.
CChris
Hi Chris,
can you post a link to your patch?
I can't find it myself.
http://sourceforge.net/search/?group_artifact_id=612384&type_of_search=artifact&group_id=95717&words=user+define+language
BR,
Loreia
I think I found it.
https://sourceforge.net/tracker/index.php?func=detail&aid=3089258&group_id=95717&atid=612384
I will put some work on it next weekend.
BR,
Lorea
Hello Lorea,
ups - I didn't expect you to be that quick ;-)
I had an oldfashioned sunday afternoon with coffe & car-racing
To give you a first quick reply: it works_ (!)_
BUT I found another issue.
When running your latest version it even looked less colorfull, because now the whole commenting failed. I couldn't believe it and was a bit paralized while staring at the UDL-Gui.
Then I saw my marked option : 'IGNORE CASE'
- then I deselected it and all highlightings appeared as wanted. :-)
Also that: keyword_;comment_ works
likewise an 'embraced' comment-sign ";" doesn't corrupt following code anymore
Ok - test 'IGNORE CASE' and see what it does to you
and I'll have a look if I can find another hidden itch and test a priv_at_long_delimiter ;-)
~~~~~~~~~~~~~~~~~
Btw: I also cross-checked your two versions and found out
a - unselecting 'IGNORE CASE' in the previous version wouldn't help
B-ut - the previous EXE combined with the newest DLL would work like both NEW do
And yes - I used the current 5.9.3 uni for a start.
And on the long - I would be very interested in an favoured ANSI version as well ;-)
Hint @ALL: if you download the current UDL-version rename Notepad++.exe -> to Notepad2++.exe, for not to override the original by accident.
have a nice sunny night and a summerly week ~ Vera
Hi Vera,
thanks a lot for your last post. It was VERY helpful. It seems I was so concerned testing everything in operators tab, that I forgot to test Ignore case option. It was indeed causing errors, and after a quick debugging session today, I discovered a bug in Scintilla. (I just went to http://www.scintilla.org to report it, but I see it is fixed in latest 2.28 version, version 2.27 that NPP is currently using still has it.)
Anyway, in StyleContext class (StyleContext.h file), function
bool MatchIgnoreCase(const char *s) {
if (MakeLowerCase(ch) != static_cast<unsigned char>(*s))
return false;
s++;
if (MakeLowerCase(chNext) != static_cast<unsigned char>(*s))
return false;
Does not work for a single character strings.
It should instead be:
bool MatchIgnoreCase(const char *s) {
if (ch != static_cast<unsigned char>(*s))
return false;
s++;
if (!*s)
return true;
if (chNext != static_cast<unsigned char>(*s))
return false;
I will make a bug report to NPP bug tracker a bit latter, so this could be fixed regardless of would Don and NPP team accept my changes to UDL.
So, I have added support for nested comments and fixed Ignore case bug in latest NPP as a result of this thread.
New binary version for testing, can be downloaded from here:
http://www.mediafire.com/?fdja8lh9afznz3t
Note that I hard-coded nested commenting feature in, so you cannot disable it (I will update dialog with this feature, but for the time being, l'll keep it hard-coded in. For testing with nested comments disabled, just use dll file in subdirectory called "nested_comments_deselected" (you need to copy it to the same directory where notepad2++.exe is.)
My next task is to study Chris' code better, so I could incorporate his changes into new UDL.
Speaking of that, Chris, can you help me with these options:
* Fix replication of prefix options on creating new language
* Fix cross styling that would take plce when changing an UDL and the active document uses a different UDL
Do you remember what was this about? I am not sure I understand it fully.
Also this:
* Surrounding a keywords with double quotes will allow it to have multiple parts. One space is equivalent to any number of spaces and tabs.
* In prefix mode, if a multiple part keyword ends with a trailing space, the next word will be part of highlighted text. Such a trailing space is ignored if prefix mode is off.
This is the thing I will "attack" first. But why is this needed? Maybe for writing pseudo-code?
BR
Loreia
A/
1/ The 4 prefix options (one per kw group) were not always carried over properly when creating a new language from existing.
2/ If you had a document opened using UDL A, open the UDL panel and change style parameters of UDL B, that would wrongly affect active documùent as it justs reports L_USER language.
B/
1/ Defining the "key word" keyword will highlight
key word
key <tab> word
and so on
2/ If prefix mode is checked for the group, "key word " will match
key word begin
key word <tab> fin
and so on. The trailing space(s) inside double quotes is ignored when prefix mode is off.
CChris
Hi,
just a quick update so this would not seem like a dead thread.
I am working on code folding right now. Progress is a bit slow due to my lack of free time these days.
So far folding forks more or less as expected, and my next step is to clean the code a bit (but optimize it and shrink it), and to add support for Folder Open and Close keywords in comments. That would allow users to fold any block of code just by creating comments like:
// foldingOpen
line 1
line 2
…
line n
// foldingClose
even in languages that don't have code folding built in. Basically, I am trying to mimic fantastic #region statement from C#.
Anyway, this is what I have done so far.
Picture should be self explanatory, but if anyone has any questions, just post them here.
Best regards,
Loreia
P.S.
I am yet to analyze all the stuff CChris wrote for UDL. There is a lot of great stuff in there, and I intend to merge all that stuff with my code. When finished, UDL will be really powerful stuff (not that it isn't powerful already, but this will be the next step)
Best regards,
Loreia
I see I made an error in that last picture, in line 19, I should be referring to line 17, not line 12.
Also I forgot to include one more case, so here it is.
Best regards,
Loreia
@loreia2:
I'm keenly aware that my code is just as densely commented as Notepad++'s proper, which you may feel lacking. If in doubt, just don't hesitate to ask me about the code at quasar850 ta rediffmail tod com
CChris
I've recently begun expanding my coding horizons with a number of languages and notations that benefit from syntax highlighting (like Backus Normal Form), and I just wanted you guys to know that I am absurdly excited to see that people are working on improving the language-definition features. There are just so many things to highlight out there (BNF! Redcode! LOLCODE! MediaWiki! Brainf***!) and most of us aren't quite skilled enough to code up a NP++ plugin.
So seriously: Thank you. Your hard work is 100% appreciated, and it will be used.
@CChris,
excellent, depending on my schedule next few weeks, I will send you an e-mail or two. I've only skimmed through your code (basically, I just read .patch files directly in Notepad2), and code looks just fine. I guess I won't have much problems reading it, but it can save me few hours if some details can be explained within context of NPP.
LexUser.cxx is what I am familiar with at this point, so my questions will probably be more about UseDefineDialog.cpp and saving stuff to .xml file.
Anyway, thanks for your help.
@Poster after CChris
Thank you for your support and appreciation. I guess I only wanted to add support for comments without spaces (which was the only thing that was breaking for me), and then somehow I got hooked :-)
It took complete rewrite of coloring algorithm to work in comments without spaces, and now I just want to expand UDL with other stuff. It is fun, but it will have to go relatively slowly because I need to balance this work with my regular obligations (which are numerous).
But I must admit it is fun to see UDL with new features added in. In fact everywhere I look now, there is always somewhere in my mind something like: "Can I add this to UDL?"
E.g. yesterday I read a blog post http://www.elpauer.org/?p=971 (it popped up on planetKDE), and I already have a new task for this weekend: adding support for line comment continuation. Funny thing is, I believe line comments continuation to next line should be outlawed, but I forgot that we have some stupid in house scripting tool that allows this "feature" and people use it extensively. It will be great if we can finally highlight that stuff properly.
Also, if you have an idea (based on one of those languages that you mention) what could be added to UDL, just post it here. Someone suggested to add support for nested comments, and when I added that in and mixed it with comment folding, I created something that gave me an idea how to significantly speed thing up when folding code. When I finish optimizations I believe speed increase (when there are many folds) will be noticeable in FoldUserDoc function.
So, don't be afraid to discuss possible new features, good ideas are always welcome.
Best regards,
Loreia
(Same user here; I finally bothered to fix my name.)
Well, the search that brought me to this thread was "nested delimiters." Specifically, I was interested in taking something like:
expression = <term> { <addop> <term> } term = <factor> { <mulop> <factor> }…and (for instance) coloring anything between <> delimiters while also bolding anything between {} delimiters, without completely overriding the formatting of the nested delimiters.
I present this more as a brainstorming spark than a "feature request," since the mainstream utility is dubious and the implementation would be complex - after all, string literals must override formatting, and you would have to figure out how to know which part of the formatting to selectively apply, and probably some other stuff I'm forgetting. On the other hand, it would synergize nicely with the vastly expanded options you've allowed for delimiter definitions.
Other than that, though, I think you may have already addressed most of the frustrations I've come across in the last few weeks. To keep things brief, I'm just going to point to everything you've done with operators and delimiters.
On a general note, I'm curious - I'm sure everyone would love to define an arbitrary number of keyword sets, delimiter sets, and so on, but we obviously can't. Is that limitation in the UDL dialogue, in the Notepad++ code, or somewhere else?
Hi Argumentum ad Stultitiam,
Nested delimiters are interesting idea, I can see how that would be useful.
It would be relatively easy to implement it for coloring of entire document, but it is a challenge to do it when re-coloring document while user is editing. You see, in case of { <addop> <term> }, if user is trying to change word "addop", I need to know that this is a say Delimiter2, but after next ">" I need to continue with Delimiter1. And if "addop" contains another subdelimiter like <add !delim3! op>, I need to carry that information too. I would need to use some STL container here, because I cannot know how deep it goes.
I'll think about it, but if code gets too complex, I'll probably skip it for time being.
Regarding arbitrary number of keywords and delimiters, I would have to dig deeper in Scintilla to see if that could be expanded, but that would be a big (really big) change. Dialog would have to be redesigned, and Coloring function would have to significantly chance its logic (keywords and delimiters would have to be checked in a while loop). This would be really unpractical. Besides, I'm surprised that you have exhausted even this amount keyword and delimiter sets (typically, I use just one or two sets)
So, I'll see about nested delimiters, but you are stuck with 4 keyword sets, and 3 delimiter sets for now.
Best regards,
Loreia
Haha, no worries. If it seems like I'm talking about stuff that's completely out there, I assure you it's only because I can't think of many practical features you haven't already addressed. Keep up the good work!
Oh, I've thought of something potentially useful, though I suspect it's utterly out of your hands: It would be lovely to be able to control the tab settings (TAB character vs. spaces, TAB width) for user-defined languages.
If I'm correct in my assumptions, then this would require completely new functionality (much like adding keyword sets), but I thought I'd throw it out there anyway.
Adding a tab settings: you need to persist the setting, display it on the dialog and maintain consistency, and then… get it applied. Notepad++ treats all user defined languages as same because they have the same lexer witth ID L_USER. And it will treat tab/spaces issues accordingly. So now we'd need to change this behaviour so that L_USER files are inquired for their actua language and then proper tab settings be applied. A better, but entailing more radical changes, idea is to have a range of individual language IDs associated to the single L_USER lexer.
Aout nested delimiters, there could be a simple way to implement them using Scintilla's indicators. Now the trouble is to include indicator details in both the definition file and the UI.
CChris
Interesting. It may be helpful to explain the actual use-case that's been spawning a lot of these questions; it may spark some ideas on your end.
As a simple example, I've been working with the MIPS assembly language (which is quite different from Notepad++'s default Assembly language). A very simple program might look like this (with 8-space tab characters):
As it stands, there are basically two options I know of to get the highlighting and formatting one would want for this language: Modify/extend the Assembly preset, or create a UDL from scratch. The downsides I have found thus far for each approach are:
Assembly Preset
No way to change line comment character from ';' to '#' (making comments incredibly distracting)
No way to set '$' as a symbol ("$8" is highlighted as an identifier, rather than a symbol and a number)
UDL
No support for highlighting hexadecimal numbers
No ability to set unique tab settings (8-space width TAB character is hardly appropriate for every UDL)
Most of this is out of scope for the UDL project, but I think that hexadecimal number support would be quite welcome and (hopefully?) easy to implement.
They somehow are. See my patch about Extra Characters for Numbers. It has a broader scopen, as it also handles cureency/format signs and readability separators between digits.
Don didn't like it originally because it would add 4 text fields on the number tab. The fieds re now on on a separate Characters tab.
With this enhancement, and after filling the relevant properties, hex support would be in UDLs.
The fields are:
* Allowed refixes for prefixed numbers. For MIPS that would be $
* Additional chars in numbers. Not used there, or perhaps : for segment:offset addresses, if that makes sense.
* Additional characters in prefixed numbers. That would be abcdefABCDEF
* Optional suffix characters: probably not used here. Targets things like 27€ or 3.6f .
Consider reconsidering, DOn?
Hi Chris and AaS,
I would just like to add that I loved Chis' patch for Extra Characters for Numbers. I LOVE the flexibility of it. And as soon as I finish my work on comments and subsequent code optimizations, I plan to further study that patch and include it in my version of UDL. I am not sure if I would just straight copy/paste it, or modify it a bit (e-mail consultations on that will follow), but that part of code is the next one on my radar.
Best regards,
Loreia
Hi Loreia, could you please say if you have some plans to introduce multipart keywords into UDL? :)
There were some older patch from Chris, but unfortunately I'm not really good to patch and use it.
Hi volander,
yes, I plan to review all of Chris' UDL patches, and then integrate them with my work. There are several of them, and so far I've just skimmed through the list, so it is hard to discuss the details at this point, but my plan is to go through entire list.
Obviously, some changes will be required in GUI, as it might be a bit "messy" to just put new elements on already crowded tabs.
Add to that my busy schedule, and you'll see that it will take some time to do it all. But the plan is to do it. I'll post intermediate binaries here for testing (and so impatient ones can use it if they can't wait for official release)
I hope, I'll have some news on it this weekend.
Best regards,
Loreia
Loreia, I'm very thankful for your answer and also for your great things in the field of UDL improving! :)
It is a really great job!
Agreed, top marks all around.
Re: GUI
Yeah, I definitely sympathize with the pain of trying to fit yet more features into the UDL dialogue. In the current NP++ release, it is literally impossible to set the background color for Delimiter 2 without hiding the Windows taskbar (Vista, 1440x900). That's not a bug fix request; just an illustration of how vertically stuffed the dialogue is.
Re: Extra Number Characters
That's straight-up awesome.
Re: CChris
I'd like to say thanks for providing just about every single useful response concerning NP++'s syntax highlighting I've ever found with Google. That's not directly related to the UDL stuff, but it's true.
About the UDL UI, perhaps you'll like this https://sourceforge.net/tracker/?func=detail&aid=3283889&group_id=95717&atid=612384 ?
There are sceenshots to give an idea.
CChris
Hi Chris,
is there anything about UDl that you didn't improve? :-)
I can't believe nothing made its way to main build (of all the UDL patches you've submitted).
Anyway, you made an interesting suggestion with your UI redesign, but for the time being, I'll just slightly redesign UDL GUI by repositioning and resizing existing elements (and adding few of my own in the process). The idea is to reduce vertical clutter, add few more "boxes", and avoid changing much in UserDefineDialog.cpp.
Basically, I don't want to spend too much time on GUI, as I am fully focused on UDL algorithm. I just need something that works for now. Latter, when algorithm is fully developed, any UI can be "soldered" on top of it.
Regarding UDL algorithm, optimizations start to show some progress, last revision is measurably faster than official build. I did just one test, on a fake 300.000 lines long file (with lots of folding), and official build took 2min 5sec to do the task, and new version took 1min 15 sec.
Code is still messy, but this test brought smile to my face, because results are promising.
Until next update,
Thank you and best regards,
Loreia
Hi,
I am excited seeing this, I was working on this https://github.com/blakmatrix/CoffeeScript_notepad_UDL for coffeescript and thought I had gotten it as good as possible within the current limits. Any plans to address issues I cane across?(i.e. regex detection, embedded statements, and if # is a comment and #{@var} is a container, and also different number format detection hex octal(0 as a prefix) decimal ect)
Keep up the good work!