Re: [ jEdit-devel ] XMode in 2.4

[mike d. <md...@st...>]

On Sun, 19 Mar 2000, Slava Pestov wrote:
> Please do. The VERSION thing is a good idea. But instead of making the
> DTD version '1.0', I think it should match the jEdit version. It will
> make things easier.

i disagree. the DTD will probably not change very often and it would be a
pain to have the XModeHandler code handle DTD "2.4pre3 - 2.4.1" different
than DTD "2.4.2 - 2.6pre3", since it would potentially require making
changes to the handler for each new jEdit version. it would be easier to
check for "1.0" versus "1.1" and change the processing accordingly.

> > i like the idea of the cache file, although i'm not sure how well the
> > rebuilding thing would work. it's not so much that rebuilding the cache
> > wouldn't work, but we would have to modify jEdit to support
> > unloading/reloading modes on-the-fly and refreshing the buffers using
> > them. otherwise, it doesn't make much sense to reflect a modified mode
> > file in one way (filename and first line globs), but not another (the
> > actual mode behavior). but, who said this wouldn't take some effort ;).
> 
> I think you misunderstood my idea... basically, the cache would store
> the label, first line and filename globs of each mode. The file names of
> newly opened files will be checked against the cache, and if a matching
> mode is found, the mode is loaded from disk for real.

your right. what i was saying does not make much sense. i think that the
performance benefit of not loading X-CBL mode every time i want to edit a
PHP file is enough that any wierdness that the cache will possibly
introduce can be dealt with if it arises.

> > this is a good idea, especially since it would involve examination and
> > possible refactoring of some of the shakier XMode internals. however, i
> > think that this (and some parts of delegation in general) would work
> > better with a new document model. is this still planned for 2.4?
> 
> Why would we need a new document for this? All the relevant info should
> just be stored in TokenMarker.lineInfo.

i guess i was thinking that two separate TokenMarkers would be used, so
the lineInfo would have to be shared, which could cause problems. however,
now that i think about it, a single GenericTokenMarker can still be user
and the DELEGATE_EXTERNAL can just load the relevant mode file and copy
the ParserRuleSets from the other mode's GenericTokenMarker to its own. i
think that they could even share the same ParserRuleSet instances. then,
the only problem that needs to be worked out is the namespace conflict
between the two MAIN ParserRuleSets (the unnamed RULES section in an XMode
mode file defines the MAIN ParserRuleSet). perhaps a perlish solution like
X-HTML::MAIN vs. X-JAVASCRIPT::MAIN could be used.

> Today, I did a tiny bit of hacking on XMode - namely, I implemented the
> PROPERTY tag and wrote some new modes (C, C++, BeanShell, IDL, JavaScript,
> Python, PostScript, Batch File and TeX). Would you like me to commit this
> to the CVS, or do you want to take a look first?

go ahead and commit these changes yourself. just let me know once you've
done it so that i can update my local copy.

> I also found a rather weird problem in XMode. In PostScript, string literals
> are delimited by ( and ), but they can also contain any number of matching
> ( and ) inside them. So for example, (hello (world, this) is (a) literal)
> is perfectly valid. To accomplish this in XMode, I wrote this:
> 
> <SPAN TYPE="LITERAL1" DELEGATE="LITERAL">
>  <BEGIN>(</BEGIN>
>  <END>)</END>
> </SPAN>
> 
> ...
> 
> <RULES SET="LITERAL" DEFAULT="LITERAL1" ESCAPE="\">
>  <SPAN TYPE="LITERAL1" DELEGATE="LITERAL">
>   <BEGIN>(</BEGIN>
>   <END>)</END>
>  </SPAN>
> </RULES>
> 
> This is all good and the above example actually works, but stuff like
> (hello \) world) doesn't. The first ")" (the one escaped with "\") ends
> the SPAN, when it shouldn't. Any ideas?

i'm not exactly sure why this is (and i should be able to tell better once
i have your X-PS mode), but it looks like it has something to do the way
XMode checks the delegate span's end rule. basically, the first rule
XMode checks (before any rule in the delegated ParserRuleSet) is the end
rule for the parent (delegating) span. if it doesn't match, then it stays
in the delegated rule set and begins checking its rules. at this point,
i'm not exactly clear on how escapes are being handled here, but they
could be broken. this recursive rule set calling is rather confusing :|.

> I've also noticed that XMode is slow. The performance when typing is ok
> (because it only has to tokenize one line but scrolling is much slower
> (because it has to tokenize all visible lines). I think we could solve
> this by storing the token list for each line in the token marker, and
> retokenizing lines only when they change (eg, by calling markTokens()
> from a document event handler). What do you think? It would simplify
> JEditTextArea internals as well.

i thought that there were problems with caching the token list that could
only be solved by abandoning the Swing Document model. is this not so?

as for XMode being slow, it is quite possible that we could realize a
substantial speed increase even without caching by improving the
efficiency of markTokensImpl() in GenericTokenMarker. some parts of it are
pretty crufty after a year of slow, incremental changes and it might be
nice for someone with a decent profiler and a machine capable of running
it to analyze what actually makes GenericTokenMarker slow. where is it
spending all its time? that's where we should focus our attention. maybe
the engine is fundamentally flawed? i hope not, but if it is, we should
either make do or replace it.

this is not to dissuade an investigation of caching, simply to state that
it is not the only viable avenue of exploration.

-md