Re: [ jEdit-devel ] Proper use of TokenMarker and LineContext

[Makarius <mak...@sk...>]

On Fri, 3 Sep 2010, Alan Ezust wrote:

> Notice the *chirp chirp* of crickets after your asked that question, eh?
>
> There is nobody who has worked closely on this code in many years. Slava 
> wanted to get rid of ChunkCache + other low level textarea classes and 
> replace them with something that is in the JDK. But since leaving, most 
> of the development on the textarea is to fix bugs and make it a 
> separable component.

This was also my impression after studying the sources a bit more.  In 
general it seems to be adequate, although some "optimizations" appear to 
target rather old platform constraints, and might even get in the way now.

For example, the policy when exactly the cache is updated, and when 
entries are deleted is still not quite clear to me.  Since the update 
seems to be triggered by getLineInfo (among other possibilities) it can be 
quite slow if the user jumps from the top of a text area to the bottom 
(say via the scrollbar).  This is because the process of producing chunks 
via token markers always iterates through the whole text, starting from 
firstInvalidLine, and our document model behind the buffer needs some time 
to build of the information for each line (about 0.1 .. 1.0 ms).  With 
thousands of lines this makes a noticeable delay.


Anyway, concerning general JDK things: Does this refer to certain editor 
models in Swing?  I was always wondering if they have been ever used 
productively, although I cannot really claim significant Java/Swing 
experience.  Recently, we had a Java hacker evaluate the editor kit of 
Netbeans, which uses all these JDK standard interfaces for styled 
documents etc., but in a very odd ways, merely emulating a line-based 
editor without much style in the end.

Compared to that I've found it much more productive to work with whatever 
jEdit happens to offer here, and try to tweak it into our direction. 
(And jEdit has generally much more style than Netbeans, of course.)


> So what you see is what you get, and if you would like to delve into the 
> code, try to figure out how it works, and document it, we would welcome 
> any contributions you may have. Even if it's just some extra code 
> comments that you think will help the next person understand it better.

OK, as a start here is one of the answers I have found so far ...

> On Tue, Aug 24, 2010 at 2:34 PM, Makarius <mak...@sk...> wrote:
>>
>> TokenMarker.markTokens only provides the current line segment, not its 
>> offset in the buffer text.  As a workaround we have experimented with a 
>> LineContext containing the last line as explicit integer.  After 
>> investigating memory profiles of jvisualvm and the jEdit sources, it 
>> seems that this is not the proper way to do it.  LineContext maintains 
>> a persistent table of all contexts ever encountered, via the "intern" 
>> method. Thus we essentially have a memory leak in our token marker.

The LineContext can in fact be tweaked to provide a consecutive line index 
into the buffer.  To get the "interning" right, one merely needs to define 
hashCode/equals on the custom class in a way that captures the intended 
semantics.  See 
http://isabelle.in.tum.de/repos/isabelle/file/12dac4b58df8/src/Tools/jEdit/src/jedit/document_model.scala#l62
how this is done right now.

Thus the static map of LineContext.intern will accumulate all kind of line 
numbers ever pushed through it.  This is adequate, although some minor 
improvement would be something like a weak hash map.  (There are more 
important performance concerns in other spots of the whole setup.)


 	Makarius