[ jEdit-users ] buffer.markTokens() produces Token object with negative values.

[Esteban B. <ebo...@da...>]

Hello,

	I'm developing a plug-in that uses Sidekick for structure browsing and
org.gjt.sp.jedit.textarea.StructureMatcher for matching language elements.

	As part of the structure parsing process, I'm using the
buffer.markTokens(...) method to retrieve the tokens in the buffer to take
advantage of the syntax rules defined in our language's editing mode. I have
noticed that, sometimes, Token objects contain negative values for either
the offset or the length fields (and perhaps in other fields, but these two
are the only ones that I check and use). It's something that happens
sporadically, at random points of a (unchanged) buffer each time.

	Here's a sketch of the parsing method I've implemented.

synchronized analize(buffer) {

	try {
		buffer.readLock();

		for each line in buffer {

			// Retrieve the tokens in the current line.
			// I tried with tokenHandler.init() as well, without success.
			tokenHandler = new DefaultTokenHandler();
			buffer.markTokens(line, tokenHandler);
			Token token = tokenHandler.getTokens();

			while ((token != null) && (token.id != Token.END)) {
				// Find out if the token is a keyword, by checking its id.
				// Do something with it if it is a top-level language structure element.
				// ...

				token = token.next();
			}
		}
	} finally {
		buffer.readUnlock();
	}
}

	I really don't understand why I'm getting Token objects with negative
values. Perhaps I'm using a method I shouldn't be using, our what I'm doing
is interfering with other jEdit processes, because, among other things, I've
noticed that syntax highlighting is sometimes affected, or that the textArea
is not painted correctly. As you can see, I've taken care of synchronizing
method invocations and of using read locks, but I still get what I think are
incorrect and inconsistent values.

	I'd appreciate any help. Perhaps someone could point out where the error
is, or even suggest an alternative implementation. I've been stuck with this
for several days, and I've run out of ideas at this point.

	Many thanks, in anticipation.

Regards,
Esteban.
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.773 / Virus Database: 520 - Release Date: 05/10/2004