#240 Skip comments with CPD

[Carlos Garces]

Hi!

I have one Tokenizer that extends from AbstractTokenizer.

With the actual code and this sample...

Dim objTest as string 'This is one comment

All strings are marked as tokens (including the comments)

This paths try to fix it. The override of tokenize function is to use it in my tokenizer for VB6 (I'll be release it soon)

Un saludo
Carlos Garcés

[Romain PELISSE]

Hi,

I look at your patch and i understand your issue, but i find your approach rather extrem. Your parser may want to discard comment as token but other may not. My Fortran parser was designed to check if students are copying each other jobs, duplicate comment are a good way to spot it... Also, people who cut'n'paste code may be inclined to cut'n'paste comment with it.

Nevertheless, would mind add a protected boolean variable that subclass to disable the 'comment as token' behavior ? I think is more subtle approach (and it does not break previous behavior of the tokenizer).

[Carlos Garces]

AbstractTokenizer.diff

[Carlos Garces]

Fixed!

I have some problems with the VB6 parser... I need handle some special chars and escape characters.

I'll be release it soon.

Un saludo
Carlos Garcés

[Romain PELISSE]

I don't think this patch is still relevant - or am I wrong ?

For what I gather from this thread, this fix was somewhat tiedly coupled to VB6 parser that has yet to released so...