The bug seems to be that QUEX_NAME(buffer_conversion_fill_region_end) invokes:
QUEX_NAME(Buffer_move_away_passed_content)(&me->buffer);
instead of:
QUEX_NAME(BufferFiller_Converter_move_away_passed_content)((QUEX_NAME(BufferFiller_Converter)<void>*)me->buffer.filler);
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
is set to 256. So, when the old content of the engine's buffer is
moved away, it always keeps this number of bytes vand pastes what
comes new after it. The 'prepare()' function do not much more but
moving the old content away. The 'finish()' function, the case
of converters, fill first a 'raw buffer', then convert its data
into the engine's buffer. Once, the limit of 256 is reached, the
engine's new content will always start at this offset.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Since stream navigation was considered meaningless for direct
buffer access, the 'tell()' function was not supposed to do
something great. Pondering over it for a while, I think, it
would make sense, to accumulate the number of characters and
reporting the character index of the current input pointer
--the way you need it.
I will also consider allowing stdin as stream input for conversions,
at least. The conversion loop is stable enough to handled interrupted
input, as it seems.
The new version, is probably out at the end of this week.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I need to retain the character offset of tokens, in order to transfer the text annotations to the tokens.
I am using quex.tell() like this:
and it works properly.
However, if I use the buffer primitives, which is the suggested solution, like this:
the offsets at some point descrease, and I obtain:
Notice that after "Anamnesis" it stays at 256 before restarting.
The relevant portion of the input is:
I am using version 0.64.8.
Any suggestion for a fix?
Further investigation shows that the value returned by tell() is reset to 256 by qlex.buffer_conversion_fill_region_finish(), whenever it is > 256.
More precisely, the count is reset to 256 within the call to:
QUEX_NAME(Buffer_move_away_passed_content)(&me->buffer);
in QUEX_NAME(buffer_conversion_fill_region_end)
in file:
quex/code_base/analyzer/member/buffer-access.i
The bug seems to be that QUEX_NAME(buffer_conversion_fill_region_end) invokes:
QUEX_NAME(Buffer_move_away_passed_content)(&me->buffer);
instead of:
QUEX_NAME(BufferFiller_Converter_move_away_passed_content)((QUEX_NAME(BufferFiller_Converter)<void>*)me->buffer.filler);
The problem still remains with function:
I suppose it is similar, but I need help in fixing it.
I fixed the last problem by replacing:
with:
also in:
but I don't know if this might affect some other uses of the function.
No. The 'move-away' of content of the raw buffer must have
happened in 'buffer_conversion_fill_region_prepare()'.
I would love to analyze this. But, to get me started, please,
provide a minimalist example (.cpp, .qx., .txt and Makefile).
Thanks
I already posted the code.
Ok, I have been working now for more than half and hour just
to collect your code together, adapt it and get something running.
Upon 'cat example.txt | lexer' the package that I attached produces:
Please, feel free to adapt the package's content so that you display
your issue.
You must use a file that is longer than 256 characters to see the problem.
The configuration parameter
is set to 256. So, when the old content of the engine's buffer is
moved away, it always keeps this number of bytes vand pastes what
comes new after it. The 'prepare()' function do not much more but
moving the old content away. The 'finish()' function, the case
of converters, fill first a 'raw buffer', then convert its data
into the engine's buffer. Once, the limit of 256 is reached, the
engine's new content will always start at this offset.
So, what is the solution to get the proper offsets from tell(), rather than the offsets from 256?
Since stream navigation was considered meaningless for direct
buffer access, the 'tell()' function was not supposed to do
something great. Pondering over it for a while, I think, it
would make sense, to accumulate the number of characters and
reporting the character index of the current input pointer
--the way you need it.
I will also consider allowing stdin as stream input for conversions,
at least. The conversion loop is stable enough to handled interrupted
input, as it seems.
The new version, is probably out at the end of this week.
Thanks, I look forward to it.
Could you please, file a short bug report for this issue. This, way
I have a reference for the regression tests.
Thanks.