Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#178 Streaming large tokens

open
Streaming (1)
1
2013-09-27
2013-09-16
No

In some cases, like when parsing a Turtle file in my scenario, there are edge cases that results in very large tokens - even GBs. So, could you implement some type of a BLOB token type that allows for streaming?

Discussion

  • Please, specify the API of 'blob' and how it is different from the accumulator.

     
  • A client may want to stream a 1 GB BLOB over a network connection, to avoid exhausting internal memory resources I would like to stream a single token that takes up 1 GB space in bulks of 4 KB - is this possible? I have looked at all the Quex accumulator examples I have found to no avail. Thank you.

     
  • Can you have something like 'sub-lexeme with maximum size 4096' defined
    in your grammar; Then you might define a mode for it. Instead of writing
    it into the accumulator, you might stream it through a socket. Or, do you
    mean the START/PAUSE/RESUME/STOP api of streaming servers?