#178 Streaming large tokens

open
Streaming (1)
1
2013-09-27
2013-09-16
Zemantics
No

In some cases, like when parsing a Turtle file in my scenario, there are edge cases that results in very large tokens - even GBs. So, could you implement some type of a BLOB token type that allows for streaming?

Discussion

  • Frank-Rene Schäfer

    Please, specify the API of 'blob' and how it is different from the accumulator.

     
  • Zemantics

    Zemantics - 2013-09-24

    A client may want to stream a 1 GB BLOB over a network connection, to avoid exhausting internal memory resources I would like to stream a single token that takes up 1 GB space in bulks of 4 KB - is this possible? I have looked at all the Quex accumulator examples I have found to no avail. Thank you.

     
  • Frank-Rene Schäfer

    Can you have something like 'sub-lexeme with maximum size 4096' defined
    in your grammar; Then you might define a mode for it. Instead of writing
    it into the accumulator, you might stream it through a socket. Or, do you
    mean the START/PAUSE/RESUME/STOP api of streaming servers?

     

Log in to post a comment.