#178 Streaming large tokens

Streaming (1)

In some cases, like when parsing a Turtle file in my scenario, there are edge cases that results in very large tokens - even GBs. So, could you implement some type of a BLOB token type that allows for streaming?


  • Frank-Rene Schäfer

    Please, specify the API of 'blob' and how it is different from the accumulator.

  • Zemantics

    Zemantics - 2013-09-24

    A client may want to stream a 1 GB BLOB over a network connection, to avoid exhausting internal memory resources I would like to stream a single token that takes up 1 GB space in bulks of 4 KB - is this possible? I have looked at all the Quex accumulator examples I have found to no avail. Thank you.

  • Frank-Rene Schäfer

    Can you have something like 'sub-lexeme with maximum size 4096' defined
    in your grammar; Then you might define a mode for it. Instead of writing
    it into the accumulator, you might stream it through a socket. Or, do you
    mean the START/PAUSE/RESUME/STOP api of streaming servers?


Log in to post a comment.