[pclasses-devel] "stream magic" support

[stephan b. <st...@s1...>]

Yo!

Christian mentioned this the other day, and i'd like to go back to it 
for a minute: "magic cookie" support.

If you don't know what that is, you really do know, you just don't know 
that you know it. When you see a shell script:

#!/bin/bash

that is a magic cookie. That "cookie" tells the OS what app can handle 
the content. gzip files have a small header, as do most other file 
types, and these can be analyzed to guess a file's type. Traditionally, 
and for ease of programming, we normally look at file extensions, but 
magic is a more reliable way of knowing the content of a file (but 
slower, of course, requiring at least a few bytes of fread()).

s11n uses this for input dispatching, so users don't need to know what 
format they're using. Clients only select the output format. The 
formatter (Serializer) writes the cookie as the first line of output, 
then proceeds to save object data in it's native format.

Reading a cookie is easy to "non-intrusively" do with files, but there 
is a subtle complication with streams: once we read the cookie we have 
extracted it from the stream and therefor cannot easily pass it to the 
parser! One option is to buffer the whole stream data after 
reading/checking the cookie, so we can pass a stringstream to the 
parser, but IMO this is not an option because streams have, by nature, 
an indeterminate size, and we don't want to read 57GB into RAM.
Another option is to change the parser's API so that we can pass the 
cookie to it. The 3rd option (the one s11n currently uses) is to simply 
consume the cookie and pass the rest on the parser. This has the slight 
advantage that the parser won't ever see his cookie, but this isn't a 
big problem (but does have the down-side that we cannot use multiple 
cookies for one parser, to differentiate versions or compatibility 
modes).

Anyway...

i bring this all up because i would like to add some IO support for P, 
such that it can load stream handlers based on... whatever. For 
example, s11n currently uses a tiny lib which can transparently 
provides gzip or bzip2 de/compression streams. That lib won't do, 
as-is, in P (it's too restrictive), but i would like to modify it to a 
more generic stream dispatcher based off of Factory.

Now comes the problem: reading cookies. Again, from files, we can read 
the cookie and pass it on to the handler. This is necessary when 
reading, e.g., a gzip file - we MUST pass all of the data to gzip. Same 
for almost any other generic stream processor. This means that writing 
cookie-based streamloader support can conceivably be done for files in 
about 20 lines of code (mostly the fopen() and accompanying calls, then 
simply delegating the real work to Factory). Files actually have one 
problem here, though: cookies won't work (AFAIK) for all binary 
formats. Some cookies require logic to decode, which means trying out 
functions on the data until we find a handler which thinks it can read 
it. In effect, this is what the 'file' (man file) does, 
using /etc/magic.

Has anyone got an idea for an interface which allows us to, halfway 
efficiently, bind to "cookie tester" functions, so we can search them 
to find a matching handler?

-- 
----- st...@s1...   http://s11n.net
"...pleasure is a grace and is not obedient to the commands
of the will." -- Alan W. Watts