Re: [naviserver-devel] whole lotta errors

[Zoran V. <zv...@ar...>]

On 15.08.2006, at 21:09, Stephen Deasey wrote:

>
> In the documents you serve, do you specify the encoding *within* the
> document, at th etop of the HTML file for example? Or are you serving
> XML in which case the default for that is utf-8 anyway (I think, off
> the top of my head...).

We put it in headers, as extension of the content-type:

    text/html; charset=utf-8

>
> I agree with Zoran. ns_conn encoding should be the way to change the
> encoding (input or output) at runtime.
>
> The mime-type header sent back to the client does need to reflect the
> encoding used, but ns_conn encoding should drive that, not the other
> way around.

Correct! At the moment, we do not mingle with ns_conn encoding
stuff, rather we set the mime-type accordingly.

>
> We can check the mime-type header for a charset declaration, and if
> it's not there, add one for the current value of ns_conn encoding.

Right.

>
> One problem to be resolved here is that Tcl encoding names do not
> match up with HTTP charset names. HTTP talks about iso-8859-1, while
> Tcl talks about iso8859-1. There are lookup routines to convert HTTP
> charset names to Tcl encoding names, but not the other way around.
> Tcl_GetEncodingName() returns the Tcl name for an encoding, not the
> charset alias we used to get the encoding.

This is just-another-nuisance we'll have to swallow. We can write
our own name-conversion between the known Tcl and HTTP encoding names.
You say:

   There are lookup routines to convert HTT charset names to Tcl  
encoding names

Who wrote them? I cannot imagine other way arround would be the problem.

>
> We could store the charset, as well as the encoding, for the conn. But
> I was wondering: could we junk all the alias stuff and, in the
> Naviserver install process, create a directory for encodings files and
> fill it with symlinks to the real Tcl encoding files, unsing the
> charset name?
>
> You call ns_conn encoding with a charset. Naviserver converts the
> charset name to a Tcl encoding name. The return value is the name of
> the encoding, which is *not* the name of the charset you passed in! I
> don't know if that's intended, but it's really confusing.

Uh.. I would not rely on that symlinking, if possible (think of  
Windows).
I believe the better way would be a HTTP->Tcl and Tcl->HTTP encoding- 
name
lookup function.

>
> Another place this trips up: In the config for the tests Michael  
> added:
>
>   ns_section "ns/mimetypes"
>   ns_param   .utf2utf_adp  "text/plain; charset=utf-8"
>   ns_param   .iso2iso_adp  "text/plain; charset=iso-8859-1"
>
>   ns_section "ns/encodings"
>   ns_param   .utf2utf_adp  "utf-8"
>   ns_param   .iso2iso_adp  "iso-8859-1"
>
> The ns/encodings are the encoding to use to read an ADP file from
> disk, accoring to extension. It solves the problem: the web designers
> editor doesn't support utf-8.  (I wonder if this is still valid any
> more?)
>
> But, the code is actually expecting Tcl encoding names here, not a
> charset, so this config is busted. It doesn't show up in the tests
> because the only alternative encoding we're using is iso-8859-1, which
> also happens to be the default.
>
> This is probably just a bug. The code uses Ns_GetEncoding() when it
> should use Ns_GetCharsetEncoding(). But that highlights another bug:
> when would you ever want to call Ns_GetEncoding()? You always want to
> take into account the charset aliases we carefully set up. This
> probably shouldn't be a public function.
>
>
> The strategy of driving the encoding from the mime-type has some other
> problems.  You have to create a whole bunch of fake mime-types /
> extension mappings just to support multiple encodings (the
> ns/mimetypes above).
>
> What if there is no extension? Or you want to keep the .adp (or
> whatever) extension, but serve content in different encodings from
> different parts of the URL tree? Currently you have to put code in
> each ADP to set the mime-type (which is always the same) explicitly,
> to set the charset as a side effect.
>
> AOLserver 4.5 has a ns_register_encoding command, which is perhaps an
> improvement on plain file extensions.
>
> Both AOLserver and our current code base have the bug where the
> character set is only set for mime-types of type text/*.  This makes a
> certain ammount of sense -- you don't want to be text encoding a
> dynamicaly generated gif, for example.
>
> However, the correct mime-type for XHTML is application/xml+html. So
> in this case, an ADP which generates XHTML will not have the correct
> encoding applied if you're relying on the mime-type.
>
>
> Here's another odity: Ns_ConSetWriteEncodedFlag() / ns_conn
> write_encoded. This is yet another way to sort-of-set the encoding,
> which is essentially a single property.
>
> The only code wich uses this is the ns_write command, and I think it
> has it backwards. By default, if Ns_ConnGetWriteEncodedFlag() returns
> false, ns_write assumes it's writing binary data. But nothing actually
> sets this flag, so ns_write doesn't encode text at all.
>
> We should remove the WRITE_ENCODED stuff.
>
> How do we handle binary data from Tcl anyway? There's a -binary switch
> to ns_return, and the write_encoded flag for ns_write.  I was
> wondering if we could just check the type of the Tcl object passed in
> to any of the ns_return-like functions to see if it's type
> "bytearray".  A byte array *could* get shimmered to a string, and then
> back again without data losss, but that's probably unlikely in
> practice.
>
>
> There's also the problem of input encodings.  If you're supporting
> multiple encodings, how do you know what encoding the query data is
> in?  A couple of solutions suggested in Rob Mayoff's guide are to put
> this in a hidden form field, or to put it in a cookie.
>
> Here's an interesting bug: You need to get the character set a form
> was encoded in, so you call ns_queryget ...  This first invokes the
> legacy ns_getform call which, among other things, pulls an file upload
> data out of the content stream and puts it into temp files.
>
> Now, you have to assume *some* encoding to get at the query data in
> the first place. So let's guess and say utf-8.  Uh oh, our hidden form
> field says iso-8859-2. OK, so we call ns_conn encoding iso-8859-2 to
> reset the encoding, and this call flushes the query data which was
> previously decoded using utf-8.
> It also flushes our uploaded files. The kicker here is uploaded files
> aren't even decoded using a text encoding, so when the query data is
> again decoded, this time using iso-8859-2, the the uploaded files will
> be exactly the same as they were before.
>
>
>
> I'm sure there's some more stuff I'm forgetting. Anyway, here's how I
> think it should be:
>
> * utf-8 by default

Yes.

> * mime-types are just mime-types

Yes.

> * always hack the mime-type for text data to add the charset

Yes (shouldn't this automatically be done by the server?)

> * text is anything sent via Ns_ConnReturnCharData()

Yes.

> * binary is a Tcl bytearray object
> * static files are served as-is, text or binary

Yes, as-is, i.e. no conversion taking place.


> * multiple encodings are handled via calling ns_conn encoding
> * folks need to do this manually. no more file extension magic
>

Yes.

> I think a nice way for folks to handle multiple encodings is to
> register a filter, which you can of course use to simulate the file
> extension scheme in place now, the AOLserver 4.5 ns_register_encoding
> stuff, and more, because it's a filter. You can also do things like
> check query data or cookies for the charset to use.
>
>
> Questions that need answered:
>
> * can we junk charset aliases in nsd/encodings.c and use a dir of  
> symlinks?

No. I hate being dependent on the distribution directory.

> * can we junk ns/encodings in 2006?

I'm afraid it will not be that fast!

> * is checking for bytearray type a good way to handle binary Tcl  
> objects?

Hm... how would you (generally) check for a byte-array?
You can get a byte-array from object but you can't
(without looking into the object type which is really not
something that is portable) say which type of object you
are dealing with.

I believe the main source of problem here is somebody
slurping the whole file and wanting to return that file
as-is i.e. w/o any translations. In that case, he/she
cound use [ns_conn encoding] to set the encoding of the
connection before calling ns_return. This way we can
strip away all the extra options from the content-returning
commands and request the writer to use ns_conn to set
the correct encoding OR to skip encoding altogether
(for example: ns_conn encoding binary).
Wouldn't that make sense?

> * does the above scheme handle all the requirements?
>

No idea! I'd say, as our requirements are pretty small, it definitely
does. But Bernd & co. have more to say as they do obviously need much
more then we do.

Cheers
Zoran