naviserver-devel

[naviserver-devel] Fw: ns_http and 8-bit enodings

[oleg <oo...@ua...>]

On Fri, 4 Sep 2020 10:31:53 +0200
Zoran Vasiljevic <zv...@ar...> wrote:

> Hi!    
Hi!

> What is the purpose of signalling the binary content way up from the
> command line? For this we have the Ns_IsBinaryMimeType(cType) test.
>  

Sometimes we have to deal with misconfigured legacy sites, which transmit data in 8-bit encoding and report "Content-Type: text/plain" without specifying charset. Such data will be broken, since it will be processed ns_http as utf-8.

Receiving binary content in this case will allow us to correctly recover data using 'encoding convertfrom' command 

>     
> > Sometimes converted strings become
> > corrupted. For example, there is a server with output encoding
> > iso-8859-2      
> 
> This is I guess out of the scope as we rely on Tcl to do 
> encoding conversions. So if this comes bad, then you must
> post a bug to the Tcl project.    

As far as I understand, this is not a Tcl problem.

ns_http uses Tcl_NewStringObj for nonbinary data received, even for 8-bit chars.
Input data for Tcl_NewStringObj must be correct utf-8 string. An 8-bit string is not.
For 8-bit strings, we must either use Tcl_NewByteArrayObj or prepare the utf-8 string using Tcl_ExternalToUtf before Tcl_NewStringObj .


Oleg.

[naviserver-devel] Fw: ns_http and 8-bit enodings

[oleg <oo...@ua...>]

On Fri, 4 Sep 2020 11:38:53 +0200
Gustaf Neumann <ne...@wu...> wrote:

> since HTTP has means to include encodings, which NaviServer uses
> acting as a server, it should behave the same way when acting as a
> client and not burdening the application to dig into the content-type
> charsets to call the right conversion stuff.  

After the patch, ns_http became similar to ns_return: understands charset and -binary )

> Having NaviServer versions leading to different
> results depending on compile flags is not a good idea.  

Totally agree. #ifdefs were inserted only for safety because this code changes the usual behavior of the ns_http command

> To get a more detailed understanding, i have to dig into your examples
> to understand whether this is indeed a problem on the Tcl side or
> in NaviServer, ... but for this, i need a certain block of time, which
> is hard to get for me right now.  

Please see my answer to Zoran. There I wrote my opinion (possibly wrong) about using the Tcl_NewStringObj function.


Oleg.

Re: [naviserver-devel] Fw: ns_http and 8-bit enodings

[Gustaf N. <ne...@wu...>]

On 04.09.20 13:53, oleg wrote:
> After the patch, ns_http became similar to ns_return: understands charset and -binary )

exactly.

Zoran:

 > OK. I see no problem with that

I've added the "-binary" flag to "ns_http" as provided by Oleg to 
bitbucket. I've also added a short documentation and the 
ns_http_charsets.test, where i've added a "-binary" flag all occurrences 
where "encoding convertfrom" is explicitly used (either we provide an 
automatic conversion or none).

I did not have time yet to dig into TCLHTTP_USE_EXTERNALTOUTF the other 
test cases depending on that. My gut feeling says "this is necessary", 
but my brain says "why this was no problem so far" and "what will be the 
collateral damage"  if we do so.

The flag TCLHTTP_USE_EXTERNALTOUTF is still in the code but needs 
further investigation. However, i am going mid of next week on holiday 
(boat), where i will not be reachable ... until then many things are 
piled up on my side, so i have no chance to look into this deeper right now.

The change form oleg has high quality, including many test cases which 
are releated to TCLHTTP_USE_EXTERNALTOUTF but not committed yet.

all the best, and many thanks to Oleg!

-g

Re: [naviserver-devel] Fw: ns_http and 8-bit enodings

[Gustaf N. <ne...@wu...>]

On 05.09.20 14:50, Gustaf Neumann wrote:
> I did not have time yet to dig into TCLHTTP_USE_EXTERNALTOUTF the 
> other test cases depending on that. My gut feeling says "this is 
> necessary", but my brain says "why this was no problem so far" and 
> "what will be the collateral damage"  if we do so.
>
> The flag TCLHTTP_USE_EXTERNALTOUTF is still in the code but needs 
> further investigation. However, i am going mid of next week on holiday 
> (boat), where i will not be reachable ... until then many things are 
> piled up on my side, so i have no chance to look into this deeper 
> right now.

My plan would be as follows: Prepare a release for the near future (i'll 
be at 10 days offline, maybe i can work on change summary etc.). For 
this release, leave the TCLHTTP_USE_EXTERNALTOUTF for the time being in 
the code (deactivated by default), and address the encoding issue in 
detail after that.

Users depending on releases will get a long list of new features with 
4.99.20 and they will not be afraid of changed encodings etc. Other 
users with complex encoding/recoding requirements can use the new 
"-binary" flag of ns_http.

all the best

-g

Re: [naviserver-devel] Fw: ns_http and 8-bit enodings

[Zoran V. <zv...@ar...>]

pls take the time off  to relax. work is not going to run away...
i am also off until 20th of sept.
have a great time!

Am 10. September 2020 12:51:24 MESZ schrieb Gustaf Neumann <ne...@wu...>:
>On 05.09.20 14:50, Gustaf Neumann wrote:
>> I did not have time yet to dig into TCLHTTP_USE_EXTERNALTOUTF the 
>> other test cases depending on that. My gut feeling says "this is 
>> necessary", but my brain says "why this was no problem so far" and 
>> "what will be the collateral damage"  if we do so.
>>
>> The flag TCLHTTP_USE_EXTERNALTOUTF is still in the code but needs 
>> further investigation. However, i am going mid of next week on
>holiday 
>> (boat), where i will not be reachable ... until then many things are 
>> piled up on my side, so i have no chance to look into this deeper 
>> right now.
>
>My plan would be as follows: Prepare a release for the near future
>(i'll 
>be at 10 days offline, maybe i can work on change summary etc.). For 
>this release, leave the TCLHTTP_USE_EXTERNALTOUTF for the time being in
>
>the code (deactivated by default), and address the encoding issue in 
>detail after that.
>
>Users depending on releases will get a long list of new features with 
>4.99.20 and they will not be afraid of changed encodings etc. Other 
>users with complex encoding/recoding requirements can use the new 
>"-binary" flag of ns_http.
>
>all the best
>
>-g
>
>
>
>_______________________________________________
>naviserver-devel mailing list
>nav...@li...
>https://lists.sourceforge.net/lists/listinfo/naviserver-devel