Re: [naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Dear all,

i have committed a change to achieve a more consistent and compliant
behavior.

Since all form and query processing of NaviServer happens via API
(ns_urldecode, ns_getform), the current architecture does not allow
direct error messages. The NaviServer philosophy is that the (Tcl)
developer should have the option to handle such cases application
specific.

We had recently changes to address this (mostly driven by vulnerability
scanners) by letting e.g. "ns_urldecode" to raise an exception when
this happens. This change completes this by also raising an exception
for "ns_getform" in such conditions. Note that raising an exception
might be a potential incompatibility for invalid data (which was
"swallowed" before).

The regression test was extended to handle such cases.

There is one more thing (in ns_connchan, so far, not able
to reproduce) that i would like to have a look on
before making the next release candidate available.

all the best

-gn

On 02.05.22 14:29, Gustaf Neumann wrote:
>
> Dear David and all,
>
> I looked into this issue, and I do not like the current situation either.
> In the current snapshot, a GET request with invalid coded
> query variables is rejected, while the POST request leads just
> to the warning, and the invalid entry is omitted.
>
> W3C [1] says in the reference for Multilingual form encoding:
> > If non-UTF-8 data is received, an error message should be sent back.
>
> This means, that the only defensible logic is to reject in both cases
> the request as invalid. One can certainly send single-byte funny character
> data in URLs, which is invalid UTF8 (e.g. "%9C" or "%E6" etc.),
> but for these requests, the charset has to be specified, either
> via content type, or via the default URL encoding in the NaviServer
> configuration... see example (2)  below.
>
> As mentioned earlier, there are increasingly many attacks with invalid
> UTF-8 data (also by vulnerability scanners), so we to be strict here.
>
> I will try to address the outstanding issues ASAP and provide then
> another RC.
>
> All the best
>
> -gn
>
> [1] https://www.w3.org/International/questions/qa-forms-utf-8
>
>
>   # POST request with already encoded form data (x-www-form-urlencoded)
>   $ curl -X POST -d "p1=a%C5%93Cb&p2=a%E6b" localhost:8100/upload.tcl
>
>   # POST request with already encoded form data, but proper encoding
>   $ curl -X POST -H "Content-Type: application/x-www-form-urlencoded; charset=iso-8859-1" -d "p2=a%E6b" localhost:8100/upload.tcl
>
>   # POST + x-www-form-urlencoded, but let curl do the encoding
>   $ curl -X POST -d "p1=aœb" -d $(echo -e 'p2=a\xE6b') localhost:8100/upload.tcl
>
>   # POST + multipart/form-data, let curl do the encoding
>   $ curl -X POST -F "p1=aœb" -F $(echo -e 'p2=a\xE6b') localhost:8100/upload.tcl
>
>   POST request with already encoded form data (x-www-form-urlencoded)
>   $ curl -X GET  "localhost:8100/upload.tcl?p1=a%C5%93Cb&p2=a%E6b"
> On 28.04.22 17:45, David Osborne wrote:
>> Hi Gustaf,
>>
>> We've been testing *4.99.24 rc1* and it seems pretty solid so far. 
>> Thanks for all the work that went into it.
>>
>> One change of behaviour that is causing us issues is the handling of 
>> invalid UTF8 characters.
>>
>> We have a system which regularly POSTs data to NaviServer - sometimes 
>> (for reasons we're looking into) the POST'ed data received by 
>> NaviServer can contain urlencoded characters which don't exist in 
>> UTF8 ( for example *%9C* instead of *%C5%93*).
>>
>> In previous versions of NaviServer, this causes an invalid character 
>> to be embedded in the data when we save it.
>>
>> Now, in version 4.99.24 we, rightly, get the warning "*Warning: 
>> decoded string is invalid UTF-8:*".
>> But the additional behaviour is that the entire form variable seems 
>> to be dropped.
>>
>> I just wanted to query if that is the intended behaviour?
>>
>> I've seen some servers convert such invalid characters to *\ufffd* 
>> (\ufffd being "replacement character" - "used to replace an incoming 
>> character whose value is unknown or unrepresentable in Unicode") - 
>> but not sure which is the correct behaviour.
>>
>> Regards,
>> Dave
>>
>>
>>
>>
>>
>> _______________________________________________
>> naviserver-devel mailing list
>> nav...@li...
>> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> -- 
> Univ.Prof. Dr. Gustaf Neumann
> Head of the Institute of Information Systems and New Media
> of Vienna University of Economics and Business
> Program Director of MSc "Information Systems"
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"