Re: [naviserver-devel] Towards NaviServer 4.99.24

[Gustaf N. <ne...@wu...>]

Dear David and all,

I looked into this issue, and I do not like the current situation either.
In the current snapshot, a GET request with invalid coded
query variables is rejected, while the POST request leads just
to the warning, and the invalid entry is omitted.

W3C [1] says in the reference for Multilingual form encoding:
 > If non-UTF-8 data is received, an error message should be sent back.

This means, that the only defensible logic is to reject in both cases
the request as invalid. One can certainly send single-byte funny character
data in URLs, which is invalid UTF8 (e.g. "%9C" or "%E6" etc.),
but for these requests, the charset has to be specified, either
via content type, or via the default URL encoding in the NaviServer
configuration... see example (2)  below.

As mentioned earlier, there are increasingly many attacks with invalid
UTF-8 data (also by vulnerability scanners), so we to be strict here.

I will try to address the outstanding issues ASAP and provide then
another RC.

All the best

-gn

[1] https://www.w3.org/International/questions/qa-forms-utf-8


  # POST request with already encoded form data (x-www-form-urlencoded)
  $ curl -X POST -d "p1=a%C5%93Cb&p2=a%E6b" localhost:8100/upload.tcl

  # POST request with already encoded form data, but proper encoding
  $ curl -X POST -H "Content-Type: application/x-www-form-urlencoded; charset=iso-8859-1" -d "p2=a%E6b" localhost:8100/upload.tcl

  # POST + x-www-form-urlencoded, but let curl do the encoding
  $ curl -X POST -d "p1=aœb" -d $(echo -e 'p2=a\xE6b') localhost:8100/upload.tcl

  # POST + multipart/form-data, let curl do the encoding
  $ curl -X POST -F "p1=aœb" -F $(echo -e 'p2=a\xE6b') localhost:8100/upload.tcl

  POST request with already encoded form data (x-www-form-urlencoded)
  $ curl -X GET  "localhost:8100/upload.tcl?p1=a%C5%93Cb&p2=a%E6b"

On 28.04.22 17:45, David Osborne wrote:
> Hi Gustaf,
>
> We've been testing *4.99.24 rc1* and it seems pretty solid so far. 
> Thanks for all the work that went into it.
>
> One change of behaviour that is causing us issues is the handling of 
> invalid UTF8 characters.
>
> We have a system which regularly POSTs data to NaviServer - sometimes 
> (for reasons we're looking into) the POST'ed data received by 
> NaviServer can contain urlencoded characters which don't exist in UTF8 
> ( for example *%9C* instead of *%C5%93*).
>
> In previous versions of NaviServer, this causes an invalid character 
> to be embedded in the data when we save it.
>
> Now, in version 4.99.24 we, rightly, get the warning "*Warning: 
> decoded string is invalid UTF-8:*".
> But the additional behaviour is that the entire form variable seems to 
> be dropped.
>
> I just wanted to query if that is the intended behaviour?
>
> I've seen some servers convert such invalid characters to *\ufffd* 
> (\ufffd being "replacement character" - "used to replace an incoming 
> character whose value is unknown or unrepresentable in Unicode") - but 
> not sure which is the correct behaviour.
>
> Regards,
> Dave
>
>
>
>
>
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"