Re: [naviserver-devel] naviserver-devel Digest, Vol 84, Issue 4

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Dear all

I've committed some code concerning the discussed urlencode reform.
The code is now conforming to RFC3986 (2005), and recommendation
of HTML 4.01  (for www-url-encoded).  It spits out warnings when
characters are included  in the "location" header fields misses
encodings (in the obvious cases). Furthermore, it contains a new
flag "-uppercase" for OAuth ... and is able to decode upper
case hex codes as well.  The new code is already running
now on OpenACS.org to see if i missed  something obvious.

I've not yet updated the documentation and the regression test.
... and - not to forget - i  am planning to raise exceptions,
when invalid hex codes are trying to be decoded.

-gn

PS: Below the more detailed commit messages:

- The new code conforms to RFC3986 (2005), which has a more precise
   and differently structured definition (e.g. no 'unwise' characters)
   of characters encoded in URLs. The coding of the query part is
   actually defined in the HTML 4.01 definitions.

- Coding of space (" ") in the path and query part is still different
   as in older versions of NaviServer (spaces are coded as "*" in the
   query part and "%20" in the path segments. This distinction is not
   not necessary according to the RFCs.

- The coding tables are documented in detail, containing design
   considerations.

- A new flag "-uppercase" added for supporting encoding for OAuth
   (RFC 5849); note that the "path" segment encoding has to be used to
   avoid coding space as "+".

- A warning is produced now, when an URL is passed to the location
   field (e.g. ns_returnredirect), which is not properly encoded.
   Only characters, which have to be always coded, are flagged.

- One can obtain the previous encoding behavior when compiling with
   "RFC1738" defined.

Am 23.03.17 um 09:00 schrieb Michael Aram:
> Dear all,
>
> I agree with Wolfgang that this should generally be taken care of by 
> the app developer. However, as there is probably much legacy code out 
> there that passes the URL unencoded to ns_returnredirect (at least in 
> our OpenACS installations there are many such places), I would opt for 
> a solution where this can be parameterized via a boolean parameter of 
> the function (b). Ideally, the default of this parameter could be 
> changed via the NaviServer config file (setting the default behavior 
> to e.g. "encode" instead of the current "leave untouched"). If this is 
> considered to much effort, at least a warning would be very useful.
>
> As a side note: if the ns_urlencode code is touched, I would suggest 
> to add a parameter that allows for choosing the case of the encoded 
> characters. The background for this is that OAuth 1 strictly requires 
> the percent-encoded characters to be uppercase only, as these become 
> part of the signature base string (without this restriction, two 
> signatures could differ only because of different encoding 
> implementations at the endpoints). See 
> https://tools.ietf.org/html/rfc5849 chapter 3.6.
>
> All the best,
> Michael
>
>
>
> On Thu, Mar 23, 2017 at 7:49 AM 
> <nav...@li... 
> <mailto:nav...@li...>> wrote:
>
>     Send naviserver-devel mailing list submissions to
>     nav...@li...
>     <mailto:nav...@li...>
>
>     To subscribe or unsubscribe via the World Wide Web, visit
>     https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>     or, via email, send a message with subject or body 'help' to
>     nav...@li...
>     <mailto:nav...@li...>
>
>     You can reach the person managing the list at
>     nav...@li...
>     <mailto:nav...@li...>
>
>     When replying, please edit your Subject line so it is more specific
>     than "Re: Contents of naviserver-devel digest..."
>
>
>     Today's Topics:
>
>        1. url-encoding and ns_returnredirect, RFC updates (Gustaf Neumann)
>        2. Re: url-encoding and ns_returnredirect, RFC updates
>           (Wolfgang Winkler)
>
>
>     ----------------------------------------------------------------------
>
>     Message: 1
>     Date: Wed, 22 Mar 2017 11:50:49 +0100
>     From: Gustaf Neumann <ne...@wu... <mailto:ne...@wu...>>
>     Subject: [naviserver-devel] url-encoding and ns_returnredirect, RFC
>             updates
>     To: Navidevel <nav...@li...
>     <mailto:nav...@li...>>
>     Message-ID: <888...@wu...
>     <mailto:888...@wu...>>
>     Content-Type: text/plain; charset="utf-8"
>
>     Dear all,
>
>     as it looks, edge is more picky about the encoding of URLs in the
>     location: header field (see e.g. recent entry in the OpenACS issue
>     tracker [1]). RFC 7231 states [2] that
>
>           Location = URI-reference
>
>     but as well:
>
>            Note: Some recipients attempt to recover from Location
>     fields that
>            are not valid URI references.  This specification does not
>     mandate
>            or define such processing, but does allow it for the sake
>     of robustness.
>
>     The BNF in [3] clear, that it has to be encoded (see snippet for path
>     segments)
>
>            URI-reference = URI / relative-ref
>            relative-ref  = relative-part [ "?" query ] [ "#" fragment ]
>            relative-part = "//" authority path-abempty
>                          / path-absolute
>                          / path-noscheme
>                          / path-empty
>
>            path-abempty  = *( "/" segment )
>            path-absolute = "/" [ segment-nz *( "/" segment ) ]
>            path-noscheme = segment-nz-nc *( "/" segment )
>            path-rootless = segment-nz *( "/" segment )
>            path-empty    = 0<pchar>
>
>
>            segment       = *pchar
>            segment-nz    = 1*pchar
>            segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims /
>     "@" )
>                          ; non-zero-length segment without any colon ":"
>            pchar         = unreserved / pct-encoded / sub-delims / ":"
>     / "@"
>
>
>     Naviserver passes the URL as is from e.g. a ns_returnredirect to the
>     "Location:" field.
>
>     So the question is, should ns
>
>     a) take care about this encoding b) take care about this encoding via
>     optional flag c) do nothing and leave  the responsibility to the
>     application programmer (current situation) d) provide a warning
>     when an
>     "obviously" unencoded url is passed to ns_returnredirect
>
>     I think, (a) is not useful, since ns can't decide from the string,
>     whether a "/" in the part is e.g. a delimiter or part of the segment.
>     Furthermore, it would break existing programs that encode already the
>     urls correctly. (b) might be useful in simple cases.
>
>     I am inclined towards (d), although an exact check for every char
>     which
>     should have been escaped might be to costly on some characters
>     (checking
>     if "%" was used just as an escape indicator, etc.); however, an
>     application developer can get hints via (d), where the
>     url-encoding was
>     probably lacking.
>
>     While looking at the nsd/urlencode.c i saw that the encoding is more
>     conservative than commented (.... "All ASCII control characters (00-1f
>     and 7f) and the URI 'delim'  and 'unwise' characters are encoded"
>     ...),
>     but it encodes as well the characters from 0x80 to 0xff. Do I
>     interprete
>     this correctly, that this refers to the differences/confusions between
>     RFC1738 (1994) and RFC1808 (1995) vs. RFC2396 (1998), see [5]. The
>     code
>     says, it conforms with RFC1738, so probably an update to at least
>     RFC2396 seems appropriate.
>
>     Comments?
>
>     -g
>
>     [1] http://openacs.org/bugtracker/openacs/bug?bug_number=3312 [2]
>     https://tools.ietf.org/html/rfc7231#page-68 [3]
>     https://tools.ietf.org/html/rfc3986#appendix-A [4]
>     https://tools.ietf.org/html/rfc2396 [5]
>     https://tools.ietf.org/html/rfc2396#appendix-G.2
>
>     -------------- next part --------------
>     An HTML attachment was scrubbed...
>
>     ------------------------------
>
>     Message: 2
>     Date: Thu, 23 Mar 2017 07:22:54 +0100
>     From: Wolfgang Winkler <wol...@di...
>     <mailto:wol...@di...>>
>     Subject: Re: [naviserver-devel] url-encoding and ns_returnredirect,
>             RFC updates
>     To: nav...@li...
>     <mailto:nav...@li...>
>     Message-ID:
>             <129...@di...
>     <mailto:129...@di...>>
>     Content-Type: text/plain; charset="windows-1252"
>
>     Hi!
>
>     I opt for version c (programmers responsibility) as url encoding
>     can be
>     tricky stuff if you don't know the context of the passed in url. A
>     flag
>     or maybe an extra proc for checking an url for problems could be
>     useful,
>     but that's something that can be done easily with ns_urlencode and
>     ns_urldecode.
>
>     Updating to RFC2396 would be most welcome. As stated in the RFC:
>
>         This document
>         defines the generic syntax of URI, including both absolute and
>         relative forms, and guidelines for their use; it revises and
>     replaces
>         the generic definitions in RFC 1738 and RFC 1808.
>
>     and
>
>         This document updates and merges "Uniform Resource Locators"
>         [RFC1738] and "Relative Uniform Resource Locators" [RFC1808]
>     in order
>         to define a single, generic syntax for all URI.
>
>     To me it seems, that the RFC1738 has been, at least in parts,
>     deprecated.
>
>     regards,
>
>     Wolfgang
>
>
>     Am 2017-03-22 um 11:50 schrieb Gustaf Neumann:
>     >
>     > Dear all,
>     >
>     > as it looks, edge is more picky about the encoding of URLs in the
>     > location: header field (see e.g. recent entry in the OpenACS issue
>     > tracker [1]). RFC 7231 states [2] that
>     >
>     >       Location = URI-reference
>     >
>     > but as well:
>     >
>     >        Note: Some recipients attempt to recover from Location
>     fields that
>     >        are not valid URI references.  This specification does
>     not mandate
>     >        or define such processing, but does allow it for the sake
>     of robustness.
>     >
>     > The BNF in [3] clear, that it has to be encoded (see snippet for
>     path
>     > segments)
>     >
>     >        URI-reference = URI / relative-ref
>     >        relative-ref  = relative-part [ "?" query ] [ "#" fragment ]
>     >        relative-part = "//" authority path-abempty
>     >                      / path-absolute
>     >                      / path-noscheme
>     >                      / path-empty
>     >
>     >        path-abempty  = *( "/" segment )
>     >        path-absolute = "/" [ segment-nz *( "/" segment ) ]
>     >        path-noscheme = segment-nz-nc *( "/" segment )
>     >        path-rootless = segment-nz *( "/" segment )
>     >        path-empty    = 0<pchar>
>     >        segment       = *pchar
>     >        segment-nz    = 1*pchar
>     >        segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims
>     / "@" )
>     >                      ; non-zero-length segment without any colon ":"
>     >        pchar         = unreserved / pct-encoded / sub-delims /
>     ":" / "@"
>     >
>     > Naviserver passes the URL as is from e.g. a ns_returnredirect to the
>     > "Location:" field.
>     >
>     > So the question is, should ns
>     >
>     > a) take care about this encoding b) take care about this
>     encoding via
>     > optional flag c) do nothing and leave  the responsibility to the
>     > application programmer (current situation) d) provide a warning when
>     > an "obviously" unencoded url is passed to ns_returnredirect
>     >
>     > I think, (a) is not useful, since ns can't decide from the string,
>     > whether a "/" in the part is e.g. a delimiter or part of the
>     segment.
>     > Furthermore, it would break existing programs that encode
>     already the
>     > urls correctly. (b) might be useful in simple cases.
>     >
>     > I am inclined towards (d), although an exact check for every char
>     > which should have been escaped might be to costly on some characters
>     > (checking if "%" was used just as an escape indicator, etc.);
>     however,
>     > an application developer can get hints via (d), where the
>     url-encoding
>     > was probably lacking.
>     >
>     > While looking at the nsd/urlencode.c i saw that the encoding is more
>     > conservative than commented (.... "All ASCII control characters
>     (00-1f
>     > and 7f) and the URI 'delim'  and 'unwise' characters are encoded"
>     > ...), but it encodes as well the characters from 0x80 to 0xff. Do I
>     > interprete this correctly, that this refers to the
>     > differences/confusions between RFC1738 (1994) and RFC1808 (1995) vs.
>     > RFC2396 (1998), see [5]. The code says, it conforms with RFC1738, so
>     > probably an update to at least RFC2396 seems appropriate.
>     >
>     > Comments?
>     >
>     > -g
>     >
>     > [1] http://openacs.org/bugtracker/openacs/bug?bug_number=3312 [2]
>     > https://tools.ietf.org/html/rfc7231#page-68 [3]
>     > https://tools.ietf.org/html/rfc3986#appendix-A [4]
>     > https://tools.ietf.org/html/rfc2396 [5]
>     > https://tools.ietf.org/html/rfc2396#appendix-G.2
>

Re: [naviserver-devel] naviserver-devel Digest, Vol 84, Issue 4

NaviServer, a high performance web server written in C and Tcl

Re: [naviserver-devel] naviserver-devel Digest, Vol 84, Issue 4