cpp-netlib-devel

[cpp-netlib-devel] soliciting ideas concerning client errors

[John P. F. <jf...@ov...>]

At the moment this a basic_client::get() request from my branch, which 
is working off the trunk interface:

response const get( basic_request<tag> const & request_, const 
connection_type connection=persistent
) {
  return request_policy::sync_process
  (request_, "GET", true, follow_redirect_, connection, 
*connection_supplier_);
};

Concerning that, I'm not satisfied with the way errors are handled (or 
lack there-of). Some errors can be coped with and still complete the 
request; others can't. The question is how to handle the former.

My solution is to differ that choice to the user on a per request basis. 
This would be used to express certain server deviations from a protocol 
standard:

IE:
//rfc1945_extended is a request_policy based off the http 1.0 spec
struct rfc1945_extended::response_deviation {
   //IE: the request would still continue even if the server didn't
         acknowledge persistence or non-persistence, or provided a 
Connection:
         not in accordance with the specification
   bool allow_incorrect_persistence;
   bool allow_version_mismatch;
   bool allow_missing_whatever_header;
   ...
};
...
response const get(const response_deviation&, const 
std::stack<boost::error_code>& ...)

This makes a throw depend on a deviation. In any case, errors up to that 
point are still pushed to the stack which, with a response fragment, 
would accompany the exception.

Dean has also personally suggested an interface similar to this to 
toggle on/off throw, which I have modified for purposes of this
discussion:

  std::stack<boost::system::error_code> errors;
  http::response response; 
  http::rfc1945_extended::response_deviation deviation(..);
  tie(response, errors) = client.get(request, deviation, http::nothrow);

Questions and comments welcome.

John

Re: [cpp-netlib-devel] soliciting ideas concerning client errors

[Glyn M. <gly...@gm...>]

Hello John,


Firstly, I have noticed a lot of activity in subversion, but I've been so
far unable to take a look at your work in depth.

2009/1/10 John P. Feltz <jf...@ov...>

>
> My solution is to differ that choice to the user on a per request basis.
> This would be used to express certain server deviations from a protocol
> standard:
>
>
<snip />

>
>
> This makes a throw depend on a deviation. In any case, errors up to that
> point are still pushed to the stack which, with a response fragment,
> would accompany the exception.
>

I think this is good, it gives the client more information about server
deviations and gives the user more control on how to deal with them.  How do
you decide the what flags to use in the response_deviation struct?  Are you
proposing this as an overload of the get method or a replacement?


Glyn

Re: [cpp-netlib-devel] soliciting ideas concerning client errors

[John P. F. <jf...@ov...>]

This would replace get() and other applicable base post/put/del/head's.

The deviations would be based off two criteria:
-The specification(ie: rfc1945) by which the request_policy processes 
the request (it's coupled with policy)
-In cases that, while still allowing processing of a get/post() etc, 
would do something counter to what the user expects from the interface, 
such as a unmatched http version or persistence.

John
 
Glyn Matthews wrote:
> Hello John,
>
>
> Firstly, I have noticed a lot of activity in subversion, but I've been so
> far unable to take a look at your work in depth.
>
> 2009/1/10 John P. Feltz <jf...@ov...>
>
>   
>> My solution is to differ that choice to the user on a per request basis.
>> This would be used to express certain server deviations from a protocol
>> standard:
>>     
> <snip />
>   
>> This makes a throw depend on a deviation. In any case, errors up to that
>> point are still pushed to the stack which, with a response fragment,
>> would accompany the exception.
>>
>>     
>
> I think this is good, it gives the client more information about server
> deviations and gives the user more control on how to deal with them.  How do
> you decide what flags to use in the response_deviation struct?  Are you
> proposing this as an overload of the get method or a replacement?
>
>
> Glyn
>
>   

Re: [cpp-netlib-devel] soliciting ideas concerning client errors

[Dean M. B. <mik...@gm...>]

Hi John,

On Mon, Jan 12, 2009 at 11:50 PM, John P. Feltz <jf...@ov...> wrote:
> This would replace get() and other applicable base post/put/del/head's.
>

Are you sure you really want to make the API's more complicated than
it currently stands?

I like the idea of doing:

using http::request;
using http::response;
typedef client<message_tag, 1, 1, policies<persistent, caching,
cookies> > http_client;
http_client client_;
request request_("http://boost.org");
response response_ = client_.get(request_);

And keeping the API simple.

> The deviations would be based off two criteria:
> -The specification(ie: rfc1945) by which the request_policy processes
> the request (it's coupled with policy)

The get/put/head/delete/post(...) functions don't have to be too
complicated. If it's already part of the policies chosen, we can have
a default deviation as part of the signature. At most we can provide
an overload to the existing API instead of replacing the simple API
we've already been presenting.

The goal is really simplicity more than really sticking hard to standards.

> -In cases that, while still allowing processing of a get/post() etc,
> would do something counter to what the user expects from the interface,
> such as a unmatched http version or persistence.
>

Actually, if you notice HTTP 1.1 is meant to be backwards compatible
to HTTP 1.0. At best, you just want to make the version information
available in the response and let the users deal with a different HTTP
version in the response rather than making the library needlessly
complicated in terms of API.

I can understand the need for the asynchronous client to be a little
bit more involved (like taking a reference to an io_service at
construction time and allowing the io_service object to be run
externally of the HTTP client) and taking functions that deal with raw
buffers or even functions that deal with already-crafted
http::response objects. However even in these situations, let's try to
be consistent with the theme of simplicity of an API.

I particularly don't like the idea that I need to set up deviations
when I've already chosen which HTTP version I want to use -- and that
deviations should complicate my usage of an HTTP client when I only
usually want to get the body of the response instead of sticking hard
to the standard. ;)

If you meant that these were to be overloads instead of replacements
(as exposed publicly through the basic_client<> specializations) then
I wouldn't have any objections to them. At this time, I need to see
the code closer to see what you intend to do with it. :)

HTH

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] soliciting ideas concerning client errors

[John P. F. <jf...@ov...>]

Dean Michael Berris wrote:
> Hi John,
>
> On Mon, Jan 12, 2009 at 11:50 PM, John P. Feltz <jf...@ov...> wrote:
>   
>> This would replace get() and other applicable base post/put/del/head's.
>>
>>     
>
> Are you sure you really want to make the API's more complicated than
> it currently stands?
>
>   
In this case: Yes, and this is only the tip of the ice-berg.
> I like the idea of doing:
>
> using http::request;
> using http::response;
> typedef client<message_tag, 1, 1, policies<persistent, caching,
> cookies> > http_client;
> http_client client_;
> request request_("http://boost.org");
> response response_ = client_.get(request_);
>
> And keeping the API simple.
>   
I do not want to confuse things by trying to equate the above exactly 
with basic_client in my branch. I'll start by saying that first -from 
implementation experience, there is no such thing as a 1.1. or 1.0 
client. At best you have an old 1.0, a revised 1.0, and 1.1. The 
specifications by which this basic_client functions off of are the only 
feasible way to identify and delineate it's behavior. This is why in my 
branch this takes the form of policies which are based off of either one 
specification, or a certain combination.

Speaking of ice-bergs, while I do appreciate the original intentions of 
simplicity behind the client's API, due to expanding implementation 
concerns and overlooked error handling issues, this view might warrant 
changing. Consider the case where prior-cached connection fails: a 
connection which was retrieved from a supplier external to the 
client-with the supplier being possibly shared by other clients. This 
poses a problem for the user. If the connection was retrieved and used 
as part of a forwarded request several levels deep, the resulting error 
isn't going to be something easily identifiable or managed. While this 
is perhaps a case for encapsulating the client completely, it all 
depends on how oblivious we expect users of this basic_client to be. At 
the moment, I had planned in the next branch release that auto 
forwarding and dealing with persistent connections to be something 
removed from the basic_client. Instead a optional wrapper would perform 
the "driving" of a forwarded request and additionally encapsulate the 
connection/resolved cache. This would take the shape of your previous 
source example and I don't see this as a significant change. If this 
could be done at the basic_client level through a policy configuration 
than I would support that as well, however for _right now_, I don't see 
an easy to way to do that.
>> The deviations would be based off two criteria:
>> -The specification(ie: rfc1945) by which the request_policy processes
>> the request (it's coupled with policy)
>>     
>
> The get/put/head/delete/post(...) functions don't have to be too
> complicated. If it's already part of the policies chosen, we can have
> a default deviation as part of the signature. At most we can provide
> an overload to the existing API instead of replacing the simple API
> we've already been presenting.
>
> The goal is really simplicity more than really sticking hard to standards.
>
>   
A default is fine.
>> -In cases that, while still allowing processing of a get/post() etc,
>> would do something counter to what the user expects from the interface,
>> such as a unmatched http version or persistence.
>>
>>     
>
> Actually, if you notice HTTP 1.1 is meant to be backwards compatible
> to HTTP 1.0. At best, you just want to make the version information
> available in the response and let the users deal with a different HTTP
> version in the response rather than making the library needlessly
> complicated in terms of API.
>   
If the user receives a version that is out of spec -in many cases they 
have a strong reason not to complete a request. This is important for 
both efficiency and compliance.
> I can understand the need for the asynchronous client to be a little
> bit more involved (like taking a reference to an io_service at
> construction time and allowing the io_service object to be run
> externally of the HTTP client) and taking functions that deal with raw
> buffers or even functions that deal with already-crafted
> http::response objects. However even in these situations, let's try to
> be consistent with the theme of simplicity of an API.
>   
I have no comment regarding a-sync usage as I've not looked into that 
issue in depth, I've only tried to make existing policy functions as 
re-usable as possible for that case.
> I particularly don't like the idea that I need to set up deviations
> when I've already chosen which HTTP version I want to use -- and that
> deviations should complicate my usage of an HTTP client when I only
> usually want to get the body of the response instead of sticking hard
> to the standard. ;)
>   
A default is fine.
> If you meant that these were to be overloads instead of replacements
> (as exposed publicly through the basic_client<> specializations) then
> I wouldn't have any objections to them. At this time, I need to see
> the code closer to see what you intend to do with it. :)
>
> HTH
>
>   
Derived overloads might work, though you run into cases of un-orthogonal 
policies (at least I have with this). That would also require 
specialization and/or sub-classing of a deviation/non-deviation rfc 
policies in my branch and I would prefer to keep the current set for now.

John

Re: [cpp-netlib-devel] soliciting ideas concerning client errors

[Dean M. B. <mik...@gm...>]

Hi John,

On Tue, Jan 13, 2009 at 2:02 PM, John P. Feltz <jf...@ov...> wrote:
> Dean Michael Berris wrote:
>> Hi John,
>>
>> On Mon, Jan 12, 2009 at 11:50 PM, John P. Feltz <jf...@ov...> wrote:
>>
>>> This would replace get() and other applicable base post/put/del/head's.
>>>
>>>
>>
>> Are you sure you really want to make the API's more complicated than
>> it currently stands?
>>
>>
> In this case: Yes, and this is only the tip of the ice-berg.

I think we may be running into some needless rigidity or complexity
here. Let me address some of the issues you've raised below.

>> I like the idea of doing:
>>
>> using http::request;
>> using http::response;
>> typedef client<message_tag, 1, 1, policies<persistent, caching,
>> cookies> > http_client;
>> http_client client_;
>> request request_("http://boost.org");
>> response response_ = client_.get(request_);
>>
>> And keeping the API simple.
>>
> I do not want to confuse things by trying to equate the above exactly
> with basic_client in my branch. I'll start by saying that first -from
> implementation experience, there is no such thing as a 1.1. or 1.0
> client. At best you have an old 1.0, a revised 1.0, and 1.1. The
> specifications by which this basic_client functions off of are the only
> feasible way to identify and delineate it's behavior. This is why in my
> branch this takes the form of policies which are based off of either one
> specification, or a certain combination.
>

I agree that based on experience there aren't strictly 1.0 or 1.1
clients -- but the original intent of the library was to keep it
simple: so simple in fact that much of the nuances of the HTTP
protocol are hidden from the user. The aim is not to treat the users
as "stupid" but to give them a consistent and simple API from which
they can write their more important application logic around.

I appreciate the idea of using policies, but the idea of the policies
as far as generic programming and design goes allows for well-defined
points of specialization. Let me try an example (untested):

template <
  class Tag,
  class Policies,
  unsigned version_major,
  unsigned version_minor
>
class basic_client : mpl:inherit_linearly<Policies>::type {
  private:
    typedef mpl:inherit_linearly<Policies>::type base;
  // ...
  basic_response<Tag> get(basic_request<Tag> const & request) {
    shared_ptr<socket> socket_ = base::init_connection(request);
    return base::perform_request(request, "GET");
  }
}

Here the example is contrived so that the idea simply is to defer
whatever implementation of init_connection is available to the
available policies. Of course this assumes that the policies used are
orthogonal. Design-wise it would be nice to enforce at compile-time
the concepts of the policies valid for the basic_client, but that's
for later. ;)

The lack of an existing HTTP Client that's "really 1.0" or "really
1.1" is not an excuse to come up with one. ;-)

So what I'm saying really is, we have a chance (since this is
basically from scratch anyway) to be able to do something that others
have avoided: writing a simple HTTP client which works when you say
it's really 1.0, or that it's really 1.1 -- in the simplest way
possible. The reason they're templates are that if you don't like how
they're implemented, then you can specialize them to your liking. :-)

> Speaking of ice-bergs, while I do appreciate the original intentions of
> simplicity behind the client's API, due to expanding implementation
> concerns and overlooked error handling issues, this view might warrant
> changing. Consider the case where prior-cached connection fails: a
> connection which was retrieved from a supplier external to the
> client-with the supplier being possibly shared by other clients. This
> poses a problem for the user. If the connection was retrieved and used
> as part of a forwarded request several levels deep, the resulting error
> isn't going to be something easily identifiable or managed. While this
> is perhaps a case for encapsulating the client completely, it all
> depends on how oblivious we expect users of this basic_client to be. At
> the moment, I had planned in the next branch release that auto
> forwarding and dealing with persistent connections to be something
> removed from the basic_client. Instead a optional wrapper would perform
> the "driving" of a forwarded request and additionally encapsulate the
> connection/resolved cache. This would take the shape of your previous
> source example and I don't see this as a significant change. If this
> could be done at the basic_client level through a policy configuration
> than I would support that as well, however for _right now_, I don't see
> an easy to way to do that.

Actually when it comes to error-handling efficiency (i.e. avoiding
exceptions which I think are perfectly reasonable to have), I would
have been happy with something like this:

template <...>
class basic_client {
  basic_response<Tag> get(basic_request<Tag> const & request);
  tuple<basic_response<Tag>, error_code> get(basic_request<Tag> const
& request, no_throw_t(*)());
}

This way, if you call 'get' with the nothrow function pointer
argument, then you've got yourself covered -- and instead of a
throwing implementation, the client can return a pair containing the
(possibly empty) basic_response<Tag> and an (possibly
default-constructed) error_code.

About expecting the users to be oblivious, yes this is part of the
point -- I envisioned not making the user worry about recoverable
errors from within the client. There's even a way to make this happen
without making the implementation too complicated. I can say I'm
working on my own refactorings, but I'm doing it as part of Friendster
at the moment so I need to clear them first before releasing as open
source.

It's more or less what I'd call a "just work" API -- and in case of
unrecoverable failures, throw/return an error.

>>> The deviations would be based off two criteria:
>>> -The specification(ie: rfc1945) by which the request_policy processes
>>> the request (it's coupled with policy)
>>>
>>
>> The get/put/head/delete/post(...) functions don't have to be too
>> complicated. If it's already part of the policies chosen, we can have
>> a default deviation as part of the signature. At most we can provide
>> an overload to the existing API instead of replacing the simple API
>> we've already been presenting.
>>
>> The goal is really simplicity more than really sticking hard to standards.
>>
>>
> A default is fine.
>>> -In cases that, while still allowing processing of a get/post() etc,
>>> would do something counter to what the user expects from the interface,
>>> such as a unmatched http version or persistence.
>>>
>>>
>>
>> Actually, if you notice HTTP 1.1 is meant to be backwards compatible
>> to HTTP 1.0. At best, you just want to make the version information
>> available in the response and let the users deal with a different HTTP
>> version in the response rather than making the library needlessly
>> complicated in terms of API.
>>
> If the user receives a version that is out of spec -in many cases they
> have a strong reason not to complete a request. This is important for
> both efficiency and compliance.

Actually, there's nothing in the HTTP 1.0 spec that says a response
that's HTTP 1.x where x != 0 is an invalid response. There's also
nothing in the spec that says that HTTP 1.1 requests cannot be
completed when the HTTP 1.0 response is received.

There are however some servers which will not accept certain HTTP
versions -- sometimes you can write an HTTP Server that will send an
HTTP error 505 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.6)
when it doesn't support the version you're trying to use as far as
HTTP is concerned. I don't know of any servers though which will not
honor HTTP 1.0 client requests and respond accordingly.

BTW, if you notice, HTTP 1.1 enables some new things that were not
possible with HTTP 1.0 (or were technically unsupported) like
streaming encodings and persistent connections / request pipelining on
both the client and server side. The specification is clear about what
the client should and should not do when certain things happen in
these persistent connections and pipelined request scenarios.

So I don't see how strict compliance cannot be possible with the
current API and with the current design of the client.

>> I can understand the need for the asynchronous client to be a little
>> bit more involved (like taking a reference to an io_service at
>> construction time and allowing the io_service object to be run
>> externally of the HTTP client) and taking functions that deal with raw
>> buffers or even functions that deal with already-crafted
>> http::response objects. However even in these situations, let's try to
>> be consistent with the theme of simplicity of an API.
>>
> I have no comment regarding a-sync usage as I've not looked into that
> issue in depth, I've only tried to make existing policy functions as
> re-usable as possible for that case.

Which I think is the wrong approach if you ask me.

The idea (and the process) of generic programming will usually lead
you from specific to generic. My best advice (?) would be to implement
the specific first, write the tests/specifications (if you're doing
Behavior Driven Development) and then refactor mercilessly in the end.

The aim of the client is to be generic as well as compliant up to a
point. HTTP 1.0 doesn't support persistent connections, although
extensions were made with 'Connection: Keep-Alive' and the
'Keep-Alive: ...' headers -- these can be supported, but they're going
to be different specializations of the basic_client.

>> I particularly don't like the idea that I need to set up deviations
>> when I've already chosen which HTTP version I want to use -- and that
>> deviations should complicate my usage of an HTTP client when I only
>> usually want to get the body of the response instead of sticking hard
>> to the standard. ;)
>>
> A default is fine.
>> If you meant that these were to be overloads instead of replacements
>> (as exposed publicly through the basic_client<> specializations) then
>> I wouldn't have any objections to them. At this time, I need to see
>> the code closer to see what you intend to do with it. :)
>>
>> HTH
>>
>>
> Derived overloads might work, though you run into cases of un-orthogonal
> policies (at least I have with this). That would also require
> specialization and/or sub-classing of a deviation/non-deviation rfc
> policies in my branch and I would prefer to keep the current set for now.
>

I think if you really want to be able to decompose the client into
multiple orthogonal policies, you might want to look into introducing
more specific extension points in the code instead of being hampered
by earlier (internal) design decisions. ;)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] soliciting ideas concerning client errors

[John P. F. <jf...@ov...>]

Dean Michael Berris wrote:
> Hi John,
>
> On Tue, Jan 13, 2009 at 2:02 PM, John P. Feltz <jf...@ov...> wrote:
>   
>> Dean Michael Berris wrote:
>>     
>>> Hi John,
>>>
>>> On Mon, Jan 12, 2009 at 11:50 PM, John P. Feltz <jf...@ov...> wrote:
>>>
>>>       
>>>> This would replace get() and other applicable base post/put/del/head's.
>>>>
>>>>
>>>>         
>>> Are you sure you really want to make the API's more complicated than
>>> it currently stands?
>>>
>>>
>>>       
>> In this case: Yes, and this is only the tip of the ice-berg.
>>     
>
> I think we may be running into some needless rigidity or complexity
> here. Let me address some of the issues you've raised below.
>
>   
>>> I like the idea of doing:
>>>
>>> using http::request;
>>> using http::response;
>>> typedef client<message_tag, 1, 1, policies<persistent, caching,
>>> cookies> > http_client;
>>> http_client client_;
>>> request request_("http://boost.org");
>>> response response_ = client_.get(request_);
>>>
>>> And keeping the API simple.
>>>
>>>       
>> I do not want to confuse things by trying to equate the above exactly
>> with basic_client in my branch. I'll start by saying that first -from
>> implementation experience, there is no such thing as a 1.1. or 1.0
>> client. At best you have an old 1.0, a revised 1.0, and 1.1. The
>> specifications by which this basic_client functions off of are the only
>> feasible way to identify and delineate it's behavior. This is why in my
>> branch this takes the form of policies which are based off of either one
>> specification, or a certain combination.
>>
>>     
>
> I agree that based on experience there aren't strictly 1.0 or 1.1
> clients -- but the original intent of the library was to keep it
> simple: so simple in fact that much of the nuances of the HTTP
> protocol are hidden from the user. The aim is not to treat the users
> as "stupid" but to give them a consistent and simple API from which
> they can write their more important application logic around.
>
>   
I'm not suggesting getting rid of this simple API. rfc based policies 
are crucial as a foundation for defining client behavior at certain 
level. I'm suggesting that level to be at the implementation. A wrapper 
can be provided with certain predefined options, be those polices or 
non-policies to do what was the original design goal for the client class.
> I appreciate the idea of using policies, but the idea of the policies
> as far as generic programming and design goes allows for well-defined
> points of specialization. Let me try an example (untested):
>
> template <
>   class Tag,
>   class Policies,
>   unsigned version_major,
>   unsigned version_minor
>   
> class basic_client : mpl:inherit_linearly<Policies>::type {
>   private:
>     typedef mpl:inherit_linearly<Policies>::type base;
>   // ...
>   basic_response<Tag> get(basic_request<Tag> const & request) {
>     shared_ptr<socket> socket_ = base::init_connection(request);
>     return base::perform_request(request, "GET");
>   }
> }
> whatever implementation of init_connection is available to the
> available policies. Of course this assumes that the policies used are
> orthogonal. Design-wise it would be nice to enforce at compile-time
> the concepts of the policies valid for the basic_client, but that's
> for later. ;)
>   
Of the necessary concerns, I see the need to provide one for supplying 
the resolved, the sockets, and logic governing both. These seem 
nonorthogonal, and so facading is more appropriate for right now. 
Additionally, I don't write code through the lens of a policy based 
strategy until I've seen what the implementation requires.
> The lack of an existing HTTP Client that's "really 1.0" or "really
> 1.1" is not an excuse to come up with one. ;-)
>   
I'm sure users wishing to optimize requests to legacy servers or servers 
which do not follow a complete 1.1 spec will find that rather 
frustrating. Especially since it something so simple to design for (see 
branch).
> So what I'm saying really is, we have a chance (since this is
> basically from scratch anyway) to be able to do something that others
> have avoided: writing a simple HTTP client which works when you say
> it's really 1.0, or that it's really 1.1 -- in the simplest way
> possible. The reason they're templates are that if you don't like how
> they're implemented, then you can specialize them to your liking. :-)
>   
This is a fine goal, however assuming that the simplest design will 
address what I believe to be all the complex concerns is flawed. Let us 
solve the complex issues _first_, then simplify. There is nothing wrong 
with the basic_clients API in principle, though I think this should 
address complex users needs first and so delegation of that class to 
single request processing + a wrapper incorporating certain policies 
with predefined behavior is the best course for now.
>> Speaking of ice-bergs, while I do appreciate the original intentions of
>> simplicity behind the client's API, due to expanding implementation
>> concerns and overlooked error handling issues, this view might warrant
>> changing. Consider the case where prior-cached connection fails: a
>> connection which was retrieved from a supplier external to the
>> client-with the supplier being possibly shared by other clients. This
>> poses a problem for the user. If the connection was retrieved and used
>> as part of a forwarded request several levels deep, the resulting error
>> isn't going to be something easily identifiable or managed. While this
>> is perhaps a case for encapsulating the client completely, it all
>> depends on how oblivious we expect users of this basic_client to be. At
>> the moment, I had planned in the next branch release that auto
>> forwarding and dealing with persistent connections to be something
>> removed from the basic_client. Instead a optional wrapper would perform
>> the "driving" of a forwarded request and additionally encapsulate the
>> connection/resolved cache. This would take the shape of your previous
>> source example and I don't see this as a significant change. If this
>> could be done at the basic_client level through a policy configuration
>> than I would support that as well, however for _right now_, I don't see
>> an easy to way to do that.
>>     
>
> Actually when it comes to error-handling efficiency (i.e. avoiding
> exceptions which I think are perfectly reasonable to have), I would
> have been happy with something like this:
>
> template <...>
> class basic_client {
>   basic_response<Tag> get(basic_request<Tag> const & request);
>   tuple<basic_response<Tag>, error_code> get(basic_request<Tag> const
> & request, no_throw_t(*)());
> }
>
> This way, if you call 'get' with the nothrow function pointer
> argument, then you've got yourself covered -- and instead of a
> throwing implementation, the client can return a pair containing the
> (possibly empty) basic_response<Tag> and an (possibly
> default-constructed) error_code.
>   
I don't see where function pointers belong in a client which intends to 
remain simple. I'm also against a no-throw parameter because it is less 
explicit. If an exception is thrown then the user knows there's an issue 
and a valid response was not returned. If a response is returned either 
way, than it is easier for the user to miss.
> About expecting the users to be oblivious, yes this is part of the
> point -- I envisioned not making the user worry about recoverable
> errors from within the client. There's even a way to make this happen
> without making the implementation too complicated. I can say I'm
> working on my own refactorings, but I'm doing it as part of Friendster
> at the moment so I need to clear them first before releasing as open
> source.
>
> It's more or less what I'd call a "just work" API -- and in case of
> unrecoverable failures, throw/return an error.
>
>   
>>>> The deviations would be based off two criteria:
>>>> -The specification(ie: rfc1945) by which the request_policy processes
>>>> the request (it's coupled with policy)
>>>>
>>>>         
>>> The get/put/head/delete/post(...) functions don't have to be too
>>> complicated. If it's already part of the policies chosen, we can have
>>> a default deviation as part of the signature. At most we can provide
>>> an overload to the existing API instead of replacing the simple API
>>> we've already been presenting.
>>>
>>> The goal is really simplicity more than really sticking hard to standards.
>>>
>>>
>>>       
>> A default is fine.
>>     
>>>> -In cases that, while still allowing processing of a get/post() etc,
>>>> would do something counter to what the user expects from the interface,
>>>> such as a unmatched http version or persistence.
>>>>
>>>>
>>>>         
>>> Actually, if you notice HTTP 1.1 is meant to be backwards compatible
>>> to HTTP 1.0. At best, you just want to make the version information
>>> available in the response and let the users deal with a different HTTP
>>> version in the response rather than making the library needlessly
>>> complicated in terms of API.
>>>
>>>       
>> If the user receives a version that is out of spec -in many cases they
>> have a strong reason not to complete a request. This is important for
>> both efficiency and compliance.
>>     
>
> Actually, there's nothing in the HTTP 1.0 spec that says a response
> that's HTTP 1.x where x != 0 is an invalid response. There's also
> nothing in the spec that says that HTTP 1.1 requests cannot be
> completed when the HTTP 1.0 response is received.
>   
These are cases that I'm not concerned with.
> There are however some servers which will not accept certain HTTP
> versions -- sometimes you can write an HTTP Server that will send an
> HTTP error 505 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.6)
> when it doesn't support the version you're trying to use as far as
> HTTP is concerned. I don't know of any servers though which will not
> honor HTTP 1.0 client requests and respond accordingly.
>
> BTW, if you notice, HTTP 1.1 enables some new things that were not
> possible with HTTP 1.0 (or were technically unsupported) like
> streaming encodings and persistent connections / request pipelining on
> both the client and server side. The specification is clear about what
> the client should and should not do when certain things happen in
> these persistent connections and pipelined request scenarios.
>
> So I don't see how strict compliance cannot be possible with the
> current API and with the current design of the client.
>
>   
>>> I can understand the need for the asynchronous client to be a little
>>> bit more involved (like taking a reference to an io_service at
>>> construction time and allowing the io_service object to be run
>>> externally of the HTTP client) and taking functions that deal with raw
>>> buffers or even functions that deal with already-crafted
>>> http::response objects. However even in these situations, let's try to
>>> be consistent with the theme of simplicity of an API.
>>>
>>>       
>> I have no comment regarding a-sync usage as I've not looked into that
>> issue in depth, I've only tried to make existing policy functions as
>> re-usable as possible for that case.
>>     
>
> Which I think is the wrong approach if you ask me.
>
>   
By existing I was referring to things like stateless items such as 
append_* in the branch. If these can't be used asynchronously I would be 
curious as to why. I've expended no other effort in regards to this.
> The idea (and the process) of generic programming will usually lead
> you from specific to generic. My best advice (?) would be to implement
> the specific first, write the tests/specifications (if you're doing
> Behavior Driven Development) and then refactor mercilessly in the end.
>
> The aim of the client is to be generic as well as compliant up to a
> point. HTTP 1.0 doesn't support persistent connections, although
> extensions were made with 'Connection: Keep-Alive' and the
> 'Keep-Alive: ...' headers -- these can be supported, but they're going
> to be different specializations of the basic_client.
>
>   
>>> I particularly don't like the idea that I need to set up deviations
>>> when I've already chosen which HTTP version I want to use -- and that
>>> deviations should complicate my usage of an HTTP client when I only
>>> usually want to get the body of the response instead of sticking hard
>>> to the standard. ;)
>>>
>>>       
>> A default is fine.
>>     
>>> If you meant that these were to be overloads instead of replacements
>>> (as exposed publicly through the basic_client<> specializations) then
>>> I wouldn't have any objections to them. At this time, I need to see
>>> the code closer to see what you intend to do with it. :)
>>>
>>> HTH
>>>
>>>
>>>       
>> Derived overloads might work, though you run into cases of un-orthogonal
>> policies (at least I have with this). That would also require
>> specialization and/or sub-classing of a deviation/non-deviation rfc
>> policies in my branch and I would prefer to keep the current set for now.
>>
>>     
>
> I think if you really want to be able to decompose the client into
> multiple orthogonal policies, you might want to look into introducing
> more specific extension points in the code instead of being hampered
> by earlier (internal) design decisions. ;)
>
>   
I am growing tired of discussing this and would prefer to just get on 
with implementing _something_ in the branch. After tomorrow I'll be 
going back to my day job and time for working on this will be limited 
and I don't want to spend that time debating design decisions in which 
there can be very little compromise.

Re: [cpp-netlib-devel] soliciting ideas concerning client errors

[Dean M. B. <mik...@gm...>]

On Wed, Jan 14, 2009 at 6:03 AM, John P. Feltz <jf...@ov...> wrote:
> Dean Michael Berris wrote:
>>
>> I agree that based on experience there aren't strictly 1.0 or 1.1
>> clients -- but the original intent of the library was to keep it
>> simple: so simple in fact that much of the nuances of the HTTP
>> protocol are hidden from the user. The aim is not to treat the users
>> as "stupid" but to give them a consistent and simple API from which
>> they can write their more important application logic around.
>>
>>
>
> I'm not suggesting getting rid of this simple API. rfc based policies
> are crucial as a foundation for defining client behavior at certain
> level. I'm suggesting that level to be at the implementation. A wrapper
> can be provided with certain predefined options, be those polices or
> non-policies to do what was the original design goal for the client class.
>

Ok.

How do you see this wrapper helping keep the API simple then?

>> I appreciate the idea of using policies, but the idea of the policies
>> as far as generic programming and design goes allows for well-defined
>> points of specialization. Let me try an example (untested):
>>
>> template <
>>   class Tag,
>>   class Policies,
>>   unsigned version_major,
>>   unsigned version_minor
>>
>> class basic_client : mpl:inherit_linearly<Policies>::type {
>>   private:
>>     typedef mpl:inherit_linearly<Policies>::type base;
>>   // ...
>>   basic_response<Tag> get(basic_request<Tag> const & request) {
>>     shared_ptr<socket> socket_ = base::init_connection(request);
>>     return base::perform_request(request, "GET");
>>   }
>> }
>> whatever implementation of init_connection is available to the
>> available policies. Of course this assumes that the policies used are
>> orthogonal. Design-wise it would be nice to enforce at compile-time
>> the concepts of the policies valid for the basic_client, but that's
>> for later. ;)
>>
>
> Of the necessary concerns, I see the need to provide one for supplying
> the resolved, the sockets, and logic governing both. These seem
> nonorthogonal, and so facading is more appropriate for right now.
> Additionally, I don't write code through the lens of a policy based
> strategy until I've seen what the implementation requires.
>

Great. So what about nested policies?

management_policy<resolver_policy, socket_policy>

Where you have many management policies:

persistent_management_policy
strict_http_1_1_management_policy
proxy_aware_http
...

And many resolver policies

async_resolver_policy
sync_resolver_policy
...

And socket creation policies

sync_connect_policy
async_connect_policy
timeout_policy
...

?

>> The lack of an existing HTTP Client that's "really 1.0" or "really
>> 1.1" is not an excuse to come up with one. ;-)
>>
>
> I'm sure users wishing to optimize requests to legacy servers or servers
> which do not follow a complete 1.1 spec will find that rather
> frustrating. Especially since it something so simple to design for (see
> branch).
>

True, but notice that HTTP 1.0 servers which return HTTP 505 errors
that says it doesn't support HTTP 1.1 should allow the user to use the
appropriate client. This logic should be provided by the user of the
library, instead of the library writers -- where our aim is to provide
a reasonably compliant *and* simple API.

This is similar to the discussion about providing a public method
'perform' with a string provided as the action and my stance doesn't
change -- the point of an HTTP client is to be reasonably compliant to
the HTTP 1.0/1.1 spec, and thus only supported or "standard" methods
are supported.

So having something like:

struct http_client_base {
  http_client_base() {}
  virtual http::response get(http::request const & request);
}

struct http_1_1_client : http_client_base {
  http_1_1_client() {}
  http::response get(http::request const & request) { return
client_.get(request); }
  private:
    basic_client<http::message_tag, 1, 1> client_;
}

struct http_1_0_client : http_client_base {
  http_1_0_client() {}
  http::response get(http::request const & request) { return
client_.get(request); }
  private:
    basic_client<http::message_tag, 1, 0> client_;
}

That's written by the user of the library to support his use case of a
dynamically determined HTTP 1.0 and HTTP 1.1 client would not be
entirely impossible. OTOH, the responsibility of
basic_client<http::message_tag, 1, 0> would be to provide a
sufficiently compliant HTTP 1.0 client, and
basic_client<http::message_tag, 1, 1> would be to provide a
sufficiently compliant HTTP 1.1 client.

>> So what I'm saying really is, we have a chance (since this is
>> basically from scratch anyway) to be able to do something that others
>> have avoided: writing a simple HTTP client which works when you say
>> it's really 1.0, or that it's really 1.1 -- in the simplest way
>> possible. The reason they're templates are that if you don't like how
>> they're implemented, then you can specialize them to your liking. :-)
>>
>
> This is a fine goal, however assuming that the simplest design will
> address what I believe to be all the complex concerns is flawed. Let us
> solve the complex issues _first_, then simplify. There is nothing wrong
> with the basic_clients API in principle, though I think this should
> address complex users needs first and so delegation of that class to
> single request processing + a wrapper incorporating certain policies
> with predefined behavior is the best course for now.
>

I still don't see how complex really the concerns are.

If you're worried about unrecoverable errors, then you have exceptions
-- which is perfectly fine. I would even go farther and say we should
have special exception types to denote different types of exceptions
all deriving from std::runtime_error.

If you're worried about unrecoverable errors *and* you don't want
exceptions, you'll use the signature I've shown below. I might even go
and change the signature from

  tuple<basic_response<Tag>, error_code> get(basic_request<Tag> const
& request, no_throw_t(*)());

to something like

  tuple<optional<basic_response<Tag> >, optional<error_code> >
get(basic_request<Tag> const & request, no_throw_t(*)());

And enforce that either the response is not defined if error_code is
defined, or that basic_response<Tag> was defined if error_code is not
defined. I might even explore the possibility of using a
boost::variant, but I'm not too comfortable with the usage semantics
of variants. This might require some more thought on my part, but I'm
thinking aloud on how I'll address the conveyance of errors from the
client side.

>>
>> Actually when it comes to error-handling efficiency (i.e. avoiding
>> exceptions which I think are perfectly reasonable to have), I would
>> have been happy with something like this:
>>
>> template <...>
>> class basic_client {
>>   basic_response<Tag> get(basic_request<Tag> const & request);
>>   tuple<basic_response<Tag>, error_code> get(basic_request<Tag> const
>> & request, no_throw_t(*)());
>> }
>>
>> This way, if you call 'get' with the nothrow function pointer
>> argument, then you've got yourself covered -- and instead of a
>> throwing implementation, the client can return a pair containing the
>> (possibly empty) basic_response<Tag> and an (possibly
>> default-constructed) error_code.
>>
>
> I don't see where function pointers belong in a client which intends to
> remain simple. I'm also against a no-throw parameter because it is less
> explicit. If an exception is thrown then the user knows there's an issue
> and a valid response was not returned. If a response is returned either
> way, than it is easier for the user to miss.
>

This is how you use that signature (with the unused function pointer):

http::client client_;
http::request request("http://www.booost.org");
http::response response;
boost::system::error_code error_code;
tie(response, error_code) = client_.get(request, http::nothrow);

The http::nothrow function would be defined as such:

struct no_throw_t;

inline no_throw_t no_throw() {
  return no_throw_t();
}

struct no_throw_t {
  private:
    no_throw_t();
    friend no_throw_t no_throw();
};

:-)

>>
>> Actually, there's nothing in the HTTP 1.0 spec that says a response
>> that's HTTP 1.x where x != 0 is an invalid response. There's also
>> nothing in the spec that says that HTTP 1.1 requests cannot be
>> completed when the HTTP 1.0 response is received.
>>
>
> These are cases that I'm not concerned with.
>

Okay, but if you were concerned with compliance when it comes to
things like persistent connections, request pipelining support, and
streaming encodings, maybe making them specializations of the basic
client using a different tag on different HTTP version numbers would
be something you'd like to explore. :-)

>>> I have no comment regarding a-sync usage as I've not looked into that
>>> issue in depth, I've only tried to make existing policy functions as
>>> re-usable as possible for that case.
>>>
>>
>> Which I think is the wrong approach if you ask me.
>>
>>
> By existing I was referring to things like stateless items such as
> append_* in the branch. If these can't be used asynchronously I would be
> curious as to why. I've expended no other effort in regards to this.

Ok. :)


>>
>> I think if you really want to be able to decompose the client into
>> multiple orthogonal policies, you might want to look into introducing
>> more specific extension points in the code instead of being hampered
>> by earlier (internal) design decisions. ;)
>>
>>
>
> I am growing tired of discussing this and would prefer to just get on
> with implementing _something_ in the branch. After tomorrow I'll be
> going back to my day job and time for working on this will be limited
> and I don't want to spend that time debating design decisions in which
> there can be very little compromise.
>

Okay.

At any rate, since the basic_client is meant to be the "world-facing"
API, it's not entirely impossible to write auxiliary classes that were
used by a specialization of the basic_client with a different tag.
What I mean by this is that for example, you can have a different tag
and a different set of (new) traits and accessors from which you can
design the basic_client specialization around.

struct non_throwing_tag;
struct persistent_connections_tag;

namespace triats {
  template <class T>
  struct http_tag;

  template <>
  struct http_tag<non_throwing_tag> {
    typedef http::message_tag type;
  };

  template <>
  struct http_tag<persistent_connections_tag> {
    typedef http::message_tag type;
  };
}

template<>
class basic_client<non_throwing_tag, 1, 0> {
  // different API of a client that doesn't throw on errors
}

template<>
class basic_client<persistent_connections_tag, 1, 0> {
  // implement a HTTP 1.0 client that supports persistent connections
}

Internally in all these specializations, you can pretty much do
whatever you want. ;-)

Then from there (and this approach) we can essentially write unit
tests that make sure the implementations of the specifializations do
what they're meant to do, and then we can look into how we can merge
the current implementation of the basic_client to accomodate or even
leverage the work you've done into the trunk. Maybe that would then
get us release 0.4 having an HTTP 1.1 client available. :D

How does that sound to you?

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] soliciting ideas concerning client errors

[John P. F. <jf...@ov...>]

Dean Michael Berris wrote:
> </snip>
>
> At any rate, since the basic_client is meant to be the "world-facing"
> API, it's not entirely impossible to write auxiliary classes that were
> used by a specialization of the basic_client with a different tag.
> What I mean by this is that for example, you can have a different tag
> and a different set of (new) traits and accessors from which you can
> design the basic_client specialization around.
>
> struct non_throwing_tag;
> struct persistent_connections_tag;
>
> namespace triats {
>   template <class T>
>   struct http_tag;
>
>   template <>
>   struct http_tag<non_throwing_tag> {
>     typedef http::message_tag type;
>   };
>
>   template <>
>   struct http_tag<persistent_connections_tag> {
>     typedef http::message_tag type;
>   };
> }
>
> template<>
> class basic_client<non_throwing_tag, 1, 0> {
>   // different API of a client that doesn't throw on errors
> }
>
> template<>
> class basic_client<persistent_connections_tag, 1, 0> {
>   // implement a HTTP 1.0 client that supports persistent connections
> }
>
> Internally in all these specializations, you can pretty much do
> whatever you want. ;-)
>
> Then from there (and this approach) we can essentially write unit
> tests that make sure the implementations of the specifializations do
> what they're meant to do, and then we can look into how we can merge
> the current implementation of the basic_client to accomodate or even
> leverage the work you've done into the trunk. Maybe that would then
> get us release 0.4 having an HTTP 1.1 client available. :D
>
> How does that sound to you?
>
>   
I'll look into it at the wrapper class level. I'm content with the 
current rfc family tree, from which is growth is easy, however those 
could be further decomposed and "infused" with tags to configure 
different client behavior as needed by the basic_client and wrapper.

John