cpp-netlib-devel — Developers Mailing List

Re: [cpp-netlib-devel] localhost_tests failures

[Dean M. C. B. <dmb...@fr...>]

Hi Kim,

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Kim Gräsman
> Sent: Tuesday, September 02, 2008 8:27 PM
> To: C++ Networking Library Developers Mailing List
> Subject: [cpp-netlib-devel] localhost_tests failures
> 
> Hi guys,
> 
> The localhost_tests suite is failing for me, out of the box, 
> with the following:
> 
> --
> Running 12 test cases...
> libs/network/test/localhost_tests.cpp(190): error in
> "text_file_query": check body(response_).length() == size failed [113
> != 117]
> libs/network/test/localhost_tests.cpp(260): fatal error in
> "cgi_query": exception thrown by r = c.get(req)
> libs/network/test/localhost_tests.cpp(271): fatal error in
> "cgi_multi_line_headers": exception thrown by r = c.get(req)
> 
> *** 3 failures detected in test suite "http 1.0 localhost tests"
> --
> 
> This is on Windows XP SP2, VC 9.0 (no service pack), Boost 1.37
> (straight from the SVN trunk) and Python 2.5.2.
> 
> I haven't investigated the latter two closely (it appears they aren't
> getting any body whatsoever), but the first one is strange. The file
> size of test.xml, according to Windows Explorer, is 117 bytes, but the
> Python script is only serving 113 bytes (I checked this with Firefox
> as a client as well).
> 

The issue has something to do with Python bugs that Divye has posted to the list earlier.

There's currently no workaround for this, since we haven't been implementing an HTTP Server in the library yet. At any rate, this doesn't manifest itself in Linux.

> So... I'm not sure where it's going wrong. Any ideas on where to start
> looking? It looks like it might be some kind of encoding mismatch,
> perhaps?
> 

Currently there's no encoding handling that goes on in cpp-netlib -- we deal with raw bytes that come from the wire, which we treat as character buffers.

Like I mentioned already, it's a Python in Windows bug. ;)

> Also, there seems to be a race condition where the tests occasionally
> launch before Python has had a chance to start, but I think that may
> be solvable by extracting the server start out into a global fixture
> (something I'm working on here -- patch coming).
> 

Looking forward to the patch. :)

HTH

--
Dean Michael Berris
Software Engineer, Friendster, Inc. 

[cpp-netlib-devel] localhost_tests failures

[K. G. <kim...@gm...>]

Hi guys,

The localhost_tests suite is failing for me, out of the box, with the following:

--
Running 12 test cases...
libs/network/test/localhost_tests.cpp(190): error in
"text_file_query": check body(response_).length() == size failed [113
!= 117]
libs/network/test/localhost_tests.cpp(260): fatal error in
"cgi_query": exception thrown by r = c.get(req)
libs/network/test/localhost_tests.cpp(271): fatal error in
"cgi_multi_line_headers": exception thrown by r = c.get(req)

*** 3 failures detected in test suite "http 1.0 localhost tests"
--

This is on Windows XP SP2, VC 9.0 (no service pack), Boost 1.37
(straight from the SVN trunk) and Python 2.5.2.

I haven't investigated the latter two closely (it appears they aren't
getting any body whatsoever), but the first one is strange. The file
size of test.xml, according to Windows Explorer, is 117 bytes, but the
Python script is only serving 113 bytes (I checked this with Firefox
as a client as well).

So... I'm not sure where it's going wrong. Any ideas on where to start
looking? It looks like it might be some kind of encoding mismatch,
perhaps?

Also, there seems to be a race condition where the tests occasionally
launch before Python has had a chance to start, but I think that may
be solvable by extracting the server start out into a global fixture
(something I'm working on here -- patch coming).

Thanks for any input on the first issue!

- Kim

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Dean M. B. <mik...@gm...>]

Hi Agustin,

Sorry this took a while. I spent an offline weekend, and only now am I
able to catch up on the Internet backlog. ;)

On Fri, Aug 29, 2008 at 1:59 PM, Agustín K-ballo Bergé
<kab...@ho...> wrote:
> Dean Michael Berris escribió:
>> 1. Encapsulating the 'method' into the http::request object.
>> 2. Making the client accept the method as a string.
>> 3. Using the "push" abstraction to convey the action.
>>
>> I decided against all three because:
>>
>> For #1:
>> - This puts undue burden to the request crafting aspect: it makes it
>> cumbersome to craft requests that you intend to re-use anyway (perform
>> a HEAD first, and if headers don't fit a certain criteria then do a
>> GET using the same resource)
>> - It complicates the http::request implementation by deviating too
>> much from the message framework/abstraction
>>
> I would expect some orthogonality among the library. That is, if a
> client sends a request then I'd expect a server to be able to receive
> one. How could this be handled under the current design, in which the
> http 'method' is not part of the request?
>

Right. Actually, here's a little rationale on why the method is not
part of the actual request object:

1. The 'method' is important only as part of the action performed.
This is orthogonal from the message payload, which is the headers, the
destination (URI), and the source (undefined).

2. Encapsulating the 'method' from the actual *act* of _requesting_
keeps the semantics clear: given this message payload
(URI+headers+body) perform a GET|POST|HEAD|PUT|DELETE with this
payload.

> According to RFC2616, "A request message from a client to a server
> includes, within the first line of that message, the method to be
> applied to the resource, (...)". I would like to see a clear
> compile-time mapping for the objects, just as I like them for HTTP
> 'methods'. Perhaps something like this:
>
>    using namespace boost::network;
>    http::resource resource("http://www.boost.org/");
>    http::request request_;
>    http::client client_;
>    http::response response_;
>    request_ = http::get(resource);
>    response_ = client_.request(request_);
>
> IMHO, not only the code looks closer to the protocol, but also detaches
> it from the underlaying transport mechanism (client_ here). It may just
> be that this is not what the library seeks for.
>

Right. However:

1. The aim of the library is to make it simple; the HTTP protocol is
by no means simple and thus certain semantics need to be introduced to
make it simpler (i.e. decouple the message payload from the actual act
of requesting).

2. The design does not encourage standalone functions that are very
hard to extend -- http::get(...) will have to return the correct
request object type given the type of the resource object. The builder
interface couples the builder as a nested type of the actual
http::request type you intend to use, making implementation and
extension easier and more deliberate -- that's easier to document as
well.

> I would appreciate some enlightening on the subject.

I can say more, but I hope this helps. :)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Glyn M. <gly...@gm...>]

Hi Agustin,

Sorry I didn't get round to this sooner.

2008/8/29 Agustín K-ballo Bergé <kab...@ho...>

>
> According to RFC2616, "A request message from a client to a server
> includes, within the first line of that message, the method to be
> applied to the resource, (...)". I would like to see a clear
> compile-time mapping for the objects, just as I like them for HTTP
> 'methods'. Perhaps something like this:
>
>    using namespace boost::network;
>    http::resource resource("http://www.boost.org/");
>    http::request request_;
>     http::client client_;
>    http::response response_;
>     request_ = http::get(resource);
>    response_ = client_.request(request_);
>
> IMHO, not only the code looks closer to the protocol, but also detaches
> it from the underlaying transport mechanism (client_ here). It may just
> be that this is not what the library seeks for.
>

The http::resource here would be equivalent to something like an URI
builder, I think.  Otherwise I don't know what you would gain from having a
free http::get function rather than making it a member of client.  I would
expect http::get to actually make a request, rather than construct a request
object.  My original concern was that the request function(s) were members
of class named "client" and I don't any difference here.

Glyn

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Agustín K-b. B. <kab...@ho...>]

Dean Michael Berris escribió:
> 1. Encapsulating the 'method' into the http::request object.
> 2. Making the client accept the method as a string.
> 3. Using the "push" abstraction to convey the action.
>
> I decided against all three because:
>
> For #1:
> - This puts undue burden to the request crafting aspect: it makes it
> cumbersome to craft requests that you intend to re-use anyway (perform
> a HEAD first, and if headers don't fit a certain criteria then do a
> GET using the same resource)
> - It complicates the http::request implementation by deviating too
> much from the message framework/abstraction
>   
I would expect some orthogonality among the library. That is, if a
client sends a request then I'd expect a server to be able to receive
one. How could this be handled under the current design, in which the
http 'method' is not part of the request?

According to RFC2616, "A request message from a client to a server
includes, within the first line of that message, the method to be
applied to the resource, (...)". I would like to see a clear
compile-time mapping for the objects, just as I like them for HTTP
'methods'. Perhaps something like this:

    using namespace boost::network;
    http::resource resource("http://www.boost.org/");
    http::request request_;
    http::client client_;
    http::response response_;
    request_ = http::get(resource);
    response_ = client_.request(request_);

IMHO, not only the code looks closer to the protocol, but also detaches
it from the underlaying transport mechanism (client_ here). It may just
be that this is not what the library seeks for.

I would appreciate some enlightening on the subject.

Agustín K-ballo Bergé.-

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Dean M. B. <mik...@gm...>]

Hi Divye,

On Fri, Aug 29, 2008 at 2:25 AM, Divye Kapoor <div...@gm...> wrote:
>
> On Thu, Aug 28, 2008 at 12:55 PM, Dean Michael Berris
> <mik...@gm...> wrote:
>>
>
> It is quite likely that applications that base their decisions on "inputs"
> from the user will have to create a switch-case logic in their client code.
> It would save a lot of work for various users if the switch-case is
> implemented as a generic perform(...) method which throws an exception on
> invalid methods.

Right. But validating the input at runtime at the library level and
throwing an exception from there is a bad idea especially if you can
avoid it at compile time.

I'd like to keep the switch-case logic to remain in the application
code, because that is something that is specific to the application
and not something the library is supposed to provide. Much like how
std::map<>'s insert returns a pair containing an iterator and a
boolean -- the boolean says whether the data was inserted, and the
library user is free to decide what to do.

The logic that's choosing which HTTP method to perform should be
specific to the application, and is something that the library should
not know about.

>>
>> >> - Type safety (?) is broken because now the users will be able to
>> >> perform 'unsupported' or 'ill-formed' HTTP requests
>> >
>
> Not necessarily, see above.
>

Actually, the idea is simple:

1. Provide a limited HTTP interface to those supported to convey
exactly what the client does support.
2. If you want a custom HTTP client that supports other "non-standard"
methods, you extend the existing client and build on top of what's
already there.

I think type-safety is the wrong word, but it's more about keeping the
interface simple.

>>
>> > I can't think of one off-hand, but I think there may be cases where it
>> > makes sense to let the calling code vary the verb dynamically.
>> > Essentially, something like:
>> >
>> >  client.request("POST", headers, body);
>>
>> >
>
> I would see client.perform(request, method) as a useful addition to the
> client interface as it handles a common use-case.
>

Right, but if method was a string, then I have a problem with it. We
can make 'method' a function signature which performs a compile-time
dispatch to the correct (supported) implementation to choose. But then
that would bring me back to just implement an interface of the actual
methods that the client does support.

>
>>
>> 1. The request object is nowhere to be found (http::request object),
>> which should encapsulate all this information (headers, body) already
>> which is of very little consequence to the client.
>
> The client requires information about the request type (GET, POST etc) and
> the request object. We are simply providing the request type as a runtime
> string rather than as a static method. I realize that it will cause a
> performance penalty due to control paths being included in the code, but
> this can be treated as a convenience function that will reduce the number of
> ugly switch-case constructs in the client code while enforcing method
> validity by exceptions.
>

Why should the library do that, when the user who needs this
functionality should do that on their own? i.e. wrap the dispatching
for their case in a function, and do the determination at runtime and
keep the library simple and intuitive without exposing a potentially
unsafe/fragile and inefficient interface?

I'd rather provide a library that provides a set of robust
implementations than support a wider range of users and end up with an
"ugly" and unsafe interface. ;-)

>> 3. This means inside client::request(...) I'll have to check whether I
>> will ignore/use the body depending on the method string provided.
>> Contrast that to a fixed interface which actually at compile time only
>> puts the code that's needed to perform the request without having to
>> do checks of what to do at runtime.
>
> Users who want such a type of dynamic functionality will have to implement
> such a dispatch table in any case. This function could simply provide a
> dispatch table for each of the typed functions as a convenient shorthand. It
> localizes code that would otherwise have to be rewritten in different
> applications. For those who already know the type of request to make,
> choosing one of the static methods would still be a better option.
>

Right, in this case since they have to implement it for their
application (because their use-case is not the use-case I want to (or
IMO, doesn't make sense) to support) then they should do it and not
ask the library to support it for the general case.

>>
>> The static mapping between the behavior of client::get and HTTP GET
>> requests is intentional because primarily:
>>
>> 1. I don't see a point in making it a run-time check when it can be
>> made clear at compile-time.
>> 2. The semantics of usage are unclear when using 'dynamic' or runtime
>> values to do the decision making that can otherwise be done at compile
>> time in a clearer manner.
>
> I would say:
> 1. Use the typed functions when you already know which type of request to
> make.
> 2. Use the generic function if the decision is based on runtime variables.
> However, be aware of the performance pitfalls and the possible exceptions.
>

Sure, but then it would stick out like a sore thumb and beg the
question: "why do we even have the typed functions when there's
already the generic (unsafe) implementation?". To avoid this, we
support just #1, because that's the best thing to do, and just give
users that need that dispatch facility implement it themselves.

>>
>> If helping the developer avoid shooting themselves in the foot matters
>> (which I really think does, a lot) then keeping the static mapping and
>> semantics intact would be a good premium. I'd think it's more a
>> feature than a deficiency that the HTTP client allows for an
>> expressive interface to allow users to "say what they mean" instead of
>> leaving a wide open door to allow users to easily get things wrong.
>> One of the goals is to make it easier to use the library, and part of
>> that is to make 'getting it wrong' harder than getting it right. :)
>
> This would count on  my list as an ease of use feature with the possible
> performance penalties dissuading me from indiscriminate use. However, I
> acknowledge the merits of retaining a static only mapping to methods. I
> would be happy sticking to the static-only mapping if this feature is felt
> to be against the good sense of making 'getting it wrong' easier.
>

I tend to be conservative in this sense: for example, I'd like to
avoid questions like:

* why doesn't `client.perform(request, "get") do what
client.perform(request,"GET") do?`
* why does client.perform(request, "PSOT") not do anything?

So before the obvious bugs get to runtime, I'd rather:

* client.psot(request) // fail, because there's no method 'psot' in client
* client.GET(request) // fail, because there's no method 'GET' in client

Hope this makes sense why I have a problem supporting a generic
'.perform' method. ;-)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Dean M. C. B. <dmb...@fr...>]

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Kim Gräsman
> Sent: Thursday, August 28, 2008 11:03 PM
> To: C++ Networking Library Developers Mailing List
> Subject: Re: [cpp-netlib-devel] Comments about the HTTP 
> client interface
> 
> Hi Glyn,
> 
> On Thu, Aug 28, 2008 at 13:54, Glyn Matthews 
> <gly...@gm...> wrote:
> >
> > 2008/8/28 Kim Gräsman <kim...@gm...>
> >>
> >> How about user_agent?
> >
> > Yes, this might be better.  "User-Agent" still needs to 
> form part of the
> > HTTP request.  Dean?
> 
> Thinking further about this, the feeling that *I am the client* is
> just as true for *I am the user-agent*, so I don't think it adds any
> value in that regard. It's just a different word. I think I like
> "client" better.
> 

I like 'client' better as well at the moment. 'user_agent' is a little clunkier, harder to remember, and harder to explain outside of HTTP.

Consider for example when we'd like to implement someday an SMTP client, or for instance an FTP client, do we say 'user_agent' for those as well? Somehow 'client' maps better to non-p2p protocols where clearly, from a client-side perspective, you'd be more explicit about using a client or hosting a server. ;-)

At any rate, a typedef of http::client as http::user_agent works as well. :-P

--
Dean Michael Berris
Software Engineer, Friendster, Inc. 

Re: [cpp-netlib-devel] cpp-netlib / pion-net integration status

[Dean M. B. <mik...@gm...>]

Hi Ken,

On Thu, Aug 28, 2008 at 9:58 PM, Ken Hausam <kh...@gm...> wrote:
> What is the status regarding the integration of pion-net into cpp-netlib?

Right now, it's on hold because it currently requires quite a lot of
work to turn pion-net into a header-only library. That falls currently
squarely on my shoulders but I'm still concentrated on implementing
parts of the HTTP client.

> I
> saw a bunch of good questions from Michael Dickey to this list back in
> April, but haven't seen much evidence of any activity since. Just curious as
> I need both an http client and http server for my project and was wondering
> if there was an alpha release of cpp-netlib that contains the http server
> code from pion-net, or if I should just build and include each project
> separately. Thanks.

cpp-netlib doesn't require to be built -- but linking an application
that uses it depends on Boost.System.

You should be alright with using pion-net and cpp-netlib together.
We'd love to know how that goes as well.

Thanks, and I hope this helps!

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[K. G. <kim...@gm...>]

Hi Divye,

On Thu, Aug 28, 2008 at 20:25, Divye Kapoor <div...@gm...> wrote:
>
> Users who want such a type of dynamic functionality will have to implement
> such a dispatch table in any case. This function could simply provide a
> dispatch table for each of the typed functions as a convenient shorthand. It
> localizes code that would otherwise have to be rewritten in different
> applications. For those who already know the type of request to make,
> choosing one of the static methods would still be a better option.

+1. That's the point I was trying to make, but much more eloquently put. Thanks!

- Kim

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Divye K. <div...@gm...>]

Hi Kim and Dean,

On Thu, Aug 28, 2008 at 12:55 PM, Dean Michael Berris <
mik...@gm...> wrote:

> Hi Kim,
>
> On Thu, Aug 28, 2008 at 2:47 PM, Kim Gräsman <kim...@gm...>
> wrote:
> >
> > On Thu, Aug 28, 2008 at 03:47, Dean Michael Berris
> > <mik...@gm...> wrote:
> >>
> >> For #2:
> >> - It puts the method crafting a 'run-time' action, which introduces
> >> if's and control-flow branches that I would like to be able to
> >> eliminate at compile time otherwise
>

It is quite likely that applications that base their decisions on "inputs"
from the user will have to create a switch-case logic in their client code.
It would save a lot of work for various users if the switch-case is
implemented as a generic perform(...) method which throws an exception on
invalid methods.

>
> >> - Type safety (?) is broken because now the users will be able to
> >> perform 'unsupported' or 'ill-formed' HTTP requests
> >
>
Not necessarily, see above.


>
> > I can't think of one off-hand, but I think there may be cases where it
> > makes sense to let the calling code vary the verb dynamically.
> > Essentially, something like:
> >
> >  client.request("POST", headers, body);
>
>
>

I would see client.perform(request, method) as a useful addition to the
client interface as it handles a common use-case.



> 1. The request object is nowhere to be found (http::request object),
> which should encapsulate all this information (headers, body) already
> which is of very little consequence to the client.


The client requires information about the request type (GET, POST etc) and
the request object. We are simply providing the request type as a runtime
string rather than as a static method. I realize that it will cause a
performance penalty due to control paths being included in the code, but
this can be treated as a convenience function that will reduce the number of
ugly switch-case constructs in the client code while enforcing method
validity by exceptions.

3. This means inside client::request(...) I'll have to check whether I
> will ignore/use the body depending on the method string provided.
> Contrast that to a fixed interface which actually at compile time only
> puts the code that's needed to perform the request without having to
> do checks of what to do at runtime.
>

Users who want such a type of dynamic functionality will have to implement
such a dispatch table in any case. This function could simply provide a
dispatch table for each of the typed functions as a convenient shorthand. It
localizes code that would otherwise have to be rewritten in different
applications. For those who already know the type of request to make,
choosing one of the static methods would still be a better option.


> The static mapping between the behavior of client::get and HTTP GET
> requests is intentional because primarily:
>
> 1. I don't see a point in making it a run-time check when it can be
> made clear at compile-time.
> 2. The semantics of usage are unclear when using 'dynamic' or runtime
> values to do the decision making that can otherwise be done at compile
> time in a clearer manner.
>
I would say:
1. Use the typed functions when you already know which type of request to
make.
2. Use the generic function if the decision is based on runtime variables.
However, be aware of the performance pitfalls and the possible exceptions.


>
> If helping the developer avoid shooting themselves in the foot matters
> (which I really think does, a lot) then keeping the static mapping and
> semantics intact would be a good premium. I'd think it's more a
> feature than a deficiency that the HTTP client allows for an
> expressive interface to allow users to "say what they mean" instead of
> leaving a wide open door to allow users to easily get things wrong.
> One of the goals is to make it easier to use the library, and part of
> that is to make 'getting it wrong' harder than getting it right. :)


This would count on  my list as an ease of use feature with the possible
performance penalties dissuading me from indiscriminate use. However, I
acknowledge the merits of retaining a static only mapping to methods. I
would be happy sticking to the static-only mapping if this feature is felt
to be against the good sense of making 'getting it wrong' easier.

> Well, 2c worth,
>
> 2c is a worth a lot from where I am. :-P
>

Same here :-)

Divye Kapoor

-- 
An idealist is one who, on noticing that a rose smells better than a
cabbage, concludes that it will also make better soup.
H. L. Mencken (1880 - 1956)
My official web site: http://people.iitr.ernet.in/shp/061305/
Webmaster: http://www.drkapoorsclinic.com
Blog: http://divyekapoor.blogspot.com

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Glyn M. <gly...@gm...>]

Kim,

2008/8/28 Kim Gräsman <kim...@gm...>

> Hi Glyn,
>
> On Thu, Aug 28, 2008 at 13:54, Glyn Matthews <gly...@gm...>
> wrote:
> >
> > 2008/8/28 Kim Gräsman <kim...@gm...>
> >>
> >> How about user_agent?
> >
> > Yes, this might be better.  "User-Agent" still needs to form part of the
> > HTTP request.  Dean?
>
> Thinking further about this, the feeling that *I am the client* is
> just as true for *I am the user-agent*, so I don't think it adds any
> value in that regard. It's just a different word. I think I like
> "client" better.
>

True, user_agent.get() and user_agent.post() are equally as problematic.
G

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[K. G. <kim...@gm...>]

Hi Glyn,

On Thu, Aug 28, 2008 at 13:54, Glyn Matthews <gly...@gm...> wrote:
>
> 2008/8/28 Kim Gräsman <kim...@gm...>
>>
>> How about user_agent?
>
> Yes, this might be better.  "User-Agent" still needs to form part of the
> HTTP request.  Dean?

Thinking further about this, the feeling that *I am the client* is
just as true for *I am the user-agent*, so I don't think it adds any
value in that regard. It's just a different word. I think I like
"client" better.

- Kim

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[K. G. <kim...@gm...>]

Hi Dean,

On Thu, Aug 28, 2008 at 09:25, Dean Michael Berris
<mik...@gm...> wrote:
>
> If helping the developer avoid shooting themselves in the foot matters
> (which I really think does, a lot) then keeping the static mapping and
> semantics intact would be a good premium. I'd think it's more a
> feature than a deficiency that the HTTP client allows for an
> expressive interface to allow users to "say what they mean" instead of
> leaving a wide open door to allow users to easily get things wrong.

Yep, I agree with that. Though I also think there's value in a
data-driveable interface -- I keep finding myself having to switch on
a set of constants in order to invoke a specific method, whose only
raison d'etre is to encapsulate the same constant values. I think
that's why I like this *as a complement*. There's no doubt in my mind
that the most common usage is simply client::get/post.

Then again, the only case I can see where the verb would vary like
that is some sort of HTTP inspector application, where you can work
from a set of headers and a body and invoke different verbs to see how
a web server reacts. It's still doable with the existing interface,
just clunkier.

Never mind me ;-)

- Kim

[cpp-netlib-devel] cpp-netlib / pion-net integration status

[Ken H. <kh...@gm...>]

What is the status regarding the integration of pion-net into cpp-netlib? I
saw a bunch of good questions from Michael Dickey to this list back in
April, but haven't seen much evidence of any activity since. Just curious as
I need both an http client and http server for my project and was wondering
if there was an alpha release of cpp-netlib that contains the http server
code from pion-net, or if I should just build and include each project
separately. Thanks.

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Glyn M. <gly...@gm...>]

Kim,

2008/8/28 Kim Gräsman <kim...@gm...>

>
> How about user_agent?
>

Yes, this might be better.  "User-Agent" still needs to form part of the
HTTP request.  Dean?

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[K. G. <kim...@gm...>]

Glyn, Dean,

On Thu, Aug 28, 2008 at 10:00, Glyn Matthews <gly...@gm...> wrote:
> Right, The word "server" is really wrong.
>>
>> Then again, thinking about client as a browser, as somebody mentioned,
>> helps, for me at least.
>
> Yeah, especially as Dean explained that he was attempting to break with
> convention with this approach.
>
> So "server" is wrong, "connection" is misleading, given the explanation
> elsewhere in this thread, "client" is perhaps the best name that in this
> context.  I can still imagine potential users being confused by this though.

How about user_agent?

- Kim

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Glyn M. <gly...@gm...>]

Hi Kim,

2008/8/28 Kim Gräsman <kim...@gm...>

> Glyn,
>
> On Thu, Aug 28, 2008 at 09:36, Glyn Matthews <gly...@gm...>
> wrote:
> >
> > In my mind, you send a GET to a server, not the client.  When I write
> code
> > like this I *am* the client, I don't *use* a client.  If I'm the only one
> > being anal about this then I'll let this point go :)
>
> I've thought about this too, but substituting "client" for "server" in
> the API became really confusing. So maybe "connection" is clearer. On
> the other hand, as Dean says, for HTTP 1.0 there is no persistent
> connection (as far as I understand it, anyway), only requests.


Right, The word "server" is really wrong.

>
> Then again, thinking about client as a browser, as somebody mentioned,
> helps, for me at least.
>

Yeah, especially as Dean explained that he was attempting to break with
convention with this approach.

So "server" is wrong, "connection" is misleading, given the explanation
elsewhere in this thread, "client" is perhaps the best name that in this
context.  I can still imagine potential users being confused by this though.

Thanks for your input,
G

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Dean M. B. <mik...@gm...>]

Hi Glyn!

On Thu, Aug 28, 2008 at 3:36 PM, Glyn Matthews <gly...@gm...> wrote:
> Hi Dean and Divye,
>
> 2008/8/28 Dean Michael Berris <mik...@gm...>
>>
>>
>> This is the intended interpretation which I was going for. :-)
>
> In my mind, you send a GET to a server, not the client.  When I write code
> like this I *am* the client, I don't *use* a client.  If I'm the only one
> being anal about this then I'll let this point go :)
>

Right, no you're not being anal I think you have a perfectly good point. :-)

In this case I think the http::client is more a meta-client (client
for a client), or a 'slave' (I'm tempted to use the terms 'robot' or
'device') which is ordered to do things on behalf of a user (in this
context, the user is the application that's using the client). We can
term this as another layer of indirection, which insulates the user
(application) from the dynamics of the protocol implementation
(connections, message formats, etc.).

>>
>> I hope at least this can make it into the documentation or in a
>> rationale document somehow... ;-)
>
> Yep, this was why I opened this subject.
>

Great! :D

>>
>> HTH!
>
> It does, thanks.

My pleasure. :)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Glyn M. <gly...@gm...>]

Hi Dean,

2008/8/28 Dean Michael Berris <mik...@gm...>

>
> What I would be inclined to support is something of a request builder
> implementation, which allows for example:
>
>  http::request::builder builder("http", "www.boost.org");
>  http::request r1 = builder.create("/");
>  http::request r2 = builder.create("/some_uri?param1=value");
>  http::request r3 = builder.create("/login.php",
> "username=user&password=password");
>  http::client client;
>  http::response response = client.get(r1);
>

I'd definitely be in favour of implementing a URI builder.


> The idea here is that we should be able to encapsulate the
> connection-mapping logic in the client between domains and actual
> sockets. That means, inside the client, it may be able to hold on to
> existing connections (in HTTP 1.1 clients) and then perform the
> correct actions.
>
> The aim of the interface is to convey that:
> - the client knows how to handle the connections so you don't have to
> worry about it
> - the request object as far as you are concerned is the base unit of
> data you can use to instruct the client what to do
> - you can craft any http::request and use it with any instance of the
> client without necessarily binding the client to a single
> destination/connection
>
> If it helps, you can think of an http::client object as an
> encapsulated browser of sorts, which can perform well-crafted requests
> on the users' behalf.
>
> This breaks the existing conventions on how HTTP clients look like and
> behave -- and existing library designs. I personally think this is a
> good thing, because we're doing it from scratch anyway. ;)
>

Fair enough ;)


>
> Besides, it makes things easier for developers (I think) because it
> allows a more consistent way of thinking about crafting HTTP requests
> and asking a client to perform a request on their behalf -- as
> compared to being encumbered with the notion that an http::connection
> is bound to a certain destination and that a request object is
> associated with a URI instead of being a fully-encapsulated HTTP
> request.
>

I agree, I think this is an improvement on other HTTP libs that I've used
(as I hoped it should be).  There's still more testing that needs to be done
to ensure that this really is as a robust approach as we want to claim.

Thanks for the detailed explanation,
Glyn

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[K. G. <kim...@gm...>]

Glyn,

On Thu, Aug 28, 2008 at 09:36, Glyn Matthews <gly...@gm...> wrote:
>
> In my mind, you send a GET to a server, not the client.  When I write code
> like this I *am* the client, I don't *use* a client.  If I'm the only one
> being anal about this then I'll let this point go :)

I've thought about this too, but substituting "client" for "server" in
the API became really confusing. So maybe "connection" is clearer. On
the other hand, as Dean says, for HTTP 1.0 there is no persistent
connection (as far as I understand it, anyway), only requests.

Then again, thinking about client as a browser, as somebody mentioned,
helps, for me at least.

Hmm,
- Kim

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Glyn M. <gly...@gm...>]

Hi Dean and Divye,

2008/8/28 Dean Michael Berris <mik...@gm...>

> On Thu, Aug 28, 2008 at 1:01 AM, Divye Kapoor <div...@gm...>
> wrote:
> >
> >
> > On Wed, Aug 27, 2008 at 6:14 PM, Glyn Matthews <gly...@gm...>
> > wrote:
> >>
> >>     using namespace boost::network;
> >>     http::request request("http://www.boost.org/");
> >>     http::client client_;
> >>     http::response response_;
> >>     response_ = client_.get(request);
> >>
> >>
> >> What exactly is the role of the client here?  I'm a little confused by
> the
> >> syntax because it reads as though you're trying to make a get request to
> an
> >> HTTP client.  Would it perhaps be better to rename http::client to
> >> http::connection?
> >
> > Currently, I read the above code as:
> > 1. frame a request for resource http://www.boost.org/
> > 2. Create a client that handles http requests/responses
> > 3. Get the resource specified by the request and provide a response.
> >
>
> This is the intended interpretation which I was going for. :-)
>

In my mind, you send a GET to a server, not the client.  When I write code
like this I *am* the client, I don't *use* a client.  If I'm the only one
being anal about this then I'll let this point go :)


> > However, I do admit that client.head(request) and client.put(request)
> don't
> > seem to be too intuitive in that respect.
> >
>
> I had been thinking about that before I even started, and I was
> thinking more to provide a compile-time and explicit mapping between
> the interfaces and the semantics of the HTTP protocol. More
> importantly, I was thinking of alternatives which were:
>
> 1. Encapsulating the 'method' into the http::request object.
> 2. Making the client accept the method as a string.
> 3. Using the "push" abstraction to convey the action.
>
> I decided against all three because:
>
...
I see your reasoning behind all these.


>
> I hope at least this can make it into the documentation or in a
> rationale document somehow... ;-)
>

Yep, this was why I opened this subject.


>
> HTH!
>

It does, thanks.
G

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Dean M. B. <mik...@gm...>]

Hi Kim,

On Thu, Aug 28, 2008 at 2:47 PM, Kim Gräsman <kim...@gm...> wrote:
>
> On Thu, Aug 28, 2008 at 03:47, Dean Michael Berris
> <mik...@gm...> wrote:
>>
>> For #2:
>> - It puts the method crafting a 'run-time' action, which introduces
>> if's and control-flow branches that I would like to be able to
>> eliminate at compile time otherwise
>> - Type safety (?) is broken because now the users will be able to
>> perform 'unsupported' or 'ill-formed' HTTP requests
>
> I can't think of one off-hand, but I think there may be cases where it
> makes sense to let the calling code vary the verb dynamically.
> Essentially, something like:
>
>  client.request("POST", headers, body);
>
> So, #2 may not be all bad--after all it's all strings in the end :)
>

Unfortunately, this is not acceptable for me because:

1. The request object is nowhere to be found (http::request object),
which should encapsulate all this information (headers, body) already
which is of very little consequence to the client.
2. The creation of the request object is by design decoupled from the
HTTP client because operations on the request object (like
adding/inspecting headers, adding/inspecting the body, other message
processing routines (conversion, transformation, serialization, etc.)
) are of no consequence (on the contrary, are actually used by) the
client.
3. This means inside client::request(...) I'll have to check whether I
will ignore/use the body depending on the method string provided.
Contrast that to a fixed interface which actually at compile time only
puts the code that's needed to perform the request without having to
do checks of what to do at runtime.

> I like the strongly named methods as they are now, but a generic
> request may be useful as well, otherwise you might _introduce_
> control-flow branches for calling the right method in cases where the
> verb is not fixed.
>

It depends what you mean by 'useful'.

One of the reasons why the library is intended to be header-only is to
minimize a lot of the run-time decision making from even happening by
making decisions at compile-time. For instance if you look at the
implementation of the http::client, there is a skeleton implementation
where the determination of whether to read the body of the response is
made/optimized at compile time (comparing HEAD and GET requests).

The static mapping between the behavior of client::get and HTTP GET
requests is intentional because primarily:

1. I don't see a point in making it a run-time check when it can be
made clear at compile-time.
2. The semantics of usage are unclear when using 'dynamic' or runtime
values to do the decision making that can otherwise be done at compile
time in a clearer manner.

So the idea really here is, the if's and control flow that is
application specific should remain application specific. For instance,
you can craft the request object in a generic manner, and do an 'if'
to find out whether you'll be using client.put(...) or
client.post(...) given the same http::request instance. I find for
example this if to be better put in the user code to convey clear
semantics, rather than be in the library where I should concentrate on
the efficiency of the implementation on a per-method level. ;-)

> That said, I can't think of a good case at the moment... As such,
> maybe better to leave it until need arises, but I don't think it can
> be entirely discounted.
>

I personally find no merit in allowing for making the method a runtime
string (and having a "generic" client::perform(...) ) unless for
instance you want to layer on top of an HTTP client a WebDAV
implementation. In this sense, I would rather have the WebDAV client
implementation inherit from the HTTP client and re-use internal
methods rather than just merely use a generic "unsafe" interface
provided by the HTTP client.

If helping the developer avoid shooting themselves in the foot matters
(which I really think does, a lot) then keeping the static mapping and
semantics intact would be a good premium. I'd think it's more a
feature than a deficiency that the HTTP client allows for an
expressive interface to allow users to "say what they mean" instead of
leaving a wide open door to allow users to easily get things wrong.
One of the goals is to make it easier to use the library, and part of
that is to make 'getting it wrong' harder than getting it right. :)

> Well, 2c worth,

2c is a worth a lot from where I am. :-P

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[K. G. <kim...@gm...>]

Hello,

On Thu, Aug 28, 2008 at 03:47, Dean Michael Berris
<mik...@gm...> wrote:
>
> For #2:
> - It puts the method crafting a 'run-time' action, which introduces
> if's and control-flow branches that I would like to be able to
> eliminate at compile time otherwise
> - Type safety (?) is broken because now the users will be able to
> perform 'unsupported' or 'ill-formed' HTTP requests

I can't think of one off-hand, but I think there may be cases where it
makes sense to let the calling code vary the verb dynamically.
Essentially, something like:

  client.request("POST", headers, body);

So, #2 may not be all bad--after all it's all strings in the end :)

I like the strongly named methods as they are now, but a generic
request may be useful as well, otherwise you might _introduce_
control-flow branches for calling the right method in cases where the
verb is not fixed.

That said, I can't think of a good case at the moment... As such,
maybe better to leave it until need arises, but I don't think it can
be entirely discounted.

Well, 2c worth,
- Kim

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Dean M. B. <mik...@gm...>]

On Thu, Aug 28, 2008 at 1:01 AM, Divye Kapoor <div...@gm...> wrote:
>
>
> On Wed, Aug 27, 2008 at 6:14 PM, Glyn Matthews <gly...@gm...>
> wrote:
>>
>>     using namespace boost::network;
>>     http::request request("http://www.boost.org/");
>>     http::client client_;
>>     http::response response_;
>>     response_ = client_.get(request);
>>
>>
>> What exactly is the role of the client here?  I'm a little confused by the
>> syntax because it reads as though you're trying to make a get request to an
>> HTTP client.  Would it perhaps be better to rename http::client to
>> http::connection?
>
> Currently, I read the above code as:
> 1. frame a request for resource http://www.boost.org/
> 2. Create a client that handles http requests/responses
> 3. Get the resource specified by the request and provide a response.
>

This is the intended interpretation which I was going for. :-)

> However, I do admit that client.head(request) and client.put(request) don't
> seem to be too intuitive in that respect.
>

I had been thinking about that before I even started, and I was
thinking more to provide a compile-time and explicit mapping between
the interfaces and the semantics of the HTTP protocol. More
importantly, I was thinking of alternatives which were:

1. Encapsulating the 'method' into the http::request object.
2. Making the client accept the method as a string.
3. Using the "push" abstraction to convey the action.

I decided against all three because:

For #1:
- This puts undue burden to the request crafting aspect: it makes it
cumbersome to craft requests that you intend to re-use anyway (perform
a HEAD first, and if headers don't fit a certain criteria then do a
GET using the same resource)
- It complicates the http::request implementation by deviating too
much from the message framework/abstraction

For #2:
- It puts the method crafting a 'run-time' action, which introduces
if's and control-flow branches that I would like to be able to
eliminate at compile time otherwise
- Type safety (?) is broken because now the users will be able to
perform 'unsupported' or 'ill-formed' HTTP requests

For #3:
- It's not intuitive (client_ << get(r1, response1) << put(r2,
response2) << delete_(r3, response3))
- It's a lot harder to implement for very little gain syntactically

So given these, I opted to implement the client so that there is a
clear mapping at compile-time of the valid HTTP methods, and a clear
interface regarding the interaction between these methods and the
request objects.

I hope at least this can make it into the documentation or in a
rationale document somehow... ;-)

HTH!

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] Comments about the HTTP client interface

[Dean M. B. <mik...@gm...>]

Hi Glyn,

On Wed, Aug 27, 2008 at 8:44 PM, Glyn Matthews <gly...@gm...> wrote:
> Hello,
>
>
> I have some questions about the way HTTP requests are made using the current
> interface, specifically this (taken from the http_1_0_test.cpp):
>
>     using namespace boost::network;
>     http::request request("http://www.boost.org/");
>     http::client client_;
>     http::response response_;
>     response_ = client_.get(request);
>
>
> What exactly is the role of the client here?  I'm a little confused by the
> syntax because it reads as though you're trying to make a get request to an
> HTTP client.  Would it perhaps be better to rename http::client to
> http::connection?
>

I haven't given this much thought, but now that you mention it, the idea of

  client.get(...)

Is to instruct the client instance to perform a get(...) given a
request instance. So it's like "sending a get(...) message to the
client instance".

> Secondly, why is the domain passed as an argument to the http::request
> constructor?  To me, this doesn't feel like the right unit of encapsulation.
>

Right.

In other network libraries (like in Windows' MFC/ATL) there is a
separation between the 'transport' or 'connection' and the individual
resources or URI's that you access given certain transports.

(see more below)

> Does the following example code make more sense?
>
>     using namespace boost::network;
>     http::connection c("http://www.boost.org");
>     http::request request1("/users/news/version_1_36_0");
>     http::response response1 = c.get(request);
>     http::request request2("/cgi-bin/path");
>     request2 << body("post_body");
>     http::response response2 = c.post(request2);
>
> Two observations about this:
> The current implementation creates a new socket for each request that is
> made, whereas the above suggestion allows a socket to remain open for
> multiple requests.  Additionally, the URI processing is currently done
> inside the request constructor.  I think it would be better to re-factor
> this into a separate package.
>

The current implementation is an HTTP 1.0 client, which by design
creates a single connection for every request. This is the reason why
the request line puts HTTP/1.0 instead of HTTP/1.1; In HTTP 1.1, the
client can keep a connection open and make multiple requests using the
same connection (this is referred to as persistent connections, and
with this the possibility of request pipelining).

What I would be inclined to support is something of a request builder
implementation, which allows for example:

  http::request::builder builder("http", "www.boost.org");
  http::request r1 = builder.create("/");
  http::request r2 = builder.create("/some_uri?param1=value");
  http::request r3 = builder.create("/login.php",
"username=user&password=password");
  http::client client;
  http::response response = client.get(r1);

The idea here is that we should be able to encapsulate the
connection-mapping logic in the client between domains and actual
sockets. That means, inside the client, it may be able to hold on to
existing connections (in HTTP 1.1 clients) and then perform the
correct actions.

The aim of the interface is to convey that:
- the client knows how to handle the connections so you don't have to
worry about it
- the request object as far as you are concerned is the base unit of
data you can use to instruct the client what to do
- you can craft any http::request and use it with any instance of the
client without necessarily binding the client to a single
destination/connection

If it helps, you can think of an http::client object as an
encapsulated browser of sorts, which can perform well-crafted requests
on the users' behalf.

This breaks the existing conventions on how HTTP clients look like and
behave -- and existing library designs. I personally think this is a
good thing, because we're doing it from scratch anyway. ;)

Besides, it makes things easier for developers (I think) because it
allows a more consistent way of thinking about crafting HTTP requests
and asking a client to perform a request on their behalf -- as
compared to being encumbered with the notion that an http::connection
is bound to a certain destination and that a request object is
associated with a URI instead of being a fully-encapsulated HTTP
request.

> Any thoughts?

I think so. ;-) See above? :-D

HTH

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.