cpp-netlib-devel — Developers Mailing List

Re: [cpp-netlib-devel] Review of http::message

[Glyn M. <gly...@gm...>]

Hi Dean,

On 15/04/2008, Dean Michael C. Berris <dmb...@fr...> wrote:
>
>
> - I was suggesting earlier to have instead a tag type which
> differentiated an http::message from a basic_message, because precisely
> to avoid the glob traits facility that we see in message_traits (and
> like the standard iterator traits that glob values together). The
> approach would be similar to:
>
>         namespace http { namespace tags {
>                 struct message_tag { };
>         }; };
>
>         namespace http { namespace traits {
>                 template <typename Tag>
>                 struct delimiters;
>
>                 template <>
>                 struct delimiters<tags::message_tag> {
>                         static char const * const STRING_CRLF =
> "\x0D\x0A";
>                         ...
>                 };
>
>                 template <typename Tag>
>                 struct header_names;
>
>                 template <>
>                 struct header<tags::message_tag> {
>                         static char const * const HEADER_HOST = "Host";
>                 };
>         }; };
>
> In the above approach, we can then even support wide character strings,
> by using a different tag to support wide characters.
>
>         namespace http {
>                 namespace tags {
>                         struct message_tag_w { };
>                 };
>
>                 namespace traits {
>                         template <> struct
> delimiters<tags::message_tag_w> {
>                                 static wchar_t const * const CRLF =
> L"\x0D\x0A";
>                                 ...
>                         };
>                 };
>         };


Maybe I'm missing something here, but isn't it an error to initialize a
static non-integral type inside a struct definition?
Regards,
Glyn

[cpp-netlib-devel] Review of http::message

[Dean M. C. B. <dmb...@fr...>]

Hi Everyone,

I took some time out to look at the implementation of the http::message
-- and right off the bat I see a few issues.

- I don't see unit test(s) for the implementation(s). I know that this
code has been adapted from an already existing library, and it would be
nice to see it in action first even in a measy unit test. I'll try
working on some later just to verify the implementation of the
http::message. Mike, can you give me (or everyone for that matter) an
idea as to how we should be using the http::message in particular
independent of the client? I can write some unit tests to check if it
still complies with the basic_message<> interface. Please let me know if
you want me to do this.

- I don't particularly like 'parser_traits' because it gives me the idea
that it contains nested types instead of constants. I don't have any
better suggestion than 'parser_constants', but somehow we need to
resolve this naming issue one way or another.

- I was suggesting earlier to have instead a tag type which
differentiated an http::message from a basic_message, because precisely
to avoid the glob traits facility that we see in message_traits (and
like the standard iterator traits that glob values together). The
approach would be similar to:

	namespace http { namespace tags {
		struct message_tag { };
	}; };

	namespace http { namespace traits {
		template <typename Tag>
		struct delimiters;

		template <>
		struct delimiters<tags::message_tag> {
			static char const * const STRING_CRLF =
"\x0D\x0A";
			...
		};

		template <typename Tag>
		struct header_names;

		template <>
		struct header<tags::message_tag> {
			static char const * const HEADER_HOST = "Host";
		};
	}; };

In the above approach, we can then even support wide character strings,
by using a different tag to support wide characters.

	namespace http { 
		namespace tags {
			struct message_tag_w { };
		};

		namespace traits {
			template <> struct
delimiters<tags::message_tag_w> {
				static wchar_t const * const CRLF =
L"\x0D\x0A";
				...
			};
		};
	};

The technique is almost like tag dispatching for methods, only this time
it's tag dispatching for type traits.

I can help with this conversion if there are no objections.

Mike, do you see a problem with libpion if we do it the way above?

- If we go with the tag dispatching approach, we can then have a
specialization on basic_message<> instead that has structural details
unique to that specialization.

Please let me know what you think guys, it looks like I'll be going to
need this _real soon_ so I now have a personal stake at making sure this
goes through as planned. ;)

--
Dean Michael Berris
Software Engineer, Friendster, Inc.
<dmb...@fr...>
+639287291459

Re: [cpp-netlib-devel] HTTP Code Conversion (WAS RE: A fewquestions)

[Dean M. C. B. <dmb...@fr...>]

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Michael Dickey
> Sent: Monday, April 07, 2008 1:54 AM
> To: C++ Networking Library Developers Mailing List
> Subject: Re: [cpp-netlib-devel] HTTP Code Conversion (WAS RE: 
> A fewquestions)
> 
[snip]
> 
> I just added a new message.hpp file that includes a 
> http::message base class, which both http::request and 
> http::response now derive from.  Do you think just 
> "http::message" works best or might this be confusing with 
> "basic_message" (since in the code the "http" namespace is 
> not explicitly qualified)?
> 

I'll check it out and see how exactly it looks like before providing
comments. One thing that came to mind was that "http::message" can just
be "basic_message<http::message_tag>" -- which can be defined as an
explicit specialization of the basic_message type. I'll look at the
implementation and I might try this idea out some time.

> 
> I was doing some performance tests & comparisons recently for 
> other reasons, and I think I disproved my assumption that 
> unordered containers are faster.  If you're doing lots of 
> lookups and the objects have long lifetimes, then I believe 
> they would be faster than ordered containers.  However, with 
> short-lived containers that do not perform lots of lookups, 
> the plain old multimap is several times faster.  I believe 
> that this is due to the overhead required to create and 
> destroy the hash tables.
> 
> So, in retrospect, I now believe that sticking with map and 
> multimap is our best bet for http objects.
> 

Sounds good to me. :)

> 
> 	I have a comment though on the 'types' namespace -- 
> would you think this
> 	would be unneccessary? I don't see a problem with having an
> 	'http::headers' struct which contained all the 
> necessary strings.
> 	Example would be something like:
> 	
> 	namespace http {
> 	 template <typename Tag=tags::default_tag>
> 	   struct headers_impl {
> 	     static char const * const CONTENT_LENGTH = 
> "Content-length";
> 	     ...
> 	   };
> 	
> 	 typedef headers_impl<> headers;
> 	}
> 	
> 	If it wouldn't be too much of a pain to use unordered 
> containers inside
> 	the basic_message<http::message_tag> and maintain a 
> consistent interface
> 	that the basic_message<> exposes (and some more unique to the
> 	basic_message<http::message_tag>) then I'd say go ahead 
> with that
> 	approach instead.
> 
> 
> I've nuked the old types files & struct.  Still a little 
> uncertain on the best naming and location.  "headers" didn't 
> really feel right to me since it also includes other 
> container types, status codes and such.  For now, I moved the 
> static functions into the new http::message base class, and 
> created a new traits.hpp file that contains all the constants 
> and data types for http messages and the http parser in new 
> message_traits<> and parser_traits<> structs.  This seems 
> like the most logical structure for now, unless you have a 
> better suggestion?
> 

What we can do is organize the contants into appropriately named wrapper
types. We can even group them with the use of traits classes. Before
going further, I can give an example here:

  namespace http { namespace constants {
    template <typename Tag = message_tag>
      struct status {
        enum { OK = 200, ... };
      };

    template <typename Tag = message_tag>
      struct mime_type {
        static char * const text_plain = "text/plain";
        static char * const text_html = "text/html";
        ...
      };
  }; // namespace constants

  }; // namespace http

I don't particularly like 'constants' because it's long and too verbose,
but I hope this brings the point across more clearly. ;)

> 
> Next thing on my list will be reading the fusion docs & 
> starting to merge over code into the http::request & 
> http::response objects...
> 

Believe me, if you've ever tried doing any sort of Boost.MPL munging
around, Boost.Fusion brings the 'higher level MPL voodoo' closer to
runtime where (most of the time) it matters most. :) But don't take my
word for it, you should try Fusion out and see for yourself. :)

> Once I have a better grasp of things and get these core http 
> classes in place, I think that converting the rest of the 
> code should move fairly smoothly..
> 

Sounds good to me. :) I'll update my working copy and look at the
http::message -- I'll write more about what I think about it later. :)

--
Dean Michael Berris
Software Engineer, Friendster, Inc.
<dmb...@fr...>
+639287291459
 

Re: [cpp-netlib-devel] HTTP Code Conversion (WAS RE: A fewquestions)

[Michael D. <mi...@mi...>]

On Mar 24, 2008, at 8:53 PM, Dean Michael C. Berris wrote:
>

>> ...
>> This does raise the question though of whether or not an
>> http_message<> class would make sense?  i.e.
>>
>> http::message<> : public basic_message<>;   (or http_message<>?)
>> http::request<> : public http::message<>;
>> http::response<> : public http::message<>;
>>
>> Perhaps this makes more sense as we add more members to the http
>> message objects...?
>>
>
> The hierarchy depicted above makes logical sense to me -- unless  
> there's
> opposition to the inheritance hierarchy above, I say go for it.
>
> On the issue of 'http_message', it would be perfectly fine for
> http::message<> to contain structural information that's unique to  
> that
> type without it having its own base separate from the basic_message<>.
> So I personally don't see the need for a different message type that's
> not derived from basic_message<> because after all, the intention of  
> the
> basic_message<> is to provide a common base for all messages to be
> defined from.
> ...

I just added a new message.hpp file that includes a http::message base  
class, which both http::request and http::response now derive from.   
Do you think just "http::message" works best or might this be  
confusing with "basic_message" (since in the code the "http" namespace  
is not explicitly qualified)?

...
>> As for using an "unordered" container, that is my personal
>> preference
>> and what we are using in pion-net.  The challenge with this is with
>> compiler support; not everyone seems to support tr1 yet, but those
>> which do not support alternatives (i.e. stdext::hash_map in
>> gcc).  We
>> have a separate "PionHashMap.hpp" header in pion that uses #define's
>> to work-around the compiler differences.
>>
>> I left it as a "multimap" in my code for the sole purpose of
>> starting
>> out keeping everything as simple as possible.  I do like the idea of
>> hashing for these headers, though since this would (in theory) help
>> increase the performance of lookups (of course, in practical use it
>> might make no difference..)  If you prefer that route too, I can
>> convert over the #define stuff we have in PionHashMap, maybe
>> by adding
>> a "unordered_map.hpp" details file?
>>
>
> I like unordered too, but I usually have not found a compelling case  
> for
> something like that except in performance critical code. Though in  
> this
> case, it might be a good decision to make early in the game. ;)

I was doing some performance tests & comparisons recently for other  
reasons, and I think I disproved my assumption that unordered  
containers are faster.  If you're doing lots of lookups and the  
objects have long lifetimes, then I believe they would be faster than  
ordered containers.  However, with short-lived containers that do not  
perform lots of lookups, the plain old multimap is several times  
faster.  I believe that this is due to the overhead required to create  
and destroy the hash tables.

So, in retrospect, I now believe that sticking with map and multimap  
is our best bet for http objects.

> I have a comment though on the 'types' namespace -- would you think  
> this
> would be unneccessary? I don't see a problem with having an
> 'http::headers' struct which contained all the necessary strings.
> Example would be something like:
>
> namespace http {
>  template <typename Tag=tags::default_tag>
>    struct headers_impl {
>      static char const * const CONTENT_LENGTH = "Content-length";
>      ...
>    };
>
>  typedef headers_impl<> headers;
> }
>
> If it wouldn't be too much of a pain to use unordered containers  
> inside
> the basic_message<http::message_tag> and maintain a consistent  
> interface
> that the basic_message<> exposes (and some more unique to the
> basic_message<http::message_tag>) then I'd say go ahead with that
> approach instead.

I've nuked the old types files & struct.  Still a little uncertain on  
the best naming and location.  "headers" didn't really feel right to  
me since it also includes other container types, status codes and  
such.  For now, I moved the static functions into the new  
http::message base class, and created a new traits.hpp file that  
contains all the constants and data types for http messages and the  
http parser in new message_traits<> and parser_traits<> structs.  This  
seems like the most logical structure for now, unless you have a  
better suggestion?

>>>> ...
>> Interesting.. so we could leave it up to the user to define the
>> container type for these (i.e. multimap or unordered_multimap)?
>>
>> Sort of makes me wonder though if we'd be giving people too
>> much rope
>> (to hang themselves with)?  Versus making it a specific type, that's
>> protected and only allows accessors?
>>
>
> Not really the users -- more like people who want to extend the  
> library.
> Remember that we will be implementing 'query_params<>' and
> 'cookie_params<>' for the tags we define. If for some reason people
> would want to extend or modify the functionality of the request  
> objects,
> then it should be entirely possible in the most un-intrusive manner.

Got it =)

...
>>> Guess I need to read up on the fusion library =)
>>
>> Would this just be for container types or does it apply to simple
>> types (like ints) as well?
>>
>
> A fusion map works like this:
> ...
> So what the fusion map allows you to do is to map values to types, so
> that you can index the values using types instead of relying on some
> form of polymorphism at runtime. Of course, you could do it with  
> member
> variables but that takes away the capability to perform compile-time
> munging/processing on the encapsulated data in the fusion container  
> (in
> this case, a fusion map).

Next thing on my list will be reading the fusion docs & starting to  
merge over code into the http::request & http::response objects...

Once I have a better grasp of things and get these core http classes  
in place, I think that converting the rest of the code should move  
fairly smoothly..

Take care,
-Mike

Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE: Nearly time tostart merging?)

[Darren G. <dar...@gm...>]

On 26/03/2008, Dean Michael C. Berris <dmb...@fr...> wrote:
>
> > -----Original Message-----
> > From: cpp...@li...
> > [mailto:cpp...@li...] On
> > Behalf Of Darren Garvey
>
> > Sent: Wednesday, March 26, 2008 4:12 AM
> > To: C++ Networking Library Developers Mailing List
>
> > Subject: Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE:
> > Nearly time tostart merging?)
>
>
> I'm not really sure if copying strings is really too inefficient
> compared to using iterator pairs. Although this may be 'premature
> optimization', it would be good to have at some point I guess.


You're right, I've vaguely benchmarked some of the CGI code and the
countless copies it does at the moment don't seem to impact *too* much. This
is probably something that should come in as and when benchmarking does (ie.
not yet).

> [snip]
> >
> >
> > I wasn't too sure about the as<> member function myself
> > either, but several people requested it and the idea already
> > has precedence in boost libraries, although I forget which
> > ones right now. Perhaps it would be better to have an
> > 'as<>-less' network::basic_param<> and then a
> > cgi::basic_param<> which derives from the former and adds the
> > as<> member function..?
> >
> > What do you think? *
> >
>
>
> I'd first like to see a use case for the basic_param<> idea -- as far as
> I see it in my head, a parameter is just a string. Although a string can
> be referred to with just a pair of iterators marking the beginning and
> the 'past end' position of a string of characters, I may need some
> convincing for the creation of a specialized type for iterator ranges.


Ok, I'll get back to you on this. TBH, I'm not entirely convinced of the
need for this myself. It keeps cropping up though so next time I feel
motivated I'll try and make a case! :)

Wouldn't a Boost.Range work with boost::lexical_cast<...>(...) ?


Hmm, not sure. It doesn't look like it to me - although it probably should
support it, no?

[snip]
>
> > This sounds like a good idea. For a readable basic_message<>
> > - ie. with the only addition being member functions
> > prepare/commit/consume - how about storage<basic_message<>> ?
> > This would allow direct reading/writing to/from the message with Asio.
>
>
> I'm not very familiar with the idea of 'storage<basic_message<> >', can
> you elaborate on that?


Sure. As I'm sure you're already aware, the Asio read/write functions only
deal with "(Mutable|Const)BufferSequence"s. boost::asio::basic_streambuf<>
provides some functions to read/write into/from it, which is what the
'storage<basic_message<>>' could mimic.

eg.
// untested
template<typename T>
class readable<basic_message<T>> : public basic_message<T>
{
public:
  typedef implementation_defined mutable_buffers_type;

  // get a mutable buffer that can be read into directly.
  mutable_buffers_type prepare(std::size_t size);

  // after reading, you call commit, marking how much data you read.
  void commit(std::size_t size);
};

template<typename T>
class writeable<basic_message<T>> : public basic_message<T>
{
public:
  typedef implementation_defined const_buffers_type;

  // get a const buffer that can be written directly.
  const_buffers_type data() const;
}:

template<typename T>
class readable_writeable<basic_message<T>> // or storage<basic_message<T>>
or whatever...
  : public readable<basic_message<T>>
  , public writeable<basic_message<T>>
{
};

Having these as adapters seems far more flexible to embedding extra member
functions into basic_message<>, I quite like this idea. In fact, the
readable_writeable<> class would probably drop right into the current CGI
codebase!

> Yeah, this is possible, but do you not think the
> > basic_message<> class should be as minimal as possible? In
> > the HTTP case you could possibly use something like:
> >
> > template<>
> > class basic_message<http::message_tag>
> >   : basic_message<tags::default_> // this might be wrong {
> >   std::string _source;
> >   std::string _destination;
> > public:
> >   std::string& source() const { return _source; }
> >   std::string& destination() const { return _destination; }
> > };
> >
> > We could then build up a message-type 'stack' on top of this
> > for the HTTP-based protocols and other 'stacks' for any other ones.
> >
>
> But this approach would break things because:
>
> basic_message<http::message_tag> http_message;
> basic_message<tags::default_> & message(http_message);
> message.source() = "Hello, World!";
> assert(http_message.source() == message.source()); // will fail


Yes, unfortunately it would. However, that makes me wonder what exactly a
'default message' is, or should be. An RFC822 Internet message? An RFC2616
HTTP message? I would prefer the former - ie. just a _header and  _body.
That way both HTTP parts and CGI parts could share a common base class.
AFAIK, specialising the basic_message<> class would mean duplicating code!

Sorry, I'm still somewhat confused as to the need for _source/_destination.
I might just be missing something. :(

The suggestion is to do something like:
>
> template <>
> struct basic_message<cgi::message_tag> {
>   ...
>   // copy constructor
>   template <typename Tag>
>   basic_message(basic_message<Tag> const & other)
>     :
>     _source(other.source()),
>     _destination(other.destination()),
>     ...
>     { };
>   ...
> };


It may seem very nit-picky, but I'd much rather avoid having to put a
superfluous source/destination parameter in each request object (the idea
would be that cgi::request, fcgi::request, etc. would all contain or derive
from basic_message<cgi::message_tag>). It's not that there would be much
performance impact - although it might show on a FastCGI daemon handling
thousands of requests - it just doesn't seem necessary.

> It all sounds very interesting. I wish I had some time to
> > start on it now!
>
> Don't we all? ;)


Hehe. I was really tempted to reapply for the GSoC along these lines -
getting the two libraries closer together and putting the CGI one up for
review. Hmm, I probably should have...

<snip>

Even so, parameterizing the allocator is another
> way to go about embedded system support. This is not one of the main
> concerns for designing the C++ Netlib _yet_, though if there's enough
> user demand for it and enough interest in making your
> phone/watch/stereo/tv/switch/router have an HTTP cilent/server, then
> that should be doable as well. :D


Yeah, this probably would be premature optimisation for the library at this
point. Anyway, I don't see why light switches shouldn't have their own
embedded servers! ;o)

Cheers,
Darren

Re: [cpp-netlib-devel] Google Summer of Code

[Glyn M. <gly...@gm...>]

Welcome Derrick,

On 26/03/2008, Derrick Hathaway <d.h...@ag...> wrote:

> I am excited to get involved and am even considering the possibility
> of sharpening my teeth on the project by defining and implementing a
> simple market data service protocol using C++ Network Library as part
> of my capstone project. Is the library at a stage where this is
> possible?


I took a brief scan through the code of your project from the link you gave
on the boost-users list.  I don't see anything that's stopping you doing
this.  C++ Network Library (CNL) doesn't offer any kind authentication
support at present, though you'll find some discussions about it in the
archives.

Thanks for introducing yourself.
Regards, Glyn

[cpp-netlib-devel] Google Summer of Code

[Derrick H. <d.h...@ag...>]

Hello C++ Network Library Developers,
I just wanted to introduce myself. My name is Derrick Hathaway, I am a
senior at Utah State University majoring in Computer Science. I have
been programming in C++ for about 8 years academically and
professionally for the last 2. As a C++ programmer I rely heavily on
Boost and the STL. I have used several of the boost libraries
including Boost.Thread, IOStreams, Serialization, Smart Pointers, and
many more. I have also recently been using ASIO to develop my capstone
project [derrickhathaway.googlepages.com].

I am interested in contributing to the project and plan to submit a
proposal to do so as a part of Google's Summer of Code program. I
first spoke with Michael Dickey about contributing to the Pion Network
Library a week ago, and have been in communication with him since
then. I have read the design documents regarding C++ Network Library
and explored the code (a little). Any suggestions would be welcomed.

I am excited to get involved and am even considering the possibility
of sharpening my teeth on the project by defining and implementing a
simple market data service protocol using C++ Network Library as part
of my capstone project. Is the library at a stage where this is
possible?

Thanks in advance.

Derrick Hathaway
d.hathaway -- at -- aggiemail.usu.edu

Re: [cpp-netlib-devel] [gsoc2008] Draft proposal for cpp-netlib andpion-net integration

[Dean M. C. B. <dmb...@fr...>]

Hi Divye!

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Divye Kapoor
> Sent: Wednesday, March 26, 2008 2:03 AM
> To: bo...@li...
> Cc: C++ Networking Library Developers Mailing List
> Subject: [cpp-netlib-devel] [gsoc2008] Draft proposal for 
> cpp-netlib andpion-net integration
> 
> Hello developers,
>    I'm attaching below a draft proposal for merging the cpp-netlib and
> pion-net projects. Its still in the making stage and I would specially
> like your feedback about the timeline and the deliverables. I've
> deliberately kept the timeline quite loose to allow for unforseen
> delays but I really want to get an estimate of how much more can be
> squeezed into the project without making the it overambitious given
> the timeframe involved (I'm pretty sure that implementation of rfc1867
> would require less than 3 weeks but I would like more feedback on this
> point). Also, I would like to know if any of the Boost library authors
> would be interested in mentoring this proposal because without them,
> this proposal is a sunken ship ;-)

It would be good (better) if you post this as well to the Boost mailing
list. (Although you may have already done that, I haven't checked my
personal mail yet so forgive me if you've already done this). That
should let the existing Boost authors that aren't on this list to be
able to respond to the message as well.

> ----
> Synopsis

[snip very well written synopsis]

I like what you intend to do, and support you 100%. However, I am a bit
worried about doing the caching which isn't really integral in the
implementation of HTTP clients/servers. There is no guarantee that
caches should be honored or even maintained by clients/servers.

What would be nice is to support proxies and cookies. I'm not sure if
libpion already does proxies, but I know it does do cookies.

[snip timeline]

I cannot say much about the timeline, because I really won't be the one
tracking progress. I'll leave this to whoever would like to mentor you
for the GSoC to decide.

> 
> Deliverables
>    A feature complete HTTP 1.1 header only implementation 
> conforming to the
> message <==> rfc data structure mapping of the cpp-netlib 
> project resulting in
> an integration of the pion-net and cpp-netlib projects 
> released under the Boost
> Software License.

I like this a lot. :D

>    Documentation, Test and Use cases for the most common 
> scenarios will be
> provided.
>    The library will compile and run successfully on atleast 2 
> platforms
> (intended Win32 and linux) and with 2 compilers (gcc 4.2 and 
> MSVC 8) in
> accordance with Boost portability policies and will follow 
> the Boost naming
> conventions.
> 

Sounds like good stuff to me.

> 
> Benefits for Boost
>    A network library supporting HTTP(S) and the most common 
> use cases will be
> ready for review.
> 
> Learning Experiences in store
>    As a result of this project, I'll gain valuable experience 
> in coding to rfc
> specifications and gain valuable insight into the working of 
> the HTTP protocol.
> In addition, I'll get a hands-on experience in working with 
> Boost.Asio and
> achieve a deeper understanding of the Boost libraries.
> 

Definitely a compelling proposal if you're asking me. ;)

> About Me
>    I'm a 19 year old student from the small town of 
> Jamshedpur, India. I'm
> pursuing my Integrated Dual Degree course in Computer Science 
> and Engineering
> (Bachelors) with a Masters specialization in Information 
> Technology at the
> Indian Institute of Technology, Roorkee. I have a strong 
> algorithmic and coding
> background. I've been ranked 2nd at the International 
> Informatics Olympiad 2005
> and 7th All India at the National Cyber Olympiad 2006. I've 
> been coding in C++
> for the past 6 years and I have been using the STL for the 
> past 3 years. Over
> the past year, I've come to use the Boost.Spirit, Boost.Bind 
> and Boost.Lambda
> libraries and I have gained valuable experience from them. 
> Now, I wish to make
> my own contribution to Boost by helping to bring network 
> abstraction to the
> Boost library.
> 

Welcome to the project, and I do hope you'll still continue to help out
even if your proposal doesn't get accepted -- although it is hard to
reject your proposal given the amount of thought and effort you've
already put into it. Have a good day, and thank you very much!

Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE: Nearly time tostart merging?)

[Dean M. C. B. <dmb...@fr...>]

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Darren Garvey
> Sent: Wednesday, March 26, 2008 4:12 AM
> To: C++ Networking Library Developers Mailing List
> Subject: Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE: 
> Nearly time tostart merging?)
> 
> 
> 
> On 25/03/2008, Dean Michael C. Berris <dmb...@fr...> wrote:
> 
> 
> 	> -----Original Message-----
> 	> From: cpp...@li...
> 	> [mailto:cpp...@li...] On
> 	> Behalf Of Darren Garvey
> 	
> 
> 
> 	> Perhaps we can open discussion about where things should and
> 	> shouldn't align and how best to do that? It'll be summer
> 	> before I have any real time to spend on this, but I'm really
> 	> keen that we can make some progress on this.
> 	>
> 	
> 	Like Glyn, I'll try looking into the code. However, I 
> may very well be
> 	concentrating more in helping out Mike with the integration of
> 	pion-net's HTTP code into the library. It would be a 
> dream come true if
> 	somehow the HTTP message code we're developing will be 
> the same one the
> 	CGI library will be using! :)
> 
> 
> This is the code in (SVN_ROOT)/branches/http_integration/ ? 
> I've had a brief look at it a couple of times, it has great 
> potential. I think the first steps towards integration of the 
> CGI library is to start using the basic_message<> class, 
> which I think should be pretty straight-forward.
> 

Great! :)

> 
> 
> 	> P.S.
> 	>
> 	> Ideas:
> 	> * add a form_parser class (as part of the public API)
> 	>    - use Boost.Spirit
> 	>    - should read/parse multipart/form-data regardless 
> of request type.
> 	>    - should allow incremental reading/parsing of 
> individual form parts
> 	
> 	I like this. :)
> 	
> 	We can even do this using the TDD approach where we can 
> write the unit
> 	test(s) independent of any subsystem, and develop the 
> parser(s) to
> 	implement to make the unit tests pass. This way, the 
> compliance to
> 	parsing multipart/form-data can be developed independently.
> 
> 
> Exactly! All this requires is using Boost.Asio's 
> (A)Sync(Read|Write)Stream concepts, so shouldn't require too 
> much isolated code.
> 

Sounds like a plan to me. :)

> 
> 
> 	> * add a param class
> 	>    - holds a boost::iterator_range to the data
> 	>    - convertible to std::string subject to a macro condition
> 	> (aka. taint mode)
> 	>    - allows easy case-insensitive comparison (possibly using
> 	> Boost.String_algo? Or the netlib transform meta-functions?)
> 	>    - allows compile-time checking of its source (eg.
> 	> env/GET/POST) to protect against XSS vulnerabilities.
> 	>    - allows easy lexical casting (eg. param::as<>)
> 	>
> 	
> 	I don't really understand what you mean by this, but if 
> you mean 'param'
> 	in the query string parameter case, then this may be a 
> good idea. I
> 	don't see though if using Boost.Any might make more 
> sense for some
> 	people who want the dynamic-ness (if there's such a 
> thing) of the value.
> 
> 
> I was referring to a possible container for any/all parsed 
> request data. The idea is that data can be read directly into 
> a basic_message<> and when it's parsed, the name/values of 
> each 'request variable' is stored in a container that just 
> *points* to the data, without copying it.
> 
> The  `param` (or `basic_param<>`) class would be little more 
> than a cheap-to-copy drop-in replacement for std::string that 
> supplied minimal networking-related member functions (eg. a 
> case-insensitive operator==() and nested typedefs like ::source_type).
> 
> I just wrote a load about how it might work, but it might be 
> better if I threw together an implementation, rather than 
> rambling on about what it could do!
> 

Sounds like a good plan to me too. :)

I'm not really sure if copying strings is really too inefficient
compared to using iterator pairs. Although this may be 'premature
optimization', it would be good to have at some point I guess.

> [snip]
> 
> 
> I wasn't too sure about the as<> member function myself 
> either, but several people requested it and the idea already 
> has precedence in boost libraries, although I forget which 
> ones right now. Perhaps it would be better to have an 
> 'as<>-less' network::basic_param<> and then a 
> cgi::basic_param<> which derives from the former and adds the 
> as<> member function..?
> 
> What do you think? *
> 

I'd first like to see a use case for the basic_param<> idea -- as far as
I see it in my head, a parameter is just a string. Although a string can
be referred to with just a pair of iterators marking the beginning and
the 'past end' position of a string of characters, I may need some
convincing for the creation of a specialized type for iterator ranges.

Wouldn't a Boost.Range work with boost::lexical_cast<...>(...) ?

> 
> 
> 	> * add prepare/consume/commit member functions to
> 	> network::basic_message<>
> 	>   - functions like in asio::streambuf.
> 	>   - would allow reading data straight from a
> 	> (asio)(A)SyncReadStream (eg. a socket), without need 
> for buffering
> 	>   - could help with incremental parsing and/or making
> 	> form_parser class request-agnostic
> 	>
> 	
> 	My suggestion instead of overloading the interface of 
> basic_message<> is
> 	to define adaptors.
> 
> 
> Ahh, interesting...
> 
> 
> 
> 	It should be feasible to have a
> 	'streamed<basic_message<> >' or 'async<basic_message<> 
> >' instance which
> 	would decorate the functionality of the basic_message 
> encapsulated
> 	within the instance. That way it can still be converted 
> into and is
> 	still interface compatible to basic_message<>, but adds 
> functionality
> 	that's typically not inherent in the actual message.
> 
> 
> ... and supplies reusable components too. :)
> 
> 
> 
> 	streamed<basic_message<http::message_tag> > streamed_message;
> 	std::string body_line;
> 	getline(body(body_stream), body_line); // synchronously 
> wait for the
> 	first body line to be streamed
> 	
> 	async<basic_message<http::message_tag> > async_message;
> 	future<std::string> body_line = body(async_message);
> 	// at some point in time
> 	cout << body_line.get();
> 	
> 	This keeps the basic_message<> interface intact and the 
> usage semantics
> 	consistent. What do you think? :D
> 
> 
> This sounds like a good idea. For a readable basic_message<> 
> - ie. with the only addition being member functions 
> prepare/commit/consume - how about storage<basic_message<>> ? 
> This would allow direct reading/writing to/from the message with Asio.
> 

I'm not very familiar with the idea of 'storage<basic_message<> >', can
you elaborate on that?

> 
> 
> 	> P.P.S.
> 	>
> 	> The basic_message<> class has embedded _source/_destination
> 	> parameters. These aren't particularly relevant to
> 	> CGI/FastCGI/SCGI and they imply a close relation between the
> 	> payload of an HTTP message and its source/destination, which
> 	> might not always be a 1:1 relationship. I wonder if they
> 	> should be moved out of the basic_message<> class to some
> 	> higher-level abstraction?
> 	
> 	This may be a good idea, but it's possible to actually 
> use SFINAE or a
> 	cgi::message_tag to specialize on the basic_message<>'s internal
> 	representation while keeping it consistent. For 
> example, you can disable
> 	using SFINAE the interface to accessing (and presence of) the
> 	_source/_destination parameters.
> 
> 
> Yeah, this is possible, but do you not think the 
> basic_message<> class should be as minimal as possible? In 
> the HTTP case you could possibly use something like:
> 
> template<>
> class basic_message<http::message_tag>
>   : basic_message<tags::default_> // this might be wrong {
>   std::string _source;
>   std::string _destination;
> public:
>   std::string& source() const { return _source; }
>   std::string& destination() const { return _destination; } };
> 
> 
> We could then build up a message-type 'stack' on top of this 
> for the HTTP-based protocols and other 'stacks' for any other ones.
> 

But this approach would break things because:

basic_message<http::message_tag> http_message;
basic_message<tags::default_> & message(http_message);
message.source() = "Hello, World!";
assert(http_message.source() == message.source()); // will fail

The suggestion is to do something like:

template <>
struct basic_message<cgi::message_tag> {
  ...
  // copy constructor
  template <typename Tag>
  basic_message(basic_message<Tag> const & other)
    : 
    _source(other.source()), 
    _destination(other.destination()), 
    ...
    { };
  ...
};

>From within the specialization, you have the chance to do almost
whatever you want with the internal implementation and external
interface -- add more methods and what not -- while keeping
compatibility with the interface of the basic_message. :)

> 
> 
> 	What do you think Darren? I hope these make sense! :)
> 
> 
> It all sounds very interesting. I wish I had some time to 
> start on it now!
> 

Don't we all? ;)

> Cheers,
> Darren
> 
> [*] From early discussion for the CGI library, some people 
> talked about embedded system support. They seem to prefer 
> very tight controls on memory allocation, so avoiding copying 
> might make these use-cases easier to support...
> 
> 

Ah, that makes sense. Even so, parameterizing the allocator is another
way to go about embedded system support. This is not one of the main
concerns for designing the C++ Netlib _yet_, though if there's enough
user demand for it and enough interest in making your
phone/watch/stereo/tv/switch/router have an HTTP cilent/server, then
that should be doable as well. :D

--
Dean Michael Berris
Software Engineer, Friendster, Inc.
<dmb...@fr...>
+639287291459
 

Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE: Nearly time to start merging?)

[Darren G. <dar...@gm...>]

Hi again Dean,

On 25/03/2008, Dean Michael C. Berris <dmb...@fr...> wrote:
>
>
> > -----Original Message-----
> > From: cpp...@li...
> > [mailto:cpp...@li...] On
> > Behalf Of Darren Garvey
>

> Perhaps we can open discussion about where things should and
> > shouldn't align and how best to do that? It'll be summer
> > before I have any real time to spend on this, but I'm really
> > keen that we can make some progress on this.
> >
>
> Like Glyn, I'll try looking into the code. However, I may very well be
> concentrating more in helping out Mike with the integration of
> pion-net's HTTP code into the library. It would be a dream come true if
> somehow the HTTP message code we're developing will be the same one the
> CGI library will be using! :)


This is the code in (SVN_ROOT)/branches/http_integration/ ? I've had a brief
look at it a couple of times, it has great potential. I think the first
steps towards integration of the CGI library is to start using the
basic_message<> class, which I think should be pretty straight-forward.

> P.S.
> >
> > Ideas:
> > * add a form_parser class (as part of the public API)
> >    - use Boost.Spirit
> >    - should read/parse multipart/form-data regardless of request type.
> >    - should allow incremental reading/parsing of individual form parts
>
> I like this. :)
>
> We can even do this using the TDD approach where we can write the unit
> test(s) independent of any subsystem, and develop the parser(s) to
> implement to make the unit tests pass. This way, the compliance to
> parsing multipart/form-data can be developed independently.


Exactly! All this requires is using Boost.Asio's (A)Sync(Read|Write)Stream
concepts, so shouldn't require too much isolated code.

> * add a param class
> >    - holds a boost::iterator_range to the data
> >    - convertible to std::string subject to a macro condition
> > (aka. taint mode)
> >    - allows easy case-insensitive comparison (possibly using
> > Boost.String_algo? Or the netlib transform meta-functions?)
> >    - allows compile-time checking of its source (eg.
> > env/GET/POST) to protect against XSS vulnerabilities.
> >    - allows easy lexical casting (eg. param::as<>)
> >
>
> I don't really understand what you mean by this, but if you mean 'param'
> in the query string parameter case, then this may be a good idea. I
> don't see though if using Boost.Any might make more sense for some
> people who want the dynamic-ness (if there's such a thing) of the value.


I was referring to a possible container for any/all parsed request data. The
idea is that data can be read directly into a basic_message<> and when it's
parsed, the name/values of each 'request variable' is stored in a container
that just *points* to the data, without copying it.

The  `param` (or `basic_param<>`) class would be little more than a
cheap-to-copy drop-in replacement for std::string that supplied minimal
networking-related member functions (eg. a case-insensitive operator==() and
nested typedefs like ::source_type).

I just wrote a load about how it might work, but it might be better if I
threw together an implementation, rather than rambling on about what it
could do!

[snip]

I wasn't too sure about the as<> member function myself either, but several
people requested it and the idea already has precedence in boost libraries,
although I forget which ones right now. Perhaps it would be better to have
an 'as<>-less' network::basic_param<> and then a cgi::basic_param<> which
derives from the former and adds the as<> member function..?

What do you think? *

> * add prepare/consume/commit member functions to
> > network::basic_message<>
> >   - functions like in asio::streambuf.
> >   - would allow reading data straight from a
> > (asio)(A)SyncReadStream (eg. a socket), without need for buffering
> >   - could help with incremental parsing and/or making
> > form_parser class request-agnostic
> >
>
> My suggestion instead of overloading the interface of basic_message<> is
> to define adaptors.


Ahh, interesting...

It should be feasible to have a
> 'streamed<basic_message<> >' or 'async<basic_message<> >' instance which
> would decorate the functionality of the basic_message encapsulated
> within the instance. That way it can still be converted into and is
> still interface compatible to basic_message<>, but adds functionality
> that's typically not inherent in the actual message.


... and supplies reusable components too. :)

streamed<basic_message<http::message_tag> > streamed_message;
> std::string body_line;
> getline(body(body_stream), body_line); // synchronously wait for the
> first body line to be streamed
>
> async<basic_message<http::message_tag> > async_message;
> future<std::string> body_line = body(async_message);
> // at some point in time
> cout << body_line.get();
>
> This keeps the basic_message<> interface intact and the usage semantics
> consistent. What do you think? :D


This sounds like a good idea. For a readable basic_message<> - ie. with the
only addition being member functions prepare/commit/consume - how about
storage<basic_message<>> ? This would allow direct reading/writing to/from
the message with Asio.

> P.P.S.
> >
> > The basic_message<> class has embedded _source/_destination
> > parameters. These aren't particularly relevant to
> > CGI/FastCGI/SCGI and they imply a close relation between the
> > payload of an HTTP message and its source/destination, which
> > might not always be a 1:1 relationship. I wonder if they
> > should be moved out of the basic_message<> class to some
> > higher-level abstraction?
>
> This may be a good idea, but it's possible to actually use SFINAE or a
> cgi::message_tag to specialize on the basic_message<>'s internal
> representation while keeping it consistent. For example, you can disable
> using SFINAE the interface to accessing (and presence of) the
> _source/_destination parameters.


Yeah, this is possible, but do you not think the basic_message<> class
should be as minimal as possible? In the HTTP case you could possibly use
something like:

template<>
class basic_message<http::message_tag>
  : basic_message<tags::default_> // this might be wrong
{
  std::string _source;
  std::string _destination;
public:
  std::string& source() const { return _source; }
  std::string& destination() const { return _destination; }
};

We could then build up a message-type 'stack' on top of this for the
HTTP-based protocols and other 'stacks' for any other ones.

What do you think Darren? I hope these make sense! :)


It all sounds very interesting. I wish I had some time to start on it now!

Cheers,
Darren

[*] From early discussion for the CGI library, some people talked about
embedded system support. They seem to prefer very tight controls on memory
allocation, so avoiding copying might make these use-cases easier to
support...

[cpp-netlib-devel] [gsoc2008] Draft proposal for cpp-netlib and pion-net integration

[Divye K. <div...@gm...>]

Hello developers,
   I'm attaching below a draft proposal for merging the cpp-netlib and
pion-net projects. Its still in the making stage and I would specially
like your feedback about the timeline and the deliverables. I've
deliberately kept the timeline quite loose to allow for unforseen
delays but I really want to get an estimate of how much more can be
squeezed into the project without making the it overambitious given
the timeframe involved (I'm pretty sure that implementation of rfc1867
would require less than 3 weeks but I would like more feedback on this
point). Also, I would like to know if any of the Boost library authors
would be interested in mentoring this proposal because without them,
this proposal is a sunken ship ;-)
Yours sincerely,
Divye Kapoor
PS: Its not yet in the prescribed format for an application, but
there's an almost one to one mapping.

----
Synopsis
   During the course of the Google Summer of Code, I would like to work on the
cpp-netlib [1] and the pion-net [2] projects to ensure that pion-lib HTTP
client-server library is seamlessly integrated with the general messaging
framework specifed in the cpp-netlib architecture [3]. This will not only
address one of the most important use cases of the Boost Asio library as it will
provide a general, cross platform implementation of network communication via
the the HTTP protocol but it will also lower the entry barrier for creating
cross-platform, network aware C++ applications.

Project
   While working on this project, I will have a large codebase to build upon (~
15000 lines in the pion-net project). To ensure robustness of the code, my first
priority will be to create a test suite with Boost.Test and Python (using the
BaseHTTPServer module) to ensure standards conforming behaviour of the initial
code.
   After a testing suite is set up and any discovered bugs are removed, I would
proceed to refactor the pion-net interface of the library to conform to the
interface exposed by the cpp-netlib message classes. This would involve
non-trivial changes in the code base as it would require conversion of the
pion-HTTP* classes to conform to the transformation layer restrictions of the
cpp-netlib classes. This might cause breaking changes to the implementation. All
such changes will be documented and appropriate test cases generated.
   Support for parsing multipart-form data (both encoding and decoding) will
then have to be coded in accordance with rfc1867 [4]. Currently, the
implementation correctly handles only GET parameters and POST parameters encoded
using url-encoding. This creates a limitation of not being able to send and
receive files and other such binary chunks of data via a POST request over the
network. A correct implementation will enable multiple files potentially as
large as 2 GB to be sent and received correctly over the HTTP protocol.
Robustness of the code will be a serious challenge in this case as disk file
security and large file sizes have to be handled. Ensuring reliable operation in
the face of timeouts and incomplete transfer will form an important
consideration of this part of the project. This will take up a bulk of the time
in the project.
   The multipart/form-data handling capabilities will then be augmented by
support for Compression headers (via gzip compression). This will enable
efficient communication on bandwidth constrained networks and will be useful
even for simple HTTP requests as this compression will be handled in a
transparent manner.
   Complete support for cacheing will then have to be built in to the HTTP
request/response model. In particular, support for the Cache-control directives
and Expires directives will have to be built into the client to prevent
unnecessary bandwidth usage.
   Documentation and examples concerning the most common use cases will be
created and in case any time is left, a Boost Extension based plugin system will
be created to allow easy extension of the HTTP protocol stack to handle custom
headers.

Timeline
Prior to actual coding:
   April 14-May 26:
       Finish off examinations at college. Examinations till 13th May.
       Get familiarized with boost extension, Boost.Test, Boost.Asio, gzip,
pion-net and cpp-netlib projects

May 26 - July 7: 6 weeks mark
         July 14: Midterm evaluation
2 weeks - Create a testing suite, ensuring automation and ironing out any
unexpected bugs.
2 weeks - Break the old interface and make it conform with the message transform
API.
3 weeks - Implement the multipart/form-data encoding and decoding algorithms.
Generate test cases. Clean up code

July 14-August 11:
1 week  - Document all the code changes and the new API to which it conforms.
1 week  - Implement the compression header support via optional gzip support
2 weeks - Implement the Cacheing directives.
 < 1 week  - Clean up supporting docs, test cases, etc.

I believe that the timeline is flexible enough to handle any unforseen bugs and
eventualities that tend to crop up yet is tight enough to keep me busy.

Deliverables
   A feature complete HTTP 1.1 header only implementation conforming to the
message <==> rfc data structure mapping of the cpp-netlib project resulting in
an integration of the pion-net and cpp-netlib projects released under the Boost
Software License.
   Documentation, Test and Use cases for the most common scenarios will be
provided.
   The library will compile and run successfully on atleast 2 platforms
(intended Win32 and linux) and with 2 compilers (gcc 4.2 and MSVC 8) in
accordance with Boost portability policies and will follow the Boost naming
conventions.


Benefits for Boost
   A network library supporting HTTP(S) and the most common use cases will be
ready for review.

Learning Experiences in store
   As a result of this project, I'll gain valuable experience in coding to rfc
specifications and gain valuable insight into the working of the HTTP protocol.
In addition, I'll get a hands-on experience in working with Boost.Asio and
achieve a deeper understanding of the Boost libraries.

About Me
   I'm a 19 year old student from the small town of Jamshedpur, India. I'm
pursuing my Integrated Dual Degree course in Computer Science and Engineering
(Bachelors) with a Masters specialization in Information Technology at the
Indian Institute of Technology, Roorkee. I have a strong algorithmic and coding
background. I've been ranked 2nd at the International Informatics Olympiad 2005
and 7th All India at the National Cyber Olympiad 2006. I've been coding in C++
for the past 6 years and I have been using the STL for the past 3 years. Over
the past year, I've come to use the Boost.Spirit, Boost.Bind and Boost.Lambda
libraries and I have gained valuable experience from them. Now, I wish to make
my own contribution to Boost by helping to bring network abstraction to the
Boost library.


[1] http://sourceforge.net/projects/cpp-netlib/
[2] http://www.pion.org/projects/pion-network-library
[3] http://cpp-netlib.wiki.sourceforge.net/Architectural+Documentation
[4] http://www.ietf.org/rfc/rfc1867.txt


-- 
Whether the chicken crossed the road or the road moved beneath the
chicken depends upon your frame of reference.
My official web site: http://www.iitr.ernet.in/studenthomepages/homepages/061305

[cpp-netlib-devel] Boost.CGI Merger (WAS RE: Nearly time to start merging?)

[Dean M. C. B. <dmb...@fr...>]

Hi Darren!

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Darren Garvey
> Sent: Monday, March 24, 2008 8:33 PM
> To: C++ Networking Library Developers Mailing List
> Subject: [cpp-netlib-devel] Nearly time to start merging?
> 
> Hello all,
> 
> I've sent a couple of random messages to this list before 
> regarding the library under development and the CGI library I 
> started working on for the last GSoC. Because of the huge 
> potential for shared code between the two libraries I really 
> hope we can find some common ground between our efforts, 
> seems silly not to. :)
> 

It's shaping up to be a good year for C++ Network programming! :)

> The CGI library is now in a 
> "mostly-works-but-is-really-ugly-in-places" state - I'm 
> blaming lack of time/practice for that! There are a couple of 
> reasonably major additions to be made when time allows 
> (they're relevant, see the postscript).
> 
> Anyway, there are some cool ideas in the netlib code. I 
> especially like using fusion::map<>s for holding 
> request/response data and of course, the common use of a 
> basic_message<> class.
> 
> Perhaps we can open discussion about where things should and 
> shouldn't align and how best to do that? It'll be summer 
> before I have any real time to spend on this, but I'm really 
> keen that we can make some progress on this.
> 

Like Glyn, I'll try looking into the code. However, I may very well be
concentrating more in helping out Mike with the integration of
pion-net's HTTP code into the library. It would be a dream come true if
somehow the HTTP message code we're developing will be the same one the
CGI library will be using! :)

> Kind regards,
> Darren
> 
> P.S.
> 
> Ideas:
> * add a form_parser class (as part of the public API)
>    - use Boost.Spirit
>    - should read/parse multipart/form-data regardless of request type.
>    - should allow incremental reading/parsing of individual form parts

I like this. :)

We can even do this using the TDD approach where we can write the unit
test(s) independent of any subsystem, and develop the parser(s) to
implement to make the unit tests pass. This way, the compliance to
parsing multipart/form-data can be developed independently.

> * add a param class
>    - holds a boost::iterator_range to the data
>    - convertible to std::string subject to a macro condition 
> (aka. taint mode)
>    - allows easy case-insensitive comparison (possibly using 
> Boost.String_algo? Or the netlib transform meta-functions?)
>    - allows compile-time checking of its source (eg. 
> env/GET/POST) to protect against XSS vulnerabilities.
>    - allows easy lexical casting (eg. param::as<>)
> 

I don't really understand what you mean by this, but if you mean 'param'
in the query string parameter case, then this may be a good idea. I
don't see though if using Boost.Any might make more sense for some
people who want the dynamic-ness (if there's such a thing) of the value.

> * add prepare/consume/commit member functions to 
> network::basic_message<>
>   - functions like in asio::streambuf.
>   - would allow reading data straight from a 
> (asio)(A)SyncReadStream (eg. a socket), without need for buffering
>   - could help with incremental parsing and/or making 
> form_parser class request-agnostic
> 

My suggestion instead of overloading the interface of basic_message<> is
to define adaptors. It should be feasible to have a
'streamed<basic_message<> >' or 'async<basic_message<> >' instance which
would decorate the functionality of the basic_message encapsulated
within the instance. That way it can still be converted into and is
still interface compatible to basic_message<>, but adds functionality
that's typically not inherent in the actual message.

streamed<basic_message<http::message_tag> > streamed_message;
std::string body_line;
getline(body(body_stream), body_line); // synchronously wait for the
first body line to be streamed

async<basic_message<http::message_tag> > async_message;
future<std::string> body_line = body(async_message);
// at some point in time
cout << body_line.get();

This keeps the basic_message<> interface intact and the usage semantics
consistent. What do you think? :D

> P.P.S.
> 
> The basic_message<> class has embedded _source/_destination 
> parameters. These aren't particularly relevant to 
> CGI/FastCGI/SCGI and they imply a close relation between the 
> payload of an HTTP message and its source/destination, which 
> might not always be a 1:1 relationship. I wonder if they 
> should be moved out of the basic_message<> class to some 
> higher-level abstraction?
> 
> 

This may be a good idea, but it's possible to actually use SFINAE or a
cgi::message_tag to specialize on the basic_message<>'s internal
representation while keeping it consistent. For example, you can disable
using SFINAE the interface to accessing (and presence of) the
_source/_destination parameters.

What do you think Darren? I hope these make sense! :)

--
Dean Michael Berris
Software Engineer, Friendster, Inc.
<dmb...@fr...>
+639287291459
 

Re: [cpp-netlib-devel] HTTP Code Conversion (WAS RE: A fewquestions)

[Dean M. C. B. <dmb...@fr...>]

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Michael Dickey
> Sent: Tuesday, March 25, 2008 12:46 AM
> To: C++ Networking Library Developers Mailing List
> Subject: Re: [cpp-netlib-devel] HTTP Code Conversion (WAS RE: 
> A fewquestions)
> 
> On Mar 23, 2008, at 7:52 PM, Dean Michael C. Berris wrote:
> 
[snip]
> >
> > Good question. There's really no straight answer yet, but I 
> was in the
> > middle of making an http::request object contain a basic_message  
> > inside,
> > and somehow make it convertible to a basic_message instead 
> of deriving
> > it from a basic_message. The reasons are:
> >
[snip]
> 
> I would like to do this hierarchy conversion right away (in the  
> branch).  There are many common aspects for HTTP requests and  
> responses, i.e. they both have name/value headers, they both have  
> content bodies, and the parsing semantics are almost identical  
> (unfortunately "almost", because things like the response 
> status code  
> influence response parsing....).   A lot of the parser code is also  
> simplified by these two concepts sharing a common base.
> 

I agree with the parser getting simplified with requests and responses
sharing a common base. I personally am not too worried about the request
parsing because cpp-netlib wasn't initially supposed to implement an
HTTP server -- so that reason was reflected in the design of building
the request object separately only for consumption of the HTTP client.
The original intention was that it should be a 'packaged request' or an
object that merely contained information about what the client should
do.

In hindsight, it would have been more consistent with the HTTP spec and
messaging nature of the protocol to model the request as a
basic_message<> extension.

> This does raise the question though of whether or not an  
> http_message<> class would make sense?  i.e.
> 
> http::message<> : public basic_message<>;   (or http_message<>?)
> http::request<> : public http::message<>;
> http::response<> : public http::message<>;
> 
> Perhaps this makes more sense as we add more members to the http  
> message objects...?
> 

The hierarchy depicted above makes logical sense to me -- unless there's
opposition to the inheritance hierarchy above, I say go for it.

On the issue of 'http_message', it would be perfectly fine for
http::message<> to contain structural information that's unique to that
type without it having its own base separate from the basic_message<>.
So I personally don't see the need for a different message type that's
not derived from basic_message<> because after all, the intention of the
basic_message<> is to provide a common base for all messages to be
defined from.

On the surface this intention does not make a lot of sense (yet), but
consider the possibility of being able to convert from one message type
to another using conversion functions -- consider the capability to
transform an HTTP response instance into an SMTP message or a serialized
basic_message for caching and retrieval later on? Building on
basic_message<> allows us to extend functionality _from the base_ and
have all others that derive from it get that functionality for free --
even if it doesn't make sense directly. For example, an IRC message may
be convertible into an XMPP message, but it doesn't make sense for a
conversion to an HTTP message to make sense (in my limited imagination)
even if it were possible.

Note that convertibility should be explicitly defined using the
transformation function(s) in the transformation layer, which all
protocols should be implementing much like how the basic_message<>
metafunctions are built.

But I digress. ;)

> >> 2) I'm new to the art of making a "header only" library and am not
> >> familiar with the techniques and best practices required for this
> >> yet.  This became an issue for me in converting over the "types"
> >> code.  What is the best way to define static constant 
> variables that
> >> will never change (strings, numbers, etc.)?  How do I avoid
> >> "duplicate
> >> symbol" errors in my compiler?
> >
> > The best way is to do something like this:
> >
> > template <typename tag=tags::default_tag>
> > struct constants {
> >  static typename tag::int_type const A_CONST = 1;
> > };
> >
> > And in places where you'd need A_CONST, you'd do this instead:
> >
> > 	constants<>::A_CONST
> >
> > I hope this helps. ;)
> 
> Interesting... so does this work around the issue with 
> compilers that  
> don't normally let you define constant values in headers.  i.e.
> 
> static const unsigned int A_CONST = 25;
> 
> would work in some compilers, but not in others.  Does this "type- 
> trick" work around that problem?
> 

It should. Consider the 'const' after the type. The problem with statics
is by default they are externalized -- but the magic with statics inside
templates is that they can be const static and accessible from the
outside. But don't take my word for it, you can try it out and see if it
works. ;)

> 
> On a totally unrelated note, I noticed that you use "typename 
> tag" in  
> the templates in some places.. shouldn't this be "typename 
> Tag" (based  
> on the Boost template parameter naming guidelines?)
> 

Ah, yes. I think we should start converting template argument names to
conform to the Boost template parameter naming guidelines too -- but
that's like the lowest priority at the moment if you ask me. ;)

> >
> > 2. We can specialize on basic_message<http::message_tag> and define
> > different storage mechanisms for the headers. It can be as 
> simple as a
> > case-insensitive std::multi_map or as complex as an
> > unordered_case_insensitive_multi_map -- which is (fortunately)  
> > entirely
> > up to us. ;)
> 
> We chose choice #2 in pion-net for a couple reasons:
> 
> a) it's much simpler & easier than having to write specialized  
> accessor functions whenever you interface with the container (which  
> performs the string conversion)
> 

I agree. :)

> b) it's probably also much faster
> 

I don't think this is too much of a concern (yet), but definitely is a
good plus to have.

> c) it allows for constants to work properly (see the http::types  
> strings constants).  i.e.
> 
> namespace http { namespace types { namespace { headers {
> 	static const std::string 
> CONTENT_LENGTH("Content-Length");	//  
> definition uses "recommended" case
> };
> 
> std::cout << types::headers::CONTENT_LENGTH << ": " <<  
> request.header(types::headers::CONTENT_LENGTH) << types::CRLF;
> 
> As for using an "unordered" container, that is my personal 
> preference  
> and what we are using in pion-net.  The challenge with this is with  
> compiler support; not everyone seems to support tr1 yet, but those  
> which do not support alternatives (i.e. stdext::hash_map in 
> gcc).  We  
> have a separate "PionHashMap.hpp" header in pion that uses #define's  
> to work-around the compiler differences.
> 
> I left it as a "multimap" in my code for the sole purpose of 
> starting  
> out keeping everything as simple as possible.  I do like the idea of  
> hashing for these headers, though since this would (in theory) help  
> increase the performance of lookups (of course, in practical use it  
> might make no difference..)  If you prefer that route too, I can  
> convert over the #define stuff we have in PionHashMap, maybe 
> by adding  
> a "unordered_map.hpp" details file?
> 

I like unordered too, but I usually have not found a compelling case for
something like that except in performance critical code. Though in this
case, it might be a good decision to make early in the game. ;)

I have a comment though on the 'types' namespace -- would you think this
would be unneccessary? I don't see a problem with having an
'http::headers' struct which contained all the necessary strings.
Example would be something like:

namespace http {
  template <typename Tag=tags::default_tag>
    struct headers_impl {
      static char const * const CONTENT_LENGTH = "Content-length";
      ...
    };

  typedef headers_impl<> headers;
}

If it wouldn't be too much of a pain to use unordered containers inside
the basic_message<http::message_tag> and maintain a consistent interface
that the basic_message<> exposes (and some more unique to the
basic_message<http::message_tag>) then I'd say go ahead with that
approach instead.

> >> 4) I have a few other container types to add to the http message
> >> implementations, namely for query string and cookie parameters (see
> >> the "query_params" and "cookie_params" typedefs in 
> types.hpp).  What
> >> would be the best approach to incorporate these into http::request?
> >>
> >
> > Ah, good question.
> >
> > If you notice, http::request is a template -- you can hijack the tag
> > parameter and require these types to be passed to the tag, or use a
> > traits (meta)function (read: type) to determine the correct type for
> > these extra containers in an http::request. Example would 
> be something
> > like this (roughly from memory):
> >
> > namespace http {
> >  template <typename tag, ... >
> >  struct request {
> >    typename query_params<tag>::type query_params;
> >    typename cookie_params<tag>::type cookie_params;
> >    ...
> >  };
> > }
> 
> Interesting.. so we could leave it up to the user to define the  
> container type for these (i.e. multimap or unordered_multimap)?
> 
> Sort of makes me wonder though if we'd be giving people too 
> much rope  
> (to hang themselves with)?  Versus making it a specific type, that's  
> protected and only allows accessors?
> 

Not really the users -- more like people who want to extend the library.
Remember that we will be implementing 'query_params<>' and
'cookie_params<>' for the tags we define. If for some reason people
would want to extend or modify the functionality of the request objects,
then it should be entirely possible in the most un-intrusive manner.

> > That would be the simplest and most flexible way to do it. Another  
> > way,
> > is to make that part of the fusion sequence encapsulated 
> inside of an
> > http::request instead of being part of the actual request 
> object. This
> > is how we currently package the data within an http::request, which
> > would make it consistent and easily iterated upon in a template
> > metaprogramming level -- for example, we might want to do 
> compile-time
> > processing for some operations instead of doing it at runtime like
> > enabling normalizations for elements in a header, requirements  
> > checking,
> > etc.
> >> 5) In general, what is the best way to add additional member (like
> >> status code, for example) variables to the http::request and
> >> http::response classes?  It doesn't look like they were
> >> intended to be
> >> extended, but maybe that is b/c the implementation is just 
> missing...
> >>
> >
> > The best way would be to add it in the fusion map used inside the
> > http::request and http::response types. Since we're doing 
> this 'header
> > only' style, we best use the header only packaging that fusion  
> > allows us
> > to have. :)
> >
> > I can go into more detail if you need more information. :)
> 
> Guess I need to read up on the fusion library =)
> 
> Would this just be for container types or does it apply to simple  
> types (like ints) as well?
> 

A fusion map works like this:

struct tags {
  struct index1 { };
  struct index2 { };
  struct index3 { };
};

namespace fusion = boost::fusion;

typedef
  fusion::map<
    fusion::pair<tags::index1, int>,
    fusion::pair<tags::index2, std::string>,
    fusion::pair<tags::index3, std::vector<int> >
  >
  my_map;

{
  my_map instance;
  fusion::at_key<tags::index1>(instance) = 1;
  fusion::at_key<tags::index2>(instance) = "Hello, World!";
  fusion::at_key<tags::index3>(instance) = std::vector<int>(0, 100);
  
  cout << fusion::at_key<tags::index2>(instance); // "Hello, World!"
};

So what the fusion map allows you to do is to map values to types, so
that you can index the values using types instead of relying on some
form of polymorphism at runtime. Of course, you could do it with member
variables but that takes away the capability to perform compile-time
munging/processing on the encapsulated data in the fusion container (in
this case, a fusion map).

> > I hope this helps Mike!
> >
> 
> Absolutely, thanks!
> 

Glad to be of help. :)
 
--
Dean Michael Berris
Software Engineer, Friendster, Inc.
<dmb...@fr...>
+639287291459
 

Re: [cpp-netlib-devel] HTTP Code Conversion (WAS RE: A few questions)

[Michael D. <mi...@mi...>]

On Mar 23, 2008, at 7:52 PM, Dean Michael C. Berris wrote:

> ...
>> 1) Why does the http::request class not derive from
>> basic_message?  I
>> noticed that http::response, OTOH does
>>
>
> Good question. There's really no straight answer yet, but I was in the
> middle of making an http::request object contain a basic_message  
> inside,
> and somehow make it convertible to a basic_message instead of deriving
> it from a basic_message. The reasons are:
>
> 1. Constructing an http::request is different from constructing a
> basic_message -- if you notice, the constructors are heavy and perform
> parsing of the URI.
>
> 2. An http::request may contain metadata that does not fit within a
> basic_message -- that being said, converting an http::request into a
> basic_message should be possible, but constructing one from a
> basic_message may be impossible to do in a straight forward manner
> (consider how you'd add the metadata like method, http version, etc.).
>
> 3. The request object is more for the consumption of an http::client
> rather than as a message that should be streamed to/from endpoints.  
> That
> being said, I agree though that it should somehow look like or at one
> point "be" a basic_message one way or another.

I would like to do this hierarchy conversion right away (in the  
branch).  There are many common aspects for HTTP requests and  
responses, i.e. they both have name/value headers, they both have  
content bodies, and the parsing semantics are almost identical  
(unfortunately "almost", because things like the response status code  
influence response parsing....).   A lot of the parser code is also  
simplified by these two concepts sharing a common base.

This does raise the question though of whether or not an  
http_message<> class would make sense?  i.e.

http::message<> : public basic_message<>;   (or http_message<>?)
http::request<> : public http::message<>;
http::response<> : public http::message<>;

Perhaps this makes more sense as we add more members to the http  
message objects...?

>> 2) I'm new to the art of making a "header only" library and am not
>> familiar with the techniques and best practices required for this
>> yet.  This became an issue for me in converting over the "types"
>> code.  What is the best way to define static constant variables that
>> will never change (strings, numbers, etc.)?  How do I avoid
>> "duplicate
>> symbol" errors in my compiler?
>
> The best way is to do something like this:
>
> template <typename tag=tags::default_tag>
> struct constants {
>  static typename tag::int_type const A_CONST = 1;
> };
>
> And in places where you'd need A_CONST, you'd do this instead:
>
> 	constants<>::A_CONST
>
> I hope this helps. ;)

Interesting... so does this work around the issue with compilers that  
don't normally let you define constant values in headers.  i.e.

static const unsigned int A_CONST = 25;

would work in some compilers, but not in others.  Does this "type- 
trick" work around that problem?


On a totally unrelated note, I noticed that you use "typename tag" in  
the templates in some places.. shouldn't this be "typename Tag" (based  
on the Boost template parameter naming guidelines?)


>> 3) A bug we recently fixed in Pion was with handling of HTTP header
>> data.  I see that that message using a multimap of strings and
>> performs case-sensitive comparisons.  I believe that headers need to
>> be stored in a case-insensitive manner (please see my types::headers
>> typedef in the new types.hpp file).  I'm not so sure that
>> this is the
>> case for other protocols though.   Should I disregard the "headers"
>> container in message and just use a different one for the
>> http classes?
> There is a transformation layer that's currently being developed and
> tested -- one of the things I've done some time ago is to enable  
> string
> transformations to be performed relatively efficiently using message
> transformation functions. We have two choices here if we intend to  
> allow
> case insensitive storage in the HTTP processing:
>
> 1. We should normalize the headers before they're put into a
> basic_message -- I think the HTTP Spec says something about the
> recommended Camel-case or Camel-Case but I'm hazy on where exactly  
> that
> is.
>
> 2. We can specialize on basic_message<http::message_tag> and define
> different storage mechanisms for the headers. It can be as simple as a
> case-insensitive std::multi_map or as complex as an
> unordered_case_insensitive_multi_map -- which is (fortunately)  
> entirely
> up to us. ;)

We chose choice #2 in pion-net for a couple reasons:

a) it's much simpler & easier than having to write specialized  
accessor functions whenever you interface with the container (which  
performs the string conversion)

b) it's probably also much faster

c) it allows for constants to work properly (see the http::types  
strings constants).  i.e.

namespace http { namespace types { namespace { headers {
	static const std::string CONTENT_LENGTH("Content-Length");	//  
definition uses "recommended" case
};

std::cout << types::headers::CONTENT_LENGTH << ": " <<  
request.header(types::headers::CONTENT_LENGTH) << types::CRLF;

As for using an "unordered" container, that is my personal preference  
and what we are using in pion-net.  The challenge with this is with  
compiler support; not everyone seems to support tr1 yet, but those  
which do not support alternatives (i.e. stdext::hash_map in gcc).  We  
have a separate "PionHashMap.hpp" header in pion that uses #define's  
to work-around the compiler differences.

I left it as a "multimap" in my code for the sole purpose of starting  
out keeping everything as simple as possible.  I do like the idea of  
hashing for these headers, though since this would (in theory) help  
increase the performance of lookups (of course, in practical use it  
might make no difference..)  If you prefer that route too, I can  
convert over the #define stuff we have in PionHashMap, maybe by adding  
a "unordered_map.hpp" details file?

>> 4) I have a few other container types to add to the http message
>> implementations, namely for query string and cookie parameters (see
>> the "query_params" and "cookie_params" typedefs in types.hpp).  What
>> would be the best approach to incorporate these into http::request?
>>
>
> Ah, good question.
>
> If you notice, http::request is a template -- you can hijack the tag
> parameter and require these types to be passed to the tag, or use a
> traits (meta)function (read: type) to determine the correct type for
> these extra containers in an http::request. Example would be something
> like this (roughly from memory):
>
> namespace http {
>  template <typename tag, ... >
>  struct request {
>    typename query_params<tag>::type query_params;
>    typename cookie_params<tag>::type cookie_params;
>    ...
>  };
> }

Interesting.. so we could leave it up to the user to define the  
container type for these (i.e. multimap or unordered_multimap)?

Sort of makes me wonder though if we'd be giving people too much rope  
(to hang themselves with)?  Versus making it a specific type, that's  
protected and only allows accessors?

> That would be the simplest and most flexible way to do it. Another  
> way,
> is to make that part of the fusion sequence encapsulated inside of an
> http::request instead of being part of the actual request object. This
> is how we currently package the data within an http::request, which
> would make it consistent and easily iterated upon in a template
> metaprogramming level -- for example, we might want to do compile-time
> processing for some operations instead of doing it at runtime like
> enabling normalizations for elements in a header, requirements  
> checking,
> etc.
>> 5) In general, what is the best way to add additional member (like
>> status code, for example) variables to the http::request and
>> http::response classes?  It doesn't look like they were
>> intended to be
>> extended, but maybe that is b/c the implementation is just missing...
>>
>
> The best way would be to add it in the fusion map used inside the
> http::request and http::response types. Since we're doing this 'header
> only' style, we best use the header only packaging that fusion  
> allows us
> to have. :)
>
> I can go into more detail if you need more information. :)

Guess I need to read up on the fusion library =)

Would this just be for container types or does it apply to simple  
types (like ints) as well?

> I hope this helps Mike!
>

Absolutely, thanks!

-Mike

Re: [cpp-netlib-devel] Nearly time to start merging?

[Glyn M. <gly...@gm...>]

On 24/03/2008, Darren Garvey <dar...@gm...> wrote:
>
> Hey Glyn,
>
> On 24/03/2008, Glyn Matthews <gly...@gm...> wrote:
> >
> >
> > On 24/03/2008, Darren Garvey <dar...@gm...> wrote:
> > >
> > >
> > > Perhaps we can open discussion about where things should and shouldn't
> > > align and how best to do that? It'll be summer before I have any real time
> > > to spend on this, but I'm really keen that we can make some progress on
> > > this.
> >
> >
> > That'd be cool.  You're under no time pressure from us.  Where can I
> > find the code for the CGI library?
> >
>
> Oh, of course. Good point! The code is in the boost sandbox:
>
>  // browser-friendly
>
> http://svn.boost.org/trac/boost/browser/sandbox/SOC/2007/cgi/branches/release
>
>  // check-out
>  http://svn.boost.org/svn/boost/sandbox/2007/cgi/branches/release
>

That should be
http://svn.boost.org/svn/boost/sandbox/SOC/2007/cgi/branches/release ;)

Unfortunately the docs are part-way through a complete rewrite. The examples
> may be more helpful at this stage. I have them (and the half-baked test
> suite) compiling on debian linux with gcc 4.1.3 and on windows with VC++9.
> The FastCGI examples only work on linux, unless you wanted to set up your
> own native acceptor and assign() it to the fcgi::acceptor - which I wouldn't
> recommend.
>

Thanks, I'll take a look through them.
Regards,
Glyn

Re: [cpp-netlib-devel] Nearly time to start merging?

[Darren G. <dar...@gm...>]

Hey Glyn,

On 24/03/2008, Glyn Matthews <gly...@gm...> wrote:
>
>
> On 24/03/2008, Darren Garvey <dar...@gm...> wrote:
> >
> >
> > Perhaps we can open discussion about where things should and shouldn't
> > align and how best to do that? It'll be summer before I have any real time
> > to spend on this, but I'm really keen that we can make some progress on
> > this.
>
>
> That'd be cool.  You're under no time pressure from us.  Where can I find
> the code for the CGI library?
>

Oh, of course. Good point! The code is in the boost sandbox:

 // browser-friendly

http://svn.boost.org/trac/boost/browser/sandbox/SOC/2007/cgi/branches/release

 // check-out
 http://svn.boost.org/svn/boost/sandbox/2007/cgi/branches/release

Unfortunately the docs are part-way through a complete rewrite. The examples
may be more helpful at this stage. I have them (and the half-baked test
suite) compiling on debian linux with gcc 4.1.3 and on windows with VC++9.
The FastCGI examples only work on linux, unless you wanted to set up your
own native acceptor and assign() it to the fcgi::acceptor - which I wouldn't
recommend.

Regards,
Darren

Re: [cpp-netlib-devel] Nearly time to start merging?

[Glyn M. <gly...@gm...>]

Hi Darren,

On 24/03/2008, Darren Garvey <dar...@gm...> wrote:
>
> Hello all,
>
> I've sent a couple of random messages to this list before regarding the
> library under development and the CGI library I started working on for the
> last GSoC. Because of the huge potential for shared code between the two
> libraries I really hope we can find some common ground between our efforts,
> seems silly not to. :)


Indeed, part of the aim of this project is to provide an umbrella under
which we can have lots of different C++ network components, so I'm glad
you'd like to share your library.

Perhaps we can open discussion about where things should and shouldn't align
> and how best to do that? It'll be summer before I have any real time to
> spend on this, but I'm really keen that we can make some progress on this.


That'd be cool.  You're under no time pressure from us.  Where can I find
the code for the CGI library?

Regards,
Glyn

[cpp-netlib-devel] Nearly time to start merging?

[Darren G. <dar...@gm...>]

Hello all,

I've sent a couple of random messages to this list before regarding the
library under development and the CGI library I started working on for the
last GSoC. Because of the huge potential for shared code between the two
libraries I really hope we can find some common ground between our efforts,
seems silly not to. :)

The CGI library is now in a "mostly-works-but-is-really-ugly-in-places"
state - I'm blaming lack of time/practice for that! There are a couple of
reasonably major additions to be made when time allows (they're relevant,
see the postscript).

Anyway, there are some cool ideas in the netlib code. I especially like
using fusion::map<>s for holding request/response data and of course, the
common use of a basic_message<> class.

Perhaps we can open discussion about where things should and shouldn't align
and how best to do that? It'll be summer before I have any real time to
spend on this, but I'm really keen that we can make some progress on this.

Kind regards,
Darren

P.S.

Ideas:
* add a form_parser class (as part of the public API)
   - use Boost.Spirit
   - should read/parse multipart/form-data regardless of request type.
   - should allow incremental reading/parsing of individual form parts
* add a param class
   - holds a boost::iterator_range to the data
   - convertible to std::string subject to a macro condition (aka. taint
mode)
   - allows easy case-insensitive comparison (possibly using
Boost.String_algo? Or the netlib transform meta-functions?)
   - allows compile-time checking of its source (eg. env/GET/POST) to
protect against XSS vulnerabilities.
   - allows easy lexical casting (eg. param::as<>)

* add prepare/consume/commit member functions to network::basic_message<>
  - functions like in asio::streambuf.
  - would allow reading data straight from a (asio)(A)SyncReadStream (eg. a
socket), without need for buffering
  - could help with incremental parsing and/or making form_parser class
request-agnostic

P.P.S.

The basic_message<> class has embedded _source/_destination parameters.
These aren't particularly relevant to CGI/FastCGI/SCGI and they imply a
close relation between the payload of an HTTP message and its
source/destination, which might not always be a 1:1 relationship. I wonder
if they should be moved out of the basic_message<> class to some
higher-level abstraction?

[cpp-netlib-devel] HTTP Code Conversion (WAS RE: A few questions)

[Dean M. C. B. <dmb...@fr...>]

Hi Mike!

Sorry it took me a while, it's been quite a weekend for me here in the
Philippines. Please see my responses below in-lined.


> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Michael Dickey
> Sent: Sunday, March 23, 2008 9:24 AM
> To: C++ Networking Library Developers Mailing List
> Subject: [cpp-netlib-devel] A few questions
> 
> I just committed a few new files into the http_integration branch.   
> These include code merged over from pion's HTTPTypes and HTTPParser  
> classes.  These are not functional yet by any means (I haven't even  
> tried compiling them), but the process helped me come up with an  
> initial set of questions:
>

Let me check these out and help you with the integration. Right off the
bat, would you happen to have tests that come with these? We use
Boost.Test for the unit testing suite, it would be great if we (or
others with spare time) could come up with test suites for these types.
 
> 1) Why does the http::request class not derive from 
> basic_message?  I  
> noticed that http::response, OTOH does
> 

Good question. There's really no straight answer yet, but I was in the
middle of making an http::request object contain a basic_message inside,
and somehow make it convertible to a basic_message instead of deriving
it from a basic_message. The reasons are:

1. Constructing an http::request is different from constructing a
basic_message -- if you notice, the constructors are heavy and perform
parsing of the URI.

2. An http::request may contain metadata that does not fit within a
basic_message -- that being said, converting an http::request into a
basic_message should be possible, but constructing one from a
basic_message may be impossible to do in a straight forward manner
(consider how you'd add the metadata like method, http version, etc.).

3. The request object is more for the consumption of an http::client
rather than as a message that should be streamed to/from endpoints. That
being said, I agree though that it should somehow look like or at one
point "be" a basic_message one way or another.

> 2) I'm new to the art of making a "header only" library and am not  
> familiar with the techniques and best practices required for this  
> yet.  This became an issue for me in converting over the "types"  
> code.  What is the best way to define static constant variables that  
> will never change (strings, numbers, etc.)?  How do I avoid 
> "duplicate  
> symbol" errors in my compiler?
> 

The best way is to do something like this:

template <typename tag=tags::default_tag>
struct constants {
  static typename tag::int_type const A_CONST = 1;
};

And in places where you'd need A_CONST, you'd do this instead:

	constants<>::A_CONST

I hope this helps. ;)

> 3) A bug we recently fixed in Pion was with handling of HTTP header  
> data.  I see that that message using a multimap of strings and  
> performs case-sensitive comparisons.  I believe that headers need to  
> be stored in a case-insensitive manner (please see my types::headers  
> typedef in the new types.hpp file).  I'm not so sure that 
> this is the  
> case for other protocols though.   Should I disregard the "headers"  
> container in message and just use a different one for the 
> http classes?
> 

There is a transformation layer that's currently being developed and
tested -- one of the things I've done some time ago is to enable string
transformations to be performed relatively efficiently using message
transformation functions. We have two choices here if we intend to allow
case insensitive storage in the HTTP processing:

1. We should normalize the headers before they're put into a
basic_message -- I think the HTTP Spec says something about the
recommended Camel-case or Camel-Case but I'm hazy on where exactly that
is.

2. We can specialize on basic_message<http::message_tag> and define
different storage mechanisms for the headers. It can be as simple as a
case-insensitive std::multi_map or as complex as an
unordered_case_insensitive_multi_map -- which is (fortunately) entirely
up to us. ;)

> 4) I have a few other container types to add to the http message  
> implementations, namely for query string and cookie parameters (see  
> the "query_params" and "cookie_params" typedefs in types.hpp).  What  
> would be the best approach to incorporate these into http::request?
> 

Ah, good question.

If you notice, http::request is a template -- you can hijack the tag
parameter and require these types to be passed to the tag, or use a
traits (meta)function (read: type) to determine the correct type for
these extra containers in an http::request. Example would be something
like this (roughly from memory):

namespace http {
  template <typename tag, ... >
  struct request {
    typename query_params<tag>::type query_params;
    typename cookie_params<tag>::type cookie_params;
    ...
  };
}

That would be the simplest and most flexible way to do it. Another way,
is to make that part of the fusion sequence encapsulated inside of an
http::request instead of being part of the actual request object. This
is how we currently package the data within an http::request, which
would make it consistent and easily iterated upon in a template
metaprogramming level -- for example, we might want to do compile-time
processing for some operations instead of doing it at runtime like
enabling normalizations for elements in a header, requirements checking,
etc.

> 5) In general, what is the best way to add additional member (like  
> status code, for example) variables to the http::request and  
> http::response classes?  It doesn't look like they were 
> intended to be  
> extended, but maybe that is b/c the implementation is just missing...
> 

The best way would be to add it in the fusion map used inside the
http::request and http::response types. Since we're doing this 'header
only' style, we best use the header only packaging that fusion allows us
to have. :)

I can go into more detail if you need more information. :)

I hope this helps Mike!

--
Dean Michael Berris
Software Engineer, Friendster, Inc.
<dmb...@fr...>
+639287291459
 

[cpp-netlib-devel] A few questions

[Michael D. <mi...@mi...>]

I just committed a few new files into the http_integration branch.   
These include code merged over from pion's HTTPTypes and HTTPParser  
classes.  These are not functional yet by any means (I haven't even  
tried compiling them), but the process helped me come up with an  
initial set of questions:

1) Why does the http::request class not derive from basic_message?  I  
noticed that http::response, OTOH does

2) I'm new to the art of making a "header only" library and am not  
familiar with the techniques and best practices required for this  
yet.  This became an issue for me in converting over the "types"  
code.  What is the best way to define static constant variables that  
will never change (strings, numbers, etc.)?  How do I avoid "duplicate  
symbol" errors in my compiler?

3) A bug we recently fixed in Pion was with handling of HTTP header  
data.  I see that that message using a multimap of strings and  
performs case-sensitive comparisons.  I believe that headers need to  
be stored in a case-insensitive manner (please see my types::headers  
typedef in the new types.hpp file).  I'm not so sure that this is the  
case for other protocols though.   Should I disregard the "headers"  
container in message and just use a different one for the http classes?

4) I have a few other container types to add to the http message  
implementations, namely for query string and cookie parameters (see  
the "query_params" and "cookie_params" typedefs in types.hpp).  What  
would be the best approach to incorporate these into http::request?

5) In general, what is the best way to add additional member (like  
status code, for example) variables to the http::request and  
http::response classes?  It doesn't look like they were intended to be  
extended, but maybe that is b/c the implementation is just missing...

Thanks,
-Mike

Re: [cpp-netlib-devel] Sourceforge tracker

[Glyn M. <gly...@gm...>]

Mike,

That's pretty old, better start a new branch from trunk.  Maybe you can
still find something useful there all the same.

G

On 22/03/2008, Michael Dickey <mi...@mi...> wrote:
>
> Sorry, I see this branch now.   I must have had an old checkout without
> realizing it.  I'm going to start playing in there this weekend.-Mike
>
> On Mar 22, 2008, at 8:15 AM, Michael Dickey wrote:
>
> Is the branch for this called "http_tests"?  That is the only branch I see
> currently in the svn repo.-Mike
>
> On Dec 12, 2007, at 12:24 PM, Glyn Matthews wrote:
>
>
>
> On 12/12/2007, Dean Michael Berris <mik...@gm...> wrote:
> >
> >
> > I like this approach. If someone would care to start this without me
> > in the meantime (I'm not getting much free time now that some of the
> > projects I'm working on are reaching the deployment stages, I'm having
> > less and less time to spare for open source development).
>
>
>
> OK, the HTTPParser task has been added to the task list and a new branch
> has been created in SVN:
> http://tinyurl.com/2ntemj
>
> So if someone wants to get started from here, go right on ahead.
> G
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace_______________________________________________
> Cpp-netlib-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
>
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/_______________________________________________
> Cpp-netlib-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel
>
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Cpp-netlib-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel
>
>

[cpp-netlib-devel] cpp-netlib & pion-net

[Michael D. <mi...@mi...>]

I started writing the Pion Network Library (first called "libpion")  
almost a year ago to start learning Boost and to help fill a need I've  
repeatedly encountered over the past decade.  There are several good  
third-party networking libraries available for C++, but none that took  
advantage of Boost and the new ASIO library that will be included in  
version 1.35.  I have a lot of experience working with HTTP, and  
needed to develop (for the third time in my career) a lightweight HTTP  
server interface.

Although I planned to release libpion as open source, I never gave  
much thought to submitting it for review to be included in Boost.   
Over time, and with support and encouragement from the community, my  
desire to do this grew. I've also been keeping up with the cpp-netlib  
project on SourceForge, which shares the same goal.  For several  
months we've been kicking around the idea of merging our efforts.

Today, I've decided to commit to winding down the Pion Network Library  
as an independent project, and merging all the code (about 15,000  
lines) into the cpp-netlib project.  This is going to be a gradual  
conversion and refactoring of code, adopting it into the framework and  
styles that have been established by the cpp-netlib folks.  My goals  
will be to enhance cpp-netlib so that it is functionally equivalent to  
the current Pion Network Library (with asynchronous and synchronous  
HTTP 1.0 and 1.1 client and server-side support), and to help prepare  
cpp-netlib for inclusion in the next major Boost release.

I expect that the transition will take at least 3 to 6 months.   
Because there are several entities actively using the Pion Network  
Library (including my own company, Atomic Labs), I plan to continue  
developing and enhancing it as an independent project at least until  
the transition has been fully completed.

Dean, Glyn and I are excited about merging the two projects and  
developer communities, and I am happy to be part of what is now the  
definitive candidate for a Boost networking protocol library.

Take care,
-Mike

Re: [cpp-netlib-devel] Sourceforge tracker

[Michael D. <mi...@mi...>]

Sorry, I see this branch now.   I must have had an old checkout  
without realizing it.  I'm going to start playing in there this weekend.
-Mike

On Mar 22, 2008, at 8:15 AM, Michael Dickey wrote:

> Is the branch for this called "http_tests"?  That is the only branch  
> I see currently in the svn repo.
> -Mike
>
> On Dec 12, 2007, at 12:24 PM, Glyn Matthews wrote:
>
>>
>>
>> On 12/12/2007, Dean Michael Berris <mik...@gm...> wrote:
>>
>> I like this approach. If someone would care to start this without me
>> in the meantime (I'm not getting much free time now that some of the
>> projects I'm working on are reaching the deployment stages, I'm  
>> having
>> less and less time to spare for open source development).
>>
>>
>> OK, the HTTPParser task has been added to the task list and a new  
>> branch has been created in SVN:
>> http://tinyurl.com/2ntemj
>>
>> So if someone wants to get started from here, go right on ahead.
>> G
>>
>> -------------------------------------------------------------------------
>> SF.Net email is sponsored by:
>> Check out the new SourceForge.net Marketplace.
>> It's the best place to buy or sell services
>> for just about anything Open Source.
>> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace_______________________________________________
>> Cpp-netlib-devel mailing list
>> Cpp...@li...
>> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/_______________________________________________
> Cpp-netlib-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel

Re: [cpp-netlib-devel] Sourceforge tracker

[Michael D. <mi...@mi...>]

Is the branch for this called "http_tests"?  That is the only branch I  
see currently in the svn repo.
-Mike

On Dec 12, 2007, at 12:24 PM, Glyn Matthews wrote:

>
>
> On 12/12/2007, Dean Michael Berris <mik...@gm...> wrote:
>
> I like this approach. If someone would care to start this without me
> in the meantime (I'm not getting much free time now that some of the
> projects I'm working on are reaching the deployment stages, I'm having
> less and less time to spare for open source development).
>
>
> OK, the HTTPParser task has been added to the task list and a new  
> branch has been created in SVN:
> http://tinyurl.com/2ntemj
>
> So if someone wants to get started from here, go right on ahead.
> G
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by:
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services
> for just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace_______________________________________________
> Cpp-netlib-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel

Re: [cpp-netlib-devel] About the change of header include in protocol/http/impl/request.hpp

[Ricardo M. <rm...@wa...>]

On Thu, 2008-03-20 at 19:02 +0530, div...@gm... wrote:
> Changed the include. Hope that compiles now.

Yes it works!.

Ricardo Muñoz,

     1. 
> Divye
> 
> On 3/20/08, div...@gm... <div...@gm...> wrote:
> > I'm sorry. I just double checked. I was using an older revision of the
> > trunk and I made the last commit changing the header file in the
> > implementation. I guess changing the header include will fix it.
> > Thanks for pointing it out. Please change the include and commit.
> >
> > Thanks,
> > Divye Kapoor
> >
> >
> > On 3/20/08, Ricardo Muñoz <rm...@wa...> wrote:
> > > Hi! first of all i am sorry not to respond in the suitable thread.
> > >
> > > I am on the last trunk revision of boost and i think that the last
> > > change in 'protocol/http/impl/request.hpp' include was wrong:
> > >
> > > ricardo@miles:~/devel/boost-trunk/boost/fusion/container$ svn info
> > > map.hpp
> > >
> > > Path: map.hpp
> > > Name: map.hpp
> > > URL: http://svn.boost.org/svn/boost/trunk/boost/fusion/container/map.hpp
> > > Repository Root: http://svn.boost.org/svn/boost
> > > Repository UUID: b8fc166d-592f-0410-95f2-cb63ce0dd405
> > > Revision: 43733
> > > Node Kind: file
> > > Schedule: normal
> > > Last Changed Author: djowel
> > > Last Changed Rev: 40392
> > > Last Changed Date: 2007-10-24 04:36:29 +0200 (Wed, 24 Oct 2007)
> > > Text Last Updated: 2008-03-18 21:00:02 +0100 (Tue, 18 Mar 2008)
> > > Checksum: cd5020bc5f53af8a4e000c796160e048
> > >
> > > Seems that map.hpp is located in container directory not on
> > > sequence/container. If i revert the change, cpp-netlib compiles without
> > > problems.
> > >
> > > I am doing something wrong?
> > >
> > > Regards,
> > >
> > > Ricardo Muñoz
> > >
> > >
> > > -------------------------------------------------------------------------
> > > This SF.net email is sponsored by: Microsoft
> > > Defy all challenges. Microsoft(R) Visual Studio 2008.
> > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > > _______________________________________________
> > > Cpp-netlib-devel mailing list
> > > Cpp...@li...
> > > https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel
> > >
> >
> >
> > --
> > Whether the chicken crossed the road or the road moved beneath the
> > chicken depends upon your frame of reference.
> > My official web site:
> > http://www.iitr.ernet.in/studenthomepages/homepages/061305
> >
> 
>