cpp-netlib-devel

[cpp-netlib-devel] Nearly time to start merging?

[Darren G. <dar...@gm...>]

Hello all,

I've sent a couple of random messages to this list before regarding the
library under development and the CGI library I started working on for the
last GSoC. Because of the huge potential for shared code between the two
libraries I really hope we can find some common ground between our efforts,
seems silly not to. :)

The CGI library is now in a "mostly-works-but-is-really-ugly-in-places"
state - I'm blaming lack of time/practice for that! There are a couple of
reasonably major additions to be made when time allows (they're relevant,
see the postscript).

Anyway, there are some cool ideas in the netlib code. I especially like
using fusion::map<>s for holding request/response data and of course, the
common use of a basic_message<> class.

Perhaps we can open discussion about where things should and shouldn't align
and how best to do that? It'll be summer before I have any real time to
spend on this, but I'm really keen that we can make some progress on this.

Kind regards,
Darren

P.S.

Ideas:
* add a form_parser class (as part of the public API)
   - use Boost.Spirit
   - should read/parse multipart/form-data regardless of request type.
   - should allow incremental reading/parsing of individual form parts
* add a param class
   - holds a boost::iterator_range to the data
   - convertible to std::string subject to a macro condition (aka. taint
mode)
   - allows easy case-insensitive comparison (possibly using
Boost.String_algo? Or the netlib transform meta-functions?)
   - allows compile-time checking of its source (eg. env/GET/POST) to
protect against XSS vulnerabilities.
   - allows easy lexical casting (eg. param::as<>)

* add prepare/consume/commit member functions to network::basic_message<>
  - functions like in asio::streambuf.
  - would allow reading data straight from a (asio)(A)SyncReadStream (eg. a
socket), without need for buffering
  - could help with incremental parsing and/or making form_parser class
request-agnostic

P.P.S.

The basic_message<> class has embedded _source/_destination parameters.
These aren't particularly relevant to CGI/FastCGI/SCGI and they imply a
close relation between the payload of an HTTP message and its
source/destination, which might not always be a 1:1 relationship. I wonder
if they should be moved out of the basic_message<> class to some
higher-level abstraction?

Re: [cpp-netlib-devel] Nearly time to start merging?

[Glyn M. <gly...@gm...>]

Hi Darren,

On 24/03/2008, Darren Garvey <dar...@gm...> wrote:
>
> Hello all,
>
> I've sent a couple of random messages to this list before regarding the
> library under development and the CGI library I started working on for the
> last GSoC. Because of the huge potential for shared code between the two
> libraries I really hope we can find some common ground between our efforts,
> seems silly not to. :)


Indeed, part of the aim of this project is to provide an umbrella under
which we can have lots of different C++ network components, so I'm glad
you'd like to share your library.

Perhaps we can open discussion about where things should and shouldn't align
> and how best to do that? It'll be summer before I have any real time to
> spend on this, but I'm really keen that we can make some progress on this.


That'd be cool.  You're under no time pressure from us.  Where can I find
the code for the CGI library?

Regards,
Glyn

Re: [cpp-netlib-devel] Nearly time to start merging?

[Darren G. <dar...@gm...>]

Hey Glyn,

On 24/03/2008, Glyn Matthews <gly...@gm...> wrote:
>
>
> On 24/03/2008, Darren Garvey <dar...@gm...> wrote:
> >
> >
> > Perhaps we can open discussion about where things should and shouldn't
> > align and how best to do that? It'll be summer before I have any real time
> > to spend on this, but I'm really keen that we can make some progress on
> > this.
>
>
> That'd be cool.  You're under no time pressure from us.  Where can I find
> the code for the CGI library?
>

Oh, of course. Good point! The code is in the boost sandbox:

 // browser-friendly

http://svn.boost.org/trac/boost/browser/sandbox/SOC/2007/cgi/branches/release

 // check-out
 http://svn.boost.org/svn/boost/sandbox/2007/cgi/branches/release

Unfortunately the docs are part-way through a complete rewrite. The examples
may be more helpful at this stage. I have them (and the half-baked test
suite) compiling on debian linux with gcc 4.1.3 and on windows with VC++9.
The FastCGI examples only work on linux, unless you wanted to set up your
own native acceptor and assign() it to the fcgi::acceptor - which I wouldn't
recommend.

Regards,
Darren

Re: [cpp-netlib-devel] Nearly time to start merging?

[Glyn M. <gly...@gm...>]

On 24/03/2008, Darren Garvey <dar...@gm...> wrote:
>
> Hey Glyn,
>
> On 24/03/2008, Glyn Matthews <gly...@gm...> wrote:
> >
> >
> > On 24/03/2008, Darren Garvey <dar...@gm...> wrote:
> > >
> > >
> > > Perhaps we can open discussion about where things should and shouldn't
> > > align and how best to do that? It'll be summer before I have any real time
> > > to spend on this, but I'm really keen that we can make some progress on
> > > this.
> >
> >
> > That'd be cool.  You're under no time pressure from us.  Where can I
> > find the code for the CGI library?
> >
>
> Oh, of course. Good point! The code is in the boost sandbox:
>
>  // browser-friendly
>
> http://svn.boost.org/trac/boost/browser/sandbox/SOC/2007/cgi/branches/release
>
>  // check-out
>  http://svn.boost.org/svn/boost/sandbox/2007/cgi/branches/release
>

That should be
http://svn.boost.org/svn/boost/sandbox/SOC/2007/cgi/branches/release ;)

Unfortunately the docs are part-way through a complete rewrite. The examples
> may be more helpful at this stage. I have them (and the half-baked test
> suite) compiling on debian linux with gcc 4.1.3 and on windows with VC++9.
> The FastCGI examples only work on linux, unless you wanted to set up your
> own native acceptor and assign() it to the fcgi::acceptor - which I wouldn't
> recommend.
>

Thanks, I'll take a look through them.
Regards,
Glyn

Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE: Nearly time tostart merging?)

[Dean M. C. B. <dmb...@fr...>]

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Darren Garvey
> Sent: Wednesday, March 26, 2008 4:12 AM
> To: C++ Networking Library Developers Mailing List
> Subject: Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE: 
> Nearly time tostart merging?)
> 
> 
> 
> On 25/03/2008, Dean Michael C. Berris <dmb...@fr...> wrote:
> 
> 
> 	> -----Original Message-----
> 	> From: cpp...@li...
> 	> [mailto:cpp...@li...] On
> 	> Behalf Of Darren Garvey
> 	
> 
> 
> 	> Perhaps we can open discussion about where things should and
> 	> shouldn't align and how best to do that? It'll be summer
> 	> before I have any real time to spend on this, but I'm really
> 	> keen that we can make some progress on this.
> 	>
> 	
> 	Like Glyn, I'll try looking into the code. However, I 
> may very well be
> 	concentrating more in helping out Mike with the integration of
> 	pion-net's HTTP code into the library. It would be a 
> dream come true if
> 	somehow the HTTP message code we're developing will be 
> the same one the
> 	CGI library will be using! :)
> 
> 
> This is the code in (SVN_ROOT)/branches/http_integration/ ? 
> I've had a brief look at it a couple of times, it has great 
> potential. I think the first steps towards integration of the 
> CGI library is to start using the basic_message<> class, 
> which I think should be pretty straight-forward.
> 

Great! :)

> 
> 
> 	> P.S.
> 	>
> 	> Ideas:
> 	> * add a form_parser class (as part of the public API)
> 	>    - use Boost.Spirit
> 	>    - should read/parse multipart/form-data regardless 
> of request type.
> 	>    - should allow incremental reading/parsing of 
> individual form parts
> 	
> 	I like this. :)
> 	
> 	We can even do this using the TDD approach where we can 
> write the unit
> 	test(s) independent of any subsystem, and develop the 
> parser(s) to
> 	implement to make the unit tests pass. This way, the 
> compliance to
> 	parsing multipart/form-data can be developed independently.
> 
> 
> Exactly! All this requires is using Boost.Asio's 
> (A)Sync(Read|Write)Stream concepts, so shouldn't require too 
> much isolated code.
> 

Sounds like a plan to me. :)

> 
> 
> 	> * add a param class
> 	>    - holds a boost::iterator_range to the data
> 	>    - convertible to std::string subject to a macro condition
> 	> (aka. taint mode)
> 	>    - allows easy case-insensitive comparison (possibly using
> 	> Boost.String_algo? Or the netlib transform meta-functions?)
> 	>    - allows compile-time checking of its source (eg.
> 	> env/GET/POST) to protect against XSS vulnerabilities.
> 	>    - allows easy lexical casting (eg. param::as<>)
> 	>
> 	
> 	I don't really understand what you mean by this, but if 
> you mean 'param'
> 	in the query string parameter case, then this may be a 
> good idea. I
> 	don't see though if using Boost.Any might make more 
> sense for some
> 	people who want the dynamic-ness (if there's such a 
> thing) of the value.
> 
> 
> I was referring to a possible container for any/all parsed 
> request data. The idea is that data can be read directly into 
> a basic_message<> and when it's parsed, the name/values of 
> each 'request variable' is stored in a container that just 
> *points* to the data, without copying it.
> 
> The  `param` (or `basic_param<>`) class would be little more 
> than a cheap-to-copy drop-in replacement for std::string that 
> supplied minimal networking-related member functions (eg. a 
> case-insensitive operator==() and nested typedefs like ::source_type).
> 
> I just wrote a load about how it might work, but it might be 
> better if I threw together an implementation, rather than 
> rambling on about what it could do!
> 

Sounds like a good plan to me too. :)

I'm not really sure if copying strings is really too inefficient
compared to using iterator pairs. Although this may be 'premature
optimization', it would be good to have at some point I guess.

> [snip]
> 
> 
> I wasn't too sure about the as<> member function myself 
> either, but several people requested it and the idea already 
> has precedence in boost libraries, although I forget which 
> ones right now. Perhaps it would be better to have an 
> 'as<>-less' network::basic_param<> and then a 
> cgi::basic_param<> which derives from the former and adds the 
> as<> member function..?
> 
> What do you think? *
> 

I'd first like to see a use case for the basic_param<> idea -- as far as
I see it in my head, a parameter is just a string. Although a string can
be referred to with just a pair of iterators marking the beginning and
the 'past end' position of a string of characters, I may need some
convincing for the creation of a specialized type for iterator ranges.

Wouldn't a Boost.Range work with boost::lexical_cast<...>(...) ?

> 
> 
> 	> * add prepare/consume/commit member functions to
> 	> network::basic_message<>
> 	>   - functions like in asio::streambuf.
> 	>   - would allow reading data straight from a
> 	> (asio)(A)SyncReadStream (eg. a socket), without need 
> for buffering
> 	>   - could help with incremental parsing and/or making
> 	> form_parser class request-agnostic
> 	>
> 	
> 	My suggestion instead of overloading the interface of 
> basic_message<> is
> 	to define adaptors.
> 
> 
> Ahh, interesting...
> 
> 
> 
> 	It should be feasible to have a
> 	'streamed<basic_message<> >' or 'async<basic_message<> 
> >' instance which
> 	would decorate the functionality of the basic_message 
> encapsulated
> 	within the instance. That way it can still be converted 
> into and is
> 	still interface compatible to basic_message<>, but adds 
> functionality
> 	that's typically not inherent in the actual message.
> 
> 
> ... and supplies reusable components too. :)
> 
> 
> 
> 	streamed<basic_message<http::message_tag> > streamed_message;
> 	std::string body_line;
> 	getline(body(body_stream), body_line); // synchronously 
> wait for the
> 	first body line to be streamed
> 	
> 	async<basic_message<http::message_tag> > async_message;
> 	future<std::string> body_line = body(async_message);
> 	// at some point in time
> 	cout << body_line.get();
> 	
> 	This keeps the basic_message<> interface intact and the 
> usage semantics
> 	consistent. What do you think? :D
> 
> 
> This sounds like a good idea. For a readable basic_message<> 
> - ie. with the only addition being member functions 
> prepare/commit/consume - how about storage<basic_message<>> ? 
> This would allow direct reading/writing to/from the message with Asio.
> 

I'm not very familiar with the idea of 'storage<basic_message<> >', can
you elaborate on that?

> 
> 
> 	> P.P.S.
> 	>
> 	> The basic_message<> class has embedded _source/_destination
> 	> parameters. These aren't particularly relevant to
> 	> CGI/FastCGI/SCGI and they imply a close relation between the
> 	> payload of an HTTP message and its source/destination, which
> 	> might not always be a 1:1 relationship. I wonder if they
> 	> should be moved out of the basic_message<> class to some
> 	> higher-level abstraction?
> 	
> 	This may be a good idea, but it's possible to actually 
> use SFINAE or a
> 	cgi::message_tag to specialize on the basic_message<>'s internal
> 	representation while keeping it consistent. For 
> example, you can disable
> 	using SFINAE the interface to accessing (and presence of) the
> 	_source/_destination parameters.
> 
> 
> Yeah, this is possible, but do you not think the 
> basic_message<> class should be as minimal as possible? In 
> the HTTP case you could possibly use something like:
> 
> template<>
> class basic_message<http::message_tag>
>   : basic_message<tags::default_> // this might be wrong {
>   std::string _source;
>   std::string _destination;
> public:
>   std::string& source() const { return _source; }
>   std::string& destination() const { return _destination; } };
> 
> 
> We could then build up a message-type 'stack' on top of this 
> for the HTTP-based protocols and other 'stacks' for any other ones.
> 

But this approach would break things because:

basic_message<http::message_tag> http_message;
basic_message<tags::default_> & message(http_message);
message.source() = "Hello, World!";
assert(http_message.source() == message.source()); // will fail

The suggestion is to do something like:

template <>
struct basic_message<cgi::message_tag> {
  ...
  // copy constructor
  template <typename Tag>
  basic_message(basic_message<Tag> const & other)
    : 
    _source(other.source()), 
    _destination(other.destination()), 
    ...
    { };
  ...
};

>From within the specialization, you have the chance to do almost
whatever you want with the internal implementation and external
interface -- add more methods and what not -- while keeping
compatibility with the interface of the basic_message. :)

> 
> 
> 	What do you think Darren? I hope these make sense! :)
> 
> 
> It all sounds very interesting. I wish I had some time to 
> start on it now!
> 

Don't we all? ;)

> Cheers,
> Darren
> 
> [*] From early discussion for the CGI library, some people 
> talked about embedded system support. They seem to prefer 
> very tight controls on memory allocation, so avoiding copying 
> might make these use-cases easier to support...
> 
> 

Ah, that makes sense. Even so, parameterizing the allocator is another
way to go about embedded system support. This is not one of the main
concerns for designing the C++ Netlib _yet_, though if there's enough
user demand for it and enough interest in making your
phone/watch/stereo/tv/switch/router have an HTTP cilent/server, then
that should be doable as well. :D

--
Dean Michael Berris
Software Engineer, Friendster, Inc.
<dmb...@fr...>
+639287291459
 

Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE: Nearly time tostart merging?)

[Darren G. <dar...@gm...>]

On 26/03/2008, Dean Michael C. Berris <dmb...@fr...> wrote:
>
> > -----Original Message-----
> > From: cpp...@li...
> > [mailto:cpp...@li...] On
> > Behalf Of Darren Garvey
>
> > Sent: Wednesday, March 26, 2008 4:12 AM
> > To: C++ Networking Library Developers Mailing List
>
> > Subject: Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE:
> > Nearly time tostart merging?)
>
>
> I'm not really sure if copying strings is really too inefficient
> compared to using iterator pairs. Although this may be 'premature
> optimization', it would be good to have at some point I guess.


You're right, I've vaguely benchmarked some of the CGI code and the
countless copies it does at the moment don't seem to impact *too* much. This
is probably something that should come in as and when benchmarking does (ie.
not yet).

> [snip]
> >
> >
> > I wasn't too sure about the as<> member function myself
> > either, but several people requested it and the idea already
> > has precedence in boost libraries, although I forget which
> > ones right now. Perhaps it would be better to have an
> > 'as<>-less' network::basic_param<> and then a
> > cgi::basic_param<> which derives from the former and adds the
> > as<> member function..?
> >
> > What do you think? *
> >
>
>
> I'd first like to see a use case for the basic_param<> idea -- as far as
> I see it in my head, a parameter is just a string. Although a string can
> be referred to with just a pair of iterators marking the beginning and
> the 'past end' position of a string of characters, I may need some
> convincing for the creation of a specialized type for iterator ranges.


Ok, I'll get back to you on this. TBH, I'm not entirely convinced of the
need for this myself. It keeps cropping up though so next time I feel
motivated I'll try and make a case! :)

Wouldn't a Boost.Range work with boost::lexical_cast<...>(...) ?


Hmm, not sure. It doesn't look like it to me - although it probably should
support it, no?

[snip]
>
> > This sounds like a good idea. For a readable basic_message<>
> > - ie. with the only addition being member functions
> > prepare/commit/consume - how about storage<basic_message<>> ?
> > This would allow direct reading/writing to/from the message with Asio.
>
>
> I'm not very familiar with the idea of 'storage<basic_message<> >', can
> you elaborate on that?


Sure. As I'm sure you're already aware, the Asio read/write functions only
deal with "(Mutable|Const)BufferSequence"s. boost::asio::basic_streambuf<>
provides some functions to read/write into/from it, which is what the
'storage<basic_message<>>' could mimic.

eg.
// untested
template<typename T>
class readable<basic_message<T>> : public basic_message<T>
{
public:
  typedef implementation_defined mutable_buffers_type;

  // get a mutable buffer that can be read into directly.
  mutable_buffers_type prepare(std::size_t size);

  // after reading, you call commit, marking how much data you read.
  void commit(std::size_t size);
};

template<typename T>
class writeable<basic_message<T>> : public basic_message<T>
{
public:
  typedef implementation_defined const_buffers_type;

  // get a const buffer that can be written directly.
  const_buffers_type data() const;
}:

template<typename T>
class readable_writeable<basic_message<T>> // or storage<basic_message<T>>
or whatever...
  : public readable<basic_message<T>>
  , public writeable<basic_message<T>>
{
};

Having these as adapters seems far more flexible to embedding extra member
functions into basic_message<>, I quite like this idea. In fact, the
readable_writeable<> class would probably drop right into the current CGI
codebase!

> Yeah, this is possible, but do you not think the
> > basic_message<> class should be as minimal as possible? In
> > the HTTP case you could possibly use something like:
> >
> > template<>
> > class basic_message<http::message_tag>
> >   : basic_message<tags::default_> // this might be wrong {
> >   std::string _source;
> >   std::string _destination;
> > public:
> >   std::string& source() const { return _source; }
> >   std::string& destination() const { return _destination; }
> > };
> >
> > We could then build up a message-type 'stack' on top of this
> > for the HTTP-based protocols and other 'stacks' for any other ones.
> >
>
> But this approach would break things because:
>
> basic_message<http::message_tag> http_message;
> basic_message<tags::default_> & message(http_message);
> message.source() = "Hello, World!";
> assert(http_message.source() == message.source()); // will fail


Yes, unfortunately it would. However, that makes me wonder what exactly a
'default message' is, or should be. An RFC822 Internet message? An RFC2616
HTTP message? I would prefer the former - ie. just a _header and  _body.
That way both HTTP parts and CGI parts could share a common base class.
AFAIK, specialising the basic_message<> class would mean duplicating code!

Sorry, I'm still somewhat confused as to the need for _source/_destination.
I might just be missing something. :(

The suggestion is to do something like:
>
> template <>
> struct basic_message<cgi::message_tag> {
>   ...
>   // copy constructor
>   template <typename Tag>
>   basic_message(basic_message<Tag> const & other)
>     :
>     _source(other.source()),
>     _destination(other.destination()),
>     ...
>     { };
>   ...
> };


It may seem very nit-picky, but I'd much rather avoid having to put a
superfluous source/destination parameter in each request object (the idea
would be that cgi::request, fcgi::request, etc. would all contain or derive
from basic_message<cgi::message_tag>). It's not that there would be much
performance impact - although it might show on a FastCGI daemon handling
thousands of requests - it just doesn't seem necessary.

> It all sounds very interesting. I wish I had some time to
> > start on it now!
>
> Don't we all? ;)


Hehe. I was really tempted to reapply for the GSoC along these lines -
getting the two libraries closer together and putting the CGI one up for
review. Hmm, I probably should have...

<snip>

Even so, parameterizing the allocator is another
> way to go about embedded system support. This is not one of the main
> concerns for designing the C++ Netlib _yet_, though if there's enough
> user demand for it and enough interest in making your
> phone/watch/stereo/tv/switch/router have an HTTP cilent/server, then
> that should be doable as well. :D


Yeah, this probably would be premature optimisation for the library at this
point. Anyway, I don't see why light switches shouldn't have their own
embedded servers! ;o)

Cheers,
Darren

[cpp-netlib-devel] Boost.CGI Merger (WAS RE: Nearly time to start merging?)

[Dean M. C. B. <dmb...@fr...>]

Hi Darren!

> -----Original Message-----
> From: cpp...@li... 
> [mailto:cpp...@li...] On 
> Behalf Of Darren Garvey
> Sent: Monday, March 24, 2008 8:33 PM
> To: C++ Networking Library Developers Mailing List
> Subject: [cpp-netlib-devel] Nearly time to start merging?
> 
> Hello all,
> 
> I've sent a couple of random messages to this list before 
> regarding the library under development and the CGI library I 
> started working on for the last GSoC. Because of the huge 
> potential for shared code between the two libraries I really 
> hope we can find some common ground between our efforts, 
> seems silly not to. :)
> 

It's shaping up to be a good year for C++ Network programming! :)

> The CGI library is now in a 
> "mostly-works-but-is-really-ugly-in-places" state - I'm 
> blaming lack of time/practice for that! There are a couple of 
> reasonably major additions to be made when time allows 
> (they're relevant, see the postscript).
> 
> Anyway, there are some cool ideas in the netlib code. I 
> especially like using fusion::map<>s for holding 
> request/response data and of course, the common use of a 
> basic_message<> class.
> 
> Perhaps we can open discussion about where things should and 
> shouldn't align and how best to do that? It'll be summer 
> before I have any real time to spend on this, but I'm really 
> keen that we can make some progress on this.
> 

Like Glyn, I'll try looking into the code. However, I may very well be
concentrating more in helping out Mike with the integration of
pion-net's HTTP code into the library. It would be a dream come true if
somehow the HTTP message code we're developing will be the same one the
CGI library will be using! :)

> Kind regards,
> Darren
> 
> P.S.
> 
> Ideas:
> * add a form_parser class (as part of the public API)
>    - use Boost.Spirit
>    - should read/parse multipart/form-data regardless of request type.
>    - should allow incremental reading/parsing of individual form parts

I like this. :)

We can even do this using the TDD approach where we can write the unit
test(s) independent of any subsystem, and develop the parser(s) to
implement to make the unit tests pass. This way, the compliance to
parsing multipart/form-data can be developed independently.

> * add a param class
>    - holds a boost::iterator_range to the data
>    - convertible to std::string subject to a macro condition 
> (aka. taint mode)
>    - allows easy case-insensitive comparison (possibly using 
> Boost.String_algo? Or the netlib transform meta-functions?)
>    - allows compile-time checking of its source (eg. 
> env/GET/POST) to protect against XSS vulnerabilities.
>    - allows easy lexical casting (eg. param::as<>)
> 

I don't really understand what you mean by this, but if you mean 'param'
in the query string parameter case, then this may be a good idea. I
don't see though if using Boost.Any might make more sense for some
people who want the dynamic-ness (if there's such a thing) of the value.

> * add prepare/consume/commit member functions to 
> network::basic_message<>
>   - functions like in asio::streambuf.
>   - would allow reading data straight from a 
> (asio)(A)SyncReadStream (eg. a socket), without need for buffering
>   - could help with incremental parsing and/or making 
> form_parser class request-agnostic
> 

My suggestion instead of overloading the interface of basic_message<> is
to define adaptors. It should be feasible to have a
'streamed<basic_message<> >' or 'async<basic_message<> >' instance which
would decorate the functionality of the basic_message encapsulated
within the instance. That way it can still be converted into and is
still interface compatible to basic_message<>, but adds functionality
that's typically not inherent in the actual message.

streamed<basic_message<http::message_tag> > streamed_message;
std::string body_line;
getline(body(body_stream), body_line); // synchronously wait for the
first body line to be streamed

async<basic_message<http::message_tag> > async_message;
future<std::string> body_line = body(async_message);
// at some point in time
cout << body_line.get();

This keeps the basic_message<> interface intact and the usage semantics
consistent. What do you think? :D

> P.P.S.
> 
> The basic_message<> class has embedded _source/_destination 
> parameters. These aren't particularly relevant to 
> CGI/FastCGI/SCGI and they imply a close relation between the 
> payload of an HTTP message and its source/destination, which 
> might not always be a 1:1 relationship. I wonder if they 
> should be moved out of the basic_message<> class to some 
> higher-level abstraction?
> 
> 

This may be a good idea, but it's possible to actually use SFINAE or a
cgi::message_tag to specialize on the basic_message<>'s internal
representation while keeping it consistent. For example, you can disable
using SFINAE the interface to accessing (and presence of) the
_source/_destination parameters.

What do you think Darren? I hope these make sense! :)

--
Dean Michael Berris
Software Engineer, Friendster, Inc.
<dmb...@fr...>
+639287291459
 

Re: [cpp-netlib-devel] Boost.CGI Merger (WAS RE: Nearly time to start merging?)

[Darren G. <dar...@gm...>]

Hi again Dean,

On 25/03/2008, Dean Michael C. Berris <dmb...@fr...> wrote:
>
>
> > -----Original Message-----
> > From: cpp...@li...
> > [mailto:cpp...@li...] On
> > Behalf Of Darren Garvey
>

> Perhaps we can open discussion about where things should and
> > shouldn't align and how best to do that? It'll be summer
> > before I have any real time to spend on this, but I'm really
> > keen that we can make some progress on this.
> >
>
> Like Glyn, I'll try looking into the code. However, I may very well be
> concentrating more in helping out Mike with the integration of
> pion-net's HTTP code into the library. It would be a dream come true if
> somehow the HTTP message code we're developing will be the same one the
> CGI library will be using! :)


This is the code in (SVN_ROOT)/branches/http_integration/ ? I've had a brief
look at it a couple of times, it has great potential. I think the first
steps towards integration of the CGI library is to start using the
basic_message<> class, which I think should be pretty straight-forward.

> P.S.
> >
> > Ideas:
> > * add a form_parser class (as part of the public API)
> >    - use Boost.Spirit
> >    - should read/parse multipart/form-data regardless of request type.
> >    - should allow incremental reading/parsing of individual form parts
>
> I like this. :)
>
> We can even do this using the TDD approach where we can write the unit
> test(s) independent of any subsystem, and develop the parser(s) to
> implement to make the unit tests pass. This way, the compliance to
> parsing multipart/form-data can be developed independently.


Exactly! All this requires is using Boost.Asio's (A)Sync(Read|Write)Stream
concepts, so shouldn't require too much isolated code.

> * add a param class
> >    - holds a boost::iterator_range to the data
> >    - convertible to std::string subject to a macro condition
> > (aka. taint mode)
> >    - allows easy case-insensitive comparison (possibly using
> > Boost.String_algo? Or the netlib transform meta-functions?)
> >    - allows compile-time checking of its source (eg.
> > env/GET/POST) to protect against XSS vulnerabilities.
> >    - allows easy lexical casting (eg. param::as<>)
> >
>
> I don't really understand what you mean by this, but if you mean 'param'
> in the query string parameter case, then this may be a good idea. I
> don't see though if using Boost.Any might make more sense for some
> people who want the dynamic-ness (if there's such a thing) of the value.


I was referring to a possible container for any/all parsed request data. The
idea is that data can be read directly into a basic_message<> and when it's
parsed, the name/values of each 'request variable' is stored in a container
that just *points* to the data, without copying it.

The  `param` (or `basic_param<>`) class would be little more than a
cheap-to-copy drop-in replacement for std::string that supplied minimal
networking-related member functions (eg. a case-insensitive operator==() and
nested typedefs like ::source_type).

I just wrote a load about how it might work, but it might be better if I
threw together an implementation, rather than rambling on about what it
could do!

[snip]

I wasn't too sure about the as<> member function myself either, but several
people requested it and the idea already has precedence in boost libraries,
although I forget which ones right now. Perhaps it would be better to have
an 'as<>-less' network::basic_param<> and then a cgi::basic_param<> which
derives from the former and adds the as<> member function..?

What do you think? *

> * add prepare/consume/commit member functions to
> > network::basic_message<>
> >   - functions like in asio::streambuf.
> >   - would allow reading data straight from a
> > (asio)(A)SyncReadStream (eg. a socket), without need for buffering
> >   - could help with incremental parsing and/or making
> > form_parser class request-agnostic
> >
>
> My suggestion instead of overloading the interface of basic_message<> is
> to define adaptors.


Ahh, interesting...

It should be feasible to have a
> 'streamed<basic_message<> >' or 'async<basic_message<> >' instance which
> would decorate the functionality of the basic_message encapsulated
> within the instance. That way it can still be converted into and is
> still interface compatible to basic_message<>, but adds functionality
> that's typically not inherent in the actual message.


... and supplies reusable components too. :)

streamed<basic_message<http::message_tag> > streamed_message;
> std::string body_line;
> getline(body(body_stream), body_line); // synchronously wait for the
> first body line to be streamed
>
> async<basic_message<http::message_tag> > async_message;
> future<std::string> body_line = body(async_message);
> // at some point in time
> cout << body_line.get();
>
> This keeps the basic_message<> interface intact and the usage semantics
> consistent. What do you think? :D


This sounds like a good idea. For a readable basic_message<> - ie. with the
only addition being member functions prepare/commit/consume - how about
storage<basic_message<>> ? This would allow direct reading/writing to/from
the message with Asio.

> P.P.S.
> >
> > The basic_message<> class has embedded _source/_destination
> > parameters. These aren't particularly relevant to
> > CGI/FastCGI/SCGI and they imply a close relation between the
> > payload of an HTTP message and its source/destination, which
> > might not always be a 1:1 relationship. I wonder if they
> > should be moved out of the basic_message<> class to some
> > higher-level abstraction?
>
> This may be a good idea, but it's possible to actually use SFINAE or a
> cgi::message_tag to specialize on the basic_message<>'s internal
> representation while keeping it consistent. For example, you can disable
> using SFINAE the interface to accessing (and presence of) the
> _source/_destination parameters.


Yeah, this is possible, but do you not think the basic_message<> class
should be as minimal as possible? In the HTTP case you could possibly use
something like:

template<>
class basic_message<http::message_tag>
  : basic_message<tags::default_> // this might be wrong
{
  std::string _source;
  std::string _destination;
public:
  std::string& source() const { return _source; }
  std::string& destination() const { return _destination; }
};

We could then build up a message-type 'stack' on top of this for the
HTTP-based protocols and other 'stacks' for any other ones.

What do you think Darren? I hope these make sense! :)


It all sounds very interesting. I wish I had some time to start on it now!

Cheers,
Darren

[*] From early discussion for the CGI library, some people talked about
embedded system support. They seem to prefer very tight controls on memory
allocation, so avoiding copying might make these use-cases easier to
support...