cpp-netlib-devel — Developers Mailing List

Re: [cpp-netlib-devel] library goals and roadmap

[Dean M. B. <mik...@gm...>]

Hi Jose!

On 6/4/07, Jose <jm...@gm...> wrote:
> Hi Dean,
>
> I've copied your doc here so that comments can be provided inline and the
> doc can be merged with the doc I sent a few minutes after you.
> I will comment on your doc in a separate message
>

Thank you very much! I like the list you also posted, it is very much
appreciated. :)

-- 
Dean Michael C. Berris
http://cplusplus-soup.blogspot.com/
mikhailberis AT gmail DOT com
+63 928 7291459

Re: [cpp-netlib-devel] library goals and roadmap

[Jose <jm...@gm...>]

Hi Dean,

I've copied your doc here so that comments can be provided inline and the
doc can be merged with the doc I sent a few minutes after you.
I will comment on your doc in a separate message

======================================================================================

C++ Networking Library
Goals and Scope


Objectives
----------

 o  Develop a high quality, portable, easy to use C++ networking library

 o  Enable users to easily extend the library

 o  Lower the barrier to entry for cross-platform network-aware C++
    applications


Goals
-----

 *  Implement a simple message implementation which can be used in
    network protocol-specific routines for inter-operability and
    to provide a generic interface to manipulating network-oriented
    messages.

 *  Implement easy to use protocol client libraries such as (but not
    limited to):

        -  HTTP 1.0/1.1
        -  (E)SMTP
        -  SNMP
        -  ICMP

 *  Implement an efficient easy to use URI class/parser.

 *  Implement a fully compliant cross-platform asynchronous DNS resolver
    either as a wrapper to external (C) libraries, or as hand-rolled
    implementation.

 *  Implement a MIME handler which builds message objects from either
    data retrieved from the network or other sources and create
    text/binary representations from existing message objects intended
    for transport over the network.


Scope
-----

 *  The library will provide a generic message class which is intended
    to be the common message type used by the protocol libraries.

 *  The library will only contain client implementations for the various
    supported protocols.

 *  The library will use only STL and Boost C++ library components,
    utilities, and libraries throughout the implementation.

 *  The library will strive to use C++ templates and template
    metaprogramming techniques in order to not require the building of
    external shared/static libraries. In other words, the library will be
    header-only and compliant with the C++ standard.


On 6/4/07, Jose <jm...@gm...> wrote:
>
>
> Hi,
>
> I would like to get a basic agreement on the goals and roadmap, so that we
> can choose where to contribute and avoid duplicating effort. Pls send me
> your comments so that
> we can get a more concrete document
>
> General library goals
> ===============
>
> - Performance (do not pay for what you don't use)
> I think this is a goal for many C++ libraries and it is specially
> important for  a c++ network library
>
> - Reusability of protocol classes
> Already discussed. Very important to the protocol designer
>
> - Protocol conformance
> I think this is key for the success of the library. There are many
> protocol implementations out there and what really hampers them is that they
> provide a partial implementation, which looks like a proof of concept but
> not a complete one. So, whatever protocols are choosen (client or/and server
> side), they should be fully implemented otherwise users will fork their own
> implementations (which is fine but contradicts the value of a library)
>
> Below is a basic list of protocols (which should be in order of priority).
> It has already been stated in the mailing list that the library goal is to
> provide reusable protocol development components and a few protocol
> implementations. Applications like web server or email server could be
> provided but it is not a goal to provide competitive application
> implementations.
>
> Protocol Roadmap
> ==============
>
> * HTTP 1.0 client
>     - URI parser
>     - request object
>     - reply parser
> * SMTP client
>     - pls send the list of key features
> * HTTP 1.1 client
>     - cookies
>     - gzip
>     - multipart mime
>     - easy interface to async DNS
> * HTTPS
> * ESMTP client
> * HTTP 1.0 server
> * HTTP 1.1 server
> * ESMTP server
>
> Example Applications
> ================
>
> Web server, email server, ...
>
>
> Pls send your comments
>
> regards
> jose
>

[cpp-netlib-devel] library goals and roadmap

[Jose <jm...@gm...>]

Hi,

I would like to get a basic agreement on the goals and roadmap, so that we
can choose where to contribute and avoid duplicating effort. Pls send me
your comments so that
we can get a more concrete document

General library goals
===============

- Performance (do not pay for what you don't use)
I think this is a goal for many C++ libraries and it is specially important
for  a c++ network library

- Reusability of protocol classes
Already discussed. Very important to the protocol designer

- Protocol conformance
I think this is key for the success of the library. There are many protocol
implementations out there and what really hampers them is that they provide
a partial implementation, which looks like a proof of concept but not a
complete one. So, whatever protocols are choosen (client or/and server
side), they should be fully implemented otherwise users will fork their own
implementations (which is fine but contradicts the value of a library)

Below is a basic list of protocols (which should be in order of priority).
It has already been stated in the mailing list that the library goal is to
provide reusable protocol development components and a few protocol
implementations. Applications like web server or email server could be
provided but it is not a goal to provide competitive application
implementations.

Protocol Roadmap
==============

* HTTP 1.0 client
    - URI parser
    - request object
    - reply parser
* SMTP client
    - pls send the list of key features
* HTTP 1.1 client
    - cookies
    - gzip
    - multipart mime
    - easy interface to async DNS
* HTTPS
* ESMTP client
* HTTP 1.0 server
* HTTP 1.1 server
* ESMTP server

Example Applications
================

Web server, email server, ...


Pls send your comments

regards
jose

[cpp-netlib-devel] RATIONALE documentation in SVN

[Dean M. B. <mik...@gm...>]

Hi Everyone!

(I don't know if 'Guys' is a politically correct term, mainly because
I don't know if we're all male here, or whether there are females on
the list too... So I'll refer to everyone as "Everyone"... If I slip
and say 'Guys' sometimes, I apologize in advance. :D)

So I've written down quickly what I would like to see in the library,
and how I want to achieve them. Please see the attached file -- your
response, comments/thoughts would be most appreciated.

I certainly hope this helps!

(The file is also available via SVN, so any changes we decide to make
on the list can be reflected in the history of the subversion
repository). :-)

-- 
Dean Michael C. Berris
http://cplusplus-soup.blogspot.com/
mikhailberis AT gmail DOT com
+63 928 7291459

Re: [cpp-netlib-devel] Adding files to the sandbox

[Jose <jm...@gm...>]

Hi Peter,

Thanks for the detailed answer.

I think a hand-written parser may be a good idea, maybe just for http uris:

user:password@host:port/file

so that the the typical (host, file) pair needed for HTTP requests can be
obtained as
quickly as possible. Additional parsing of the file segment can be provided
when needed (e.g. for server apps).

What do you think ?

regards
Jose

On 03 Jun 2007 23:17:55 +0200, Peter Simons <si...@cr...> wrote:
>
> Hi Jose,
>
> you are right, there is a spirit-based URI parser in mini-httpd.
> Unfortunately, it is incomplete insofar as that it understands
> HTTP URLs only and doesn't recognize literal IPv6 host names. In
> addition, the code is a bit messy because it was written several
> years ago, at a time where Spirit didn't have the sophisticated
> actor infrastructure it has these days.
>
> In my experience, the greatest challenge when parsing an URI is
> not the parser, it is the resulting data structure. The URI class
> mini-httpd uses in fine for mini-httpd, but it certainly is far
> from generic.
>
> RFC 2396 comes with a state-machine for parsing URIs, by the way,
> and it's pretty wild:
>
> |    ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?
> |     12            3  4          5       6  7        8 9
>
> The relevant sub-match states are:
>
> |    scheme    = $2
> |    authority = $4
> |    path      = $5
> |    query     = $7
> |    fragment  = $9
>
> In terms of performance, I doubt that there is much of a
> difference between Spirit and Boost.Regex in this context. The
> main difference is that Boost.Regex must be linked whereas Spirit
> is a header-only library. That may or may not matter to our
> users; it's hard to tell.
>
> One disadvantage of Spirit is that compile-time goes through the
> roof even for trivial grammars. Another problem is that Spirit
> relies one rather sophisticated magic to be thread-safe. Compiled
> regular expressions, however, are immutable and can be used by
> any number of threads concurrently without synchronization.
>
> A hand-written parser might be slightly faster than either Spirit
> or Boost.Regex. It's definitely harder to get right, though. :-)
>
> Best regards,
> Peter
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Cpp-netlib-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel
>

[cpp-netlib-devel] Binary message class

[<rhy...@gm...>]

Hi, all

Currently, the cpp-netlib is only text protocol oriented, since the 
underlying data storage facility is actually std::string (definitely 
will be replaced by std::basic_string<>). This maybe enough for 
protocols like http, but definitely not for binary protocols like socks. 
And, I would like to have a binary protocol friendly network framework. 
How about you guys? ;-)

For binary data processing abilities, I suggest the following:

 * reference counted binary buffer

    To avoid memory copies, reference counted binary buffer would help a 
lot.  I think boost::shared_ptr and boost::asio::basic_streambuf<> may 
help. The asio basic_streambuf<> template is automatically resizable and 
uses a vector<> for the underlying data storage. With Boost.Pool for 
memory management, I think it may be an excellent choice.

 * standard compliant marshalling/unmarshalling

    Binary messages should be constructed easily with 
marshalling/unmarshalling facilities. The library provides facilities to 
marshal/unmarshal basic data types such as different kinds of (unsigned) 
integers, string, sequence and array, and user can define their own 
overloaded marshal/unmarshal methods to deal with user defined data 
structures:

        namespace network {
            typedef basic_message<tag::binary_> bin_message;
        } // namespace network

        struct payload {
            ...
        };

        bin_message msg;
        payload p;
        uint32_t len = sizeof( payload )

        msg
            << source( ... )
            << dest( ... ) << dest( ... )
            << marshal( len ) << marshal( p );

    It also would be fine to have a standard compliant 
marshalling/demarshalling policy, may be CORBA CDR could be a choice?

In one word, I'd like to have a binary message class that plays the role 
of ACE_Message_Block and ACE_InputCDR/ACE_OutputCDR. What about your 
opinions, guys?

Cheers
Cheng

Re: [cpp-netlib-devel] Adding files to the sandbox

[Peter S. <si...@cr...>]

Hi Jose,

you are right, there is a spirit-based URI parser in mini-httpd.
Unfortunately, it is incomplete insofar as that it understands
HTTP URLs only and doesn't recognize literal IPv6 host names. In
addition, the code is a bit messy because it was written several
years ago, at a time where Spirit didn't have the sophisticated
actor infrastructure it has these days.

In my experience, the greatest challenge when parsing an URI is
not the parser, it is the resulting data structure. The URI class
mini-httpd uses in fine for mini-httpd, but it certainly is far
from generic.

RFC 2396 comes with a state-machine for parsing URIs, by the way,
and it's pretty wild:

 |    ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?
 |     12            3  4          5       6  7        8 9

The relevant sub-match states are:

 |    scheme    = $2
 |    authority = $4
 |    path      = $5
 |    query     = $7
 |    fragment  = $9

In terms of performance, I doubt that there is much of a
difference between Spirit and Boost.Regex in this context. The
main difference is that Boost.Regex must be linked whereas Spirit
is a header-only library. That may or may not matter to our
users; it's hard to tell.

One disadvantage of Spirit is that compile-time goes through the
roof even for trivial grammars. Another problem is that Spirit
relies one rather sophisticated magic to be thread-safe. Compiled
regular expressions, however, are immutable and can be used by
any number of threads concurrently without synchronization.

A hand-written parser might be slightly faster than either Spirit
or Boost.Regex. It's definitely harder to get right, though. :-)

Best regards,
Peter

Re: [cpp-netlib-devel] Adding files to the sandbox

[Jose <jm...@gm...>]

On 6/3/07, Glyn Matthews <gly...@gm...> wrote:
>
> OK thanks Dean, I was trying to use SVN import.
>
> I added some stuff to parse URIs using boost spirit, it doesn't quite work
> (there is something wrong with the way I've done the domain names), but I
> hope I can open it up comments and improvements.
>


Hi Glyn,

Peter's server daemon code already has a complete URI parser (I think!).

Given that URI parsing is the most common operation, I wonder whether there
would be a performance advantage by parsing the URI via a state machine vs
spirit

regards
jose

Re: [cpp-netlib-devel] Adding files to the sandbox

[Dean M. B. <mik...@gm...>]

On 03 Jun 2007 18:17:47 +0200, Peter Simons <si...@cr...> wrote:
> Hi Glyn,
>
>  > The list has been rather quiet in the last week. Has anyone
>  > made any progress?
>
> Well, I find it hard to contribute because I don't know into
> which direction this project is headed. That makes it difficult
> to tell what exactly would constitute progress. I'll remain an
> interested observer until I have a better understanding of
> cpp-netlib's scope and goals.
>

Hi Peter!

I understand. Let me deliver the intended direction, scope, and goals
in a document (currently still under development) and hope to hear
what you think. I certainly value the insights everybody have been
sharing into what the networking library should and should not
contain, and how and how not things should be done.

I admit, I'm not very good at writing documentation -- but I do think
a lot in terms of actual implementation. I find it hard to write down
directions, goals, and scope but I would gladly implement it in C++
then have people try and use it. Case in point are the other libs I've
developed (the BDD interface and Runtime Dynamic Dispatcher) where the
implementations are code complete, but lacking in documentation -- it
seems that someone always needs to ask a question first before I be
able to explain what it does or doesn't do.

So consider this a call for help in that regard. :D

(If only blogging about it would be good enough... I would have done
that a long time ago) ;-)

> Best regards,
> Peter
>

Thanks Peter!

-- 
Dean Michael C. Berris
http://cplusplus-soup.blogspot.com/
mikhailberis AT gmail DOT com
+63 928 7291459

Re: [cpp-netlib-devel] Adding files to the sandbox

[Dean M. B. <mik...@gm...>]

Hi Glyn!

On 6/3/07, Glyn Matthews <gly...@gm...> wrote:
> OK thanks Dean, I was trying to use SVN import.
>
> I added some stuff to parse URIs using boost spirit, it doesn't quite work (there is something wrong with the way I've done the domain names), but I hope I can open it up comments and improvements.
>

Sounds like something worth looking at. :) Thanks Glyn!

> The list has been rather quiet in the last week.  Has anyone made any progress?

I personally haven't -- the work at Friendster had been keeping me
busy for the past week, not enough time to do any sort of forward
progress...

Look for a bit more documentation though, I've been concentrating a
lot in distilling the thoughts and actually coming out with a
clear-cut document saying what I'd like the networking library to
contain (of course, which also takes into account the suggestions and
'want to haves' we've pretty much listed down in a different thread).

If the pace is not fast enough, I'd welcome any sort of help in doing
that -- we have the Sourceforge Wiki available, and I have enabled
everyone to be able to write there.

I shold be getting back into implementing the transformation layers of
the message class, and hopefully have a (very) simple HTTP client and
asynchronous resolver wrapping the Boost.Asio resolver class and some
non-trivial examples. Hopefully the projects I'm part of in Friendster
start getting finished and so I can spend more time open sourcing the
memcache client library we've developed in house and this networking
library. :)

Until then, I'd appreciate all the help and feedback from everyone.

So if you have thoughts, and questions, I'd love to hear from you!

Thanks again Glyn. :)

-- 
Dean Michael C. Berris
http://cplusplus-soup.blogspot.com/
mikhailberis AT gmail DOT com
+63 928 7291459

Re: [cpp-netlib-devel] Adding files to the sandbox

[Peter S. <si...@cr...>]

Hi Glyn,

 > The list has been rather quiet in the last week. Has anyone
 > made any progress?

Well, I find it hard to contribute because I don't know into
which direction this project is headed. That makes it difficult
to tell what exactly would constitute progress. I'll remain an
interested observer until I have a better understanding of
cpp-netlib's scope and goals.

Best regards,
Peter

Re: [cpp-netlib-devel] Adding files to the sandbox

[Glyn M. <gly...@gm...>]

OK thanks Dean, I was trying to use SVN import.

I added some stuff to parse URIs using boost spirit, it doesn't quite work
(there is something wrong with the way I've done the domain names), but I
hope I can open it up comments and improvements.

The list has been rather quiet in the last week.  Has anyone made any
progress?
G

On 03/06/07, Dean Michael Berris <mik...@gm...> wrote:
>
> Hi Glyn!
>
> You should be able to check out the sandbox, then using your
> subversion client add the files to your local working copy, then do a
> commit. I'll be online through Yahoo Messenger and Google Talk later,
> I can try and help you out if you're having trouble with the sandbox
> and subversion.
>
> Hope this helps!
>
>
> On 6/3/07, Glyn Matthews <gly...@gm...> wrote:
> > Guys,
> >
> > I have something I want to put in the sandbox, how do I add them to the
> > repository?
> >
> > Regards,
> > Glyn
> >
>
>
> --
> Dean Michael C. Berris
> http://cplusplus-soup.blogspot.com/
> mikhailberis AT gmail DOT com
> +63 928 7291459
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Cpp-netlib-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel
>

Re: [cpp-netlib-devel] Adding files to the sandbox

[Dean M. B. <mik...@gm...>]

Hi Glyn!

You should be able to check out the sandbox, then using your
subversion client add the files to your local working copy, then do a
commit. I'll be online through Yahoo Messenger and Google Talk later,
I can try and help you out if you're having trouble with the sandbox
and subversion.

Hope this helps!


On 6/3/07, Glyn Matthews <gly...@gm...> wrote:
> Guys,
>
> I have something I want to put in the sandbox, how do I add them to the
> repository?
>
> Regards,
> Glyn
>


-- 
Dean Michael C. Berris
http://cplusplus-soup.blogspot.com/
mikhailberis AT gmail DOT com
+63 928 7291459

[cpp-netlib-devel] Adding files to the sandbox

[Glyn M. <gly...@gm...>]

Guys,

I have something I want to put in the sandbox, how do I add them to the
repository?

Regards,
Glyn

Re: [cpp-netlib-devel] Sandbox added into SVN

[Glyn M. <gly...@gm...>]

TGlzdCwKCk9uIDI5LzA1LzA3LCDBrLPHIDxyaHl0aG0ubWFpbEBnbWFpbC5jb20+IHdyb3RlOgo+
Cj4gSGksIEdseW4KPgo+IEFjdHVhbGx5LCB0aGUgcmVhc29uIHRoYXQgSSBzdWdnZXN0IHRvIHNl
dCB1cCBzdWNoIGEgc2FuZGJveCBpcyB0bwo+IGNvbGxlY3QgYWxyZWFkeSB3ZWxsLWZvcm1lZCBh
bmQgcmV1c2FibGUgc25pcHBldHMgdGhhdCBtYXkgY29udHJpYnV0ZQo+IGZyb20geW91IGFsbC4g
Oi0pIEFsdGhvdWdoIEkgdGhpbmsgd2UgYXJlIChtYWlubHkpIHN0aWxsIGluIHRoZSBkZXNpZ24K
PiBzdGFnZSwgSSdkIGxpa2UgdG8gc2VlIHNvbWUgcHJvdG90eXBlcywgcG9zc2libGUgdXNlIGNh
c2VzLCBhbmQgZXZlbgo+IHBzdWVkb2NvZGUsIHNpbmNlIGl0J3MgYSBTQU5EQk9YIDotRAoKCgpP
Sywgc28gYmVmb3JlIEkgZ2V0IHJlYWxseSBjYXJyaWVkIGF3YXksIGRvZXMgYW55b25lIGtub3cg
b2YgYW55CmltcGxlbWVudGF0aW9ucyBvZiBSRkMgMjM5Nj8KCkcK

Re: [cpp-netlib-devel] Sandbox added into SVN

[<rhy...@gm...>]

Hi, Glyn

Actually, the reason that I suggest to set up such a sandbox is to 
collect already well-formed and reusable snippets that may contribute 
from you all. :-) Although I think we are (mainly) still in the design 
stage, I'd like to see some prototypes, possible use cases, and even 
psuedocode, since it's a SANDBOX :-D

Glyn Matthews wrote:
> Cheng,
>
> On 28/05/07, 连城 <rhy...@gm...> wrote:
>>
>> Hi, all
>>
>> I've discussed with Dean and decided to set up a sandbox outside the
>> trunk. Since the word "network" is such a concept that covers a lot, I
>> think everyone has his/her own ideas and reusable code snippets that
>> might contribute to the project. (Shamelessly, currently I myself
>> haven't got such pieces of snippets :-[ , but I'll squeeze my brain to
>> contribute some :-) ).
>
>
> That's great!
>
> Do you think it might be useful to start implementing things like 
> protocol
> parsers here?
>
> Glyn
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> ------------------------------------------------------------------------
>
> _______________________________________________
> Cpp-netlib-devel mailing list
> Cpp...@li...
> https://lists.sourceforge.net/lists/listinfo/cpp-netlib-devel
>   

Re: [cpp-netlib-devel] Sandbox added into SVN

[Dean M. B. <mik...@gm...>]

Hi Glyn!

On 5/29/07, Glyn Matthews <gly...@gm...> wrote:
>
>  Do you think it might be useful to start implementing things like protocol
> parsers here?
>

Definitely! Please, if you feel like you can start on some part of the
library now, please go ahead. :) I'd love to see your code up at the
sandbox as well. :)

Then we can discuss the code in the mailing list. :)

-- 
Dean Michael C. Berris
http://cplusplus-soup.blogspot.com/
mikhailberis AT gmail DOT com
+63 928 7291459

Re: [cpp-netlib-devel] Sandbox added into SVN

[Glyn M. <gly...@gm...>]

Q2hlbmcsCgpPbiAyOC8wNS8wNywgwayzxyA8cmh5dGhtLm1haWxAZ21haWwuY29tPiB3cm90ZToK
Pgo+IEhpLCBhbGwKPgo+IEkndmUgZGlzY3Vzc2VkIHdpdGggRGVhbiBhbmQgZGVjaWRlZCB0byBz
ZXQgdXAgYSBzYW5kYm94IG91dHNpZGUgdGhlCj4gdHJ1bmsuIFNpbmNlIHRoZSB3b3JkICJuZXR3
b3JrIiBpcyBzdWNoIGEgY29uY2VwdCB0aGF0IGNvdmVycyBhIGxvdCwgSQo+IHRoaW5rIGV2ZXJ5
b25lIGhhcyBoaXMvaGVyIG93biBpZGVhcyBhbmQgcmV1c2FibGUgY29kZSBzbmlwcGV0cyB0aGF0
Cj4gbWlnaHQgY29udHJpYnV0ZSB0byB0aGUgcHJvamVjdC4gKFNoYW1lbGVzc2x5LCBjdXJyZW50
bHkgSSBteXNlbGYKPiBoYXZlbid0IGdvdCBzdWNoIHBpZWNlcyBvZiBzbmlwcGV0cyA6LVsgLCBi
dXQgSSdsbCBzcXVlZXplIG15IGJyYWluIHRvCj4gY29udHJpYnV0ZSBzb21lIDotKSApLgoKClRo
YXQncyBncmVhdCEKCkRvIHlvdSB0aGluayBpdCBtaWdodCBiZSB1c2VmdWwgdG8gc3RhcnQgaW1w
bGVtZW50aW5nIHRoaW5ncyBsaWtlIHByb3RvY29sCnBhcnNlcnMgaGVyZT8KCiBHbHluCg==

[cpp-netlib-devel] Sandbox added into SVN

[<rhy...@gm...>]

Hi, all

I've discussed with Dean and decided to set up a sandbox outside the 
trunk. Since the word "network" is such a concept that covers a lot, I 
think everyone has his/her own ideas and reusable code snippets that 
might contribute to the project. (Shamelessly, currently I myself 
haven't got such pieces of snippets :-[ , but I'll squeeze my brain to 
contribute some :-) ).

Cheers
Cheng

Re: [cpp-netlib-devel] Security, Authentication and Encryption

[Dean M. B. <mik...@gm...>]

Hi Glyn!

On 5/24/07, Glyn Matthews <gly...@gm...> wrote:
> Boost Networkers,
>

I like the sound of that. ;-)

>
> A few times on this list we have mentioned security without ever yet going
> into any detail about it.  I believe this a rather fundamental aspect of
> network programming and it would be wrong to design a networking library
> without addressing this.  However, I do think that this is a whole other
> application area and probably beyond the scope of boost.network.  But what
> do other people on this list thing?
>

I agree that security is a very broad topic which should live better
in a different library. I however think that security is integral in
some protocols aside from HTTP and SMTP, and should be made a protocol
specific thing. We should strike a balance and be as complete as we
can as far as implementing the protocols are concerned -- without
being too ambitious as to implement all the possible variations of a
protocol (i.e. HTTP 1.0/1.1 would be enough, supporting only a few
security/authentication methodologies -- same goes for SMTP/ESMTP
implementing the most common security/authentication methodologies).

So I'm guessing, that an smtp::credentials<> type would be different
from an http::credentials<> type, and that an http::session<>'s
authentication implementation would be different from an
smtp::session<>'s authentication implementation as well.

> My thoughts:
> I am not aware of anything in boost, or that's been proposed, in the domain
> of security.  This is an area that moves rapidly and I can imagine its very
> difficult for a library designer to keep abreast of all the changes.  Also
> it can be quite tightly coupled to a particular protocol.  However, there
> are definitely algorithms and techniques which are used repeatedly by
> application developers and which definitely belong in a re-usable library or
> framework.
>

I personally do not intend to implement security routines -- i'd much
rather defer that to other libraries (libgcrypt, or GNU TLS, etc.). I
do however intend to support some of these to be usable in the
library, depending on the protocols using it.

> Does anyone think it is possible to develop security software for our own
> requirements and eventually fork should someone have the interest or will to
> take it on?  What do people normally do when they require authentication /
> encryption?  I often find it difficult to set out a clear policy of this,
> and the code I and my co-developers write is often rather disorganised.
> Therefore I think its a good candidate for considering developing a library
> or framework for addressing this, or at the very least determining a clear
> boost.network policy.
>

I think it should be possible to abstract the authentication and
security parts using the 'traits' idiom, where we can store certain
types in traits container classes which defer the implementation of
specific security implementations in more detail. I'd like to use an
example of what I mean just to be more precise:

namespace policies {
#ifdef __ENABLE_GNU_TLS__
  namespace gnu {
    struct tls {
      typedef gnu_tls_authentication_module auth;
      typedef gnu_tls_encryption_methodology encryption;
      typedef some_other_gnu_hashing hash;
    };
  } // namespace gnu
#endif
#ifdef __ENABLE_WIN_TLS___
  namespace win {
    struct tls {
      typedef win_tls_authentication_module auth;
      typedef win_tls_encryption_methodology encryption;
      typedef some_other_win_hashing hash;
  }
#endif
} // namespace policies

namespace http { // for example
  template <class AuthPolicy>
  struct session {
    // use the AuthPolicy contained types here
    // like 'typename AuthPolicy::auth' or 'typename AuthPolicy::encryption'
  };
}

We can even write simple wrappers to adapt the API of the different
implementations to fit in the model that we will use. This should
allow for easy extension as we go along.

> Any comments?
> G
>

Thanks for asking the very important question and providing insight. I
certainly hope we can figure something out this early, so that we can
explore them as soon as we can with code. :-)

-- 
Dean Michael C. Berris
http://cplusplus-soup.blogspot.com/
mikhailberis AT gmail DOT com
+63 928 7291459

Re: [cpp-netlib-devel] Insights to the Message Class and Network Routines

[Dean M. B. <mik...@gm...>]

Hi Peter!

On 23 May 2007 19:42:26 +0200, Peter Simons <si...@cr...> wrote:
>
> it feels like we have different perspectives on the subject of
> networking. I mostly care about the server side. You seem to have
> more experience with the client side. Consequently, our views
> disagree on several occasions. This is good thing, because by
> finding a solution that we can agree on, we will come up with a
> solution that is better than any one either of us could have come
> up with alone.
>

I agree. When I proposed this networking library, I was more concerned
about implementing common client protocols, and have been pretty hazy
(or not very dedicated) to writing implementations for the server
side.

Your insights have opened my eyes to some issues I've failed to
consider on the server side implementation details, and now I'm doing
a lot of re-thinking about what the most productive approach would be.

> The difficult bit is to handle disagreements constructively.
>

Indeed. I however, am very open to suggestions and comments, so please
keep them coming. I certainly appreciate the input at any level. :-)

> After thinking some ideas through, I feel that I might have been
> aiming too high. An HTTP implementation is beyond our scope. I
> see it like this: when I want to write a sophisticated web
> application, I don't bother with an HTTP server. I write a CGI or
> FastCGI that works with _any_ HTTP server. Even assuming that I
> would have to commit to one particular HTTP server, frankly, I'd
> probably write an Apache module instead of a policy-driven
> handler functor for Boost.Network.
>

On the server side, I agree that it might be too ambitious to
implement the HTTP protocol and provide a "hook-into-skeleton"
implementation. But I'm not afraid to try and fail in this regard. :-)

So we can definitely forgo the server side HTTP implementation. While
I do agree that it would make sense to just write an application that
talked FastCGI or whatnot instead of using a http::server<...>, in my
work at Friendster I've seen that this does not scale well for
thousands of connections at a time.

I however am looking more at a very flexible HTTP client library. One
which any C++ programmer with STL experience will be able to use right
off the bat with minimal mental cartwheels.

> I wonder what our users hope to find in this library when it's
> finished. Please, everyone feel encouraged to post a list of
> things you would personally like to have; things you miss when
> writing network-oriented C++ code. Here is what I'd like to have:
>
[snipped great list]

I'd like to add the following to that:

o  A set of primitive networking routines like a "blocking read that
times out" or a "self-reconnecting socket connection" and "a raw
packet builder/sender".

o  A minimal but comprehensive HTTP client implementation

o  A minimal but comprehensive SMTP client implementation

o  A flexible interface/framework where I can add more protocol
implementations easily, or with very little effort (example, I want to
implement a new binary protocol, I just need to add a few classes
which talk a certain "language" and then leverage on whatever
Boost.Network already has)

So indeed, a URI class and a MIME handler would be good to have. I'm
still not sure though how we could make that work without dynamic
runtime programming (virtual, all that inefficient stuff).

>
> If Boost.Network offers high-quality solutions for those
> problems, I'll be a user. I also feel that getting those classes
> implemented and documented up to the standards Boost expects will
> be a major effort that's measured in months, not weeks.
>

Yes, but I am positive we can get a minimal HTTP/SMTP client going in
a couple weeks time. ;-) So the sooner I'm able to commit some more
stuff sitting on my machine, the better it would look as the days go
by. :-)

If anybody also has stuff you'd like to put into the subversion,
please feel free to do so. The sooner we can evaluate code from other
people, the faster this process could go. :-) (Take this as a call for
help, and if you can pick up from where the code currently already
is).

> Anyway, we have also talked about DNS:
>
>  > Remember though that Chris already has a resolver in
>  > Boost.Asio...
>
> Well, Asio has a wrapper for getnameinfo(). That is a portable
> system call that gives synchronous access to the resolver the OS
> uses. The system resolver is very convenient and great to have,
> but it is a synchronous resolver. A great many applications can't
> use it. Furthermore, the system resolver knows only about A
> records and PTR records. You can resolve names to addresses or
> addresses to names, but you cannot look up a mail exchanger or a
> TXT record. For all but the simplest applications, the system
> resolver is insufficient.
>

Ah, okay. Then that would be a good thing to implement, if the DNS
client that works on most platforms is inadequate, having one in
Boost.Network would make sense. Unless there are already better
implementations that do the job already...

> I feel an asynchronous DNS resolver is beyond the scope of Asio
> or Boost.Network. Providing convenient and type-safe access to
> the asynchronous resolvers other people have written, however,
> might be worthwhile.
>

I think implementing a server would be out of bounds of the
Boost.Network scope, but a header-only client can be very attractive.
:-)

> Take care,
> Peter
>

Thanks very much Peter for the insights! :-) They definitely help. :-)

-- 
Dean Michael C. Berris
http://cplusplus-soup.blogspot.com/
mikhailberis AT gmail DOT com
+63 928 7291459

[cpp-netlib-devel] Security, Authentication and Encryption

[Glyn M. <gly...@gm...>]

Boost Networkers,


A few times on this list we have mentioned security without ever yet going
into any detail about it.  I believe this a rather fundamental aspect of
network programming and it would be wrong to design a networking library
without addressing this.  However, I do think that this is a whole other
application area and probably beyond the scope of boost.network.  But what
do other people on this list thing?

My thoughts:
I am not aware of anything in boost, or that's been proposed, in the domain
of security.  This is an area that moves rapidly and I can imagine its very
difficult for a library designer to keep abreast of all the changes.  Also
it can be quite tightly coupled to a particular protocol.  However, there
are definitely algorithms and techniques which are used repeatedly by
application developers and which definitely belong in a re-usable library or
framework.

Does anyone think it is possible to develop security software for our own
requirements and eventually fork should someone have the interest or will to
take it on?  What do people normally do when they require authentication /
encryption?  I often find it difficult to set out a clear policy of this,
and the code I and my co-developers write is often rather disorganised.
Therefore I think its a good candidate for considering developing a library
or framework for addressing this, or at the very least determining a clear
boost.network policy.

Any comments?
G

Re: [cpp-netlib-devel] Insights to the Message Class and Network Routines

[Glyn M. <gly...@gm...>]

Peter,

On 23 May 2007 19:42:26 +0200, Peter Simons <si...@cr...> wrote:

>
> I wonder what our users hope to find in this library when it's
> finished. Please, everyone feel encouraged to post a list of
> things you would personally like to have; things you miss when
> writing network-oriented C++ code.


I'll put my user's hat on for the moment:

1) A set of easy-to-use common protocols.  When using C++ I find myself
rewriting a lot of this kind of code.

2) The ability to create and use custom protocols within the network
framework.  In my current position, I find myself communicating with exotic
hardware that uses their own protocols.  I'd love to have a way to just use
these within a C++ network framework.

3) Support for encryption and authentication.

Peter, I'd go along with all four points in your list too.

I use C++ a lot in my work and I look with envy at other languages which
have much better standard library support for network programming.  To my
mind, this is a major flaw with the current C++ libraries, that there is no
standard support for something as important as network programming.  Of
course, the recent discussions on this list demonstrate that is far from
being simple even to decide what kind of support C++ programmers require
with approaches from server and client perspectives.  Another difficulty is
that programmers quite different points of access (for want of a better
term) to programming networking applications.  What I mean by that is that
different applications need to be written on different network layers and it
is important that a library expose an interface to each layer.


Anyway, we have also talked about DNS:
> I feel an asynchronous DNS resolver is beyond the scope of Asio
> or Boost.Network. Providing convenient and type-safe access to
> the asynchronous resolvers other people have written, however,
> might be worthwhile.


I agree with this.  More generally, I think its important that we provide
simple, type-safe means for interfacing with other existing network C++
libraries and applications.

G

Re: [cpp-netlib-devel] Insights to the Message Class and Network Routines

[Alex O. <al...@gm...>]

Hello all

On 23 May 2007 19:42:26 +0200, Peter Simons <si...@cr...> wrote:
> Hi Dean,
>
> it feels like we have different perspectives on the subject of
> networking. I mostly care about the server side. You seem to have
> more experience with the client side. Consequently, our views
> disagree on several occasions. This is good thing, because by
> finding a solution that we can agree on, we will come up with a
> solution that is better than any one either of us could have come
> up with alone.
>
> The difficult bit is to handle disagreements constructively.
>
> After thinking some ideas through, I feel that I might have been
> aiming too high. An HTTP implementation is beyond our scope. I
> see it like this: when I want to write a sophisticated web
> application, I don't bother with an HTTP server. I write a CGI or
> FastCGI that works with _any_ HTTP server. Even assuming that I
> would have to commit to one particular HTTP server, frankly, I'd
> probably write an Apache module instead of a policy-driven
> handler functor for Boost.Network.
>
> I wonder what our users hope to find in this library when it's
> finished. Please, everyone feel encouraged to post a list of
> things you would personally like to have; things you miss when
> writing network-oriented C++ code. Here is what I'd like to have:
>
>  (1) A class to represent any kind of URI. I want to be able to
>      read an URI by saying "Uri uri; cin >> uri;", and there it
>      is. I would also like to be able to write open(uri) to
>      obtain a stream socket that is connected to the address the
>      URI represents. (libcurl does something like this in C.)
>
>  (2) Uri's know "mail:user@domain", so clearly (1) requires a
>      class to represent an e-mail address. I frequently need
>      lists of addresses too.

I'm not sure, that mailto: should be treated as an uri at SMTP messages

>  (3) Parsers for all kinds of HTTP and E-Mail headers. I have
>      "Date: Wed May 23 19:05:46 CEST 2007\r\n", and now I want to
>      know what time that is. The same goes for
>      "If-Modified-Since:" and a lot of other headers.
>
>  (4) A class to represent multi-part MIME messages. MIME is used
>      in e-mail and in HTTP, yet in practice it's notoriously hard
>      to handle. A good MIME Message class should support
>      registering content-type-specific data handlers. Issuing a
>      GPG certificates -- or verifying one -- is something such
>      user-supplied function could do.

Yes, MIME support is important for all of protocols in current time.

In our parsers, that we implemented on my previous work  (currently
closed source) we had presented the message as a "lazy" structure,
that parse data when we try to access to underlying elements (usually
by name).
So for example, HTTP request consists from:
Method URL Version
Headers
empty line
Body

every of elements had parsers, associated with them, and it was
possible to change parsers at runtime (especially for POST's).  For
headers, also was a list of mappings Name -> parser, that allow us to
transparently parse data in headers, and represent them as a string or
corresponding structure (members of that, also was available via named
map).
And when members of structure was changed, the text representation
also will rebuild, when requested

Here small example:

node_type uri("uri", request["uri"].value(), new
mimer::http::uri_nparser<std::string>);
cout << "query=" << uri["query"].value() << std::endl;
// .... process data ...
request["uri"]=newUri; // modify it

-- 
With best wishes,                    Alex Ott, MBA
http://alexott.blogspot.com/
http://alexott-ru.blogspot.com/
http://content-filtering.blogspot.com/
http://xtalk.msk.su/~ott/

Re: [cpp-netlib-devel] Insights to the Message Class and Network Routines

[Peter S. <si...@cr...>]

Hi Dean,

it feels like we have different perspectives on the subject of
networking. I mostly care about the server side. You seem to have
more experience with the client side. Consequently, our views
disagree on several occasions. This is good thing, because by
finding a solution that we can agree on, we will come up with a
solution that is better than any one either of us could have come
up with alone.

The difficult bit is to handle disagreements constructively.

After thinking some ideas through, I feel that I might have been
aiming too high. An HTTP implementation is beyond our scope. I
see it like this: when I want to write a sophisticated web
application, I don't bother with an HTTP server. I write a CGI or
FastCGI that works with _any_ HTTP server. Even assuming that I
would have to commit to one particular HTTP server, frankly, I'd
probably write an Apache module instead of a policy-driven
handler functor for Boost.Network.

I wonder what our users hope to find in this library when it's
finished. Please, everyone feel encouraged to post a list of
things you would personally like to have; things you miss when
writing network-oriented C++ code. Here is what I'd like to have:

 (1) A class to represent any kind of URI. I want to be able to
     read an URI by saying "Uri uri; cin >> uri;", and there it
     is. I would also like to be able to write open(uri) to
     obtain a stream socket that is connected to the address the
     URI represents. (libcurl does something like this in C.)

 (2) Uri's know "mail:user@domain", so clearly (1) requires a
     class to represent an e-mail address. I frequently need
     lists of addresses too.

 (3) Parsers for all kinds of HTTP and E-Mail headers. I have
     "Date: Wed May 23 19:05:46 CEST 2007\r\n", and now I want to
     know what time that is. The same goes for
     "If-Modified-Since:" and a lot of other headers.

 (4) A class to represent multi-part MIME messages. MIME is used
     in e-mail and in HTTP, yet in practice it's notoriously hard
     to handle. A good MIME Message class should support
     registering content-type-specific data handlers. Issuing a
     GPG certificates -- or verifying one -- is something such
     user-supplied function could do.

If Boost.Network offers high-quality solutions for those
problems, I'll be a user. I also feel that getting those classes
implemented and documented up to the standards Boost expects will
be a major effort that's measured in months, not weeks.

Anyway, we have also talked about DNS:

 > Remember though that Chris already has a resolver in
 > Boost.Asio...

Well, Asio has a wrapper for getnameinfo(). That is a portable
system call that gives synchronous access to the resolver the OS
uses. The system resolver is very convenient and great to have,
but it is a synchronous resolver. A great many applications can't
use it. Furthermore, the system resolver knows only about A
records and PTR records. You can resolve names to addresses or
addresses to names, but you cannot look up a mail exchanger or a
TXT record. For all but the simplest applications, the system
resolver is insufficient.

I feel an asynchronous DNS resolver is beyond the scope of Asio
or Boost.Network. Providing convenient and type-safe access to
the asynchronous resolvers other people have written, however,
might be worthwhile.

Take care,
Peter