naviserver-tracker Mailing List for NaviServer (Page 6)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Feature Requests item #1151137, was opened at 2005-02-24 16:01
Message generated for change (Settings changed) made by seryakov
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=719009&aid=1151137&group_id=130646

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: C-API
Group: None
>Status: Closed
Resolution: None
Priority: 5
Submitted By: Vlad Seryakov (seryakov)
Assigned to: Vlad Seryakov (seryakov)
Summary: New driver API and Udp module

Initial Comment:
Hi guys,

Attached is minor driver extensions which do not change
existing drivers  but add new functionality. There are
some cosmetic changes, like moving some fields in the
Ns_Sock/Ns_Driver structres so they can be accessed
publically and made some private functions public but
functionality preserved as before.

I included udp driver as an example of new API, and
also added ns_sha1 command in the tclmisc.c, it is just
one command and it is uses practically everywhere.

-------------------------------------------------------------------------
To test udp driver i use new ns_udp command:

ossweb:nscp 8> ns_udp send 127.0.0.1 5060 "GET /
HTTP/1.0\n\n"

HTTP/1.0 200 OK
MIME-Version: 1.0
Date: Thu, 24 Feb 2005 05:39:50 GMT
Server: NaviServer/4.0.10
Content-Type: text/html; charset=iso-8859-1
Content-Length: 661
Connection: close

<HEAD><TITLE>Seryakov's Family Intranet</TITLE></HEAD>
.....

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-05-17 19:24

Message:
Logged In: YES 
user_id=184124

I am attaching new nsudp driver, it uses existing driver sock,
just couple places where NS_DRIVER_UDP flag is checked for
initializing and not performing accept calls. ns_udp module
is simple driver like nssock.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-05-04 18:01

Message:
Logged In: YES 
user_id=184124

I implemented my SIP driver completely in separate threads
and do not use Conn/Queue or jobs at all. When comparing
performance with SER(sip proxy), NS still behind because of
multi-threading overhead while SER forks many children
listening on the same socket, same way as apache 1.x used to
do. Also, NS/AS/Tcl memory allocator is slower than
non-threaded allocator. My first version was using
Conn/SockCallbacks but it was so slow, i could not process
more than 500 requests per second. Now i can do 9000
requests per second on the same server. So, having UDP
implemented in the core is not that important for me but
still, using  UDP as signaling/messaging protocol instead of
HTTP or implementing other not that performance-needed
drivers would be great.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-03-18 18:17

Message:
Logged In: YES 
user_id=184124

>I'm happy to see any of the core changed to add this
>support, but it needs to be justified.  I don't want to
>support multiple different ways of doing the same thing.

No, it is not the same thing. To build different drivers,
API and tools should be flexible enough, if we have only one
way to add a driver, so only drivers which fit will be(if)
added, and if something usefull will come up, will we need
to change the core again to fit different mehtod?

Basically you offered the same thing that i am talking
about: if we supply new DriverThread for UDP driver, the
only common is how to queue it, public NsQueueConn. The rest
is driver/thread specific. Currently, i can live with
specific thread proc using sock callbacks, but when driver
threads will be available, i can switch to it, but still
once i read data from UDP, i need to build Sock/Conn and
queue it. This is what i am offering, make NsQueueConn
public plus/mines changing some other function but just to
only support public NsQueueConn. IT will not conflict with
future global driver changes.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-03-18 14:28

Message:
Logged In: YES 
user_id=184124

One good for having NsQueueConn public is to be able to
submit connection threads from memory, without actual
protocol/driver connection. I have data to spawn connection
thread for and i just submit it. This way my server uses
thread/connection pooling without implementing my own thread
management(for example i do this now in my monitoring, but i
could use conn threads instead with better control over the
queue).

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-03-18 14:23

Message:
Logged In: YES 
user_id=184124

Yes, it will work in the long run. Currently i do not have
any problems with having only one DriverThread but having
different driver threads with specific ThreadProc would be
nice. It is just last time i tried new 4.1 driver, i could
not work with it using Mozilla, somehow all connections got
stale and mozilla was excepting more and more data from the
server while server thought it sent everything already. My
point: it needs more testing and even AOL did not start
testing it.

In the short run i am fine with ability to use
sockcallbacks, i just need to be able to submit Sock to the
ConnQueue, couple of small functions i added. If we can
allow it for now and wait till 4.1 driver will be stable i
would be okay.

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-03-18 13:03

Message:
Logged In: YES 
user_id=87254

I'm not against making NsQueueConn() public per-se, I am
against publishing it as a work-around to creating a good
API for pluggable protocols.

I'm keen to see great support for text based and binary
protocols, packet based and stream based protocols,
implementations which want to use a listen thread and
multiple worker threads, and those which want to do all work
in a single thread.

I'm happy to see any of the core changed to add this
support, but it needs to be justified.  I don't want to
support multiple different ways of doing the same thing.

Maybe we should backport the changes in 4.1 to our tree
before we start hacking into it?  We could leave out the adp
changes etc. as I believe someone said they were broken.  We
should make a point release before doing this.  What do you
think?

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-03-18 12:46

Message:
Logged In: YES 
user_id=87254

Sorry, I don't have a patch to show how UDP/Unix sockets
could be integrated into nssock.  The 4.1 code base would
seem to be a better starting point as it creates a new
thread for each loaded comm driver.

Here's how it could work:

Unix domain socket support is easy.  Use the same syntax as
for the binder for the address param of the driver module in
the config file.  Call the correct socket listen routine
depending on type within NsStartDrivers().  The nssock
module should require no changes to work with this socket type.

UDP sockets are harder.  The 4.1 code base has changed
NsStartDrivers() to spin off a thread for each loaded comm
module.  As the requirements for stream and packet sockets
are different, a different routine would be passed to
Ns_ThreadCreate() for each type.

I would rename DriverThread() StreamThread() and add a new
routine, PacketThread().  Much of the contents of
DriverThread() can be junked for handling packet based
protocols, but not all should.  The new QueueWait callbacks
and pre-queue filters are just as applicable here as for
stream based protocols.

The advantage of this is that there is a clear distinction
between network transport, protocol, and application. 
Network transports such as SSL/TLS, SSH, etc. can be swapped
in below different protocol implementations.  Many
applications can be built on a protocol implementation if it
provides the standard hooks (registered proc, filters,
QueueWait callbacks, pre-queue filteres...).

Is this enough to give you an idea?  It seems feasable to
me, what do you think?

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-03-18 04:31

Message:
Logged In: YES 
user_id=184124

Do we have any conclusions about this? Soon i will have to
develop some application with support for SIP over UDP, i
would like to do it on Naviserver and not on my hacked AS.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-03-04 16:44

Message:
Logged In: YES 
user_id=184124

And again, i have nothing against it, i said it before, what
i am offering is a littl ebit different, for network based
protocols driver stuff is kay, i just want to be able to
submit conn threads from within the server, not through the
main driver.
If i have already request data, through shared memory, file
or other way, i want to submit wroker thread directly, not
by connecting to myself and submitting data using HTTP or
other protocol i will have to implement just to queue te
request. This is for other applications, not HTTP/Web based,
more like messaging/authentication servers and alike.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-03-04 14:41

Message:
Logged In: YES 
user_id=184124

Can you provide what you have to modify in driver/nssock, i
think that will not be easy and will require too many
modifications.

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-03-04 10:08

Message:
Logged In: YES 
user_id=87254

What do you think of my suggestion that the nssock module
and nsd/driver.c could be easily modified to handle udp and
unix domain sockets when specified in the config file?

A change like that combined with the work you've already
done to the binder would mean no development is required to
handle all socket types, and would be transparent to the
upper layers.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-26 17:10

Message:
Logged In: YES 
user_id=184124

Stephan,
I am suggesting a compromise, let's put ParseProc and
DriverSockQueu patches, they can coexists easily but will
offer two way to add new drivers. Once we have new drivers
working we will see what can be modified/adjusted, for now
we do not have anything except my several drivers i use in
the production and i use my own loop for smtp driver to keep
processing and state machine in one place.
Again, i have nothing against your method, i used to do
similar in my previous versions but now i need to be
completely free to implement aditional driver and i think we
should have this ability.

Zoran, what do you think?

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-25 22:57

Message:
Logged In: YES 
user_id=184124

Try to implement HTTP over UDP as i provided in the patch,
you will need to modify core driver again, this is one example.

Next, if i want to implement small driver that do not
interact with web part of aolserver, no filters, it just
need to do one small thing, it should be fast but will
handle very many requests. I do not need filters, callbacks,
i need my own main loop. 
I do not understand why you are insisting that all drivers
should go through http driver's main loop. If i implement
dns server and web interface to it, why they should compete
in the same driver's main loop and go through registered
filters if they share only backend database structure?

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-02-25 22:22

Message:
Logged In: YES 
user_id=87254

You don't have to use registered filters or procs.  It's an
option for protocol implementations where the initial accept
and parsing take place in the driver thread, and processing
takes place in a conn thread.

Once you submit to the queue however, either a filter, a
registered proc, or a registered proxy proc must handle the
request.  It makes no difference whether you submit
explicitly with Ns_DriverSockQueue or implicitly after your
Ns_ParseProc completes.

How much of the request structure you fill in is up to you,
it's all optional.  If you choose not to use it there is no
overhead.  It doesn't have to be a text based protocol for
this to make sense.  You might fill in the request structure
if you wanted to enable people to handle different request
types via C or Tcl.  Or, you could store binary data in
Ns_Cls storage and access it from registered procs etc.

In the case where you want to handle connection
accept/parse/reponse processing in the same thread, nothing
needs to be added.  You can do that today with
Ns_SockCallback (or by placing all your code in an
Ns_ParseProc and returning the response from there)

Can you be more specific about the overhead of using an
Ns_ParseProc?  Exactly what memory is allocated, what code
is run that should not..?

What specifically can you not do with the Ns_ParseProc
interface that you need the Ns_Driver* routines for?  I'm
looking for concrete examples.

As far as I can tell, with Ns_ParseProc you write less code,
get more options, and it takes advantage of infrastructure
to give you more speed.  I must be missing something, but
you'll have to explain it to me in more detail because I
don't get it.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-25 20:43

Message:
Logged In: YES 
user_id=184124

Okay, the biggest problem with your way i see that i am
enforced to go through http driver, even with my parse proc,
i need to know how it works to implement my driver so
correct hooks/filters will be called. If i want skip some
parts i am not able to do it, http driver works only one way. 
Another issue is if something will not fit into current
driver, additional hook needs to be introduced and core
driver needs to be modified again.

If my driver fits http-like/text mode paradigm, reusing http
driver is the easiest way, but if i need something specific,
all extra step to mimic the connection as http request, so 
all other parts of the http driver will not fail is just
unnecesssary extra efforts.

And i you mentioned, using Ns_DriverSockQueue is low level
function and requires from the developer full attention for
bulding the driver, but if this is what i want, why not. I
can spawn my own thread if some callbacks are long running
Tcl scripts and queue connection from my own loop. If i omly
want to received packets, decode them and submit connection,
i do not need all http driver infrusturcture, why to enforce
using it?

So we can have at least 3 ways for supporting multiprotocols:
- standard callbacks, completely different threads, no
connection pooling (Exists now)

- Ns_DriverSockQueue, new thread or callback thread, reusing
connection pooling. All filters/traces are reused as well

- driver Parse proc in the http driver to reuse drivers
thread and connection pooling with all filters/traces

All 3 methiods can co-exists and do not interfere with each
other, that's my point, and all they add just couple extra
functions. I would implement all methods, i have 3 drivers i
need to run: smtp, dns, snmp, can add imap in the future

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-02-25 20:31

Message:
Logged In: YES 
user_id=87254

"Each way has its own drawbacks..."

That's what I'm asking.  What are the drawbacks for each
approach we've come up with so far?

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-25 15:23

Message:
Logged In: YES 
user_id=184124

I am not arguing with you about your ParseProc driver
extension, it is very usefull and makes driver more flexible.

I just want to add couple of new API functions that will allow
developer to queue connections from any place, that's it.
How drivers will be written and how developer will decide to
handle it is up to each particular developer. But nobody
will be locked up in only one way of doing it.

Each way has its own drawbacks so we have multiple choices
and developer will decide for itself how it should be.

Both our ways do not mutually exclusive.

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-02-25 08:28

Message:
Logged In: YES 
user_id=87254

Ah I see you're right, an extra thread is not created for
each incoming request.  'udpThread' is perhaps not the best
name for the socket callback though... :-)

I still don't see the advantage of using Ns_SockCalback.  A
single thread is created by the server to handle all
registered callbacks, including those from ns_sockcallback.
 At runtime, while this single thread is running tcl code to
handle one callback, your dirver callback cannot run, no new
connections will be accepted, etc.  Why is it better to use
the Ns_SockCallback thread rather than a driver thread?

Maybe I'm reading this wrong, but how do you handle the case
where the request arrives as more than one packet? 
Ns_DriverSockRead() is called from the 'udpThread' socket
callback.  The only return value checked is NS_OK, but
couldn't this also be SOCK_MORE?

How would you handle things like keepalive?  I think you'd
have to reimplement that in the Ns_SockCallback thread.

Re the proxy stuff, Ns_RegisterRequest() and
Ns_RegisterProxyRequest() seem very simillar.  With
Ns_RegisterRequest(), filters are run and you get the choice
of using C or Tcl.  Ns_RegisterProxyRequest() offers no
advantage that I can see -- you still need a complete
Request structure, even if you just ignore it.

The comm driver initialization should probably be changed to
automatically handle unix domain sockets.  You add /foo
rather that 127.0.0.1 in the config file and it knows to
create the correct type of socket.  nssock and nsopenssl
wouldn't have to be modified at all.

UDP is different.  There are two types of protocols: single
packet, such as DNS or RADIUS; multi packet, as used in some
streaming media and p2p protocols.

I think for the single packet case we again might want to
modify the driver code to automatically handle it.  No
read-ahead is necessary, there's only one packet per
request, so it's placed in the request buffer and passed on
to the next stage, which is parsing.  Here, a return code of
SOCK_MORE would be illegal.

Every multi-packet UDP protocol will require custom
framing/sequencing and the developer will have to create a
new socket driver.

Taking RADIUS as an example, which is a single packet UDP
protocol, you'd create a very simple Ns_ParseProc who's only
job is to check that the number of bytes specified in the
header actually arrived, and return SOCK_READY.  A default
request structure is created for you so the only other thing
you have to do is Ns_RegisterRequest() for the '/' URL. 
Within your request proc, call Ns_ConnContent() and parse
the buffer.

Now, you do have a number of other options to make this more
flexible.  You could parse the request in your Ns_ParseProc
and then fill out the request structure.  e.g. the different
RADIUS message types could be expressed as HTTP verbs.  This
buys you flexibility.  Now you can Ns_RegisterRequest() a
different routine for each RADIUS message type, and someone
reusing your code can override your default implementation.
 You can also handle some message types in C, and others in Tcl.

You might decide to put some useful information about the
RADIUS request in the URL line.  Now you get logging for free.

You might decide to parse the key/value pairs from the
RADIUS request into query vars or headers.  Now you don't
have to write a bunch of RADIUS specific Tcl commands to
access the data, should you want to handle that from Tcl. 
ns_queryget or ns_conn headers  will work.

How far you want to go is up to you, the writer of the
protocol module.  But at the low end all you need is an
Ns_ParseProc and usually a single registered request proc,
which is a lot less code to write than the Ns_Driver* stuff,
I think.

I'm sure it's not perfect!  So what's wrong with it? Why
would it suck to write a RADIUS protocol module as I've just
described, to take one example?

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-25 05:51

Message:
Logged In: YES 
user_id=184124

I think we need to collect all solutions and then see where
are going, i am holding to port all my old/running modules
because i do not know how naviserver will handle foreign
protocols.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-25 05:48

Message:
Logged In: YES 
user_id=184124

yes, using proxy can solve immediate requirements without
hacking NsConnProc by adding hooks to call driver specific C
functions.

If i need my smtp server and main loop is in C, i need
somehow call it
in the connection thread. Using registered proxy function i
can do it now, i do not need filter/traces. This is for
completely new modules.

I can implement main loop in the module as Tcl command and
then call it in the connection filter, it is possible, it
will just require many different parts to be in place and
still filter should be registered as Tcl proc which will
call another driver main loop.

sometimes low level stuff makes things easier and simpler:-)))

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-25 05:45

Message:
Logged In: YES 
user_id=184124

In all my installations i need sha module but it wasn't in
the distribution, so i need to download it or repackage
aolserver to include nssha1. That is my point, it is just
one simple function use more often than something like
ns_jpegsize or so.

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-02-25 05:42

Message:
Logged In: YES 
user_id=87254

Well I don't know about the wisdom of adding sha1 to the
core at this stage  :-)

I see your point though, encryption and hashing functions
are almost universally required for systems/server work. 
Maybe we need to consider adding a new encryption module to
the core distribution.  Like nsdb it would export a C API
via libnscypher.so (or whatever) as well as the Tcl module
nscypher.so.

Times have changed and things like the openssl libraries are
common on all platforms so it's not the big deal it used to
be to add such dependiencies.

Such a module should make implementing SSL in nsopenssl easier.

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-25 05:41

Message:
Logged In: YES 
user_id=184124

No, i do not spawn new thread, i use callback feature of the
server, when socket is ready, server calls provided callback
ad that callback just submits the socket to connection pool.
if pool is full, Ns_DriverSockQueue will return NS_TIMEOUT,
you can retry.
Yes, it is low-level, for high level, HTTP driver provides a
lot of functionality, it could be extended like you did, but
still it is HTTP driver hacked. If i want completely new
driver, like RADIUS server, http driver will not help me, i
need low level stuff, and it is there already.

To reuse resource limiting, i added Ns_DriverSockQueue
function, so
new conections can be queue instead of creating new threads.

I do not have anything against your patch for extending
current driver, it is very usefull. i am offering new API
for low level drivers.

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-02-25 05:36

Message:
Logged In: YES 
user_id=87254

The proxy stuff is too late in the cycle to do much.  The
request has to be fully parsed by then (for read-ahead).

If a proxy function is available, then all filters, the auth
phase, registered procs, cleanup procs etc. are bypassed.  A
lot of that stuff can be very useful for non-HTTP protocols.

I don't think handling stuff via a proxy function buys you much.

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-02-25 05:32

Message:
Logged In: YES 
user_id=87254

I don't understand what you're trying to acheive here (well,
apart from multi-protocol... :-).  The newly exposed
Ns_Driver* entry points are quite low level, and so the
implementor of the new protocol is left to do a lot of the
heavy lifting.

For example, the way I read it you have to create your own
listen socket and register a callback.  Every time a new
reaquest comes in a thread is spawned to handle it.  From
that thread you then submit the parsed request to one of the
conn threads.

Excessive thread creation and message passing between
threads is not going to perform well.  And it seems you have
to write more code than e.g. the example POP3 driver I
posted some time ago.

You're also not taking advantage of the other facilities
that the server offers.  What happens if 1000 connections
arrive, do you spawn 1000 threads?  You could of course code
up some limit checks, but this already exists.  What if a
client sends you a continuous stream of data, 2GB...  etc.

By using the driver hooks to provide the new protocol
parser, you deny yourself the opportunity to use something
like the nsopenssl module.  This should work just fine for
protocols like SMTP, IMAP, POP3 and probably others.

Anyway, I think one of the most carefully coded aspects of
the server is it's attention to resource usage.  That goes
for IO, context switching, memory, etc.  It's espescialy
nice that most of the time you're not even aware that all
this work is being done for you.  I'd like new protocol
drivers to be able to transparently take advantage of that.

Could you take a look at my old POP3 demo driver?  It's the
attachment nspopd-0.3.tar.bz2 over here:

http://sourceforge.net/tracker/index.php?func=detail&aid=973010&group_id=3152&atid=353152

It's not obvious that anything interesting is going on, so
it's not much to look at.  But actually, conn socket
read-ahead is happening eficiently in one thread with async
IO, the conn threads are treated as a precious resource
(heavy-weight Tcl interps) and are allocated at the last
minute, there's an easy API in C and Tcl to implement the
actual reading of data from the INBOX (could be from the
file system, db etc.).

You've got a lot of experience writing servers, what do you
think is wrong with this model?  What can it not do, or what
could it do better?

----------------------------------------------------------------------

Comment By: Vlad Seryakov (seryakov)
Date: 2005-02-24 21:29

Message:
Logged In: YES 
user_id=184124

Another thing, once we can submit connections from any
place, no need to build any drivers, even in C, i can
register new proxy proc and set protocol field in my
request, so when submitted, connection will run registered
proxy proc.
for example: 
 in my smtp driver/module, i create driver, register proxy
for smtp: protocol, register callback for the socket. Once
connection accepted, in my module i submit that connection
to the queue with request-protocl set to smtp:.
queue.c will call my proxy handler, which is C function.

No need to add anything else. This way even standard
aolserver can be extended without touching precious http
driver thread. Sorry for sarcasm.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=719009&aid=1151137&group_id=130646

2005	Jan	Feb (61)	Mar (153)	Apr (39)	May (10)	Jun (15)	Jul (15)	Aug (2)	Sep	Oct (17)	Nov (2)	Dec (13)
2006	Jan (18)	Feb (9)	Mar	Apr	May	Jun	Jul (7)	Aug (1)	Sep (2)	Oct	Nov (1)	Dec
2007	Jan (8)	Feb (3)	Mar	Apr	May (2)	Jun	Jul (2)	Aug	Sep	Oct	Nov	Dec
2008	Jan	Feb	Mar	Apr (6)	May (1)	Jun	Jul	Aug	Sep	Oct	Nov	Dec

naviserver-tracker Mailing List for NaviServer (Page 6)

NaviServer, a high performance web server written in C and Tcl

naviserver-tracker — SourceForge Tracker Notifications