Menu
▾
▴
naviserver-devel
|
From: Vlad S. <vl...@cr...> - 2008-08-29 01:05:13
|
> + * tests/http_byteranges.test: Add a couple of tests for ranges on > + larger files to exercise the file descriptor path rather than the > + in memory mmap or cache path. Unfortunately, this actually > + triggers the writer thread and this has no support of ranges, > + which I guess is where it's actually most needed. TODO... > + I really like last changes, at the same time i need writer support because my tests showed that serving big files with conn threads are waste of resources. At the same time, being able to support range in the writer make sense. -- Vlad Seryakov vl...@cr... http://www.crystalballinc.com/vlad/ |
|
From: Vlad S. <vl...@cr...> - 2008-09-11 21:57:23
|
will do, it may take a while Stephen Deasey wrote: > On Fri, Aug 29, 2008 at 2:12 AM, Vlad Seryakov <vl...@cr...> wrote: >> And sent email without finishing the thought :-)) >> >> Maybe parsing for range should done early enough and saved in the >> request, so whoever then will be sending data will use it. >> >> Just a thought from east coast :-)) >> > > > Here's a couple of patches as a follow on to recent checkins to cvs: > > registerdriver.diff > setlength.diff > range-sendfile.diff > > The first changes the prototype of the Driver callback. Actually it > removes it and adds a callback for each function: send, receive, keep, > close, and adds a new one: sendfile. > > The second is just a clarification on the handling of length headers, > but it's needed for the third. > > The third changes the range code to use the new sendfile support. I > changed it round so that instead of saying SendRange you say > ParseRange. You pass it an array of Ns_FileVec's, it looks at the > headers and it fills it in for you. I haven't tried, but this should > allow you to allocate an array of Ns_FileVec's and associate it with > the socket for the writer thread. We might have to also fixup the > timeout handling in the Ns_Sock* stuff -- It seems to always want to > do a poll timeout the way it's currently written. > > > I didn't want to check any of this in because the API change will > break half a dozen of your driver modules. So whenever you get a > chance (no rush) try out the registerdriver.diff patch and see if you > can get your modules working with it. > > I did notice a couple of odd things. NsDriverClose() get called twice > for each socket. Not sure why, but I don't think it should. I've > changed the semantics: it used to be "Notify the driver that we are > going to close", now we delegate the close to the driver, which for > nssock is simply ns_sockclose(). The advantage is that for other > modules, like your nsudp, you should be able to handle whatever needs > to be handled. For example nsudp wants to keep the socket open. > Previously this was handled with special logic in the driver thread > checking for if UDP, else !UDP. I think this complication is where > the DriverClose x2 bug crept in. > > I haven't thoroughly checked it, but I'm 95% confident that you could > handle all the special cases for the current alternative drivers more > cleanly by delegating to the driver callbacks and removing all the > special logic to do with NS_DRIVER_* (UDP, UNIX, QUEUE_ONACCEPT, > QUEUE_ONREAD). Have another look and see what you think. > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > > ------------------------------------------------------------------------ > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
|
From: Stephen D. <sd...@gm...> - 2008-09-11 22:06:31
|
On Thu, Sep 11, 2008 at 10:53 PM, Vlad Seryakov <vl...@cr...> wrote: > will do, it may take a while > no problem |
|
From: Vlad S. <vl...@cr...> - 2008-08-29 01:12:43
|
And sent email without finishing the thought :-)) Maybe parsing for range should done early enough and saved in the request, so whoever then will be sending data will use it. Just a thought from east coast :-)) Vlad Seryakov wrote: >> + * tests/http_byteranges.test: Add a couple of tests for ranges on >> + larger files to exercise the file descriptor path rather than the >> + in memory mmap or cache path. Unfortunately, this actually >> + triggers the writer thread and this has no support of ranges, >> + which I guess is where it's actually most needed. TODO... >> + > > I really like last changes, at the same time i need writer support > because my tests showed that serving big files with conn threads are > waste of resources. > At the same time, being able to support range in the writer make sense. > -- Vlad Seryakov vl...@cr... http://www.crystalballinc.com/vlad/ |
|
From: Stephen D. <sd...@gm...> - 2008-08-29 01:37:44
|
On Fri, Aug 29, 2008 at 2:12 AM, Vlad Seryakov <vl...@cr...> wrote:
>>> + * tests/http_byteranges.test: Add a couple of tests for ranges on
>>> + larger files to exercise the file descriptor path rather than the
>>> + in memory mmap or cache path. Unfortunately, this actually
>>> + triggers the writer thread and this has no support of ranges,
>>> + which I guess is where it's actually most needed. TODO...
>>> +
>>
>> I really like last changes, at the same time i need writer support
>> because my tests showed that serving big files with conn threads are
>> waste of resources.
>> At the same time, being able to support range in the writer make sense.
>>
>> And sent email without finishing the thought :-))
>
> Maybe parsing for range should done early enough and saved in the
> request, so whoever then will be sending data will use it.
>
> Just a thought from east coast :-))
I was thinking of adding support for sendfile().
It's different on every platform, so having looked at a few, I was
going to make our interface follow the Solaris senfilev() model which
basically allows an arbitrary number of interleaved memory and file
ranges. It's an almost exact fit for our byterange code :-)
Here's the man page:
http://docs.sun.com/app/docs/doc/816-5172/sendfilev-3ext?a=view
So we'd end up with an array of those sendfilev_t-like structures
describing the data to send. Not sure if they should go in the
conn/request, or... But maybe it should be a public interface,
unlike the Range struct which is kind of a private thing atm.
|
|
From: Stephen D. <sd...@gm...> - 2008-09-11 21:52:35
|
On Fri, Aug 29, 2008 at 2:12 AM, Vlad Seryakov <vl...@cr...> wrote: > And sent email without finishing the thought :-)) > > Maybe parsing for range should done early enough and saved in the > request, so whoever then will be sending data will use it. > > Just a thought from east coast :-)) > Here's a couple of patches as a follow on to recent checkins to cvs: registerdriver.diff setlength.diff range-sendfile.diff The first changes the prototype of the Driver callback. Actually it removes it and adds a callback for each function: send, receive, keep, close, and adds a new one: sendfile. The second is just a clarification on the handling of length headers, but it's needed for the third. The third changes the range code to use the new sendfile support. I changed it round so that instead of saying SendRange you say ParseRange. You pass it an array of Ns_FileVec's, it looks at the headers and it fills it in for you. I haven't tried, but this should allow you to allocate an array of Ns_FileVec's and associate it with the socket for the writer thread. We might have to also fixup the timeout handling in the Ns_Sock* stuff -- It seems to always want to do a poll timeout the way it's currently written. I didn't want to check any of this in because the API change will break half a dozen of your driver modules. So whenever you get a chance (no rush) try out the registerdriver.diff patch and see if you can get your modules working with it. I did notice a couple of odd things. NsDriverClose() get called twice for each socket. Not sure why, but I don't think it should. I've changed the semantics: it used to be "Notify the driver that we are going to close", now we delegate the close to the driver, which for nssock is simply ns_sockclose(). The advantage is that for other modules, like your nsudp, you should be able to handle whatever needs to be handled. For example nsudp wants to keep the socket open. Previously this was handled with special logic in the driver thread checking for if UDP, else !UDP. I think this complication is where the DriverClose x2 bug crept in. I haven't thoroughly checked it, but I'm 95% confident that you could handle all the special cases for the current alternative drivers more cleanly by delegating to the driver callbacks and removing all the special logic to do with NS_DRIVER_* (UDP, UNIX, QUEUE_ONACCEPT, QUEUE_ONREAD). Have another look and see what you think. |
|
From: Vlad S. <vl...@cr...> - 2008-09-12 02:07:53
|
I agree about making modules take care as much as possible. But here are the questions i need to resolve: - init, socket needs to be created, UDP or TCP or Unix, so driver needs initProc - for module like UDP, driver needs to know not to accept or we need another proc called acceptProc, default will be using accept and treat socket as TCP but drier can ovwerwrite with its own - queuing, at some point, after accept or later driver need to queue socket. The reason behind QUEUE_ON_ACCEPT to take care basic stuff without writing same code for creating separate thread for slitening and accepting sockets. SMTP is a good example in nssmtp. QUEUE_ONREAD is usefull in case of UDP sockets, after reading UDP packet call registered proc So, i need to resolve these issues and i would like to move as much as possible into drivers. Any suggestions? Stephen Deasey wrote: > On Fri, Aug 29, 2008 at 2:12 AM, Vlad Seryakov <vl...@cr...> wrote: >> And sent email without finishing the thought :-)) >> >> Maybe parsing for range should done early enough and saved in the >> request, so whoever then will be sending data will use it. >> >> Just a thought from east coast :-)) >> > > > Here's a couple of patches as a follow on to recent checkins to cvs: > > registerdriver.diff > setlength.diff > range-sendfile.diff > > The first changes the prototype of the Driver callback. Actually it > removes it and adds a callback for each function: send, receive, keep, > close, and adds a new one: sendfile. > > The second is just a clarification on the handling of length headers, > but it's needed for the third. > > The third changes the range code to use the new sendfile support. I > changed it round so that instead of saying SendRange you say > ParseRange. You pass it an array of Ns_FileVec's, it looks at the > headers and it fills it in for you. I haven't tried, but this should > allow you to allocate an array of Ns_FileVec's and associate it with > the socket for the writer thread. We might have to also fixup the > timeout handling in the Ns_Sock* stuff -- It seems to always want to > do a poll timeout the way it's currently written. > > > I didn't want to check any of this in because the API change will > break half a dozen of your driver modules. So whenever you get a > chance (no rush) try out the registerdriver.diff patch and see if you > can get your modules working with it. > > I did notice a couple of odd things. NsDriverClose() get called twice > for each socket. Not sure why, but I don't think it should. I've > changed the semantics: it used to be "Notify the driver that we are > going to close", now we delegate the close to the driver, which for > nssock is simply ns_sockclose(). The advantage is that for other > modules, like your nsudp, you should be able to handle whatever needs > to be handled. For example nsudp wants to keep the socket open. > Previously this was handled with special logic in the driver thread > checking for if UDP, else !UDP. I think this complication is where > the DriverClose x2 bug crept in. > > I haven't thoroughly checked it, but I'm 95% confident that you could > handle all the special cases for the current alternative drivers more > cleanly by delegating to the driver callbacks and removing all the > special logic to do with NS_DRIVER_* (UDP, UNIX, QUEUE_ONACCEPT, > QUEUE_ONREAD). Have another look and see what you think. > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > > ------------------------------------------------------------------------ > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel -- Vlad Seryakov vl...@cr... http://www.crystalballinc.com/vlad/ |
|
From: Stephen D. <sd...@gm...> - 2008-09-14 20:46:06
|
On Fri, Sep 12, 2008 at 3:08 AM, Vlad Seryakov <vl...@cr...> wrote: > I agree about making modules take care as much as possible. > > But here are the questions i need to resolve: > > - init, socket needs to be created, UDP or TCP or Unix, so driver needs > initProc > > - for module like UDP, driver needs to know not to accept or we need > another proc called acceptProc, default will be using accept and treat > socket as TCP but drier can ovwerwrite with its own > > - queuing, at some point, after accept or later driver need to queue > socket. The reason behind QUEUE_ON_ACCEPT to take care basic stuff > without writing same code for creating separate thread for slitening and > accepting sockets. SMTP is a good example in nssmtp. > QUEUE_ONREAD is usefull in case of UDP sockets, after reading UDP packet > call registered proc > > So, i need to resolve these issues and i would like to move as much as > possible into drivers. > > Any suggestions? Yeah, create listen and accept callbacks. I think TCP drivers should work fine -- I did that ages ago with the example pop3 driver. The UDP though... Maybe they should just read the packet in the accept callback, stash it, fake-up the HTTP request line in the Read callback, and the conn thread can then access the stashed packet directly. I'll have another go at it later, but here's some cleaned and split patches anyway, if you get a chance. |
|
From: Vlad S. <vl...@cr...> - 2008-09-14 20:56:34
|
You can go ahead and patch CVS directly, i agree with the approach and all other modules i will port one by one. Not all i use now, mostly UDP and Syslog, so it is not a big deal. I though as well about reading in acceptProc and queue, it just needs public Queue function and the rest can be done inside each module. Stephen Deasey wrote: > On Fri, Sep 12, 2008 at 3:08 AM, Vlad Seryakov <vl...@cr...> wrote: >> I agree about making modules take care as much as possible. >> >> But here are the questions i need to resolve: >> >> - init, socket needs to be created, UDP or TCP or Unix, so driver needs >> initProc >> >> - for module like UDP, driver needs to know not to accept or we need >> another proc called acceptProc, default will be using accept and treat >> socket as TCP but drier can ovwerwrite with its own >> >> - queuing, at some point, after accept or later driver need to queue >> socket. The reason behind QUEUE_ON_ACCEPT to take care basic stuff >> without writing same code for creating separate thread for slitening and >> accepting sockets. SMTP is a good example in nssmtp. >> QUEUE_ONREAD is usefull in case of UDP sockets, after reading UDP packet >> call registered proc >> >> So, i need to resolve these issues and i would like to move as much as >> possible into drivers. >> >> Any suggestions? > > > Yeah, create listen and accept callbacks. > > I think TCP drivers should work fine -- I did that ages ago with the > example pop3 driver. > > The UDP though... Maybe they should just read the packet in the > accept callback, stash it, fake-up the HTTP request line in the Read > callback, and the conn thread can then access the stashed packet > directly. > > I'll have another go at it later, but here's some cleaned and split > patches anyway, if you get a chance. > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > > ------------------------------------------------------------------------ > > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel -- Vlad Seryakov vl...@cr... http://www.crystalballinc.com/vlad/ |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X