middle-man-developers Mailing List for Middleman filtering proxy server (Page 3)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

On Monday 12 January 2004 08:10 am, Riadh Elloumi wrote:
> > When do you think we should do the next release? There's been a
> > _huge_ number of changes since 1.9.1, including a few pretty
> > serious bugs that some people have complained about,
> > so I'd like to get it out as soon as possible.. then we can
> > benefit from many people testing it.
>
> Yes, fully agree with you. But before the release, we must update the
> documentation to be suitable with the changes in the web interface. I'll do
> this and try to make it more structured than a simple HTML page.

Nice, the documentation is about due for an overhaul. I _hate_ writing documentation, I'm glad you don't feel the same way :)

> I also
> planned to integrate the ICAP client on the proxy, but it seem to be more
> much work than I expected. The problem is that the code processing the
> requests and the responses in main.c and protocol.c is not enough clear and
> structured. My idea is to make independent modules called services like
> rewrite, keyword, etc and ICAP can be a service among these services. We
> can also break the code into shared modules (*.so) that we can load and
> unload on demand... I think this will take a big work and may not be ready
> for the next release, so let's make the 2.0 !

I gave this some though before, one thing we have working in our favor is there isn't really any inter-dependence between most of the features.
We can add hooks throughout the code that modules can register to be called at, some of the hook locations I think we'll need are:

- after client connecting (for access)
- after client request header, after parsing first time but before second parsing (for header rewrite)
- after client request header, after second parsing (for header filtering)
(in case you're wondering, parsing of the header is done twice to make url commands work with the header rewrite feature)
- before connecting (for forwarding and redirecting)
- after connecting (this is where control would be passed onto the module handling the protocol, if we decide to make that a module too).
- after receiving server header, before parsing (for header rewrite) (done inside the protocol module)
- after receiving server header, after parsing (for header filtering) (done inside the protocol module)
- before transfering (to determine if it needs to be buffered; for rewrite, keywords, external, et al.) (done inside protocol module)
- while transfering, from socket READEVENT callback (for prefetching)
- after transfering if it was buffered (post processing; for rewrite, keywords, external, prefetching, et al.). (done inside protocol module)
- after transfer completes (transfer limiting)

We'd also need a way for the modules to select which order they're called in (for example, before connecting redirecting has to happen before forwarding)

We'd also have to reserve different return values from the callbacks to determine what to do, for example:

the 'after client connecting' hook, to determine if the client should be allowed to continue.
the 'before connecting' hook', to determine if we should continue or not (redirect_do may send a 302 redirect, and skip to the next client request).
the 'before transfering' hook, to determine if the file should be buffered.

Before we even start this though (if we indeed decide to go this way), I'd like to have a rock solid plan on everything that needs to be done, I don't want to get halfway
through converting everything into modules and realise there's something we overlooked in the design that we can't do (cleanly anyways).

Anyhow, that's definitely post-2.0; I'm pretty happy with the way things are ATM.. I just want to get it cleaned up and out the door :)

>
> > Things seem to be pretty stable for me, I've been running some
> > stress tests with siege and haven't had any problems.
> > One thing I was concerned about was memory leaks, but after
> > running a stress test for about 8 hours (left it on while I
> > slept) memory usage topped off at 80mb
> > (I'm not sure why the hell it's growing that big, but it didn't
> > get any bigger). Maybe we should put part of your malloc
> > debugging code back in, the part that
> > frees up the memory when a SIGHUP is received.. then we can use
> > valgrind to see if any leaks exist.
>
> Ok, I'll do it. I have never heared before about Siege, I'll try it in my
> tests!
>
> Here is my TODO list
>
> - before I forget: replace regexp by ip range lists in access section

I forgot about that too :) 

Did you see the other things I put in the TODO file (the statistics gathering and per-user config files)? 
Do you think it's a good idea? I poke around mailing lists of other proxy servers sometimes looking for ideas (hehe!) and 
cookie caching is a popular request.

Oh, and multilingual support is a good idea; unfortunately english is my only language (and I'm pretty bad at that), so I'll have to leave that to your capable hands :)
I'll do any coding work it involves though.

> - remove any negative file size
> - update the documentation
> - integrate the memory leak detection code
>
> I'm happy so much improvements due to our team work!

Me too :) developement has never moved this fast.

G'day,
Jason McLaughiln

-

Oh drat these computers, they're so naughty and so complex - I could pinch them!
- Marvin the Martian

middle-man-developers Mailing List for Middleman filtering proxy server (Page 3)

middle-man-developers — Middleman developers mailing list