pypes-user

[Eric G. <eg...@py...>]

Thanks Michael,

Don't be too surprised by the similarities because I've been a fan a  
Kamaelia for some time now. I really like the style of messaging  
passing so many of my ideas came from Kamaelia. Although there are  
similarities, I don't think Pypes is quite as ambitious. Pypes started  
out as a way of feeding huge volumes of data to a search index. It was  
originally linear and I was using function composition to chain  
together the new style generators in Python (PEP 342).

Although this worked well, it made it a bit tough to setup at runtime  
(thinking in terms of UI work). I also wanted the ability to publish  
to multiple locations and/or branch based on certain attributes in a  
packet. I turned to Stackless because I think it's a really cool. I  
love seeing these sort of old school concepts resurrected (like Flow- 
Based Programming).

Too many of us become complacent with "what is" and just assume that  
it's the best (or only) way to do things. Both Stackless (coroutines)  
and Flow-Based programming challenge some very fundamental concepts in  
computer science. More importantly, they seem to be ideal concepts  
that are quite applicable to the sort of problems that we're trying to  
solve with these frameworks.

Quite honestly I see pypes (Visual Design Studio) as more of an ETL  
style framework and for this reason, I haven't addressed the issue of  
cycles. I wanted to keep things as simple as possible because lots of  
complexity leads to the potential for more bugs. With this ETL  
mindset, performance is a priority. A typical installation of pypes  
(VDS) in an enterprise setting is 3 quad Xeon machines all processing  
content in parallel. It's not uncommon to push 15 million documents  
through the system.

If something fails after 3 million documents, we have to go back and  
start the feed over again which can really start to eat up time as  
well as the customer's patience. For this reason simplicity is also a  
priority because it allows us to minimize the possibility for errors.

The UI work was really inspired by Yahoo Pipes. I hadn't done any Ajax  
or Javascript coding prior to this and I would have never thought this  
type of UI was possible had it not been for Yahoo Pipes. The idea of a  
web UI was really cool because I had been writing an HTTP service  
layer on top of pypes that exposed a REST API where external  
applications could inject content into the system by issuing HTTP POST  
commands. This style of processing is easy to scale out using hardware  
load balancing.

The fact that I was able to build the UI as a web application still  
baffles me. Best of all, it really cuts down on dependencies. Visual  
Design Studio is pure Python with no C-extensions so it's simple to  
install. Of course, C-extensions can be shipped/built separately to  
address specific performance concerns. Right now I'm using Elementtree  
which is shipped with Python 2.6 and it's blazing fast (about twice as  
fast as libxml -- yes I deal with a LOT of XML content).

I have some Bayesian and Fisher classifiers as well as tools for  
creating decision trees since I deal with a lot of taxonomy mapping.  
Information Extraction is another focal point.

If pypes gains a following I suspect we'll see some RSS mashup tools  
surface. I've been in a few organizations that would love to leverage  
Yahoo Pipes but are afraid because there's no SLA. Yahoo Pipes also  
doesn't allow you to write custom components and both these problems  
are address in pypes.

At any rate, thanks for taking the time to check out pypes and provide  
some feedback. I could definitely see some collaboration in our  
future. Will you be at PyCon this year by any chance? I thought I saw  
a tweet stating that it wasn't looking too promising.

-Eric