freespeech-overflow

[Overflow] DL and PL integration

[Brad C. <cha...@ar...>]

Hey guys;
	As I mentioned to Jarl yesterday, Jean-Marc and I had been
talking a bit about starting to more closely integrate the BL and PL,
especially with respect to returning errors and calculated values in a
node to be displayed. We kind of came up with a two step plan for
starting on this:

1. Write a stream class for the PL which derives from the C++ iostream
class. This class will be responsible for communicating directly with the
BL through a CORBA interface. The idea is that when nodes are running
they can call this stream class like:

returnValueStream << 'This is a node name' << 'This is some value info';

This kind of abstracts the actual communication to this stream class, and
allows you to turn streams on and off.

2. Write the CORBA interface and supporting code so that the stream class
can communicate with the BL. 

I started working towards this second goal and wrote an initial IDL file
for the communication, which I attached. The basic idea is that the BL
implements an interface through which the PL can return errors and
values. The PL implements a callback interface for whenever it returns an
error, so that the BL can either stop or continue a run when an error
occurs.

This is just a first start to work with, and I already have a number of
questions about the design:

1. How will we distinguish between different BL processes submitted to
the PL? There are two ways I could think of to do it:

a. Have a different BlReporter submitted for each run. The PL would then
somehow have to keep track of which BlReporter object is which.

b. Have an id number assigned to each submitted process that would be
given to a generic BlReporter (or we could have a general BL interface
that gave specific BlReporters based on id numbers). In this case the PL
and BL would have to track these id numbers.

2. How can the stream class return the stop or continue messages back to
the node? I'm not sure how we can achieve this kind of callback with the
stream setup, but this is probably just me missing something.

Anyways, this is just something to get started with. I'm excited to see
this start getting hashed out and seeing the parts be more integrated!
Talk with you all soon.

Brad

[Overflow] Re: DL and PL integration

[jarl v. k. <ja...@ca...>]

>

Hi All,

| 2. Write the CORBA interface and supporting code so that the stream class
| can communicate with the BL.

This will change the old plan of linking the PL and BL to one binairy. There's no
real prob as long as we setup a good authentication meganism between the 2 layers.

>
> The basic idea is that the BL
> implements an interface through which the PL can return errors and
> values. The PL implements a callback interface for whenever it returns an
> error, so that the BL can either stop or continue a run when an error
> occurs.

Sound logical.. and I cant thing of any to complain about, so you got my vote :)

>
> b. Have an id number assigned to each submitted process that would be
> given to a generic BlReporter (or we could have a general BL interface
> that gave specific BlReporters based on id numbers). In this case the PL
> and BL would have to track these id numbers.

This we already have, the Universal Resource Id (URI), look @ the DL-BL idl.

>
> 2. How can the stream class return the stop or continue messages back to
> the node? I'm not sure how we can achieve this kind of callback with the
> stream setup, but this is probably just me missing something.

I dont quite see what you're pointing at, can you explain a bit more?

Tnnx 4 the Bl2Pl IDL Brad..  I'll think about it, you'll hear my comments on it soon.

bye,
jarl

Re: [Overflow] Re: DL and PL integration

[Jean-Marc V. <jea...@he...>]

> | 2. Write the CORBA interface and supporting code so that the stream class
> | can communicate with the BL.
> 
> This will change the old plan of linking the PL and BL to one binairy. There's no
> real prob as long as we setup a good authentication meganism between the 2 layers.

I still think linking the PL with the BL is the right thing to do... instead of
using CORBA, we could just use simple callbacks, which would be much simpler and
faster when exchanging large data.

	Jean-Marc

-- 
Jean-Marc Valin
Universite de Sherbrooke - Genie Electrique
va...@ge...

Re: [Overflow] Re: DL and PL integration

[Brad C. <cha...@ar...>]

Jean-Marc wrote:
>  I still think linking the PL with the BL is the right thing to do...
>  instead of using CORBA, we could just use simple callbacks, which
would be
>  much simpler and faster when exchanging large data.

The thing I really want is to be able to write nodes that work in Piper
in languages other than C++. For instance, the AlBeanBox and ISYS nodes
are written in java -- I think it would be really nice if we could plug
these into the same node framework. In general, it would be nice if
people had a number of language choices for writing node extensions in.

I will freely admit that I don't really understand how this will work,
but I really want to work towards it. I see having a CORBA layer for
these callbacks will at least be a small step towards this goal.

What do you guys think? Am I being crazy here as well? Is there any way
we are going to be able to do this?

BTW, would it be helpful if we had an IRC session to talk about this and
try to iron some things out? I believe Jarl is 6 hours behind Jean-Marc
and I, so if we want to try this, maybe we should try to for sometime
between 4pm and 6pm (Jean-Marc and my time) someday next week. What do
you guys think? I'm pretty open during these times next week (although
Wednesday is the worst, but I can rearrange), so it anyone wants to
propose a time to do talk, I would be very up for it. I just think it
might help us hash through some stuff a little faster and is easier than
e-mail sometimes...

Brad

Re: [Overflow] Re: DL and PL integration

[Jean-Marc V. <jea...@he...>]

First, for Jarl, I would suggest that you subscribe to the Overflow mailing list
by going to 
http://lists.sourceforge.net/mailman/listinfo/freespeech-overflow
It's fairly low-trafic, since there's only Brad and me so far!

> Jean-Marc wrote:
> >  I still think linking the PL with the BL is the right thing to do...
> >  instead of using CORBA, we could just use simple callbacks, which
> would be
> >  much simpler and faster when exchanging large data.
> 
> The thing I really want is to be able to write nodes that work in Piper
> in languages other than C++. For instance, the AlBeanBox and ISYS nodes
> are written in java -- I think it would be really nice if we could plug
> these into the same node framework. In general, it would be nice if
> people had a number of language choices for writing node extensions in.

I think there are other ways to achieve Overflow 'language bindings'. Note that
using CORBA for BL-PL communication doesn't help in that respect. What's
important is that Overflow C++ nodes can communicate with nodes written in other
languages. I think that the best way to achieve that is to write a special node
for every language we want to support. That node would just be an empty node
that would call the 'foreign' node, either with a function call, as a process,
with CORBA, ... that is left to the one implementing the language binding. As
the the PL itself, it could still be linked in the BL. I really see the BL as an
executable that runs on each machine and that links with the Overflow libraries
to do the local processing.

> BTW, would it be helpful if we had an IRC session to talk about this and
> try to iron some things out? I believe Jarl is 6 hours behind Jean-Marc
> and I, so if we want to try this, maybe we should try to for sometime
> between 4pm and 6pm (Jean-Marc and my time) someday next week. What do
> you guys think? I'm pretty open during these times next week (although
> Wednesday is the worst, but I can rearrange), so it anyone wants to
> propose a time to do talk, I would be very up for it. I just think it
> might help us hash through some stuff a little faster and is easier than
> e-mail sometimes...

So far, monday and tuesday are fine with me.

	Jean-Marc

-- 
Jean-Marc Valin
Universite de Sherbrooke - Genie Electrique
va...@ge...

[Overflow] BL thoughts

[Jean-Marc V. <jea...@he...>]

Hi,

I've been thinking about how to integrate the BL with the PL and here are my
thoughts on that for now.

I'm assuming the BL will be split in a "master" process, running where the UI is
running, and a bunch of "slave" processes on remote (and local) machines. I
think the master could easiliy be linked with the DL, so communications would be
easy (function calls). 

As for the "slaves", I see them linking with Overflow. That way, the BL slave
would call getOutput() on the PL sub-network, get the result and send that to
another BL process. Since the BL would link with the overflow (PL) libraries,
communications would be made by functions/callbacks. For instance, the first
node of a distributed section would call a function in the BL and ask for its
incoming data. In the same way, for probes and viewers, we would have a node
that would call a BL function (either directly or through a method from a new
iostream). Here's with an example how I see the whole thing.

We have the following processing:
A->B->C->D   (each letter is a Node performing a certain operation)
Suppose the user (or the system) decides to break-up the processing in three
parts:
A -> (B->C) -> D

The BL will spawn 3 processes (we don't care where they run for now). Now, let's
consider the second process doing B->C, the BL will construct an Overflow
network:

N->B->C , where N is a special BL communication node.
When the second BL is asked to compute its result (remember that we're going
backwards, using a pull), it will call getOutput() on node C. In order to
perform its calculation, node C will call getOutput() on node B, which will
getOutput() on node N. Node N will then call a BL function that will ask for its
input. The BL function will ask (CORBA/IPC/socket/...) the first BL and return
the answer it got. Node N will then return the result to node B, which will
calculate something and return to node C, and at the end, the second BL will
have the answer. It will send that answer to the third BL in the same way that
the first BL sent it to the second. (is it clear or have I lost everybody?). If
we suppose that each of the "slave" BL was running on a different machine, we
have that each machine only needs one process (or thread) running. The only
inter-process communication will happen between different BL parts, so both the
PL and DL will not have to care about that. Does that make sense? 

Now let's say that in our precious example node C was a viewer. When C gets the
result from B, it will tell (with a function call) its BL to display some data.
Its BL slave will send that to its BL "master", which will send it (function
call) to the DL, which will know what to do with it: either send it to the GUI
or log it in a file for a batch processing (or even discard it when the user
doesn't want to see it).

What do you think of that?

	Jean-Marc

-- 
Jean-Marc Valin
Universite de Sherbrooke - Genie Electrique
va...@ge...

[Overflow] Re: BL thoughts

[Jean-Marc V. <jea...@he...>]

> Hmm... not really :) I was thinking about the node that will be the glue between
> instances of BL's. Building the glue between BL and PL will be a quite similair task.
> I'm currently stuck with porting the C Corba code to C++. I asked Brad to have a look
> and I ordered a book about this subject.

I don't understand what kind of "communication node" you're talking about. Is an
IRC chat at the Brad suggested OK for you (4pm EST, 10pm for you?)?

	Jean-Marc

-- 
Jean-Marc Valin
Universite de Sherbrooke - Genie Electrique
va...@ge...

[Overflow] Re: DL and PL integration

[Brad C. <cha...@ar...>]

jarl wrote:
>  This will change the old plan of linking the PL and BL to one binairy.
>  There's no real prob as long as we setup a good authentication
meganism
>   between the 2 layers.

Well, basically I guess I was just thinking about the CORBA layer only
being sort of between node communication from the PL to the BL. The
reason I think this is a good idea, is so that nodes can be written in
languages other than C++. I am really into this, since I think it will
help us use nodes from other projects (like ISYS and ALBeanBox) although
I'm not sure how yet :-).


[having id numbers that the BL would have to keep track of]
>  This we already have, the Universal Resource Id (URI), look @ the
>  DL-BL idl.

Right, although I'm not sure I understand how this will hook in with the
way Overflow uses unique 'names' as id for nodes. Have you already coded
this in the BL? Sorry, I am waaaaay behind on where the BL code is at
:-<.
 
I wrote:
>  > 2. How can the stream class return the stop or continue messages
> >  back to the node? I'm not sure how we can achieve this kind of
callback
> >  with the stream setup, but this is probably just me missing
something.

Jarl wrote:
>  I dont quite see what you're pointing at, can you explain a bit more?

Well, what I was thinking of is this:

-> The node registers some kind of error. It pauses the run because of
the error and then calls the error stream.

errorStream << 1 << 'some kind of error';

-> The error stream recieves the error. It creates a PlErrorCallback and
calls the BL with an error message.

-> The BL recieves the error message, passing it on to the DL and then
the UI, where the user decides to continue the run.

-> The BL gets this message and calls runContinue() on the
PlErrorCallback.

-> Now my big question mark -- how does this info get back to the node so
it knows to keep processing getOutputs()? Am I misunderstanding the
design here?

Brad

Does this make sense?