NeuroGrid - P2P Bookmark Organiser

Hi Josip,

Thanks for the comments.  I think you are probably right, but the latest 
code has done away with the persistence engines completely.  

I made a lot of progress with torque, and so the latest release of the 
ng-core works with torque as the persistance layer rather than with the 
PersistenceEngine code I developed for NGX.

I haven't checked the new code into cvs, partly because I'm not sure how 
much of the torque stuff in there as well.

Anyways grab the ng-core release and see what I mean.

CHEERS> SAM

Josip Almasi wrote:

>Hi!
>
>Just had a look at the code.
>Comment from KeywordPersistenceEngine:
>
>  // not sure about all this - seems like we are going to want to be
>able to detect the kind
>  // of object that is being persisted and act accordingly rather than
>having to write out a separate
>  // class for each object - not sure ...
>  
>  // could have switch based on object type -- call other particular
>methods, or indeed classes based on class ...
>  //???
>
>IMHO right.
>PersistenceEngine could be one (maybe abstract) class. Any switches
>needed can be optimised using some static final boolean attributes, i.e.
>if(ONLYONE && x_result.size() > 1)
>{
>  throw new DBException("Two entries for this
>"+x_result[0].getClass().getName()+" in persistent storage");
>}
>(DBException should extend RuntimeException)
>
>Vectors and arrays make the code unreadable. Introducing Maps instead
>should improve it, but may require writing a brand new Map
>implementation and hacking into torque BaseClass etc.
>
>Anyway I'd like to do it. Should I explore further in this direction?
>
>My cvs seems to work now... I don't know about my timing, but summer
>vacation is generally good time for some teamwork:)
>
>btw:
>  public static PersistenceEngine getPersistenceEngine()
>  {
>    if(o_created == false)	
>    {
>      o_persistence_engine = new UserDBPersistenceEngine();
>      o_created = true;
>    }
>    return o_persistence_engine;
>  }
>
>can be written as
>
>private static o_persistence_engine PersistenceEngine = new
>UserDBPersistenceEngine();
>public static PersistenceEngine getPersistenceEngine() {
>  return o_persistence_engine;
>}
>
>(or something alike, don't remember exactly)
>
>Regards...
>
>
>
>_______________________________________________________________
>
>Don't miss the 2002 Sprint PCS Application Developer's Conference
>August 25-28 in Las Vegas - http://devcon.sprintpcs.com/adp/index.cfm?source=osdntextlink
>
>_______________________________________________
>neurogrid-devel mailing list
>neurogrid-devel@...
>https://lists.sourceforge.net/lists/listinfo/neurogrid-devel
>
>
>  
>

Josip Almasi wrote:

>Sam Joseph wrote:
>  
>
>>Thanks for the comments.  I think you are probably right, but the latest
>>code has done away with the persistence engines completely.
>>
>>Anyways grab the ng-core release and see what I mean.
>>    
>>
>
>I did, just looked at wrong place:)
>So persistence is done. What's next?
>
Well right now I'm writing a torque adaptor for Mckoi so that you can 
have download a package with built in database, and then I really need 
to get the transport layer abstraction sorted.  It will still use HTTP 
initially, but it should become easy to change that once we have the 
right abstraction.

Other than that I've got Jetty (another servlet engine) running and I'm 
going to try and get that to work as an HTTP proxy so that NG can pick 
up more information about how users use nonNG search engines.  And then 
there's the interface to redesign, but at least that's in a documented 
markup language now (Velocity) so if I can tempt back any of the 
interface designers they should have an easier time.

CHEERS> SAM

Sam Joseph wrote:
> 
> Well right now I'm writing a torque adaptor for Mckoi so that you can
> have download a package with built in database, and then I really need
> to get the transport layer abstraction sorted.  It will still use HTTP
> initially, but it should become easy to change that once we have the
> right abstraction.

Sounds good.

> Other than that I've got Jetty (another servlet engine) running and I'm
> going to try and get that to work as an HTTP proxy so that NG can pick
> up more information about how users use nonNG search engines.  And then

Sounds even better. Do you plan to use it as caching proxy?

Well, I had a brief look at torque-ng now.
In NeuroGridTestSuite.init(), locations of files are hardcoded i.e.
Torque.init("./Torque.properties");
It's fine when started from propmt, but I'm just getting used to jEdit
AntFarm:)
And it may be a good idea to run tests from servlet container...
I usually specify locations of properties relative to a class file, and
here's how I get a class location:
public static URL getLocation( Object obj ) {
  return
obj.getClass().getProtectionDomain().getCodeSource().getLocation();
}
->
Torque.init(getLocation(this)+"../../Torque.properties");
or something alike.

And, NeuroGridSearch seems nice:) I could use some more comments, but I
need to learn more about torque before asking more questions.
Good work!

Regards...

Josip Almasi wrote:

>>Other than that I've got Jetty (another servlet engine) running and I'm
>>going to try and get that to work as an HTTP proxy so that NG can pick
>>up more information about how users use nonNG search engines.  And then
>>    
>>
>
>Sounds even better. Do you plan to use it as caching proxy?
>
Not particularly.  I mean most browsers do automatic caching already. 
 Although it would be useful to be able to control which pages were 
cached, i.e. to store some cached copies for longer - perhaps the ones 
that are bookmarked.  I've always set up the URI data model to have a 
local copy reference.

Assuming that we can get jetty to act as a proxy then there are a lot of 
possibilities, but I'm not even sure if it can be done.  I'm gonna give 
it a go and let you know.

>Well, I had a brief look at torque-ng now.
>In NeuroGridTestSuite.init(), locations of files are hardcoded i.e.
>Torque.init("./Torque.properties");
>It's fine when started from propmt, but I'm just getting used to jEdit
>AntFarm:)
>And it may be a good idea to run tests from servlet container...
>
Sure, although part of the point of this release is that it is not 
dependent on servlets at all

>I usually specify locations of properties relative to a class file, and
>here's how I get a class location:
>public static URL getLocation( Object obj ) {
>  return
>obj.getClass().getProtectionDomain().getCodeSource().getLocation();
>}
>->
>Torque.init(getLocation(this)+"../../Torque.properties");
>or something alike.
>
Sounds good - if I put the code in CVS, you wanna implement this for me? 
 I'd really like to get the Mckoi version out by the end of this week, 
and the changes you suggest are good ones.

Anyway, I'll put the code in CVS.

>And, NeuroGridSearch seems nice:) I could use some more comments,
>
More comments - right - will give that a go

> but I
>need to learn more about torque before asking more questions.
>Good work!
>
Thanks.

Torque has some reasonable documentation .  Part 4 "writing a sample 
application" of the tutorial is a good place to start:

http://jakarta.apache.org/turbine/torque/tutorial.html

There are also a couple of how-tos with more reference oriented material:

http://jakarta.apache.org/turbine/torque/peers-howto.html
http://jakarta.apache.org/turbine/torque/criteria-howto.html

CHEERS> SAM

Sam Joseph wrote:

> Anyway, I'll put the code in CVS. 

Code is now in CVS - you can grab it using:

cvs 
-d:pserver:anonymous@... login

press return at the login

cvs -z3 
-d:pserver:anonymous@... co 
neurogrid/torque-ng

follow the readme in the torque-ng dir to build everything.  I've only 
included the templates for postgresql, mysql and mckoisql (not 
completely finished), but I can add others if people ask, and all the 
templates are in the sourceforge release at:

http://sourceforge.net/project/showfiles.php?group_id=26907&release_id=94107

CHEERS> SAM

Sam Joseph wrote:
> 
> Not particularly.  I mean most browsers do automatic caching already.
>  Although it would be useful to be able to control which pages were
> cached, i.e. to store some cached copies for longer - perhaps the ones
> that are bookmarked.  I've always set up the URI data model to have a
> local copy reference.

I'm thinking of something for some time now... like: you ask your node
about something, and get some url. You click on url and get the data.
Then, I ask the same node the same thing and get the same url. If we're
in the same building/city/continent and document resides outside, we
will save some bandwidth if I get the data from proxy. And more
important, I can get the data even if ddos attack destroys
root-servers.net.
Actually I'm fascinated with possibilites of information travelling
through the net where needed:)

> Assuming that we can get jetty to act as a proxy then there are a lot of
> possibilities, but I'm not even sure if it can be done.  I'm gonna give
> it a go and let you know.

Hm afaik http proxy is not a big deal, at least it's well documented.
And about caching proxy, I found some iirc URLCache thing on
alphaworks.ibm.com. I don't like it and it's not free, so I'll have to
write something better:)
Anyway it caches everything retreived via URL.openConnection().

> Sure, although part of the point of this release is that it is not
> dependent on servlets at all

One from my wishlist:)

> Sounds good - if I put the code in CVS, you wanna implement this for me?

Yes, I'll do it probably today.

> Torque has some reasonable documentation .  Part 4 "writing a sample
> application" of the tutorial is a good place to start:

Thanks, actually I did read this when you introduced torque, but it was
months ago:)

Regards...

Sam Joseph wrote:
> 
> >obj.getClass().getProtectionDomain().getCodeSource().getLocation();
> >}
> >->
> >Torque.init(getLocation(this)+"../../Torque.properties");
> >or something alike.
> >
> Sounds good - if I put the code in CVS, you wanna implement this for me?

Won't work this way - I get full path of ant.jar :)

So I did it with sysproperty arg of junit task:

      <sysproperty key="TorqueConfig" file="./Torque.properties" />
      <sysproperty key="BuildConfig" file="${build.properties}"/>
      <sysproperty key="Log4jConfig" file="./conf/log4j.properties"/>

and

      Torque.init(System.getProperty( "TorqueConfig" ));
      Uri.init(System.getProperty( "Log4jConfig" )); 
...

so now I can build NG from jEdit, much faster than from prompt.
OK to commit build-torque.xml and NeuroGridTestSuite.java ?

But some tests failed. I'm using mysql, so rollback fail is OK. But:
Failing to retrieve Event - assume deleted
org.apache.torque.TorqueException: Failed to select one and only one
row.

at
com.neurogrid.om.BaseUriTriplePeer.retrieveByPK(BaseUriTriplePeer.java:555)

at
com.neurogrid.om.BaseUriTriplePeer.retrieveByPK(BaseUriTriplePeer.java:530)

at com.neurogrid.om.BaseEvent.getUriTriple(BaseEvent.java:499)

at com.neurogrid.middle.EventTest.testCreate(EventTest.java:259)

?

Regards...

Josip Almasi wrote:

>Won't work this way - I get full path of ant.jar :)
>
>So I did it with sysproperty arg of junit task:
>
>      <sysproperty key="TorqueConfig" file="./Torque.properties" />
>      <sysproperty key="BuildConfig" file="${build.properties}"/>
>      <sysproperty key="Log4jConfig" file="./conf/log4j.properties"/>
>
>and
>
>      Torque.init(System.getProperty( "TorqueConfig" ));
>      Uri.init(System.getProperty( "Log4jConfig" )); 
>...
>
>so now I can build NG from jEdit, much faster than from prompt.
>
top stuff

>OK to commit build-torque.xml and NeuroGridTestSuite.java ?
>
sure thing

is jedit free? good?

>
>But some tests failed. I'm using mysql, so rollback fail is OK. But:
>Failing to retrieve Event - assume deleted
>org.apache.torque.TorqueException: Failed to select one and only one
>row.
>
>at
>com.neurogrid.om.BaseUriTriplePeer.retrieveByPK(BaseUriTriplePeer.java:555)
>
>at
>com.neurogrid.om.BaseUriTriplePeer.retrieveByPK(BaseUriTriplePeer.java:530)
>
>at com.neurogrid.om.BaseEvent.getUriTriple(BaseEvent.java:499)
>
>at com.neurogrid.middle.EventTest.testCreate(EventTest.java:259)
>
>?
>  
>
Yeah, that error occurs the second time you run on a database because 
the delete tests have removed some ids that are still represented in 
other tables - I need to remove all delete operations for databases like 
mysql that don't have foreign key support.

I'll have to remember to sort that out.  You should be able to get the 
errors to go away by making sure that the "neurogrid" database is 
recreated from scratch when you run the tests.

CHEERS> SAM

Josip Almasi wrote:

>Sam Joseph wrote:
>  
>
>>Well I think the proxy itself is not a big deal.  In fact I managed to
>>find a java proxy implementation.  The question is whether the servlet
>>engine itself has the flexibility to support the proxy.   I know that
>>tomcat definitely does not, and that jetty might.
>>    
>>
>
>Well now when you put it like that, imho servlet can't act like proxy.
>
would you care to elaborate on that.  I mean it might not be a terribly 
efficient way to implement a proxy.  But if we want to examine all the 
data flowing backwards and forward between the browser and some search 
engine, then a proxy seems a good way to go and servlets would seem to 
be a good way to hook different things in there.

Here's a link related to someone who previously struggled (and gave up) 
when trying to turn tomcat into a proxy:

http://archives2.real-time.com/rte-tomcat/2000/Jun/msg02417.html

The suggestion being that tomcat's URL mappings are not powerful enough 
to support a proxy simulation.

Actually just reading back on my friends notes I can see he is saying 
that Jetty has a lower level API than the servlets, called the 
ResourceHandlerAPI, so I guess it sounds like a servlet can't simulate a 
proxy because conforming to the servlet spec would prevent it.  Does 
that make sense?

So I guess what I am proposing with Jetty is that there would be some 
proxy module built on this Resource HandlerAPI which would be able to 
access the appropriate NG functions in response to browser activity.

Of course not being able to access the servlet methods would be a pain - 
maybe we would have to build alot of extra stuff, or maybe not, cos how 
much functionality do servlets provide that I need?  We shall see .....

CHEERS> SAM

Sam Joseph wrote:
> 
> >Well now when you put it like that, imho servlet can't act like proxy.
> >
> would you care to elaborate on that.  I mean it might not be a terribly
> efficient way to implement a proxy.  But if we want to examine all the
> data flowing backwards and forward between the browser and some search
> engine, then a proxy seems a good way to go and servlets would seem to
> be a good way to hook different things in there.

Well I'm not _sure_... it's that catch with servlets: a servlet is
required to close the stream... then those tcp_keepalive things may
require writing a socket server... then, how do I know if my client uses
http 1.0 or 1.1? It's required by spec... then, how does a servlet
service OPTIONS method? And so on - too many questions.
Just started reading http://www.ietf.org/rfc/rfc2616.txt
8.1.3 Proxy Servers
8.2.3 Use of the 100 (Continue) Status
10.3.6 305 Use Proxy
14.9 Cache-Control
... must this, must not that, should something - too complicated.

> when trying to turn tomcat into a proxy:
> 
> http://archives2.real-time.com/rte-tomcat/2000/Jun/msg02417.html
> 
> The suggestion being that tomcat's URL mappings are not powerful enough
> to support a proxy simulation.

Wouldn't agree on that. Proxy must _not_ rewrite any urls.

> Actually just reading back on my friends notes I can see he is saying
> that Jetty has a lower level API than the servlets, called the
> ResourceHandlerAPI, so I guess it sounds like a servlet can't simulate a
> proxy because conforming to the servlet spec would prevent it.  Does
> that make sense?

Yes. A proxy is generally expected to keep connections alive longer than
a usual server. But servlet api has (afaik) nothing alike. So maybe some
container api can help.

> So I guess what I am proposing with Jetty is that there would be some
> proxy module built on this Resource HandlerAPI which would be able to
> access the appropriate NG functions in response to browser activity.
> 
> Of course not being able to access the servlet methods would be a pain -
> maybe we would have to build alot of extra stuff, or maybe not, cos how
> much functionality do servlets provide that I need?  We shall see .....

Yeah, let's hack muffin and jigsaw :)
btw this jigsaw thing is http 1.1 proxy, it's not GPL but maybe the code
is readable - available at w3c.org.
And squid (http://www.squid-cache.org/), imho very good docs, but it's
something else.

Regards...

Sam Joseph wrote:
> 
> >OK to commit build-torque.xml and NeuroGridTestSuite.java ?
> >
> sure thing

done.

> is jedit free? good?

GPL, and best:)
I stared using it about two years ago couse it was the only editor that
worked on both windoze and OSes. In the meantime it turned into IDE, has
bunch of plugins - decompiler, debugger, ant, tomcat, ftp... and it's
still editor:)
See yourself, it's on sourceforge: http://jedit.sf.net/

Regards...

dixit Josip Almasi:
> > is jedit free? good?
> 
> GPL, and best:)
> I stared using it about two years ago couse it was the only editor that
> worked on both windoze and OSes.

long before jedit, there was emacs and vi on more operating systems
(including windows) than there are currently jvms for, and which therefore
jedit is limited to.

> In the meantime it turned into IDE, has
> bunch of plugins - decompiler, debugger, ant, tomcat, ftp... and it's
> still editor:)

all things that emacs did before. plus, you can easily use emacs and vi over
ssh.
-- 
dog
falco | raptor | ukrmma#18
sol lucet omnibus ... praeter anglorum

dog wrote:
> 
> dixit Josip Almasi:
> > > is jedit free? good?
> >
> > GPL, and best:)
> > I stared using it about two years ago couse it was the only editor that
> > worked on both windoze and OSes.
> 
> long before jedit, there was emacs and vi on more operating systems
> (including windows) than there are currently jvms for, and which therefore
> jedit is limited to.
> 
> > In the meantime it turned into IDE, has
> > bunch of plugins - decompiler, debugger, ant, tomcat, ftp... and it's
> > still editor:)
> 
> all things that emacs did before. plus, you can easily use emacs and vi over
> ssh.

I knew it, someone allways bites it:))))
NHF it was a spam-like joke:)
As for jedit, i just like it.

Regards...