seeks-users — User forum

[Seeks-users] How many threads ?

[Loic D. <lo...@da...>]

Hi,

I'm still strugling and unsure if the implementation of the chord
algorithm should be done using event based I/O instead of the sequential
approach described in the papers and currently implemented in
DHTVirtualNode.

While thinking about it, I realize that in the current implementation
there needs to be a thread for each lookup of the host responsible for a
given key. In a million node DHT, such a lookup will take up to 10
seconds (see below for the reasoning). Even in a much smaller DHT it
takes a lot longer than is acceptable for a blocking operation.

As a result, the number of threads required by the current implementation is

1) for stabilisation
2) for handling incoming requests
N) for N simultaneous requests

That is 2 + N threads total.

Cheers

* Execution time of the routing and access algorithms

  The finger table is refreshed one entry after the other independantly.
     Updating is done by looking up the host responsible for a given key.
  The successor list is refreshed by asking the successor.
  The successor and the predecessor is refreshed by a simple decision based
     on an incoming message.

  Finding the host responsible for a give key is the longest operation
  as it requires to contact log(N) nodes. Let say a DHT contains 1
  million nodes, ~13 nodes will be contacted to figure out which host
  needs to be contacted for a given finger table entry. A contact is
  made of a) asking for the closest predecessor, b) asking its
  successor to the predecessor retrieved in step a).  How long will
  these 13 * 2 = 26 contacts take ? Assuming each connection timesout
  when not receiving an answer within 300ms, the total time to
  complete the operation is 26 * 300 = 7800ms, 7.8ms.

Re: [Seeks-users] Negative feedback on personalized search results

[Emmanuel B. <ebe...@se...>]

On Mon, Jan 10, 2011 at 01:32:40AM +0100, Loic Dachary wrote:
> Congratulations :-)
> > The negative feedback button is to be released with the forthcoming
> > 0.3.2 stable version of Seeks. Our experimental public node has been 
> > upgraded with the new code, http://www.seeks-project.info/search_exp.php
> >   
> Where is the "negative feedback button" located on the page ?

As of today you need to setup your own 'experimental' node, or use
a public one that is up to date.

Em.

Re: [Seeks-users] DHT implementation

[Emmanuel B. <ebe...@se...>]

Hey Loic,

sounds great, and besides that I like the ascii description,
I don't have any important remarks.

Maybe the layering of the CNX and REMOTE objects will be
clearer to me when reading the code.

My understanding is that "All MESSAGEs carry the KEY of the NODE from which
the MESSAGE ORIGINates" because the ROUTER object does caching of 
hosts and ports. Otherwise there is no need for a node to send its
key, am I right ?

Em.

On Mon, Jan 10, 2011 at 01:26:53AM +0100, Loic Dachary wrote:
> Hi,
> 
> I spent most of the day trying to figure out a simple architecture for
> routing the
> messages between the nodes of the DHT. I summarized my findings in the
> following
> document and I would welcome feedbacks or questions.
> 
> http://redmine.seeks-project.info/attachments/download/71/IMPLEMENTATION
> 
> Cheers
> 

Re: [Seeks-users] Negative feedback on personalized search results

[Pablo J. <me...@ou...>]

Hi,
Le 10/01/2011 01:32, Loic Dachary a écrit :
> Congratulations :-)
>> The negative feedback button is to be released with the forthcoming
>> 0.3.2 stable version of Seeks. Our experimental public node has been 
>> upgraded with the new code, http://www.seeks-project.info/search_exp.php
>>   
> Where is the "negative feedback button" located on the page ?
You have to do a search that gives personnalised results (you can try
seeks), and then a light-red cross appears at the end of the snippet
(just after « (Cached) Archive - Similar - Type »)

Pablo

Re: [Seeks-users] Negative feedback on personalized search results

[Loic D. <lo...@da...>]

Congratulations :-)
> The negative feedback button is to be released with the forthcoming
> 0.3.2 stable version of Seeks. Our experimental public node has been 
> upgraded with the new code, http://www.seeks-project.info/search_exp.php
>   
Where is the "negative feedback button" located on the page ?

Cheers

[Seeks-users] DHT implementation

[Loic D. <lo...@da...>]

Hi,

I spent most of the day trying to figure out a simple architecture for
routing the
messages between the nodes of the DHT. I summarized my findings in the
following
document and I would welcome feedbacks or questions.

http://redmine.seeks-project.info/attachments/download/71/IMPLEMENTATION

Cheers

[Seeks-users] Remote user databases

[Emmanuel B. <ebe...@se...>]

Hi all,

The current 'experimental' branch introduces, with commit 
704ba486cf961ff799d7e00d7be9173d921e2df9 the ability to use remote user
databases.

In practice, this means that you can setup a Tokyo Tyrant server for
your user database containing personalized information (your clicks,
queries and navigation), serving the database over your local network
for example. You can then use the same database from any of your Seeks 
nodes. 

While this may not sound better as to setup a remote Seeks instance, 
it is the first step towards the ability to exchange data fragments among
remote user databases. So technically, this is the root basis for later
sharing of queries and experience on their results.

Be careful though if you intend to setup a remote user database, as 
it does not include any rights (read/write) management, nor secure
(e.g. SSL) connection yet.

Em.

[Seeks-users] Negative feedback on personalized search results

[Emmanuel B. <ebe...@se...>]

Hi all,

recent work on the Seeks code base has added the long planned feature
of being able to reject unwanted search results.

One of Seeks' main objectives is to give users better control over
search algorithms and their results. Until now Seeks was learning
from users clicks and recommending some results among results returned
by conventional engines. This learning was 'passive', not
requiring any specific intervention from users.

This positive reinforcement was a first step towards better personalization
and sharing of experience and knowledge among users. But the lack of
negative feedback was blocking users from having a finer control 
over their results.

Seeks now implements a 'rejection' button next to any personalized
results. As a remainder personalized results are those using one
or more users' past clicks to boost up a result's ranking.
The 'rejection' button applies a negative feedback to both the 
result's URL and domain. It thus blocks the result from being
boosted up and from impacting other users' similar searches.

The negative feedback is overridden when the URL is being clicked.

In a not to distant future, not only personalized results, but all
results will be put under similar control.

These are preliminary steps before the introduction of a publishing
mecanismm allowing users to 'push' results they like to search
groups directly, and others to reject them.

The negative feedback button is to be released with the forthcoming
0.3.2 stable version of Seeks. Our experimental public node has been 
upgraded with the new code, http://www.seeks-project.info/search_exp.php

For developpers and testers, the feature is now part of
the 'experimental' of our git repository.

Em.

[Seeks-users] Short 'under the hood' Seeks talk at FOSDEM 2011 Data Devroom

[Emmanuel B. <ebe...@se...>]

Hi Seeksers,

FYI our submission to the FOSDEM Data Devroom this year has been accepted.

Therefore there will be a short 15 mins talk on Saturday the 5th at 6pm.

I'll outline the main machine learning and similarity analysis algorithms
that are currently implemented and will give some insights about where
we are heading. This latter part includes possibly more machine learning
for local results personalization and data selection.

Below is the submission as I did send it.

-------------------------------------------------------------

* Speaker: Emmanuel Benazera

* Biography: Emmanuel Benazera received a master degree in applied mathematics and social sciences
  from University Paris-Dauphine, France in 1999, and Ph.D. degree in computer science
  in 2003 from Paul Sabatier University, France. He was a visiting scientist at the NASA
  Ames Research Center for two years before he joined the robotics department of
  DFKI/University of Bremen in Germany. Then he was momentarily with LAAS-CNRS in Toulouse,
  France. His main research interests include automated decision making & planning, machine
  learning, information retrieval and p2p networks.

* Title: How Seeks let you do your Web search at home:

* Abstract: Seeks is a free and open P2P design and application for enabling social websearch.
 Its specific purpose is to regroup users whose queries are similar so they can share both the
 query results and their experience on these results.

 Seeks is designed to run as a proxy on a user's machine. As such it studies the user's
 behavior and uses these data locally to improve websearch results. Other features include
 automated similarity analysis for text and images, automated clustering of textual content,
 and most discriminant words highlighting.

 Seeks relies on a mixture of machine learning & similarity analysis tools applied
 to Web search and results personalization. These tools are implemented with one goal in mind:
 give more control to the user in its searches. Typically, machine learning and information
 retrieval algorithms help squeeze more information out of the data. However, Seeks always
 let the user decide what matters to him. It does so by studying user clicks and navigation,
 and using this data to make recommendations (results, queries, URLs). This talk intends
 to outline some of these tools, and to show how their careful integration benefit many
 searches.

 In a near future, Seeks instances will talk to each other through a P2P overlay network (DHT),
 so that users can share their experience over similar queries.

* Audience: intermediate

-----------------------------------------------------------------------------------------------

Happy New Year celebrations,

Em.

Re: [Seeks-users] Identity and Access Management

[Emmanuel B. <ebe...@se...>]

Hi,

thanks for the pointers.

At this time there are two security threads to be considered within Seeks:

1/ routing on the DHT requires that peers can be certified. There are two 
  reasons for this:
 
 * secure routing, certificates being a typical counter-measure to Sybil
   attack;

 * trust rings within the main DHT ring itself so that users trusting each
   others can build their own trusty sub-network.

2/ remote access to search profiles. In a near future Seeks will allow
  deporting the user database containing a user's search profile (i.e.
  queries + clicked URLs, expect more fields in the future).
  This will have two applications:

  * users will be able to use a single remote database from all their
  machines, thus centralizing / securing their own data as they'd like
  (instead of carrying the data on their local machine). Where to put
  the database remains the user's decision.

  * public institutions / users will be able to share their profiles
  to so-called 'followers'. These followers will be able to use the
  shared profiles to experience their searches' results under a different
  filter. Typically, searching for solution to a problem say on a Debian
  system, a user would consider using a 'Debian' labelled profile.

The security solutions in the four contexts above are open, and many
questions remain.

Typically, is there a single solution that could address both 1/ & 2/ ?
This, I believe, would be the first question to answer.

In my view, 1/ & 2/ are slightly different. 1/ requires a set of certificates,
along with a central or decentralized authority to issue / revoke them.

About 2/, my current understanding is that there are two needs:
 
 a/ a control of read/write permmissions on remote user databases. This
 for shared databases to remain under the control of their owners;

 b/ a certification system to ensure that a profile (user db) is truely
 provided by the institution / user who claims to own (and fill) it.

Right now, I guess that b/ could be addressed outside Seeks, by serving
the profile from a (certified) website. However, a certification scheme
built-in Seeks would be a plus in the longer term I believe.

Much discussion is needed on the topics above, and much remain to be 
done and decided. I will try to help as much as I can with design and
solution techniques, but I fear my knowledge and current understanding
of these matters is largely insufficient.

Em.

On Tue, Dec 21, 2010 at 03:30:42AM +0100, Alexandre CABROL PERALES wrote:
> Dear seeks contributors,
> i heard about important topic to be discussed for seeks project
> about identity and access management.
> 
> I'm currently working on in my professionnal life, so i push my
> first contribution.
> 
> I found interesting article about existing opensource framework to
> implement IAM in a project.
> 
> http://openliberty.org/wiki/index.php/ProjectAris
> 
> And other solutions which already exists:
> http://www.openliberty.org/wiki/index.php/Existing_Identity_Systems
> 
> Anybody is interested in IAM solution for seeks project and wanna
> talk about ?

Re: [Seeks-users] DHT code evolution (repost)

[Loic D. <lo...@da...>]

Hi,

As discussed on IRC today, I think there is one class missing :
DHTStorage that would handle the data associated with the keys. I would
be responsible for moving the data from one node to the other, should a
new node join or leave the ring. I think it would allow to remove the
SGVirtualNode + SGNode (if you look at them they don't really do much
more than encoding/decoding and delegating to sg_manager) and have
sg_manager be a derivative of DHTStorage.

Cheers

Re: [Seeks-users] DHT code evolution (repost)

[Loic D. <lo...@da...>]

>   
>> * reducing to a single thread instead of two (with libevent)
>>     
>
> You mean, instead of one for the RPC server + one for the stabilization
> calls in client mode ?
>
>   
Yes.
>> * use UDT instead of UDP
>>     
>
> What would be the rational behind this change given that the protocol buffer
> messages are very small at this time, between 54 and 128 bytes.
>
>   
I'm not entirely sure. I researched it today and found features
important to the DHT
1) multiplexing communication channels allowing for an asynchronous version
    of do_rpc_call (nice to have)
2) keeps information about the connection health which is essential to
figuring
    out if a node is dead or alive (required)
3) the application can use the UDT file descriptor to estabish a
communication
    channel using the same port without disrupting the DHT messages
(nice to have)
4) NAT traversal (nice to have)
    
>> * cache DNS resolution
>>     
I think getaddrinfo_a does the trick (
http://redmine.seeks-project.info/issues/288 ). The fallback is the
standard getaddrinfo which only has the disadvantage of blocking.  This
system call  is apparently available on every GNU/Linux distributions
which should be good for most users willing to go fast with seeks.

>> * use a pool of remote nodes modeling their accessibity accross virtual
>> nodes
>>     
>
> This would be shared board of information about the quality and freshness
> of known nodes and data transfers to them, right ?
>
>   
Yes. And it turns out UDT already collects the necessary information.
>
>> * replace the protobuf wrappers with a hierarchy of derived classes
>>     
>
> Can you please elaborate about what you mean with this modification ?
>
>   
When the server receives a message, it needs to unserialize it to
extract the DHTKey and route the message to the appropriate virtual
node. It should be done by registering a factory to the Transport
object. The factory would be called with the incoming buffer and return
a Message based object. The searchgroup factory woud be derived from the
l1 factory and able to interpret searchgroup specific messages.

It is necessary to avoid the double unserialization that is currently done.
> I agree. Everything that is about to be released should be as clean as
> possible. I will only try to avoid postponing the release of p2p for
> reasons such as building up more features than needed for a safe basic
> start.
>  
>   
I agree and I think we will have a hard time handling security and
keeping it simple.
> The restructured code makes the DHT much better than what it was before
> you put your magic spell on it :) The unit tests are a blessing. 
>
>   
:-)
>
> Exactly. This doesn't mean we have to come up with a framework as advanced
> as maidsafe before the first p2p release, but that the ground should be
> clean and prepare for gradual improvements, theoretical and empirical.
>   
I think we can do simpler.
> The latter is another reason to release a p2p-enabled version in a not
> too distant future.
>
>   
Yes, this is my main concern. Trying to find the right balance.
> Again, thanks for the good work, and all the time you've put on this
> difficult part of Seeks' pool of code.
>   
It is exciting to explore all aspects of this delicate and yet essential
piece of the puzzle ;-)

Cheers

Re: [Seeks-users] DHT code evolution (repost)

[Emmanuel B. <ebe...@se...>]

On Wed, Dec 22, 2010 at 10:05:17AM +0100, Loic Dachary wrote:
> Hi,
> 
> I completed a rewrite of the stabilization of the successor list of the
> seeks DHT today, with the associated tests. It is available at
> http://seeks.dachary.org/seeks.git

Thanks this is great work.
 
> In my opinion the changes that shoud be done to estabish a sound base
> for the DHT are the following:
> 
> * replacing the stabilization loop with libevent

OK. This is ticket #39:
http://redmine.seeks-project.info/issues/39

> * reducing to a single thread instead of two (with libevent)

You mean, instead of one for the RPC server + one for the stabilization
calls in client mode ?

> * use UDT instead of UDP

What would be the rational behind this change given that the protocol buffer
messages are very small at this time, between 54 and 128 bytes.

> * cache DNS resolution

OK.

> * use a pool of remote nodes modeling their accessibity accross virtual
> nodes

This would be shared board of information about the quality and freshness
of known nodes and data transfers to them, right ?

> * use Location instead of DHTKey + NetAddress

OK.

> * replace the protobuf wrappers with a hierarchy of derived classes

Can you please elaborate about what you mean with this modification ?

> Each of them taken individually are relatively easy to agree on. All
> together they look like more work than necessary to get started. It
> would certainly be possible to have a working DHT without doing this
> ground work. But I believe it would seriously jeopardize our ability to
> build on safe grounds.

I agree. Everything that is about to be released should be as clean as
possible. I will only try to avoid postponing the release of p2p for
reasons such as building up more features than needed for a safe basic
start.
 
> In the past weeks I wrote tests and extensively explored the existing
> code. I don't think I missed an area that would contradict the
> diagnostic above. I adapted and refactored locally to the best of my
> abilities. But I believe it is time, using our combined experiences, to
> restructure the code to match all of the above constraints at once.

The restructured code makes the DHT much better than what it was before
you put your magic spell on it :) The unit tests are a blessing. 

I can help with the refactoring at all levels. My attention is more to
soundness of the theoretical foundations of the DHT. All of this can be
discussed here.
 
> One could argue that it would be a waste of time to carefull restructure
> and that it is more important to get something working quickly. But if
> that was the case, chosing a ready made solution such as maidsafe would
> be a lot quicker. The main reason to implement a DHT is to make it an
> integral part of seeks instead of a tool box imported from a third party.

Exactly. This doesn't mean we have to come up with a framework as advanced
as maidsafe before the first p2p release, but that the ground should be
clean and prepare for gradual improvements, theoretical and empirical.
The latter is another reason to release a p2p-enabled version in a not
too distant future.

Again, thanks for the good work, and all the time you've put on this
difficult part of Seeks' pool of code.

Em.

[Seeks-users] DHT code evolution (repost)

[Loic D. <lo...@da...>]

Hi,

I completed a rewrite of the stabilization of the successor list of the
seeks DHT today, with the associated tests. It is available at
http://seeks.dachary.org/seeks.git

In my opinion the changes that shoud be done to estabish a sound base
for the DHT are the following:

* replacing the stabilization loop with libevent
* reducing to a single thread instead of two (with libevent)
* use UDT instead of UDP
* cache DNS resolution
* use a pool of remote nodes modeling their accessibity accross virtual
nodes
* use Location instead of DHTKey + NetAddress
* replace the protobuf wrappers with a hierarchy of derived classes

Each of them taken individually are relatively easy to agree on. All
together they look like more work than necessary to get started. It
would certainly be possible to have a working DHT without doing this
ground work. But I believe it would seriously jeopardize our ability to
build on safe grounds.

In the past weeks I wrote tests and extensively explored the existing
code. I don't think I missed an area that would contradict the
diagnostic above. I adapted and refactored locally to the best of my
abilities. But I believe it is time, using our combined experiences, to
restructure the code to match all of the above constraints at once.

One could argue that it would be a waste of time to carefull restructure
and that it is more important to get something working quickly. But if
that was the case, chosing a ready made solution such as maidsafe would
be a lot quicker. The main reason to implement a DHT is to make it an
integral part of seeks instead of a tool box imported from a third party.

Cheers

[Seeks-users] Identity and Access Management

[Alexandre C. P. <ale...@la...>]

Dear seeks contributors,
i heard about important topic to be discussed for seeks project about 
identity and access management.

I'm currently working on in my professionnal life, so i push my first 
contribution.

I found interesting article about existing opensource framework to 
implement IAM in a project.

http://openliberty.org/wiki/index.php/ProjectAris

And other solutions which already exists:
http://www.openliberty.org/wiki/index.php/Existing_Identity_Systems

Anybody is interested in IAM solution for seeks project and wanna talk 
about ?

[Seeks-users] seeks on NetBSD

[Erin N. <eri...@fr...>]

hi all,

after seeing the project at the RMLL, I tried to install a seeks node on 
my NetBSD server but failed... My first error was:
In file included from ../proxy/seeks_proxy.h:27,
                  from mrf.cpp:22:
../proxy/proxy_dts.h:42:38: error: pcreposix.h: No such file or directory
../proxy/proxy_dts.h:43:18: error: pcre.h: No such file or directory
../proxy/pcrs.h:146: error: ISO C++ forbids declaration of 'pcre' with 
no type
../proxy/pcrs.h:146: error: expected ';' before '*' token
../proxy/pcrs.h:147: error: ISO C++ forbids declaration of 'pcre_extra' 
with no type
../proxy/pcrs.h:147: error: expected ';' before '*' token
../proxy/pcrs.h: In constructor 'sp::pcrs_job::pcrs_job()':
../proxy/pcrs.h:137: error: class 'sp::pcrs_job' does not have any field 
named '_pattern'
../proxy/pcrs.h:137: error: class 'sp::pcrs_job' does not have any field 
named '_hints'
../proxy/proxy_dts.h: At global scope:
../proxy/proxy_dts.h:257: error: ISO C++ forbids declaration of 
'regex_t' with no type
../proxy/proxy_dts.h:257: error: expected ';' before '*' token
../proxy/proxy_dts.h:258: error: ISO C++ forbids declaration of 
'regex_t' with no type
../proxy/proxy_dts.h:258: error: expected ';' before '*' token
../proxy/proxy_dts.h: In constructor 'sp::url_spec::url_spec()':
../proxy/proxy_dts.h:227: error: class 'sp::url_spec' does not have any 
field named '_preg'
../proxy/proxy_dts.h:227: error: class 'sp::url_spec' does not have any 
field named '_tag_regex'
gmake[3]: *** [mrf.lo] Error 1

after some look in the Makefile, this error disappear by manually adding 
the include path in AM_CPPFLAGS in the Makefile inside lsh directory (I 
know, it's only a hack, a more permanent solution has to found... And I 
will be happy to find it after succeeded to install a public node on my 
server ^^), but a second error appear and I don't know what to do:

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -Wall -Wno-deprecated 
-g -pipe -I../utils/ -I../proxy/ -I /usr/pkg/include/ -g -O2 -MT 
LSHUniformHashTableHamming.lo -MD -MP -MF 
.deps/LSHUniformHashTableHamming.Tpo -c LSHUniformHashTableHamming.cpp  
-fPIC -DPIC -o .libs/LSHUniformHashTableHamming.o
LSHSystem.h:62: error: expected unqualified-id before numeric constant
LSHSystem.h: In constructor 'lsh::LSHSystem::LSHSystem(const unsigned 
int&, const unsigned int&)':
LSHSystem.h:40: error: expected identifier before numeric constant
LSHSystem.h:40: error: expected `(' before numeric constant
LSHSystem.h:40: error: expected `{' before numeric constant
gmake[3]: *** [LSHUniformHashTableHamming.lo] Error 1

is there some solution for this one?