gnugle-developers Mailing List for Gnugle

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Potentially interesting link:

http://www.cyc.com/cyc/technology/cycrandd/areasofrandd_dir/distributedai

Hi Marco & al.,

just wanted to mention to you that I made a few changes & additions
to the Wiki.

Joe

Dear Open Source developer

I am doing a research project on "Fun and Software Development" in which I kindly invite you to participate.
You will find the online survey under http://fasd.ethz.ch/qsf/. The questionnaire consists of 53 questions and you will need about 15 minutes to complete it.

With the FASD project (Fun and Software Development) we want to define the motivational significance of fun when software developers decide to engage in Open Source projects. What is special about our research project is that a similar survey is planned with software developers in commercial firms. This procedure allows the immediate comparison between the involved individuals and the conditions of production of these two development models. Thus we hope to obtain substantial new insights to the phenomenon of Open Source Development.

With many thanks for your participation,
Benno Luthiger

PS:
The results of the survey will be published under http://www.isu.unizh.ch/fuehrung/blprojects/FASD/.
We have set up the mailing list fa...@we... for this study. Please see http://fasd.ethz.ch/qsf/mailinglist_en.html for registration to this mailing list.

_______________________________________________________________________

Benno Luthiger
Swiss Federal Institute of Technology Zurich
8092 Zurich

Mail: benno.luthiger(at)id.ethz.ch
_______________________________________________________________________
Hi Joe!

> Thanks for the update.  The wiki at 
> 
> http://www.marcolange.de/gnugle/gnugle-wiki/index.php/Main_Page 
> 
> looks nicer than the old one.

This is certainly a matter of taste, I liked the look of the PHPWiki 
better, but it was a non-stable development version (the stable version 
had too few features) and was missing some features (mainly file upload, 
and LaTeX also comes in handy) I wanted to have.

> Keep up the good work and if you have any reading assignments,
> please post them to the list.  

Tomorrow (i.e. today, its 4.12 AM) is a holiday here in Germany, perhaps 
I will be able to take a look at some of the IPTPS papers I found on the 
internet and summarize and comment them in the wiki. Maybe I will have 
some reading suggestions then.

> PS.  I just had a quirky thought that perhaps Index Replication
> http://www.marcolange.de/gnugle/gnugle-wiki/index.php/Index_Replication
> might be fruitfully related to genetic algorithms...

I am definitively no expert what genetic algorithms are concerned, but I 
  am always open to new ideas and suggests, so I would really like you 
to explain that in more detail. Best would be in the wiki.

Greetings,

   Marco

Hi Marco,

Thanks for the update.  The wiki at 

http://www.marcolange.de/gnugle/gnugle-wiki/index.php/Main_Page 

looks nicer than the old one.

Keep up the good work and if you have any reading assignments,
please post them to the list.  

Thanks,

Joe

PS.  I just had a quirky thought that perhaps Index Replication

http://www.marcolange.de/gnugle/gnugle-wiki/index.php/Index_Replication

might be fruitfully related to genetic algorithms...

Hi!

Although there was absolutely no progress in the project during the last 
weeks due to time issues, I managed to migrate the Wiki to Media Wiki 
1.2.5 which offers some advantages over the PhpWiki:
   * more stable
   * additional features, e.g. LaTeX markup (not yet enabled, soon to
       come), page locking, file uploads etc.

Please create your used account my following the "Log In" link in the 
upper right corner the first time you visit the wiki.

I hope I will make some progress on the project within the following two 
weeks. I wanted to evaluate some approches for structured P2P networks 
(such as CAN, Chord, Kademlia) more in detail and analyse their 
advantages and disadvantages and analyse whether they could be used or 
adapted for our needs.

Thanks,
   Marco

Hi!

> One of the issues that I've been interested in is "web scanning"
> software -- for example, a program that would automatically begin a
> search starting at the current page you are browsing and then return
> pages that match certain keywords that lie within some neighborhood
> of that page.
> 
> Perhaps Gnugle could have something like this built in.

Yes, such a feature is definitely planned, implemented as a HTTP proxy. 
By combined the web crawling with a proxy server, we have several 
advantages for the user:

   - The web crawling can also be used for pre-caching documents linked
     by the current page, thereby being able to return the web page
     from its cache when the user clicks the link.

   - The web crawling bandwidth is not wasted from the user's point of
     view, the user has no disadvantage of having his computer doing the
     web crawling.

   - This proxy function can also be used to generate access statistics
     and ratings for the indexed web pages.

But the whole proxy thing has one big disadvantage: It is not anonymous. 
Potential adversaries in the network could be able to track your web 
surfing. We somehow have to avoid this and do not yet have a idea how.

We surely plan to implement such a feature, but not in the first version 
of Gnugle, as the implementation of a proxy server is rather technical. 
For the first version we want to concentrate on an efficient protocol 
and indexing.

Best regards,
   Marco

One of the issues that I've been interested in is "web scanning"
software -- for example, a program that would automatically begin a
search starting at the current page you are browsing and then return
pages that match certain keywords that lie within some neighborhood
of that page.

Perhaps Gnugle could have something like this built in.

Hi Joe!

> I am interested in your project, having had the same or similar idea
> myself on various occaisions.
> 
> Please send me and my friend Tim (copied above) some more
> information about the project.

Thank you, Joe and Tim, very much for your interest in the project. For 
the time being, we are still in a very early planning phase of the 
project, and currently we are mainly gathering information about 
suitable network topographies and protocols. We have not released any 
source code yet.

We do have a project wiki [2] on our project home page [1], where we 
have already discussed some ideas. But unfortunately, some of the ideas 
are already out-of-date.

Our first plan was to use an unstructured peer-to-peer network, where 
the index is simply distributed among all peers in the network without 
specific structure. To find all entries in the index for a given search 
phrase, a broadcast (TTL-flooding) of the query to all peers would be 
necessary, which is basically the search algorithm used by the Gnutella 
network.

After some discussions with Michael Christen, who has initiated a 
similar project [3] (distributed indexing, but no web crawling, only 
using transparent proxies for indexing), it became clear that this 
approach is not scalable enough for large networks, also see this paper 
[6] for information about the scalability issues of the Gnutella network.

In the meantime, we have realized that we need a structure peer-to-peer 
network for our needs, as only structured peer-to-peer networks provide 
scalability. A structured peer-to-peer network is basically a 
content-addressable network (CAN), i.e. given a search phrase, one can, 
through some simple calculations (mainly hash functions), determine on 
which peer the requested information is stored. This concept is also 
known as a distributed hash table (DHT), as it basically provides the 
actions

   - get(key)
   - put(key, value)
   - delete(key)

in a distributed manner. With the help of such a DHT, we can quickly map 
search phrases to peers in the network. But there still remain open 
problems:

   - How do we cope with the high volatility of peers in the network,
     with peers entering and leaving the network at any time.

   - How does our protocol provide sufficient load-balancing. Certain
     key phrases will surely be queried more frequently than others.
     Taking into consideration that Google answers 200 million query per
     day, a single peer might not be able to handle

   - How to take the underlying Internet topography into account. One of
     the main problems of current P2P networks implementations deployed
     is their disregard for the underlying Internet topology. Most P2P
     networks form an overlay network on top of the Internet at
     application level, providing their own routing algorithms. The main
     problem is that neighborhoods in the network graph of the overlay
     network are often no neighborhoods in the network graph of the
     Internet, thus resulting in high latencies and high cost due to
     excessive peering between different networks. We should take this
     into account in the design of our networks, although he will need
     a trade-off between this problem and the scalability and stability
     of the network.

We would be very glad if you would like to join our project. As you can 
see, this is not a project we can start to code quickly, as we first 
still have to do some kind of research to work out suitable protocols 
and algorithms for our needs. Interesting material about peer-to-peer 
networks and P2P network protocols can be found in the proceedings of 
the IPTPS 2002 [4] and 2003 [5] and following the citations in these 
papers. Nearly all of these papers can be found on Citeseer. Another 
interesting paper about the feasibility of P2P search engines, which 
shows the problems to be overcome, is [7].

Please feel free to join our gnugle-developers mailing list.

Best regards,
   Marco

[1] http://gnugle.sourceforge.net
[2] http://www.marcolange.de/gnugle/gnugle-wiki/
[3] http://www.anomic.de/AnomicHTTPProxy/index.html
[4] http://www.cs.rice.edu/Conferences/IPTPS02/
[5] http://iptps03.cs.berkeley.edu/program.html
[6] http://www.cs.rice.edu/Conferences/IPTPS02/165.pdf
[7] http://iptps03.cs.berkeley.edu/final-papers/search_feasibility.ps

Florian Bremer wrote:

>>What kind of simulation do you have in mind exactly? Is it only the 
>>network topography or do you want to simulate information propagation 
>>and indexing issues, too? I like the idea of having a tool for network 
>>topography because the project is very interesting from the research 
>>point of view. In addition, this tool would not only be limited to 
>>Gnugle but could be of greater use.
> 
> 
> I thought about simulating network topography and some aspects of
> information propagation.  
> 
> Crawling and indexing functionality should be (almost) ignored and
> replaced with a very simple dummy technique:
> 
> Instead of websites, a node crawls a number N (member of a set NS). If
> the simulation environment intercepts a crawl-request, it delivers the
> result res(N) and a random amount of new numbers (each member of NS),
> which represent links found in the indexed "site". Res(N) can be a
> hash-function or something like this.  

The idea is good. Every indexed keyword is represented by a number. 
Links are simulated by random numbers.

> Althoug this is a quick'n'dirty hardcore simplification, i believe, that
> it will help to answer the following questions :
> 
> -What's the probability P, that a search request is answered
> successfully?
> -How does P depend on redundancy-level of the information stored among
> the different node?
> -How does P depend on the amount of connected nodes, the cardinality of
> NS (in the figurative sense this means the total amount of websites) and
> other parameters?
> 
> Other aspects we adumbrated in the wiki like "Network information
> propagation" / "Adding nodes to the network" could simulated in am more
> detailed manner. 
> 
> _VERY_ short implementation idea (too tired to keep on
> writing/thinking):
> 
> -main program is provides an infinite loop
> 
> -new nodes are added to the network with a specified probablitiy; the
> node, it has to connect to, is chosen randomly among the connected nodes
> -in each cycle, each node (represented as a object) executes a method
> (idle, connect, disconnect, crawl, do search query, ...) with a
> specified probability
> -transmitting a query/answer between two nodes (without using a third
> node as interstation) takes one cycle

A good idea, I will keep that in mind, when my mind is clear again,

Greetings,
              Marco

On Wed, Jan 28, 2004 at 08:03:17PM +0100, Marco Lange wrote:

> What kind of simulation do you have in mind exactly? Is it only the=
=20
> network topography or do you want to simulate information propagati=
on=20
> and indexing issues, too? I like the idea of having a tool for netw=
ork=20
> topography because the project is very interesting from the researc=
h=20
> point of view. In addition, this tool would not only be limited to=
=20
> Gnugle but could be of greater use.

I thought about simulating network topography and some aspects of
information propagation. =20

Crawling and indexing functionality should be (almost) ignored and
replaced with a very simple dummy technique:

Instead of websites, a node crawls a number N (member of a set NS). I=
f
the simulation environment intercepts a crawl-request, it delivers th=
e
result res(N) and a random amount of new numbers (each member of NS),
which represent links found in the indexed "site". Res(N) can be a
hash-function or something like this. =20

Althoug this is a quick'n'dirty hardcore simplification, i believe, t=
hat
it will help to answer the following questions :

-What's the probability P, that a search request is answered
successfully?
-How does P depend on redundancy-level of the information stored amon=
g
the different node?
-How does P depend on the amount of connected nodes, the cardinality =
of
NS (in the figurative sense this means the total amount of websites) =
and
other parameters?

Other aspects we adumbrated in the wiki like "Network information
propagation" / "Adding nodes to the network" could simulated in am mo=
re
detailed manner.=20

_VERY_ short implementation idea (too tired to keep on
writing/thinking):

-main program is provides an infinite loop

-new nodes are added to the network with a specified probablitiy; the
node, it has to connect to, is chosen randomly among the connected no=
des
-in each cycle, each node (represented as a object) executes a method
(idle, connect, disconnect, crawl, do search query, ...) with a
specified probability
-transmitting a query/answer between two nodes (without using a third
node as interstation) takes one cycle

> I hope I will be able to do a bit more the next days, but the last =
three=20
> days, I was "disabled" by a nerving cold and sniffles.

Fortunately, i've already left my yearly cold behind me ;o)

Gru=DF
Florian

=3D=3D=3D
"Irgendwie hab ich das schon mal erlebt. - Ach ja!
Gestern, vorgestern und eigentlich jeden Tag.
Das ist deprimierend." (Bernd das Brot)

GnuPG: 0x7E3055F3

Florian Bremer wrote:

> At the beginning I thought, that it we would be a goog idea to start
> implementing nodes with a limited functional range, 
> but as real client-"server" applications using sockets to communicate
> with one another.
> 
> But now i've got a new idea: What's about writing an simulating
> environment (standalone program) first? 
> 
> Of cause, we would ignore / simplify several issues, but such an
> environment would sufficient, to consider the following points:
> 
> -How does the network structure envolve, when many nodes are added?
> (maybe we could visuialize this for a "small" amount of nodes)
> -What's the possibility of generating nodes, that can't be removed from
> the network without dividing it into 2 disjunctive parts?
> -From which parameters does the point above depend?
> -How does the propagation of queries amount the nodes work?
> -...
> 
> I think such an environment would bring up / answer many interesting
> questions related to graph theory, which will be important for us.

What kind of simulation do you have in mind exactly? Is it only the 
network topography or do you want to simulate information propagation 
and indexing issues, too? I like the idea of having a tool for network 
topography because the project is very interesting from the research 
point of view. In addition, this tool would not only be limited to 
Gnugle but could be of greater use.

> P.S: As i have to write some examinations in the next weeks, there will
> be few activity until 11th februrary ;o(

I hope I will be able to do a bit more the next days, but the last three 
days, I was "disabled" by a nerving cold and sniffles.

Greetings,
            Marco

Hi!

At the beginning I thought, that it we would be a goog idea to start
implementing nodes with a limited functional range, 
but as real client-"server" applications using sockets to communicate
with one another.

But now i've got a new idea: What's about writing an simulating
environment (standalone program) first? 

Of cause, we would ignore / simplify several issues, but such an
environment would sufficient, to consider the following points:

-How does the network structure envolve, when many nodes are added?
(maybe we could visuialize this for a "small" amount of nodes)
-What's the possibility of generating nodes, that can't be removed from
the network without dividing it into 2 disjunctive parts?
-From which parameters does the point above depend?
-How does the propagation of queries amount the nodes work?
-...

I think such an environment would bring up / answer many interesting
questions related to graph theory, which will be important for us.

P.S: As i have to write some examinations in the next weeks, there will
be few activity until 11th februrary ;o(

Greetings,
Florian

===
"Irgendwie hab ich das schon mal erlebt. - Ach ja!
Gestern, vorgestern und eigentlich jeden Tag.
Das ist deprimierend." (Bernd das Brot)

GnuPG: 0x7E3055F3

* Marco Lange <ma...@ma...> [040124 12:37]:

Hi Marco,

> To what the discussion about the language is concerned I would propose 
> English so that non-german-speaking people who are interested in the 
> project can retrieve information from the mailing lists archives later 
> in the project.

 full ack!

> Yes :) IRC could be good for brainstorming sessions and so on. I am not 
> an IRC user at all, but we could use IRC as a means of quickly 
> discussion ideas and proposals. The disadvantage of IRC is, that, in 
> contrast to our wiki or the mailinglist, it is not archived. Thus, when 
> we should happen to have lengthier important discussion on IRC, we 
> should sum up the main results in the Wiki and send the IRC log to the 
> mailinglist for archival.

 Ok. I'm already in #gnugle at freenode. Hopefully you'll join also.

18:10 -!- con-fuse [~dk...@pc...] has joined #gnugle
18:10 -!- ServerMode/#gnugle [+n] by saberhagen.freenode.net
18:10 [Users #gnugle]

> BTW: The mailinglist can be configured to set a Reply-To header to the 
> mailinglist. Shall I do this?

 I personally use mutt as MUA and this list is configured as mailinglist
in my .muttrc. When I wanna reply to the list I just press shift-L and the
reply to is set correctly. So if someone sets a follow-up header it will
also send CC to the one. In short: If you use a good and rfc compatible
MUA this should be no problem and not needed to change the Reply-To
header.

peace,
 daniel

-- 
Daniel K. Gebhart                               http://dkg.con-fuse.org/
Institute for Computer Science                   University of Innsbruck
Fon: (office): +43 512 507 6444     Mail: <daniel.gebhart(at)uibk.ac.at>

* Marco Lange <mai...@ma...> [040124 12:45]:

Hi Marco,

> Oh interesting, I will have to read this. I just proposed Ruby because I 
> know it better than Python and I personally like it more. But that is 
> only a personal preference, and I have nothing at all against Python.

 I think python get's more and more important to the opensource community
and a lot of new stuff is developed in python. I can't see this effect in
ruby. But I've to say, that I know too less about ruby.

peace,
 daniel

-- 
Daniel K. Gebhart                               http://dkg.con-fuse.org/
Institute for Computer Science                   University of Innsbruck
Fon: (office): +43 512 507 6444     Mail: <daniel.gebhart(at)uibk.ac.at>

On Sat, Jan 24, 2004 at 12:38:45PM +0100, Marco Lange wrote:

> BTW: The mailinglist can be configured to set a Reply-To header to the 
> mailinglist. Shall I do this?

Yes!

Greetings
Florian

===
"Irgendwie hab ich das schon mal erlebt. - Ach ja!
Gestern, vorgestern und eigentlich jeden Tag.
Das ist deprimierend." (Bernd das Brot)

GnuPG: 0x7E3055F3

Hi!

>>Ist doch perfekt, da wir so oder so vorhatten, zumindest am Anfang viele
>>Features in Skriptsprachen zu implementieren. Im Wiki hatten wir diesbez?glich
>>mal ein paar Gedanken ausgetauscht und an Ruby gedacht.
> 
>  Ich denke auch, dass wir uns da gut ergaenzen koennen werden. ;) Btw.
> wuerde ich Python statt Ruby vorziehen.  
> 
> http://c2.com/cgi/wiki?PythonVsRuby

Oh interesting, I will have to read this. I just proposed Ruby because I 
know it better than Python and I personally like it more. But that is 
only a personal preference, and I have nothing at all against Python.

Best regards,
   Marco

Hi!

To what the discussion about the language is concerned I would propose=20
English so that non-german-speaking people who are interested in the=20
project can retrieve information from the mailing lists archives later=20
in the project.

>> Wie stehts mit IRC? Btw, sollten wir in der ML englisch schreiben?
>=20
> Also, ich f=FCr meinen Teil nutze (bisher) eigentlich keinen IRC. Thema
> englisch: Ich halte es eigentlich f=FCr ganz sinnvoll, die ganze
> "offizielle" Kommunikation auf englisch zu betreiben.=20

Yes :) IRC could be good for brainstorming sessions and so on. I am not=20
an IRC user at all, but we could use IRC as a means of quickly=20
discussion ideas and proposals. The disadvantage of IRC is, that, in=20
contrast to our wiki or the mailinglist, it is not archived. Thus, when=20
we should happen to have lengthier important discussion on IRC, we=20
should sum up the main results in the Wiki and send the IRC log to the=20
mailinglist for archival.

BTW: The mailinglist can be configured to set a Reply-To header to the=20
mailinglist. Shall I do this?

CU,
   Marco

* Florian Bremer <Flo...@rw...> [040124 12:19]:

Hi Florian,

> Ist doch perfekt, da wir so oder so vorhatten, zumindest am Anfang viele
> Features in Skriptsprachen zu implementieren. Im Wiki hatten wir diesbez=
=FCglich
> mal ein paar Gedanken ausgetauscht und an Ruby gedacht.

 Ich denke auch, dass wir uns da gut ergaenzen koennen werden. ;) Btw.
wuerde ich Python statt Ruby vorziehen. =20

http://c2.com/cgi/wiki?PythonVsRuby

> Also, ich f=FCr meinen Teil nutze (bisher) eigentlich keinen IRC. Thema

 sollten wir das aendern? :)

> englisch: Ich halte es eigentlich f=FCr ganz sinnvoll, die ganze
> "offizielle" Kommunikation auf englisch zu betreiben.=20

 Ich auch. Beginnen wir damit im naechsten Thread. :)

> =3D=3D=3D

 Deine Sig. Trennung ist kaputt und sollte idealerweise "-- " lauten.

lieben Gruss,
 daniel

--=20
Daniel K. Gebhart                               http://dkg.con-fuse.org/
Institute for Computer Science                   University of Innsbruck
Fon: (office): +43 512 507 6444     Mail: <daniel.gebhart(at)uibk.ac.at>

On Sat, Jan 24, 2004 at 11:50:20AM +0100, Daniel K. Gebhart wrote:

> ...
>=20
>  Bei mir sieht die Sache vermutlich gerade umgekehrt aus. :) Von C =
habe
> ich fast gar keine Ahnung, dafuer beschaeftige ich mich aber viel m=
it
> Scriptsprachen, die einem in der Systemadministration zur Hand gehe=
n
> koennen. (Shell, Perl, Python,..).=20

Ist doch perfekt, da wir so oder so vorhatten, zumindest am Anfang vi=
ele
Features in Skriptsprachen zu implementieren. Im Wiki hatten wir dies=
bez=FCglich
mal ein paar Gedanken ausgetauscht und an Ruby gedacht.

> Linux verwende ich nun seit.. ahm..
> mal ueberlegen.. seit ca. 5 Jahren oder so. Ich bin sehr aktiv im D=
ebian
> Projekt und Crux (CLC Maintainer). Beruflich habe ich mich hauptsae=
chlich
> mit Solaris rumzuschlagen.
>=20
> ...
>=20
>  Wie stehts mit IRC? Btw, sollten wir in der ML englisch schreiben?

Also, ich f=FCr meinen Teil nutze (bisher) eigentlich keinen IRC. The=
ma
englisch: Ich halte es eigentlich f=FCr ganz sinnvoll, die ganze
"offizielle" Kommunikation auf englisch zu betreiben.=20

<vorgeschobener Grund>
Dann wird unser Projekt viel attraktiver f=FCr potentielle Entwickler=
 aus
dem Ausland.
</vorgeschobener Grund>

<echter Grund>
Ich muss endlich meine Sprachkenntnisse wieder ein wenig auffrischen =
;o)
</echter Grund>

Gru=DF
Florian

=3D=3D=3D
"Irgendwie hab ich das schon mal erlebt. - Ach ja!
Gestern, vorgestern und eigentlich jeden Tag.
Das ist deprimierend." (Bernd das Brot)

GnuPG: 0x7E3055F3

* Marco Lange <ma...@ma...> [040123 20:36]:

Hi Marco,

> Unsere Kenntnisse sind im Bereich UNIX-Programmierung noch recht gering,=
=20
> ich habe lediglich erste Gehversuche hinter mir, daf=FCr beherrsche ich=
=20
> Ruby ziemlich gut und ein wenig Python. C beherrschen wir aber beide=20
> ganz gut. Wir beide sind seit etwa 12 - 18 Monaten vollst=E4ndige=20
> Linuxuser, vorher habe ich zumindest etwa 2 Jahre parallel Linux laufen=
=20
> gehabt, zwischendurch auch mal FreeBSD.

 Bei mir sieht die Sache vermutlich gerade umgekehrt aus. :) Von C habe
ich fast gar keine Ahnung, dafuer beschaeftige ich mich aber viel mit
Scriptsprachen, die einem in der Systemadministration zur Hand gehen
koennen. (Shell, Perl, Python,..). Linux verwende ich nun seit.. ahm..
mal ueberlegen.. seit ca. 5 Jahren oder so. Ich bin sehr aktiv im Debian
Projekt und Crux (CLC Maintainer). Beruflich habe ich mich hauptsaechlich
mit Solaris rumzuschlagen.

> PS: Kommunikation mit uns ambesten =FCber die Mailingliste von SF oder=20
> =FCber die Adresse von der ich sende und Florians Adresse im Cc.

 Wie stehts mit IRC? Btw, sollten wir in der ML englisch schreiben?

lieben Gruss,
 daniel

--=20
Daniel K. Gebhart                               http://dkg.con-fuse.org/
Institute for Computer Science                   University of Innsbruck
Fon: (office): +43 512 507 6444     Mail: <daniel.gebhart(at)uibk.ac.at>

gnugle-developers Mailing List for Gnugle

gnugle-developers — Discussions about the development of Gnugle