xlattice-devel — For XLattice developers

[XLattice-devel] util-0.3 release

[Jim D. <jd...@di...>]

This release adds support for Web Start (JNLP) version numbers and brings
XLattice Version class in line with this, to allow XLattice components to
be distributed using JNLP.

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure

[XLattice-devel] from a discussion of a p2p IRC network

[Jim D. <jd...@di...>]

On Fri, 20 Aug 2004, *** wrote:

> Nothing written down yet but I can give you a summary. Remember one of the
> main points is to keep the protocol simple and light but also be able to
> scale fairly well. I want clones very easy to make. This also means that

What is a 'clone' in this context?

> it's not going to be very secure. I'm not really trying to stop all
> adversaries, just trying to raise the cost. There's not going to be any form
> of identity, any one can pretend to be anyone else. It's going to be pretty
> lawless but it will be equal.
>
> Messages
>
>   The are 2 types of messages. Network status messages and chat messages.
> They are different in how they are handled but they use the same header.
>
>   Msg Header
>     Id: GUID; id of this message
>     Target: GUID; 2 uses, id of the node that sent the message if it's a
> status message (GUID) OR it's a hash of channel/nick name of recepient if
> it's a chat message (MD5)

>     Cost: HASH; this is a special hash that is used to limit the rate that
> messages can be created, it takes a few seconds to create this hash but very
> little time to check it (think hashcash)

Burning CPU won't bother anyone with significant resources.  Rule of
thumb is that every feature added doubles the implementation cost of
the system.  This particular feature has no obvious benefit, but will
certainly introduce bugs, add quite signficantly to build costs, and
irritate users.

>     Flags: byte; this tells what type of message it is,
> Link/Delink/Keepalive (not routed)/Chat
>     TTL: byte; status messages have a max TTL of 3, chat messages have
> infinite TTL (255)
>     Length: dword; length of the data following this header
>
> Routing
>
>   Routing is only done in small groups, each node only knows about it's
> neighbors up to 3 hops away. To inform each of the neighbors of a connection
> or disconnection a status message is sent to all nodes with a TTL of 3. The
> reason for this is that I am using Efa flooding algorithm, it helps cut down
> the number duplicate messages. Each message is only routed by a node once,
> duplicates are dropped.

There are several EFAs, acronym is a problem.  I assume you mean
Juang Li's.

Her Efa is part of a larger proposal, ESCP, "An Efficient
Clustered Super-Peer Architecture for P2P Networks".  That is,
it's a server-assisted peer-to-peer network, like XLattice in fact.

She claims to have benchmarked the algorithm and found it efficient,
but publishes no source code.  She also claims that the algorithm
will be efficient in an environment without superpeers, but doesn't
justify this assertion at all.

From what I can see from reading the literature, Efa has no way
of coping with random leaves and joins, especially the sort of
frequent leaves and joins that can be expected if impatient users
are involved.  Specifically, when people leave and you aren't aware
of it, you won't forward some messages because you "know" that the
message is going to be routed by the node that left.

Also, from a fairly casual look at the proposal it appears that traffic
will be distributed very unevenly, with low-index nodes bearing a
disproportionate share of the load.  If node IDs are self-assigned,
people will inevitably start gaming the system.

An evaluation of Efa would involve a lot more thinking and also knocking
up a simulation.  Certainly the approach would have to be modified to make
it practical.  I would guess that this might be worth doing.  That is, Efa
does not appear to be a working solution, but it looks like it is
reasonably close to something that might work.

>   Link Header
>     Id: GUID; id of the node from which this was received
>     Address: dword; ip address of sender
>     Port: word; port of sender
>
> Link messages are like a traceroute. The originating node sends out a link
> message with 0 data. Each node that forwards it adds the link header of the
> last node and passes it on. This means that by the time the message reaches
> a TTL of 0 it can have info of up to 3 different nodes.
>
> sender node > milddle node > middle node > end node
>
> These messages are used to build a map of the surronding network with the
> current node has the central point. This map is used with the Efa algorithm
> to predict duplicate messages and prevent them.

If there are no superpeers, how do you join the network?  How do you
locate other nodes?  If you do, why should they reply to your messages?
If they do, what stops you from DOSing them?

In Li Juang's system, a joining node is assigned a set of neighbors.
It contacts the neighbors, and they reply with neighbor tables.  The
new node collects these tables, waits, then sends its neighbors its
own neighbor tables so that they can recompute.  All very civilized --
and very fragile.

> Delink messages are the opposite of a link message except that only the id
> is used for each node because the other information (address/port) is
> already known.
>
> Keepalives are just empty headers used to keep the TCP connection alive
> across routers/proxies that timeout after so long with no data. They are
> only sent when no other messages have been sent in the last 30 seconds and
> are not routed or take up any resources.

Why 30 seconds?  Is this tunable?  If so, how?

> Chatting
>
>   The style is similar to IRC. People select a nick and join a channel to

Select a nick??  How do you avoid collisions?  Who defines channels?

> chat as a group or they can privately (not very private) message each other.
> To keep the channels/nicknames seperate the name of the channel is hashed
> and used as a 128bit key for the text to be encrypted with AES. The key is
> then hashed again and sent along with the encrypted text. Only people that
> know the channel name, aka "joined", are able to read the text. Nicknames
> are merely private channels, but anyone that knows the nickname can also
> read any messages received by that nickname. This could be overcome with
> private/public keys but that's out of scope for my purposes, remember I'm
> just trying to raise the cost of my adversaries and the network is designed
> to be more free than trusted.

IRC and similar things (XMPP) are server-centric systems.  I am certain
that it is possible to respecify IRC and XMPP to operate with server-
assisted p2p networks.  I think that it's possible to do this if some of
the p2p nodes involved form a Byzantine arbiter.  (So you have a p2p
network in which some self-selected subset of the nodes also act as
Byzantine servers; no other type of servers would seem to be absolutely
necessary.)  I don't see how it's possible on a pure p2p network.

Freenet tries to operate this way.  All peers are equal. The result is
chaos, or you might want to say that it is lawless but equal ;-)  People
who make their machines available as Freenet nodes find that they are
constantly churning their disks and choking their network connections but
very little of any value trickles through the network.  Part of this is
caused by people with a sense of humour who continuously dump trash into
the network, part by poor design, in particular a design which provides no
defense against users, neither the malicious nor the incompetent.

IRC or XMPP involves a significant amount of state.  Any p2p network
design for such a protocol must begin by devising a way for sharing and
maintaining that state across the network.  As far as I can see, this
inevitably involves some subset of the nodes forming a Byzantine cluster.
So the key design issue is how to create a stable, long-lived Byzantine
cluster out of independent nodes which the designer must assume are
operated by the malicious and the incompetent.

Probably a better design would involve a loose confederation of such
clusters, but you can't have that until you have a working, stable,
long-lived Byzantine cluster.

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure

[XLattice-devel] projmgr-0.2 release

[Jim D. <jd...@di...>]

This is a bugfix release, bringing XLattice's project management
component up to date with the other two released components,
util-0.2 and corexml-0.2.

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure

[XLattice-devel] corexml-0.2 release

[Jim D. <jd...@di...>]

This release fixes some minor bugs in XLattice's core XML facility
and adds the implementation of the corexml.bind package, which enables
programmers to automate the creation of Java objects from XML input files.

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure

[XLattice-devel] util-0.2 release

[Jim D. <jd...@di...>]

This is a minor bugfix release, one of a set of simultaneous
releases of XLattice components (util-0.2, corexml-0.2, and projmgr-0.2).

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure

[XLattice-devel] projmgr-0.1 release

[Jim D. <jd...@di...>]

This is a preliminary release of XLattice's ProjMgr component,
used for management of software development of XLattice and
similar multi-components.  It depends upon the recently released
util-0.1 and corexml-0.1 components.

The release is packaged as a zip file which unzips into a ./xlattice
subdirectory.  In addition to packages included with other
components, ProjMgr depends upon the Antlr lexer/parser generator,
which is included with the distribution.

Linux/FreeBSD/UNIX users can run unit tests by typing

  cd xlattice/projmgr
  ./build.sh test

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure

[XLattice-devel] corexml-0.1 release

[Jim D. <jd...@di...>]

This is a preliminary release of the CoreXml component, containing
XML facilities to be used by most other XLattice components.  It
depends upon the recently released util-0.1 module.

The release is packaged as a zip file which unzips into a ./xlattice
subdirectory.  Necessary jars, including Ant and JUnit, are included
in xlattice/lib.  The corexml-0.1 release jar is in xlattice/lib/xlattice.
Linux/FreeBSD/UNIX users can run unit tests by typing

  cd xlattice/corexml
  ./build.sh test

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure

[XLattice-devel] util-0.1 release

[Jim D. <jd...@di...>]

This is a preliminary release of utilities to be used by most or all
XLattice components, including the Version superclass and the
cmdline package.

The release is packaged as a zip file which unzips into a ./xlattice
subdirectory.  Necessary jars, including Ant and JUnit, are included
in xlattice/lib.  The util-0.1 release jar is in xlattice/lib/xlattice.
Linux/FreeBSD/UNIX users can run unit tests by typing

  cd xlattice/util
  ./build.sh test

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure

[XLattice-devel] FASD project: Online survey launched

[<ben...@id...>]

Dear Open Source developer

I am doing a research project on "Fun and Software Development" in which I kindly invite you to participate.
You will find the online survey under http://fasd.ethz.ch/qsf/. The questionnaire consists of 53 questions and you will need about 15 minutes to complete it.

With the FASD project (Fun and Software Development) we want to define the motivational significance of fun when software developers decide to engage in Open Source projects. What is special about our research project is that a similar survey is planned with software developers in commercial firms. This procedure allows the immediate comparison between the involved individuals and the conditions of production of these two development models. Thus we hope to obtain substantial new insights to the phenomenon of Open Source Development.


With many thanks for your participation,
Benno Luthiger


PS:
The results of the survey will be published under http://www.isu.unizh.ch/fuehrung/blprojects/FASD/.
We have set up the mailing list fa...@we... for this study. Please see http://fasd.ethz.ch/qsf/mailinglist_en.html for registration to this mailing list.

_______________________________________________________________________

Benno Luthiger
Swiss Federal Institute of Technology Zurich
8092 Zurich

Mail: benno.luthiger(at)id.ethz.ch
_______________________________________________________________________

[XLattice-devel] Re: XLattice (Re: building a better Free net)

[Jim D. <jd...@di...>]

On Wed, 12 Nov 2003, [iso-8859-1] Some Guy wrote:

> If someone sics an army of zombie winblows machines on your webserver
> to take it down, you could say he was censoring you.  Ebay and
> Microsoft got hit for example.  A P2P network can distribute nodes and
> data so that most people can't figuar out where the data is or who has
> it.
>
> Security in P2P is very hard, but doable.

Any specific ideas?

> > > The MNet crowd has this:
> > > http://mnet.sourceforge.net/EGTPv1_Architecture.txt
> >
> > Well, I'll read their documentation in the morning, but at a quick glance
> > it's quite different.  It's a single shared file system without anonymity
> > and it doesn't look very modular.  The expectation is that XLattice will
> > be used for building a variety of networks and that a user can
> > simultaneously belong to more than one XLattice network.
>
> MNet has the modual, the Evil Genious Transfer Protocol (EGTP),
> they're thinking of breaking off from their main project.  I was
> suggesting you might want to use it for connectivity.  I don't know

EGTP is the Mojo Nation communications library.  It's designed for one
huge network organized around a central server (or servers).  It may be
written in Python; the documentation isn't all that clear.

I will download the software and have a look at it later this week.

> > > Maybe we'd like to use it for connectivity.  I'm not sure since it's
> > > in python.  My big requirement: it must use select() instead of 2
> > > threads per connection.  That just blows.
> > >
> > > > Ideally it would support > chat and a distributed file system
> > > directly.  So you could build a > real-time strategy game on top of
> > > it, or implement something like Freenet > over it.  4or5 times lag, is
> > > too much for most real time games.  You might be able to chat on it.
> >
> > What do you mean by 4 or 5 times lag?
>
> Oh, I was thinking you wanted to communicate through a hypercube.
> I'm hoping to make a DHT with about 4 or 5 hops max, which would be
> than most of the projects I know about.  To play shooters though you
> don't want anything to go more than 1 hop.

The idea is to use the hypercube for fairly distributing broadcast
traffic.  Where there is a reason for node-node communications, it
should be direct.  One hop.

For broadcast, the assumption is that the sender has peers or sets
up connections with peers in sub-hypercubes.  So if you have an
M+N=D-dimensional hypercube, the message would be copied to 2^M
peers, each of whom would be responsible for an N-dimensional sub-
hypercube.  They would distribute the message in the same fashion,
sending it to 2^M peers, each responsible for a smaller hypercube,
and so on, recursively.

> PThreads and Berkely Socket Functions work in win32 and unix, and
> probably just about everything.
>
> > See http://xlattice.sourceforge.net
> > It's a draft, written this evening.
>
> Ok, looking at that it seems like you're tring to do a lot of stuff
> I'm not.  I'm not saying we can't work together; it just seems like
> you're talking about doing all kinds of stuff.  There are tons of hard
> problems to solve in P2P, you probably don't want take them all on at
> once.

The idea is to build a low-level infrastructure that will support a
variety of applications.  The infrastructure should be simple, although
it can be elaborated over time.

One of the problems with Freenet appears to be a lack of layering.  That
is, the various parts of the code should be cleanly separated, but aren't:
it's all mixed together.  If XLattice is done right, it should be possible
to implement Freenet on top of it.  The lower level code, the XLattice
bit, would be fully tested and reliable -- and have no knowledge at all of
Freenet.  This in itself would make the Freenet clone more reliable.

Care to elaborate on your other ideas for making a better Freenet?

> Have you heard of Waste?  I haven't tried it, but it's supposed to be
> a nice P2P app for chatting and file sharing between friends.  That
> sounds like some of what you're wanting to do.

I finally found it on the Web.  There appears to be a bit of a copyright
problem ;-)

------------------------------------------------------------------------
NOTICE OF UNAUTHORIZED SOFTWARE

An unauthorized copy of Nullsoft's copyrighted software was briefly posted
on this website on or about Wednesday May 28, 2003. The software was
identified as "WASTE" (the "Software") and includes the files
"waste-setup.exe", "waste-source.zip", "waste-source.tar.gz" and any
additional files contained in these files.

Nullsoft is the exclusive owner of all right, title and interest in the
Software. The posting of the Software on this website was not authorized
by Nullsoft.
-------------------------------------------------------------------------

--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881

[XLattice-devel] test

[Jim D. <jd...@di...>]

[XLattice-devel] test 2

[Jim D. <jd...@di...>]