[Blocks-development] Future direction: data services need network services need netwo rks

[Michael S. <msp...@dp...>]

I'm a bottom-up design person.  It's difficult for me to think about high
and broad concepts without being able to see details.

Despite this, I would like to start a broad concept discussion, and then tie
the discussion into some immediate low-level network design tasks.

-----

High-level subject:  replacing all client/server tasks with peer-to-peer
service.

I could never quite put my finger on what was bothering me, back in 1995 and
1996.  I knew I didn't like the increasing corporate involvement and the
'tiered' structure the internet was adopting.  I never saw how that affects
peer-to-peer until recently.  That's what makes this technology so exciting.

The whole point of a p2p network is to distribute information...so the
services available on a p2p network must be information services of some
kind.

We've already had many different ideas for the kinds of service that can run
on a peer-to-peer network.  Filesharing is obvious.  Distributed chat and
message boards (moderated even) are also interesting.  A distributed
voice-over-IP gateway is even more interesting.

These are all 'data services'.  Each one uses several 'network services' in
a special way:

Napster uses:  centralized search and indexing, distributed file store, and
direct file data transfer.
Blocks uses:  distributed search and indexing, distributed file store, and
routed file data transfer.
An IRC file-trading room uses:  no content search or indexing, distributed
host directories, centralized file stores, and direct file data transfer.

And that's just filesharing, folks.  ^_^

-----

Mid-level subject:  defining a library of network services which can form
any arbitrary data service

I wonder if we can't create a taxonomy of the properties of a network
service, and then implement a wide range of network services to meet every
possible need.

What properties could a network service have?  Yesterday and the day before
I talked about a network service being 'batch' or 'interactive'.  I'll
clarify that here:

An interactive service is a service that is extremely sensitive to latency.
If someone were to implement an encrypted, tunneled VOIP gateway over a p2p
network, the traffic would be extremely sensitive to latency.  If a node on
the network tries to cache data and retransmit it at a time of lower demand,
several seconds or minutes of silence would be perceived on the telephone
conversation.  Similarly if a 'delayed transmission' trick were done in an
IRC-like chat room, people would be annoyed by the result.

A batch service is a service that isn't sensitive to latency at all.  If I'm
using a 28.8 modem and I'm trying to download a copy of
Crouching_Tiger_Hidden_Dragon.asf, I already know that the transfer is going
to take more than 24 hours.  If someone delays sending me data for a couple
hours (because I'm waiting in a queue, perhaps) then I won't mind that much.

(Both scenarios would actually be pretty silly -- no current p2p network can
guarantee the low latency required for tunneled VOIP, and it's difficult for
a network to guarantee that one large file will be successfully delivered
slowly over a 24-hour-plus period.  But we aren't done designing yet.  It
may be possible in the future for someone to pay a quantity of 'mojo' to all
hosts along a route to get 'guaranteed' data delivery service.)

FIXME:  Can someone help with the classification of other network service
types?  Remember that a network service is what you get when you break apart
an existing data service (Blocks, Gnutella, Napster, ICQ, Slashdot,
whatever) into its requisite parts.

Some other network services:  data storage, indexing, traffic forwarding,
cataloging and grouping, authentication, authorization.

These still need to be classified:  what kinds of data storage are there
(encrypted or not, short term or long term, etc.), what kind of indexing
(how trusted, how distributed), and so forth.

-----

Low-level subject:  expanding the types of network service available through
Blocks.

All of these things require a transport.  I feel (and humbly and
respectfully submit) that our short-term goals should be:

1)  Harden the network underpinnings of Blocks.  They're pretty well
hardened right now.  Testing, simulation, and analysis.
2)  Strengthen the one existing data service we have running on Blocks right
now:  file sharing.  Find weak points and vulnerabilities, scaling problems,
etc.  Don't try to bolt a brand new data service type on top of Blocks just
yet (like I was trying to do with the web-of-trust...sorry...).
3)  Complete an analysis of data service types and the required network
service types.  Drool over all the cool ideas that emerge from this
analysis.
4)  Incrementally design and implement network services, and turn them into
new and innovative data services as quickly as possible.  Once we start
doing this (if we ever get this far) we'll start attracting a lot of
attention.

-----

It is asking too much to think that just because I have a crazy idea and go
type an essay about it, everybody else on the mailing list will immediately
make time to absorb all that verbiage and type a reply the same day.  Sorry
for all the traffic, guys.

-----

We still have our page on Sourceforge.  (Did anyone else notice, humorously,
that www.sourceforget.net points to the Free Haven Project?  ^_^ )

PG, if you still remember your login for SourceForge, could you grant me
admin rights to the (dormant) project?  I already have developer rights
there...but I'd like to add and structure documentation and such.  I'll
figure out how the whole thing works if you'd like, and help people figure
out the site.  A quick link to the project page on SourceForge is:
http://sourceforge.net/projects/blocks

Just for the record...I do not consider myself to have any control
whatsoever over the Blocks project.  I just want to be our SourceForge
janitor/custodian.  :)

-----

When responding with thoughts, please try and keep in mind what 'level' your
thoughts are on.  If someone proposes "I think we should use Java" then
that's a low-level concept.  It's an implementation detail.  After some
introspection the person might realize that there was a high-level concept
he used to arrive at the low-level concept:  "I think the project should use
an open, interpreted language that doesn't require an expensive compiler.
Plus, I already know Java and would like to contribute."

That is...if you say "I think we should use Java" you might get some
resistance, because we haven't had enough high-level discussion to decide
what language would be ideal for the project.  If you say "I think we should
allow people to extend the client in arbitrary ways without needing special
programming experience or a compiler" and people agree, then the people
might be more receptive to the suggestion of Java.

Sorry if that seemed like I was preaching...

-----

Thank you *very* much Erik for giving me your time on IRC last night.  Most
of the collected and organized thought you've seen in this email wouldn't
have come together without the high-level discussion he and I had last
night.  (If my thoughts seem neither collected nor organized...blame it on
him.  ^_^ )

--Spence
bl...@ms...