[Blocks-development] Filesharing concept discussion: limited resources, and queueing...

[Michael S. Jr. <m...@ms...>]

I was originally going to write up a long 'paper' about this...but I'm lazy
tonight.

I've had some ideas gnawing at me for the past while, and if any of you are
still out there...I'd like to bounce the ideas off of you guys before I try
to post some incoherent ramblings to infoanarchy or shouldexist or
something.

I'd like to talk about some potential features for filesharing systems:

1)  queues

Does anyone use fserv's on IRC?  I had started to write a technical
description of how IRC fserv's work, so a layperson or an academic could
understand them...someone let me know if you're interested in hearing it and
I'll finish and post it...

I think queuing or time-shifting file requests might help with the problem
of hosts getting swamped with requests, trying to maintain lots of slow
transfers and then failing them all.

I think it'd be nice to see a status of 'queued, 250 min until start'
instead of 'pending' or 'queued'.

This concept could probably also be applied to other services which were
originally designed to provide immediate service:  an ftp server could let a
transfer start but 'queue' it, sending data at 1 byte per 30 seconds, until
it's ready to start the transfer at full speed.

2)  less interactivity of service...batch service?

There's more client demand for data than server bandwidth available to
provide that data.  Servers will get swamped...and even though all users
have the expectation that their requests will all be handled immediately,
some requests will be handled slower than others, and some won't be handled
at all.

Also, there are different kinds of data on a network:  file advertisement or
search request data, network utility data, and file or content data.
Obviously network utility data should have higher priority than other types,
because the network is the transport the other types of data use.

Perhaps we should consider searches that take 24 hours to collect results,
but collect them in a more reliable way?

Perhaps we should consider 'server connections' to not equal TCP
connections?  What's wrong with a 'connection' that sends low volumes of
high-reliability traffic through e-mail, perhaps even using anonymous
remailers?  (Ideas:  with a *very* low volume of traffic, you could encode
your messages so they look human-readable.  With a mail host tolerant of
high volumes, you could have a more 'reliable' high-volume connection, with
a 10 MB buffer on the ISP's mail server, automatic dropping of inbound
connections with the buffer gets full, more tolerance of disconnects and
transient network conditions, etc.)

Perhaps some connections can be through DNS tunnels, etc.

3)  More human interaction

Once you have 'batch' communications, you can process and optimize the
communications in more effective ways...perhaps even involving a human in
some of the processing.

And once you have human interaction...some day in the future...maybe then my
crazy web-of-trust idea becomes valid.  As it was shown to me a while
back...currently there is no need for content sorting or filtering, no
matter how 'strong' the filtering might be.  There isn't a flood of content
out there right now.

What's everyone's thoughts on all this?

----

BTW, early February I accepted a position as a programmer at a credit card
processing company here in town.  I can't believe I was always afraid of
Windows programming -- now that I have (financial) motivation to sit down
and figure the stuff out, I'm doing great!

Perhaps it's time to reinvent Blocks...but this time with two or more
programmers.  ^_^

----

With the recent developments in OpenNap server operators coming under
attack, it's becoming clear:  no matter how legally sound your position is,
someone with power can still lash out at you if they can find out who you
are.

I've been thinking about how Blocks was designed.  If I'm remembering
correctly...if we don't readvertise files to new connections, and if you
make *sure* that you aren't connected to someone you don't trust at the time
you share a file, it's impossible for someone to pull a MediaEnforcer on the
Blocks network.

Think about that for a minute:  not even Freenet can claim that level of
immunity.  On Freenet, the act of requesting content (to check for its
presence) causes the content to be replicated to that machine.  There's some
legal protection there, although I'm not a lawyer.  However, the attacker
can merely go to the ISP and say "I represent this really big organization,
and we would appreciate it if you stopped this user from running this
perfectly legal service.  Besides, the user is consuming a much higher than
average portion of our upstream bandwidth, isn't it?  It'll save you money
in the long run to remove users like this from your service! *nudge nudge*"
A Freenet node can be shut down merely because it's a node on an *unpopular*
network.

Blocks doesn't have established port numbers.  With a little work, it could
even have a difficult-to-guess protocol, requiring either some 'hash cash'
or a bit of information that proves you learned about the IP and port
through the network, not through a host scan.

Ooh...wouldn't that be cool?  Your first connection to the first server
would require that you connect to a port, and without reading *any* data,
compute and send 50-million-operations worth of hash-cash to a 'mute'
socket, before the other end would even talk to you.  That would certainly
chill efforts to scan hosts for open ports, checking to see if any ports are
running Blocks nodes.

----

If my tone just changed...it's because within the last three paragraphs I
suddenly realized that Blocks is providing something that *no other service*
can provide right now.  It's enough to make me want to pick it up and dust
it off and start hacking again.

I just installed a new 40 GB hard disk.  Once I confirm that UDMA is OFF
(it's a CRC-skipping Western Digital...it was cheap) I'll put a 30 GB Blocks
cache up on michael.mspencer.net port 8017 ^_^

Have some anime.  Cowboy Bebop is definitely top-shelf anime, if you don't
mind reading subtitles.  And it has nothing at all to do with cowboys or
be-bop.  Ignore the funny name and give it a chance.  ^_^

----

In an attempt to make a 'real programmer' out of me, the CTO has had me
flowcharting our existing code.  For the past two weeks.  I think I'm
starting to like it.  *shudders*

I'm going to try to create a flowchart of the current version of Blocks.
Once I understand the code (and can help others understand the code, through
flowcharts), I'd like to start working on Blocks myself.  Probably starting
with a Windows-only ugly stepchild version.

(I can see it now -- instead of falling into the usual trap of making a full
dialog page for connection information, I'll be like Napster...but I'll say
"Disconnected"  "Kinda connected"  "Connected"  "Very connected")

That does it.  I need to go to sleep.  Please, let me know what you think
about both the broad filesharing system ideas and about dusting Blocks off
and standing it on its feet again.

Also...I've started hosting the Blocks homepage (kinda) at
http://www.mspencer.net/blocks

I'm missing the graphics though.  Can someone mail those to me, if you have
them?

--Spence
bl...@ms...