Re: [Waterken-server] k2v bugfix

[Stephen C. <js...@gm...>]

Hi Tyler,
Yes we(*) saw short writes here. This is the Java version we're using
(64-bit, Linux)

java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)

I also found this Hadoop discussion, in which they accept a patch for a
"writeFully" method: https://issues.apache.org/jira/browse/HDFS-3531

(Incidentally I've had a look at the OpenJDK source and couldn't see any
reason why short writes couldn't occur, either.)

The background to all this is that we're investigating scaling Waterken and
vat storage on disk was an issue for us. Since k2v seemed to be on the
horizon, we did a Cassandra implementation for it.

We then did a quick spike to write an adapter between n2v and k2v (during
the course of which we saw the aforementioned issue). We also noticed
several cases where File objects were used to hold Vat hierarchies: we
added an abstraction to get away from this.

We're also not sure that we're using the K2V abstractions correctly: do you
intend for there to be a single K2V object in the system or one per
vat/folder?

However we're not particularly happy with this solution, and ultimately of
course, we'd like to be using the official version of Waterken, with k2v
baked in. Would you be able to say when this might happen?

Thanks,
Steve

(*) I'm working with Hunter Kelly, who has already asked a few questions on
the list.

On 7 August 2012 06:31, Tyler Close <tyl...@gm...> wrote:

> Hi Stephen,
>
> It looks like you're suggesting that the code should loop on the
> FileChannel.write() call, instead of throwing when all of the data is
> not written. AFAICT, the spec for FileChannel.write() is that all of
> the data is supposed to be written on the first call and a loop is not
> required. In particular, the docs for the write() method say:
>
> "Unless otherwise specified, a write operation will return only after
> writing all of the r requested bytes"
>
> And, AFAICT, the FileChannel docs don't specify otherwise. Have you
> come across a use case where you've seen the FileChannel do a short
> write?
>
> BTW, I'm curious who the "we" are and what you're using the K2V code
> for. Can you share?
>
> Thanks,
> --Tyler
>
> On Fri, Aug 3, 2012 at 4:06 AM, Stephen Crane <js...@gm...> wrote:
> > Hi Tyler,
> > Here's a small bugfix to Trie.Update.commit() which fixes short writes
> (we
> > replaced the "else" clause):
> >
> >             if (DoubleSyncInsteadOfChecksum == checksumValue) {
> >                 if (address - SizeOfChecksum - SizeOfSeparator -
> > startAddress !=
> >                         out.write(nodes, 0, nodes.length - 2)) {
> >                     throw new IOException();
> >                 }
> >                 out.force(false);
> >                 if (SizeOfSeparator + SizeOfChecksum !=
> >                         out.write(nodes, nodes.length - 2, 2)) {
> >                     throw new IOException();
> >                 }
> >             } else {
> >                 long n = address - startAddress;
> >                 while (n > 0) {
> >                     n -= out.write(nodes);
> >                 }
> >             }
> >
> > (Apologies for the lack of patch, the editor reformatted your code.)
> >
> > Cheers,
> > Steve
> >
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Waterken-server mailing list
> > Wat...@li...
> > https://lists.sourceforge.net/lists/listinfo/waterken-server
> >
>
>
>
> --
> "Waterken News: Capability security on the Web"
> http://waterken.sourceforge.net/recent.html
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Waterken-server mailing list
> Wat...@li...
> https://lists.sourceforge.net/lists/listinfo/waterken-server
>