prevayler-coders Mailing List for Prevayler
Brought to you by:
jsampson,
klauswuestefeld
Menu
▾
▴
prevayler-coders — People interested in the inner workings of the Prevayler project.
You can subscribe to this list here.
| 2004 |
Jan
|
Feb
|
Mar
|
Apr
(13) |
May
(83) |
Jun
(8) |
Jul
(18) |
Aug
(33) |
Sep
(132) |
Oct
(16) |
Nov
(32) |
Dec
(32) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2005 |
Jan
(35) |
Feb
(21) |
Mar
(7) |
Apr
(11) |
May
(18) |
Jun
|
Jul
(21) |
Aug
(14) |
Sep
(7) |
Oct
|
Nov
(16) |
Dec
|
| 2006 |
Jan
(15) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(57) |
Jul
|
Aug
(15) |
Sep
(3) |
Oct
(1) |
Nov
|
Dec
|
| 2007 |
Jan
|
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
(9) |
Jul
(7) |
Aug
|
Sep
(2) |
Oct
(1) |
Nov
|
Dec
(1) |
| 2008 |
Jan
|
Feb
(4) |
Mar
(10) |
Apr
(20) |
May
|
Jun
(5) |
Jul
(34) |
Aug
(2) |
Sep
|
Oct
(2) |
Nov
(2) |
Dec
|
| 2009 |
Jan
|
Feb
|
Mar
(5) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Klaus W. <kla...@gm...> - 2009-03-13 06:43:45
|
Nice :) On Thu, Mar 12, 2009 at 3:30 PM, Justin T. Sampson <ju...@kr...> wrote: > Okay -- Jacob, Jon, are you still around? Do you remember the steps you > took? > > I think I'll go ahead and do the "manual upload of artifacts" described here > to submit prevayler-2.3.jar: > > http://maven.apache.org/guides/mini/guide-central-repository-upload.html > > I'm actually starting to use Maven for building Prevayler itself (it really > helps with figuring out dependencies), so I'll be able to do the preferred > "sync'ing your own repository to the central repository automatically" > approach in the future. I guess I'll use the SourceForge web hosting space > as the intermediary repository since the Maven bot already has ssh access > there (basically, I have to use Maven to push releases to an arbitrary > repository that the central repository can then pull from). > > As for "prevayler" vs. "org.prevayler" for the groupId, it looks like they > now require the inverted-domain form for new projects, but old projects can > of course continue to use whatever groupId they've used before. I'll > definitely use "prevayler" for 2.3, but I'm considering switching to > "org.prevayler" to have kind of a clean starting point since I'll be using > Maven for the build and splitting out some subgroups and modules. > > Cheers, > Justin > > > On Wed, Mar 11, 2009 at 6:30 PM, Klaus Wuestefeld > <kla...@gm...> wrote: >> >> Jacob might have been involved too... >> >> On Wed, Mar 11, 2009 at 10:20 PM, Klaus Wuestefeld >> <kla...@gm...> wrote: >> > Jon Tirsen, IIRC >> > >> > >> > On Wed, Mar 11, 2009 at 8:49 PM, Justin T. Sampson <ju...@kr...> >> > wrote: >> >> Howdy, >> >> >> >> Who was involved in putting past Prevayler releases into the Maven >> >> central >> >> repository? I'd like to get 2.3 in there (and 2.4 when it's ready). >> >> >> >> I'm also curious (maybe I'll try to ask the Maven folks about this as >> >> well) >> >> about using "prevayler" vs. "org.prevayler" as the groupId. They seem >> >> to >> >> prefer the latter form these days, so I wonder if it's worth moving. >> >> >> >> Cheers, >> >> Justin >> >> >> >> -- >> >> Agile Focus - http://agilefocus.com/ >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are >> >> powering Web 2.0 with engaging, cross-platform capabilities. Quickly >> >> and >> >> easily build your RIAs with Flex Builder, the Eclipse(TM)based >> >> development >> >> software that enables intelligent coding and step-through debugging. >> >> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com >> >> _______________________________________________ >> >> Prevayler-coders mailing list >> >> Pre...@li... >> >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >> >> >> >> >> > >> >> >> ------------------------------------------------------------------------------ >> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are >> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and >> easily build your RIAs with Flex Builder, the Eclipse(TM)based development >> software that enables intelligent coding and step-through debugging. >> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com >> _______________________________________________ >> Prevayler-coders mailing list >> Pre...@li... >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > > > -- > Agile Focus - http://agilefocus.com/ > > ------------------------------------------------------------------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > |
|
From: Justin T. S. <ju...@kr...> - 2009-03-12 19:02:55
|
Okay -- Jacob, Jon, are you still around? Do you remember the steps you took? I think I'll go ahead and do the "manual upload of artifacts" described here to submit prevayler-2.3.jar: http://maven.apache.org/guides/mini/guide-central-repository-upload.html I'm actually starting to use Maven for building Prevayler itself (it really helps with figuring out dependencies), so I'll be able to do the preferred "sync'ing your own repository to the central repository automatically" approach in the future. I guess I'll use the SourceForge web hosting space as the intermediary repository since the Maven bot already has ssh access there (basically, I have to use Maven to push releases to an arbitrary repository that the central repository can then pull from). As for "prevayler" vs. "org.prevayler" for the groupId, it looks like they now require the inverted-domain form for new projects, but old projects can of course continue to use whatever groupId they've used before. I'll definitely use "prevayler" for 2.3, but I'm considering switching to "org.prevayler" to have kind of a clean starting point since I'll be using Maven for the build and splitting out some subgroups and modules. Cheers, Justin On Wed, Mar 11, 2009 at 6:30 PM, Klaus Wuestefeld <kla...@gm... > wrote: > Jacob might have been involved too... > > On Wed, Mar 11, 2009 at 10:20 PM, Klaus Wuestefeld > <kla...@gm...> wrote: > > Jon Tirsen, IIRC > > > > > > On Wed, Mar 11, 2009 at 8:49 PM, Justin T. Sampson <ju...@kr...> > wrote: > >> Howdy, > >> > >> Who was involved in putting past Prevayler releases into the Maven > central > >> repository? I'd like to get 2.3 in there (and 2.4 when it's ready). > >> > >> I'm also curious (maybe I'll try to ask the Maven folks about this as > well) > >> about using "prevayler" vs. "org.prevayler" as the groupId. They seem to > >> prefer the latter form these days, so I wonder if it's worth moving. > >> > >> Cheers, > >> Justin > >> > >> -- > >> Agile Focus - http://agilefocus.com/ > >> > >> > ------------------------------------------------------------------------------ > >> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > >> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > >> easily build your RIAs with Flex Builder, the Eclipse(TM)based > development > >> software that enables intelligent coding and step-through debugging. > >> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com > >> _______________________________________________ > >> Prevayler-coders mailing list > >> Pre...@li... > >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders > >> > >> > > > > > ------------------------------------------------------------------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > -- Agile Focus - http://agilefocus.com/ |
|
From: Klaus W. <kla...@gm...> - 2009-03-12 01:36:58
|
Jacob might have been involved too... On Wed, Mar 11, 2009 at 10:20 PM, Klaus Wuestefeld <kla...@gm...> wrote: > Jon Tirsen, IIRC > > > On Wed, Mar 11, 2009 at 8:49 PM, Justin T. Sampson <ju...@kr...> wrote: >> Howdy, >> >> Who was involved in putting past Prevayler releases into the Maven central >> repository? I'd like to get 2.3 in there (and 2.4 when it's ready). >> >> I'm also curious (maybe I'll try to ask the Maven folks about this as well) >> about using "prevayler" vs. "org.prevayler" as the groupId. They seem to >> prefer the latter form these days, so I wonder if it's worth moving. >> >> Cheers, >> Justin >> >> -- >> Agile Focus - http://agilefocus.com/ >> >> ------------------------------------------------------------------------------ >> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are >> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and >> easily build your RIAs with Flex Builder, the Eclipse(TM)based development >> software that enables intelligent coding and step-through debugging. >> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com >> _______________________________________________ >> Prevayler-coders mailing list >> Pre...@li... >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >> >> > |
|
From: Klaus W. <kla...@gm...> - 2009-03-12 01:20:54
|
Jon Tirsen, IIRC On Wed, Mar 11, 2009 at 8:49 PM, Justin T. Sampson <ju...@kr...> wrote: > Howdy, > > Who was involved in putting past Prevayler releases into the Maven central > repository? I'd like to get 2.3 in there (and 2.4 when it's ready). > > I'm also curious (maybe I'll try to ask the Maven folks about this as well) > about using "prevayler" vs. "org.prevayler" as the groupId. They seem to > prefer the latter form these days, so I wonder if it's worth moving. > > Cheers, > Justin > > -- > Agile Focus - http://agilefocus.com/ > > ------------------------------------------------------------------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > |
|
From: Justin T. S. <ju...@kr...> - 2009-03-11 23:55:24
|
Howdy, Who was involved in putting past Prevayler releases into the Maven central repository? I'd like to get 2.3 in there (and 2.4 when it's ready). I'm also curious (maybe I'll try to ask the Maven folks about this as well) about using "prevayler" vs. "org.prevayler" as the groupId. They seem to prefer the latter form these days, so I wonder if it's worth moving. Cheers, Justin -- Agile Focus - http://agilefocus.com/ |
|
From: Klaus W. <kla...@gm...> - 2008-11-02 01:12:19
|
Cool :) On Sat, Nov 1, 2008 at 8:09 PM, Justin T. Sampson <ju...@kr...> wrote: > Howdy Coders, > > I'd like to push for a release in the next month or two. I have modest > goals, focused on requests that have come across the mailing list in > the past several months, including a couple of submitted patches. See > below for a list. I'll leave some of my more controversial ideas (such > as exception handling and other internal rearchitecting) for later. > > By the way, Java 1.4 is officially dead as of yesterday: > http://java.sun.com/products/archive/eol.policy.html > > None of the features I have in mind will require Java 5.0, so it > doesn't matter much for this release. I'll go ahead and keep Prevayler > 2.4 compatible with Java 1.4, but I expect to ignore Java 1.4 > compatibility immediately after this release. > > I plan to build and test Prevayler 2.4 using Java 5.0 on Mac OS X and > also test using Java 1.4, Java 5.0, and Java 6 on Linux. I would > expect to drop Java 1.4 and add Java 7 on Linux for testing subsequent > releases. > > Here are the features I have in mind: > > 1. Don't hang the system when there's an IOException from the journal. > Prevayler should certainly stop handling requests, but should throw an > exception rather than hanging every thread that calls into it. > (Requested by Graham Barr on August 13. I already implemented this a > while ago -- commit e219167 on the "cleanup" branch -- so I just need > to merge it back into the "master" branch.) > > 2. Return the file location of the snapshot from takeSnapshot(). > (Patch submitted by William Pietri on April 17.) > > 3. Provide an API for cleaning up old journal files. (Requested by > Steven Benjamin on April 15.) > > 4. Read snapshot #0 on startup if it exists. (Requested by Achint > Srivastava on June 24.) > > 5. Fix an issue with serialization in Java 6 when a classloader is > specified. (Patch submitted by Tony Cabot on April 15.) > > Cheers, > Justin > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > |
|
From: Justin T. S. <ju...@kr...> - 2008-11-01 22:09:40
|
Howdy Coders, I'd like to push for a release in the next month or two. I have modest goals, focused on requests that have come across the mailing list in the past several months, including a couple of submitted patches. See below for a list. I'll leave some of my more controversial ideas (such as exception handling and other internal rearchitecting) for later. By the way, Java 1.4 is officially dead as of yesterday: http://java.sun.com/products/archive/eol.policy.html None of the features I have in mind will require Java 5.0, so it doesn't matter much for this release. I'll go ahead and keep Prevayler 2.4 compatible with Java 1.4, but I expect to ignore Java 1.4 compatibility immediately after this release. I plan to build and test Prevayler 2.4 using Java 5.0 on Mac OS X and also test using Java 1.4, Java 5.0, and Java 6 on Linux. I would expect to drop Java 1.4 and add Java 7 on Linux for testing subsequent releases. Here are the features I have in mind: 1. Don't hang the system when there's an IOException from the journal. Prevayler should certainly stop handling requests, but should throw an exception rather than hanging every thread that calls into it. (Requested by Graham Barr on August 13. I already implemented this a while ago -- commit e219167 on the "cleanup" branch -- so I just need to merge it back into the "master" branch.) 2. Return the file location of the snapshot from takeSnapshot(). (Patch submitted by William Pietri on April 17.) 3. Provide an API for cleaning up old journal files. (Requested by Steven Benjamin on April 15.) 4. Read snapshot #0 on startup if it exists. (Requested by Achint Srivastava on June 24.) 5. Fix an issue with serialization in Java 6 when a classloader is specified. (Patch submitted by Tony Cabot on April 15.) Cheers, Justin |
|
From: Klaus W. <kla...@gm...> - 2008-10-27 03:01:45
|
Let this be the official Prevayler repository. Point to it from the site. See you, Klaus. On Mon, Oct 27, 2008 at 12:01 AM, Justin T. Sampson <ju...@kr...> wrote: > Okay, folks, I've finally published my Git repository for Prevayler in > what I believe will be a useful manner: > > http://github.com/jsampson/prevayler > > GitHub has a very nice interface for browsing a repository, and makes > it really easy to create your own repositories which mirror official > ones and then submit patches to the maintainer. Read up on installing > and using Git here: > > http://github.com/guides > > In particular, you can use this easy process for submitting changes to > me for inclusion in the next release: > > http://github.com/guides/fork-a-project-and-submit-your-modifications > > This repository includes ALL the code from ALL prior repositories. > You'll see tags ranging from "v0.01.001" all the way up to "v2.3". > > The tip of the "master" branch is equivalent to the "v2.3" tag. The > other branches are as follows: > > "sourceforge_head": The tip of the SourceForge CVS repository at the > moment it was abandoned for Codehaus. > > "codehaus_head": The tip of the Codehaus CVS repository at the moment > it was abandoned for Subversion on prevayler.org. > > "bugfix": The branch created during Prevayler 2 development for > merging changes from the main branch for release. > > "cleanup": The post-2.3 changes I was working on a while ago, > immediately before any Java 5 code was introduced. > > "java5_experiment": Additional post-2.3 changes, including introducing > Java 5 generics. > > Cheers, > Justin > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > |
|
From: Justin T. S. <ju...@kr...> - 2008-10-27 02:01:38
|
Okay, folks, I've finally published my Git repository for Prevayler in what I believe will be a useful manner: http://github.com/jsampson/prevayler GitHub has a very nice interface for browsing a repository, and makes it really easy to create your own repositories which mirror official ones and then submit patches to the maintainer. Read up on installing and using Git here: http://github.com/guides In particular, you can use this easy process for submitting changes to me for inclusion in the next release: http://github.com/guides/fork-a-project-and-submit-your-modifications This repository includes ALL the code from ALL prior repositories. You'll see tags ranging from "v0.01.001" all the way up to "v2.3". The tip of the "master" branch is equivalent to the "v2.3" tag. The other branches are as follows: "sourceforge_head": The tip of the SourceForge CVS repository at the moment it was abandoned for Codehaus. "codehaus_head": The tip of the Codehaus CVS repository at the moment it was abandoned for Subversion on prevayler.org. "bugfix": The branch created during Prevayler 2 development for merging changes from the main branch for release. "cleanup": The post-2.3 changes I was working on a while ago, immediately before any Java 5 code was introduced. "java5_experiment": Additional post-2.3 changes, including introducing Java 5 generics. Cheers, Justin |
|
From: Justin T. S. <ju...@kr...> - 2008-08-14 15:10:13
|
I changed that to a different mechanism at some point -- but I think it was after the 2.3 release. I'll take a look later today to find the revision that contains the change. The idea is that if there's a failure in the journal in any thread, Prevayler needs to stop processing transactions from all threads. Someone's original attempt was to hang the thread that saw an exception while it still holds an exclusive lock within Prevayler, such that every other thread would also hang when trying to acquire that lock. The change I made was to set an error flag and then propogate the exception, such that any other thread could get in, see the error flag set, and throw an exception itself, allowing all the threads to cleanup other resources & carry on or quit. Cheers, Justin On 8/13/08, gb...@ty... <gb...@ty...> wrote: > Is there a design reason for the hang() method in PersistentJournal which > simply hangs up the current thread when it can't either create or append > to a journal? > > We find that on load testing, temporary resource failures ( ulimit > excessions and/or running out of disk-space ) cause threads performing > prevayler transactions to hang ( and thus to lock out all the resources > held by those threads ). > > Since the hang() method is obviously deliberate, it would suggest to me > that there is some utility in having a bunch of hung threads - does > prevayler provide some means of inspecting these and doing something with > them? > > Any help much appreciated, > > Graham > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > |
|
From: <gb...@ty...> - 2008-08-14 06:28:00
|
Is there a design reason for the hang() method in PersistentJournal which simply hangs up the current thread when it can't either create or append to a journal? We find that on load testing, temporary resource failures ( ulimit excessions and/or running out of disk-space ) cause threads performing prevayler transactions to hang ( and thus to lock out all the resources held by those threads ). Since the hang() method is obviously deliberate, it would suggest to me that there is some utility in having a bunch of hung threads - does prevayler provide some means of inspecting these and doing something with them? Any help much appreciated, Graham |
|
From: Bill B. <bi...@mo...> - 2008-07-07 17:46:07
|
Interesting. From what I know about Oracle, there's a tuning parameter that controls whether it syncs on each transaction (for a given connection), which is what we have (you can turn off batching completely if you want). I don't remember what our TPS rate was when we were tuning. We're using QDBM for our storage, which is extremely fast but represents only part of the persistence that Prevayler does (since a prevayler transaction accounts for changes to the object model as well as storage): http://qdbm.sourceforge.net/ We need to upgrade it to Tokyo Cabinet, but I haven't had time yet (http://tokyocabinet.sourceforge.net/). I guess you would classify our system as a type of application server. We've deployed a touch screen app in nursing homes with it and we are talking with some defense groups about various applications which are pretty diverse. Bill Klaus Wuestefeld wrote: > Prevayler batches transactions from different threads* and executes > them only after sync'd. > > I got 700 transactions** per second with 1500 threads running on a > 700MHz Celeron notebook some years ago. > > ** Each transaction created an object, updated another and deleted a third. > > Do you batch data coming from the same thread? > > I suppose you are doing more SCADA than OLTP, no? > > In that case you really dont need durability. > > See you, Klaus. > > > On Thu, Jul 3, 2008 at 8:20 PM, Bill Burdick <bi...@mo...> wrote: > >> Doesn't batching implicitly fail to guarantee durability anyway? >> >> Bill >> |
|
From: Klaus W. <kla...@gm...> - 2008-07-04 00:07:58
|
Prevayler batches transactions from different threads* and executes them only after sync'd. I got 700 transactions** per second with 1500 threads running on a 700MHz Celeron notebook some years ago. ** Each transaction created an object, updated another and deleted a third. Do you batch data coming from the same thread? I suppose you are doing more SCADA than OLTP, no? In that case you really dont need durability. See you, Klaus. On Thu, Jul 3, 2008 at 8:20 PM, Bill Burdick <bi...@mo...> wrote: > Doesn't batching implicitly fail to guarantee durability anyway? > > Bill > > > Klaus Wuestefeld wrote: >> >> Can you return to the user before the second sync is done? >> >> In your scheme, you guarantee consistency but not durability if you do >> that, I think. >> >> >> >> On Thu, Jul 3, 2008 at 12:24 PM, Bill Burdick <bi...@mo...> >> wrote: >> >>> >>> Sounds good. As for the journal in our system we could put in some >>> verification codes like that but since we can eliminate the second sync, >>> I >>> don't think it would really gain anything and it would lengthen the file >>> by >>> quite a lot (our entries can be very small). >>> >>> Bill >>> >>> >>> Justin T. Sampson wrote: >>> >>>> >>>> On Wed, Jul 2, 2008 at 2:42 PM, Bill Burdick <bi...@mo...> >>>> wrote: >>>> >>>> >>>> >>>>> >>>>> That works if you can reliably detect a corrupted batch. If garbage is >>>>> allowed, that may not be possible, especially if that garbage happens >>>>> to >>>>> be >>>>> old blocks reused from other journal files. Take the case where you >>>>> are >>>>> running lots of prevayler apps with very small journals. If there is a >>>>> failure, the allowed garbage might be a reused block from a different >>>>> journal file that would appear not to be corrupt, but would actually be >>>>> the >>>>> wrong data (if none of the "real" data made it to disk before the power >>>>> failure). [...] >>>>> >>>>> >>>> >>>> As of Prevayler 2.3, every transaction in the journal is preceded by a >>>> header giving the transaction sequence number, the transaction >>>> timestamp, and the size of the serialized transaction. >>>> >>>> For a simple truncation, without actual garbage data, the size of the >>>> serialized transaction guarantees detection. >>>> >>>> For the odd case of data from an old journal file somehow ending up in >>>> a corrupted journal file, the transaction sequence number guarantees >>>> detection. >>>> >>>> I could imagine adding a checksum to make it easier to detect real >>>> garbage data. That could be done in the journaling code or in the >>>> serializer -- the GZIPSerializer, for example, runs the gzip algorithm >>>> on each transaction, which I believe includes a checksum. Even as it >>>> is, I imagine there's a pretty good chance that even the default >>>> JavaSerializer would detect most kinds of garbage, as long as the >>>> garbage doesn't match the grammar of the Java serialized form. >>>> >>>> Cheers, >>>> Justin >>>> >>>> >>>> ------------------------------------------------------------------------- >>>> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >>>> Studies have shown that voting for your favorite open source project, >>>> along with a healthy diet, reduces your potential for chronic lameness >>>> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >>>> _______________________________________________ >>>> Prevayler-coders mailing list >>>> Pre...@li... >>>> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >>>> >>>> >>>> >>> >>> ------------------------------------------------------------------------- >>> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >>> Studies have shown that voting for your favorite open source project, >>> along with a healthy diet, reduces your potential for chronic lameness >>> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >>> _______________________________________________ >>> Prevayler-coders mailing list >>> Pre...@li... >>> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >>> >>> >>> >> >> ------------------------------------------------------------------------- >> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >> Studies have shown that voting for your favorite open source project, >> along with a healthy diet, reduces your potential for chronic lameness >> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >> _______________________________________________ >> Prevayler-coders mailing list >> Pre...@li... >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >> >> > > > ------------------------------------------------------------------------- > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > Studies have shown that voting for your favorite open source project, > along with a healthy diet, reduces your potential for chronic lameness > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > |
|
From: Bill B. <bi...@mo...> - 2008-07-03 23:18:00
|
Doesn't batching implicitly fail to guarantee durability anyway? Bill Klaus Wuestefeld wrote: > Can you return to the user before the second sync is done? > > In your scheme, you guarantee consistency but not durability if you do > that, I think. > > > > On Thu, Jul 3, 2008 at 12:24 PM, Bill Burdick <bi...@mo...> wrote: > >> Sounds good. As for the journal in our system we could put in some >> verification codes like that but since we can eliminate the second sync, I >> don't think it would really gain anything and it would lengthen the file by >> quite a lot (our entries can be very small). >> >> Bill >> >> >> Justin T. Sampson wrote: >> >>> On Wed, Jul 2, 2008 at 2:42 PM, Bill Burdick <bi...@mo...> >>> wrote: >>> >>> >>> >>>> That works if you can reliably detect a corrupted batch. If garbage is >>>> allowed, that may not be possible, especially if that garbage happens to >>>> be >>>> old blocks reused from other journal files. Take the case where you are >>>> running lots of prevayler apps with very small journals. If there is a >>>> failure, the allowed garbage might be a reused block from a different >>>> journal file that would appear not to be corrupt, but would actually be >>>> the >>>> wrong data (if none of the "real" data made it to disk before the power >>>> failure). [...] >>>> >>>> >>> As of Prevayler 2.3, every transaction in the journal is preceded by a >>> header giving the transaction sequence number, the transaction >>> timestamp, and the size of the serialized transaction. >>> >>> For a simple truncation, without actual garbage data, the size of the >>> serialized transaction guarantees detection. >>> >>> For the odd case of data from an old journal file somehow ending up in >>> a corrupted journal file, the transaction sequence number guarantees >>> detection. >>> >>> I could imagine adding a checksum to make it easier to detect real >>> garbage data. That could be done in the journaling code or in the >>> serializer -- the GZIPSerializer, for example, runs the gzip algorithm >>> on each transaction, which I believe includes a checksum. Even as it >>> is, I imagine there's a pretty good chance that even the default >>> JavaSerializer would detect most kinds of garbage, as long as the >>> garbage doesn't match the grammar of the Java serialized form. >>> >>> Cheers, >>> Justin >>> >>> ------------------------------------------------------------------------- >>> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >>> Studies have shown that voting for your favorite open source project, >>> along with a healthy diet, reduces your potential for chronic lameness >>> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >>> _______________________________________________ >>> Prevayler-coders mailing list >>> Pre...@li... >>> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >>> >>> >>> >> ------------------------------------------------------------------------- >> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >> Studies have shown that voting for your favorite open source project, >> along with a healthy diet, reduces your potential for chronic lameness >> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >> _______________________________________________ >> Prevayler-coders mailing list >> Pre...@li... >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >> >> >> > > ------------------------------------------------------------------------- > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > Studies have shown that voting for your favorite open source project, > along with a healthy diet, reduces your potential for chronic lameness > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > |
|
From: Justin T. S. <ju...@kr...> - 2008-07-03 19:52:42
|
Well, queries aren't supposed to modify the prevalent system at all,
so it's actually okay for a query to throw an Error. The system can
keep on running without experiencing inconsistency.
The only exception might be if you're doing some kind of lazy
computations inside queries -- conceptually read-only, but actually
read-write. In that case, queries must not run concurrently with each
other anyway.
-Justin
On Wed, Jul 2, 2008 at 11:57 PM, Klaus Wuestefeld
<kla...@gm...> wrote:
> Justin,
>
> Do you see any way of extending this "no inconsistency" guarantee for
> queries running in parallel when one of them throws an Error?
>
> Will some sort of gate (AtomicReference<Boolean>?), just before the
> queries return, work? Or might the scheduler play tricks on us?
>
> Klaus.
>
>
> On Thu, Jul 3, 2008 at 3:50 AM, Klaus Wuestefeld
> <kla...@gm...> wrote:
>>> There's a remaining bug -- while transactions will now be prevented
>>> from executing since the journal has been closed, queries are still
>>> allowed to execute against the corrupted prevalent system.
>>
>> Yup. Fixed below.
>>
>> Anything else? Is that it?
>>
>> Is this what 90% of all apps (that fit in RAM) out there need in terms
>> of persistence?
>>
>> Klaus
>>
>> --------------------
>>
>> import java.io.EOFException;
>> import java.io.IOException;
>>
>> public class PrevaylerJr {
>> private final Object _system;
>> private final AcidOutputStream _journal;
>>
>> public PrevaylerJr(Object initialState, String storageFile) throws Exception {
>> AcidInputStream input = new AcidInputStream(storageFile);
>> try {
>> initialState = input.readObject();
>>
>> while (true)
>> ((Command)input.readObject()).executeOn(initialState);
>>
>> } catch (EOFException expected) {}
>>
>> _system = initialState;
>> _journal = new AcidOutputStream(storageFile, _system);
>> }
>>
>> synchronized public Object executeTransaction(Command transaction)
>> throws Exception {
>> checkSystemConsistency();
>> _journal.append(transaction);
>> return tryToExecute(transaction);
>> }
>>
>> synchronized public Object executeQuery(Command query) {
>> checkSystemConsistency();
>> return tryToExecute(query);
>> }
>>
>> private Object tryToExecute(Command command) {
>> try {
>> return command.executeOn(_system);
>> } catch (Error err) {
>> _journal.close();
>> throw err;
>> }
>> }
>>
>> private void checkSystemConsistency() {
>> if (_journal.isClosed())
>> throw new IllegalStateException("An Error such as OutOfMemoryError
>> or StackOverflowError was thrown while executing some previous
>> transaction or query. The system might be in an inconsistent state.");
>> }
>>
>> public static interface Command {
>> Object executeOn(Object system);
>> }
>> }
>>
>> class AcidInputStream {
>>
>> public AcidInputStream(String fileName) throws IOException {}
>> public Object readObject() throws IOException {return null;}
>>
>> }
>>
>> class AcidOutputStream {
>>
>> /** Atomically and durably replaces the previous journal file with
>> the new entry using file rename manoeuvres.*/
>> public AcidOutputStream(String fileName, Object firstEntry) throws
>> IOException {}
>> public void append(Object entry) throws IOException {}
>> public void close() {}
>> public boolean isClosed() {return false;}
>>
>> }
>>
>
> -------------------------------------------------------------------------
> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> Studies have shown that voting for your favorite open source project,
> along with a healthy diet, reduces your potential for chronic lameness
> and boredom. Vote Now at http://www.sourceforge.net/community/cca08
> _______________________________________________
> Prevayler-coders mailing list
> Pre...@li...
> https://lists.sourceforge.net/lists/listinfo/prevayler-coders
>
|
|
From: Klaus W. <kla...@gm...> - 2008-07-03 18:54:35
|
Can you return to the user before the second sync is done? In your scheme, you guarantee consistency but not durability if you do that, I think. On Thu, Jul 3, 2008 at 12:24 PM, Bill Burdick <bi...@mo...> wrote: > Sounds good. As for the journal in our system we could put in some > verification codes like that but since we can eliminate the second sync, I > don't think it would really gain anything and it would lengthen the file by > quite a lot (our entries can be very small). > > Bill > > > Justin T. Sampson wrote: >> >> On Wed, Jul 2, 2008 at 2:42 PM, Bill Burdick <bi...@mo...> >> wrote: >> >> >>> >>> That works if you can reliably detect a corrupted batch. If garbage is >>> allowed, that may not be possible, especially if that garbage happens to >>> be >>> old blocks reused from other journal files. Take the case where you are >>> running lots of prevayler apps with very small journals. If there is a >>> failure, the allowed garbage might be a reused block from a different >>> journal file that would appear not to be corrupt, but would actually be >>> the >>> wrong data (if none of the "real" data made it to disk before the power >>> failure). [...] >>> >> >> As of Prevayler 2.3, every transaction in the journal is preceded by a >> header giving the transaction sequence number, the transaction >> timestamp, and the size of the serialized transaction. >> >> For a simple truncation, without actual garbage data, the size of the >> serialized transaction guarantees detection. >> >> For the odd case of data from an old journal file somehow ending up in >> a corrupted journal file, the transaction sequence number guarantees >> detection. >> >> I could imagine adding a checksum to make it easier to detect real >> garbage data. That could be done in the journaling code or in the >> serializer -- the GZIPSerializer, for example, runs the gzip algorithm >> on each transaction, which I believe includes a checksum. Even as it >> is, I imagine there's a pretty good chance that even the default >> JavaSerializer would detect most kinds of garbage, as long as the >> garbage doesn't match the grammar of the Java serialized form. >> >> Cheers, >> Justin >> >> ------------------------------------------------------------------------- >> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >> Studies have shown that voting for your favorite open source project, >> along with a healthy diet, reduces your potential for chronic lameness >> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >> _______________________________________________ >> Prevayler-coders mailing list >> Pre...@li... >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >> >> > > > ------------------------------------------------------------------------- > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > Studies have shown that voting for your favorite open source project, > along with a healthy diet, reduces your potential for chronic lameness > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > |
|
From: Bill B. <bi...@mo...> - 2008-07-03 15:27:33
|
Now those are some awesome TShirts!
Bill
Klaus Wuestefeld wrote:
> I'll also throw in a free "It's Good to be the King" Sovereign
> Computing TShirt while we're at it...
>
>
> On Thu, Jul 3, 2008 at 4:13 AM, Klaus Wuestefeld
> <kla...@gm...> wrote:
>
>> :)
>>
>> MVPs* in the PrevaylerJr game we are playing on this list will be
>> awarded a free TShirt with the entire PrevaylerJr code on it.
>>
>> No excuses: The entire code is only some 20 semicolons long and is below.
>>
>> Open issue:
>> - Implement AcidInputStream and AcidOutputStream.
>>
>> Extended play:
>> - How can we allow the AcidOutputStream to perform batched syncs
>> with minimum impact on PrevaylerJr's design?
>>
>> *Justin, I would say Bill Burdick has earned his TShirt already, no?
>>
>>
>> ----------------
>>
>>
>> import java.io.EOFException;
>> import java.io.IOException;
>>
>> public class PrevaylerJr {
>> private final Object _system;
>> private final AcidOutputStream _journal;
>>
>> public PrevaylerJr(Object initialState, String storageFile) throws Exception {
>> AcidInputStream input = new AcidInputStream(storageFile);
>> try {
>> initialState = input.readObject();
>>
>> while (true)
>> ((Command)input.readObject()).executeOn(initialState);
>>
>> } catch (EOFException expected) {}
>>
>> _system = initialState;
>> _journal = new AcidOutputStream(storageFile, _system);
>> }
>>
>> synchronized public Object executeTransaction(Command transaction)
>> throws Exception {
>> checkSystemConsistency();
>> _journal.append(transaction);
>> return tryToExecute(transaction);
>> }
>>
>> synchronized public Object executeQuery(Command query) {
>> checkSystemConsistency();
>> return tryToExecute(query);
>> }
>>
>> private Object tryToExecute(Command command) {
>> try {
>> return command.executeOn(_system);
>> } catch (Error err) {
>> _journal.close();
>> throw err;
>> }
>> }
>>
>> private void checkSystemConsistency() {
>> if (_journal.isClosed())
>> throw new IllegalStateException("An Error such as OutOfMemoryError
>> or StackOverflowError was thrown while executing some previous
>> transaction or query. The system might be in an inconsistent state.");
>> }
>>
>> public static interface Command {
>> Object executeOn(Object system);
>> }
>> }
>>
>> class AcidInputStream {
>>
>> public AcidInputStream(String fileName) throws IOException {}
>> public Object readObject() throws IOException {return null;}
>>
>> }
>>
>> class AcidOutputStream {
>>
>> /** Atomically and durably replaces the previous journal file with
>> the new entry using file rename manoeuvres.*/
>> public AcidOutputStream(String fileName, Object firstEntry) throws
>> IOException {}
>> public void append(Object entry) throws IOException {}
>> public void close() {}
>> public boolean isClosed() {return false;}
>>
>> }
>>
>>
>
> -------------------------------------------------------------------------
> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> Studies have shown that voting for your favorite open source project,
> along with a healthy diet, reduces your potential for chronic lameness
> and boredom. Vote Now at http://www.sourceforge.net/community/cca08
> _______________________________________________
> Prevayler-coders mailing list
> Pre...@li...
> https://lists.sourceforge.net/lists/listinfo/prevayler-coders
>
>
|
|
From: Bill B. <bi...@mo...> - 2008-07-03 15:26:38
|
Sounds good. As for the journal in our system we could put in some verification codes like that but since we can eliminate the second sync, I don't think it would really gain anything and it would lengthen the file by quite a lot (our entries can be very small). Bill Justin T. Sampson wrote: > On Wed, Jul 2, 2008 at 2:42 PM, Bill Burdick <bi...@mo...> wrote: > > >> That works if you can reliably detect a corrupted batch. If garbage is >> allowed, that may not be possible, especially if that garbage happens to be >> old blocks reused from other journal files. Take the case where you are >> running lots of prevayler apps with very small journals. If there is a >> failure, the allowed garbage might be a reused block from a different >> journal file that would appear not to be corrupt, but would actually be the >> wrong data (if none of the "real" data made it to disk before the power >> failure). [...] >> > > As of Prevayler 2.3, every transaction in the journal is preceded by a > header giving the transaction sequence number, the transaction > timestamp, and the size of the serialized transaction. > > For a simple truncation, without actual garbage data, the size of the > serialized transaction guarantees detection. > > For the odd case of data from an old journal file somehow ending up in > a corrupted journal file, the transaction sequence number guarantees > detection. > > I could imagine adding a checksum to make it easier to detect real > garbage data. That could be done in the journaling code or in the > serializer -- the GZIPSerializer, for example, runs the gzip algorithm > on each transaction, which I believe includes a checksum. Even as it > is, I imagine there's a pretty good chance that even the default > JavaSerializer would detect most kinds of garbage, as long as the > garbage doesn't match the grammar of the Java serialized form. > > Cheers, > Justin > > ------------------------------------------------------------------------- > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > Studies have shown that voting for your favorite open source project, > along with a healthy diet, reduces your potential for chronic lameness > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > |
|
From: Klaus W. <kla...@gm...> - 2008-07-03 07:18:39
|
I'll also throw in a free "It's Good to be the King" Sovereign
Computing TShirt while we're at it...
On Thu, Jul 3, 2008 at 4:13 AM, Klaus Wuestefeld
<kla...@gm...> wrote:
> :)
>
> MVPs* in the PrevaylerJr game we are playing on this list will be
> awarded a free TShirt with the entire PrevaylerJr code on it.
>
> No excuses: The entire code is only some 20 semicolons long and is below.
>
> Open issue:
> - Implement AcidInputStream and AcidOutputStream.
>
> Extended play:
> - How can we allow the AcidOutputStream to perform batched syncs
> with minimum impact on PrevaylerJr's design?
>
> *Justin, I would say Bill Burdick has earned his TShirt already, no?
>
>
> ----------------
>
>
> import java.io.EOFException;
> import java.io.IOException;
>
> public class PrevaylerJr {
> private final Object _system;
> private final AcidOutputStream _journal;
>
> public PrevaylerJr(Object initialState, String storageFile) throws Exception {
> AcidInputStream input = new AcidInputStream(storageFile);
> try {
> initialState = input.readObject();
>
> while (true)
> ((Command)input.readObject()).executeOn(initialState);
>
> } catch (EOFException expected) {}
>
> _system = initialState;
> _journal = new AcidOutputStream(storageFile, _system);
> }
>
> synchronized public Object executeTransaction(Command transaction)
> throws Exception {
> checkSystemConsistency();
> _journal.append(transaction);
> return tryToExecute(transaction);
> }
>
> synchronized public Object executeQuery(Command query) {
> checkSystemConsistency();
> return tryToExecute(query);
> }
>
> private Object tryToExecute(Command command) {
> try {
> return command.executeOn(_system);
> } catch (Error err) {
> _journal.close();
> throw err;
> }
> }
>
> private void checkSystemConsistency() {
> if (_journal.isClosed())
> throw new IllegalStateException("An Error such as OutOfMemoryError
> or StackOverflowError was thrown while executing some previous
> transaction or query. The system might be in an inconsistent state.");
> }
>
> public static interface Command {
> Object executeOn(Object system);
> }
> }
>
> class AcidInputStream {
>
> public AcidInputStream(String fileName) throws IOException {}
> public Object readObject() throws IOException {return null;}
>
> }
>
> class AcidOutputStream {
>
> /** Atomically and durably replaces the previous journal file with
> the new entry using file rename manoeuvres.*/
> public AcidOutputStream(String fileName, Object firstEntry) throws
> IOException {}
> public void append(Object entry) throws IOException {}
> public void close() {}
> public boolean isClosed() {return false;}
>
> }
>
|
|
From: Klaus W. <kla...@gm...> - 2008-07-03 07:13:23
|
:)
MVPs* in the PrevaylerJr game we are playing on this list will be
awarded a free TShirt with the entire PrevaylerJr code on it.
No excuses: The entire code is only some 20 semicolons long and is below.
Open issue:
- Implement AcidInputStream and AcidOutputStream.
Extended play:
- How can we allow the AcidOutputStream to perform batched syncs
with minimum impact on PrevaylerJr's design?
*Justin, I would say Bill Burdick has earned his TShirt already, no?
----------------
import java.io.EOFException;
import java.io.IOException;
public class PrevaylerJr {
private final Object _system;
private final AcidOutputStream _journal;
public PrevaylerJr(Object initialState, String storageFile) throws Exception {
AcidInputStream input = new AcidInputStream(storageFile);
try {
initialState = input.readObject();
while (true)
((Command)input.readObject()).executeOn(initialState);
} catch (EOFException expected) {}
_system = initialState;
_journal = new AcidOutputStream(storageFile, _system);
}
synchronized public Object executeTransaction(Command transaction)
throws Exception {
checkSystemConsistency();
_journal.append(transaction);
return tryToExecute(transaction);
}
synchronized public Object executeQuery(Command query) {
checkSystemConsistency();
return tryToExecute(query);
}
private Object tryToExecute(Command command) {
try {
return command.executeOn(_system);
} catch (Error err) {
_journal.close();
throw err;
}
}
private void checkSystemConsistency() {
if (_journal.isClosed())
throw new IllegalStateException("An Error such as OutOfMemoryError
or StackOverflowError was thrown while executing some previous
transaction or query. The system might be in an inconsistent state.");
}
public static interface Command {
Object executeOn(Object system);
}
}
class AcidInputStream {
public AcidInputStream(String fileName) throws IOException {}
public Object readObject() throws IOException {return null;}
}
class AcidOutputStream {
/** Atomically and durably replaces the previous journal file with
the new entry using file rename manoeuvres.*/
public AcidOutputStream(String fileName, Object firstEntry) throws
IOException {}
public void append(Object entry) throws IOException {}
public void close() {}
public boolean isClosed() {return false;}
}
|
|
From: Klaus W. <kla...@gm...> - 2008-07-03 06:57:38
|
Justin,
Do you see any way of extending this "no inconsistency" guarantee for
queries running in parallel when one of them throws an Error?
Will some sort of gate (AtomicReference<Boolean>?), just before the
queries return, work? Or might the scheduler play tricks on us?
Klaus.
On Thu, Jul 3, 2008 at 3:50 AM, Klaus Wuestefeld
<kla...@gm...> wrote:
>> There's a remaining bug -- while transactions will now be prevented
>> from executing since the journal has been closed, queries are still
>> allowed to execute against the corrupted prevalent system.
>
> Yup. Fixed below.
>
> Anything else? Is that it?
>
> Is this what 90% of all apps (that fit in RAM) out there need in terms
> of persistence?
>
> Klaus
>
> --------------------
>
> import java.io.EOFException;
> import java.io.IOException;
>
> public class PrevaylerJr {
> private final Object _system;
> private final AcidOutputStream _journal;
>
> public PrevaylerJr(Object initialState, String storageFile) throws Exception {
> AcidInputStream input = new AcidInputStream(storageFile);
> try {
> initialState = input.readObject();
>
> while (true)
> ((Command)input.readObject()).executeOn(initialState);
>
> } catch (EOFException expected) {}
>
> _system = initialState;
> _journal = new AcidOutputStream(storageFile, _system);
> }
>
> synchronized public Object executeTransaction(Command transaction)
> throws Exception {
> checkSystemConsistency();
> _journal.append(transaction);
> return tryToExecute(transaction);
> }
>
> synchronized public Object executeQuery(Command query) {
> checkSystemConsistency();
> return tryToExecute(query);
> }
>
> private Object tryToExecute(Command command) {
> try {
> return command.executeOn(_system);
> } catch (Error err) {
> _journal.close();
> throw err;
> }
> }
>
> private void checkSystemConsistency() {
> if (_journal.isClosed())
> throw new IllegalStateException("An Error such as OutOfMemoryError
> or StackOverflowError was thrown while executing some previous
> transaction or query. The system might be in an inconsistent state.");
> }
>
> public static interface Command {
> Object executeOn(Object system);
> }
> }
>
> class AcidInputStream {
>
> public AcidInputStream(String fileName) throws IOException {}
> public Object readObject() throws IOException {return null;}
>
> }
>
> class AcidOutputStream {
>
> /** Atomically and durably replaces the previous journal file with
> the new entry using file rename manoeuvres.*/
> public AcidOutputStream(String fileName, Object firstEntry) throws
> IOException {}
> public void append(Object entry) throws IOException {}
> public void close() {}
> public boolean isClosed() {return false;}
>
> }
>
|
|
From: Klaus W. <kla...@gm...> - 2008-07-03 06:49:56
|
> There's a remaining bug -- while transactions will now be prevented
> from executing since the journal has been closed, queries are still
> allowed to execute against the corrupted prevalent system.
Yup. Fixed below.
Anything else? Is that it?
Is this what 90% of all apps (that fit in RAM) out there need in terms
of persistence?
Klaus
--------------------
import java.io.EOFException;
import java.io.IOException;
public class PrevaylerJr {
private final Object _system;
private final AcidOutputStream _journal;
public PrevaylerJr(Object initialState, String storageFile) throws Exception {
AcidInputStream input = new AcidInputStream(storageFile);
try {
initialState = input.readObject();
while (true)
((Command)input.readObject()).executeOn(initialState);
} catch (EOFException expected) {}
_system = initialState;
_journal = new AcidOutputStream(storageFile, _system);
}
synchronized public Object executeTransaction(Command transaction)
throws Exception {
checkSystemConsistency();
_journal.append(transaction);
return tryToExecute(transaction);
}
synchronized public Object executeQuery(Command query) {
checkSystemConsistency();
return tryToExecute(query);
}
private Object tryToExecute(Command command) {
try {
return command.executeOn(_system);
} catch (Error err) {
_journal.close();
throw err;
}
}
private void checkSystemConsistency() {
if (_journal.isClosed())
throw new IllegalStateException("An Error such as OutOfMemoryError
or StackOverflowError was thrown while executing some previous
transaction or query. The system might be in an inconsistent state.");
}
public static interface Command {
Object executeOn(Object system);
}
}
class AcidInputStream {
public AcidInputStream(String fileName) throws IOException {}
public Object readObject() throws IOException {return null;}
}
class AcidOutputStream {
/** Atomically and durably replaces the previous journal file with
the new entry using file rename manoeuvres.*/
public AcidOutputStream(String fileName, Object firstEntry) throws
IOException {}
public void append(Object entry) throws IOException {}
public void close() {}
public boolean isClosed() {return false;}
}
|
|
From: Justin T. S. <ju...@kr...> - 2008-07-03 00:13:44
|
On Wed, Jul 2, 2008 at 2:42 PM, Bill Burdick <bi...@mo...> wrote: > That works if you can reliably detect a corrupted batch. If garbage is > allowed, that may not be possible, especially if that garbage happens to be > old blocks reused from other journal files. Take the case where you are > running lots of prevayler apps with very small journals. If there is a > failure, the allowed garbage might be a reused block from a different > journal file that would appear not to be corrupt, but would actually be the > wrong data (if none of the "real" data made it to disk before the power > failure). [...] As of Prevayler 2.3, every transaction in the journal is preceded by a header giving the transaction sequence number, the transaction timestamp, and the size of the serialized transaction. For a simple truncation, without actual garbage data, the size of the serialized transaction guarantees detection. For the odd case of data from an old journal file somehow ending up in a corrupted journal file, the transaction sequence number guarantees detection. I could imagine adding a checksum to make it easier to detect real garbage data. That could be done in the journaling code or in the serializer -- the GZIPSerializer, for example, runs the gzip algorithm on each transaction, which I believe includes a checksum. Even as it is, I imagine there's a pretty good chance that even the default JavaSerializer would detect most kinds of garbage, as long as the garbage doesn't match the grammar of the Java serialized form. Cheers, Justin |
|
From: Bill B. <bi...@mo...> - 2008-07-02 21:40:07
|
That works if you can reliably detect a corrupted batch. If garbage is allowed, that may not be possible, especially if that garbage happens to be old blocks reused from other journal files. Take the case where you are running lots of prevayler apps with very small journals. If there is a failure, the allowed garbage might be a reused block from a different journal file that would appear not to be corrupt, but would actually be the wrong data (if none of the "real" data made it to disk before the power failure). The double sync with end markers guarantees detection in all of the cases we came up with, because the stuff you write is after the end marker and only after the first sync completes do you overwrite the previous end marker with a "noop" (which "activates" the new end marker). Actually, I guess you don't even need the second sync -- it's OK if it doesn't happen, because of the protection implicit in the technique. So I guess that makes this discussion kind of moot. Thanks -- that will improve performance! :) Performance has been very good. The journalled app is a switch from using Sqlite to QDBM. From what I remember (we tested performance in March), I believe the QDBM version was 10-12 times faster than the Sqlite version and journalling added a 25% overhead (so with journalling, it was roughly 7.5-8 times faster). I haven't tested single vs. double sync, because I need hard guarantees that we won't lose data during power loss, etc. and I haven't gotten a single sync method yet that guarantees that (unless I can verify the possibilities with journaling file systems and strict appending, etc.) By the way, we had originally used Prevayler for this, but the amount of data became too huge to use it. The laptops had 1G of RAM and there were many hundreds of megs of data -- that plus the other apps we had to run just wouldn't all fit in memory. Bill Klaus Wuestefeld wrote: > OK, but even if it does allow the last written batch to be garbage, > cant you simply ignore the last corrupted batch, instead of doing two > syncs? > > How does your double sync performance compare with single sync? > > See you, Klaus. > > > On Wed, Jul 2, 2008 at 2:09 PM, Bill Burdick <bi...@mo...> wrote: > >> I don't think it is possible for a crash to corrupt a part of the file which >> you are not changing, at least not with a journaling file system that >> preserves file metadata (size, etc). I'm using XFS and from what I hear, >> NTFS is pretty good -- I think NTFS only journals metadata. I'm not an >> expert in file systems, so I'm not sure what all of the guarantees are. My >> guess is that strict appending (like what prevayler does) probably gives you >> the best guarantees. >> >> Given that a file system that journals metadata and only strict appending, >> do power failures only truncate data and not introduce garbage? If so, I >> think this would provide all of the guarantees I need, because I could just >> write a length before each batch and verify with the remaining file size, >> which would remove the need for end markers and the extra sync. >> >> >> Bill >> >> >> >> >> Klaus Wuestefeld wrote: >> >>> Prevayler also syncs batches of transactions. They are only executed >>> after the sync for their batch completes. >>> >>> Prevayler detects corrupt batches and ignores the last one if it is >>> corrupt, as if it never had been written, which is ok, since its >>> transactions never got executed on the system in the first place. >>> >>> Why is it you cant simply ignore the last corrupt batch? >>> >>> A related issue: >>> Is it possible for a crash during an append to a file to corrupt the >>> last cluster, corrupting what had previously been completely syncd? >>> >>> See you, Klaus. >>> >>> >>> On Tue, Jul 1, 2008 at 8:36 PM, Bill Burdick <bi...@mo...> >>> wrote: >>> >>> >>>> Sync isn't atomic for system failures. A power failure can happen before >>>> a >>>> sync completes (and did many times with our system, because users were >>>> allowing unattended laptops to run out of charge during heavy updating). >>>> Because of this, if you take out the first sync and only use the last >>>> sync, >>>> and there IS a failure during the sync, there could be a "noop" followed >>>> by >>>> an incomplete journal record. The presence of a "noop" instead of an >>>> "end" >>>> indicates that the next record is known not to be corrupted (in our >>>> system, >>>> there is actually a batch of records after the "noop", not just a single >>>> record, so you don't have to sync for every record), so we only write it >>>> after the first sync completes. >>>> >>>> Bill >>>> >>>> >>>> Klaus Wuestefeld wrote: >>>> >>>> >>>>>> The reason for the two syncs is that you can't guarantee the >>>>>> order that dirty pages will be written to disk >>>>>> >>>>>> >>>>>> >>>>> Doesn't sync guarantee that all its dirty pages have been written? >>>>> >>>>> >>>>> ------------------------------------------------------------------------- >>>>> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >>>>> Studies have shown that voting for your favorite open source project, >>>>> along with a healthy diet, reduces your potential for chronic lameness >>>>> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >>>>> _______________________________________________ >>>>> Prevayler-coders mailing list >>>>> Pre...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >>>>> >>>>> >>>>> >>>>> >>>> ------------------------------------------------------------------------- >>>> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >>>> Studies have shown that voting for your favorite open source project, >>>> along with a healthy diet, reduces your potential for chronic lameness >>>> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >>>> _______________________________________________ >>>> Prevayler-coders mailing list >>>> Pre...@li... >>>> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >>>> >>>> >>>> >>>> >>> ------------------------------------------------------------------------- >>> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >>> Studies have shown that voting for your favorite open source project, >>> along with a healthy diet, reduces your potential for chronic lameness >>> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >>> _______________________________________________ >>> Prevayler-coders mailing list >>> Pre...@li... >>> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >>> >>> >>> >> ------------------------------------------------------------------------- >> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >> Studies have shown that voting for your favorite open source project, >> along with a healthy diet, reduces your potential for chronic lameness >> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >> _______________________________________________ >> Prevayler-coders mailing list >> Pre...@li... >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >> >> >> > > ------------------------------------------------------------------------- > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > Studies have shown that voting for your favorite open source project, > along with a healthy diet, reduces your potential for chronic lameness > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > |
|
From: Klaus W. <kla...@gm...> - 2008-07-02 17:16:10
|
OK, but even if it does allow the last written batch to be garbage, cant you simply ignore the last corrupted batch, instead of doing two syncs? How does your double sync performance compare with single sync? See you, Klaus. On Wed, Jul 2, 2008 at 2:09 PM, Bill Burdick <bi...@mo...> wrote: > I don't think it is possible for a crash to corrupt a part of the file which > you are not changing, at least not with a journaling file system that > preserves file metadata (size, etc). I'm using XFS and from what I hear, > NTFS is pretty good -- I think NTFS only journals metadata. I'm not an > expert in file systems, so I'm not sure what all of the guarantees are. My > guess is that strict appending (like what prevayler does) probably gives you > the best guarantees. > > Given that a file system that journals metadata and only strict appending, > do power failures only truncate data and not introduce garbage? If so, I > think this would provide all of the guarantees I need, because I could just > write a length before each batch and verify with the remaining file size, > which would remove the need for end markers and the extra sync. > > > Bill > > > > > Klaus Wuestefeld wrote: >> >> Prevayler also syncs batches of transactions. They are only executed >> after the sync for their batch completes. >> >> Prevayler detects corrupt batches and ignores the last one if it is >> corrupt, as if it never had been written, which is ok, since its >> transactions never got executed on the system in the first place. >> >> Why is it you cant simply ignore the last corrupt batch? >> >> A related issue: >> Is it possible for a crash during an append to a file to corrupt the >> last cluster, corrupting what had previously been completely syncd? >> >> See you, Klaus. >> >> >> On Tue, Jul 1, 2008 at 8:36 PM, Bill Burdick <bi...@mo...> >> wrote: >> >>> >>> Sync isn't atomic for system failures. A power failure can happen before >>> a >>> sync completes (and did many times with our system, because users were >>> allowing unattended laptops to run out of charge during heavy updating). >>> Because of this, if you take out the first sync and only use the last >>> sync, >>> and there IS a failure during the sync, there could be a "noop" followed >>> by >>> an incomplete journal record. The presence of a "noop" instead of an >>> "end" >>> indicates that the next record is known not to be corrupted (in our >>> system, >>> there is actually a batch of records after the "noop", not just a single >>> record, so you don't have to sync for every record), so we only write it >>> after the first sync completes. >>> >>> Bill >>> >>> >>> Klaus Wuestefeld wrote: >>> >>>>> >>>>> The reason for the two syncs is that you can't guarantee the >>>>> order that dirty pages will be written to disk >>>>> >>>>> >>>> >>>> Doesn't sync guarantee that all its dirty pages have been written? >>>> >>>> >>>> ------------------------------------------------------------------------- >>>> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >>>> Studies have shown that voting for your favorite open source project, >>>> along with a healthy diet, reduces your potential for chronic lameness >>>> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >>>> _______________________________________________ >>>> Prevayler-coders mailing list >>>> Pre...@li... >>>> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >>>> >>>> >>>> >>> >>> ------------------------------------------------------------------------- >>> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >>> Studies have shown that voting for your favorite open source project, >>> along with a healthy diet, reduces your potential for chronic lameness >>> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >>> _______________________________________________ >>> Prevayler-coders mailing list >>> Pre...@li... >>> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >>> >>> >>> >> >> ------------------------------------------------------------------------- >> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! >> Studies have shown that voting for your favorite open source project, >> along with a healthy diet, reduces your potential for chronic lameness >> and boredom. Vote Now at http://www.sourceforge.net/community/cca08 >> _______________________________________________ >> Prevayler-coders mailing list >> Pre...@li... >> https://lists.sourceforge.net/lists/listinfo/prevayler-coders >> >> > > > ------------------------------------------------------------------------- > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > Studies have shown that voting for your favorite open source project, > along with a healthy diet, reduces your potential for chronic lameness > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > _______________________________________________ > Prevayler-coders mailing list > Pre...@li... > https://lists.sourceforge.net/lists/listinfo/prevayler-coders > > |
