prevayler-discussion — General discussion about project Prevayler.

[Prevayler-discussion] CRaC - Coordinated Restore at Checkpoint

[Klaus W. <kla...@gm...>]

"CRaC effectively takes a snapshot of the Java process when it is fully
warmed up, then uses that snapshot to launch any number of JVMs from this
captured state. Not all existing Java programs can run without modification
as all resources need to be explicitly checkpointed and restored using the
CRaC API."

https://docs.azul.com/core/crac/crac-introduction

Re: [Prevayler-discussion] Well... I'm nearly running on java 17

[Rick R. <ri...@le...>]

Hi Justin,

While I certainly agree that a) it's a bit weird and b) it's kinda 
useful,  I think the right thing to do (in theory) would be to store 
gzips as test resources at different java version points.   Then you 
could get the same effect with simple end to end tests.

However, it is inconceivable to me that gzip formats are not backwards 
compatible.  So I presume that worst case, a manual unzip and rezip 
might be enough to recover but in reality, it should just work.   I 
don't have a 1.6 zipped journal to use.

BUT .. as I personally don't have old data to worry about, AND prevayler 
is not in terribly active development, I think there is some case to be 
made for just fixing the test (as I have done).

I would be interested in a survey of active implementations, and of 
course, I am only too happy to PR my changes, if maintainers feel like 
that is desired.

For the time being I am just building my own prevayler-core and factory 
and using that.


R



On 8/6/22 9:49 AM, Justin T. Sampson wrote:
> Hi Rick,
>
> Thanks for sharing your results!
>
> It looks like I wrote that test myself, 13 years ago. It's a bit weird 
> that I did a string comparison, since the assertion failure ends up 
> looking pretty weird.
>
> In a sense, we shouldn't really be asserting the exact contents of the 
> journal, since all we care about is that the transaction got 
> compressed somehow and can be recovered successfully.
>
> One hiccup with that is the fact that running GZIP on a small 
> transaction will tend to increase its size, not decrease it, so we 
> can't simply assert that the journal is smaller than it was without 
> GZIP. :) Maybe if we really care about it, we would generate a large 
> transaction and assert that it does, in fact, get smaller with GZIP.
>
> On the other hand, having an assertion for the exact contents of the 
> journal is kind of nice to catch changes like you're seeing, because 
> it's a little surprising the GZIP format would change. Hopefully it 
> is, at least, a backward-compatible change, because you wouldn't want 
> your old GZIP'd journals to become unrecoverable due to a later Java 
> upgrade.
>
> One approach might be to try fiddling with the settings of the 
> Deflater underlying the GZIPOutputStream that our GZIPSerializer class 
> is using. It's held in a protected field (def) on the output stream 
> instance, so it can be accessed and configured by extending the 
> GZIPOutputStream class (which could be done inline with an anonymous 
> class). According to the JDK docs, there were some fields added to 
> Deflater in Java 7, so maybe their defaults have changed. It's 
> possible that you could get it to behave like it used to, and leave 
> the assertion alone.
>
> Adding the extra character to the assertion might be okay, but my main 
> concern is that without understanding where the character comes from, 
> we'll end up with a test that depends on the Java version. It will 
> fail for all versions < N, for whatever version N introduced the 
> change that you're seeing.
>
> Cheers,
> Justin
>
>
>
>
> On Fri, Aug 5, 2022 at 5:20 PM Rick Ross via Prevayler-Discussion 
> <pre...@li...> wrote:
>
>     So I dusted off 2.7 and did some tweaking to get compiling.   I
>     converted broken tests to Junit5 with assert4J and let the legacy
>     engine handle the rest.
>
>     So far so good, but there is an error I'm not really following.
>
>     In
>
>     JournalSerializerTest.testCompressedJournal()
>
>     I get a single character error.
>
>     [...]
>
>
>
> _______________________________________________
> To unsubscribe go to the end of this page:http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" --http://www.prevayler.org

Re: [Prevayler-discussion] Well... I'm nearly running on java 17

[Justin T. S. <ju...@kr...>]

Hi Rick,

Thanks for sharing your results!

It looks like I wrote that test myself, 13 years ago. It's a bit weird that
I did a string comparison, since the assertion failure ends up looking
pretty weird.

In a sense, we shouldn't really be asserting the exact contents of the
journal, since all we care about is that the transaction got
compressed somehow and can be recovered successfully.

One hiccup with that is the fact that running GZIP on a small transaction
will tend to increase its size, not decrease it, so we can't simply assert
that the journal is smaller than it was without GZIP. :) Maybe if we really
care about it, we would generate a large transaction and assert that it
does, in fact, get smaller with GZIP.

On the other hand, having an assertion for the exact contents of the
journal is kind of nice to catch changes like you're seeing, because it's a
little surprising the GZIP format would change. Hopefully it is, at least,
a backward-compatible change, because you wouldn't want your old GZIP'd
journals to become unrecoverable due to a later Java upgrade.

One approach might be to try fiddling with the settings of the Deflater
underlying the GZIPOutputStream that our GZIPSerializer class is using.
It's held in a protected field (def) on the output stream instance, so it
can be accessed and configured by extending the GZIPOutputStream class
(which could be done inline with an anonymous class). According to the JDK
docs, there were some fields added to Deflater in Java 7, so maybe their
defaults have changed. It's possible that you could get it to behave like
it used to, and leave the assertion alone.

Adding the extra character to the assertion might be okay, but my main
concern is that without understanding where the character comes from, we'll
end up with a test that depends on the Java version. It will fail for all
versions < N, for whatever version N introduced the change that you're
seeing.

Cheers,
Justin




On Fri, Aug 5, 2022 at 5:20 PM Rick Ross via Prevayler-Discussion <
pre...@li...> wrote:

> So I dusted off 2.7 and did some tweaking to get compiling.   I converted
> broken tests to Junit5 with assert4J and let the legacy engine handle the
> rest.
>
> So far so good, but there is an error I'm not really following.
>
> In
>
> JournalSerializerTest.testCompressedJournal()
>
> I get a single character error.
>
> [...]
>

[Prevayler-discussion] Well... I'm nearly running on java 17

[Rick R. <ri...@le...>]

So I dusted off 2.7 and did some tweaking to get compiling.   I 
converted broken tests to Junit5 with assert4J and let the legacy engine 
handle the rest.

So far so good, but there is an error I'm not really following.

In

JournalSerializerTest.testCompressedJournal()

I get a single character error.


expected:

"1A;withQuery=false;systemVersion=1;executionTime=1000002
   ‹|bSHË,*.
ß½=â
   1B;withQuery=false;systemVersion=2;executionTime=1000004
   ‹|bS(NMÎÏK
(VU
   1A;withQuery=false;systemVersion=3;executionTime=1000006
   ‹|bS(ÉÈ,J
ìs~T
   "
  but was:

"1A;withQuery=false;systemVersion=1;executionTime=1000002
   ‹|bÿSHË,*.
ß½=â
   1B;withQuery=false;systemVersion=2;executionTime=1000004
   ‹|bÿS(NMÎÏK
(VU
   1A;withQuery=false;systemVersion=3;executionTime=1000006
   ‹|bÿS(ÉÈ,J
ìs~T
   "


It's not obvious at first, but lines 2,4 and 6 each have an extra character

expected:

   "
   ‹|bSHË,*.
ß½=â
   ‹|bS(NMÎÏK
(VU
   ‹|bS(ÉÈ,J
ìs~T
   "
  but was:

   "
   ‹|bÿSHË,*.
ß½=â
   ‹|bÿS(NMÎÏK
(VU
   ‹|bÿS(ÉÈ,J
ìs~T
   "

Notice that there is an extra
ÿ

after the |b

I'm toying with just throwing it in there and carrying.    Does anyone 
have any thoughts?   I thought maybe the compression algorithm bumped a 
version or something and that is what changed.   It sure looks like a 
header to me:


Anyway.. hope everyone is well.


Rick

Re: [Prevayler-discussion] Happy 20th Anniversary, Prevayler!

[Klaus W. <kla...@gm...>]

Wow! Thanks Justin! :D


On Sun, Oct 17, 2021 at 3:06 AM Justin T. Sampson <ju...@kr...>
wrote:

> According to my records, Klaus Wuestefeld released Prevayler version
> 00.01.001 as a zip file on October 16th, 2001, ushering in an era of high
> performance and productivity for us all. Happy 20th anniversary!
>
> Git didn't exist back then, but when we migrated to Git years later, I
> included all of the early releases as the initial commits for posterity:
>
>
> https://github.com/prevayler/prevayler/commits/master?after=f52736da5a2d7d87ef7a5cc4c2d0597df535b1cc+629&branch=master
>
> Cheers,
> Justin
>

[Prevayler-discussion] Happy 20th Anniversary, Prevayler!

[Justin T. S. <ju...@kr...>]

According to my records, Klaus Wuestefeld released Prevayler version
00.01.001 as a zip file on October 16th, 2001, ushering in an era of high
performance and productivity for us all. Happy 20th anniversary!

Git didn't exist back then, but when we migrated to Git years later, I
included all of the early releases as the initial commits for posterity:

https://github.com/prevayler/prevayler/commits/master?after=f52736da5a2d7d87ef7a5cc4c2d0597df535b1cc+629&branch=master

Cheers,
Justin

[Prevayler-discussion] Pravega

[Robert F. <ro...@de...>]

Hi everybody,

I recently discovered pravega.io, a stream storage platform. Browsing through the docs I noticed the StateSynchronizer:
http://pravega.io/docs/latest/state-synchronizer/

Same concept as prevayler but with some key differences. Anyone here have experience with pravega?

--
Robert Friberg
Devrex Labs
Memstate, OrigoDB

Re: [Prevayler-discussion] Chainvayler

[hakan e. <hak...@gm...>]

Thanks Marcel :)

Good question indeed, I was also thinking myself in what real life cases
replication only mode can be used.

Besides demo and testing purposes, I guess general answer is, to share data
among nodes which doesnt need to be persisted. Some cases comes to my mind:

* Shared state machine or configuration, assuming eventually consistent
nature of Chainvayler is acceptable.
* Some in memory event bus. Following the observer pattern, attach
listeners to Chained objects, so changes to one object in a node will be
notified to other nodes. This can be a quite handy way of sharing events,
without an external dependency like AWS SNS.

Best,
Hakan



On Wed, Apr 29, 2020 at 1:53 PM Marcel Oerlemans <mar...@gm...>
wrote:

> Hi Hakan,
>
> Love the work and the write-up.
> Was wondering: why would one want to disable the persistence part? (not
> criticizing, just wondering when that would be wanted, when using chain- or
> prevayler)
>
> Met vriendelijke groet,
> Marcel Oerlemans
>
>
> On Sat, Apr 25, 2020 at 1:43 PM hakan eryargi <hak...@gm...>
> wrote:
>
>> Thanks Rick :)
>>
>> Those are two very different projects and might be better delivered as
>>> separate things, even though you can turn them off independently
>>
>>
>> I'm targeting enterprise domain and they need both. Postvayler,
>> predecessor of Chainvayler, was doing transparent persistence. But
>> enterprise need scalability or at least redundancy. So it was on the back
>> of my mind for long time for how to replicate this. Either one
>> (replication/persistence) has their own specific use cases, possibly
>> persistence a lot more, like all desktop applications can benefit from it,
>> but for the overall goal, eliminating databases, we need  replication and
>> persistence at the same time.
>>
>> Adoption and interest are so hard to generate.
>>
>>
>> So true indeed.
>>
>> It feels like many people don’t have the vision or look deeper, or maybe
>> I’m very biased and exaggerating the possible value.
>>
>> What could you build, dirt simple, that would get people excited about
>>> the possibility.
>>
>>
>> I was thinking similar, so a few years ago spent some time and adapted
>> Spring's famous "pet clinic" sample to Postvayler. It integrates with
>> Spring's load time weaving mechanism and persistence root can be
>> @Autowired.
>> https://github.com/raftAtGit/spring-petclinic
>>
>> Many additional things are possible here. For example, autowiring Chained
>> non root objects based on JsonPath (or XPath) like syntax. For example:
>>
>> @RequestMapping("/deposit/{accountId}")
>> void depositMoney(
>>     @PathVariable int accountId,
>>     @Path("$.bank.account[{accountId}]") Account account) {
>>
>>        // account with accountId will be injected. as long as Bank class has a Map, getAccount(int) method or similar.
>>        account.deposit(amount);
>> }
>>
>> But these are just additional benefits. Not directly related to
>> Postvayler/Chainvayler. That's why I didnt even mention them in the blog.
>>
>> I believe, if it ever happens, the breaking point will happen when some
>> large and/or famous company uses Chainvayler in a real life project, and
>> proves it can actually be used to eliminate databases, provides much better
>> performance and reduce development costs.
>>
>> Best,
>> Hakan
>>
>>
>> On Fri, Apr 24, 2020 at 7:44 PM Rick Ross <ri...@le...> wrote:
>>
>>> I like the concept a lot.   Isn't the dream to just annotate objects for
>>> persistence and whatever rules you need to apply?     To marry that to a
>>> shared model across JVMs is very interesting.
>>>
>>>
>>> My first thoughts are :
>>>
>>> Those are two very different projects and might be better delivered as
>>> separate things, even though you can turn them off independently
>>>
>>> Adoption and interest are so hard to generate.  Without them, it's just
>>> a personal project.   I mean no offense by that, but success breeds
>>> success.  We've all been there.  What could you build, dirt simple, that
>>> would get people excited about the possibility.
>>>
>>>     My first thought there is the classic LAMP stack needs one less
>>> letter, right?
>>>
>>>     It strikes me that with persistence managed entirely as objects, it
>>> wouldn't be to hard to autogenerate basic crud REST calls.   Once you have
>>> annotated your object, it is automatically REST available, with full crud.
>>>
>>>
>>>     Or perhaps a connector that could work with Angular and provide safe
>>> persistent storage, right off the bat.
>>>
>>>
>>> Just spitballing here.   I'll keep tabs on this though.  It's
>>> interesting.
>>>
>>>
>>> Rick
>>>
>>>
>>>
>>> On 4/22/20 11:35 PM, hakan eryargi wrote:
>>>
>>> Hey guys,
>>>
>>> Remember my PoC project *Postvayler* many years ago? It was making
>>> POJOs persistent transparently based on @annotations and of course
>>> Prevayler.
>>>
>>> Recently I took the idea one step forward and extended the mechanism to
>>> also replicate POJOs among JVMs.
>>>
>>> Here is my blog post about it:
>>>
>>> https://medium.com/@hakan.eryargi/transparent-replication-and-persistence-for-pojo-graphs-805d9ae0f776
>>>
>>>
>>> Here is the project *Chainvayler* itself:
>>> https://github.com/raftAtGit/Chainvayler
>>>
>>> Personally I find the idea really promising :)
>>>
>>> Cheers,
>>> Hakan (r a f t)
>>>
>>> PS: I'm somehow got out from the email list, re-registered myself.
>>>
>>> On Mon, Dec 2, 2013 at 12:12 AM hakan eryargi <hak...@gm...>
>>> wrote:
>>>
>>>> i'm experimenting with the idea. so far not bad. just put what I've
>>>> done to GitHub. any suggestions, critics and contributions are welcome:)
>>>> https://github.com/raftAtGit/Postvayler
>>>>
>>>> apologies for the name, it sounded like a nice word play:)
>>>>
>>>> to run the sample, run the Compiler class
>>>> with raft.postvayler.samples.bank.Bank argument and then run
>>>> raft.postvayler.samples.bank.Main
>>>>
>>>> how it works:
>>>> compiler injects bytecode to root class and all @Persistent classes
>>>> that can be accesible* from root.
>>>>
>>>> root class contains an injected object pool, where each @Persistent
>>>> class is assigned a Long id and put into this object pool when created.
>>>> this pool will be* a WeakValueMap so objects which are not attached to root
>>>> will be garbage collected at some time.
>>>>
>>>> each @Persist method in a @Persistent class is instrumented like this:
>>>> @Persist
>>>> ReturnType doSomething(someParams) {..}
>>>>
>>>> doSomehing is copied to __postvayler__doSomething(..). the original doSomehing
>>>> becomes:
>>>>
>>>> ReturnType doSomething(someParams) {
>>>>     if (!there is postvayler context)
>>>>         return __postvayler__doSomething(..);
>>>>     if (we are in a prevayler transaction)
>>>>         return __postvayler__doSomething(..);
>>>>     return prevayler.execute(new MethodTransaction(..));
>>>> }
>>>>
>>>> the arguments to doSomehing(..) is transformed in MethodTransaction such
>>>> that references to @Persistent objects are replaced with Reference's,
>>>> which will later be restored from object pool. seems as this also solves
>>>> the Baptism problem.
>>>>
>>>> looks good to me for now:)
>>>>
>>>> cheers,
>>>> *r a f t*
>>>>
>>>> * not implemented yet
>>>>
>>>> On Wed, Nov 27, 2013 at 11:07 AM, Naveen Chawla <nav...@gm...>
>>>> wrote:
>>>>
>>>>> Sure, here:
>>>>> http://docs.oracle.com/javase/tutorial/java/generics/methods.html
>>>>>
>>>>> As for consistency, any multi-changes explicitly synchronized over
>>>>> prevalentObject can be auto-batched.
>>>>>
>>>>>
>>>>> On 27 November 2013 00:25, Klaus Wuestefeld <ma...@kl...> wrote:
>>>>>
>>>>>> > It is ordinary generic method syntax.
>>>>>>
>>>>>> Could you point me to some doc on this syntax? I couldnt find it here:
>>>>>> http://docs.oracle.com/javase/tutorial/extra/generics/methods.html
>>>>>>
>>>>>> > The beginning and end of a transaction would simply be any change
>>>>>> to any
>>>>>> > non-transient field.
>>>>>>
>>>>>> Suppose a transaction needs to update two fields to be correct but the
>>>>>> system crashes after the first. Will the system not be inconsistent on
>>>>>> recovery?
>>>>>>
>>>>>> Klaus
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Rapidly troubleshoot problems before they affect your business. Most
>>>>>> IT
>>>>>> organizations don't have a clear picture of how application
>>>>>> performance
>>>>>> affects their revenue. With AppDynamics, you get 100% visibility into
>>>>>> your
>>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>>>> AppDynamics Pro!
>>>>>>
>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>>>> _______________________________________________
>>>>>> To unsubscribe go to the end of this page:
>>>>>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>>>>> _______________________________________________
>>>>>> "Databases in Memoriam" -- http://www.prevayler.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>>>> organizations don't have a clear picture of how application performance
>>>>> affects their revenue. With AppDynamics, you get 100% visibility into
>>>>> your
>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>>> AppDynamics Pro!
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> To unsubscribe go to the end of this page:
>>>>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>>>> _______________________________________________
>>>>> "Databases in Memoriam" -- http://www.prevayler.org
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>> _______________________________________________
>>> "Databases in Memoriam" -- http://www.prevayler.org
>>>
>>> _______________________________________________
>> To unsubscribe go to the end of this page:
>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>> _______________________________________________
>> "Databases in Memoriam" -- http://www.prevayler.org
>>
> _______________________________________________
> To unsubscribe go to the end of this page:
> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
>

Re: [Prevayler-discussion] Chainvayler

[Marcel O. <mar...@gm...>]

Hi Hakan,

Love the work and the write-up.
Was wondering: why would one want to disable the persistence part? (not
criticizing, just wondering when that would be wanted, when using chain- or
prevayler)

Met vriendelijke groet,
Marcel Oerlemans


On Sat, Apr 25, 2020 at 1:43 PM hakan eryargi <hak...@gm...>
wrote:

> Thanks Rick :)
>
> Those are two very different projects and might be better delivered as
>> separate things, even though you can turn them off independently
>
>
> I'm targeting enterprise domain and they need both. Postvayler,
> predecessor of Chainvayler, was doing transparent persistence. But
> enterprise need scalability or at least redundancy. So it was on the back
> of my mind for long time for how to replicate this. Either one
> (replication/persistence) has their own specific use cases, possibly
> persistence a lot more, like all desktop applications can benefit from it,
> but for the overall goal, eliminating databases, we need  replication and
> persistence at the same time.
>
> Adoption and interest are so hard to generate.
>
>
> So true indeed.
>
> It feels like many people don’t have the vision or look deeper, or maybe
> I’m very biased and exaggerating the possible value.
>
> What could you build, dirt simple, that would get people excited about the
>> possibility.
>
>
> I was thinking similar, so a few years ago spent some time and adapted
> Spring's famous "pet clinic" sample to Postvayler. It integrates with
> Spring's load time weaving mechanism and persistence root can be
> @Autowired.
> https://github.com/raftAtGit/spring-petclinic
>
> Many additional things are possible here. For example, autowiring Chained
> non root objects based on JsonPath (or XPath) like syntax. For example:
>
> @RequestMapping("/deposit/{accountId}")
> void depositMoney(
>     @PathVariable int accountId,
>     @Path("$.bank.account[{accountId}]") Account account) {
>
>        // account with accountId will be injected. as long as Bank class has a Map, getAccount(int) method or similar.
>        account.deposit(amount);
> }
>
> But these are just additional benefits. Not directly related to
> Postvayler/Chainvayler. That's why I didnt even mention them in the blog.
>
> I believe, if it ever happens, the breaking point will happen when some
> large and/or famous company uses Chainvayler in a real life project, and
> proves it can actually be used to eliminate databases, provides much better
> performance and reduce development costs.
>
> Best,
> Hakan
>
>
> On Fri, Apr 24, 2020 at 7:44 PM Rick Ross <ri...@le...> wrote:
>
>> I like the concept a lot.   Isn't the dream to just annotate objects for
>> persistence and whatever rules you need to apply?     To marry that to a
>> shared model across JVMs is very interesting.
>>
>>
>> My first thoughts are :
>>
>> Those are two very different projects and might be better delivered as
>> separate things, even though you can turn them off independently
>>
>> Adoption and interest are so hard to generate.  Without them, it's just a
>> personal project.   I mean no offense by that, but success breeds success.
>> We've all been there.  What could you build, dirt simple, that would get
>> people excited about the possibility.
>>
>>     My first thought there is the classic LAMP stack needs one less
>> letter, right?
>>
>>     It strikes me that with persistence managed entirely as objects, it
>> wouldn't be to hard to autogenerate basic crud REST calls.   Once you have
>> annotated your object, it is automatically REST available, with full crud.
>>
>>
>>     Or perhaps a connector that could work with Angular and provide safe
>> persistent storage, right off the bat.
>>
>>
>> Just spitballing here.   I'll keep tabs on this though.  It's interesting.
>>
>>
>> Rick
>>
>>
>>
>> On 4/22/20 11:35 PM, hakan eryargi wrote:
>>
>> Hey guys,
>>
>> Remember my PoC project *Postvayler* many years ago? It was making POJOs
>> persistent transparently based on @annotations and of course Prevayler.
>>
>> Recently I took the idea one step forward and extended the mechanism to
>> also replicate POJOs among JVMs.
>>
>> Here is my blog post about it:
>>
>> https://medium.com/@hakan.eryargi/transparent-replication-and-persistence-for-pojo-graphs-805d9ae0f776
>>
>>
>> Here is the project *Chainvayler* itself:
>> https://github.com/raftAtGit/Chainvayler
>>
>> Personally I find the idea really promising :)
>>
>> Cheers,
>> Hakan (r a f t)
>>
>> PS: I'm somehow got out from the email list, re-registered myself.
>>
>> On Mon, Dec 2, 2013 at 12:12 AM hakan eryargi <hak...@gm...>
>> wrote:
>>
>>> i'm experimenting with the idea. so far not bad. just put what I've done
>>> to GitHub. any suggestions, critics and contributions are welcome:)
>>> https://github.com/raftAtGit/Postvayler
>>>
>>> apologies for the name, it sounded like a nice word play:)
>>>
>>> to run the sample, run the Compiler class
>>> with raft.postvayler.samples.bank.Bank argument and then run
>>> raft.postvayler.samples.bank.Main
>>>
>>> how it works:
>>> compiler injects bytecode to root class and all @Persistent classes that
>>> can be accesible* from root.
>>>
>>> root class contains an injected object pool, where each @Persistent
>>> class is assigned a Long id and put into this object pool when created.
>>> this pool will be* a WeakValueMap so objects which are not attached to root
>>> will be garbage collected at some time.
>>>
>>> each @Persist method in a @Persistent class is instrumented like this:
>>> @Persist
>>> ReturnType doSomething(someParams) {..}
>>>
>>> doSomehing is copied to __postvayler__doSomething(..). the original doSomehing
>>> becomes:
>>>
>>> ReturnType doSomething(someParams) {
>>>     if (!there is postvayler context)
>>>         return __postvayler__doSomething(..);
>>>     if (we are in a prevayler transaction)
>>>         return __postvayler__doSomething(..);
>>>     return prevayler.execute(new MethodTransaction(..));
>>> }
>>>
>>> the arguments to doSomehing(..) is transformed in MethodTransaction such
>>> that references to @Persistent objects are replaced with Reference's,
>>> which will later be restored from object pool. seems as this also solves
>>> the Baptism problem.
>>>
>>> looks good to me for now:)
>>>
>>> cheers,
>>> *r a f t*
>>>
>>> * not implemented yet
>>>
>>> On Wed, Nov 27, 2013 at 11:07 AM, Naveen Chawla <nav...@gm...>
>>> wrote:
>>>
>>>> Sure, here:
>>>> http://docs.oracle.com/javase/tutorial/java/generics/methods.html
>>>>
>>>> As for consistency, any multi-changes explicitly synchronized over
>>>> prevalentObject can be auto-batched.
>>>>
>>>>
>>>> On 27 November 2013 00:25, Klaus Wuestefeld <ma...@kl...> wrote:
>>>>
>>>>> > It is ordinary generic method syntax.
>>>>>
>>>>> Could you point me to some doc on this syntax? I couldnt find it here:
>>>>> http://docs.oracle.com/javase/tutorial/extra/generics/methods.html
>>>>>
>>>>> > The beginning and end of a transaction would simply be any change to
>>>>> any
>>>>> > non-transient field.
>>>>>
>>>>> Suppose a transaction needs to update two fields to be correct but the
>>>>> system crashes after the first. Will the system not be inconsistent on
>>>>> recovery?
>>>>>
>>>>> Klaus
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>>>> organizations don't have a clear picture of how application performance
>>>>> affects their revenue. With AppDynamics, you get 100% visibility into
>>>>> your
>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>>> AppDynamics Pro!
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> To unsubscribe go to the end of this page:
>>>>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>>>> _______________________________________________
>>>>> "Databases in Memoriam" -- http://www.prevayler.org
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>>> organizations don't have a clear picture of how application performance
>>>> affects their revenue. With AppDynamics, you get 100% visibility into
>>>> your
>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>> AppDynamics Pro!
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> To unsubscribe go to the end of this page:
>>>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>>> _______________________________________________
>>>> "Databases in Memoriam" -- http://www.prevayler.org
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>> _______________________________________________
>> "Databases in Memoriam" -- http://www.prevayler.org
>>
>> _______________________________________________
> To unsubscribe go to the end of this page:
> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
>

Re: [Prevayler-discussion] Chainvayler

[hakan e. <hak...@gm...>]

Thanks Rick :)

Those are two very different projects and might be better delivered as
> separate things, even though you can turn them off independently


I'm targeting enterprise domain and they need both. Postvayler, predecessor
of Chainvayler, was doing transparent persistence. But enterprise need
scalability or at least redundancy. So it was on the back of my mind for
long time for how to replicate this. Either one (replication/persistence)
has their own specific use cases, possibly persistence a lot more, like all
desktop applications can benefit from it, but for the overall goal,
eliminating databases, we need  replication and persistence at the same
time.

Adoption and interest are so hard to generate.


So true indeed.

It feels like many people don’t have the vision or look deeper, or maybe
I’m very biased and exaggerating the possible value.

What could you build, dirt simple, that would get people excited about the
> possibility.


I was thinking similar, so a few years ago spent some time and adapted
Spring's famous "pet clinic" sample to Postvayler. It integrates with
Spring's load time weaving mechanism and persistence root can be
@Autowired.
https://github.com/raftAtGit/spring-petclinic

Many additional things are possible here. For example, autowiring Chained
non root objects based on JsonPath (or XPath) like syntax. For example:

@RequestMapping("/deposit/{accountId}")
void depositMoney(
    @PathVariable int accountId,
    @Path("$.bank.account[{accountId}]") Account account) {

       // account with accountId will be injected. as long as Bank
class has a Map, getAccount(int) method or similar.
       account.deposit(amount);
}

But these are just additional benefits. Not directly related to
Postvayler/Chainvayler. That's why I didnt even mention them in the blog.

I believe, if it ever happens, the breaking point will happen when some
large and/or famous company uses Chainvayler in a real life project, and
proves it can actually be used to eliminate databases, provides much better
performance and reduce development costs.

Best,
Hakan


On Fri, Apr 24, 2020 at 7:44 PM Rick Ross <ri...@le...> wrote:

> I like the concept a lot.   Isn't the dream to just annotate objects for
> persistence and whatever rules you need to apply?     To marry that to a
> shared model across JVMs is very interesting.
>
>
> My first thoughts are :
>
> Those are two very different projects and might be better delivered as
> separate things, even though you can turn them off independently
>
> Adoption and interest are so hard to generate.  Without them, it's just a
> personal project.   I mean no offense by that, but success breeds success.
> We've all been there.  What could you build, dirt simple, that would get
> people excited about the possibility.
>
>     My first thought there is the classic LAMP stack needs one less
> letter, right?
>
>     It strikes me that with persistence managed entirely as objects, it
> wouldn't be to hard to autogenerate basic crud REST calls.   Once you have
> annotated your object, it is automatically REST available, with full crud.
>
>
>     Or perhaps a connector that could work with Angular and provide safe
> persistent storage, right off the bat.
>
>
> Just spitballing here.   I'll keep tabs on this though.  It's interesting.
>
>
> Rick
>
>
>
> On 4/22/20 11:35 PM, hakan eryargi wrote:
>
> Hey guys,
>
> Remember my PoC project *Postvayler* many years ago? It was making POJOs
> persistent transparently based on @annotations and of course Prevayler.
>
> Recently I took the idea one step forward and extended the mechanism to
> also replicate POJOs among JVMs.
>
> Here is my blog post about it:
>
> https://medium.com/@hakan.eryargi/transparent-replication-and-persistence-for-pojo-graphs-805d9ae0f776
>
>
> Here is the project *Chainvayler* itself:
> https://github.com/raftAtGit/Chainvayler
>
> Personally I find the idea really promising :)
>
> Cheers,
> Hakan (r a f t)
>
> PS: I'm somehow got out from the email list, re-registered myself.
>
> On Mon, Dec 2, 2013 at 12:12 AM hakan eryargi <hak...@gm...>
> wrote:
>
>> i'm experimenting with the idea. so far not bad. just put what I've done
>> to GitHub. any suggestions, critics and contributions are welcome:)
>> https://github.com/raftAtGit/Postvayler
>>
>> apologies for the name, it sounded like a nice word play:)
>>
>> to run the sample, run the Compiler class
>> with raft.postvayler.samples.bank.Bank argument and then run
>> raft.postvayler.samples.bank.Main
>>
>> how it works:
>> compiler injects bytecode to root class and all @Persistent classes that
>> can be accesible* from root.
>>
>> root class contains an injected object pool, where each @Persistent class
>> is assigned a Long id and put into this object pool when created. this pool
>> will be* a WeakValueMap so objects which are not attached to root will be
>> garbage collected at some time.
>>
>> each @Persist method in a @Persistent class is instrumented like this:
>> @Persist
>> ReturnType doSomething(someParams) {..}
>>
>> doSomehing is copied to __postvayler__doSomething(..). the original doSomehing
>> becomes:
>>
>> ReturnType doSomething(someParams) {
>>     if (!there is postvayler context)
>>         return __postvayler__doSomething(..);
>>     if (we are in a prevayler transaction)
>>         return __postvayler__doSomething(..);
>>     return prevayler.execute(new MethodTransaction(..));
>> }
>>
>> the arguments to doSomehing(..) is transformed in MethodTransaction such
>> that references to @Persistent objects are replaced with Reference's,
>> which will later be restored from object pool. seems as this also solves
>> the Baptism problem.
>>
>> looks good to me for now:)
>>
>> cheers,
>> *r a f t*
>>
>> * not implemented yet
>>
>> On Wed, Nov 27, 2013 at 11:07 AM, Naveen Chawla <nav...@gm...>
>> wrote:
>>
>>> Sure, here:
>>> http://docs.oracle.com/javase/tutorial/java/generics/methods.html
>>>
>>> As for consistency, any multi-changes explicitly synchronized over
>>> prevalentObject can be auto-batched.
>>>
>>>
>>> On 27 November 2013 00:25, Klaus Wuestefeld <ma...@kl...> wrote:
>>>
>>>> > It is ordinary generic method syntax.
>>>>
>>>> Could you point me to some doc on this syntax? I couldnt find it here:
>>>> http://docs.oracle.com/javase/tutorial/extra/generics/methods.html
>>>>
>>>> > The beginning and end of a transaction would simply be any change to
>>>> any
>>>> > non-transient field.
>>>>
>>>> Suppose a transaction needs to update two fields to be correct but the
>>>> system crashes after the first. Will the system not be inconsistent on
>>>> recovery?
>>>>
>>>> Klaus
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>>> organizations don't have a clear picture of how application performance
>>>> affects their revenue. With AppDynamics, you get 100% visibility into
>>>> your
>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>>> AppDynamics Pro!
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> To unsubscribe go to the end of this page:
>>>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>>> _______________________________________________
>>>> "Databases in Memoriam" -- http://www.prevayler.org
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>> organizations don't have a clear picture of how application performance
>>> affects their revenue. With AppDynamics, you get 100% visibility into
>>> your
>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>> AppDynamics Pro!
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> To unsubscribe go to the end of this page:
>>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>> _______________________________________________
>>> "Databases in Memoriam" -- http://www.prevayler.org
>>>
>>>
>>
>
> _______________________________________________
> To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
>
>

Re: [Prevayler-discussion] Chainvayler

[Justin T. S. <ju...@kr...>]

Nice write-up! Thanks for sharing.

Cheers,
Justin

On Wed, Apr 22, 2020 at 11:36 PM hakan eryargi <hak...@gm...>
wrote:

> Hey guys,
>
> Remember my PoC project *Postvayler* many years ago? It was making POJOs
> persistent transparently based on @annotations and of course Prevayler.
>
> Recently I took the idea one step forward and extended the mechanism to
> also replicate POJOs among JVMs.
>
> Here is my blog post about it:
>
> https://medium.com/@hakan.eryargi/transparent-replication-and-persistence-for-pojo-graphs-805d9ae0f776
>
>
> Here is the project *Chainvayler* itself:
> https://github.com/raftAtGit/Chainvayler
>
> Personally I find the idea really promising :)
>
> Cheers,
> Hakan (r a f t)
>
> PS: I'm somehow got out from the email list, re-registered myself.
>
> On Mon, Dec 2, 2013 at 12:12 AM hakan eryargi <hak...@gm...>
> wrote:
>
>> i'm experimenting with the idea. so far not bad. just put what I've done
>> to GitHub. any suggestions, critics and contributions are welcome:)
>> https://github.com/raftAtGit/Postvayler
>>
>> apologies for the name, it sounded like a nice word play:)
>>
>> to run the sample, run the Compiler class
>> with raft.postvayler.samples.bank.Bank argument and then run
>> raft.postvayler.samples.bank.Main
>>
>> how it works:
>> compiler injects bytecode to root class and all @Persistent classes that
>> can be accesible* from root.
>>
>> root class contains an injected object pool, where each @Persistent class
>> is assigned a Long id and put into this object pool when created. this pool
>> will be* a WeakValueMap so objects which are not attached to root will be
>> garbage collected at some time.
>>
>> each @Persist method in a @Persistent class is instrumented like this:
>> @Persist
>> ReturnType doSomething(someParams) {..}
>>
>> doSomehing is copied to __postvayler__doSomething(..). the original doSomehing
>> becomes:
>>
>> ReturnType doSomething(someParams) {
>>     if (!there is postvayler context)
>>         return __postvayler__doSomething(..);
>>     if (we are in a prevayler transaction)
>>         return __postvayler__doSomething(..);
>>     return prevayler.execute(new MethodTransaction(..));
>> }
>>
>> the arguments to doSomehing(..) is transformed in MethodTransaction such
>> that references to @Persistent objects are replaced with Reference's,
>> which will later be restored from object pool. seems as this also solves
>> the Baptism problem.
>>
>> looks good to me for now:)
>>
>> cheers,
>> *r a f t*
>>
>> * not implemented yet
>>
>> On Wed, Nov 27, 2013 at 11:07 AM, Naveen Chawla <nav...@gm...>
>> wrote:
>>
>>> Sure, here:
>>> http://docs.oracle.com/javase/tutorial/java/generics/methods.html
>>>
>>> As for consistency, any multi-changes explicitly synchronized over
>>> prevalentObject can be auto-batched.
>>>
>>>
>>> On 27 November 2013 00:25, Klaus Wuestefeld <ma...@kl...> wrote:
>>>
>>>> > It is ordinary generic method syntax.
>>>>
>>>> Could you point me to some doc on this syntax? I couldnt find it here:
>>>> http://docs.oracle.com/javase/tutorial/extra/generics/methods.html
>>>>
>>>> > The beginning and end of a transaction would simply be any change to
>>>> any
>>>> > non-transient field.
>>>>
>>>> Suppose a transaction needs to update two fields to be correct but the
>>>> system crashes after the first. Will the system not be inconsistent on
>>>> recovery?
>>>>
>>>> Klaus
>>>>
>>>

[Prevayler-discussion] Chainvayler

[hakan e. <hak...@gm...>]

Hey guys,

Remember my PoC project *Postvayler* many years ago? It was making POJOs
persistent transparently based on @annotations and of course Prevayler.

Recently I took the idea one step forward and extended the mechanism to
also replicate POJOs among JVMs.

Here is my blog post about it:
https://medium.com/@hakan.eryargi/transparent-replication-and-persistence-for-pojo-graphs-805d9ae0f776


Here is the project *Chainvayler* itself:
https://github.com/raftAtGit/Chainvayler

Personally I find the idea really promising :)

Cheers,
Hakan (r a f t)

PS: I'm somehow got out from the email list, re-registered myself.

On Mon, Dec 2, 2013 at 12:12 AM hakan eryargi <hak...@gm...>
wrote:

> i'm experimenting with the idea. so far not bad. just put what I've done
> to GitHub. any suggestions, critics and contributions are welcome:)
> https://github.com/raftAtGit/Postvayler
>
> apologies for the name, it sounded like a nice word play:)
>
> to run the sample, run the Compiler class
> with raft.postvayler.samples.bank.Bank argument and then run
> raft.postvayler.samples.bank.Main
>
> how it works:
> compiler injects bytecode to root class and all @Persistent classes that
> can be accesible* from root.
>
> root class contains an injected object pool, where each @Persistent class
> is assigned a Long id and put into this object pool when created. this pool
> will be* a WeakValueMap so objects which are not attached to root will be
> garbage collected at some time.
>
> each @Persist method in a @Persistent class is instrumented like this:
> @Persist
> ReturnType doSomething(someParams) {..}
>
> doSomehing is copied to __postvayler__doSomething(..). the original doSomehing
> becomes:
>
> ReturnType doSomething(someParams) {
>     if (!there is postvayler context)
>         return __postvayler__doSomething(..);
>     if (we are in a prevayler transaction)
>         return __postvayler__doSomething(..);
>     return prevayler.execute(new MethodTransaction(..));
> }
>
> the arguments to doSomehing(..) is transformed in MethodTransaction such
> that references to @Persistent objects are replaced with Reference's,
> which will later be restored from object pool. seems as this also solves
> the Baptism problem.
>
> looks good to me for now:)
>
> cheers,
> *r a f t*
>
> * not implemented yet
>
> On Wed, Nov 27, 2013 at 11:07 AM, Naveen Chawla <nav...@gm...>
> wrote:
>
>> Sure, here:
>> http://docs.oracle.com/javase/tutorial/java/generics/methods.html
>>
>> As for consistency, any multi-changes explicitly synchronized over
>> prevalentObject can be auto-batched.
>>
>>
>> On 27 November 2013 00:25, Klaus Wuestefeld <ma...@kl...> wrote:
>>
>>> > It is ordinary generic method syntax.
>>>
>>> Could you point me to some doc on this syntax? I couldnt find it here:
>>> http://docs.oracle.com/javase/tutorial/extra/generics/methods.html
>>>
>>> > The beginning and end of a transaction would simply be any change to
>>> any
>>> > non-transient field.
>>>
>>> Suppose a transaction needs to update two fields to be correct but the
>>> system crashes after the first. Will the system not be inconsistent on
>>> recovery?
>>>
>>> Klaus
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>> organizations don't have a clear picture of how application performance
>>> affects their revenue. With AppDynamics, you get 100% visibility into
>>> your
>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
>>> AppDynamics Pro!
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> To unsubscribe go to the end of this page:
>>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>> _______________________________________________
>>> "Databases in Memoriam" -- http://www.prevayler.org
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Rapidly troubleshoot problems before they affect your business. Most IT
>> organizations don't have a clear picture of how application performance
>> affects their revenue. With AppDynamics, you get 100% visibility into your
>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
>> Pro!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
>> _______________________________________________
>> To unsubscribe go to the end of this page:
>> http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>> _______________________________________________
>> "Databases in Memoriam" -- http://www.prevayler.org
>>
>>
>

Re: [Prevayler-discussion] Prevayler Repo Evolution

[Justin T. S. <ju...@kr...>]

This is really cool. :)

Note that this video was generated in 2015, so it doesn't show the most
recent activity.

Also note that the history starts in 2001, long before Git even existed. :)
That is because all of our prior history was carefully preserved when we
migrated to Git. The first 15 commits[*] were imported from zip files that
Klaus was distributing for Prevayler 1 back in 2001-2003, and the next few
years of history were imported from two different CVS servers where
Prevayler 2 development was hosted before moving to GitHub.

[*]
https://github.com/prevayler/prevayler/commits/master?after=8701eb06f1314980405c3b19db00b6facc8a394f+629


On Wed, Jul 31, 2019 at 6:52 AM Klaus Wuestefeld <kla...@gm...>
wrote:

> https://www.youtube.com/watch?v=ut-ZYY3S3WQ
>

[Prevayler-discussion] Prevayler Repo Evolution

[Klaus W. <kla...@gm...>]

https://www.youtube.com/watch?v=ut-ZYY3S3WQ

Re: [Prevayler-discussion] Checkpoint/Restore In Userspace (CRIU) for Prevalent Snapshots

[Christian S. <sor...@gm...>]

No, I didn't. But that would indeed be a very fast restore mechanism.

If only https://openjdk.java.net/jeps/310 included "heap/object instances".
;-)

Cheers,
Christian

[Prevayler-discussion] Checkpoint/Restore In Userspace (CRIU) for Prevalent Snapshots

[Klaus W. <kla...@gm...>]

Hi, have any of you tried using CRIU or some similar solution for JVM
process or heap snapshots?

https://criu.org/Main_Page

That would be great for fast start up times, avoiding serialization time.

Cheers, Klaus

Re: [Prevayler-discussion] reading data from "journal"-files

[Justin S. <jsa...@gu...>]

peacecop kalmer wrote:

> Is there a way to read the data from an existing .journal-file
> like converting it into XML or CSV or any other human-readable
> format? I have lots of journal-files that has been used for
> storing data and I need this data for further analyses.
> However, I don't have the source code of the application itself
> that uses these journal-files.

The missing source code makes it potentially tricky. Are you at
least able to run the application? And does the application have
an option to trigger a Prevayler snapshot?

Conceptually, the journal files aren't really "storing data" in
the sense of a database table. They are more like a log of
changes, so ideally you would bring up the application, let it
read all the journals, and then get the data you need from the
application's memory. A Prevayler snapshot, as opposed to a
journal, contains a dump of the objects in memory that result
from replaying all the journals.

If you're lucky, the application is already configured to
generate snapshots in XML format. The default is to use Java
serialization, which is less convenient than XML but is at
least a well-defined format that you might be able to figure
out how to process.

Cheers,
Justin

[Prevayler-discussion] reading data from "journal"-files

[peacecop kalmer: <pan...@gm...>]

Is there a way to read the data from an existing /.journal/-file like 
converting it into XML or CSV or any other human-readable format? I have 
lots of /journal/-files that has been used for storing data and I need 
this data for further analyses. However, I don't have the source code of 
the application itself that uses these /journal/-files.

-- 

*peacecop kalmer:*

*Instituudi tee 3-27 *

*EE76902 * *Harku *

*Harku*

*Harjumaa*

*+372 652 4228 *

*+372  5620 4556 *

*__________________________________________________________________ *

Re: [Prevayler-discussion] Apache Mnemonic

[Robert F. <rob...@de...>]

> still playing with the idea to re-implement Prevayler with modern Java tools.

Go for it! For inspiration have a look at memstate, a reimplementation of origodb, the prevayler equivalent for dotnet, written in C#.

Origodb processes 3k commands per second, memstate uses group commit (command batching) and comes close to 100k CPS depending on the type of storage backend.
The other key difference is that the prevalent system is built from the stream of commands arriving from the backing store. That means we can have multiple nodes where any node can accept commands. A node simply sends commands to storage asynchronously and waits for them to return on the subscription channel before responding to the caller. The key here is that the storage backend provides a total order of all commands. We are focusing on 2 different storage backends: postgres with its notification feature and EventStore, a time series database which has built in push subscriptions. Postgres is slower but the ability to run on AWS Aurora or RDS is extremely valueable. We also looked at both kafka and kinesis but we found end to end latency to be too poor!

OrigoDB had its own replication protocol where each replica received commands directly from the primary node, synchronous or asynchronous. Synchronous replication provides stronger consistency across all the nodes. With memstate a node can lag behind.
Mnemonic looks like something entirely different, it transparently maps objects on the heap to non-volatile memory.

--
Robert Friberg
Devrex Labs
Memstate, OrigoDB

From: Christian Stein <sor...@gm...>
Reply-To: "pre...@li..." <pre...@li...>
Date: Sunday, 28 January 2018 at 17:51
To: "pre...@li..." <pre...@li...>
Subject: [Prevayler-discussion] Apache Mnemonic

Hi Prevayler group,

still playing with the idea to re-implement Prevayler with modern Java tools.

Now I found https://mnemonic.apache.org

I haven't tried that libary, yet. But looks promising.

What do you think?

Cheers,
Christian

[Prevayler-discussion] Apache Mnemonic

[Christian S. <sor...@gm...>]

Hi Prevayler group,

still playing with the idea to re-implement Prevayler with modern Java
tools.

Now I found https://mnemonic.apache.org

I haven't tried that libary, yet. But looks promising.

What do you think?

Cheers,
Christian

Re: [Prevayler-discussion] Linode Announces High Memory Instances

[Christian S. <sor...@gm...>]

   - Hewlett Packard Enterprise introduces the world’s largest
   single-memory computer
   - The prototype unveiled today contains *160 terabytes of memory*
   - HPE expects the architecture to easily scale to a nearly-limitless
   pool of memory

Let's Prevayler run on this box...

Source: https://news.hpe.com/content-hub/memory-driven-computing/

On Fri, Feb 17, 2017 at 11:13 PM, Klaus Wuestefeld <
kla...@gm...> wrote:

> https://blog.linode.com/2017/02/14/high-memory-instances-and-5-linodes/
>
> They come in handy for Prevayler-based systems.
>
> See you, Klaus :)
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> To unsubscribe go to the end of this page: http://lists.sourceforge.net/
> lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
>

Re: [Prevayler-discussion] Report from using Kafka as Prevalence journal

[Marcel O. <mar...@gm...>]

Sometimes I feel we are like wizards, with our own Arcane incantations ;)

Much thanks for the Docker line though!

On Mon, Mar 27, 2017 at 2:52 PM, Karl Wettin <kar...@ko...> wrote:

> Kafka doesn't have to be that tricky, at least not to get started with it:
>
> https://github.com/spotify/docker-kafka/
>
> docker run -p 2181:2181 -p 9092:9092 --rm  --env
> ADVERTISED_HOST=10.40.1.52 --env ADVERTISED_PORT=9092 --name kafka -h kafka
> spotify/kafka
>
> // kafka properties:
> config.put("auto.offset.reset", "earliest");
> config.put("group.id", UUID.randomUUID().toString());
>
> That's pretty much all configuration you need.
>
>
>                 kalle
>
> > On 14 Mar 2017, at 23:42, Robert Friberg <rob...@de...>
> wrote:
> >
> > Hi,
> >
> > We’ve been using this approach as well with OrigoDB. A key strength not
> mentioned by Karl is that additional consumers of the transaction stream
> can generate arbitrary read models, either in memory or on disk, in real
> time or on demand.
> >
> > Kafka is very mature and popular but it requires a LOT of knowledge to
> install, configure and operation. Ask me how I know 😊
> > To understand the kafka model, it’s easy to think of a persistent
> pub/sub message bus. A producer publishes messages to a topic and consumers
> subscribe to topics.
> >
> > Yes, message ordering is preserved and the delivery guarantee is at
> least once, so you should probably have a deduplication step in the
> pipeline.
> >
> > Kafka is relatively slow if you write a single message using a producer
> and wait for the message using a consumer. You can configure the poll
> interval to be more aggressive but sending a single transaction over the
> wire, flushing to disk on the kafka broker(s) and then reading it back will
> be slow. Guessing TPS is in the hundreds, but haven’t measured. You could
> scale a bit by accepting writes at multiple nodes but better to go async
> and have multiple threads write to kafka using a single producer (yes,
> producers are thread safe) and then correlate with the transaction when it
> arrives at the consumer.
> >
> > This async approach using EventStore instead of Kafka, we can easily do
> 100K write transactions per second. The EventStore  client batches messages
> on the write side and uses push subscriptions on the read side, so there is
> no extra latency due to polling. Here is the jvm client for EventStore:
> https://github.com/EventStore/EventStore.JVM
> >
> > One trick to avoid stale reads is to pass a sequential version number
> with a query. The query will wait for the prevalent system to be at least
> the required version before executing.
> >
> > Note that there are degrees of eventual consistency, and the
> inconsistency here is only temporal. Any given state is consistent in
> itself, it just might not be the most recent consistent state.
> >
> > --
> > Robert Friberg
> > Building OrigoDB at Devrex Labs
> > +46733839080
> >
> > From: Ralph Johnson [mailto:jo...@cs...]
> > Sent: den 14 mars 2017 22:13
> > To: Open discussion about the Prevayler project. <
> pre...@li...>
> > Subject: Re: [Prevayler-discussion] Report from using Kafka as
> Prevalence journal
> >
> > i was thinking of doing something similar, i.e. using a messaging system
> to keep processes using Prevayler in synch.    But the systems I was
> looking at would ensure that transactions would arrive in the same order
> for all processes.   It sounds like Kafka doesn't do that.  Or did I not
> understand?
> >
> > I looked up Kafka after reading your message.  I've never looked at it
> before.   It looks well-designed.   You say "write speed is crippled".  How
> many transactions per second can it deliver?
> >
> > -Ralph Johnson
> >
> > On Tue, Mar 14, 2017 at 4:02 AM, Karl Wettin <kar...@ko...>
> wrote:
> > You chose to allow Open discussion about the Prevayler project. (
> pre...@li...) even though this message
> failed authentication
> > Click to disallow
> > This message is eligible for Automatic Cleanup! (kar...@ko...)
> Add cleanup rule | More info
> >
> > Perhaps some of you find this interesting.
> >
> >
> > For a little while now we've been running a Prevalance pattern that use
> Kafka as journal to handle multiple load balanced and availability nodes of
> the same data store. Transactions have unique identities and when executed
> by the user on a local node they are sent to Kafka, waited upon to bounce
> back from Kafka and then executed on the local node using a CountDownLatch
> to make sure the transaction has been distributed. Calls on a local node
> are kept in order other using a ConcurrentReadWriteLock. It's very
> simplistic, some 200 lines of code.
> >
> >
> > The obvious drawbacks is that:
> >
> > * Write speed is crippled, bound to the Kafka polling rate. This will of
> course also affect the read speed at the local node on which the
> transaction was created, as the node will be locked down for further
> queries and transaction until the waited upon transaction has been received
> and executed.
> >
> > * The nodes are out of sync at a level equal to the network latency.
> Queries executed on different nodes can for a short time return different
> results but there is eventual consistency. One can however often implement
> transactions and queries in a manner so that this never occurs (e.g. never
> request data that has not been created or is about to be updated), or
> simply by accepting this caveat as a system limitation.
> >
> > For us, none of the above seems to be a problem at this point.
> >
> >
> > Well implemented transactions with validation before broadcasting a
> transaction and the same validation before executing the transaction seems
> to handle when multiple nodes simultaneously attempt to broadcast
> incompatible transactions, then only the first to arrive to Kafka will
> succeed and the others will receive a validation exception. This does
> however mean that the journal might contain transactions that indeed will
> fail.
> >
> > I suppose it would be possible to use Kafka as an inter-node write-lock
> if one is OK with adding even more latency, and thus removing a few of the
> caveats. But I'm not going there voluntarily.
> >
> >
> > All in all it seems to be working great.
> >
> > So far there are no snapshots, but the idea is to schedule starting up a
> new service node to which no clients will connect, take a snapshot and then
> shut it down again.
> >
> >
> >                 k
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > _______________________________________________
> > To unsubscribe go to the end of this page: http://lists.sourceforge.net/
> lists/listinfo/prevayler-discussion
> > _______________________________________________
> > "Databases in Memoriam" -- http://www.prevayler.org
> >
> >
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot______
> _________________________________________
> > To unsubscribe go to the end of this page: http://lists.sourceforge.net/
> lists/listinfo/prevayler-discussion
> > _______________________________________________
> > "Databases in Memoriam" -- http://www.prevayler.org
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> To unsubscribe go to the end of this page: http://lists.sourceforge.net/
> lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
>

Re: [Prevayler-discussion] Report from using Kafka as Prevalence journal

[Karl W. <kar...@ko...>]

Kafka doesn't have to be that tricky, at least not to get started with it:

https://github.com/spotify/docker-kafka/

docker run -p 2181:2181 -p 9092:9092 --rm  --env ADVERTISED_HOST=10.40.1.52 --env ADVERTISED_PORT=9092 --name kafka -h kafka spotify/kafka

// kafka properties:
config.put("auto.offset.reset", "earliest");
config.put("group.id", UUID.randomUUID().toString());

That's pretty much all configuration you need.


		kalle

> On 14 Mar 2017, at 23:42, Robert Friberg <rob...@de...> wrote:
> 
> Hi,
>  
> We’ve been using this approach as well with OrigoDB. A key strength not mentioned by Karl is that additional consumers of the transaction stream can generate arbitrary read models, either in memory or on disk, in real time or on demand.
>  
> Kafka is very mature and popular but it requires a LOT of knowledge to install, configure and operation. Ask me how I know 😊
> To understand the kafka model, it’s easy to think of a persistent pub/sub message bus. A producer publishes messages to a topic and consumers subscribe to topics.
>  
> Yes, message ordering is preserved and the delivery guarantee is at least once, so you should probably have a deduplication step in the pipeline.
>  
> Kafka is relatively slow if you write a single message using a producer and wait for the message using a consumer. You can configure the poll interval to be more aggressive but sending a single transaction over the wire, flushing to disk on the kafka broker(s) and then reading it back will be slow. Guessing TPS is in the hundreds, but haven’t measured. You could scale a bit by accepting writes at multiple nodes but better to go async and have multiple threads write to kafka using a single producer (yes, producers are thread safe) and then correlate with the transaction when it arrives at the consumer.
>  
> This async approach using EventStore instead of Kafka, we can easily do 100K write transactions per second. The EventStore  client batches messages on the write side and uses push subscriptions on the read side, so there is no extra latency due to polling. Here is the jvm client for EventStore:https://github.com/EventStore/EventStore.JVM
>  
> One trick to avoid stale reads is to pass a sequential version number with a query. The query will wait for the prevalent system to be at least the required version before executing.
>  
> Note that there are degrees of eventual consistency, and the inconsistency here is only temporal. Any given state is consistent in itself, it just might not be the most recent consistent state.
>  
> --
> Robert Friberg
> Building OrigoDB at Devrex Labs
> +46733839080
>  
> From: Ralph Johnson [mailto:jo...@cs...] 
> Sent: den 14 mars 2017 22:13
> To: Open discussion about the Prevayler project. <pre...@li...>
> Subject: Re: [Prevayler-discussion] Report from using Kafka as Prevalence journal
>  
> i was thinking of doing something similar, i.e. using a messaging system to keep processes using Prevayler in synch.    But the systems I was looking at would ensure that transactions would arrive in the same order for all processes.   It sounds like Kafka doesn't do that.  Or did I not understand?
>  
> I looked up Kafka after reading your message.  I've never looked at it before.   It looks well-designed.   You say "write speed is crippled".  How many transactions per second can it deliver?
>  
> -Ralph Johnson
>  
> On Tue, Mar 14, 2017 at 4:02 AM, Karl Wettin <kar...@ko...> wrote:
> You chose to allow Open discussion about the Prevayler project. (pre...@li...) even though this message failed authentication 
> Click to disallow 
> This message is eligible for Automatic Cleanup! (kar...@ko...) Add cleanup rule | More info
> 
> Perhaps some of you find this interesting.
> 
> 
> For a little while now we've been running a Prevalance pattern that use Kafka as journal to handle multiple load balanced and availability nodes of the same data store. Transactions have unique identities and when executed by the user on a local node they are sent to Kafka, waited upon to bounce back from Kafka and then executed on the local node using a CountDownLatch to make sure the transaction has been distributed. Calls on a local node are kept in order other using a ConcurrentReadWriteLock. It's very simplistic, some 200 lines of code.
> 
> 
> The obvious drawbacks is that:
> 
> * Write speed is crippled, bound to the Kafka polling rate. This will of course also affect the read speed at the local node on which the transaction was created, as the node will be locked down for further queries and transaction until the waited upon transaction has been received and executed.
> 
> * The nodes are out of sync at a level equal to the network latency. Queries executed on different nodes can for a short time return different results but there is eventual consistency. One can however often implement transactions and queries in a manner so that this never occurs (e.g. never request data that has not been created or is about to be updated), or simply by accepting this caveat as a system limitation.
> 
> For us, none of the above seems to be a problem at this point.
> 
> 
> Well implemented transactions with validation before broadcasting a transaction and the same validation before executing the transaction seems to handle when multiple nodes simultaneously attempt to broadcast incompatible transactions, then only the first to arrive to Kafka will succeed and the others will receive a validation exception. This does however mean that the journal might contain transactions that indeed will fail.
> 
> I suppose it would be possible to use Kafka as an inter-node write-lock if one is OK with adding even more latency, and thus removing a few of the caveats. But I'm not going there voluntarily.
> 
> 
> All in all it seems to be working great.
> 
> So far there are no snapshots, but the idea is to schedule starting up a new service node to which no clients will connect, take a snapshot and then shut it down again.
> 
> 
>                 k
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
> 
>  
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
> To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org

Re: [Prevayler-discussion] Report from using Kafka as Prevalence journal

[Robert F. <rob...@de...>]

Hi,

We’ve been using this approach as well with OrigoDB. A key strength not mentioned by Karl is that additional consumers of the transaction stream can generate arbitrary read models, either in memory or on disk, in real time or on demand.

Kafka is very mature and popular but it requires a LOT of knowledge to install, configure and operation. Ask me how I know 😊
To understand the kafka model, it’s easy to think of a persistent pub/sub message bus. A producer publishes messages to a topic and consumers subscribe to topics.

Yes, message ordering is preserved and the delivery guarantee is at least once, so you should probably have a deduplication step in the pipeline.

Kafka is relatively slow if you write a single message using a producer and wait for the message using a consumer. You can configure the poll interval to be more aggressive but sending a single transaction over the wire, flushing to disk on the kafka broker(s) and then reading it back will be slow. Guessing TPS is in the hundreds, but haven’t measured. You could scale a bit by accepting writes at multiple nodes but better to go async and have multiple threads write to kafka using a single producer (yes, producers are thread safe) and then correlate with the transaction when it arrives at the consumer.

This async approach using EventStore instead of Kafka, we can easily do 100K write transactions per second. The EventStore  client batches messages on the write side and uses push subscriptions on the read side, so there is no extra latency due to polling. Here is the jvm client for EventStore: https://github.com/EventStore/EventStore.JVM

One trick to avoid stale reads is to pass a sequential version number with a query. The query will wait for the prevalent system to be at least the required version before executing.

Note that there are degrees of eventual consistency, and the inconsistency here is only temporal. Any given state is consistent in itself, it just might not be the most recent consistent state.

--
Robert Friberg
Building OrigoDB at Devrex Labs
+46733839080

From: Ralph Johnson [mailto:jo...@cs...]
Sent: den 14 mars 2017 22:13
To: Open discussion about the Prevayler project. <pre...@li...>
Subject: Re: [Prevayler-discussion] Report from using Kafka as Prevalence journal

i was thinking of doing something similar, i.e. using a messaging system to keep processes using Prevayler in synch.    But the systems I was looking at would ensure that transactions would arrive in the same order for all processes.   It sounds like Kafka doesn't do that.  Or did I not understand?

I looked up Kafka after reading your message.  I've never looked at it before.   It looks well-designed.   You say "write speed is crippled".  How many transactions per second can it deliver?

-Ralph Johnson

On Tue, Mar 14, 2017 at 4:02 AM, Karl Wettin <kar...@ko...<mailto:kar...@ko...>> wrote:
[Boxbe]<https://www.boxbe.com/overview>[http://www.boxbe.com/stfopen?tc_serial=29310543226&tc_rand=1341927045&utm_source=stf&utm_medium=email&utm_campaign=ANNO_AFA&utm_content=001]You chose to allow Open discussion about the Prevayler project. (pre...@li...<mailto:pre...@li...>) even though this message failed authentication
Click to disallow<https://www.boxbe.com/anno?tc_serial=29310543226&tc_rand=1341927045&utm_source=stf&utm_medium=email&utm_campaign=ANNO_AFA&utm_content=001&action=authfail&set=false&key=uC3uVkXGuUtcik6vMPkTsS2nWCSgICUoD8MbmnjcqxE%3D&token=9g%2F5SLD81HsuGVR2emB%2Fb8Z0p17qbKMSZwt%2BSOrggii4%2BFX%2FDDF2XX4XpbUu3%2FAt>
This message is eligible for Automatic Cleanup! (kar...@ko...<mailto:kar...@ko...>) Add cleanup rule<https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DuC3uVkXGuUv4Wb99Bjk%252BGPzxrK8ieOUyl1gov9ijHh0%253D%26token%3DSD1vQhkW0iDi7vbXrjcW75E90Fv8TPUuQv%252F8Bp%252FfxG%252BdxXeDTckFZPjXSccVRfu6syLPHQ7ZbxY4WpnqFAZOtsDWbcqZb7TIcB6%252FDYZQScYaHrBGiLBgzuf4GVNlIGNmPhb0qMB82OBQKgZg7pSWHw%253D%253D&tc_serial=29310543233&tc_rand=130196421&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> | More info<http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=29310543233&tc_rand=130196421&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>

Perhaps some of you find this interesting.


For a little while now we've been running a Prevalance pattern that use Kafka as journal to handle multiple load balanced and availability nodes of the same data store. Transactions have unique identities and when executed by the user on a local node they are sent to Kafka, waited upon to bounce back from Kafka and then executed on the local node using a CountDownLatch to make sure the transaction has been distributed. Calls on a local node are kept in order other using a ConcurrentReadWriteLock. It's very simplistic, some 200 lines of code.


The obvious drawbacks is that:

* Write speed is crippled, bound to the Kafka polling rate. This will of course also affect the read speed at the local node on which the transaction was created, as the node will be locked down for further queries and transaction until the waited upon transaction has been received and executed.

* The nodes are out of sync at a level equal to the network latency. Queries executed on different nodes can for a short time return different results but there is eventual consistency. One can however often implement transactions and queries in a manner so that this never occurs (e.g. never request data that has not been created or is about to be updated), or simply by accepting this caveat as a system limitation.

For us, none of the above seems to be a problem at this point.


Well implemented transactions with validation before broadcasting a transaction and the same validation before executing the transaction seems to handle when multiple nodes simultaneously attempt to broadcast incompatible transactions, then only the first to arrive to Kafka will succeed and the others will receive a validation exception. This does however mean that the journal might contain transactions that indeed will fail.

I suppose it would be possible to use Kafka as an inter-node write-lock if one is OK with adding even more latency, and thus removing a few of the caveats. But I'm not going there voluntarily.


All in all it seems to be working great.

So far there are no snapshots, but the idea is to schedule starting up a new service node to which no clients will connect, take a snapshot and then shut it down again.


                k
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org

Re: [Prevayler-discussion] Report from using Kafka as Prevalence journal

[Ralph J. <jo...@cs...>]

i was thinking of doing something similar, i.e. using a messaging system to
keep processes using Prevayler in synch.    But the systems I was looking
at would ensure that transactions would arrive in the same order for all
processes.   It sounds like Kafka doesn't do that.  Or did I not understand?

I looked up Kafka after reading your message.  I've never looked at it
before.   It looks well-designed.   You say "write speed is crippled".  How
many transactions per second can it deliver?

-Ralph Johnson

On Tue, Mar 14, 2017 at 4:02 AM, Karl Wettin <kar...@ko...> wrote:

> [image: Boxbe] <https://www.boxbe.com/overview> You chose to allow Open
> discussion about the Prevayler project. (prevayler-discussion@lists.
> sourceforge.net) even though this message failed authentication
> Click to disallow
> <https://www.boxbe.com/anno?tc_serial=29310543226&tc_rand=1341927045&utm_source=stf&utm_medium=email&utm_campaign=ANNO_AFA&utm_content=001&action=authfail&set=false&key=uC3uVkXGuUtcik6vMPkTsS2nWCSgICUoD8MbmnjcqxE%3D&token=9g%2F5SLD81HsuGVR2emB%2Fb8Z0p17qbKMSZwt%2BSOrggii4%2BFX%2FDDF2XX4XpbUu3%2FAt>
> This message is eligible for Automatic Cleanup! (kar...@ko...) Add
> cleanup rule
> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DuC3uVkXGuUv4Wb99Bjk%252BGPzxrK8ieOUyl1gov9ijHh0%253D%26token%3DSD1vQhkW0iDi7vbXrjcW75E90Fv8TPUuQv%252F8Bp%252FfxG%252BdxXeDTckFZPjXSccVRfu6syLPHQ7ZbxY4WpnqFAZOtsDWbcqZb7TIcB6%252FDYZQScYaHrBGiLBgzuf4GVNlIGNmPhb0qMB82OBQKgZg7pSWHw%253D%253D&tc_serial=29310543233&tc_rand=130196421&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
> | More info
> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=29310543233&tc_rand=130196421&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001>
>
> Perhaps some of you find this interesting.
>
>
> For a little while now we've been running a Prevalance pattern that use
> Kafka as journal to handle multiple load balanced and availability nodes of
> the same data store. Transactions have unique identities and when executed
> by the user on a local node they are sent to Kafka, waited upon to bounce
> back from Kafka and then executed on the local node using a CountDownLatch
> to make sure the transaction has been distributed. Calls on a local node
> are kept in order other using a ConcurrentReadWriteLock. It's very
> simplistic, some 200 lines of code.
>
>
> The obvious drawbacks is that:
>
> * Write speed is crippled, bound to the Kafka polling rate. This will of
> course also affect the read speed at the local node on which the
> transaction was created, as the node will be locked down for further
> queries and transaction until the waited upon transaction has been received
> and executed.
>
> * The nodes are out of sync at a level equal to the network latency.
> Queries executed on different nodes can for a short time return different
> results but there is eventual consistency. One can however often implement
> transactions and queries in a manner so that this never occurs (e.g. never
> request data that has not been created or is about to be updated), or
> simply by accepting this caveat as a system limitation.
>
> For us, none of the above seems to be a problem at this point.
>
>
> Well implemented transactions with validation before broadcasting a
> transaction and the same validation before executing the transaction seems
> to handle when multiple nodes simultaneously attempt to broadcast
> incompatible transactions, then only the first to arrive to Kafka will
> succeed and the others will receive a validation exception. This does
> however mean that the journal might contain transactions that indeed will
> fail.
>
> I suppose it would be possible to use Kafka as an inter-node write-lock if
> one is OK with adding even more latency, and thus removing a few of the
> caveats. But I'm not going there voluntarily.
>
>
> All in all it seems to be working great.
>
> So far there are no snapshots, but the idea is to schedule starting up a
> new service node to which no clients will connect, take a snapshot and then
> shut it down again.
>
>
>                 k
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> To unsubscribe go to the end of this page: http://lists.sourceforge.net/
> lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
>
>