[Zookeeper-developer] [ zookeeper-Patches-1844561 ] fast sync between the leader and the follower.

[SourceForge.net <no...@so...>]

Patches item #1844561, was opened at 2007-12-05 02:05
Message generated for change (Comment added) made by mahadevkonar
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1008546&aid=1844561&group_id=209147

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: server
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Mahadev Konar (mahadevkonar)
Assigned to: Nobody/Anonymous (nobody)
Summary: fast sync between the leader and the follower.

Initial Comment:
this patch implements a fast sync between the leader adn the follower. THe leader maintains a queue of 500 requests in memory that have been committed and when a follower joins sends the diff to the follower. In case the follower is really behind, it will get the whole snapshot. In case the follower is ahead of the leader it will truncate the logs and then proceed.. 

----------------------------------------------------------------------

>Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-13 22:14

Message:
Logged In: YES 
user_id=1926680
Originator: YES

with some more changes/documentation ... thanks ben
File Added: fast-sync-follower-leader-5.patch

----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-13 21:40

Message:
Logged In: YES 
user_id=1926680
Originator: YES

thanks ben... for the first one... since removing from outstanding changes
does not gurantee that the changes have been applied to the datatree, i am
doing that later so that I am sure its been applied to the datatree before
I add it to the committedLog. Doing it before would be ok as well but i
feel comfortable with this control flow.

for the second .. nice catch... you are right. I fixed that ... uploading
a new patch.

for the third, you are right this will not catch this case... but since
the max zxid committed will be much much greater than the followers zxid,
it will get the snapshot. The code gets a little messy if I have to take
care of all the epoch transactions. In this case the follower gets the
whole snapshot... what do you think?
File Added: fast-sync-follower-leader-4.patch

----------------------------------------------------------------------

Comment By: Benjamin Reed (breed)
Date: 2007-12-13 21:24

Message:
Logged In: YES 
user_id=154690
Originator: NO

FinalRequestProcessor: Shouldn't the zks.addCommittedProposal happen
inside the sync block of zks.outstandingChanges? I think that switch should
happen atomically. 

Follower: in the case of Leader.DIFF it would be nice to have a comment
that says after loadData // Now process the diffs that come from the leader
as normal updates. Also, the zxid of truncate is the last zxid to KEEP not
the first to DELETE right? In truncate() it looks like you are DELETEing
zxid and up.

FollowerHandler: It looks like you are missing a TRUNC case: Imagine the
follower has seen (1,32) (where 1 is epoch and 32 is the count) and the
leader has seen and committed: (1,31), (2,0), (2,1), (2,1). It doesn't look
like (1,32) will be truncated from the follower's log.


----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-13 19:38

Message:
Logged In: YES 
user_id=1926680
Originator: YES

uploading the patch becaue the previous one conflicts with the trunk..
File Added: fast-sync-follower-leader-3.patch

----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-12 20:37

Message:
Logged In: YES 
user_id=1926680
Originator: YES

can you guys review this patch before I go off for vacation? i want to get
it in so that it does not get stale.

----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-07 23:06

Message:
Logged In: YES 
user_id=1926680
Originator: YES

this patch applies to the current trunk. No specific changes.
File Added: fast-sync-follower-leader-2.patch

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1008546&aid=1844561&group_id=209147