[Zookeeper-developer] [ zookeeper-Patches-1844561 ] fast sync between the leader and the follower.

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Patches item #1844561, was opened at 2007-12-05 02:05
Message generated for change (Comment added) made by mahadevkonar
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1008546&aid=1844561&group_id=209147

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: server
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Mahadev Konar (mahadevkonar)
Assigned to: Mahadev Konar (mahadevkonar)
Summary: fast sync between the leader and the follower.

Initial Comment:
this patch implements a fast sync between the leader adn the follower. THe leader maintains a queue of 500 requests in memory that have been committed and when a follower joins sends the diff to the follower. In case the follower is really behind, it will get the whole snapshot. In case the follower is ahead of the leader it will truncate the logs and then proceed.. 

----------------------------------------------------------------------

>Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-14 18:33

Message:
Logged In: YES 
user_id=1926680
Originator: YES

fixed the synchronization problem with outstanding queue and committed
queue. THanks ben for piointing it out. I also fixed the typo. I havent
fixed the epoch in this patch. I agree with flavio that we can fix it with
tuple but it will complicate things a little. I would vote for getting this
in and later working on epochs to be handled. 

File Added: fast-sync-follower-leader-6.patch

----------------------------------------------------------------------

Comment By: Benjamin Reed (breed)
Date: 2007-12-14 16:15

Message:
Logged In: YES 
user_id=154690
Originator: NO

1. - Yeah, epoch transitions are messy. We should handle them eventually,
but it would be nice to get an initial framework in. The problem comes up
in the following scenario: follower has (1,36), (1,37) and the leader has
(1,36),(2,0),(2,1),(2,2),(3,0),(3.1). When the follower syncs to the leader
the leader needs to know the ending txid of previous epochs to know how to
truncate. It's not hard to do, but it would be sync to get the syncing
correct before we add handling of previous epochs.

2. - I think Mahadev meant to handle this, but he has a typo in
FollowerZooKeper server. He "overrode" addCommittedRequest instead of
addCommittedProposal. We should really start using the @Override attribute
to catch this kind of stuff.

Also, I still think that the addCommittedProposal needs to go in the
synchronized block since we have a race condition such that a request may
be in between the the OutstandingChanges and CommittedProposals lists so
that the request doesn't get sent to the follower on the initial sync up.

----------------------------------------------------------------------

Comment By: fpj (fpj)
Date: 2007-12-14 15:23

Message:
Logged In: YES 
user_id=1926444
Originator: NO

It looks good to me, although I have a couple more questions/suggestions:

1- It's not clear if it would be messy to handle epoch transitions as all
you would need is an operator to determine if a pair succeeds, preceeds, or
is equal to another pair (epoch, txid); 

2- Is it true that all servers maintain this buffer of committed requests?
If the buffer does not span multiple epochs, then I don't understand the
reason for having all servers maintaining the buffer as it seems that only
the leader would need to maintain it.

-Flavio  

----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-13 22:14

Message:
Logged In: YES 
user_id=1926680
Originator: YES

with some more changes/documentation ... thanks ben
File Added: fast-sync-follower-leader-5.patch

----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-13 21:40

Message:
Logged In: YES 
user_id=1926680
Originator: YES

thanks ben... for the first one... since removing from outstanding changes
does not gurantee that the changes have been applied to the datatree, i am
doing that later so that I am sure its been applied to the datatree before
I add it to the committedLog. Doing it before would be ok as well but i
feel comfortable with this control flow.

for the second .. nice catch... you are right. I fixed that ... uploading
a new patch.

for the third, you are right this will not catch this case... but since
the max zxid committed will be much much greater than the followers zxid,
it will get the snapshot. The code gets a little messy if I have to take
care of all the epoch transactions. In this case the follower gets the
whole snapshot... what do you think?
File Added: fast-sync-follower-leader-4.patch

----------------------------------------------------------------------

Comment By: Benjamin Reed (breed)
Date: 2007-12-13 21:24

Message:
Logged In: YES 
user_id=154690
Originator: NO

FinalRequestProcessor: Shouldn't the zks.addCommittedProposal happen
inside the sync block of zks.outstandingChanges? I think that switch should
happen atomically. 

Follower: in the case of Leader.DIFF it would be nice to have a comment
that says after loadData // Now process the diffs that come from the leader
as normal updates. Also, the zxid of truncate is the last zxid to KEEP not
the first to DELETE right? In truncate() it looks like you are DELETEing
zxid and up.

FollowerHandler: It looks like you are missing a TRUNC case: Imagine the
follower has seen (1,32) (where 1 is epoch and 32 is the count) and the
leader has seen and committed: (1,31), (2,0), (2,1), (2,1). It doesn't look
like (1,32) will be truncated from the follower's log.

----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-13 19:38

Message:
Logged In: YES 
user_id=1926680
Originator: YES

uploading the patch becaue the previous one conflicts with the trunk..
File Added: fast-sync-follower-leader-3.patch

----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-12 20:37

Message:
Logged In: YES 
user_id=1926680
Originator: YES

can you guys review this patch before I go off for vacation? i want to get
it in so that it does not get stale.

----------------------------------------------------------------------

Comment By: Mahadev Konar (mahadevkonar)
Date: 2007-12-07 23:06

Message:
Logged In: YES 
user_id=1926680
Originator: YES

this patch applies to the current trunk. No specific changes.
File Added: fast-sync-follower-leader-2.patch

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1008546&aid=1844561&group_id=209147