I was looking for an example of a DHT implementation for a couple of projects I'm working on when I stumbled across NChord. I'm excited to see there is a project and it seems to have a lot of design goals similar to mine.
Unfortunately, I ran into a problem almost immediately running the NChordServer. I started up the first instance on port 666 and then started up a second on 667 with localhost 666 for a seed connection and I repeatedly get the same series of errors:
1/21/2009 10:13:45 PM DAVROS:667 (97112D2AB2EB4DC5) > : Chord Service registered on port 667.
1/21/2009 10:13:50 PM DAVROS:667 (97112D2AB2EB4DC5) > : Joining Ring @ localhost:666.
1/21/2009 10:13:51 PM DAVROS:667 (97112D2AB2EB4DC5) > : New Successor DAVROS:666 (FED6D1D5A92ECB63).
1/21/2009 10:14:05 PM DAVROS:667 (97112D2AB2EB4DC5) > : New Predecessor DAVROS:666 (FED6D1D5A92ECB63).
1/21/2009 10:14:22 PM DAVROS:667 (97112D2AB2EB4DC5) > : Unable to contact initial seed node localhost:666 (7660D37162011541). Re-Joining...
1/21/2009 10:14:22 PM DAVROS:667 (97112D2AB2EB4DC5) > : Joining Ring @ localhost:666.
1/21/2009 10:14:22 PM DAVROS:667 (97112D2AB2EB4DC5) > : Error occured during ReJoin (This BackgroundWorker is currently busy and cannot run multiple tasks concurrently.)
This essentially repeats from New Successor but without further New Predecessor lines.
The first instance on 666 only sees the first join and spits out the new Successor and Predecessor lines for 667 but does nothing.
I'm running this on Vista Ultimate x64 SP1 and built with VS2008 targetting the 2.0 runtime with AnyCPU. There are some mentions of the error message on forums basically saying if you get it, you did something wrong in your code because you are trying to re-use a BackgroundWorker before it has completed it's last task.
I thought for a bit it might be something you only see when two nodes can't communicate but the first node definitely sees the second try to join which would seem to indicate otherwise.
I'll poke around and see if I can find where it coming from and how to fix it but I thought I'd let you know in case it's putting off others that might participate if it seemed to work out of the box.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I added a stack trace to the error output. It appears it's happening in ChordInstance.Maintenance.cs on line 56 in the method StartMaintenance() with a call to m_StabilizeSuccessors.RunWorkerAsync(); at least in the release version.
I pulled down the latest from SVN to see if that would help but it suffers the same problem.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK, I think I located the problem... two problems, rather. The first problem is that I didn't realize you are generating the keys based on the hostname+port and stuffing that into m_SeedNode.ID then trying to compare it to what the seed node actually said it's key was. I was trying to connect to localhost:666 but the node in question used DAVROS:666 to generate it's own key.
That's easily fixed obviously and was my fault for not knowing the way you generate keys (which, by the way, will absolutely not work on the open internet) but it seems to have exposed a separate issue in the logic of Join starting the maintenance background workers which then calles ReJoin which will sometimes call Join thus starting the background workers again and throwing an error.
So, use the machine name when specifying the seed unless you want to go recode the way keys are made so it's less dependent on things that probably can't reasonably be known outside of private network.
For internet usage, you'd probably have to set m_SeedNode to whatever the seed tells you it's key is on the first successful connection. I don't know enough about why m_SeedNode.ID wouldn't be the Successor Node to know if there's something you'd have to be worry about when doing that.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
sorry, somehow i seem to have missed this thread. thanks for the great detective work on this issue.
for open internet use, yes, changing the seed from hostname to something more suitable (IP address, mac address, some other seed) would be a good thing.
i imagine the fix for this issue would be to check whether or not the background worker is running on-rejoin, and if it is, either re-queue the start of the process, or to drop the re-start alltogether. i haven't seen this problem manifest itself in my testing long ago but it does seem to make sense.
thanks for pointing it out Jason! i appreciate and welcome your contributions to the nChord project.
all the best,
--andrew
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
@ Jason ... How did u fix it .. ? Evn I am getting the same error ...
@ Andrew .. Have you used some network simulator to simulate nodes .. ? When I give the command line rguements as
80 localhost 81 .... it says invalid seed node .... and whn i give 80 192.168.66.125 80 (thats my pcs ip address) ... it gives same result as of Jason's ... !
How can I run it on a network .. ? Where it will have multiple nodes .. ?
Plz reply
Thanks
Pallavi
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
@ Jason ... How did u fix it .. ? Evn I am getting the same error ...
@ Andrew .. Have you used some network simulator to simulate nodes .. ? When I give the command line rguements as
80 localhost 81 .... it says invalid seed node .... and whn i give 80 192.168.66.125 80 (thats my pcs ip address) ... it gives same result as of Jason's ... !
How can I run it on a network .. ? Where it will have multiple nodes .. ?
Plz reply
Thanks
Pallavi
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Welcome to Help
I was looking for an example of a DHT implementation for a couple of projects I'm working on when I stumbled across NChord. I'm excited to see there is a project and it seems to have a lot of design goals similar to mine.
Unfortunately, I ran into a problem almost immediately running the NChordServer. I started up the first instance on port 666 and then started up a second on 667 with localhost 666 for a seed connection and I repeatedly get the same series of errors:
1/21/2009 10:13:45 PM DAVROS:667 (97112D2AB2EB4DC5) > : Chord Service registered on port 667.
1/21/2009 10:13:50 PM DAVROS:667 (97112D2AB2EB4DC5) > : Joining Ring @ localhost:666.
1/21/2009 10:13:51 PM DAVROS:667 (97112D2AB2EB4DC5) > : New Successor DAVROS:666 (FED6D1D5A92ECB63).
1/21/2009 10:14:05 PM DAVROS:667 (97112D2AB2EB4DC5) > : New Predecessor DAVROS:666 (FED6D1D5A92ECB63).
1/21/2009 10:14:22 PM DAVROS:667 (97112D2AB2EB4DC5) > : Unable to contact initial seed node localhost:666 (7660D37162011541). Re-Joining...
1/21/2009 10:14:22 PM DAVROS:667 (97112D2AB2EB4DC5) > : Joining Ring @ localhost:666.
1/21/2009 10:14:22 PM DAVROS:667 (97112D2AB2EB4DC5) > : Error occured during ReJoin (This BackgroundWorker is currently busy and cannot run multiple tasks concurrently.)
This essentially repeats from New Successor but without further New Predecessor lines.
The first instance on 666 only sees the first join and spits out the new Successor and Predecessor lines for 667 but does nothing.
I'm running this on Vista Ultimate x64 SP1 and built with VS2008 targetting the 2.0 runtime with AnyCPU. There are some mentions of the error message on forums basically saying if you get it, you did something wrong in your code because you are trying to re-use a BackgroundWorker before it has completed it's last task.
I thought for a bit it might be something you only see when two nodes can't communicate but the first node definitely sees the second try to join which would seem to indicate otherwise.
I'll poke around and see if I can find where it coming from and how to fix it but I thought I'd let you know in case it's putting off others that might participate if it seemed to work out of the box.
I added a stack trace to the error output. It appears it's happening in ChordInstance.Maintenance.cs on line 56 in the method StartMaintenance() with a call to m_StabilizeSuccessors.RunWorkerAsync(); at least in the release version.
I pulled down the latest from SVN to see if that would help but it suffers the same problem.
OK, I think I located the problem... two problems, rather. The first problem is that I didn't realize you are generating the keys based on the hostname+port and stuffing that into m_SeedNode.ID then trying to compare it to what the seed node actually said it's key was. I was trying to connect to localhost:666 but the node in question used DAVROS:666 to generate it's own key.
That's easily fixed obviously and was my fault for not knowing the way you generate keys (which, by the way, will absolutely not work on the open internet) but it seems to have exposed a separate issue in the logic of Join starting the maintenance background workers which then calles ReJoin which will sometimes call Join thus starting the background workers again and throwing an error.
So, use the machine name when specifying the seed unless you want to go recode the way keys are made so it's less dependent on things that probably can't reasonably be known outside of private network.
For internet usage, you'd probably have to set m_SeedNode to whatever the seed tells you it's key is on the first successful connection. I don't know enough about why m_SeedNode.ID wouldn't be the Successor Node to know if there's something you'd have to be worry about when doing that.
sorry, somehow i seem to have missed this thread. thanks for the great detective work on this issue.
for open internet use, yes, changing the seed from hostname to something more suitable (IP address, mac address, some other seed) would be a good thing.
i imagine the fix for this issue would be to check whether or not the background worker is running on-rejoin, and if it is, either re-queue the start of the process, or to drop the re-start alltogether. i haven't seen this problem manifest itself in my testing long ago but it does seem to make sense.
thanks for pointing it out Jason! i appreciate and welcome your contributions to the nChord project.
all the best,
--andrew
@ Jason ... How did u fix it .. ? Evn I am getting the same error ...
@ Andrew .. Have you used some network simulator to simulate nodes .. ? When I give the command line rguements as
80 localhost 81 .... it says invalid seed node .... and whn i give 80 192.168.66.125 80 (thats my pcs ip address) ... it gives same result as of Jason's ... !
How can I run it on a network .. ? Where it will have multiple nodes .. ?
Plz reply
Thanks
Pallavi
@ Jason ... How did u fix it .. ? Evn I am getting the same error ...
@ Andrew .. Have you used some network simulator to simulate nodes .. ? When I give the command line rguements as
80 localhost 81 .... it says invalid seed node .... and whn i give 80 192.168.66.125 80 (thats my pcs ip address) ... it gives same result as of Jason's ... !
How can I run it on a network .. ? Where it will have multiple nodes .. ?
Plz reply
Thanks
Pallavi