Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

maxDataServicesPerHost / SSDs / RAID

Help
hauptmac
2014-05-30
2014-06-11
  • hauptmac
    hauptmac
    2014-05-30

    Hello, thank you for sharing this great graph database!

    I am experimenting with it and I am wondering why I should have multiple data services per host since there would be more effort for the coordination between the processes.

    If I have multiple CPUs (with several cores) - is it a good idea to have one process / data service per cpu?

    Regarding multiple SSDs: Do you recommend a RAID or one dedicated SSD per data service on a host?

    Thank you so much in advance and best regards from Munich!

     
    • Bryan Thompson
      Bryan Thompson
      2014-05-30

      One DS per compute node is appropriate. Each data service can use many hardware threads. The code base does not yet use async IO (we have only recently converged on java 7 due to support requirements for java 6 customers) so threads are still required to schedule IOs.

      Enterprise grade PCIe flash is a better choice than raid of SSD disks. A single SSD will provide more IOPS than raided spinning disk. One SSD per DS is sufficient depending on the data scale.

      Thanks,
      Bryan

      On May 30, 2014, at 6:07 AM, "Claudius Hauptmann" hauptmac@users.sf.net wrote:

      Hello, thank you for sharing this great graph database!

      I am experimenting with it and I am wondering why I should have multiple data services per host since there would be more effort for the coordination between the processes.

      If I have multiple CPUs (with several cores) - is it a good idea to have one process / data service per cpu?

      Regarding multiple SSDs: Do you recommend a RAID or one dedicated SSD per data service on a host?

      Thank you so much in advance and best regards from Munich!

      maxDataServicesPerHost / SSDs / RAID

      Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

      To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

       
      Attachments
      • hauptmac
        hauptmac
        2014-05-30

        Hello Bryan,

        thank you for your advice! So I will start a bigdata federation with 1 node, 1 DS and 1 ssd (the fastest one can buy).

        As far as I understand I can add additional ds nodes to the Federation and send a HUP to update all nodes. Therefore I have two more questions:

        Will some of the already existing triples automatically be moved to the new nodes?

        How can I "cool down" and "empty" an older ds into a newer ds to remove it from the federation (maybe because the hardware is no longer fast enough)?

        Thanks,
        Claudius

         
        • Bryan Thompson
          Bryan Thompson
          2014-05-30

          Bigdata redistributes shards to balance the write pressure on the indices. If you change the number of data services and apply new writes, then it will begin to redistribute shards.

          It might be pretty easy to script this as well. What you need to do is force the data service to close out the existing journal (used to absorb writes) and open a new one. This is done through the DS API using a force overflow option. That will start the asynchronous processing that handles the dynamic sharding. The only catch is that it might not initiate shard moves without write pressure on the database. If so, then automating this might require a trigger that is understood by the DS (code change).

          There is no mechanism to automate that cool down. You would want to look at the DS operations for forcing shard moves. This would need to be a modal logic change. There would need to be a way to mark a given DS as no longer a target for shard moves and then have the DS redistribute the shards. Simple enough conceptually.

          Bryan

          On May 30, 2014, at 11:09 AM, "hauptmac" hauptmac@users.sf.net wrote:

          Hello Bryan,

          thank you for your advice! So I will start a bigdata federation with 1 node, 1 DS and 1 ssd (the fastest one can buy).

          As far as I understand I can add additional ds nodes to the Federation and send a HUP to update all nodes. Therefore I have two more questions:

          Will some of the already existing triples automatically be moved to the new nodes?

          How can I "cool down" and "empty" an older ds into a newer ds to remove it from the federation (maybe because the hardware is no longer fast enough)?

          Thanks,
          Claudius

          maxDataServicesPerHost / SSDs / RAID

          Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

          To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

           
          Attachments
  • hauptmac
    hauptmac
    2014-05-30

    Hello Bryan,

    thanks! It looks like the shouldMove method of the AsynchronousOverflowTask could be a good starting point for "cooling down" and stopping a data service. MayBe there is a chance to put a "cool down" flag somewhere which is read by the shouldMove method.

    http://www.bigdata.com/docs/api/com/bigdata/resources/AsynchronousOverflowTask.html

    I will have a deeper look into the code to figure out how to get this done and where to trigger the AsynchronousOverflowTask.

    Claudius

     
    • hauptmac
      hauptmac
      2014-05-30

      After some investigation the following two classes seem to be helpful. An export followed by a reconfiguration and an import (which is not that fast) may not be the best solution. Especially since bigdata has got very cool and advanced load balancing mechanisms that move data around. I hope I will find some time to implement a cool down + shut down feature for nodes that should be removed from the federation.

      http://www.bigdata.com/docs/api/com/bigdata/resources/OverflowManager.html

      http://www.bigdata.com/docs/api/com/bigdata/journal/WriteExecutorService.html

       
      • Bryan Thompson
        Bryan Thompson
        2014-05-31

        Yes, those are key classes. However, the main integration point is the AsynchronousOverflowFactory, which is responsible for the actually dynamic sharding decisions and would be the class to schedule the moves of the shards from the data service that is being shutdown to other services in the cluster. The data service that is being shutdown would also need to refuse shard moves onto that service. I would study the move index partition task to see where to best introduce the logic to refuse a shard move.

        The metadata about the data service state could go into zookeeper or jini/river. Or it could be a simple, transient state directly set through the IDataService API. The latter is probably the easiest to implement and perhaps the most appropriate.

        The move of the shards from the service that you are shutting down should be done by dispersing the shards in inverse proportion to the existing write load on the other services. However, it should not be done too rapidly as the data service must still accept writes for shards on that service. You may have 1000s of shard that need to be moved. That can take a while if there is also a lot of write pressure on those shards.

        The asynchronous processing should not take so long that the journal absorbing writes becomes significantly overextended. If it does, it will create a higher asynchronous overflow processing burden for the following iteration (once the new journal fills up). It is easy enough to balance this by having the service refuse to schedule new moves of shards away from itself once the journal is already at its nominal capacity. It is ok for the journal to extend beyond that nominal capacity, but you will eventually get into trouble if too many buffered writes need to be cleared from the next journal.

        If you use shared disk, e.g., a parallel file system, then you can just failover the service onto another node. The new node takes over the same service and accesses the same files in the shared file system. The shard moves are only required for a shared nothing deployment.

        Another way to handle this is to synchronize the disk from the node you want to shutdown to a new node that is not a federation member. You then would fail over the data service at the top of the asynchronous overflow handler and have the new machine take over the responsibility for the same index partitions. This approach is more dependent on how the service deployments are actually managed.

        If you work up and test a patch and can sign our contributor license agreement, then I can review it and merge it into the main code base.

        Thanks,
        Bryan

        On May 30, 2014, at 6:51 PM, "hauptmac" hauptmac@users.sf.net wrote:

        After some investigation the following two classes seem to be helpful. An export followed by a reconfiguration and an import (which is not that fast) may not be the best solution. Especially since bigdata has got very cool and advanced load balancing mechanisms that move data around. I hope I will find some time to implement a cool down + shut down feature for nodes that should be removed from the federation.

        http://www.bigdata.com/docs/api/com/bigdata/resources/OverflowManager.html

        http://www.bigdata.com/docs/api/com/bigdata/journal/WriteExecutorService.html

        maxDataServicesPerHost / SSDs / RAID

        Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

        To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

         
        Attachments
  • hauptmac
    hauptmac
    2014-06-11

    thank you for the detailed answer and suggestions! I will try to find a solution as soon as we need to remove nodes and I think the agreement will be no problem.