IDataService services aren't starting up?

Help
glitch253
2014-05-09
2014-06-10
  • glitch253
    glitch253
    2014-05-09

    Hey all -- we've deployed an installation of BigData in scale-out mode across 22 nodes (1 head node and 21 data nodes). It seems that data nodes which are working correctly have both an IServicesManagerService and an IDataService -- but on 7 of my nodes the IDataService won't start.

    More importantly we believe this causes the nanoSparqlServer.sh to not function correctly -- starting the nanoSparqlServer with my correct namespace and relevant port yields a 503 service undefined when we navigate there with a browser.

    Any information would be welcomed with much appreciation. I've pasted the output from my ./listServices below.

    JVM version is "Oracle Corporation 1.7.0_45".
    It probably doesn't matter, but Zookeeper nodes are also being used by Apache Accumulo.

    In the detail.log file there are a lot of these --
    "WARN [com.bigdata.service.jini.JiniFederation.scheduledService1] com.bigdata.service.jini.lookup.AbstractCachingServiceClient.getServiceItem(AbstractCachingServiceClient.java:293) 2014-05-08 20:47:39,760 - No matching service."

    Waiting 5000ms for service discovery.
    Zookeeper is running.
    Discovered 1 jini service registrars.
    192.168.2.210
    Discovered 41 services
    Discovered 0 stale bigdata services.
    Discovered 40 live bigdata services.
    Discovered 1 other services.
    Bigdata services by serviceIface:
    There are 22 instances of com.bigdata.jini.start.IServicesManagerService on 22 hosts
    There are 1 instances of com.bigdata.journal.ITransactionService on 1 hosts
    There are 1 instances of com.bigdata.service.IClientService on 1 hosts
    There are 14 instances of com.bigdata.service.IDataService on 14 hosts
    There are 1 instances of com.bigdata.service.ILoadBalancerService on 1 hosts
    There are 1 instances of com.bigdata.service.IMetadataService on 1 hosts
    Bigdata services by hostname:
    There are 2 live bigdata services on dcloud1.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud10.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud11.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud12.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 1 live bigdata services on dcloud13.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 2 live bigdata services on dcloud14.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 1 live bigdata services on dcloud15.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 2 live bigdata services on dcloud16.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 1 live bigdata services on dcloud17.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 live bigdata services on dcloud18.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 live bigdata services on dcloud19.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 2 live bigdata services on dcloud2.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 1 live bigdata services on dcloud20.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 live bigdata services on dcloud21.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 2 live bigdata services on dcloud3.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud4.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud5.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud6.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud7.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud8.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 2 live bigdata services on dcloud9.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.service.IDataService services
    There are 5 live bigdata services on dhead.ccri.com
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.journal.ITransactionService services
    There are 1 com.bigdata.service.IClientService services
    There are 1 com.bigdata.service.ILoadBalancerService services
    There are 1 com.bigdata.service.IMetadataService services

    [hdfs@dhead bin]$

    Thanks very much for any help!

     
    • Bryan Thompson
      Bryan Thompson
      2014-05-09

      The IServicesManager pulls the configuration information from zookeeper, examines the rules for starting services, and starts services.

      You can try the broadcast_sighup script. The IServicesManager will notice this, pull the data from zookeeper, and then apply the rules and attempt to start any local services.

      You should also examine the constraints in the configuration file that define which nodes should have what kinds of services and make sure that there are nodes for the rest of the data services to start on.

      There are a number of log outputs as well as the output from the crontab job that should be running the bigdata script that starts the IServicesManagerService. You should check the logs for error messages related to a failed service start.

      The DataServices host the index partitions. They need to be up for operations to proceed correctly.

      Thanks,
      Bryan

      From: Mike P glitch253@users.sf.net<mailto:glitch253@users.sf.net>
      Reply-To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
      Date: Thursday, May 8, 2014 9:06 PM
      To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
      Subject: [bigdata:discussion] IDataService services aren't starting up?

      Hey all -- we've deployed an installation of BigData in scale-out mode across 22 nodes (1 head node and 21 data nodes). It seems that data nodes which are working correctly have both an IServicesManagerService and an IDataService -- but on 7 of my nodes the IDataService won't start.

      More importantly we believe this causes the nanoSparqlServer.sh to not function correctly -- starting the nanoSparqlServer with my correct namespace and relevant port yields a 503 service undefined when we navigate there with a browser.

      Any information would be welcomed with much appreciation. I've pasted the output from my ./listServices below.

      JVM version is "Oracle Corporation 1.7.0_45".
      It probably doesn't matter, but Zookeeper nodes are also being used by Apache Accumulo.

      In the detail.log file there are a lot of these --
      "WARN [com.bigdata.service.jini.JiniFederation.scheduledService1] com.bigdata.service.jini.lookup.AbstractCachingServiceClient.getServiceItem(AbstractCachingServiceClient.java:293) 2014-05-08 20:47:39,760 - No matching service."

      Waiting 5000ms for service discovery.
      Zookeeper is running.
      Discovered 1 jini service registrars.
      192.168.2.210
      Discovered 41 services
      Discovered 0 stale bigdata services.
      Discovered 40 live bigdata services.
      Discovered 1 other services.
      Bigdata services by serviceIface:
      There are 22 instances of com.bigdata.jini.start.IServicesManagerService on 22 hosts
      There are 1 instances of com.bigdata.journal.ITransactionService on 1 hosts
      There are 1 instances of com.bigdata.service.IClientService on 1 hosts
      There are 14 instances of com.bigdata.service.IDataService on 14 hosts
      There are 1 instances of com.bigdata.service.ILoadBalancerService on 1 hosts
      There are 1 instances of com.bigdata.service.IMetadataService on 1 hosts
      Bigdata services by hostname:
      There are 2 live bigdata services on dcloud1.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud10.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud11.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud12.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 1 live bigdata services on dcloud13.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 2 live bigdata services on dcloud14.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 1 live bigdata services on dcloud15.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 2 live bigdata services on dcloud16.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 1 live bigdata services on dcloud17.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 live bigdata services on dcloud18.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 live bigdata services on dcloud19.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 2 live bigdata services on dcloud2.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 1 live bigdata services on dcloud20.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 live bigdata services on dcloud21.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 2 live bigdata services on dcloud3.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud4.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud5.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud6.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud7.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud8.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 2 live bigdata services on dcloud9.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.service.IDataService services
      There are 5 live bigdata services on dhead.ccri.com
      There are 1 com.bigdata.jini.start.IServicesManagerService services
      There are 1 com.bigdata.journal.ITransactionService services
      There are 1 com.bigdata.service.IClientService services
      There are 1 com.bigdata.service.ILoadBalancerService services
      There are 1 com.bigdata.service.IMetadataService services

      [hdfs@dhead bin]$

      Thanks very much for any help!


      IDataService services aren't starting up?https://sourceforge.net/p/bigdata/discussion/676946/thread/7bc27c1e/?limit=50#abac


      Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

      To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

       
      Attachments
  • glitch253
    glitch253
    2014-05-09

    BIGDATA_RELEASE_1_3_0 -- from the branches section of SVN

     
  • glitch253
    glitch253
    2014-05-09

    Okay, good call on the log4j thing -- I should have looked at that sooner. So that's good that you pointed it out. There's a warning that looks like it might be the cause of my problems:

    So it looks like its trying to find a .war file somewhere -- but I'm not entirely sure where that's supposed to be.

    WARN [main] org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:506) 2014-05-09 11:14:04,338 - Failed startup of context o.e.j.w.WebAppContext@511c7571{/bigdata,null,null}{bigdata-war/src}
    java.io.FileNotFoundException: bigdata-war/src
    at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:493)
    at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:72)
    at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:460)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:496)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:125)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:125)
    at org.eclipse.jetty.server.Server.start(Server.java:358)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
    at org.eclipse.jetty.server.Server.doStart(Server.java:325)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at com.bigdata.rdf.sail.webapp.NanoSparqlServer.main(NanoSparqlServer.java:339)

     
    Last edit: glitch253 2014-05-09
    • Bryan Thompson
      Bryan Thompson
      2014-05-09

      Ok. This could be a problem with the 1.3.0 branch after the merge down of the RDR branch. It is probably not staging the web application files correctly for the scale-out deployment.

      That RDR => main branch merge was done very recently. We have not tested the scale-out cluster yet. That happens mainly during active cluster development and during QA for a release (which is coming up, so I would have hit this soon).

      You can checkout an earlier revision and it should not have this problem. Try prior to r8160 (r8160 is the merge back of the RDR branch to the main development branch).

      The web app is now correctly modeled in bigdata-war/src in SVN. Prior to the RDR merge down it was not modeled correctly. There is going to be something about the scale-out deployment that is assuming the old structuring of the web app.

      Thanks,
      Bryan

      From: Mike P glitch253@users.sf.net<mailto:glitch253@users.sf.net>
      Reply-To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
      Date: Friday, May 9, 2014 12:14 PM
      To: "[bigdata:discussion]" 676946@discussion.bigdata.p.re.sf.net<mailto:676946@discussion.bigdata.p.re.sf.net>
      Subject: [bigdata:discussion] IDataService services aren't starting up?

      Okay, good call on the log4j thing -- I should have looked at that sooner. So that's good that you pointed it out. Where's a warning that looks like it might be the cause of my problems:

      So it looks like its trying to find a .war file somewhere -- but I'm not entirely sure where that's supposed to be.

      WARN [main] org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:506) 2014-05-09 11:14:04,338 - Failed startup of context o.e.j.w.WebAppContext@511c7571{/bigdata,null,null}{bigdata-war/src}
      java.io.FileNotFoundException: bigdata-war/src
      at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:493)
      at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:72)
      at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:460)
      at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:496)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:125)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:125)
      at org.eclipse.jetty.server.Server.start(Server.java:358)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
      at org.eclipse.jetty.server.Server.doStart(Server.java:325)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at com.bigdata.rdf.sail.webapp.NanoSparqlServer.main(NanoSparqlServer.java:339)


      IDataService services aren't starting up?https://sourceforge.net/p/bigdata/discussion/676946/thread/7bc27c1e/?limit=25#cfd2


      Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

      To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

       
      Attachments
      • glitch253
        glitch253
        2014-05-09

        Ok I'll try it out -- thanks for the info and your patience!

         
        • glitch253
          glitch253
          2014-05-09

          Okay so I checked the tags/1_3_0_RELEASE this time.

          Starting jini services on the head node gives me the "timeout value is negative" that I pasted below.

          And it looks like that causes some other services not to start right. All datanodes are functioning correctly.

          Here's the listServices.sh output for the head node:

          There are 3 live bigdata services on dhead.ccri.com
          There are 1 com.bigdata.jini.start.IServicesManagerService services
          There are 1 com.bigdata.journal.ITransactionService services
          There are 1 com.bigdata.service.IClientService services

          Looks like I'm missing the metadata service and the loadbalancer service.

          Just out of curiosity I tried starting up the nanosparqlserver and I got this:

          ERROR: 1 main com.bigdata.Banner$1.uncaughtException(Banner.java:111): Uncaught exception in thread
          java.lang.IllegalStateException: Unknown tag: Arg

          ERROR: 15551 main com.bigdata.jini.start.ServicesManagerStartupTask.startJiniCoreServices(ServicesManagerStartupTask.java:257): Could not start jini services: java.lang.IllegalArgumentException: timeout value is negative
          java.lang.IllegalArgumentException: timeout value is negative
          at java.lang.Object.wait(Native Method)
          at com.bigdata.jini.start.config.JiniCoreServicesConfiguration.getServiceRegistrars(JiniCoreServicesConfiguration.java:440)
          at com.bigdata.jini.start.process.JiniCoreServicesProcessHelper.startCoreServices(JiniCoreServicesProcessHelper.java:150)
          at com.bigdata.jini.start.ServicesManagerStartupTask.startJiniCoreServices(ServicesManagerStartupTask.java:252)
          at com.bigdata.jini.start.ServicesManagerStartupTask.doStartup(ServicesManagerStartupTask.java:156)
          at com.bigdata.jini.start.ServicesManagerStartupTask.call(ServicesManagerStartupTask.java:105)
          at com.bigdata.jini.start.AbstractServicesManagerService.setup(AbstractServicesManagerService.java:306)
          at com.bigdata.jini.start.AbstractServicesManagerService.start(AbstractServicesManagerService.java:240)
          at com.bigdata.jini.start.ServicesManagerServer$AdministrableServicesManagerService.start(ServicesManagerServer.java:564)
          at com.bigdata.jini.start.ServicesManagerServer$AdministrableServicesManagerService.start(ServicesManagerServer.java:404)
          at com.bigdata.service.jini.AbstractServer.<init>(AbstractServer.java:851)
          at com.bigdata.jini.start.ServicesManagerServer.<init>(ServicesManagerServer.java:354)
          at com.bigdata.jini.start.ServicesManagerServer.main(ServicesManagerServer.java:372)

           
          • Bryan Thompson
            Bryan Thompson
            2014-05-09

            I would suspect a bad install.

            I will be running through the scale-out cluster QA soon in preparation for 1.3.1. If you wait for that, you can run the most current code. For the 1.3.0 release, I would check your install.

            Bryan

            On May 9, 2014, at 3:28 PM, "Mike P" glitch253@users.sf.net wrote:

            Okay so I checked the tags/1_3_0_RELEASE this time.

            Starting jini services on the head node gives me the "timeout value is negative" that I pasted below.

            And it looks like that causes some other services not to start right. All datanodes are functioning correctly.

            Here's the listServices.sh output for the head node:

            There are 3 live bigdata services on dhead.ccri.com
            There are 1 com.bigdata.jini.start.IServicesManagerService services
            There are 1 com.bigdata.journal.ITransactionService services
            There are 1 com.bigdata.service.IClientService services

            Looks like I'm missing the metadata service and the loadbalancer service.

            Just out of curiosity I tried starting up the nanosparqlserver and I got this:

            ERROR: 1 main com.bigdata.Banner$1.uncaughtException(Banner.java:111): Uncaught exception in thread
            java.lang.IllegalStateException: Unknown tag: Arg

            ERROR: 15551 main com.bigdata.jini.start.ServicesManagerStartupTask.startJiniCoreServices(ServicesManagerStartupTask.java:257): Could not start jini services: java.lang.IllegalArgumentException: timeout value is negative
            java.lang.IllegalArgumentException: timeout value is negative
            at java.lang.Object.wait(Native Method)
            at com.bigdata.jini.start.config.JiniCoreServicesConfiguration.getServiceRegistrars(JiniCoreServicesConfiguration.java:440)
            at com.bigdata.jini.start.process.JiniCoreServicesProcessHelper.startCoreServices(JiniCoreServicesProcessHelper.java:150)
            at com.bigdata.jini.start.ServicesManagerStartupTask.startJiniCoreServices(ServicesManagerStartupTask.java:252)
            at com.bigdata.jini.start.ServicesManagerStartupTask.doStartup(ServicesManagerStartupTask.java:156)
            at com.bigdata.jini.start.ServicesManagerStartupTask.call(ServicesManagerStartupTask.java:105)
            at com.bigdata.jini.start.AbstractServicesManagerService.setup(AbstractServicesManagerService.java:306)
            at com.bigdata.jini.start.AbstractServicesManagerService.start(AbstractServicesManagerService.java:240)
            at com.bigdata.jini.start.ServicesManagerServer$AdministrableServicesManagerService.start(ServicesManagerServer.java:564)
            at com.bigdata.jini.start.ServicesManagerServer$AdministrableServicesManagerService.start(ServicesManagerServer.java:404)
            at com.bigdata.service.jini.AbstractServer.(AbstractServer.java:851)
            at com.bigdata.jini.start.ServicesManagerServer.(ServicesManagerServer.java:354)
            at com.bigdata.jini.start.ServicesManagerServer.main(ServicesManagerServer.java:372)

            IDataService services aren't starting up?

            Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

            To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

             
            Attachments
            • Bryan Thompson
              Bryan Thompson
              2014-05-15

              I have filed a ticket for this [1]. The issue shows up in the single
              machine NSS starts that we use for QA. It is blocking the 1.3.1 release
              and will be resolved shortly.

              Thanks,
              Bryan

              [1] http://trac.bigdata.com/ticket/939 (NSS does not start from command
              line: bigdata-war/src not found)

              On Fri, May 9, 2014 at 6:44 PM, Bryan Thompson thompsonbry@users.sf.netwrote:

              I would suspect a bad install.

              I will be running through the scale-out cluster QA soon in preparation for
              1.3.1. If you wait for that, you can run the most current code. For the
              1.3.0 release, I would check your install.

              Bryan

              On May 9, 2014, at 3:28 PM, "Mike P" glitch253@users.sf.net wrote:

              Okay so I checked the tags/1_3_0_RELEASE this time.

              Starting jini services on the head node gives me the "timeout value is
              negative" that I pasted below.

              And it looks like that causes some other services not to start right. All
              datanodes are functioning correctly.

              Here's the listServices.sh output for the head node:

              There are 3 live bigdata services on dhead.ccri.com
              There are 1 com.bigdata.jini.start.IServicesManagerService services
              There are 1 com.bigdata.journal.ITransactionService services
              There are 1 com.bigdata.service.IClientService services

              Looks like I'm missing the metadata service and the loadbalancer service.

              Just out of curiosity I tried starting up the nanosparqlserver and I got
              this:

              ERROR: 1 main com.bigdata.Banner$1.uncaughtException(Banner.java:111):
              Uncaught exception in thread
              java.lang.IllegalStateException: Unknown tag: Arg

              ERROR: 15551 main
              com.bigdata.jini.start.ServicesManagerStartupTask.startJiniCoreServices(ServicesManagerStartupTask.java:257):
              Could not start jini services: java.lang.IllegalArgumentException: timeout
              value is negative
              java.lang.IllegalArgumentException: timeout value is negative
              at java.lang.Object.wait(Native Method)
              at
              com.bigdata.jini.start.config.JiniCoreServicesConfiguration.getServiceRegistrars(JiniCoreServicesConfiguration.java:440)
              at
              com.bigdata.jini.start.process.JiniCoreServicesProcessHelper.startCoreServices(JiniCoreServicesProcessHelper.java:150)
              at
              com.bigdata.jini.start.ServicesManagerStartupTask.startJiniCoreServices(ServicesManagerStartupTask.java:252)
              at
              com.bigdata.jini.start.ServicesManagerStartupTask.doStartup(ServicesManagerStartupTask.java:156)
              at
              com.bigdata.jini.start.ServicesManagerStartupTask.call(ServicesManagerStartupTask.java:105)
              at
              com.bigdata.jini.start.AbstractServicesManagerService.setup(AbstractServicesManagerService.java:306)
              at
              com.bigdata.jini.start.AbstractServicesManagerService.start(AbstractServicesManagerService.java:240)
              at
              com.bigdata.jini.start.ServicesManagerServer$AdministrableServicesManagerService.start(ServicesManagerServer.java:564)
              at
              com.bigdata.jini.start.ServicesManagerServer$AdministrableServicesManagerService.start(ServicesManagerServer.java:404)
              at com.bigdata.service.jini.AbstractServer.(AbstractServer.java:851)
              at
              com.bigdata.jini.start.ServicesManagerServer.(ServicesManagerServer.java:354)
              at
              com.bigdata.jini.start.ServicesManagerServer.main(ServicesManagerServer.java:372)

              IDataService services aren't starting up?

              Sent from sourceforge.net because you indicated interest in
              https://sourceforge.net/p/bigdata/discussion/676946/

              To unsubscribe from further messages, please visit
              https://sourceforge.net/auth/subscriptions/


              IDataService services aren't starting up?http://sourceforge.net/p/bigdata/discussion/676946/thread/7bc27c1e/?limit=25#cfd2/44cb/f0a0/923b/6e26

              Sent from sourceforge.net because you indicated interest in
              https://sourceforge.net/p/bigdata/discussion/676946/

              To unsubscribe from further messages, please visit
              https://sourceforge.net/auth/subscriptions/

               
    • Bryan Thompson
      Bryan Thompson
      2014-05-21

      The 1.3.1 release is out. The cluster should start smoothly.
      Bryan

      On May 9, 2014, at 12:14 PM, "Mike P" glitch253@users.sf.net wrote:

      Okay, good call on the log4j thing -- I should have looked at that sooner. So that's good that you pointed it out. Where's a warning that looks like it might be the cause of my problems:

      So it looks like its trying to find a .war file somewhere -- but I'm not entirely sure where that's supposed to be.

      WARN [main] org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:506) 2014-05-09 11:14:04,338 - Failed startup of context o.e.j.w.WebAppContext@511c7571{/bigdata,null,null}{bigdata-war/src}
      java.io.FileNotFoundException: bigdata-war/src
      at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:493)
      at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:72)
      at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:460)
      at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:496)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:125)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:125)
      at org.eclipse.jetty.server.Server.start(Server.java:358)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
      at org.eclipse.jetty.server.Server.doStart(Server.java:325)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at com.bigdata.rdf.sail.webapp.NanoSparqlServer.main(NanoSparqlServer.java:339)

      IDataService services aren't starting up?

      Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/bigdata/discussion/676946/

      To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

       
      Attachments
      • glitch253
        glitch253
        2014-06-10

        Bryan,

        I missed this message earlier, but thank you very much for the follow up! I'll give the deployment another shot and let you know how it goes.