Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

cant remove DomainHealth_WLDFHarvesterModule

Help
2011-02-16
2013-04-25
  • Hi
    I deployed Domainheallth v09 on WLS 10.3.2 (solaris 32bit). I Targeted it only to the Admin and used the -appversion switch with the weblogic.Deployer command.

    After three days, the UI was only showing stats for two server instances out of the 11 in the domain (the admin and 1 managed).
    I looked at the WLDF Module that it creates automatically and it was Targeted to 'Part of cluster - but all managed servers' on all clusters, not to the option which indicates the whole clusters.
    So I figured that this must be affecting the harvester.

    I re-targeted the Module to the clusters and when I went to commit the changes, I got the error saying I have to restart the servers. I restarted the Admin server, I tried undeploying the app, but it won´t let me unless I restart the managed servers in the whole domain. That is a bit of a problem in a production environment. 'Doable' but doesn't look good. Maybe before the deployment of domainhealth there were pending task which requiere server restart.

    The logs show that it attempts to delete the prior version of the WLDF Module, and tries to recreate it but gets that error.
    Thank you

    ####<Feb 16, 2011 3:39:10 PM CLST> <Warning> <Deployer> <myserver> <Webpay3gAdmin> < ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1297881550368> <BEA-149004> <Failures were detected while initiating remove task for application 'DomainHealth_WLDFHarvesterModule'.>
    ####<Feb 16, 2011 3:39:10 PM CLST> <Warning> <Deployer> <myserver> <Webpay3gAdmin> < ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1297881550370> <BEA-149078> <Stack trace for message 149004
    weblogic.management.DeploymentException: Attempt to operate 'remove' on null BasicDeploymentMBean for deployment DomainHealth_WLDFHarvesterModule. Operation can not be performed until server is restarted.
            at weblogic.deploy.internal.targetserver.DeploymentManager.assertDeploymentMBeanIsNonNull(DeploymentManager.java:1319)
            at weblogic.deploy.internal.targetserver.DeploymentManager.findDeploymentMBean(DeploymentManager.java:1360)
            at weblogic.deploy.internal.targetserver.DeploymentManager.createOperation(DeploymentManager.java:1122)
            at weblogic.deploy.internal.targetserver.DeploymentManager.createOperations(DeploymentManager.java:1406)
            at weblogic.deploy.internal.targetserver.DeploymentManager.handleUpdateDeploymentContext(DeploymentManager.java:162)
            at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.updateDeploymentContext(DeploymentServiceDispatcher.java:155)
            at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doUpdateDeploymentContextCallback(DeploymentReceiverCallbackDeliverer.java:133)
            at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.updateDeploymentContext(DeploymentReceiverCallbackDeliverer.java:27)
            at weblogic.deploy.service.internal.statemachines.targetserver.ReceivedPrepare.callDeploymentReceivers(ReceivedPrepare.java:203)
            at weblogic.deploy.service.internal.statemachines.targetserver.ReceivedPrepare.handlePrepare(ReceivedPrepare.java:112)
            at weblogic.deploy.service.internal.statemachines.targetserver.ReceivedPrepare.receivedPrepare(ReceivedPrepare.java:52)
            at weblogic.deploy.service.internal.targetserver.TargetRequestImpl.run(TargetRequestImpl.java:211)
            at weblogic.deploy.service.internal.transport.CommonMessageReceiver$1.run(CommonMessageReceiver.java:457)
            at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
            at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
    >
    ####<Feb 16, 2011 3:39:10 PM CLST> <Critical> <DomainHealth> <myserver> <Webpay3gAdmin> < ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1297881550841> <BEA-000000> <Unable to configure WDLF Harvester module. Cause: javax.management.RuntimeMBeanException: weblogic.descriptor.BeanAlreadyExistsException: Bean already exists: "weblogic.diagnostics.descriptor.WLDFHarvestedTypeBeanImpl@37b0ed6(/Harvester/HarvestedTypes)">
    ####<Feb 16, 2011 3:39:11 PM CLST> <Error> <DomainHealth> <myserver> <Webpay3gAdmin> < ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1297881551475> <BEA-000000> <Statistics Retriever Background Service - first time processing initialisation failed - will attempt initialisation again after a pause (this can be caused if the server takes a while to start-up, in which case, the retry should sort things out). Cause: domainhealth.core.jmx.WebLogicMBeanException: DomainHealth failedto configure WDLF Harvester module and retirement policy>

     
  • Rebooting all instances made the error go away.
    But still, domainhealth shows no data for instances on the remote machine. There is data only for managed servers running in the same machine that the Admin is on.
    So I turn off WLDF harvesting and went back to the old JMX polling.
    I am now getting the data from all instances.

     
  • Paul Done
    Paul Done
    2011-02-25

    Hi

    Sorry you had problems with the WLDF backend. This is strange because I've tested that this works again very recently from multiple machines. Do you have a multi-homed Solaris machine (ie. multiple IP addresses per machine?

    Note: You should always just target the Admin Server for deployment of DH, regardless of how many managted servers in the domain.,

    Thx

    Paul

     
  • It happened again.
    I manualy removed the WLDF Module of version 0.9 and then undeployed the application.
    After that, I deployed version 1.0 (including machinestats02) and got the error "Bean already exists". Details below.

    I tried bouncing all managed servers and admin server (not at the same time) thinking maybe the mbean was tied to an instance, but no luck.
    Any clues?

    ####<Dec 16, 2011 3:21:51 PM CLST> <Critical> <DomainHealth> <machine> <AdminPortalWLSProd> <domainhealth.backend.retriever.RetrieverBackgroundService> <weblogic> <> <> <1324059711743> <BEA-000000> <Unable to configure WDLF Harvester module. Cause: javax.management.RuntimeMBeanException: weblogic.descriptor.BeanAlreadyExistsException: Bean already exists: "weblogic.diagnostics.descriptor.WLDFHarvestedTypeBeanImpl@7ebe0df4(/Harvester/HarvestedTypes)">
    ####<Dec 16, 2011 3:21:52 PM CLST> <Error> <DomainHealth> <machine> <AdminPortalWLSProd> <domainhealth.backend.retriever.RetrieverBackgroundService> <weblogic> <> <> <1324059712718> <BEA-000000> <Statistics Retriever Background Service - first time processing initialisation failed - will attempt initialisation again after a pause (this can be caused if the server takes a while to start-up, in which case, the retry should sort things out). Cause: domainhealth.core.jmx.WebLogicMBeanException: DomainHealth failed to configure WDLF Harvester module and retirement policy>

    I also tried
    java weblogic.Admin -url t3://allmachines:port -username username DELETE -mbean somestring
    But can't find the correct "somestring"

     
  • Paul Done
    Paul Done
    2012-01-23

    Looks like a problem with WLS admin not clearing up WLDF (not a problem with DomainHealth directly but just WLS admin). If all else fails, may be worth shutting down all servers, then go to <domainhome>/config/config.xml and using vi or other text editor, search for any remenants of DH deployments or WLDF modules or retirement policies in there (eg. <wldf-data-retirement-by-age>). Also delete any DH related .xml files from  <domainhome>/config//diagnostics sub-dir>.

    Paul

     
  • I did everything you said.
    DH deployed without errors.
    Thank you very much!