Menu ▾ ▴

#1313 osaf: opensaf does not start when long dn object is present in imm.db and cluster is reset

never
duplicate
Mathi Naickan
None
defect
log
-
4.6
major
2015-09-24
2015-04-13
Sirisha Alla
No

This is observed on changeset 6377 (46FC Tag). The system is up with single pbe and 50k objects. Long dns was enabled. There is one long dn object in the cluster.

Syslog on SC-1:

Apr 9 15:49:14 SLES-64BIT-SLOT1 osafimmnd[10731]: WA Setting attr longDnsAllowed to 0 in opensafImm=opensafImm,safApp=safImmService not allowed when long RDN exists inside object: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxattrName_testAdminOwnerClear_SubLevelScope_1011

Now the cluster is reset. Nodes in the cluster fail to come up with the following reason:

Apr 13 13:04:55 SLES-64BIT-SLOT1 osafimmnd[3439]: NO Persistent Back End OI attached, pid: 3465
Apr 13 13:04:55 SLES-64BIT-SLOT1 osafimmnd[3439]: NO Implementer connected: 1 (OpenSafImmPBE) <10, 2010f>
Apr 13 13:04:55 SLES-64BIT-SLOT1 osafimmnd[3439]: NO implementer for class 'OpensafImm' is OpenSafImmPBE => class extent is safe.
Apr 13 13:04:55 SLES-64BIT-SLOT1 osafimmpbed: NO Update epoch 20 committing with ccbId:100000003/4294967299
Apr 13 13:04:56 SLES-64BIT-SLOT1 osafimmnd[3439]: NO PBE-OI established on this SC. Dumping incrementally to file imm.db
Apr 13 13:05:34 SLES-64BIT-SLOT1 opensafd[3378]: ER Timed-out for response from LOGD
Apr 13 13:05:34 SLES-64BIT-SLOT1 opensafd[3378]: ER
Apr 13 13:05:34 SLES-64BIT-SLOT1 opensafd[3378]: ER Going for recovery
Apr 13 13:05:34 SLES-64BIT-SLOT1 opensafd[3378]: ER Trying To RESPAWN /usr/lib64/opensaf/clc-cli/osaf-logd attempt #1
Apr 13 13:05:34 SLES-64BIT-SLOT1 opensafd[3378]: ER Sending SIGKILL to LOGD, pid=3452
Apr 13 13:05:49 SLES-64BIT-SLOT1 osaflogd[3500]: Started
Apr 13 13:05:49 SLES-64BIT-SLOT1 osaflogd[3500]: WA read_logsv_configuration(). All attributes could not be read
Apr 13 13:05:49 SLES-64BIT-SLOT1 osaflogd[3500]: NO Log config system: high 0 low 0, application: high 0 low 0
Apr 13 13:05:49 SLES-64BIT-SLOT1 osaflogd[3500]: NO log root directory is: /var/log/opensaf/saflog
Apr 13 13:05:49 SLES-64BIT-SLOT1 osaflogd[3500]: NO LOG data group is:
Apr 13 13:05:49 SLES-64BIT-SLOT1 osaflogd[3500]: NO LGS_MBCSV_VERSION = 4
Apr 13 13:05:49 SLES-64BIT-SLOT1 osaflogd[3500]: saImmOmSearchInitialize FAILED, rc = 13
Apr 13 13:06:29 SLES-64BIT-SLOT1 opensafd[3378]: ER Timed-out for response from LOGD
Apr 13 13:06:29 SLES-64BIT-SLOT1 opensafd[3378]: ER Could Not RESPAWN LOGD
Apr 13 13:06:29 SLES-64BIT-SLOT1 opensafd[3378]: ER
Apr 13 13:06:29 SLES-64BIT-SLOT1 opensafd[3378]: ER Trying To RESPAWN /usr/lib64/opensaf/clc-cli/osaf-logd attempt #2
Apr 13 13:06:29 SLES-64BIT-SLOT1 opensafd[3378]: ER Sending SIGKILL to LOGD, pid=3495
Apr 13 13:06:44 SLES-64BIT-SLOT1 osaflogd[3546]: Started
Apr 13 13:06:44 SLES-64BIT-SLOT1 osaflogd[3546]: WA read_logsv_configuration(). All attributes could not be read
Apr 13 13:06:44 SLES-64BIT-SLOT1 osaflogd[3546]: NO Log config system: high 0 low 0, application: high 0 low 0
Apr 13 13:06:44 SLES-64BIT-SLOT1 osaflogd[3546]: NO log root directory is: /var/log/opensaf/saflog
Apr 13 13:06:44 SLES-64BIT-SLOT1 osaflogd[3546]: NO LOG data group is:
Apr 13 13:06:44 SLES-64BIT-SLOT1 osaflogd[3546]: NO LGS_MBCSV_VERSION = 4
Apr 13 13:06:44 SLES-64BIT-SLOT1 osaflogd[3546]: saImmOmSearchInitialize FAILED, rc = 13
Apr 13 13:07:24 SLES-64BIT-SLOT1 opensafd[3378]: ER Timed-out for response from LOGD
Apr 13 13:07:24 SLES-64BIT-SLOT1 opensafd[3378]: ER Could Not RESPAWN LOGD
Apr 13 13:07:24 SLES-64BIT-SLOT1 opensafd[3378]: ER
Apr 13 13:07:24 SLES-64BIT-SLOT1 opensafd[3378]: ER FAILED TO RESPAWN
Apr 13 13:07:24 SLES-64BIT-SLOT1 osaffmd[3419]: exiting for shutdown
Apr 13 13:07:24 SLES-64BIT-SLOT1 osafimmd[3429]: exiting for shutdown
Apr 13 13:07:24 SLES-64BIT-SLOT1 osafimmnd[3439]: NO No IMMD service => cluster restart, exiting
Apr 13 13:07:24 SLES-64BIT-SLOT1 osafimmpbed: WA PBE lost contact with parent IMMND - Exiting
Apr 13 13:07:24 SLES-64BIT-SLOT1 osafrded[3410]: exiting for shutdown
Apr 13 13:07:24 SLES-64BIT-SLOT1 kernel: [ 1630.782513] TIPC: Disabling bearer <eth:eth0>
Apr 13 13:07:24 SLES-64BIT-SLOT1 kernel: [ 1630.782518] TIPC: Lost link <1.1.1:eth0-1.1.4:eth0> on network plane A
Apr 13 13:07:24 SLES-64BIT-SLOT1 kernel: [ 1630.782521] TIPC: Lost contact with <1.1.4>
Apr 13 13:07:24 SLES-64BIT-SLOT1 kernel: [ 1630.782526] TIPC: Lost link <1.1.1:eth0-1.1.3:eth0> on network plane A
Apr 13 13:07:24 SLES-64BIT-SLOT1 kernel: [ 1630.782527] TIPC: Lost contact with <1.1.3>
Apr 13 13:07:24 SLES-64BIT-SLOT1 kernel: [ 1630.782534] TIPC: Left network mode
Apr 13 13:07:24 SLES-64BIT-SLOT1 kernel: [ 1630.782551] NET: Unregistered protocol family 30
Apr 13 13:07:24 SLES-64BIT-SLOT1 kernel: [ 1630.782554] TIPC: Deactivated
Apr 13 13:07:24 SLES-64BIT-SLOT1 opensafd: Starting OpenSAF failed</eth:eth0>

syslog, imm and logd traces are attached.

1 Attachments
slot1.tar.bz2

Discussion

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-04-13

    Question: is this an old test that worked on 5.0 ?

     

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-04-13
     

  • Sirisha Alla

    Sirisha Alla - 2015-04-13

    This is not an old test. Stale object because of another test bumped into this scenario.

     

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-04-13
    • status: unassigned --> accepted
    • assigned_to: Anders Bjornerstedt
     

  • Neelakanta Reddy

    Neelakanta Reddy - 2015-04-13

    Before starting cluster services export
    export SA_ENABLE_EXTENDED_NAMES=1
    Then opensaf will start sucessfully

    May be logsv missed exporting the variable.

     

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-04-13
    • status: accepted --> assigned
    • assigned_to: Anders Bjornerstedt --> Neelakanta Reddy
     

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-04-13

    Reassigning to Neel

     

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-04-20
    • summary: opensaf does not start when long dn object is present in imm.db and cluster is reset --> IMM: opensaf does not start when long dn object is present in imm.db and cluster is reset
     

  • Neelakanta Reddy

    Neelakanta Reddy - 2015-04-22
    • summary: IMM: opensaf does not start when long dn object is present in imm.db and cluster is reset --> osaf: opensaf does not start when long dn object is present in imm.db and cluster is reset
    • Component: imm --> osaf
     

  • Neelakanta Reddy

    Neelakanta Reddy - 2015-04-22

    From the readme:

    To enable the extended SaNameT format, the application source code has to be compiled with the SA_EXTENDED_NAME_SOURCE preprocessor macro defined, and the environment variable SA_ENABLE_EXTENDED_NAMES must be set to the value 1 before the first call to any SAF API function.

    Problem reported in this case is :

    Log service is searching from root. If an application is searching from root and longDns are present, then there is a chance that longdn variable may be searched and ERR_TOO_LONG will be returned.

    The workaround fro the reported problem is to export SA_ENABLE_EXTENDED_NAMES=1

    Logd traces:

    Apr 22 11:21:43.608245 osaflogd [4217:imma_om_api.c:5557] << saImmOmAccessorInitialize
    Apr 22 11:21:43.608252 osaflogd [4217:imma_om_api.c:6545] >> search_init_common
    Apr 22 11:21:43.611918 osaflogd [4217:imma_om_api.c:6913] T3 Search initialize failed:13
    Apr 22 11:21:43.611959 osaflogd [4217:imma_om_api.c:6948] << search_init_common

    Immnd traces:

    Apr 22 11:21:43.608296 osafimmnd [4199:immsv_evt.c:5414] T8 Received: IMMND_EVT_A2ND_SEARCHINIT (16) from 2010f
    Apr 22 11:21:43.608306 osafimmnd [4199:immnd_evt.c:0881] T2 SEARCH INIT for root:(null)
    Apr 22 11:21:43.608313 osafimmnd [4199:ImmModel.cc:1386] TR Allocating iterator searchOp:0x1cd9310
    Apr 22 11:21:43.608331 osafimmnd [4199:ImmModel.cc:10363] >> searchInitialize
    Apr 22 11:21:43.608496 osafimmnd [4199:ImmModel.cc:10621] TR SEARCH_NON_EXTENDED_NAMES filter: Object name is too long: tEstAttr8=zzzzzzzzzzzzzzzzyyyyyyyyyyyyyyyyyyyxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxattrName_testAdminOwnerClear_SubLevelScope_1011
    Apr 22 11:21:43.608505 osafimmnd [4199:ImmModel.cc:10861] << searchInitialize

     

  • Neelakanta Reddy

    Neelakanta Reddy - 2015-04-22
    • status: assigned --> unassigned
    • assigned_to: Neelakanta Reddy --> Mathi Naickan
     

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-08-13
     

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-08-13

    I think this ticket should be closed as invalid.
    The mechanism works as documented.

    The only relevant defect related to this incident is #1430 and it has been fixed.
    https://sourceforge.net/p/opensaf/tickets/1430/
    That is an application that does a search with a scope that does not include any
    long DN object should not get hit by any longDn error.

     

  • Anders Widell

    Anders Widell - 2015-08-14

    It seems that the LOG service performs an unnecessarily broad search (starting at the root of the IMM database). Instead, it could search the tree below safApp=safLogService

     

  • elunlen

    elunlen - 2015-08-14

    The log service should search from root "safApp=safLogService". See ticket [#1452]
    Note: It is still not allowed to use a long DN for naming a log stream configuration object. Long DN is not supported by log service

     

    Related

    Tickets: #1452

  • Anders Bjornerstedt

    Anders Bjornerstedt - 2015-09-24
    • status: unassigned --> duplicate
    • Component: osaf --> log
    • Version: 4.6 FC --> 4.6
    • Milestone: 4.5.1 --> never
     

Log in to post a comment.

MongoDB Logo MongoDB