#520 Mds: Tune MDS logging to minimal & informative

4.7.FC
fixed
None
enhancement
mds
-
4.3.1
minor
2015-09-22
2013-07-25
No

Minimize the MDS logging to only in case of required so that it can not reach 1 Mb of log rotation range/size sooner .

amfnd core dump is produced when amfnd main thread (10720) is waiting for a pthread mutex, gl_mds_library_mutex, which is held by the mds thread (10723).
The amf watchdog detects this (no healthchecks received) and sends an abort signal to the amfnd. Holding a mutex during file operations in MDS is not correct and should be corrected. (HR50165)

#0 0x00007f7830d70294 in __lll_lock_wait () from /lib64/libpthread.so.0

(gdb) p gl_mds_library_mutex

$1 = {data = {lock = 2, count = 1, owner = 10723, nusers = 1, kind = 1, spins = 0, list = {prev = 0x0, next = 0x0}},

size = " ã ) ", ' ' <repeats 22="" times="">, align = 4294967298}

(gdb) info thr

Id Target Id Frame

4 Thread 0x7f7832263b00 (LWP 10723) 0x00007f783083e20d in write () from /lib64/libc.so.6

3 Thread 0x7f7832283b00 (LWP 10722) 0x00007f7830844f53 in select () from /lib64/libc.so.6

2 Thread 0x7f7832243b00 (LWP 10724) 0x00007f7830d7076d in read () from /lib64/libpthread.so.0

  • 1 Thread 0x7f7832286700 (LWP 10720) 0x00007f7830d70294 in __lll_lock_wait () from /lib64/libpthread.so.0

Related

Tickets: #520
Tickets: #793

Discussion

1 2 > >> (Page 1 of 2)
  • A V Mahesh (AVM)

    • summary: Mds: MDS thread is holding a mutex during write(s) causing amfnd to get aborted. --> Mds: Tune MDS logging to minimal & informative
    • Description has changed:

    Diff:

    --- old
    +++ new
    @@ -1,3 +1,5 @@
    +Minimize the  MDS logging  to  only in case of  required so that it can not reach  1 Mb of log rotation range/size sooner .
    +
    
     amfnd core dump is produced when amfnd main thread (10720) is waiting for a pthread mutex, gl_mds_library_mutex, which is held by the mds thread (10723).
     The amf watchdog detects this (no healthchecks received) and sends an abort signal to the amfnd. Holding a mutex during file operations in MDS is not correct and should be corrected. (HR50165)
    
    • Priority: major --> minor
     
  • A V Mahesh (AVM)

     
  • A V Mahesh (AVM)

    On 1/21/2014 5:30 PM, Hans Feldt wrote:

    To improve readability of the mds.log. I have an idea to make it look like syslog.
    For example instead of:
    Dec 19 16:44:16.938714 <4288733188> NOTIFY |BEGIN MDS LOGGING| PID=3181|ARCH=0|64bit=1

    we would have e.g:
    Dec 19 16:44:16.938714 osafimmd[3181] NOTIFY |BEGIN MDS LOGGING| ARCH=0|64bit=1

    What do you think about that?
    /Hans

     
  • A V Mahesh (AVM)

    • Type: defect --> enhancement
     
  • A V Mahesh (AVM)

    • status: unassigned --> assigned
    • assigned_to: A V Mahesh (AVM)
    • Milestone: future --> 4.5.FC
     
  • A V Mahesh (AVM)

    • status: assigned --> accepted
     
  • Mathi Naickan

    Mathi Naickan - 2014-07-14

    As a first step, We can aim for
    1) printing service names (as strings) instead of service ids
    2) printing process(binary) names along with pid
    3) Instead of adest/vdest address, print the service name(as strings) or the process name

     
  • Mathi Naickan

    Mathi Naickan - 2014-08-14
    • status: accepted --> review
     
  • Anders Widell

    Anders Widell - 2014-08-15
    • Milestone: 4.5.FC --> 4.5.0
     
  • A V Mahesh (AVM)

    • status: review --> fixed
     
  • A V Mahesh (AVM)

    changeset: 5787:1ae3fab58c88
    branch: opensaf-4.5.x
    parent: 5785:ea9d40a93b09
    user: A V Mahesh mahesh.valla@oracle.com
    date: Mon Sep 15 14:42:43 2014 +0530
    summary: mds: change mds logging prefix readability [#520]

    changeset: 5788:47bb76fa989b
    branch: opensaf-4.5.x
    user: A V Mahesh mahesh.valla@oracle.com
    date: Mon Sep 15 14:45:07 2014 +0530
    summary: mds: use ncsmds_svc_names mapping array of ncsmds_svc_id for logging [#520]

    changeset: 5789:be5d69ad7c82
    branch: opensaf-4.5.x
    user: A V Mahesh mahesh.valla@oracle.com
    date: Mon Sep 15 14:46:34 2014 +0530
    summary: mds: update adest details for log readability [#520]

    changeset: 5790:43f758230e5f
    branch: opensaf-4.5.x
    user: A V Mahesh mahesh.valla@oracle.com
    date: Mon Sep 15 14:47:33 2014 +0530
    summary: mds: update logs according for readability [#520]

    changeset: 5791:e970f1e8ff6b
    parent: 5786:dcae8bcf532f
    user: A V Mahesh mahesh.valla@oracle.com
    date: Mon Sep 15 14:51:41 2014 +0530
    summary: mds: change mds logging prefix readability [#520]

    changeset: 5792:f8dbeb5ee3c0
    user: A V Mahesh mahesh.valla@oracle.com
    date: Mon Sep 15 14:52:40 2014 +0530
    summary: mds: use ncsmds_svc_names mapping array of ncsmds_svc_id for logging [#520]

    changeset: 5793:a9233c37c652
    user: A V Mahesh mahesh.valla@oracle.com
    date: Mon Sep 15 14:53:20 2014 +0530
    summary: mds: update adest details for log readability [#520]

    changeset: 5794:c01bcaff9ca7
    tag: tip
    user: A V Mahesh mahesh.valla@oracle.com
    date: Mon Sep 15 14:54:02 2014 +0530
    summary: mds: update logs according for readability [#520]

     

    Related

    Tickets: #520

  • A V Mahesh (AVM)

    • status: fixed --> assigned
     
  • A V Mahesh (AVM)

    Further improvements will done in nest release

     
  • A V Mahesh (AVM)

    • Milestone: 4.5.0 --> 4.6.FC
     
  • A V Mahesh (AVM)

    If service name is external application if applications is not registered in SVC_ID list ( 256, The range below is for OpenSAF external application useage)
    the mds log is printing empty string. this need to be improved .

    Dec 31 2:50:03.903356 osafpmnd[5400] NOTIFY |MDTM: svc up event for svc_id = , subscri. by svc_id = pwe_id=1 Adest = <rem_node<span>[2]:dest_tipc_id_ref[471531569]>
    Dec 31 2:50:03.903456 osafpmnd[5400] NOTIFY |MDTM: svc up event for svc_id = , subscri. by svc_id = pwe_id=1 Adest = <rem_node<span>[2]:dest_tipc_id_ref[471531569]>
    Dec 31 2:50:03.903473 osafpmnd[5400] NOTIFY |MDTM: svc up event for svc_id = , subscri. by svc_id = pwe_id=1 Adest = <nodeid[1]:osafpmnd[5400]

    Improvemnt : if applications is not registered in SVC_ID list , it can shown as `external_svc[PID]

     
  • A V Mahesh (AVM)

     
  • A V Mahesh (AVM)

    • status: assigned --> review
     
  • A V Mahesh (AVM)

     
  • A V Mahesh (AVM)

    changeset: 6328:df70aadfb2e6
    branch: opensaf-4.5.x
    parent: 6325:1c92c3ecf486
    user: A V Mahesh mahesh.valla@oracle.com
    date: Thu Mar 12 13:23:16 2015 +0530
    summary: mds: part of mds log improvements [#520]

    changeset: 6327:2a6bc699b5eb
    user: A V Mahesh mahesh.valla@oracle.com
    date: Thu Mar 12 13:21:49 2015 +0530
    summary: mds: part of mds log improvements [#520]

     

    Related

    Tickets: #520

  • A V Mahesh (AVM)

    Next we need to improve the log strings which can be understood by end user/debugger ( make them some meaning full to end user )

     
  • A V Mahesh (AVM)

    • status: review --> assigned
     
  • Mathi Naickan

    Mathi Naickan - 2015-03-13
    • Milestone: 4.6.FC --> 5.0
     
  • A V Mahesh (AVM)

    • Milestone: 5.0 --> 4.7-Tentative
     
  • Praveen

    Praveen - 2015-09-17
    • status: assigned --> review
     
  • A V Mahesh (AVM)

    changeset: 6846:e8c3ec079da6
    tag: tip
    user: A V Mahesh mahesh.valla@oracle.com
    date: Fri Sep 18 13:33:58 2015 +0530
    summary: mds: improved mds logging [#520]

    As if NO Function changes and the current changes will reflect only in DEBUG mode
    (export MDS_LOG_LEVEL=5) , and #520 Ticket is continuous process for each release,
    I pushed the changes , any review comment will be incorporated subsequently.

    After 4.7 Tag this will be reopened for next release.

     

    Related

    Tickets: #520

1 2 > >> (Page 1 of 2)

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks