WritingPlugins

Tony Asleson

Overview

The libStorageMgmt library uses a plug-in architecture to accommodate the differences in storage arrays. Storage arrays can provide block and/or network file system storage. Each storage array product will require a vendor unique plug-in if the vendor does not support SMI-S[SMI-S:1] or one of the other plug-ins that are currently available.

Architecture

Plug-ins in libStorageMgmt are different than many other plug-in designs. Traditionally a plug-in is a shared object that is loaded dynamically at run-time into the same address space as the plug-in process, this is not the case for libStorageMgmt. To ensure a software license agnostic design and for fault isolation, the plug-ins execute in their own address space as stand alone executables with inter-process communication (IPC) between the client and plug-in. The details of the IPC mechanism are not required as the library abstracts the IPC in the plug-in application programming interface (API).

Plug-in operation

Sequence of events

When a client application or the libStorageMgmt command line (lsmcli) utilizes the library the following sequence occurs:

  1. The library uses the uniform resource identifier (URI) and parses out which plug-in was specified. For example LSMCLI_URI=sim:// refers to the simulator plug-in.

  2. The library uses the plug-in name "sim" and looks for the Unix domain socket in the socket directory. The default directory is /var/run/lsm/ipc. This can be changed for the client library at run-time by specifying the LSM_UDS_PATH environment variable.

  3. Once found the client library opens the Unix domain socket, this causes the lsmd daemon to accept the connection from the client. Based on which file that was opened tells the daemon which plug-in is requested to be used. The daemon then forks and execs the plug-in passing the socket descriptor on the command line to plug-in.

  4. At this point the client process has a direct connection to the plug-in.

  5. The lsmd is no longer in the path and goes back to sleep waiting for another process to open a socket.

Benefits of this design

  • If daemon dies or is killed, existing client plug-in sessions remain.
  • If a plug-in crashes the client process will remain operational.
  • The daemon needs to know nothing of the IPC protocol and is very simple.
  • The plug-in can be closed source if so required by the vendor.

Writing a plug-in

Writing a plug-in involves implementing the required plug-in interface. The library has a plug-in API for both the C and Python programming languages. Any language that supports sockets and text can theoretically be utilized to write a plug-in, but the library provides the abstraction that hides this complexity.

General plug-in guidelines

These guidelines pertain to plug-in design regardless of programming language used.

Threading/multi-process

The library does not provide locking and it doesn't keep any global state. Thus it is valid for a client to have a separate plug-in instance in use for each thread or process. Plug-ins should anticipate that multiple instances of themselves can and possibly will be running at the same time to different arrays. Running concurrent plug-ins to the same array, unless explicitly supported by the array is not a typical use case. As the library provides a mechanism for multiple long running operations, multiple plug-in instances for the same array are not needed.

Plug-ins execute with non-root privileges

To reduce potential of local exploits, plug-ins have reduced privileges. Please take this
into account when writing and designing your plug-in.

Plug-ins should not store local persistent state

As the array state could change independently from libStorageMgmt it is important that the plug-in not store state on the system the plug-in is executing on.

Plug-in lifetime

The client API provides for a handle that is opened and closed for each plug-in instance. During this time the plug-in is free to cache whatever data they deem necessary to provide correct operation. Note: The library is planned to provide some form of generic caching in the future to avoid repeated data retrieval from the array. When using the lsmcli tool, the lifetime is for only one command.

Logging

Plug-ins should log errors to syslog. Helper function exist to facilitate this in the library.

Errors

The library uses well defined error codes to remain language agnostic. C does not support exceptions. Additional error data can be retrieved when they occur to provide textual error messages and optionally debug data from the plug-in or the array itself. It is the library callers responsibility to retrieve this additional information after an error occurs and before issuing another command. If additional error data exists and other functions are called the additional error information will be lost.

For computer languages that do support exceptions, a custom exception class which contains the error code and additional information is provided.

As a plug-in writer, please try to match the correct error to the situation at hand. If this is not possible return a generic error code with a meaningful text and/or additional information.

Location and naming

  • Plug-ins are located in one directory, currently that is /usr/bin (This is configurable via the lsmd command line)

  • Plug-ins require a name format of *_lsmplugin (When the daemon starts is iterates in the directory enumerating them)

Job control

The methods to set and get the time-out are used to specify how long the plug-in waits for a response from the array. If an operation cannot safely complete within the time-out the call should return a job id so that the client can check on the status of the operation at a later point in time. Job ids are free form strings and are plug-in defined. The plug-in implementation needs to determine everything about the asynchronous operation from this string between invocations of the plug-in.

Writing a plug-in (step by step)

  • Pick a unique name so that your main executable has the form name_lsmplugin
  • The base functions/methods are required for all plug-ins
    • get/set timeout
    • startup/shutdown
    • job status
    • job free
    • capabilities
    • plug-in information
    • pools
    • systems
For python
  • Implement the interface that supports the level of functionality you wish to provide (see iplugin.py). Most plug-ins will either inherit from IStorageAreaNetwork or INfs or both if your plug-in supports block and network file systems.
  • Call plug-in runner passing the name of the class and the command line arguments to it for processing and executing the run method

    1
    2
    3
    4
    5
    6
    7
    8
    #!/usr/bin/env python
    import sys
    
    from lsm.pluginrunner import PluginRunner
    from lsm.simulator import StorageSimulator
    
    if __name__ == '__main__':
        PluginRunner(StorageSimulator, sys.argv).run()
    
  • Note: During development you can call your plug-in directly on the command line for easier debug

For C plug-ins
  • Include the required header file "#include <libstoragemgmt/libstoragemgmt_plug_interface.h"
  • Implement which callback functions you plan on supporting plus the required ones
  • Pass the command line count and args to the library with load and unload functions

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    #include <libstoragemgmt/libstoragemgmt_plug_interface.h>
    #include <stdlib.h>
    #include <stdint.h>
    
    static char name[] = "Simple limited plug-in example";
    static char version [] = "0.01";
    
    struct plugin_data {
        uint32_t tmo;
        /* All your other variables as needed */
    };
    
    /* Create the functions you plan on implementing that
        match the callback signatures */
    static int tmoSet(lsm_plugin_ptr c, uint32_t timeout, lsm_flag flags )
    {
        int rc = LSM_ERR_OK;
        struct plugin_data *pd = (struct plugin_data*)lsm_private_data_get(c);
        /* Do something with state to set timeout */
        pd->tmo = timeout;
        return rc;
    }
    
    static int tmoGet(lsm_plugin_ptr c, uint32_t *timeout, lsm_flag flags )
    {
        int rc = LSM_ERR_OK;
        struct plugin_data *pd = (struct plugin_data*)lsm_private_data_get(c);
        /* Do something with state to get timeout */
        *timeout = pd->tmo;
        return rc;
    }
    
    /* Setup the function addresses in the appropriate
        required callback structure */
    static struct lsm_mgmt_ops_v1 mgmOps = {
        tmoSet,
        tmoGet,
        NULL,
        NULL,
        NULL,
        NULL,
        NULL
    };
    
    int load( lsm_plugin_ptr c, const char *uri, const char *password,
                            uint32_t timeout, lsm_flag flags )
    {
        /* Do plug-in specific init. and setup callback structures */
        struct plugin_data *data = (struct plugin_data *)
                                    malloc(sizeof(struct plugin_data));
    
        if (!data) {
            return LSM_ERR_NO_MEMORY;
        }
    
        /* Call back into the framework */
        int rc = lsm_register_plugin_v1( c, data, &mgmOps, NULL, NULL, NULL);
        return rc;
    }
    
    int unload( lsm_plugin_ptr c, lsm_flag flags)
    {
        /* Get a handle to your private data and do clean-up */
        struct plugin_data *pd = (struct plugin_data*)lsm_private_data_get(c);
        free(pd);
        return LSM_ERR_OK;
    }
    
    int main(int argc, char *argv[] )
    {
        return lsm_plugin_init_v1(argc, argv, load, unload, name, version);
    }
    

For more details please look at the source code for the included plug-ins

  • The python simulated plug-in sim

  • The C simulated plug-in simc


Related

Wiki: Architecture
Wiki: DevelopLibstorageMgmt