Perl Plugin support

2011-03-25
2013-04-16
  • Mark McCorkle
    Mark McCorkle
    2011-03-25

    I read somewhere that you might be exposing plugin development to us perl hackers.  Is that still on the books?

    While I'm waiting, I've been using your DebugTreeDump VcItem to point at data files I'm creating to render useful information such as my entire Nagios (http://www.nagios.org/) system status:
    http://www.devteam.org/users/mccorkle/images/zoom1.png
    http://www.devteam.org/users/mccorkle/images/zoom2.png
    http://www.devteam.org/users/mccorkle/images/zoom3.png

    I actually have a LOT more of the raw data (including historical data) being shown in my current version (over in the text area), but I didn't include that in my screenshots as it might be "sensitive" to my company.  I've also started another plugin that will draw the nmap output (network detection) overlaying with my nagios helping a sysadmin see stuff on his network that he isn't aware of or isn't monitoring yet.  Lots of good can be done in this arena. 

    I know this is a nasty hack on top of the DebugTree plugin, but I'd love to start with a perl version of the DebugTree and build it up to rendering complex datasets like this (with images, clickable actions, view modifiers, etc).  Alternatively, I could finish my proof of concept and make a wish list of the 5 or 10 things that DebugTree needs to be a drop in replacement and offer a bounty for someone who is a c++ developer to add those for me.  Let me know which makes more sense for you. 

    I've shopped this idea (and screenshots) around to my other system admin friends and they love it so far.  I've also got a designer that could help me dress it up.  Would you be interested in a plugin inside Eaglemode that offers that kind of functionality being made available? 

     
  • Oliver Hamann
    Oliver Hamann
    2011-03-27

    I find the application you are aiming for would be a very useful and exciting example for a zoomable user interface. I still have the plan to develop a scripting API for perl and other languages. But I haven't much time for the project, and I thought to do the Android port and some other things first. Though, a bounty could change everything *g*. Anyway, what do you think are the 5 or 10 things you would require from DebugTree (emTreeDump)?

     
  • Mark McCorkle
    Mark McCorkle
    2011-03-28

    Three immediate features that would allow me to get to where I'm dreaming right now (without the perl API ready):
    * ability to place a tag which loads either local or remote svg images
    * reload of xml data file without moving view (an F5/Reload Files goes back up to
    * ability to have a children element point at another element in the tree somewhere - so that if I have multiple views (host view, service view, status view), I can just point into the tree for the data, not have to re-supply it

    I had others in mind, but when I went to articulate them, they didn't make sense (specifically I dropped clickable target links and html/css formatting of the text area for layout). 

    --

    An example of the emTreeDump XML for "ability to place a tag which loads either local or remote svg images":

    title = "server2"
    text = "Current State: reachable \n Current Load: 0.30 \n Current Memory: 35% free"
    bgcolor = {80 100 120 255}
    fgcolor = {255 255 255 255}
    children = {
    {
    title="Current Memory Usage Graph"
    type="img"
    src="http://www.aserver.com/dynamicMemoryUsagePng.cgi?servername=server2&timestamp=1301334064"
    children={
      title="History Memory Usage Graph"
      type="img"
      src="http://www.aserver.com/dynamicMemoryUsagePng.cgi?servername=server2&timestamp=1301334064&range=historic"
    }
    }
    {
    title = "Current Memory Usage Details"
    text = "            total       used       free     shared    buffers     cached\nMem:       1026212     969444      56768          0        652     503784\n-/+ buffers/cache:     465008     561204\nSwap:      2040244      27224    2013020\n"
    bgcolor = {80 100 120 255}
    fgcolor = {255 255 255 255}
    }
    }
    

    An example for "ability to have a children element point at another element in the tree somewhere":

    title = "servers in state 0"
    text = "Total Servers in this state: 25"
    bgcolor = {80 100 120 255}
    fgcolor = {255 255 255 255}
    children = {
    {
    target="server2"
    }
    {
    target="server5"
    }
    }
    


    or perhaps just view a child element directly:

    title = "server's using > 50% of their ram"
    text = "Total Servers in this state: 2"
    bgcolor = {80 100 120 255}
    fgcolor = {255 255 255 255}
    children = {
    {
    target="server14.Current Memory Usage Graph"
    }
    {
    target="server15.Current Memory Usage Graph"
    }
    }
    

    The goal here would be to logically group the information that needs to be viewed together, but still provide the zoom down to the details. I'm envisioning children rendered similar to infinite "test panel" embedding with the 4 children in the upper right corner of each test panel. 


    In response to placing a bounty on the perl api (or these changes for emTreeDump), I'd can only really put a paltry $100 on either, but would it be better for me to sponsor you a GetSatisfaction.com account and volunteer my time to manage the community to collect feature requests and pledges from the community larger than myself?

     
  • Mark McCorkle
    Mark McCorkle
    2011-03-29

    I re-discovered mk Livestatus for nagios and now have a much easier / faster way of generating my xml tree dump, so I'll be adding an order of magnitude more content into the report from nagios into eagle mode (1,318 distinct points of data, each could possible be per host, and per service).  I will follow up with a screenshot of the current working state when I'm done.  Is there a better place for me to show you my work in progress, or should I keep the conversation in this forum thread? 

     
  • Oliver Hamann
    Oliver Hamann
    2011-03-29

    Well, I am thinking of extending the tree nodes by just one attribute: file. It would simply use the Eagle Mode file plugin interface to show the contents of any file in the same way as the file manager does.

    Thereby you could show local SVG images, or other image files type, or texts, or whatever.

    In order to load a file from an URL, a simple emTempConv plugin could be written, which temporarily "converts" a file that contains an URL into the file the URL points to, by calling wget or so. The disadvantage would be that there is a tunnel into which the user has to zoom before the file is shown (like with ZIP files for example).

    In addition, the file attribute would allow to split the tree into multiple tree files, and to show individual parts (tree files) at multiple places.

    Last but not least you could write an emTempConv plugin that calls your scripts for creating the tree files dynamically when the user zooms in.

    The result would not look very nice, but it is a quick & easy solution. It is much easier than writing the scripting API, or even extending the tree nodes the way you suggested.

    What do you say, would it be helpful to you? If so, I think I could include the file attribute in the next release, which will come early May. I could also provide you with an example of the mentioned emTempConv plugins.

    The only reward I would like to have is a success story about your project on the Eagle Mode homepage. I think this would be cool. Maybe you could provide me with a description and some screenshots or a video or just a link when you are done.

     
  • Oliver Hamann
    Oliver Hamann
    2011-03-29

    Is there a better place for me to show you my work in progress, or should I keep the conversation in this forum thread?

    You can post to the forums or to my email address, what ever you prefer.

     
  • Mark McCorkle
    Mark McCorkle
    2011-04-07

    The file attribute would be a huge benefit. With the tmpConvert plugins and wget requests, I could add all of the data too heavy to deliver to the client directly (such as the svgs and pngs) and render them server side (where the data lives) or server side real time if the data updates to often to schedule its delivery to the viewing client. Brilliant suggestion. I look forward to it in the next release.

    In my development, I have come up with another method for using eaglemode tools like what I've been working on. At my day job, I have both Linux openvz, BSD Jails, and Amazon EC2 nodes in my network here, and I would like to manage them in eaglemode. Since i can detect those hosts parent child relationships via tools and config files, i can display them in eaglemode using the same methods as the nagios data.

    I will then write commands to do all the normal management of the virtual machines at put those in my panel.  With that, we'd have network detection, monitoring, and some basic management tools in place. 

    Is there any way to (easily) display commands in the control panel when an element is selected in debugTree and pass the name or value of the element selected to that command (same as when I select a file in file viewer and click a command)?

     
  • Oliver Hamann
    Oliver Hamann
    2011-04-09

    Okay, I'll see if I can also add a command attribute. This should not be too difficult. I am currently busy in preparations for exhibiting Eagle Mode on the LinuxTag. My plan is to release version 0.81.0 one or two weeks before. If everything works well, I am sure there is time to include the file and command attributes. Otherwise I would deliver it as a patch after the exhibition.

     
  • Mark McCorkle
    Mark McCorkle
    2011-04-09

    That is excellent news.  I would love to have my part ready by then as well.  I have the mysql backended version working in a read only method now, but I am targetting a much heavier data set using mk_livestatus.  Even without the the file attribute, I can get the data set much more filled out and impressive if you'd like to include a quick set of screenshots.  Just lett m know when you'd need my stuff by.

     
  • Oliver Hamann
    Oliver Hamann
    2011-04-10

    I think it would be a very nice idea to pin a sheet of paper about your work on the wall of the exhibition booth. So it would be great if you could send two or three screenshots for that (within about the next two and a half weeks).

     
  • Mark McCorkle
    Mark McCorkle
    2011-04-11

    As I was figuring out a way to integrate data from another web application, I happened upon wkhtmltoimage which effectively takes a url and converts it into a PNG / JPG image.  With your file attribute and the ability to render images, what I'll be doing is scheduling up my local scripts to pull down the remote web application's html, check for changes, and create new png images of the web application if they have changed since the last time it checked.  This way I'll have all of the data for a server renderable in one place (inside eaglemode). 

     
  • Oliver Hamann
    Oliver Hamann
    2011-04-24

    Today I am releasing version 0.81.0. As I promised, the emTreeDump file format got two new attributes for you: commands and files. Here comes an example, make sure to name it example.emTreeDump because it shows itself recursively:

    #%rec:emTreeDump%#
    frame = rectangle
    bgcolor = {200 255 200 255}
    fgcolor = {0 0 0 255}
    title = "Tree Dump Example"
    text = "This node shows a\nfile, a directory,\nand a child node.\n\nIn addition, there\nare two commands in\nthe control panel"
    commands = {
        {
            caption = "Hello"
            args = {
                "/bin/sh"
                "-c"
                "exec $EM_DIR/bin/emShowStdDlg message Hello 'Hello World'"
                # Hint: /bin/sh is needed to resolve $EM_DIR here.
            }
        }
        {
            caption = "List Dir"
            args = {
                "xterm"
                "-e"
                "echo \"contents of current directory:\"; ls -l; read"
            }
        }
    }
    files = {
        "example.emTreeDump"
        "/"
    }
    children = {
        {
            frame = hexagon
            bgcolor = {200 200 255 255}
            fgcolor = {0 0 0 255}
            title = "Child node"
            text = ""
            commands = {}
            files = {}
            children = {}
        }
    }
    
     
  • Mark McCorkle
    Mark McCorkle
    2011-05-20

    Thanks for 0.81 and a few long sessions hacking things together, I'm proud to present the following results.  Since I last checked in, I've switched my backend from using the ndoutils mySQL data (which really couldn't deal well with more than 30-40 hosts with a 60 second refresh)  over to using mk_livestatus, where I can get all the data and render out the treeDump file in 0.443s.  I've also switched over from printing out all the treeDump content directly from my perl code to using perl's Text::Template engine to render the treeDump in a much cleaner way.  I've still got way too many escaped and double escaped quotes and double quotes in my perl code, but I'll continue to refactor there until that is cleaned up as well. 

    The way that this whole system works in my environment is that I have a piece of perl code that cron runs every 60 seconds.  This code merges the mk_livestatus live data from nagios and a template designed to work as a treeDump, and drops that file to the local disk in /tmp/nagios.emTreeDump.  I then have another cron job that every 60 seconds copies that /tmp/nagios.emTreeDump over to my local laptop and it then calls eaglemode -reload to update my view.   All of the server descriptive data (see eagleNagios4.png and eagleNagios5.png) is being generated once a night on my local laptop also via a quick cron job.  The final content that is rendered is done from my local file system.  Below are quick snapshots of my work in progress - but all of the hard work I expected is done at this point, and I just need to smooth some of the edges before I push all this code up here for everyone to share. 

    My top level view with the critical, warning, and unknown status of services on my network plainly obvious:
    http://www.devteam.org/users/mccorkle/images/eagleNagios1.png

    In my second view, you'll see the "text" area of the main view being used to let me know how fresh this data is - as well as a little more detail on what each of those little boxes contain:
    http://www.devteam.org/users/mccorkle/images/eagleNagios2.png

    In the third view, I've zoomed down enough that I can see all my hosts and their monitored services.  Normally, I wouldn't have this much red on that map (as only critical services are red) but I've still got a little bug in the code that I'm ironing out:
    http://www.devteam.org/users/mccorkle/images/eagleNagios3.png

    In this view, I've zoomed down to a host that I only monitor the ping of that host - and you can start to see where I use the "files" attribute to show off a png that gives details about this host:  
    http://www.devteam.org/users/mccorkle/images/eagleNagios4.png

    I've zoomed a little more down on the content of the files attribute.  This file is drawn by another open source tool called wkhtmltopdf which takes a snapshot of my live server and network documentation tool (I use the open source racktables.org) and then I use imageMagick to convert the pdf into one tall png which I point to with the new "files" attribute:
    http://www.devteam.org/users/mccorkle/images/eagleNagios5.png

    Here I've zoomed down to the details of a service that is being monitored (the ping service in this case) and you can see the raw details of the most recent monitoring check.
    http://www.devteam.org/users/mccorkle/images/eagleNagios6.png

    Once I see an alert of any sort on a host, I can then click on the host name and it will active the two "commands" that I've defined thanks to that new attribute.  The first command simply opens up an xterm and ssh's to that host, seen here:
    http://www.devteam.org/users/mccorkle/images/eagleNagios7.png

    The second command is much more useful, in that it does the top 4 or 5 things that I also do to diagnose a server's health when I connect to it and displays those in a terminal, waiting for any keystroke before closing the terminal:
    http://www.devteam.org/users/mccorkle/images/eagleNagios8.png

    I've still got a lot of things to iron out (namely that I have 3 different Linux / BSD variants which can't have the same commands run on them and the rendering of the html to png isn't perfect yet) but it gets the idea out there pretty well and will be useful in my day to day doing of my job. 

    I've also been doing a lot of work in Amazon Web Services lately, and I've been dreaming up a nice interface for AWS being built in Eaglemode using some of the same techniques that I've used here.  I will stay focused on the eaglemode / nagios work for now, but that is likely where I'll head next once I get this work to its logical conclusion. 

     
  • Mark McCorkle
    Mark McCorkle
    2011-06-22

    Just as a follow up on this hack, I've been using it daily since I put it together, and it has lots of benefits over the traditional web interface such as:
    1) traditionally, I had to see an error, click on it to see the context of other services on the box, or click on the overview to see the context of this box within the entire cluster - with EagleMode, I can see all of this at once
    2) also in the traditional web interface, I had to click on an event to get more details on the error.  With my little hack, not only do I get all the raw details that Nagios knows (which I format into the text view with whitespace enough that it is easy to read), I can use the commands ability to click a single button and do a full server diagnostic (via ssh and authorized_keys it is brilliantly smooth) and see the immediate situation of a box having a problem with very little effort. 

    I've got the following features left on my todo list:
    * add all ports and service details view the a tmpConvert view of the host
    * add graph images from pnp4Nagios also via the tmpConvert method of fetching urls
    * group hosts into their host and service groups in a distinct view (to make finding hosts easier)
    * importing disconnected html data into the view of hosts (not just from nagios, but also from my ticketing system where I track alerts, and my system documentation database web interface where I track the config of a server).  I'm currently using a tool that takes a website and converts it into images, then I stitch the images together and include them in the view.  Lots more work there. 

    I've been showing this live system off to some trusted friends and have gotten a few requests on instructions to deploy this elsewhere, but all of my work is still just a hack and is split into work on the local machine via cron jobs and work on the remote servers that provide so much of the raw data that is needed for all of this to be viewable.