Consolidating log info into reports

Matt
2011-03-21
2013-05-28
1 2 > >> (Page 1 of 2)
  • Matt

    Matt - 2011-03-21

    I really like the level of detail you can get form the logs and pics from pykeylogger, but, as with any raw data, it would be much more useful and convenient in an organized, easy-to-browse format. It's not that I plan on using this software so much that I can't get when I need from it on occasion, but I feel like it would be a fun and relatively easy task to integrate some data organization and presentation modules into pykeylogger.

    I'm considering making modifications to do things like include a simple HTML file in a zipped log package that takes the log output and displays it in a chronological, easy-to-read fashion, such as, having a scrollable text box that displays human-readable translations of keystrokes, and listing mouseclick-pics in a table.

    More importantly, I'm trying to think of a way to do it all so that where ever the log output is directed (to an FTP directory for example), one could simply browse to that directory (assuming it's web-hosted) and navigate through the formatted log out put without having to man-handle the files themselves, or even unzip the archives (should archives be used).

    Any thoughts, recommendations on methods, or ideas for further features in a system like this?

    I'll have to familiarize myself with Python, as I've never previously done anything with it but hack existing code. Hopefully my Java/Perl/PHP experience will aid this process.

    P.S. I see there is something similar in the PYKL planned features list ("Create log viewer"), but I'm thinking of something a bit more web-friendly, without the need for another software piece.

     
  • Swift

    Swift - 2011-03-21

    How exactly do you want the logs to be formatted? You mentioned "human-readable translations of keystrokes". But that doesn't make much sense to me. What exactly isn't readable with the way logs are currently being recorded? If you could write up a simple dummy html file for me, representing how you want your logs to display, then it would be easier to understand what you're trying to accomplish. A simple template is all we need. :)

    Also, your "more important" paragraph is relatively simple to fix. All you would need to do is after having the log output to html, run a batch script to merge all the logs (all the .html files) into one index.html . And this is the file that would upload to your FTP directory. Therefore, when you navigate to the directory through a browser, it would open the logs with your nice tables and "human-readable translations of keystrokes".

     
  • Matt

    Matt - 2011-03-21

    I took a closer look at the log files and they weren't as difficult as I thought they were. Previously I had only logged a bunch of  presses and such, so I didn't get to see much of the behavior and didn't really realize how well PYKL separated entries and such. I was going to suggest having it intelligently organize plain text, while separating out special keys, but I see it already does that, but I still feel it can be optimized for readability. On top of that, many keys show up as e.g. , which is a lot of fluff for the eye to overcome, Also, if a key is held down, you get multiple instances, and if you do multiple things within the same application you will have multiple entries all with much of the same info for the recorded executable/PID and such. This is great from a logging standpoint, and I wouldn't change the log itself at all, but it makes quite sluggish to actually troll through the log to find what you're looking for. What I'm talking about  is not a change to the logging, but more of an intelligent front-end (like I say, similar to the log-reader on the 'planned features list'). Except that we're not talking about a separate program to interpret logs, but a module within the exiting PYKL that outputs an additional 'log view' if you will and sends it on it's way along with the raw log. Of course I'm sure you understand all of that. I often catch myself being overly verbose.

    My main ideas to alleviate the strain of raw log-trolling are:
    1) translate output like "" into something more immediately recognizable (e.g. "")
    2) allow for grouping of similar entries (e.g. entires from the same executable), these could even be collapsible
    3) alternatively simply show all log entries in a sortable table, like any other database of events (think MS Event Viewer)
    4) upon thinking over #1 again, you could even go a step further and display small graphics to represent non-standard keys. For efficiency's sake you'd probably want to store these somewhere in the web-accessible final destination of the logs. However, for flexibility, the location of these images could be dynamically changeable by submitting a path to the html page (via JS or PHP or some such).

    Concerning my "more importantly" paragraph, ideally you wouldn't even need to upload a separate index or batch file to the FTP/Web location. The idea is to have as expandable and maintenance-free an index of log data as possible. You could simply have each page that arrives from the keylogged machine do a quick scan for other log files in the same directory and provide links to them at the top/bottom of each page for example. This could be made more efficient (for instances with LOTS of logs, by only having the script scan and create links for pages with nearby date/time ranges, effectively creating a dynamic pager.

    A big point I'm interested in exploring is, to minimize on potential data corruption/overhead/transfer bulk, whether all of the log data _including_ the html file (of course at this point I've gone far beyond HTML into CGI, but we'll say HTML for simplicity) could be transferred in a single archived format. One archive representing one report. It wouldn't be hard to have any particular, _extracted_ page do a quick extraction of any other unextracted reports. But I'm trying to think of a way to easily extract that initial report.

    At this point I should point out that all of what I'm saying here is mostly me thinking out loud on what is ultimately just a frivolous project (taken in scope of PYKL). And thinking back I realize I've neglected to account for how one would find these web accessible reports in the first place without either knowing their filenames/dates, using the webserver's directory browsing capability, or actually having an initial index file that does all the heavy lifting without the need for sending all of this extra stuff across with the log archives.

    My initial concerns were to avoid having a separate front-end piece, so that all would be needed would be PYKL running, and a web browser, but on second thought, perhaps going with the front-end piece would be much easier, and it wouldn't require any tinerking with the PYKL core, which is a fine example of "don't fix what's not broken".

    Anyway, food for thought.

     
  • Swift

    Swift - 2011-03-21

    Everything you mentioned is already possible. Nothing complex at all, no worries :) In fact, I can whip up something if you give me about ten minutes for you, so that you can see it all indeed is possible within a matter of minutes. After my demo, I can tell you how I did it.

    What I'm going to do:
    1. Change "KeyName:" etc to something tiny; you kindly suggested "" . Good idea.
    2. Collapsible entries based on window handle id.
    3. Sortable entries based on your defined column >> eg, a-z: handle id, application directory, date, time, username, etc
    4. Index of log data with ability to sort by latest log file

     
  • Swift

    Swift - 2011-03-21

    Well here's an example of my current log file with #1 finished:

    20110321|1935|noprocname|3408906|Matt|Untitled - Notepad|notepad is now open[rtrn]here we will test our [tab]special [tab]c
    20110321|1936|noprocname|3408906|Matt|Untitled - Notepad|haracters[ctrl][rtrn]contrls [ctrl][ctrl][ctrl][back][back][back][back][back][back][back][back][back][back][back][back][back][back][back][back]
    
     
  • Swift

    Swift - 2011-03-21

    There was an error in my code, but it's fixed now. Here's the updated log file:

    20110321|1941|noprocname|7145744|Matt|Untitled - Notepad|[tab][rtrn][rtrn][back][back]
    20110321|1942|noprocname|66278|Matt|dist|[Lctrl][Rctrl][F12]
    20110321|1943|C:\dist\o01Zk98M.exe|3278726|Matt||[Lctrl][Rctrl][F11]
    

    Just showing you #1 is finished. Now for 2-4..That's easily done in excel, but you want it in Python. Hm.

     
  • Matt

    Matt - 2011-03-21

    You're quick ;]

     
  • Swift

    Swift - 2011-03-21

    Here's an excel macro to do #3. All you do is open your log file in excel, and then run this macro.
    Before:

    After the macro:

    From here you can sort by clicking the arrows ( i clicked the arrow next to date )

    Sub MakeItPretty()
        Columns("A:A").Select
        Selection.TextToColumns Destination:=Range("A1"), DataType:=xlDelimited, _
            TextQualifier:=xlDoubleQuote, ConsecutiveDelimiter:=False, Tab:=False, _
            Semicolon:=False, Comma:=False, Space:=False, Other:=True, OtherChar _
            :="|", FieldInfo:=Array(Array(1, 5), Array(2, 1), Array(3, 1), Array(4, 1), Array(5, _
            1), Array(6, 1), Array(7, 1)), TrailingMinusNumbers:=True
        Columns("A:A").EntireColumn.AutoFit
        Range("A1").Select
        Selection.EntireRow.Insert
        ActiveCell.FormulaR1C1 = "Date"
        Range("B1").Select
        ActiveCell.FormulaR1C1 = "Time"
        Range("C1").Select
        ActiveCell.FormulaR1C1 = "Dir"
        Range("D1").Select
        ActiveCell.FormulaR1C1 = "WindowID"
        Range("E1").Select
        ActiveCell.FormulaR1C1 = "User"
        Range("F1").Select
        ActiveCell.FormulaR1C1 = "Title"
        Range("G1").Select
        ActiveCell.FormulaR1C1 = "Keys"
        Range("A1:G1").Select
        Selection.AutoFilter
    End Sub
    
     
  • Swift

    Swift - 2011-03-21

    Oops, wrong link for the 3rd image (sorting). Correct link:

    (Wish they let us edit our posts)

     
  • Swift

    Swift - 2011-03-22

    This macro can become very useful. I have a log file of over 800 entries, and by simply running the macro, it turns it into this:
    (I've blurred it due to sensitive information, notice i'm on row 339. I've also frozen the top row so it shows on the 1st row whenever i scroll down)


    And updated macro with 1st row frozen ( so it shows when you scroll ):

    Sub MakeItPretty()
        Columns("A:A").Select
        Selection.TextToColumns Destination:=Range("A1"), DataType:=xlDelimited, _
            TextQualifier:=xlDoubleQuote, ConsecutiveDelimiter:=False, Tab:=False, _
            Semicolon:=False, Comma:=False, Space:=False, Other:=True, OtherChar _
            :="|", FieldInfo:=Array(Array(1, 5), Array(2, 1), Array(3, 1), Array(4, 1), Array(5, _
            1), Array(6, 1), Array(7, 1)), TrailingMinusNumbers:=True
        Columns("A:A").EntireColumn.AutoFit
        Range("A1").Select
        Selection.EntireRow.Insert
        ActiveCell.FormulaR1C1 = "Date"
        Range("B1").Select
        ActiveCell.FormulaR1C1 = "Time"
        Range("C1").Select
        ActiveCell.FormulaR1C1 = "Dir"
        Range("D1").Select
        ActiveCell.FormulaR1C1 = "WindowID"
        Range("E1").Select
        ActiveCell.FormulaR1C1 = "User"
        Range("F1").Select
        ActiveCell.FormulaR1C1 = "Title"
        Range("G1").Select
        ActiveCell.FormulaR1C1 = "Keys"
        Range("A1:G1").Select
        Selection.AutoFilter
        With ActiveWindow
            .SplitColumn = 0
            .SplitRow = 1
        End With
        ActiveWindow.FreezePanes = True
    End Sub
    
     
  • Matt

    Matt - 2011-03-22

    Looks good. Just for kicks I started working on a simple php viewer anyway. I'll post it up once it does anything useful ;]

     
  • Matt

    Matt - 2011-03-22

    Well it's not sortable yet, but here's what I whipped up so far: http://www.mattseng.com/personal/pyklv/v1.0/

    Just for a little bandwidth protection there's a password on this page:
    user: pyklv
    pass: ucHQ35GM

     
  • Swift

    Swift - 2011-03-22

    Now that's awesome. Could you message me the source to your viewer? I'd love to chec it out, I feel like I could implement sortable table headers easily for you, if not you can as well.

     
  • Matt

    Matt - 2011-03-22

    Hope you don't mind that it's strictly PHP ;]

    Sent.

     
  • Swift

    Swift - 2011-03-22

    Oh man, you know a lot more about php than I do haha. Do you think you could setup your script to import and store the data in a mysql server, whereupon after storage it deletes the log file and only reads logs from the database?

     
  • Matt

    Matt - 2011-03-22

    I certainly could. I had briefly considered that, but couldn't really come up with any reasons why it might be desirable or preferable over just reading logs.

    Of course if you went all out with a fully web accessible control panel that tracks multiple devices (or log sources) it would pretty sweet I guess.

     
  • Swift

    Swift - 2011-03-22

    Well here's sortable done on the detailed_logs:
    http://pykeylogger.tk/h/log01/

    Try clicking on one of the headers.
    ( I messed up your css, sorry lol)

     
  • Matt

    Matt - 2011-03-22

    Very nice.

     
  • Swift

    Swift - 2011-03-22

    Any luck on mysql?

     
  • Matt

    Matt - 2011-03-22

    I haven't started, and I probably won't have time this week. CGI is one thing but CGI+SQL takes like 4 times more debugging.

     
  • Swift

    Swift - 2011-03-22

    Aw okay. By the way, I was able to "minify" the packaged source down to 14 files from the original 34 files.

    It's cleaner : )

     
  • Matt

    Matt - 2011-03-23

    JOOC, what all did you remove? I'm not familiar enough with the source to know what it all does, so I'm curious what isn't needed.

     
  • Matt

    Matt - 2011-03-23

    Well I could look my source and compare, but I should say I'm curious _why_ it isn't all needed.

     
  • Swift

    Swift - 2011-03-23


    Here's just a few of them I deleted. None of these were being called in pykeylogger, thus they were just sitting there not doing anything. If you check the top of each class for what's being imported, you won't find any of these there. (If you do let me know lol). Also, library.zip is embedded into pykeylogger.exe using py2exe's nozip option. Eliminating the .zip file but increasing the .exe's file size to around 4mb.

    And w9xpopen.exe isn't needed as well, since w9xpopen is for ancient windows machines (95/98).

     
  • Swift

    Swift - 2011-03-23

    Although _ssl.pyd is needed, since we are using TLS in pykeylogger . But it's not necessary if you don't use the email function

     
1 2 > >> (Page 1 of 2)

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks