After a 287 day hiatus (Wow, has it really been that long since our internal DAD Dev server died??) we're back at the development grindstone. We've stood up a new XServe with a DAD development instance running.
Now that things are up and running (on Windows again) we need to take a step back and see what's going on with the scheduler. When we first ripped out the Java and replaced it with Perl we discovered some bugs in the Windows implementation of the Perl Process code. There are still some artifacts of this left over.... read more
You may have noticed that the project has been dormant for quite a while. I'm very glad to let you know that things will begin cranking up again over the next few months. The first item to look for will be an example Virtual Machine for VMWare.
What we've done is created an Ubuntu VM with the DAD server pre-installed with some sample data. This will give you the opportunity to explore the interface, see what sorts of things DAD can do and even put up a test deployment without trying to build or install DAD yourself!... read more
Greetings one and all.
I've been poking at the SQL queries for quite some time and puzzling over why a query for, let's say, all Postfix events comes back nearly instantly while a query for only Postfix events in the last 60 minutes takes upwards of three minutes. I've found the answer:
The MySQL optimizer is a cranky beast.
Of course, I absolutely love MySQL. The performance is fantastic, but the query optimizer does make some odd decisions sometimes. The reason for the super slow queries based on timestamps is that the query optimizer seemed to believe that it was far faster to pull in everything based on the event ID even though the main limiting factor was the timestamp. Since we're limiting on timestamp and indexing on the event ID, the database server essentially had to scan every event. As you can imagine, this is very bad. ;)... read more
In our hurry to get the major 0.06 release out the door we neglected to update the installation information files. We also discovered that we had forgotten to include the maximum packet size configuration change requirements in the checklist/script. This release corrects these problems.
After many months I think that we can confidently say that the main schema for the database (ie, that of the events) is stable, so we've rolled up all of the most recent changes and pushed our version up to 0.06!
There are still some significant outstanding interface issues, so you should expect another release in about two weeks. Here are some notable outstanding issues:
Currently, you cannot use the query modeling tools to successfully save a query. There are a few items missing from the page and it's still built around designing a SQL query rather than doing string searches. This is about 60% done in the dev environment.... read more
You'll be glad to hear that we've spent some significant time working on converting the alerting system over to use the new database format. We're well on our way and have introduced a new function into the "Reports.pm" file and made alerting even easier!
If you take a look in the "RemoteDesktop.pl" alert, you'll see that this is really the only required code:
#Read in and evaluate the configuration values... read more
We've pushed a change into SVN that allows you to configure the database credentials and location for the Java scheduler. The new file is located in DAD/dbconfig.jv.
Currently, there are several configuration files floating around. The first is mentioned above. The next is the config file "dbconfig.ph" located in DAD/jobs/. This config file is for all of the Perl jobs. Aggregator.ph is located in DAD/jobs/Log Parser/ and is used to configure various aspects of the aggregator. The last configuration file is located in DAD/web/config/dbconfig.php. This file is used by the PHP site to configure the database location and credentials.... read more
We thought that you'd be thrilled to hear that the schema has been finalized for event storage. We went through a long and painful exercise in breaking down the events further, storing unique fields in addition to unique strings, but the insert performance was incredibly poor. Even in a very small test environment the insert process using stored procedures was painfully slow. The only solution would be to introduce a virtual method of storing unique fields in memory like we are with unique strings, but the complications involved were just more than we are willing to tackle at this time.... read more
Just a quick Dev update for you!
First off, we've put several stored procedures into the database to allow for easier searching. Unfortunately, while we knew that MySQL has some known performance issues with subquery optimization, especially in stored procedures, we had no idea how severe the problem is.
A simple search for two arbitrary pieces of text with a dataset of 5 million events could easily send the SQL thread into a "Preparing" status that lasts 300+ seconds before the PHP thread times out and gives up. Clearly this is unacceptable. We've recoded the dynamic string query functions within the PHP and the Perl reports module. The same query through this code returns in under 2 seconds. Clearly, there're some optimization issues!!!
As you can tell, there has been some lag time since the last post, but there is news!
Behind the scenes, we've spent a lot of time experimenting with further normalizing the data down into a unique_fields table, but at least for now, we've decided to shelve that project. It's a great idea from the point of view of search and storage efficiency, but when it comes to actually inserting the data into the database, the speed is absolutely unacceptable.... read more
I've been watching the forums lately and have noticed that a number of people have been having a variety of installation issues. To be quite honest, I had lost touch with the installation process since I've been working off of my dev version for about a year now.
Yesterday, I built a new dev server to verify that the migration process from the old to the new aggregator and query format on an X64 box. In the process, I discovered that the install, particularly on a 64 bit system, is rather tricky. In fact, for 64 bit systems, the install script won't work at all!!... read more
We've just committed the newest changes to Aggregator V2 into the tree. This aggregator is slightly more efficient with the new schema than the one we were working with earlier in the week but it is still painfully slow when it comes to inserting data into the database. On the test rig it is steadily falling behind the event rate of 100+ servers. Even so, we're looking good and making headway!... read more
A few months ago we spent a lot of time creating the log carver and the ability to create log carving rules in the web interface. For this reason, the really wonderful news that we have is a bit disappointing since we spent so much time on the carver!
Since we've modified the database so that every word in the entire event is indexed for every event, it has more or less eliminated the need for the regular expression carving. You can now find any event based on any combination of words that you're looking for!... read more
If you're tracking the DAD project, you'll find this to be extremely important news. Over the past six to nine months, a variety of tables were added and other database changes made that were not the most optimal. We are happy to announce today that a significant redesign has begun.
The most significant and important change is that the way that events are stored is being completely changed. All of the events will now be stored in a table called 'events' and the entire content of the events will be full text indexed through a normalization process.... read more
I'm sure by now that you all thought that this project had died, but it's not true! We've had a lot happening in the background and have started pushing out SVN updates.
At this point in time, we do expect to release another official version before the end of August, but we would recommend in the meantime that you seriously consider using SVN to pull down a copy and then maintaining your copy through SVN checkout.... read more
Well, we didn't quite make our goals for this afternoon, but it is very important that we not delay the release of the re-engineered scheduler, so we're pushing out 0.05 Interim. In the next day or so please look for 0.05.1 at the least, perhaps 0.06. 0.06 will include the ability to upgrade your DAD in place regardless of schema changes which we are sure will be a welcome change!
Please look for a new release to be available sometime on Monday. Features will include:
* Fixed scheduler (No more heap error!)
* More built-in alert jobs!
* Ability to schedule backups of tables!
* Additional alerting mechanism for better correlation of events
* Interface for managing retention times
* Ability to update queries through the interface
* Ability to assign RBAC to queries
* Much more!!... read more
We've been receiving a number of questions recently related to how much data DAD can handle effectively. To give you some idea, we can tell you that one of the first adopters of DAD is currently handling more than 1.2 billion events in their installation. How many events are you handling? Please send us feedback and let us know!
We would like to invite all in the community who are actually using DAD to please feel free to submit feature requests using the Sourceforge Tracker system. We will try to prioritize and assign out the work for those that seem reasonable and useful.
In order to determine which features are most desirable we are also recommending that if you see a feature request that you could benefit from, please "bump" the request with a note so that we have some idea of what matters to who.... read more
The memory leak in the scheduler turned out to be a known issue in the JDBC driver. Apparently, closing a result set is not enough to allow the memory to be released to the garbage collector. We've re-engineered how the database interface to the scheduler works so that it now passes a job object rather than a result set object which has effectively resolved the memory leak.
We are currently out of the office so do not have the resources to track this down at the moment, however we have observed that the scheduler that had been running happily for many days has suddenly started throwing out of memory errors. More than likely there is a dangling reference to an object that is preventing garbage collection.
The effect of this bug is that when the Scheduler quits, all periodic jobs will stop running. However, the Syslog and Aggregator jobs will continue functioning indefinitely so log data is not being lost as far as we can currently tell.... read more
The 0.04.1 release of DAD fails to mention that the PHP GD2 module must also be installed during the PHP installer process.
We have received scattered reports that the latest version of PHP has changed the default installation paths and options. While this is a good thing, it also means that it renders our installation instructions invalid. We will repost a modified 0.04 release later this evening to address these issues. For now, it appears that the simple workaround is:
* Install Apache first
* When you install PHP 5.2.1 tell it that you want the MYSQL and MYSQLI additional modules installed
* Tell PHP to configure itself for Apache 2.0.x
* Allow it to reconfigure Apache.... read more
DAD is now largely feature complete when it comes to log management. This is a major milestone. It means that you can do the following with DAD right now:
*Aggregate and alert on Windows event logs
*Aggregate, carve and alert on Syslogs
*Carve arbitrary log file formats and alert on them
*Schedule arbitrary jobs to run periodically or persistently
*Edit carving rules online through the web interface... read more
We are very happy to announce that a functional version of the Syslog daemon and the log carver have been pushed into the SVN tree!
If you are using the SVN version you will need to write log parsing rules manually at the moment, but this will change by the weekend. Essentially you need to create two rules. The first rule is a regex to match lines and the second rule is a regex to extract fields out of the matched line. When you perform the matches, you can use extended Perl regular expressions and you can specify up to 24 backreferences that will be expanded out and inserted as individual fields in the database.... read more