Has it been considered that the syslog transmission style could be used to send XALT entries to a syslog aggregator like splunk, Loggly, Logstash or Greylog. These could be used to store the raw events (with other logging data) and extracted via a query to the aggregators API.
Extraction could be a specific query, or to harvest the entries to go into the XALT database, however I can see that it may be beneficial to not use the XALT database and to figure out how to query the log aggregator directly.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We (Robert and myself - XALT developers) have focused on the library and
executable launch tracking primarily. We originally had only planned to
do the simple file transfer method, but then we realized that making the
transfer mechanism arbitrary so we could support direct to database, file
transfer, and our syslog method was the way to go. We also realized that
by doing so, it would be relatively easy (just time and effort) to support
other transfer mechanisms.
Having said that, I think you are talking about the phase where we
“scrape” the syslog files and move the data to the database and instead
propose using a syslog aggregator (I am assuming the data still goes to
syslog first). No, we did not explicitly talk about using a syslog
aggregator, but it would be easy to support in my opinion since it would
be simple to just not use the existing scraping script and use say write
another script to move/copy the data to the aggregator instead. And then
all the queries can be done in the aggregator.
If you don’t want the data to go to syslog first and you want one of these
aggregators to collect the data (skipping syslog), then you probably need
to write a new “transmission” function — I bet it would look a lot like
the syslog method. The thing about the syslog method is that a lot of
that is set up outside of XALT in the syslog configuration files. So you
might do the same kind of configuration for another tool, and then make
sure the transmission method logs to the new tools. It is probably more
complicated than this, but at a high level that should be roughly how it
would work.
If others have discussed this for XALT, I am not aware.
Has it been considered that the syslog transmission style could be used
to send XALT entries to a syslog aggregator like splunk, Loggly, Logstash
or Greylog. These could be used to store the raw events (with other
logging data) and extracted via a query to the aggregators API.
Extraction could be a specific query, or to harvest the entries to go
into the XALT database, however I can see that it may be beneficial to
not use the XALT database and to figure out how to query the log
aggregator directly.
Has it been considered that the syslog transmission style could be used to send XALT entries to a syslog aggregator like splunk, Loggly, Logstash or Greylog. These could be used to store the raw events (with other logging data) and extracted via a query to the aggregators API.
Extraction could be a specific query, or to harvest the entries to go into the XALT database, however I can see that it may be beneficial to not use the XALT database and to figure out how to query the log aggregator directly.
Aaron,
We (Robert and myself - XALT developers) have focused on the library and
executable launch tracking primarily. We originally had only planned to
do the simple file transfer method, but then we realized that making the
transfer mechanism arbitrary so we could support direct to database, file
transfer, and our syslog method was the way to go. We also realized that
by doing so, it would be relatively easy (just time and effort) to support
other transfer mechanisms.
Having said that, I think you are talking about the phase where we
“scrape” the syslog files and move the data to the database and instead
propose using a syslog aggregator (I am assuming the data still goes to
syslog first). No, we did not explicitly talk about using a syslog
aggregator, but it would be easy to support in my opinion since it would
be simple to just not use the existing scraping script and use say write
another script to move/copy the data to the aggregator instead. And then
all the queries can be done in the aggregator.
If you don’t want the data to go to syslog first and you want one of these
aggregators to collect the data (skipping syslog), then you probably need
to write a new “transmission” function — I bet it would look a lot like
the syslog method. The thing about the syslog method is that a lot of
that is set up outside of XALT in the syslog configuration files. So you
might do the same kind of configuration for another tool, and then make
sure the transmission method logs to the new tools. It is probably more
complicated than this, but at a high level that should be roughly how it
would work.
If others have discussed this for XALT, I am not aware.
-----Original Message-----
From: Aaron Hicks aethylred@users.sf.net
Reply-To: "[xalt:discussion]" general@discussion.xalt.p.re.sf.net
Date: Wednesday, September 16, 2015 at 8:46 PM
To: "[xalt:discussion]" general@discussion.xalt.p.re.sf.net
Subject: [xalt:discussion] Using syslog transmission to store XALT data in
a syslog aggregator