Menu
▾
▴
#43 Can't parse rejik log
Hello!
I have an error about redirector log.
I conf file the string is
redirector_log_format #year#-#mon#-#day# #hour# #list#:#tmp# #ip# #user# #tmp#/#tmp#/#url#/#end#
And the example of log is
2017-03-27 17:02:49 CHATS: 10.0.0.42 - d.yakushev@triatsystems.ru registrar-rr.prod.registrar.skype.com:443 (urls rule: skype.com)
But when I make report I see the next info
SARG: Reading redirector log file "/var/log/rejik/redirector.log" SARG: Creating top users report... SARG: Creating download report... SARG: Creating top sites report... SARG: Creating report to list who visisted what site... SARG: (info) Denied report not produced because it is empty
What does #tmp# mean in template? How to correct the template?
#tmp#ignores the token.It's difficult to tell how to fix the template from that single example. So let's actually parse the string to show how sarg does it.
Sarg searches the log line for the first character found in the template after a token. The beginning of the template is
#year#-. Therefore, sarg searches for the first "-" in the log line and takes everything before the "-" character as part of the year. Then it continues and searches for#mon#-. Sarg takes everything between the previous "-" and the next "-" as the month number. And so on.As a result, the log line is parsed like this:
#year#-== "2017"#mon#-== "03"#day#<space>== "27"#hour#<space>== "17:02:49"#list#:== "CHATS"#tmp#<space>== ""#ip#<space>== "10.0.0.42"#user#<space>== "-"#tmp#/!! no match found as "/" is nowhere in what is left of the stringTo solve your problem, you must look at the strings you have in your log and find a pattern that can be used to extract the
#url#.If you expect to extract "registrar-rr.prod.registrar.skype.com" as the url (i.e. interrupt the url at the first colon), use
If the ":" isn't always present or should be part of the url, use a space to anchor the token end at what looks like " (urls rule:…" in the example:
Hello!
Here is a piece of redirectors log.
I used this template
redirector_log_format #year#-#mon#-#day# #hour# #list#:#tmp# #ip# #user#@#tmp# #url# #end#
But report is short and without users, see attachment.
Your template redirector_log_format #year#-#mon#-#day# #hour# #list#:#tmp# #ip# #user# #url# #end# doesn't work.
Attachment.
And warnings in log:
I used the following template on your sample file
Lines were extracted as expected or, at the very least, I could not spot any error. See attached screenshot.
You are right, my first throw was wrong. I had to add a
#tmp#to ignore the "-" before the user name and I removed the domain name as you did.Hello!
Beautiful! It works!
And what to do with entries which are not shown into the report? Does it mean that they are repeated? Or it's just to reduce the report? Is it possible to show them all?
Every entry from your sample file is taken into account in the report I generated for the screen capture but only 10 are shown as requested by the
squidguard_report_limitoption (10 by default).Therefore, if you want to see the six hidden entries for a.levdik, set
squidguard_report_limitto a higher value or to zero to disable the limit.If, on the other hand, some entries are really ignored from the full rejik log, then, it means those entries don't match the template. What is special about those entries? Do you see how they could fail to be parsed by the redirector template?
And one more question.
I have a script which is launched by cron.
All log files are combined in $LOG_FILES and read by sarg, but what about redirector's log? Only one mention is in sarg.conf, but what to do if I have some redirector.log files which are gziped. Is it possible to launch sarg from CLI and point to redirector's logs similar to squid log files?
No, this cycle is not needed. You may just pass
exec /usr/bin/sarg var/log/squid/access.log*Wildcard is allowed for access log files.
In relation to redirector's log files, it cannot be passed in CLI. Sarg reads the config and log files mentioned in it. You should rotate and squeeze them after processing by sarg. I guess so far it's the best option.
In addition to what Evgeniy wrote, you may further simplify the script by replacing
-d $YESTERDAYwith-d day-1. It saves runningdateto calculateYESTERDAYin the script. The syntax for-dis explained in the wiki page [Command line options].Beside, I believe there is an error in your script:
-t 23:59:59is after the&>/dev/nullon the command line. I believe option -t is ignored in that case which is good as it would truncate your report to only show accesses made exactly at 23:59:59.Related
Wiki: Command line options
I found the option for CLI
Seems to me that it's what I need. Can I use it for Rejik logs? And gzipped logs?
Of course it's possible to do as Evgeniy adviced but if I want to have redirector log for a month I have to wait and rotate it only after making a report.
You are right, that option can read uncompressed Rejik log.
But I just committed a change to read compressed logs too. It is a one line change that was the obvious thing to do.
The new feature is available in the git master branch. It is sarg version 2.4. I recommend you clone a working copy of the source and build it.
Frederic, thanks a lot for hint with days.