Menu
▾
▴
Re: [mysar-support] New importer version released
|
From: Tomas F. <to...@ce...> - 2007-05-31 14:30:15
|
After some more testing, I believe it's a problem on my side only. I import the IP address in a special format and although I did the testing with the standard squid log format, some changes I had done to the tableSummaries db table made the combined unique key not unique enough. So the MySQL statement failed because the database believed there was an identical record already. I am not sure if anyone will understand it from my explanation though ;) But the point is, I am 99% sure I caused this problem myself. Your importer is working great! My apologies... By the way, I hacked/modified my version to, optionally, import the IP address unaltered / in human readable format as it appears in the log - i.e. not converting it to the net short format. If anyone is interested I can send you the modified version. It will probably break some Mysar functionality though. On the other hand, it's just an option in the config file so you can turn it off. Tomas ----- Original Message ----- From: "Cassiano Martin" <cas...@at...> To: "mysar support mailing list" <mys...@li...> Sent: Wednesday, May 30, 2007 8:41 PM Subject: Re: [mysar-support] New importer version released > I've tested here, got no errors. It imported it normally. > There is a new test version on my site. > > http://www.polaco.pro.br/mysar/testing > > so, do the following: > > invoke the importer using the -g switch. this command will run a > database creator. > it will ask you for the mysql admin password, a database name, and a > mysar user and password. > create a new database, giving it a different name, and it will not touch > yours current database. > modify the config file, to point to the new database, and run it with > the log file that makes it crash. > > I'm going on with debugging. I'll try to reproduce the error here. > > Tomas France escreveu: >> Hi! >> >> I apologize for the delay. I haven't had a chance for more testing lately >> but it should be easy to replicate the error: >> >> ------------------ >> >> 1) Rotate the logs with squid -k rotate to have an empty access.log >> 2) Copy & paste the below 2 lines into it: >> >> 1180265352.705 313 123.123.123.123 TCP_MISS/200 1701 GET domain.com:443 >> testuser DIRECT/123.123.13.123 text/html >> 1180265352.782 348 123.123.123.123 TCP_MISS/200 1717 GET domain.com:443 >> testuser DIRECT/123.123.13.123 text/css >> >> (The email will probably wreck the lines somewhat...) >> >> 3) In the mysar MySQL database set the "lastTimestamp" value in the >> config >> table to a lower timestamp then the log record, for example to >> "1170265352.100". >> 4) Run the mysar offline importer >> >> ------------------ >> >> Doing so on my server gives me this error: >> >> "2007-05-30 14:01:07 src/mysar.c:484 -->>Statement Error in main() >> Duplicate >> entry '2007-05-27-2-6-6-7' for key 2". >> >> My server is running on CentOS 4.5 and CPanel. I converted my MySQL >> tables >> to InnoDB but that should have no impact on the error I believe. Also, I >> am >> running the latest stable MySQL - version 5.0.41 (Intel Compiler binaries >> downloaded directly from mysql.com). I compiled the offline importer with >> GCC 4.2.0 and another version with ICC 9.1.047. Both versions give me the >> same error when trying to import the above mentioned records. >> >> Tomas >> >> >> >> ----- Original Message ----- >> From: "Cassiano Martin" <cas...@at...> >> To: "mysar support mailing list" <mys...@li...> >> Sent: Monday, May 28, 2007 12:37 PM >> Subject: Re: [mysar-support] New importer version released >> >> >> >>> Tomas, could you supply me, a part of your logfile? >>> I've tested here, and I still got no errors. >>> >>> Its very weird, because I havent touched the part of file reading >>> routine... >>> The split section is white spaces, TABs are ignored. >>> Its on my plans to change it for a more efficient way, but no progress >>> at all. >>> >>> Send me a part of the log, that crashes the imported, so I can find the >>> it more easy. >>> >>> []'s >>> Cassiano Martin >>> >>> Tomas France escreveu: >>> >>>> Hi everyone, >>>> >>>> I have been testing it and it looks like it does not happen all the >>>> time. >>>> The main problem seems to be indeed in the timestamp, more than one >>>> problem >>>> actually. >>>> >>>> If between the timestamp & the next field is a tab instead of space, it >>>> causes the segmentation fault error. If I manually remove the tab and >>>> replace it with a single space, the error is gone but there is a >>>> problem >>>> with importing still - probably caused by the fact the importer >>>> improperly >>>> parses the part of the timestamp after the dot symbol (or perhaps >>>> ignores >>>> it >>>> altogether?). >>>> >>>> >>>> >>>>>> just a small note: the importer need these flags do be not touched: >>>>>> TCP_IMS_HIT, TCP_REFRESH_HIT, TCP_HIT, TCP_NEGATIVE_HIT, TCP_MEM_HIT >>>>>> << >>>>>> >>>>>> >>>> Not really, it works just fine without them. This is the logformat I am >>>> using: >>>> >>>> logformat idc_custom %ts.%03tu %6tr 1 X/X %<st X %ru %un X X >>>> >>>> The number "1" is there instead of the IP address and is the physical >>>> server >>>> identificator. But for the record, I have tested the new version of >>>> importer >>>> with the standard squid log format as well. >>>> >>>> I have not had the described problem(s) with the older version of >>>> importer. >>>> I will keep testing the new version. I realize I have a very custom >>>> setup >>>> but I always try to test everything with the standard setup too. >>>> >>>> Tomas >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Cassiano Martin" <cas...@at...> >>>> To: "mysar support mailing list" <mys...@li...> >>>> Sent: Monday, May 28, 2007 12:07 AM >>>> Subject: Re: [mysar-support] New importer version released >>>> >>>> >>>> >>>> >>>>> Hi! >>>>> >>>>> I need to know exactly how the error happened. I've tested here, and I >>>>> still got no errors. >>>>> When you imported the log, have you modified the file? or imported it >>>>> directly from squid? >>>>> >>>>> just a small note: the importer need these flags do be not touched: >>>>> TCP_IMS_HIT, TCP_REFRESH_HIT, TCP_HIT, TCP_NEGATIVE_HIT, TCP_MEM_HIT >>>>> I also cant understand why this: entry "2007-05-27-4-1-5-7' for key 2" >>>>> the date is wrong, and the executed statement was updateSumsOut. >>>>> >>>>> >>> ------------------------------------------------------------------------- >>> This SF.net email is sponsored by DB2 Express >>> Download DB2 Express C - the FREE version of DB2 express and take >>> control of your XML. No limits. Just data. Click to get it now. >>> http://sourceforge.net/powerbar/db2/ >>> _______________________________________________ >>> mysar-support mailing list >>> mys...@li... >>> https://lists.sourceforge.net/lists/listinfo/mysar-support >>> >>> >>> >> >> >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by DB2 Express >> Download DB2 Express C - the FREE version of DB2 express and take >> control of your XML. No limits. Just data. Click to get it now. >> http://sourceforge.net/powerbar/db2/ >> _______________________________________________ >> mysar-support mailing list >> mys...@li... >> https://lists.sourceforge.net/lists/listinfo/mysar-support >> >> >> > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > mysar-support mailing list > mys...@li... > https://lists.sourceforge.net/lists/listinfo/mysar-support > > |