[Assp-test] Antwort: Re: rebuild spamdb - runtime

[Thomas E. <Tho...@th...>]

>just to delete files

Yes this would be an option - but I've found a better way (IMHO) :)
ASSP will move such files out of the corpus to another location - so it is 
possible to request them via BlockReport-resend and to manage the files in 
the MaillogTail GUI.

Thomas



Von:    Scott MacLean <as...@ho...>
An:     ASSP development mailing list <ass...@li...>
Datum:  26.03.2012 21:41
Betreff:        Re: [Assp-test] rebuild spamdb - runtime



How about an option just to delete files in the corpus that cause 
excessive scanning times? That would be a lot more admin-friendly.

At 08:54 AM 3/26/2012, Thomas Eckardt wrote:

>Hi all,
>
>investigating an IMHO too long rebuild runtime issue - I assumed a bug in
>the code - I've found that this issue was caused by only a few files in
>the corpus. For some of the files it has taken one minute or more to
>finish processing them.
>This large file processing time not depends on the size of the files, it
>only depends on the contents. All of the files are extremly HTML encoded
>and contains a very very large number of HTML tags.
>For example: I removed 6 files from the corpus and the rebuild time is
>decreased from 20 to 10 minutes.
>
>Such a very long processing time could lead in to long time record locks
>for Spamdb, Whitelistdb and Redlistdb - which could cause any other 
thread
>to run in to a stuck mode (get locked) and it will peak the CPU usage for
>a single core to 100% for a long time.
>
>The next relase will monitor the processing time for every file. The list
>of the longest 10 runtimes will be written to the rebuild log for every
>corpus folder, if the processing time for a single file is larger than 
one
>second.  The file 'rebuilddebug.txt' will contain the complete list of
>files with a processing time > 1 second.
>
>
>Thomas
>
>
>
>DISCLAIMER:
>*******************************************************
>This email and any files transmitted with it may be confidential, legally
>privileged and protected in law and are intended solely for the use of 
the
>
>individual to whom it is addressed.
>This email was multiple times scanned for viruses. There should be no
>known virus in this email!
>*******************************************************
>
>
>
>------------------------------------------------------------------------------
>This SF email is sponsosred by:
>Try Windows Azure free for 90 days Click Here
>http://p.sf.net/sfu/sfd2d-msazure
>_______________________________________________
>Assp-test mailing list
>Ass...@li...
>https://lists.sourceforge.net/lists/listinfo/assp-test
------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Assp-test mailing list
Ass...@li...
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
*******************************************************