On my Win98SE system with my test data I cannot see any difference in "Remove All" performance between RC4 and RC5.
When re-reading the "other" thread I noticed that Bill's test data used consecutive slots (1 to 968) while my test data used higher slot numbers which were not consecutive.
After carefully trimming the test data to ensure consecutive slot numbers were used, I re-ran the "Remove All" tests several times:
RC4 = 75 secs, 77 secs
RC5 = 76 secs, 76 secs
My test data uses slots 2755 to 3339, Bills data used slots 1 to 968. I used Win98SE, Bill used WinXP Pro SP2.
Perhaps it is a Win98SE limitation or I am doing something silly?
Brian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
>Perhaps it is a Win98SE limitation or I am doing something silly?
You didn't, by chance, permanently set your popfile.db to synchronous=off, in which case the temporary change that POPFile makes wouldn't have any impact.
You can check the status with the commandline utility...
sqlite popfile.db
SQLite version 2.8.15
Enter ".help" for instructions
sqlite> pragma default_synchronous;
synchronous
-----------
1
sqlite>
The number returned tells you the setting, 0=OFF, 1 = NORMAL, 2 = FULL
Scott
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
>> You didn't, by chance, permanently set your popfile.db to synchronous=off, <<
Here is what I found when I checked the setting:
SQLite version 2.8.15
Enter ".help" for instructions
sqlite> pragma default_synchronous;
1
sqlite>
I've found that I can speed things up by a factor of around 4 if I delete the message history files manually (120 secs get reduced to just under 30 secs) so being FAT-based might be significant.
Brian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have (at last!) been able to find a difference between RC4 and RC5 on my Win98SE system when using "Remove All".
My test data consists of 809 messages, ranging in size from 873 bytes to 657.9 KB (average size 9.79 KB).
I tested RC4 and RC5 using the test database and all 809 message files and then tested RC4 and RC5 again with just the database (i.e. left the 'messages' folder completely empty). Each test was repeated 5 times.
RC4 with message files = average 131.2 secs (min 131, max 132)
RC5 with message files = average 128.2 secs (min 124, max 130)
RC4 with no message files = average 30.6 secs (min 29, max 33)
RC5 with no message files = average 20 secs (all 5 runs took 20 sec each)
So it looks like the processing involved in actually deleting the message files from the 'messages' folder was masking the effect of the changes in RC5
Brian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Are you using a 4200 RPM laptop hard drive -- or even a 5400 RPM drive -- by chance? I usually have so many files in my messages folder that it can literally take half a day to delete them all -- and that's on a 4 disk 10,000 RPM RAID-5 array.
--
Tom Voss
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No, it is a 7,200 RPM ATA-100 HDD connected via a Promise ATA-100 controller.
Tried defragmenting the disk but it made no difference.
Perhaps VCACHE or some other settings could be tweaked (it's been a long time since I changed anything there - the last time I ran any tests, HDD performance seemed to be as expected)
Brian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
When Tom asked about the HDD specifications, I couldn't remember the details so I asked Google. One of the results mentioned a firmware update for the disk so I installed it.
There is still no real difference between RC4 and RC5 when the 809 message files exist but the new firmware has reduced this "Remove All" time by almost 22% which was a nice surprise.
Brian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well I think ftlight was right and the way in which I applied helphand's async. delete fix didn't actually fix the problem.
Here's RC5 that should do the job:
http://getpopfile.org/popfile-0.22.2RC5.zip
http://getpopfile.org/popfile-0.22.2RC5-windows.zip
The change is also in CVS.
John.
That fixed it! 515 messages removed in about 12 seconds.
Bill
Three cheers for the open source paradigm.
--
Jim
All right. That did the trick. Some more benchmarking results.
The old ones (as a baseline)
Remove 3272 messages with
rc1 378 seconds
rc4 384 seconds
3321 messages with rc5: 134 seconds.
And if you want to know how those 134 seconds are spent:
If I comment out the unlink and the directory cleanup in release_slot, removing those 3321 messages takes 22 seconds.
If I comment out the cleanup stuff only: 25 seconds.
And the funniest result of them all: If only the unlink is commented out: 271 seconds.
Manni
On my Win98SE system with my test data I cannot see any difference in "Remove All" performance between RC4 and RC5.
When re-reading the "other" thread I noticed that Bill's test data used consecutive slots (1 to 968) while my test data used higher slot numbers which were not consecutive.
After carefully trimming the test data to ensure consecutive slot numbers were used, I re-ran the "Remove All" tests several times:
RC4 = 75 secs, 77 secs
RC5 = 76 secs, 76 secs
My test data uses slots 2755 to 3339, Bills data used slots 1 to 968. I used Win98SE, Bill used WinXP Pro SP2.
Perhaps it is a Win98SE limitation or I am doing something silly?
Brian
>Perhaps it is a Win98SE limitation or I am doing something silly?
You didn't, by chance, permanently set your popfile.db to synchronous=off, in which case the temporary change that POPFile makes wouldn't have any impact.
You can check the status with the commandline utility...
sqlite popfile.db
SQLite version 2.8.15
Enter ".help" for instructions
sqlite> pragma default_synchronous;
synchronous
-----------
1
sqlite>
The number returned tells you the setting, 0=OFF, 1 = NORMAL, 2 = FULL
Scott
>> You didn't, by chance, permanently set your popfile.db to synchronous=off, <<
Here is what I found when I checked the setting:
SQLite version 2.8.15
Enter ".help" for instructions
sqlite> pragma default_synchronous;
1
sqlite>
I've found that I can speed things up by a factor of around 4 if I delete the message history files manually (120 secs get reduced to just under 30 secs) so being FAT-based might be significant.
Brian
I have (at last!) been able to find a difference between RC4 and RC5 on my Win98SE system when using "Remove All".
My test data consists of 809 messages, ranging in size from 873 bytes to 657.9 KB (average size 9.79 KB).
I tested RC4 and RC5 using the test database and all 809 message files and then tested RC4 and RC5 again with just the database (i.e. left the 'messages' folder completely empty). Each test was repeated 5 times.
RC4 with message files = average 131.2 secs (min 131, max 132)
RC5 with message files = average 128.2 secs (min 124, max 130)
RC4 with no message files = average 30.6 secs (min 29, max 33)
RC5 with no message files = average 20 secs (all 5 runs took 20 sec each)
So it looks like the processing involved in actually deleting the message files from the 'messages' folder was masking the effect of the changes in RC5
Brian
Are you using a 4200 RPM laptop hard drive -- or even a 5400 RPM drive -- by chance? I usually have so many files in my messages folder that it can literally take half a day to delete them all -- and that's on a 4 disk 10,000 RPM RAID-5 array.
--
Tom Voss
>> Are you using a 4200 RPM laptop hard drive <<
No, it is a 7,200 RPM ATA-100 HDD connected via a Promise ATA-100 controller.
Tried defragmenting the disk but it made no difference.
Perhaps VCACHE or some other settings could be tweaked (it's been a long time since I changed anything there - the last time I ran any tests, HDD performance seemed to be as expected)
Brian
When Tom asked about the HDD specifications, I couldn't remember the details so I asked Google. One of the results mentioned a firmware update for the disk so I installed it.
There is still no real difference between RC4 and RC5 when the 809 message files exist but the new firmware has reduced this "Remove All" time by almost 22% which was a nice surprise.
Brian