Dear all,
I wanted to optimize my music collection containing about 4,000 mp3 files. After optimization I found that about 10% of files were deleted. Do you have any ideas why? the screenshot is added.
I did some experiments and found that the problem does not relate to plugins. The same deletion problem was confirmed for other files, see DOC and PDF as examples. Considering that procession is 0 sec, it seems that no plugins were run and the optimizer just deleted the files. I think the optimizer have to check the final file size and if it is zero, restore original version.
Before I found the problem I processed several big archives. Some files could be restored from recycling, however it seems that about 5 % of the files are lost.
Sure I can share. But it seems that the problem is not repeatable. Next time the FO optimizes those files. I suggest you to check the presence of the file after running plugins. It will not fix the source of problem, but prevent losing of files as it happened to me.
P.S. If it helps, I used my archive HDD connected via USB.
1) Plugin execution is checked for being correct.
2) If it is the optimize size is checked. It required fille to exists. It should be greater than 8 bytes and less than the original one.
3) If so it is replaced, and if no, nothing happens.
Are you running it over a removable drive or something that could make files not being written?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for the explanations, but problem still exists and its critical.
I like your program, because it significantly simplifies optimization
process, combing advantages of many plugins. However, this problem makes
its usage very risky. File deletion typically happens, when processing big
number of files, so it is hard to detect that several files were deleted. I
hope you can find the exact problem.
I want to pay your attention that the processing time of the deleted files
is 0sec, so I can guess, that plugins were not properly executed, or were
not executed at all. Are there possible synchronization problems,
considering that I detected this problem, when using USB HDD? Maybe delayed
file operations? Try to complete them before going next.
I also recommend at the end of the work to check the existence of all files
in the list.
P.S. It could be fantastic, if you add option to clear metadata of many
files, where pdf and jpg are the most important.
P.P.S. I found that the FO deletes track information in mp3. Is it correct
behavior? Is it possible to turn it off?
P.P.S. If the file is corrupted or encrypted, the plugin may hang. I faced
with this problem in pdf files, which were encrypted by the company
security system. Could you check the consistency/readability of the files
before running plugins?
1) Plugin execution is checked for being correct.
2) If it is the optimize size is checked. It required fille to exists. It
should be greater than 8 bytes and less than the original one.
3) If so it is replaced, and if no, nothing happens.
Are you running it over a removable drive or something that could make
files not being written?
I will add to my todo listo to debug and try to check possible failured during the invocation to the plugins.
Regarding the option of clearing metadata it is not the scope of FileOptimizer, but I will not discard creating a new tool for bulk removing it in several file types.
For MP3 processing I am using MP3Packer and if "Copy metadata" is enabled FO will pass it the -t and -s flags which should not stripg metadata according to the plugin documentation:
-t Strip non-MP3 data at the beginning (mainly ID3v2 tags)
-s Strip non-MP3 data at the end (mainly tags)
If not, it would be better to report it to MP3Packer developers.
As for encryption, I did some quick tests to overcome the problem. Essentially I check if the reduction is excesive and if so, discard it. Of course this is a task of the plugin itself.
Thanks.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
One more consideration. Your program reports the final size of the deleted file as 0%, so it knows that the file was deleted. You may check the presence of file in the code, where 0% was obtained, and if it is zero, restore from the recycling.
Regarding mp3 metadata. I did not modify any setting. I checked and found that the checkbox to copy data is set. However, in the optimized files, there were no any D3v2 tags. Could you check that the plugin is called properly? If yes, I can report to developers.
Regarding encryption. I agree that it is plugin problem, but its affects usage of FO. Can you perform more checks of the file content? For example, our company security soft encrypts files and they begin from “<## NASCA DRM FILE - VER1.00 ##>”, so it can be easily detected that the pdf or docx file is encrypted.
P.S. If you create a good tool, for metadata deletion, it could be a great job. As far as I searched, there is no a good program for such simple action. The best I found was "ExifCleaner", however it still has some issues.
Have a nice day,
Anton
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear all,
I wanted to optimize my music collection containing about 4,000 mp3 files. After optimization I found that about 10% of files were deleted. Do you have any ideas why? the screenshot is added.
Thank you Anton.
Can you try to isolate the plugin failing?
How can I do it?
On Sun, Aug 18, 2024, 20:07 Nikkho nikkho@users.sourceforge.net wrote:
Sure. Take look at the FAQ section: https://nikkhokkho.sourceforge.io/?page=FileOptimizer
See how to enable the log, and then how to disable a plugin until you get the one failing.
I did some experiments and found that the problem does not relate to plugins. The same deletion problem was confirmed for other files, see DOC and PDF as examples. Considering that procession is 0 sec, it seems that no plugins were run and the optimizer just deleted the files. I think the optimizer have to check the final file size and if it is zero, restore original version.
Before I found the problem I processed several big archives. Some files could be restored from recycling, however it seems that about 5 % of the files are lost.
Indeed this is what FO is doing. Unless the plugin returns OK, optimized file is not keep. Can you attach a sample of file, please?
Sure I can share. But it seems that the problem is not repeatable. Next time the FO optimizes those files. I suggest you to check the presence of the file after running plugins. It will not fix the source of problem, but prevent losing of files as it happened to me.
P.S. If it helps, I used my archive HDD connected via USB.
It is hard to catch. I run the optimization process over those three files 10 times and never happened. But as you say, it is only happening sometimes.
The thing is that the checks in FO are even more exhaustive than you suggest, if you take a look to it you will see: https://sourceforge.net/p/nikkhokkho/code/HEAD/tree/trunk/FileOptimizer/Source/cppMain.cpp#l3047
1) Plugin execution is checked for being correct.
2) If it is the optimize size is checked. It required fille to exists. It should be greater than 8 bytes and less than the original one.
3) If so it is replaced, and if no, nothing happens.
Are you running it over a removable drive or something that could make files not being written?
Thank you for the explanations, but problem still exists and its critical.
I like your program, because it significantly simplifies optimization
process, combing advantages of many plugins. However, this problem makes
its usage very risky. File deletion typically happens, when processing big
number of files, so it is hard to detect that several files were deleted. I
hope you can find the exact problem.
I want to pay your attention that the processing time of the deleted files
is 0sec, so I can guess, that plugins were not properly executed, or were
not executed at all. Are there possible synchronization problems,
considering that I detected this problem, when using USB HDD? Maybe delayed
file operations? Try to complete them before going next.
I also recommend at the end of the work to check the existence of all files
in the list.
P.S. It could be fantastic, if you add option to clear metadata of many
files, where pdf and jpg are the most important.
P.P.S. I found that the FO deletes track information in mp3. Is it correct
behavior? Is it possible to turn it off?
P.P.S. If the file is corrupted or encrypted, the plugin may hang. I faced
with this problem in pdf files, which were encrypted by the company
security system. Could you check the consistency/readability of the files
before running plugins?
Sincerely,
Anton
On Thu, Aug 22, 2024, 14:17 Nikkho nikkho@users.sourceforge.net wrote:
I will add to my todo listo to debug and try to check possible failured during the invocation to the plugins.
Regarding the option of clearing metadata it is not the scope of FileOptimizer, but I will not discard creating a new tool for bulk removing it in several file types.
For MP3 processing I am using MP3Packer and if "Copy metadata" is enabled FO will pass it the -t and -s flags which should not stripg metadata according to the plugin documentation:
-t Strip non-MP3 data at the beginning (mainly ID3v2 tags)
-s Strip non-MP3 data at the end (mainly tags)
If not, it would be better to report it to MP3Packer developers.
As for encryption, I did some quick tests to overcome the problem. Essentially I check if the reduction is excesive and if so, discard it. Of course this is a task of the plugin itself.
Thanks.
One more consideration. Your program reports the final size of the deleted file as 0%, so it knows that the file was deleted. You may check the presence of file in the code, where 0% was obtained, and if it is zero, restore from the recycling.
Regarding mp3 metadata. I did not modify any setting. I checked and found that the checkbox to copy data is set. However, in the optimized files, there were no any D3v2 tags. Could you check that the plugin is called properly? If yes, I can report to developers.
Regarding encryption. I agree that it is plugin problem, but its affects usage of FO. Can you perform more checks of the file content? For example, our company security soft encrypts files and they begin from “<## NASCA DRM FILE - VER1.00 ##>”, so it can be easily detected that the pdf or docx file is encrypted.
P.S. If you create a good tool, for metadata deletion, it could be a great job. As far as I searched, there is no a good program for such simple action. The best I found was "ExifCleaner", however it still has some issues.
Have a nice day,
Anton
Have checked MP3 processing and it should not strip metadata. According to the log file it executes mp3packer without -t -s
Start: 2024/08/22 17:34:48 End: 2024/08/22 17:34:49 Level: 9 Original: 5414467 Optimized: 5350569 Errorlevel: 0 Input: C:\Users\JavierG\Desktop\MP3\Lennox Annie - (10) - It
S Alright , BabyS Coming.mp3 Output: Plugin: MP3packer (1/1) Commandline: D:\PROYECTOS\FileOptimizer\Plugins64\mp3packer.exe -z -a "" -A -f "C:\Users\JavierG\Desktop\MP3\Lennox Annie - (10) - ItS Alright , BabyS Coming.mp3" "C:\Users\JavierG\AppData\Local\Temp\FileOptimizer_Output_2794_Lennox Annie - (10) - ItS Alright , BabyS Coming.mp3"You can enable log level too, and double check it is correct.
Regarding encription, this mechanism will only detect DRM by NASCA, but not other encryption such as the built in PDF encruption.