Menu
▾
▴
1 Attachments
#5244 Webmin consume memory and process of cpu
While we were downloading a 300 Megabytes file from the file manager, the webmin started consuming the processing average up to 8 and the memory at 3GB, causing the server not to process correctly.
The only option to solve the incident was to completely stop the webmin service, after which the average and the memory will return to normal values.
In the logs I can not find a reason for the failure.
Related
Were you uploading the file to the Webmin server, or downloading from it?
Download from the server.
Last edit: Ilia 2019-03-15
To provide more information about this case, on that same server where we have a 60G partition of type ext4; that later we had to convert to btrfs, because it contains hundreds of thousands of files with very small sizes (12K to 32K).
When we search or try to explore the repository and we have to work with many sub-levels or tabs through the File Manager, it also gave us the case that the average was raised to values 25 or more and the memory was completely consumed by the service of webmin.
As always the only way to solve is to completely stop the service and in the worst case reboot the server.
Note: it happened twice as much ext4 as with btrfs.
Note: Ubuntu Server 14.04 and 18.04
Last edit: Andrés González Tenorio 2019-03-15
I think, I see the cause of this issue. File Manager in Authentic Theme, at current state, reads directory completely, without using any paginations. It's a big draw back, when used with large amount of files in directory.
I am planning to add an option to File Manager, that would use server-side pagination, if needed.
At first, it was avoided, in favour of having instant search on the table, containing all files in directory. It's great still but makes things broken on folders containing thousands of files.
Andrés, for now, can you double test, if you face the same issue using un-themed version of File Manager, like in Gray Theme? I am asking this, as you initially mentioned a problem was caused upon downloads.
Andrés - do you have a single directory containing 100k files, or 100k across the whole filesystem?
The file repository is a BTRFS partition of 50G.
It contains a total of 1038090 of directories and 3097870 of files, each directory contains at least 3 files.
Note files and directories increase daily
The files vary in size between 2K and 50K.
The name of the directories is 50 numeric characters, example 99931121800310136180100100003030000000004134913251
An example of the names of the files that are in each directory are:
99931121800310136180100100003030000000004134913251.xml
99931121800310136180100100003030000000004134913251-signed.xml
99931121800310136180100100003030000000004134913251-answer.xml
Estructure example
/repository/
├── root
│ ├── 999
│ │ ├── 99931121800310136180100100003030000000004134913251
│ │ │ ├── 99931121800310136180100100003030000000004134913251.xml
│ │ │ ├── 99931121800310136180100100003030000000004134913251-signed.xml
│ │ │ ├── 99931121800310136180100100003030000000004134913251-answer.xml
Last edit: Andrés González Tenorio 2019-03-19
What's the largest number of files or subdirectories in a single directory?
I will give information of some of the directories with more files:
Last edit: Andrés González Tenorio 2019-03-20
Andrés, when you try accessing a directory with ~20.000 files, does it even load? Which theme you have been using?
The theme we used is Authentic Theme
Yes, we have been able to enter directories that have more than 20000 subdirectories. But as we search subdirectories and files, or return to the root and look for other subdirectories, the performance is getting worse to such an extent that the webmin had to be closed. And in the worst moments, stop the service completely.
For the past few days I have been reworking File Manager under Authentic Theme, so it could read and display properly directories with large amount of files. I should finish it later today. I will let you know when it's done, so you could try it out.
I have added support for listing 1m+ files with ease. Install latest development theme version to give it a try.
I have found another possible cause that could cause server going out of memory. Latest commits to Authentic Theme would fix that.
Please try latest dev version.
Hi
I goin to try it.
Thank you.
El mié., 27 de mar. de 2019 5:36 a.m., Ilia Rostovtsev rostovtsev@users.sourceforge.net escribió:
Related
Bugs: #5244