While adding >64k files (complex big archive) let phex 3.2.106 slow down and consumes 90% of a dualcore athlon64 CPU and far to much memory.
If I keep under 30k files, it all works fine.
Each shared file consumes a little memory this is not avoidable, we already tried optimized the amount of memory used a while ago and there might be some bytes here or there left to optimize. But somewhere will be a limit..
Currently a shared file with a path length of about 200 character will consume at least 450 bytes memory. A directory that contains a shared file will consume at least 350 bytes memory. Additionally comes the search tree which will also consume about 400 bytes per file and the query routing information which will use 2MB for a high number of files. All this alone will make up at least 60MB RAM and it will likely not yet include everything. So lets say you can calculate with around 1MB of RAM needed per 1000 shared files.
Now lets come to the CPU problem. When you first share the files Phex will need to hash all the files. This will take some time and uses CPU and high IO activity. If you share that many files it could take a few hours until every file is hashed. Specially if the files are large.
If the hashing is finished the CPU activity will go down. There might still be higher CPU usage though, because of the very large search keyword tree that every query will look through when trying to find files.
I will do some benchmarking, but at least what I can say is that Phex is usually not very CPU intensive. So if your RAM and CPU is not able to handle this high amount of files I would recommend sharing less files. AFAIK there are other popular file sharing programs that limit shared files, I don't think this is the better alternative.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The problem here is that the CPU usage is high when one has many shared files (I also experienced the problem). More exactly: It seems to keep low until a certain threshhold of files is reached. When the number of files exceed that threshhold the CPU suddenly leaps very high.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Each shared file consumes a little memory this is not avoidable, we already tried optimized the amount of memory used a while ago and there might be some bytes here or there left to optimize. But somewhere will be a limit..
Currently a shared file with a path length of about 200 character will consume at least 450 bytes memory. A directory that contains a shared file will consume at least 350 bytes memory. Additionally comes the search tree which will also consume about 400 bytes per file and the query routing information which will use 2MB for a high number of files. All this alone will make up at least 60MB RAM and it will likely not yet include everything. So lets say you can calculate with around 1MB of RAM needed per 1000 shared files.
Now lets come to the CPU problem. When you first share the files Phex will need to hash all the files. This will take some time and uses CPU and high IO activity. If you share that many files it could take a few hours until every file is hashed. Specially if the files are large.
If the hashing is finished the CPU activity will go down. There might still be higher CPU usage though, because of the very large search keyword tree that every query will look through when trying to find files.
I will do some benchmarking, but at least what I can say is that Phex is usually not very CPU intensive. So if your RAM and CPU is not able to handle this high amount of files I would recommend sharing less files. AFAIK there are other popular file sharing programs that limit shared files, I don't think this is the better alternative.
This bug was also spotted in i2phex:
- second post of http://forum.i2p2.de/viewtopic.php?t=3513
The problem here is that the CPU usage is high when one has many shared files (I also experienced the problem). More exactly: It seems to keep low until a certain threshhold of files is reached. When the number of files exceed that threshhold the CPU suddenly leaps very high.