Too many opened file error
Brought to you by:
r_x
Menu
▾
▴
#75 Too many opened file error
I try use Mobile Atlas Creator 1.8 alpha 3 with turaterkep mapsource to convert Trekbuddy tarred atlas. The tiles converted to 256x256pixel 8bit png image. I use UHU-Linux 2.1. I use previously downloaded tiles. Now i set the max. tile age to 1 years. The conversion run offline (all tiles exist, my internet resource limited).
The 7-12 numbered map conversion ok. The 13 numbered map failed with "too many opened file error". See some tracking data, and the error log bellow.
I see the the conversion window, and i check and tracking the running process:
Second progress bar active: Downloading tiles for map number 12 opened file number very quickly increased, but bellow 1024.
Third progress bar active: Map creation, number of opened files lower, average 65.
Second progress bar active: Downloading tiles for map number 13 opened file number very quickly increased.
The opened files is 1024, the error messages displayed in MOBAC. The conversion failed. I see cat /proc/sys/fs/file-max
limit is not problem. But the user space file limit is equal (1024).
cat /proc/sys/fs/file-max
192348
ulimit -n
1024
ps -C "sh start.sh" -o pid=
2486
ps -C "java -Xms64m -Xmx512M -jar Mobil" -o pid=
2487
Before running conversion, MOBAC running:
ls -l /proc/2486/fd | wc -l
5
ls -l /proc/2487/fd | wc -l
58
cat /proc/sys/fs/file-nr
3776 0 192348
Atlas creating, after first error occured:
ls -l /proc/2487/fd | wc -l
1024
cat /proc/sys/fs/file-nr
4544 0 192348
Version: Mobile Atlas Creator 1.8 alpha 3 (1086)
Platform: Linux (2.6.23.9-1) (gnome)
Distribution name: 2.1 (Bumm)
Java VM: Java HotSpot(TM) Client VM (1.6.0_18-b07)
Mapsources rev: 1085
Thread: AtlasThread 1
Error hierarchy:
MapCreationException: java.io.FileNotFoundException: /home/orion/MOBAC/atlases/H_2010-03-21_231421/H/H 13 (0)/H 13 (0).map (Túl sok nyitott fájl)
FileNotFoundException: /home/orion/MOBAC/atlases/H_2010-03-21_231421/H/H 13 (0)/H 13 (0).map (Túl sok nyitott fájl)
mobac.exceptions.MapCreationException: java.io.FileNotFoundException: /home/orion/MOBAC/atlases/H_2010-03-21_231421/H/H 13 (0)/H 13 (0).map (Túl sok nyitott fájl)
at mobac.program.atlascreators.TrekBuddyCustom.createMap(TrekBuddyCustom.java:54)
at mobac.program.AtlasThread.createMap(AtlasThread.java:299)
at mobac.program.AtlasThread.createAtlas(AtlasThread.java:151)
at mobac.program.AtlasThread.run(AtlasThread.java:85)
Caused by: java.io.FileNotFoundException: /home/orion/MOBAC/atlases/H_2010-03-21_231421/H/H 13 (0)/H 13 (0).map (Túl sok nyitott fájl)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at mobac.program.atlascreators.TrekBuddy.writeMapFile(TrekBuddy.java:70)
at mobac.program.atlascreators.TrekBuddyCustom.createMap(TrekBuddyCustom.java:33)
... 3 more
As I wrote in the forum I need the list of open files for the Java process. The total number itself is not enough.
User the command "pfiles {PID}" and attach the list to this tracker item or send it to me via mail so that I can see what part of MOBAC causes the problem.
psof -a -p 2077 > before.txt
psof -a -p 2077 > after.txt
ls -lat /proc/2077/fd > after_proc.txt
pfiles not exist on my system (UHU-Linux 2.1). I try searhing source code, but no succes.
Some forum suggest swap pfiles to lsof -a -p <pid>
List files attached.
ps -C "java -Xms64m -Xmx512M -jar Mobil" -o pid=
2077
psof -a -p 2077 > before.txt
Too many opened files error occured
psof -a -p 2077 > after.txt
ls -lat /proc/2077/fd > after_proc.txt
OK, the provided files show that the error text "too many open files" is very misleading - not files but network sockets are meant.
I remember that there was another Linux user who observed many unclosed connections in his DSL-Router when using MOBAC. Looks like on some Linux systems Java has problems closing open TCP connections.
I not have DSL-Router or any router. I think the main problem the download error handling. If the network is ok, and the tile exist on the server all ok. The number of opened network socket small.
If network is down, lots of network socket not closed, and the MOBAC go to "too many opened files error" if the limit(uname -n) occured.
If the network is ok, but tiles not exist on server (in turaterkeo mapsource is normal, empty tiles not in server), the some or more network socket not closed. The number of unclosed network socket lower than network is down. I think, this is depend the number of not exist tiles.
I think now understand why the error "randomly" occured. If the download ok, i think the not properly closed network socket closed ok when converting the tiles. In conversion the number of opened files is small. After the conversion ok, when downloading the next map tiles, the number of opened network socket increase again.
Sorry, uname -n = ulimit -n
The current version of MOBAC (1.8 alpha4) i think the Hungarian map source is disabled? If i want to processing to Trekbuddy atlas format or Oziexplorer i get error message the output format not compatible with this map source.
Correct, a small bug - corrected in alpha 4a
1.8 alpha5a error log
Thanks the update. Now the Trekbuddy tarred output working with MOBAC 1.8 alpha5a. But, MOBAC crashing and exit vhen conversion is running. See the attached error log. I try again the conversion, and crash again. I monitoring the running process, The opened files not too high. I don't know what is wrong.
hs_err_pid3187.log
I try the 1.8alpha 8b the crash problem is same. Any idea to fix this error. See the attached error log: hs_err_pid3187.log
I am sorry, but as far as I know MOBAC doesn't do anything "bad" that could cause the crash. I assume it is an incompatibility of the JRE with your environment (which you could also call a Bug of the JRE).
A Java program that crashes the JRE as it happen on your system should be theoretically impossible. But it happens and therefore the problem is located in the JVM not in MOBAC.