#155 Number crunching with Intel MKLibrary runs slow on openMosix
Hello
I am using a heavey number crunching program compiled
with Intel fortran compiler 7.2 and Intel Math Kernel
Library 7.0.
1. When the program is run on the home node (runhome),
it runs at 100% processor usage until it finishes, as
it should.
2. When the program is run a different node (i.e. a
node which does not have the /home/ partition mounted
locally) it runs at low CPU usage (between 16% and 60%,
sometimes 99%) instead of running 99.9 % throughout.
The program then takes correspondingly longer to
complete on the other nodes. There are no other
programs running on these nodes which could influence
performance.
Any suggestions would be welcome.
Notes:
1. A program run takes several hours. It uses the
BLAS/LAPACK components of IntelMKL; matrix
diagonalization, etc is the main task (Density
Functional Theory).
2.1. Input: 3 files: two <10k and one of ~300k
2.2. Output: 8 files, four are < 50k, one file of 5.5
MB and two files of 10 MB.
3. Several instances of these programs are run
simultaneously (one per processor), sharing some of the
input files but writing their own output.
4. A similar program compiled with gfc without MKL
(which makes the program slower), and which uses a
single small input file and a single output file of ~1
MB, always runs at 99.9% on any node.
Thank you in advance
Regards
ELom
Logged In: YES
user_id=1196698
Does you code migrate at all?
I am using Intel8.1 compiler and any attempt to use
migration ends up
with killed process (lost connection with the node).
If I run it with nomig or on dedicted node ( mosrun -# -F ),
no problem.
Is it possible to have Intel compiled code migrating at all?
Regards,
Tomasz
Logged In: YES
user_id=1021609
Elom-
You may be having I/O issues slow things down. Try using
the mosrun options to biase the cluster towards CPU instead
of I/O operations.
Logged In: NO
Hello
Thanks for your responses
1. I am using mosrun option -c to bias towards CPU.
Unfortunately the program is running slowly with this option
enabled.
2. About migration: I generally do not let this program
migrate, but fix it to specific nodes (mosrun -#). I do note
that some other programs compiled with ifc do migrate.
However, with this particular program, the Intel Math Kernel
Libraries determine at runtime which library to use on a
given processor. If a job starts running on a P-IV and then
migrates to a P-III it always crashes. However, the other
way round it survives sometimes, but not always.
Another point: apparently openMosix cannot migrate threaded
programs (yet). I think this may be part of the source of
segfaults. At least the Intel MathKernelLibraries use
threads, and possibly the compiler too.
I think it should be possible to let Intel compiled programs
migrate, if you do not use threads and do not use processor
specific optimizations (except if you have identical
machines in your cluster).
(PS: I am using Intel Fortran Compiler 7.0. Other users of
this program who compile with IFC 8 rather than 7.0 have
reported problems with programs segfaulting (on plain Linux,
not openMosix, I assume), while some different programs
segfault when compiled with 7.0 but are OK with Intel
Fortran 8!)
Regards
ELom
Logged In: YES
user_id=1283799
Originator: NO
Reported on 2.4 => Moshe