The VXL Project

William A. Hoffman wrote:
> Hi,
> 
> Does anyone have any thoughts or ideas on how to make the netlib stuff thread safe.
> This subject has been on the list before, but I thought there might be some new ideas
> to fix the problem.

I had thought that the best start would be to use a recent version of 
LAPACK which avoid some of the old fortran constructs (used by the 
LINPACK code we currently use) that translate to static variables.

Ian.

At 03:07 PM 6/13/2006, Ian Scott wrote:
>William A. Hoffman wrote:
>>Hi,
>>Does anyone have any thoughts or ideas on how to make the netlib stuff thread safe.
>>This subject has been on the list before, but I thought there might be some new ideas
>>to fix the problem.
>
>I had thought that the best start would be to use a recent version of LAPACK which avoid some of the old fortran constructs (used by the LINPACK code we currently use) that translate to static variables.
This is an interesting page on the topic:

http://www.cisst.org/cnetlib/cvstrac/cvstrac.cgi/wiki?p=CisstNetlibIntroduction

It sounds like they did not find a solution.   They are claiming that LAPACK is
not thread safe either.

-Bill

I hadn't realised that f2c had never been updated to FORTRAN-90 - Bummer

We could compile LAPACK 3e for a number of platforms using the compiler 
at g95.com and distribute binaries in v3p. If we also included the 
LAPACK 3e source, some people could do it themselves. Any one else could 
use the existing non-thread-safe version of LINPACK.

Alternatively we could get the best version of CLAPCK we can find, and 
manually remove all the static variables ourselves.

Either sounds like a lot of work.

Ian.

William A. Hoffman wrote:
> At 03:07 PM 6/13/2006, Ian Scott wrote:
>> William A. Hoffman wrote:
>>> Hi,
>>> Does anyone have any thoughts or ideas on how to make the netlib stuff thread safe.
>>> This subject has been on the list before, but I thought there might be some new ideas
>>> to fix the problem.
>> I had thought that the best start would be to use a recent version of LAPACK which avoid some of the old fortran constructs (used by the LINPACK code we currently use) that translate to static variables.
> This is an interesting page on the topic:
> 
> http://www.cisst.org/cnetlib/cvstrac/cvstrac.cgi/wiki?p=CisstNetlibIntroduction
> 
> It sounds like they did not find a solution.   They are claiming that LAPACK is
> not thread safe either.
> 
> -Bill
>

At 10:54 AM 6/14/2006, Ian Scott wrote:
>I hadn't realised that f2c had never been updated to FORTRAN-90 - Bummer
>
>We could compile LAPACK 3e for a number of platforms using the compiler 
>at g95.com and distribute binaries in v3p. If we also included the 
>LAPACK 3e source, some people could do it themselves. Any one else could 
>use the existing non-thread-safe version of LINPACK.
>
>Alternatively we could get the best version of CLAPCK we can find, and 
>manually remove all the static variables ourselves.
>
>Either sounds like a lot of work.

How much of netlib is actually used by vnl?   Is there an easy way to tell?
One idea I had was to make each of the netlib.c files a c++ class, and make
all the statics members of the class.   You might be able to automate that
with a perl script or something and get it 90% of the way.

-Bill

William A. Hoffman wrote:

> How much of netlib is actually used by vnl?   Is there an easy way to tell?

I've tried running a Dart coverage, with the .NoDartCoverage files 
removed from v3p/netlib - but whether the dashboard is taking a long 
time to update (20 minutes so far) or the submission hasn't worked - I 
don't know. If nothing happens, you can trying running a manual coverage 
test, and see which bits of code aren't currently tested.

If you add the all the complex, and float versions of the covered files, 
that would be a good estimate. It wouldn't surprise me if a file was 
only added to v3p/netlib, because some code in vnl_algo actually needs it.

A completely alternative solution, that I had been planning* to use for 
some of our private code's mutable data members is to write a thread 
local storage class template. A netlib_tls<int> would guarantee to give 
you a separate int in each thread. Importing something from boost might 
be a good start to write such a class. Then simply replace each static 
double with a netlib_tls<double>. You could even write something in C, 
which is currently used by almost all of netlib, but it might be harder 
to script the replacement of all the static variables then.

Ian.

* - In the sense of - when we really, absolutely, unavoidably, have to 
start using threads, I'll think properly about how to do this.

At 12:10 PM 6/14/2006, Ian Scott wrote:
>William A. Hoffman wrote:
>
>>How much of netlib is actually used by vnl?   Is there an easy way to tell?
>
>I've tried running a Dart coverage, with the .NoDartCoverage files removed from v3p/netlib - but whether the dashboard is taking a long time to update (20 minutes so far) or the submission hasn't worked - I don't know. If nothing happens, you can trying running a manual coverage test, and see which bits of code aren't currently tested.
>
>If you add the all the complex, and float versions of the covered files, that would be a good estimate. It wouldn't surprise me if a file was only added to v3p/netlib, because some code in vnl_algo actually needs it.
>
>A completely alternative solution, that I had been planning* to use for some of our private code's mutable data members is to write a thread local storage class template. A netlib_tls<int> would guarantee to give you a separate int in each thread. Importing something from boost might be a good start to write such a class. Then simply replace each static double with a netlib_tls<double>. You could even write something in C, which is currently used by almost all of netlib, but it might be harder to script the replacement of all the static variables then.

You might be able to modify f2c to output the correct types.


>Ian.
>
>* - In the sense of - when we really, absolutely, unavoidably, have to start using threads, I'll think properly about how to do this.

Well, we do have some funding to work on this problem.   So, if you have more ideas, please
send them our way.  Also, it seems that cpu's are now mostly dual core and parallel software 
is becoming a must to take advantage of the hardware.

-Bill

I've also noticed that f2c has a "-a" option that changes the default
storage type from "static" to "auto"; this may get rid of a number of
the static variables. However, I noticed that the fortran code has
common blocks, so static variables may be unavoidable. I didn't look
to see what the common variables were for, and whether they'd have any
effect on threading.

Amitha.

> It wouldn't surprise me if a file was only added to v3p/netlib,
> because some code in vnl_algo actually needs it.

That's indeed the case.

>From what I remember when I was digging into the v3p/netlib code,
removing statics and common blocks manually would be very difficult, if
not virtually impossible. Additional proof being the apparent failure
to do so in LAPACK.

Ian's suggestion with templated data types (to replace all int, float,
double in v3p/netlib) sounds like a good idea to me.

--	Peter.

Amitha Perera wrote:
> I've also noticed that f2c has a "-a" option that changes the default
> storage type from "static" to "auto"; this may get rid of a number of
> the static variables. However, I noticed that the fortran code has
> common blocks, so static variables may be unavoidable. I didn't look
> to see what the common variables were for, and whether they'd have any
> effect on threading.

I tried converting a few .f files from linpack with this command:

   f2c -A -a -C++ -c -E file.f

The resulting code looks much more thread friendly.  I'm considering 
converting all of linpack with these options and updating v3p/netlib. 
Any remaining global storage that is not constant could be fixed with 
the thread-local-storage template suggestion.

I notice that netlib.h has all the prototypes for netlib functions.  Was 
there a process used to generate all the .c files and netlib.h the first 
time?

Thanks,
-Brad

> Was there a process used to generate all the .c files and netlib.h
the
> first time?

netlib.h was generated by hand.

--	Peter.

Brad King wrote:
> Amitha Perera wrote:
>> I've also noticed that f2c has a "-a" option that changes the default
>> storage type from "static" to "auto"; this may get rid of a number of
>> the static variables. However, I noticed that the fortran code has
>> common blocks, so static variables may be unavoidable. I didn't look
>> to see what the common variables were for, and whether they'd have any
>> effect on threading.
> 
> I tried converting a few .f files from linpack with this command:
> 
>    f2c -A -a -C++ -c -E file.f
> 
> The resulting code looks much more thread friendly.  I'm considering 
> converting all of linpack with these options and updating v3p/netlib. 

Converting with the command

f2c -A -a -C++ -c -E -P

seems to generate thread-friendly code and a separate .P file with a 
prototype for each function.

I wrote a program that uses vnl_svd in multiple threads at once.  The 
program gets wrong answers and/or crashes if more than one thread is 
used with the current netlib.  Then I downloaded the latest [cdsz]svdc.f 
source files and their blas dependencies from netlib.org.  I converted 
them to C using the above command, and integrated them into v3p using 
the procedure I propose below.  After updating vnl_svd to use the new 
netlib code the multi-threaded program works fine.

It appears this approach will work for most cases.  Therefore I propose 
the following transition scheme to convert incrementally to a 
thread-safe netlib:

1.) Put the latest version of libf2c sources in v3p/netlib/libf2c.

2.) Put the latest version of blas and linpack .f files and their 
converted .c files in v3p/netlib/blas and v3p/netlib/linpack.

3.) Create a v3p_netlib library that contains the new code.

4.) Use a v3p_netlib_mangle.h header that contains lines like

   #define dsvdc_ v3p_netlib_dsvdc_

in order to mangle all the types and functions in the new netlib code to 
have v3p_netlib_ prefixes.

5.) Convert the .P files generated by f2c to .h files with all the 
symbols converted to their mangled names.  Generate a 
v3p_netlib_prototypes.h header that includes all of them.

6.) Manually add v3p_netlib_const in places where const-ness is needed 
in the API when calling from vnl.  This macro will be defined to "const" 
when building code outside the v3p_netlib code and defined to empty 
inside the code.  It avoids the need to sweep const-correctness through 
the converted code while still enforcing const-ness in the API.

7.) Create a v3p_netlib.h header that packages all the above together in 
a single API encapsulated in the prefix-namespace "v3p_netlib_".

I have these steps done in a working checkout of vxl.  Only the svd code 
has been downloaded from linpack so far.  All the vnl tests pass, and 
vnl_svd is now thread-safe.

If there is no disagreement I plan to commit these changes and then 
incrementally sweep through the rest of the linpack routines needed by vnl.

Comments?
-Brad

Great!
Please go ahead ...

--	Peter.

Brad,

> I have these steps done in a working checkout of vxl.  Only the svd code 
> has been downloaded from linpack so far.  All the vnl tests pass, and 
> vnl_svd is now thread-safe.

Would it be possible to move to LAPACK at the same time? I believe that 
it is significantly more numerically accurate and faster than linpack? I 
don't know how much more work it would be, but I suspect it wouldn't be 
trivial.

Either way, fixing the thread safety of v3p/netlib is an important and 
overdue improvement.

Ian.