CSR matrix to compressed_matrix transfer creates memory allocation error

[tario]

Hello Karli,

I have still problem with the data transfer from Foam to ViennaCL. I am now converting the Foam LDU matrix to CSR, the format the ViennaCL compressed_matrix uses but I get a memory allocation error, when I run a simulation. I am now using the .set() functionality instead of a loop.

Here the first lines of the error messages:

*** Error in `simpleFoam': malloc(): memory corruption: 0x00000000036b8a50 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fca161337e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7fca1613e13e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fca16140184]
/opt/openfoam5/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(malloc+0x25)[0x7fca17684455]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x18)[0x7fca16a32e78]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znam+0x9)[0x7fca16a32f19]
/home/tario/OpenFOAM/tario-5.0/platforms/linux64GccDPInt32Opt/lib/libCLFoam.so(_ZN8viennacl7backend13memory_createERNS0_10mem_handleEmRKNS_7contextEPKv+0x59)[0x7fca101636d9]
/home/tario/OpenFOAM/tario-5.0/platforms/linux64GccDPInt32Opt/lib/libCLFoam.so(_ZN6clFoam7csr2vclERKN4Foam9lduMatrixERN8viennacl17compressed_matrixIdLj1EEE+0xf9)[0x7fca1015dae9]
/home/tario/OpenFOAM/tario-5.0/platforms/linux64GccDPInt32Opt/lib/libCLFoam.so(_ZN6clFoam22clPBiCGStab_solve_FoamERKN4Foam9lduMatrixERNS0_5FieldIdEERKS5_iddRNS0_17SolverPerformanceIdEERKNS0_8UPtrListIKNS0_17lduInterfaceFieldEEEh+0x1ad)[0x7fca1017560d]
/home/tario/OpenFOAM/tario-5.0/platforms/linux64GccDPInt32Opt/lib/libCLFoam.so(_ZNK4Foam11clPBiCGStab5solveERNS_5FieldIdEERKS2_h+0x2e4)[0x7fca1015cd14]
simpleFoam[0x45dfd3]
simpleFoam[0x46ba5a]
simpleFoam[0x423d17]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fca160dc830]
simpleFoam[0x426fe9]
======= Memory map: ========
00400000-0049a000 r-xp 00000000 08:01 920647                             /opt/openfoam5/platforms/linux64GccDPInt32Opt/bin/simpleFoam
0069a000-0069c000 r--p 0009a000 08:01 920647                             /opt/openfoam5/platforms/linux64GccDPInt32Opt/bin/simpleFoam
0069c000-0069d000 rw-p 0009c000 08:01 920647                             /opt/openfoam5/platforms/linux64GccDPInt32Opt/bin/simpleFoam
01f92000-045eb000 rw-p 00000000 00:00 0                                  [heap]
7fca04000000-7fca04021000 rw-p 00000000 00:00 0 
7fca04021000-7fca08000000 ---p 00000000 00:00 0 
7fca0b3cc000-7fca0b3cd000 ---p 00000000 00:00 0 
7fca0b3cd000-7fca0bbcd000 rw-p 00000000 00:00 0 
7fca0bbcd000-7fca0bbce000 ---p 00000000 00:00 0 
7fca0bbce000-7fca0c3ce000 rw-p 00000000 00:00 0 
7fca0c3ce000-7fca0c3cf000 ---p 00000000 00:00 0 
7fca0c3cf000-7fca0cbcf000 rw-p 00000000 00:00 0 

And these are my new import functions:

void ldu2csr
(
    const Foam::lduMatrix & matrix, 
    uint* cols, 
    uint* rows,
    ScalarType* vals
) 
{
    int n_rows = matrix.diag().size();

    //
    //  Calculate each row size. Sizes are shifted, because in the next part
    //  array rows is modified twice.
    //
    for (int i=0; i < matrix.upper().size() ; i++) 
    {
    int ri1 = matrix.lduAddr().lowerAddr()[i]; 
    int ri2 = matrix.lduAddr().upperAddr()[i];
    rows[ri1+2] ++;
        rows[ri2+2] ++;
    };

    //
    //  Compute row offsets. Offsets are shifted by one positions, 
    //  because they are used as START positions while filling values.
    //
    rows[0] = 0 ;
    rows[1] = 0 ;
    for(int i=1; i<n_rows; i++) 
    {
    rows[i+1] += rows[i];
    };

    //
    //  Fill in CSR matrix.
    //  Order below is important to keep column indices sorted.
    //

    // fill/convert lower triangle into CSR format
    for (int i=0; i < matrix.lower().size() ; i++) 
    {
    int row    = matrix.lduAddr().upperAddr()[i] +1;
    int column = matrix.lduAddr().lowerAddr()[i];

    int idx = rows[row];
    vals[idx] = matrix.lower()[i];
    cols[idx] = column;
    rows[row]++;
    };

    // fill/convert diagonal into CSR format
    for (int i=0; i<matrix.diag().size(); i++) 
    {
    int idx = rows[i+1];
    vals[idx] = matrix.diag()[i];
    cols[idx] = i; // i is row and column index
    rows[i+1]++;
    };

    // fill/convert upper triangle into CSR format
    for (int i=0; i < matrix.upper().size() ; i++) 
    {
    int row    = matrix.lduAddr().lowerAddr()[i] +1;
    int column = matrix.lduAddr().upperAddr()[i];

    int idx = rows[row];
    vals[idx] = matrix.upper()[i];
    cols[idx] = column;
    rows[row]++;
    };
}

void csr2vcl
(
    const Foam::lduMatrix & matrix, 
    viennacl::compressed_matrix<ScalarType> & vcl_matrix
)
{
    uint n = matrix.diag().size() ;
    uint nnz = matrix.lower().size() + matrix.upper().size() + matrix.diag().size() ;

    // allocate memory for CSR sparse matrix 
    ScalarType * vals = (ScalarType *)calloc(nnz, sizeof(ScalarType));
    uint * cols = (uint *)calloc(nnz, sizeof(uint)); // error here wih size of row_jumber, col_buffer...?
    uint * rows = (uint *)calloc(n + 1, sizeof(uint)); // error here wih size of row_jumber, col_buffer...?

    clFoam::ldu2csr(matrix, rows,cols,vals);
    vcl_matrix.set(rows,cols,vals,n+1,nnz,nnz);   // error here wih size of row_jumber, col_buffer...?

    // free and release the matrix memory
    free(rows); free(cols); free(vals);  //colloc() 
}

Can you spot the error causing the memory allocation problem?

T.

[tario]

An update:

I changed memory allocation to malloc, as this is apparently Foam style:

Now the function looks like that, memory allocation seems ok but another issue appears see further down:

void csr2vcl
(
    const Foam::lduMatrix & matrix, 
    viennacl::compressed_matrix<ScalarType> & vcl_matrix
)
{
    uint n = matrix.diag().size() ;
    uint nnz = matrix.lower().size() + matrix.upper().size() + matrix.diag().size() ;

    // allocate memory for CSR sparse matrix 
    ScalarType * vals = (ScalarType *)malloc(sizeof(ScalarType) * nnz);
    uint * cols = (uint *)malloc(sizeof(uint) * nnz); 
    uint * rows = (uint *)malloc(sizeof(uint) * (n+1));

    // Different options of matrix allocation, here for 10 values
    // int *ptr = (int *) malloc(sizeof(int) * 10 );
    // int *ptr = (int *) calloc(10, (sizeof(int)));

    clFoam::ldu2csr(matrix,rows,cols,vals);
    vcl_matrix.set(rows,cols,vals,n+1,nnz,nnz);   // error here wih size of row_jumber, col_buffer...?

    // free and release the matrix memory
    free(rows); free(cols); free(vals);  //colloc() 
}

New error:

#0  Foam::error::printStack(Foam::Ostream&) at ??:?
#1  Foam::sigSegv::sigHandler(int) at ??:?
#2  ? in "/lib/x86_64-linux-gnu/libc.so.6"
#3  clFoam::ldu2csr(Foam::lduMatrix const&, unsigned int*, unsigned int*, double*) at ??:?
#4  clFoam::csr2vcl(Foam::lduMatrix const&, viennacl::compressed_matrix<double, 1u>&) at ??:?
#5  clFoam::clPBiCGStab_solve_Foam(Foam::lduMatrix const&, Foam::Field<double>&, Foam::Field<double> const&, int, double, double, Foam::SolverPerformance<double>&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char) at ??:?
#6  Foam::clPBiCGStab::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const at ??:?
#7  ? at ??:?
#8  ? at ??:?
#9  ? at ??:?
#10  __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
#11  ? at ??:?
Segmentation fault (core dumped)
Mo 4. Sep 23:53:19 CEST 2017   END running simpleFoam

[tario]

I managed to start one case which usually runs in a few seconds on one core and has very low memory requirements, well be low 1Gb.

When I ran my application with ViennaCL, the memory usage jumped to 32gb + 18.5Gb swap, while core utilization was low across all 8 threads so the openMP backend seems to work.

=> there's a hugh memory leak which I can't find.

[Karl Rupp]

Hi tario,
please run your code through valgrind in order to locate memory leaks. Your call to the .set()-member should be

vcl_matrix.set(rows,cols,vals,n,n,nnz);

Best regards,
Karli