Re: [OctDev] Sparse matrix assign problem From: David Bateman - 2010-12-28 14:58 ```Please keep responses on the list as the list is an invaluable source of information for people that have similar problems, and going offline deprives them of this information. w4nderlust wrote: > First thing: Thanks really much :) > then some questions: > 1) what's the difference between .cdx and .xcdx ? This is just to know. > The "x" before data in Sparse and Full matrices and ridx and cidx in Sparse matrices prevents some bounds checking. If you "know" that your indexing is correct, using xridx rather than ridx then your code will be faster, but will it will crash octave if your indexing is incorrect. > 2) how can owrking on the transpose of a help performances? > The compressed column format of Octave means that working with a Sparse matrix column by column is many times faster than working with a matrix row by row. The reason is that working by columns we can access the data sequentially in "data" and "ridx" whereas as not only can't we do that when working by rows, we have to traverse all of the elements of a column of the sparse matrix to find the offset into "data" and "ridx" for a particular row. > 3) if i run this code octave crashes... why? (using octave 3.2.4) > > >>>> SA = sprand(100, 100, 0.2); >>>> SB = sprand(100, 100, 0.2); >>>> tic; spmaxmin_ST_new(SA, SB); toc; >>>> > octave: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) > &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd))) > ) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__buil > tin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) > & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long > )old_end & pagemask) == 0)' failed. > panic: Aborted -- stopping myself... > attempting to save variables to `octave-core'... > > some other times it gives me just: > > panic: Segmentation fault -- stopping myself... > > some ideas? > Thanks > Because my assumption that the maximum number of non zeros elements in the return matrix is nnzA+nnzB is incorrect. I suggest counting the number of elements in the returned matrix first as this can be done at significantly lower cost than the full calculation, sizing the return matrix correctly and then removing the call to maybe_compress as its no longer needed. Try the attached David ```

[OctDev] Sparse matrix assign problem w4nderlust <w4nderlust@gm...>
 Re: [OctDev] Sparse matrix assign problem From: Søren Hauberg - 2010-12-27 14:42 ```man, 27 12 2010 kl. 15:37 +0100, skrev w4nderlust: > Hello everyone. > Compiling an octfile i'm experiencing a compile error. > spmaxmin_ST.cc:62: error: lvalue required as left operand of assignment > > that's how my code works: > //i create a sparse matrix > SparseMatrix c = SparseMatrix(rowsA, colsB, rowsA*colsB); > > //i create a constart temp matrix as suggested in the documentation > const SparseMatrix tmp(c); > > //somewhere in my code i do > if (mx != 0) > > tmp(i, j) = mx; > //mx is a double and it is initialized > > the problem is: > tmp(i, j) = mx; > > i don't know what's wrong because in the documentation there are a lot > of examples with this way of accessing the matrix... > Can someone please help me? I don't know the sparse matrix API that well, but it looks like you have declared a constant matrix tmp (constant means that it will not be changed during the rest of the program). You then alter this matrix (by 'tmp(i, j) = mx'), which is not possible if the matrix is constant. I don't know this trick with the constant temp matrix, but it seems like that is the source of your troubles. Søren ```

 Re: [OctDev] Sparse matrix assign problem From: Søren Hauberg - 2010-12-27 16:11 ```Hi Some practical comments: * Please keep the list CC'ed (press reply to all) instead of just sending me a private mail; that way others can help. * If you send code, please send ALL relevant code. The code you send does NOT compile (you have not included relevant headers, defined HELP or NUM_ARG and you have not declared calc_tnorm). * Please send the code as an attached file so that we don't have to copy go through the trouble of creating a new file and copying the contents of an e-mail to this file (this is relevant not only to make our lives easier, but also to keep the encoding). * Please tell us what you did to compile the code. * Please tell us what you did to run the code. Søren man, 27 12 2010 kl. 15:54 +0100, skrev w4nderlust: > I did that because it was suggested in the api manual. > Anyway, if i remove it it compiles, but when i run it in octave i get > this error: > error: Sparse::SparseRep::elem (octave_idx_type, octave_idx_type): sparse matri > x filled > > here's the full code: > > DEFUN_DLD (spmaxmin_ST, args, nargout, HELP) > > { > > if (args.length() != NUM_ARG) > > print_usage(); > > else > > { > > SparseMatrix a = args(0).sparse_matrix_value(); > > SparseMatrix b = args(1).sparse_matrix_value(); > > dim_vector dimsA = a.dims(); > > int rowsA = dimsA(0); > > int colsA = dimsA(1); > > > > dim_vector dimsB = b.dims(); > > int rowsB = dimsB(0); > > int colsB = dimsB(1); > > > > SparseMatrix c = SparseMatrix(rowsA, colsB, rowsA*colsB); > > dim_vector dimsC = c.dims(); > > int rowsC = dimsC(0); > > int colsC = dimsC(1); > > > > //const SparseMatrix tmp(c); > > int i,j,k; > > double mx,mn; > > if (colsA != rowsB) > > print_usage(); > > else > > { > > for (i = 0; i < rowsC; i++) > > { > > for(j = 0; j < colsC; j++) > > { > > mx = 0; > > for (k = b.cidx(j); k <= (b.cidx(j+1) - 1); k++) > > { > > mn = calc_tnorm(a(i, b.ridx(k)), b.data(k)); > > if (mn > mx) > > mx = mn; > > } > > if (mx != 0) > > c(i, j) = mx; > > } > > } > > c.maybe_compress(); > > return octave_value(c); > > } > > } > > } > > > > 2010/12/27 Søren Hauberg : > > man, 27 12 2010 kl. 15:37 +0100, skrev w4nderlust: > >> Hello everyone. > >> Compiling an octfile i'm experiencing a compile error. > >> spmaxmin_ST.cc:62: error: lvalue required as left operand of assignment > >> > >> that's how my code works: > >> //i create a sparse matrix > >> SparseMatrix c = SparseMatrix(rowsA, colsB, rowsA*colsB); > >> > >> //i create a constart temp matrix as suggested in the documentation > >> const SparseMatrix tmp(c); > >> > >> //somewhere in my code i do > >> if (mx != 0) > >> > >> tmp(i, j) = mx; > >> //mx is a double and it is initialized > >> > >> the problem is: > >> tmp(i, j) = mx; > >> > >> i don't know what's wrong because in the documentation there are a lot > >> of examples with this way of accessing the matrix... > >> Can someone please help me? > > > > I don't know the sparse matrix API that well, but it looks like you have > > declared a constant matrix tmp (constant means that it will not be > > changed during the rest of the program). You then alter this matrix (by > > 'tmp(i, j) = mx'), which is not possible if the matrix is constant. I > > don't know this trick with the constant temp matrix, but it seems like > > that is the source of your troubles. > > > > Søren > > > > ```

 Re: [OctDev] Sparse matrix assign problem From: w4nderlust - 2010-12-27 19:12 Attachments: spmaxmin_ST.cc ```Sorry for everything, i'm attaching the full code. to compile i simply did call: mkoctfile spmaxmin_ST.cc and to run it i reated 2 sparse matrices A and B and then: tic; spmaxmin_ST(A, B); toc; 2010/12/27 Søren Hauberg : > Hi > > Some practical comments: > >      * Please keep the list CC'ed (press reply to all) instead of just >        sending me a private mail; that way others can help. >      * If you send code, please send ALL relevant code. The code you >        send does NOT compile (you have not included relevant headers, >        defined HELP or NUM_ARG and you have not declared calc_tnorm). >      * Please send the code as an attached file so that we don't have >        to copy go through the trouble of creating a new file and >        copying the contents of an e-mail to this file (this is relevant >        not only to make our lives easier, but also to keep the >        encoding). >      * Please tell us what you did to compile the code. >      * Please tell us what you did to run the code. > > Søren > > man, 27 12 2010 kl. 15:54 +0100, skrev w4nderlust: >> I did that because it was suggested in the api manual. >> Anyway, if i remove it it compiles, but when i run it in octave i get >> this error: >> error: Sparse::SparseRep::elem (octave_idx_type, octave_idx_type): sparse matri >> x filled >> >> here's the full code: >> >> DEFUN_DLD (spmaxmin_ST, args, nargout, HELP) >> >> { >> >>       if (args.length() != NUM_ARG) >> >>               print_usage(); >> >>       else >> >>       { >> >>               SparseMatrix a = args(0).sparse_matrix_value(); >> >>               SparseMatrix b = args(1).sparse_matrix_value(); >> >>               dim_vector dimsA = a.dims(); >> >>               int rowsA = dimsA(0); >> >>               int colsA = dimsA(1); >> >> >> >>               dim_vector dimsB = b.dims(); >> >>               int rowsB = dimsB(0); >> >>               int colsB = dimsB(1); >> >> >> >>               SparseMatrix c = SparseMatrix(rowsA, colsB, rowsA*colsB); >> >>               dim_vector dimsC = c.dims(); >> >>               int rowsC = dimsC(0); >> >>               int colsC = dimsC(1); >> >> >> >>               //const SparseMatrix tmp(c); >> >>               int i,j,k; >> >>               double mx,mn; >> >>               if (colsA != rowsB) >> >>                       print_usage(); >> >>               else >> >>               { >> >>                       for (i = 0; i < rowsC; i++) >> >>                       { >> >>                               for(j = 0; j < colsC; j++) >> >>                               { >> >>                                       mx = 0; >> >>                                       for (k = b.cidx(j); k <= (b.cidx(j+1) - 1); k++) >> >>                                       { >> >>                                               mn = calc_tnorm(a(i, b.ridx(k)), b.data(k)); >> >>                                               if (mn > mx) >> >>                                                       mx = mn; >> >>                                       } >> >>                                       if (mx != 0) >> >>                                               c(i, j) = mx; >> >>                               } >> >>                       } >> >>                       c.maybe_compress(); >> >>                       return octave_value(c); >> >>               } >> >>       } >> >> } >> >> >> >> 2010/12/27 Søren Hauberg : >> > man, 27 12 2010 kl. 15:37 +0100, skrev w4nderlust: >> >> Hello everyone. >> >> Compiling an octfile i'm experiencing a compile error. >> >> spmaxmin_ST.cc:62: error: lvalue required as left operand of assignment >> >> >> >> that's how my code works: >> >> //i create a sparse matrix >> >> SparseMatrix c = SparseMatrix(rowsA, colsB, rowsA*colsB); >> >> >> >> //i create a constart temp matrix as suggested in the documentation >> >> const SparseMatrix tmp(c); >> >> >> >> //somewhere in my code i do >> >> if (mx != 0) >> >> >> >>     tmp(i, j) = mx; >> >> //mx is a double and it is initialized >> >> >> >> the problem is: >> >> tmp(i, j) = mx; >> >> >> >> i don't know what's wrong because in the documentation there are a lot >> >> of examples with this way of accessing the matrix... >> >> Can someone please help me? >> > >> > I don't know the sparse matrix API that well, but it looks like you have >> > declared a constant matrix tmp (constant means that it will not be >> > changed during the rest of the program). You then alter this matrix (by >> > 'tmp(i, j) = mx'), which is not possible if the matrix is constant. I >> > don't know this trick with the constant temp matrix, but it seems like >> > that is the source of your troubles. >> > >> > Søren >> > >> > > > > ```

 Re: [OctDev] Sparse matrix assign problem From: Søren Hauberg - 2010-12-27 20:10 ```Sorry if I came off a bit harsh in my previous mail. I had a quick look at your code; I can tell you what the problem is, but I cannot tell you how to solve as I don't know what your code is supposed to do and I don't know the SparseMatrix class very well (perhaps other people on the list can help). Basically 'a(i, b.xridx(k))' (the first argument to the 'calc_tnorm' call) gives you problems as this actually creates an element in the matrix, which is not what you want. Now you could avoid this by declaring 'a' and 'b' as being constant. This does, however, not work in your case as you are calling the 'xridx' function which is non-const. Not sure what the solution is as that depends on what you actually want to accomplish. Søren man, 27 12 2010 kl. 20:12 +0100, skrev w4nderlust: > Sorry for everything, i'm attaching the full code. > to compile i simply did call: > mkoctfile spmaxmin_ST.cc > and to run it i reated 2 sparse matrices A and B and then: > tic; spmaxmin_ST(A, B); toc; > > > > 2010/12/27 Søren Hauberg : > > Hi > > > > Some practical comments: > > > > * Please keep the list CC'ed (press reply to all) instead of just > > sending me a private mail; that way others can help. > > * If you send code, please send ALL relevant code. The code you > > send does NOT compile (you have not included relevant headers, > > defined HELP or NUM_ARG and you have not declared calc_tnorm). > > * Please send the code as an attached file so that we don't have > > to copy go through the trouble of creating a new file and > > copying the contents of an e-mail to this file (this is relevant > > not only to make our lives easier, but also to keep the > > encoding). > > * Please tell us what you did to compile the code. > > * Please tell us what you did to run the code. > > > > Søren > > > > man, 27 12 2010 kl. 15:54 +0100, skrev w4nderlust: > >> I did that because it was suggested in the api manual. > >> Anyway, if i remove it it compiles, but when i run it in octave i get > >> this error: > >> error: Sparse::SparseRep::elem (octave_idx_type, octave_idx_type): sparse matri > >> x filled > >> > >> here's the full code: > >> > >> DEFUN_DLD (spmaxmin_ST, args, nargout, HELP) > >> > >> { > >> > >> if (args.length() != NUM_ARG) > >> > >> print_usage(); > >> > >> else > >> > >> { > >> > >> SparseMatrix a = args(0).sparse_matrix_value(); > >> > >> SparseMatrix b = args(1).sparse_matrix_value(); > >> > >> dim_vector dimsA = a.dims(); > >> > >> int rowsA = dimsA(0); > >> > >> int colsA = dimsA(1); > >> > >> > >> > >> dim_vector dimsB = b.dims(); > >> > >> int rowsB = dimsB(0); > >> > >> int colsB = dimsB(1); > >> > >> > >> > >> SparseMatrix c = SparseMatrix(rowsA, colsB, rowsA*colsB); > >> > >> dim_vector dimsC = c.dims(); > >> > >> int rowsC = dimsC(0); > >> > >> int colsC = dimsC(1); > >> > >> > >> > >> //const SparseMatrix tmp(c); > >> > >> int i,j,k; > >> > >> double mx,mn; > >> > >> if (colsA != rowsB) > >> > >> print_usage(); > >> > >> else > >> > >> { > >> > >> for (i = 0; i < rowsC; i++) > >> > >> { > >> > >> for(j = 0; j < colsC; j++) > >> > >> { > >> > >> mx = 0; > >> > >> for (k = b.cidx(j); k <= (b.cidx(j+1) - 1); k++) > >> > >> { > >> > >> mn = calc_tnorm(a(i, b.ridx(k)), b.data(k)); > >> > >> if (mn > mx) > >> > >> mx = mn; > >> > >> } > >> > >> if (mx != 0) > >> > >> c(i, j) = mx; > >> > >> } > >> > >> } > >> > >> c.maybe_compress(); > >> > >> return octave_value(c); > >> > >> } > >> > >> } > >> > >> } > >> > >> > >> > >> 2010/12/27 Søren Hauberg : > >> > man, 27 12 2010 kl. 15:37 +0100, skrev w4nderlust: > >> >> Hello everyone. > >> >> Compiling an octfile i'm experiencing a compile error. > >> >> spmaxmin_ST.cc:62: error: lvalue required as left operand of assignment > >> >> > >> >> that's how my code works: > >> >> //i create a sparse matrix > >> >> SparseMatrix c = SparseMatrix(rowsA, colsB, rowsA*colsB); > >> >> > >> >> //i create a constart temp matrix as suggested in the documentation > >> >> const SparseMatrix tmp(c); > >> >> > >> >> //somewhere in my code i do > >> >> if (mx != 0) > >> >> > >> >> tmp(i, j) = mx; > >> >> //mx is a double and it is initialized > >> >> > >> >> the problem is: > >> >> tmp(i, j) = mx; > >> >> > >> >> i don't know what's wrong because in the documentation there are a lot > >> >> of examples with this way of accessing the matrix... > >> >> Can someone please help me? > >> > > >> > I don't know the sparse matrix API that well, but it looks like you have > >> > declared a constant matrix tmp (constant means that it will not be > >> > changed during the rest of the program). You then alter this matrix (by > >> > 'tmp(i, j) = mx'), which is not possible if the matrix is constant. I > >> > don't know this trick with the constant temp matrix, but it seems like > >> > that is the source of your troubles. > >> > > >> > Søren > >> > > >> > > > > > > > ```

 Re: [OctDev] Sparse matrix assign problem From: David Bateman - 2010-12-27 20:53 Attachments: spmaxmin_ST_new.cc ```Søren Hauberg wrote: > Sorry if I came off a bit harsh in my previous mail. > > I had a quick look at your code; I can tell you what the problem is, but > I cannot tell you how to solve as I don't know what your code is > supposed to do and I don't know the SparseMatrix class very well > (perhaps other people on the list can help). > > Basically 'a(i, b.xridx(k))' (the first argument to the 'calc_tnorm' > call) gives you problems as this actually creates an element in the > matrix, which is not what you want. Now you could avoid this by > declaring 'a' and 'b' as being constant. This does, however, not work in > your case as you are calling the 'xridx' function which is non-const. > > Not sure what the solution is as that depends on what you actually want > to accomplish. > > No time to really look at this, but does the attached help? The major changes are - Declare the matrices a and b as const to ensure that a.data, etc don't add elements to these matrices.. - Limit the number of elements to create in the sparse matrix c to a.nnz()+b.nnz() as this seems to be the maximum it can have to me, though it can easily be less - Operate on the transpose of the matrix C as the compressed row storage of the Octave sparse matrix makes this more logical. Do a final transpose to get the data in the right sense You might be able to get things faster if you eliminate the a(i,b.xridx(k)) call. This might be done by either working with the transpose of a, though at the cost of more memory. -- David Bateman dbateman@... 35 rue Gambetta +33 1 46 04 02 18 (Home) 92100 Boulogne-Billancourt FRANCE +33 6 72 01 06 33 (Mob) ```

 Re: [OctDev] Sparse matrix assign problem From: David Bateman - 2010-12-28 14:58 Attachments: spmaxmin_ST_new.cc ```Please keep responses on the list as the list is an invaluable source of information for people that have similar problems, and going offline deprives them of this information. w4nderlust wrote: > First thing: Thanks really much :) > then some questions: > 1) what's the difference between .cdx and .xcdx ? This is just to know. > The "x" before data in Sparse and Full matrices and ridx and cidx in Sparse matrices prevents some bounds checking. If you "know" that your indexing is correct, using xridx rather than ridx then your code will be faster, but will it will crash octave if your indexing is incorrect. > 2) how can owrking on the transpose of a help performances? > The compressed column format of Octave means that working with a Sparse matrix column by column is many times faster than working with a matrix row by row. The reason is that working by columns we can access the data sequentially in "data" and "ridx" whereas as not only can't we do that when working by rows, we have to traverse all of the elements of a column of the sparse matrix to find the offset into "data" and "ridx" for a particular row. > 3) if i run this code octave crashes... why? (using octave 3.2.4) > > >>>> SA = sprand(100, 100, 0.2); >>>> SB = sprand(100, 100, 0.2); >>>> tic; spmaxmin_ST_new(SA, SB); toc; >>>> > octave: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) > &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd))) > ) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__buil > tin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) > & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long > )old_end & pagemask) == 0)' failed. > panic: Aborted -- stopping myself... > attempting to save variables to `octave-core'... > > some other times it gives me just: > > panic: Segmentation fault -- stopping myself... > > some ideas? > Thanks > Because my assumption that the maximum number of non zeros elements in the return matrix is nnzA+nnzB is incorrect. I suggest counting the number of elements in the returned matrix first as this can be done at significantly lower cost than the full calculation, sizing the return matrix correctly and then removing the call to maybe_compress as its no longer needed. Try the attached David ```

 Re: [OctDev] Sparse matrix assign problem From: w4nderlust - 2010-12-28 16:07 ```Thanks a lot, now it works nicely (i also tried not to count the real nel and put all to nonzero and then compress, but the results are really equivalent). now for 100*100 matrices it works twice faster than the full version, but for 700x700 matrices it is slower (15% slower than the full version). I'll try to use the transpose of A and let's see if it gets faster. P.S. sorry for the reply and not reply all, i always forget.... 2010/12/28 David Bateman : > Please keep responses on the list as the list is an invaluable source of > information for people that have similar problems, and going offline > deprives them of this information. > > w4nderlust wrote: >> First thing: Thanks really much :) >> then some questions: >> 1) what's the difference between .cdx and .xcdx ? This is just to know. >> > > The "x" before data in Sparse and Full matrices and ridx and cidx in > Sparse matrices prevents some bounds checking. If you "know" that your > indexing is correct, using xridx rather than ridx then your code will be > faster, but will it will crash octave if your indexing is incorrect. > >> 2) how can owrking on the transpose of a help performances? >> > The compressed column format of Octave means that working with a Sparse > matrix column by column is many times faster than working with a matrix > row by row. The reason is that working by columns we can access the data > sequentially in "data" and "ridx" whereas as not only can't we do that > when working by rows, we have to traverse all of the elements of a > column of the sparse matrix to find the offset into "data" and "ridx" > for a particular row. > >> 3) if i run this code octave crashes... why? (using octave 3.2.4) >> >> >>>>> SA = sprand(100, 100, 0.2); >>>>> SB = sprand(100, 100, 0.2); >>>>> tic; spmaxmin_ST_new(SA, SB); toc; >>>>> >> octave: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) >>  &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd))) >> ) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__buil >> tin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) >> & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long >> )old_end & pagemask) == 0)' failed. >> panic: Aborted -- stopping myself... >> attempting to save variables to `octave-core'... >> >> some other times it gives me just: >> >> panic: Segmentation fault -- stopping myself... >> >> some ideas? >> Thanks >> > Because my assumption that the maximum number of non zeros elements in > the return matrix is nnzA+nnzB is incorrect. I suggest counting the > number of elements in the returned matrix first as this can be done at > significantly lower cost than the full calculation, sizing the return > matrix correctly and then removing the call to maybe_compress as its no > longer needed. Try the attached > > David > > > ```