I have an error during copy from ublas::compressed_matrix to viennaCL::compressed_matrix
It work good for matrices with nonzeros about 500000, but if my ublas matrix is large, for example 10 millions, the result viennaCL::compressed matrix is empty, has all zeros as non-zero values.
I use OpenMP on CPU.
If anybody faced with such a problem?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
are you referring to 10 million nonzeros, or 10 million rows and columns? Can you provide the matrix in matrix market format in order to reproduce the problem? Maybe you run out of memory?
Best regards,
Karli
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Karl,
I now have tested it for compressed matrix with 6 millions rows and colums. If it has about 50 nonzeros in each row, viennacl::copy doesn't work. But if it has only 30 or lesss nonzeros in each row, everything is good.
No, there are no problems with lack of memory. I am testing it on 64 gb machine and also on NUMA workstation with 1TB.
I found such a problem during calculation my FEM problems.
If I doing: solve(A, B, tag, precond) with A, B, precond - ublas objects, everything is good. But no parallel.
But if I doing copy before solve to viennaCL objects, solve return with NAN estimate tolerance. This is because Jacoby precond is zero. And precond is zero because copy doesn't work.
Such a big matrix in matrix market format has size of 1.4 gb. I can provide it, but it is better to provide code, as I think, which generate such a matrix. I added it as an attachment.
So it is just a test matrix, in real FEM solver I have even bigger matrices with more nonzeros for each row.
I had long time of debugging, and saw unsigned_int in many functions of viennacl::compressed_matrix. For example in function copy type of row_jumper and col_buffer must be unsigned int as I see. Does it mean that I have a limitation on number of nonzeros connected with unsigned int maximum? The error with copy ocure much before such sizes, but I can't understand if I have such a limit.
Last edit: Dmitriy 2020-03-24
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So, as I see this problems really connected with very large indexes.
In file: cpu_ram.hpp (line 94)
My size_in_bytes was about 2.3e9.
After classical cast: long(size_in_bytes) it becomes negative and data don't copy.
So I replaced (long) to (long long) and everything is ok.
But are there any good way to work with indexes larger that (unsigned int)?
I mean if I have number of nonzeros > 4.6e9?
Of course, I can rewrite function copy and a lot of othwer functions in compressed_matrix.hpp to replace (unsigned int) to (vcl_size_t) but it is not good decision))
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you are on Windows, you should attempt a 64-bit build in order to get 64-bit long. On Linux and MacOS 'long' is usually 64 bits anyway. It won't help with the unsigned int overflow, though.
There is no easy way to get around the unsigned int issue without making a significant impact on the overall code. It's certainly doable, but will take some time. ViennaCL had primarily GPUs in mind, for which memory is limited to a few GB, thus the GPU runs out of RAM before unsigned int overflows. With OpenMP on a workstation the situation is apparently different. Let me check the code and I'll get back to you.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Dmitriy,
I'll give the conversion to 64-bit integers a shot in a separate branch in the developer repository. It will take a day or two until I get to it, though.
Best regards,
Karli
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi carl. It is a great news. Let me sat thanks for your work. It really great.
Actually I replaced unsigned int occurrences in compressed_matrix.hpp, sparse_matrix_hpp and in many places for couple of iterative solvers and now it seems to work fine eve for system of:
rows = 210 mlns, columns = 210mls and with about 50 nonzeros in each row. I tested it in my fem geomechanics and hydrodynaics code, seems to work fine.
But of course, I have not capabilities to do it for each preconditioner and each iterative solver.
It will be a great if you will release a branch with 64-bit integers.
You a doing a great job.
Best regards,
Dmitry
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have an error during copy from ublas::compressed_matrix to viennaCL::compressed_matrix
It work good for matrices with nonzeros about 500000, but if my ublas matrix is large, for example 10 millions, the result viennaCL::compressed matrix is empty, has all zeros as non-zero values.
I use OpenMP on CPU.
If anybody faced with such a problem?
Hi Dmitriy,
are you referring to 10 million nonzeros, or 10 million rows and columns? Can you provide the matrix in matrix market format in order to reproduce the problem? Maybe you run out of memory?
Best regards,
Karli
Hi Karl,
I now have tested it for compressed matrix with 6 millions rows and colums. If it has about 50 nonzeros in each row, viennacl::copy doesn't work. But if it has only 30 or lesss nonzeros in each row, everything is good.
No, there are no problems with lack of memory. I am testing it on 64 gb machine and also on NUMA workstation with 1TB.
I found such a problem during calculation my FEM problems.
If I doing: solve(A, B, tag, precond) with A, B, precond - ublas objects, everything is good. But no parallel.
But if I doing copy before solve to viennaCL objects, solve return with NAN estimate tolerance. This is because Jacoby precond is zero. And precond is zero because copy doesn't work.
Such a big matrix in matrix market format has size of 1.4 gb. I can provide it, but it is better to provide code, as I think, which generate such a matrix. I added it as an attachment.
So it is just a test matrix, in real FEM solver I have even bigger matrices with more nonzeros for each row.
Best regards,
Dmitry
I had long time of debugging, and saw unsigned_int in many functions of viennacl::compressed_matrix. For example in function copy type of row_jumper and col_buffer must be unsigned int as I see. Does it mean that I have a limitation on number of nonzeros connected with unsigned int maximum? The error with copy ocure much before such sizes, but I can't understand if I have such a limit.
Last edit: Dmitriy 2020-03-24
So, as I see this problems really connected with very large indexes.
In file: cpu_ram.hpp (line 94)
My size_in_bytes was about 2.3e9.
After classical cast: long(size_in_bytes) it becomes negative and data don't copy.
So I replaced (long) to (long long) and everything is ok.
But are there any good way to work with indexes larger that (unsigned int)?
I mean if I have number of nonzeros > 4.6e9?
Of course, I can rewrite function copy and a lot of othwer functions in compressed_matrix.hpp to replace (unsigned int) to (vcl_size_t) but it is not good decision))
Ah, so the suspected 32-bit overflow. :-/
If you are on Windows, you should attempt a 64-bit build in order to get 64-bit long. On Linux and MacOS 'long' is usually 64 bits anyway. It won't help with the unsigned int overflow, though.
There is no easy way to get around the
unsigned intissue without making a significant impact on the overall code. It's certainly doable, but will take some time. ViennaCL had primarily GPUs in mind, for which memory is limited to a few GB, thus the GPU runs out of RAM beforeunsigned intoverflows. With OpenMP on a workstation the situation is apparently different. Let me check the code and I'll get back to you.Hi Dmitriy,
I'll give the conversion to 64-bit integers a shot in a separate branch in the developer repository. It will take a day or two until I get to it, though.
Best regards,
Karli
Hi carl. It is a great news. Let me sat thanks for your work. It really great.
Actually I replaced unsigned int occurrences in compressed_matrix.hpp, sparse_matrix_hpp and in many places for couple of iterative solvers and now it seems to work fine eve for system of:
rows = 210 mlns, columns = 210mls and with about 50 nonzeros in each row. I tested it in my fem geomechanics and hydrodynaics code, seems to work fine.
But of course, I have not capabilities to do it for each preconditioner and each iterative solver.
It will be a great if you will release a branch with 64-bit integers.
You a doing a great job.
Best regards,
Dmitry