Hi! CodeGeeX has 13B parameters, even under FP16 format, the model itself takes around 27GB RAM. Thus it requires GPUs with at least 32GB RAM.
If you want to run it on RTX 3080, there are two possibilities. First, if you want to run it on a single card, you need to frequently switch data between CPU RAM and GPU RAM. It is feasible but will be extremely slow (depending on the speed of PCI-E on your hardware). Second, you may also run it on multiple cards (4 * 3080) using pipeline techniques, i.e., divide the model into multiple parts and put them on different cards, then do the forward calculation of each part sequentially. It is faster than the first solution.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello, was optimizing the number of model parameters part of your research priorities ? If so, what methods or strategies have you put in place? If not, do you plan to look into this subject soon ?
Thank you.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello, was optimizing the number of model parameters part of your research priorities ? If so, what methods or strategies have you put in place? If not, do you plan to look into this subject soon ? Thank you.
The optimization of the model inference is definitely an important future work. It is not necessarily reducing the number of parameters, since the model should have enough model capacity to learn different languages (especially in the case of multilingual models). Instead, we plan to use methods like quantization (e.g., compress FP16 to INT8) or faster implementations (C++ implementation, kernel fusion, etc). We are currently working on this, and the solution will also be publicly available.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Originally posted by: Stanislas0
Hi! CodeGeeX has 13B parameters, even under FP16 format, the model itself takes around 27GB RAM. Thus it requires GPUs with at least 32GB RAM.
If you want to run it on RTX 3080, there are two possibilities. First, if you want to run it on a single card, you need to frequently switch data between CPU RAM and GPU RAM. It is feasible but will be extremely slow (depending on the speed of PCI-E on your hardware). Second, you may also run it on multiple cards (4 * 3080) using pipeline techniques, i.e., divide the model into multiple parts and put them on different cards, then do the forward calculation of each part sequentially. It is faster than the first solution.
Originally posted by: Ehplodor
Hello, was optimizing the number of model parameters part of your research priorities ? If so, what methods or strategies have you put in place? If not, do you plan to look into this subject soon ?
Thank you.
Originally posted by: Stanislas0
The optimization of the model inference is definitely an important future work. It is not necessarily reducing the number of parameters, since the model should have enough model capacity to learn different languages (especially in the case of multilingual models). Instead, we plan to use methods like quantization (e.g., compress FP16 to INT8) or faster implementations (C++ implementation, kernel fusion, etc). We are currently working on this, and the solution will also be publicly available.