In the machine learning field, especially deep learning, GPU acceleration has gotten a lot of attention. But is it really that using GPU is faster than using CPU in training every model?
As we all know, GPU is also called Graphics Processing Unit. It was initially designed to speed up the process when creating and rendering graphics, to be more specific, to accelerate when calculating each pixel based on complex mathematics of color and lighting models. Although it is easy for the CPU to calculate one pixel, it is burdensome if it is high-resolution images, not to mention videos, especially when the calculation has to be done one by one. Additionally, CPU also needs to handle a lot of other tasks.
On the other hand, the main advantage of GPU, compared to CPU, is its ability to parallel computing and data throughput. GPU usually consists of hundreds or thousands of cores that can do calculations in parallel. A GPU core may not be as powerful and smart as a CPU core. The GPU cores are specialized to deal with massive calculations with similar operations, such as matrices operations. And they can do it in parallel. But for tasks that are not like image rendering and may need to wait for the result from the last calculation to do the next calculation, GPU is not necessary to be faster.
For many machine learning or deep learning tasks, lots of data is usually required to train a model and GPU parallelization can provide great help. For example, for an image classification model, the training images are independent for the models to learn from so that the models can be trained simultaneously, which is where GPU can be utilized. Popular deep learning frameworks like PyTorch and Tensorflow mostly support using GPU to train models.