Yes, the bottleneck is your hardware. Increase your buffer size as your system is not fast enough to keep up. The first most important device that contributes to how low a buffer size you can use is your audio device, provided that you have at least a 7200 RPM disk (or an SSD equivalent).

Decreasing the buffer size decreases latency, but at the same time consumes more CPU power. Aside from that, if your disk is not fast enough, there is almost always a limit to how low the buffer size can go. Disks of 5400RPM usually cannot go lower than 256.

If you are using a MIDI controller or an external software sequencer, then latency is important. However, a buffer of 512 will suffice. If you are only using the mouse, increase the buffer size until 1024 so big projects will play back without stuttering.