The Asio driver supports to query the actual audio delay introduced by the driver. We should use this information to get a better overall delay value estimation.
Maybe for the Jack and Core Audio interfaces also such a function exists.
At least for the Windows ASIO driver I have implemented to query the latency from the driver. MacOS and Linux may at a later time follow. But for this I will create a new ticket eventually.