system: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux
gnuplot version: 5.0.2 (2015-12-24)
cairo version: 1.14.0
The attached example is generated with the following code:
set terminal pdfcairo font ",9" fontscale 1
set output "./example.pdf"
plot sin(x)
However, the actual font size is 15px=12pt as shown by Inkscape. epscairo terminal produces graph with the same (12pt) font size.
There are so many scaling factors involved that it is not realistic to expect the final visual font size to match the nominal font size in the "set terminal" command. This is particularly true when the plot produced by gnuplot is embedded in a larger document with its own layer of rescaling.
That is why the "fontscale" option exists, so that you can keep the nominal fontsize the same across multiple gnuplot terminal types but tweak the eventual printed font size to suit the final document.