The fontsize argument to gp_cairo_set_font is an integer, and this creates some surprising discretization issues for me on the pdfcairo terminal (e.g., that font sizes in the produced PDF can't be anything between 8 and 10.67, which appear to correspond to arguments of 3 and 4 to gp_cairo_set_font). I see no obvious reason why that argument couldn't just be a double (it's setting a double in plot_struct).
I'm seeing the issue on Windows, gnuplot 5.2.5, and setting the terminal with the following line will yield an 8 point font, changing the font size to 11 will yield a 10.67 point font:
set term pdfcairo size 3.5,2.0 font 'Times New Roman, 10' round fontscale 0.375
Good idea.
I made this change in the source for 5.3 and will do the same for 5.2 (but it didn't make yesterday's 5.2.6 release package)