The dnsmasq configuration generated when setting the boot filename on the DHCP services page appears to be incorrect.
To reproduce, enter a filename (and next server of course) in the provided field. It doesn't matter if this is done globally or on a per host reservation basis, the result is the same.

When clicking save the following output is generated in the system log:
Jan 7 18:30:33 <daemon.crit> protostar dnsmasq[4294]: bad dhcp-option at line 8 of /var/etc/dnsmasq/dhcpd.conf
Jan 7 18:30:33 <daemon.crit> protostar dnsmasq[4294]: bad dhcp-option at line 8 of /var/etc/dnsmasq/dhcpd.conf
Jan 7 18:30:33 <daemon.crit> protostar dnsmasq[4294]: FAILED to start up
This is the contents of the dnsmasq/dhcpd.conf file:
dhcp-option=option:domain-name, "domain.com"
dhcp-authoritative
dhcp-leasefile=/var/db/dnsmasq.dhcpd.leases
dhcp-range=set:lan,10.8.8.50,static
dhcp-option=tag:lan,option:dns-server,10.8.8.1
dhcp-option=tag:lan,option:router,10.8.8.1
dhcp-option=tag:lan,option:tftp-server,10.8.8.2;
dhcp-option=tag:lan,option:boot-filename,"undionly.kpxe.vmw-hardwired"
According to the sample configuration provided by dns/dnsmasq, the correct syntax should be using dhcp-boot as a single line containing both the filename and server to boot from:
# Set the boot filename for netboot/PXE. You will only need
# this if you want to boot machines over the network and you will need
# a TFTP server; either dnsmasq's built-in TFTP server or an
# external one. (See below for how to enable the TFTP server.)
#dhcp-boot=pxelinux.0
# The same as above, but use custom tftp-server instead machine running dnsmasq
#dhcp-boot=pxelinux,server.name,192.168.1.100
I've just posted a 'fix'. this will require both a filename and next server to be set. next-server and tftp-server and clients that use bootp vs dhcp seems to be an area of mixed results. there are old clients like some ip phones, that might need dhcp and bootp options. hopefully with setting dhcp-no-override, it will work with all clients.