On Linux these days TTYs are locked using flock() along with ioctl(fd, TIOCEXCL).
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734086
https://linux.die.net/man/4/tty_ioctl
If locking is to be done using lock files, it should be done with liblockdev.
https://linux.die.net/man/3/lockdev
Modern Linux distros no longer uses /var/lock for TTY locks. This path seems to be hard-coded into the serial library. These should now go in /var/lock/lockdev. liblockdev includes a setgid helper app to handle permissions required to write lock files and ensures they are written to the correct location.
https://systemd-devel.freedesktop.narkive.com/vCJLfMo2/headsup-var-lock-and-var-lock-lockdev
Anonymous
Yes, this is hard-coded into the RXTX library. Apparently, this library is no longer being supported (at least, I haven't seen any updates in the past 7 years).
I have a prototype fix converting to using the JSSC library instead of the RXTX library, which will solve the file locking issue, but I'm not sure if it will solve the DTR twitching issue.
Been using JSSC with kernel-based file locking for a while.