[Libusbx-devel] [PATCH] detach-kernel-driver: return -EBUSY if another libusb app is active

[Hans de G. <hde...@re...>]

Currently applications for devices which only are accessed from userspace
can use claim / release interface to make sure they don't get in each others
way. The same however does not work for applications which first need to detach
a "native" / in kernel driver, as this detach will not only detach native
drivers but also the usbfs driver, thus stealing the device from another
userspace / libusb app.

This patch fixes libusb_detach_kernel_driver to only detach "real" kernel
drivers and not the special usbfs driver used for userspace access to
USB devices. If the usbfs driver is found LIBUSB_ERROR_BUSY will be
returned.

Signed-off-by: Hans de Goede <hde...@re...>
---
 libusb/core.c           |    1 +
 libusb/os/linux_usbfs.c |    6 ++++++
 2 files changed, 7 insertions(+)

diff --git a/libusb/core.c b/libusb/core.c
index eccc216..18f56ca 100644
--- a/libusb/core.c
+++ b/libusb/core.c
@@ -1546,6 +1546,7 @@ int API_EXPORTED libusb_detach_kernel_driver(libusb_device_handle *dev,
  * \returns 0 on success
  * \returns LIBUSB_ERROR_NOT_FOUND if no kernel driver was active
  * \returns LIBUSB_ERROR_INVALID_PARAM if the interface does not exist
+ * \returns LIBUSB_ERROR_BUSY if the interface is in use by another libusb app
  * \returns LIBUSB_ERROR_NO_DEVICE if the device has been disconnected
  * \returns LIBUSB_ERROR_NOT_SUPPORTED on platforms where the functionality
  * is not available
diff --git a/libusb/os/linux_usbfs.c b/libusb/os/linux_usbfs.c
index dad7536..72fd2b4 100644
--- a/libusb/os/linux_usbfs.c
+++ b/libusb/os/linux_usbfs.c
@@ -1522,12 +1522,18 @@ static int op_detach_kernel_driver(struct libusb_device_handle *handle,
 {
 	int fd = _device_handle_priv(handle)->fd;
 	struct usbfs_ioctl command;
+	struct usbfs_getdriver getdrv;
 	int r;
 
 	command.ifno = interface;
 	command.ioctl_code = IOCTL_USBFS_DISCONNECT;
 	command.data = NULL;
 
+	getdrv.interface = interface;
+	r = ioctl(fd, IOCTL_USBFS_GETDRIVER, &getdrv);
+	if (r == 0 && strcmp(getdrv.driver, "usbfs") == 0)
+		return LIBUSB_ERROR_BUSY;
+
 	r = ioctl(fd, IOCTL_USBFS_IOCTL, &command);
 	if (r) {
 		if (errno == ENODATA)
-- 
1.7.10.4