Discussion on ML was positive towards iSCSI. The gains are portability and enabling networked devices, While that sounds nice, it's a big job implementing so in all likelihood it won't happen.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Have you looked at this again recently? LIO is now in mainline, and the API for backstores is pretty simple (see struct se_subsystem_api).
The main issue I see there is that configuration is done via configfs, so cdemud's relationship with vhba would be inverted - the user would trigger config changes via configfs, and the vhba LIO backstore would signal cdemud with what it wants.
On the other hand, the way LIO works allows adding LUNs on the fly, so there would no longer be any need to balance "ran out of devices" against "my /dev is horrendously cluttered".
It might even be an opportunity for a rearchitecting - have a central cdemu daemon that listens to the kernel over netlink, the kernel requests a disc image, the central daemon spawns a dedicated per-image process.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OPs idea here was to avoid having to maintain kernel code altogether. Directly supporting iSCSI in the daemon is something that would have allowed for that. Although there are disadvantages such as performance overhead, writing lots of code for iSCSI support that can be obtained by other means (reinvent the wheel).
I did have a look in the meantime, and noticed LIO got merged. I also noticed a few virtual HBAs getting merged in the kernel and it hit me it might be possible to use of of those.
The option you're suggesting of rewriting VHBA from using the SCSI layer to use the LIO API seems interesting. But I won't pretend knowing a lot about kernel coding, so I'll ask you instead: what would the advantages be?
From what I could gather VHBA would be written to work essentially like the PSCSI backing store, and channel CDBs directly into the LIO framework. What is to keep someone from setting up PSCSI on a CDEMU device and do the same? There are some cool things that can be done in LIO, it can utilize virtio-scsi to send SCSI data in a passthrough manner to VMs such as QEMU and KVM, but again... can't one do this using PSCSI and have the same performance?
Netlink and forking only seems to add overhead at no obvious (to me) benefit.
I'd really love to hear some selling points here...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You are correct LIO can be hooked up to cdemu via pscsi, in order to add the ability to export CDemu drives over iscsi, vhost-scsi, etc. The thing is, implementing vhba as a LIO backstore has other benefits. One of these is that LIO backstores can add LUNs on the fly, so you don't need to set the number of devices when starting cdemud - they can be created on-demand in essentially unlimited quantities, and removed the same way.
Also, netlink and forking were kind of 'toss it out there' things from me, not a 'standard' part of the API, but I think it may help if I explain more.
For one, neither is on the hot path. There would be a central daemon listening on netlink, but it's basically a jobserver. It hears a request 'load disc /home/baz/foo.iso', spawns a handler for /home/baz/foo.iso (with some kind of cookie so it can hook up with the request), and then that handler sticks around and handles all IO to foo.iso from then until it is detached. It doesn't even really need to fork, it could thread or be event-based, but forking has the benefit that if the central daemon dies, the handler is still around so mounted discs don't get terminated, and if one handler dies (say, a malicious disc in a format with a less-durable parser) neither the central daemon nor the other handlers are affected (beyond the central daemon receiving SIGCHLD, at least)
The handler can talk to the kernel any way it wants, and it's the performance-sensitive part - I think reusing the vhost ring-buffer system might be a neat way to do it, since it's already known to perform well.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Another thought I had, which could eliminate the issues with a central daemon and permissions. In LIO, when you add a device you can pass arguments to it; the only argument required for every backstore is name, but backstores can define parameters and indicate that some are mandatory. Here's my thought:
1.) Have a mandatory 'user' parameter when adding a device to the VHBA backstore (in LIO, adding devices is done via configfs)
2.) Have cdeumu's load command use a privileged helper that always uses the calling user's name there
3.) Pass that down to the central dispatcher daemon from the design I described
3.) When the central daemon spawns the helper, it gives the helper what it needs to communicate with the kernel (vhost file descriptor, whatever), drops privileges to the calling user, and then exec()s. That way, images are always opened with the privileges of the user calling 'cdemu load', avoiding information disclosure. Maybe even make open()ing the emulated device give the same result as the attempted open() of the image, if it failed.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Agh, that last 3 should be a 4. Anyway, while someone could get around that by writing to configfs directly, only root can do that, and they can read any file regardless.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The thing
is, implementing vhba as a LIO backstore has other benefits. One of these
is that LIO backstores can add LUNs on the fly, so you don't need to set
the number of devices when starting cdemud - they can be created on-demand
in essentially unlimited quantities, and removed the same way.
Well... granted cdemu doesn't do this today but it would be relatively simple to do.
the kernel module already supports dynamic device creation and so it will not be a chore to
make the deamon do dynamic device allocation as well. of course this will
also have to mean adding a dbus-service to add devices.
and.. let's see...
your second point was permissions. that's IMO taken care of by dbus session busses.
i will admit that security and such is a secondary objective, but the developers earlier agreed that being open about the issue was sufficient.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Basic security is also provided by VFS permissions on the vhba control device of course. Crude, but it works.
I should also say that forking is not a bad idea. If I were to use CDEMU for some server-type use that's exactly how I would want the arcitecture to be. However cdemu is typically something you use on a desktop computer and you rarely use more than one or two devices. Thus the value of crash-isolation you get from forking, is IMO overrated for the typical use case. It's IMO not worth the inherent drawbacks from forking.
I'll look into having the daemon support dynamic device creation. That's something worth doing.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think the main question here is, which is the lesser evil - keeping our VHBA module in step with changes in kernel (which was the main reason for OP), or having to redesign the whole CDEmu architecture to LIO? Keep in mind that architectural changes in daemon will bring along changes in the clients as well, since if I understand correctly, they would have to directly manipulate with configfs?
Personally, I don't see dynamic addition/removal of LUNs as a big selling point. As Henrik pointed out, VHBA module already supports dynamic device allocation. While on that topic, I don't find supporting it in daemon and clients as something really worth the trouble...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm a bit surprised to see this bug closed while the initial comments seemed quite positive?
Wasn't it feasible to use one of the existing systems in the kernel? Respectively did the kernel developers really reject the driver as it is - or did they just say a "would be nice if you could use that instead"?
Inclusion in mainline would be still beneficial, I think.
It simply makes CDEmu available to more users (even if they still need userland support) and probably better supported for any new versions.
Cheers,
Chris.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've been in contact with some kernel coders to ask for some guidance on getting it merged.
I'll post updates as I have them.
Got some pretty good advice.
I'll be researching some alternatives to vhba.
Alternatives include iSCSI, STGT and SCST.
iSCSI doesn't look trivial to implement
SCST isn't in mainline kernel but it is well-documented.
STGT suffers from lack of documentation.
Discussion on ML was positive towards iSCSI. The gains are portability and enabling networked devices, While that sounds nice, it's a big job implementing so in all likelihood it won't happen.
Have you looked at this again recently? LIO is now in mainline, and the API for backstores is pretty simple (see struct se_subsystem_api).
The main issue I see there is that configuration is done via configfs, so cdemud's relationship with vhba would be inverted - the user would trigger config changes via configfs, and the vhba LIO backstore would signal cdemud with what it wants.
On the other hand, the way LIO works allows adding LUNs on the fly, so there would no longer be any need to balance "ran out of devices" against "my /dev is horrendously cluttered".
It might even be an opportunity for a rearchitecting - have a central cdemu daemon that listens to the kernel over netlink, the kernel requests a disc image, the central daemon spawns a dedicated per-image process.
OPs idea here was to avoid having to maintain kernel code altogether. Directly supporting iSCSI in the daemon is something that would have allowed for that. Although there are disadvantages such as performance overhead, writing lots of code for iSCSI support that can be obtained by other means (reinvent the wheel).
I did have a look in the meantime, and noticed LIO got merged. I also noticed a few virtual HBAs getting merged in the kernel and it hit me it might be possible to use of of those.
The option you're suggesting of rewriting VHBA from using the SCSI layer to use the LIO API seems interesting. But I won't pretend knowing a lot about kernel coding, so I'll ask you instead: what would the advantages be?
From what I could gather VHBA would be written to work essentially like the PSCSI backing store, and channel CDBs directly into the LIO framework. What is to keep someone from setting up PSCSI on a CDEMU device and do the same? There are some cool things that can be done in LIO, it can utilize virtio-scsi to send SCSI data in a passthrough manner to VMs such as QEMU and KVM, but again... can't one do this using PSCSI and have the same performance?
Netlink and forking only seems to add overhead at no obvious (to me) benefit.
I'd really love to hear some selling points here...
You are correct LIO can be hooked up to cdemu via pscsi, in order to add the ability to export CDemu drives over iscsi, vhost-scsi, etc. The thing is, implementing vhba as a LIO backstore has other benefits. One of these is that LIO backstores can add LUNs on the fly, so you don't need to set the number of devices when starting cdemud - they can be created on-demand in essentially unlimited quantities, and removed the same way.
Also, netlink and forking were kind of 'toss it out there' things from me, not a 'standard' part of the API, but I think it may help if I explain more.
For one, neither is on the hot path. There would be a central daemon listening on netlink, but it's basically a jobserver. It hears a request 'load disc /home/baz/foo.iso', spawns a handler for /home/baz/foo.iso (with some kind of cookie so it can hook up with the request), and then that handler sticks around and handles all IO to foo.iso from then until it is detached. It doesn't even really need to fork, it could thread or be event-based, but forking has the benefit that if the central daemon dies, the handler is still around so mounted discs don't get terminated, and if one handler dies (say, a malicious disc in a format with a less-durable parser) neither the central daemon nor the other handlers are affected (beyond the central daemon receiving SIGCHLD, at least)
The handler can talk to the kernel any way it wants, and it's the performance-sensitive part - I think reusing the vhost ring-buffer system might be a neat way to do it, since it's already known to perform well.
Another thought I had, which could eliminate the issues with a central daemon and permissions. In LIO, when you add a device you can pass arguments to it; the only argument required for every backstore is name, but backstores can define parameters and indicate that some are mandatory. Here's my thought:
1.) Have a mandatory 'user' parameter when adding a device to the VHBA backstore (in LIO, adding devices is done via configfs)
2.) Have cdeumu's load command use a privileged helper that always uses the calling user's name there
3.) Pass that down to the central dispatcher daemon from the design I described
3.) When the central daemon spawns the helper, it gives the helper what it needs to communicate with the kernel (vhost file descriptor, whatever), drops privileges to the calling user, and then exec()s. That way, images are always opened with the privileges of the user calling 'cdemu load', avoiding information disclosure. Maybe even make open()ing the emulated device give the same result as the attempted open() of the image, if it failed.
Agh, that last 3 should be a 4. Anyway, while someone could get around that by writing to configfs directly, only root can do that, and they can read any file regardless.
I love enthusiasts.
Well... granted cdemu doesn't do this today but it would be relatively simple to do.
the kernel module already supports dynamic device creation and so it will not be a chore to
make the deamon do dynamic device allocation as well. of course this will
also have to mean adding a dbus-service to add devices.
and.. let's see...
your second point was permissions. that's IMO taken care of by dbus session busses.
i will admit that security and such is a secondary objective, but the developers earlier agreed that being open about the issue was sufficient.
Basic security is also provided by VFS permissions on the vhba control device of course. Crude, but it works.
I should also say that forking is not a bad idea. If I were to use CDEMU for some server-type use that's exactly how I would want the arcitecture to be. However cdemu is typically something you use on a desktop computer and you rarely use more than one or two devices. Thus the value of crash-isolation you get from forking, is IMO overrated for the typical use case. It's IMO not worth the inherent drawbacks from forking.
I'll look into having the daemon support dynamic device creation. That's something worth doing.
I think the main question here is, which is the lesser evil - keeping our VHBA module in step with changes in kernel (which was the main reason for OP), or having to redesign the whole CDEmu architecture to LIO? Keep in mind that architectural changes in daemon will bring along changes in the clients as well, since if I understand correctly, they would have to directly manipulate with configfs?
Personally, I don't see dynamic addition/removal of LUNs as a big selling point. As Henrik pointed out, VHBA module already supports dynamic device allocation. While on that topic, I don't find supporting it in daemon and clients as something really worth the trouble...
Hey Henrik.
Rok just pointed me here.
I'm a bit surprised to see this bug closed while the initial comments seemed quite positive?
Wasn't it feasible to use one of the existing systems in the kernel? Respectively did the kernel developers really reject the driver as it is - or did they just say a "would be nice if you could use that instead"?
Inclusion in mainline would be still beneficial, I think.
It simply makes CDEmu available to more users (even if they still need userland support) and probably better supported for any new versions.
Cheers,
Chris.