[Infiniband-access_layer] Shared memory

[Fab T. <fti...@in...>]

Folks,

I'd like to propose some changes in the shared memory and shmid =
management
in IBAL to better support user-mode applications.

First up are issues with ib_reg_shared:
- ib_reg_shared entry point for user-mode will fail if the input h_mr is =
not
within the same process context as the registration.  This prevents
ib_reg_shared from working across multiple processes.  This is a kernel
proxy issue in IBAL.  There are two options that I can think of to solve
this.  The first would require keeping a global map of all memory =
regions so
that the input h_mr could be found/validated/reference counted for the
ib_reg_shared call.  This would allow a user-mode process to call
ib_reg_shared against a kernel-mode h_mr.  The second option is to keep =
a
proxy-global map of all memory handles that would behave in the same =
way,
but would prevent a user-mode app from calling ib_reg_shared with a =
kernel
h_mr.

- ib_reg_shared needs to check that the page mappings match between the
input h_mr and the requested vaddr before succeeding the call - note =
however
that this check should be performed by the HCA driver, but I don't =
believe
that the Mellanox VPD for IBAL does this today.  IBAL must also check =
that
the input h_mr exists on the HCA represented by the input h_pd.

Second issue is SHMID management, which seems to exist mainly to support
DAPL.  The current ib_create_shmid and ib_reg_shmid don't quite meet the
DAPL usage model: DAPL doesn't want to have a master/slave relationship =
-
there should not be any ordering requirement.  I would like to propose =
two
changes.  The first would merge ib_create_shmid and ib_reg_shmid into a
single call ib_reg_shmid that would figure out itself if it was the =
first
call or a subsequent call.  ib_reg_shmid would then need to track the =
page
mappings so that use on multiple HCAs would be supported.  The shmid =
object
would not be associated with any particular HCA, and would have a =
per-HCA
list of memory regions to determine whether to call ib_reg_mem or
ib_reg_shared (based on whether existing memory region handles are =
available
for the target HCA).  The shmid object would then auto-destroy when the =
last
memory region associated with it is deregistered.

Further, I believe it would be beneficial to extend the shmid identifier
from an integer to a byte array.  This would remove the need for DAPL to
manage cookie-to-shmid relationships, and remove the need for DAPL to =
have a
kernel agent.  Additionally, it would help reduce the potential for ID
namespace collision between different applications by allowing a more
natural identifier to be specified.

Thoughts?  Comments?

- Fab