dapl-announce — DAPL announcements

[Dapl-announce] new uDAPL co-maintainer

[James L. <jle...@ne...>]

In recognition of his many contributions to the DAPL project, Arlin 
Davis is joining the project as an official co-maintainer. Arlin and I 
will collaborate on DAPL maintenance and development decisions.

james

--
James Lentini | Network Appliance | 781-768-5359 | jle...@ne...

[Dapl-announce] uDAPL/kDAPL Gamma 3.1 now available

[James L. <jle...@ne...>]

		   Release Notes for 
		Gamma 3.1 DAPL Release
		   May 31, 2006


        DAPL GAMMA 3.1 RELEASE NOTES

        This release of the DAPL reference implementation 
        is timed to coincide with the first release of the 
        Open Fabrics (www.openfabrics.org) software stack.
        This release adds support for this new stack, which 
        is now the native Linux RDMA stack.
        
        This release also adds a new licensing option. In 
        addition to the Common Public License and BSD License,
	the code can now be licensed under the terms of the GNU 
        General Public License (GPL) version 2.

        NEW SINCE Gamma 3.0

        - GPL v2 added as a licensing option
        - OpenFabrics (aka OpenIB) gen2 verbs support
        - dapltest support for Solaris 10

        * BUG FIXES

        + Fixed a disconnect event processing race
        + Fix to destroy all QPs on IA close
        + Removed compiler warnings
        + Removed unused variables
        + And many more...

        DAPL GAMMA 3.0 RELEASE NOTES

        This is the first release based on version 1.2 of the spec. There 
        are some components, such a shared receive queues (SRQs), which 
        are not implemented yet. 

        Once again there were numerous bug fixes submitted by the 
        DAPL community.

        NEW SINCE Beta 2.06

        - DAT 1.2 headers
        - DAT_IA_HANDLEs implemented as small integers
	- Changed default device name to be "ia0a"
        - Initial support for Linux 2.6.X kernels
        - Updates to the OpenIB gen 1 provider 

        * BUG FIXES

        + Updated Makefile for differentiation between OS releases. 
        + Updated atomic routines to use appropriate API
        + Removed unnecessary assert from atomic_dec. 
        + Fixed bugs when freeing a PSP.
        + Fixed error codes returned by the DAT static registry.
        + Kernel updates for dat_strerror.
        + Cleaned up the transport layer/adapter interface to use DAPL 
          types rather than transport types.
        + Fixed ring buffer reallocation.
        + Removed old test/udapl/dapltest directory.
        + Fixed DAT_IA_HANDLE translation (from pointer to int and 
          vice versa) on 64-bit platforms.

	DAP BETA 2.06 RELEASE NOTES

	We are not planning any further releases of the Beta series,
	which are based  on the 1.1 version of the spec. There may be
	further releases for bug fixes, but we anticipate the DAPL
	community to move to the new 1.2 version of the spec and the
	changes mandated in the reference implementation.

	The biggest item in this release is the first inclusion of the
	OpenIB Gen 1 provider, an item generating a lot of interest in
	the IB community. This implementation has graciously been
	provided by the Mellanox team. The kdapl implementation is in
	progress, and we imagine work will soon begin on Gen 2.

	There are also a handful of bug fixes available, as well as a long
	awaited update to the endpoint design document.

	NEW SINCE Beta 2.05

	- OpenIB gen 1 provider support has been added
	- Added dapls_evd_post_generic_event(), routine to post generic 
	  event types as requested by some providers. Also cleaned up 
	  error reporting.
	- Updated the endpoint design document in the doc/ directory.

	* BUG FIXES

	+ Cleaned up memory leak on close by freeing the HCA structure;
	+ Removed bogus #defs for rdtsc calls on IA64.
	+ Changed daptest thread types to use internal types for 
	  portability & correctness
	+ Various 64 bit enhancements & updates
	+ Fixes to conformance test that were defining CONN_QUAL twice
	  and using it in different ways
	+ Cleaned up private data handling in ep_connect & provider 
	  support: we now avoid extra copy in connect code; reduced
	  stack requirements by using private_data structure in the EP;
	  removed provider variable.
	+ Fixed problem in the dat conformance test where cno_wait would
	  attempt to dereference a timer value and SEGV.
	+ Removed old vestiges of depricated POLLING_COMPLETIONS 
	  conditionals.

	DAPL BETA 2.05 RELEASE NOTES

	This was to be a very minor release, the primary change was
	going to be the new wording of the DAT license as contained in
	the header for all source files. But the interest and
	development occurring in DAPL provided some extra bug fixes, and
	some new functionality that has been requested for a while.

	First, you may notice that every single source file was
	changed. If you read the release notes from DAPL BETA 2.04, you
	were warned this would happen. There was a legal issue with the
	wording in the header, the end result was that every source file
	was required to change the word 'either of' to 'both'. We've
	been putting this change off as long as possible, but we wanted
	to do it in a clean drop before we start working on DAT 1.2
	changes in the reference implementation, just to keep things
	reasonably sane.

	kdapltest has enabled three of the subtests supported by
	dapltest. The Performance test in particular has been very
	useful to dapltest in getting minima and maxima. The Limit test
	pushes the limits by allocating the maximum number of specific
	resources. And the FFT tests are also available.

	Most vendors have supported shared memory regions for a while,
	several of which have asked the reference implementation team to
	provide a common implementation. Shared memory registration has
	been tested on ibapi, and compiled into vapi. Both InfiniBand
	providers have the restriction that a memory region must be
	created before it can be shared; not all RDMA APIs are this way,
	several allow you to declare a memory region shared when it is
	registered. Hence, details of the implementation are hidden in
	the provider layer, rather than forcing other APIs to do
	something strange.

	This release also contains some changes that will allow dapl to
	work on Opteron processors, as well as some preliminary support
	for Power PC architecture. These features are not well tested
	and may be incomplete at this time.

	Finally, we have been asked several times over the course of the
	project for a canonical interface between the common and
	provider layers. This release includes a dummy provider to meet
	that need. Anyone should be able to download the release and do
	a:
	   make VERBS=DUMMY

	And have a cleanly compiled dapl library. This will be useful
	both to those porting new transport providers, as well as those
	going to new machines.

	The DUMMY provider has been compiled on both Linux and Windows
	machines.


	NEW SINCE Beta 2.4
	- kdapltest enhancements:
	  * Limit subtests now work
	  * Performance subtests now work.
	  * FFT tests now work.

	- The VAPI headers have been refreshed by Mellanox

	- Initial Opteron and PPC support.

	- Atomic data types now have consistent treatment, allowing us to
	  use native data types other than integers. The Linux kdapl
	  uses atomic_t, allowing dapl to use the kernel macros and
	  eliminate the assembly code in dapl_osd.h

	- The license language was updated per the direction of the
	  DAT Collaborative. This two word change affected the header
	  of every file in the tree.

	- SHARED memory regions are now supported.

	- Initial support for the TOPSPIN provider.

	- Added a dummy provider, essentially the NULL provider. It's
	  purpose is to aid in porting and to clarify exactly what is
	  expected in a provider implementation.

	- Removed memory allocation from the DTO path for VAPI

	- cq_resize will now allow the CQ to be resized smaller. Not all
	  providers support this, but it's a provider problem, not a
	  limitation of the common code.

	* BUG FIXES

	+ Removed spurious lock in dapl_evd_connection_callb.c that
	  would have caused a deadlock.
	+ The Async EVD was getting torn down too early, potentially
	  causing lost errors. Has been moved later in the teardown
	  process.
	+ kDAPL replaced mem_map_reserve() with newer SetPageReserved()
	  for better Linux integration.
	+ kdapltest no longer allocate large print buffers on the stack,
	  is more careful to ensure buffers don't overflow.
	+ Put dapl_os_dbg_print() under DAPL_DBG conditional, it is
	  supposed to go away in a production build. 
	+ dapltest protocol version has been bumped to reflect the
	  change in the Service ID.
	+ Corrected several instances of routines that did not adhere
	  to the DAT 1.1 error code scheme.
	+ Cleaned up vapi ib_reject_connection to pass DAT types rather
	  than provider specific types. Also cleaned up naming interface
	  declarations and their use in vapi_cm.c; fixed incorrect
	  #ifdef for naming.  
	+ Initialize missing uDAPL provider attr, pz_support.
	+ Changes for better layering: first, moved
	  dapl_lmr_convert_privileges to the provider layer as memory
	  permissions are clearly transport specific and are not always
	  defined in an integer bitfield; removed common routines for
	  lmr and rmr. Second, move init and release setup/teardown
	  routines into adapter_util.h, which defined the provider
	  interface.
	+ Cleaned up the HCA name cruft that allowed different types
	  of names such as strings or ints to be dealt with in common
	  code; but all names are presented by the dat_registry as
	  strings, so pushed conversions down to the provider
	  level. Greatly simplifies names.
	+ Changed deprecated true/false to DAT_TRUE/DAT_FALSE.
	+ Removed old IB_HCA_NAME type in favor of char *.
	+ Fixed race condition in kdapltest's use of dat_evd_dequeue. 
	+ Changed cast for SERVER_PORT_NUMBER to DAT_CONN_QUAL as it
	  should be. 
	+ Small code reorg to put the CNO into the EVD when it is
	  allocated, which simplifies things. 
	+ Removed gratuitous ib_hca_port_t and ib_send_op_type_t types,
	  replaced with standard int.
	+ Pass a pointer to cqe debug routine, not a structure. Some
	  clean up of data types.
	+ kdapl threads now invoke reparent_to_init() on exit to allow
	  threads to get cleaned up.



	DAPL BETA 2.04 RELEASE NOTES

	The big changes for this release involve a more strict adherence
	to the original dapl architecture. Originally, only InfiniBand
	providers were available, so allowing various data types and
	event codes to show through into common code wasn't a big deal.

	But today, there are an increasing number of providers available
	on a number of transports. Requiring an IP iWarp provider to
	match up to InfiniBand events is silly, for example.

	Restructuring the code allows more flexibility in providing an
	implementation.

	There are also a large number of bug fixes available in this
	release, particularly in kdapl related code.

	Be warned that the next release will change every file in the
	tree as we move to the newly approved DAT license. This is a
	small change, but all files are affected.

	Future releases will also support to the soon to be ratified DAT
	1.2 specification.

	This release has benefited from many bug reports and fixes from
	a number of individuals and companies. On behalf of the DAPL
	community, thank you!


	NEW SINCE Beta 2.3

	- Made several changes to be more rigorous on the layering
	  design of dapl. The intent is to make it easier for non
	  InfiniBand transports to use dapl. These changes include:
	  
	  * Revamped the ib_hca_open/close code to use an hca_ptr
	    rather than an ib_handle, giving the transport layer more
	    flexibility in assigning transport handles and resources.

	  * Removed the CQD calls, they are specific to the IBM API;
	    folded this functionality into the provider open/close calls.

	  * Moved VAPI, IBAPI transport specific items into a transport
	    structure placed inside of the HCA structure. Also updated
	    routines using these fields to use the new location. Cleaned
	    up provider knobs that have been exposed for too long.

	  * Changed a number of provider routines to use DAPL structure
	    pointers rather than exposing provider handles & values. Moved
	    provider specific items out of common code, including provider
	    data types (e.g. ib_uint32_t).

	  * Pushed provider completion codes and type back into the
            provider layer. We no longer use EVD or CM completion types at
            the common layer, instead we obtain the appropriate DAT type
            from the provider and process only DAT types.

	  * Change private_data handling such that we can now accommodate
            variable length private data.

	- Remove DAT 1.0 cruft from the DAT header files.

	- Better spec compliance in headers and various routines.

	- Major updates to the VAPI implementation from
          Mellanox. Includes initial kdapl implementation

	- Move kdapl platform specific support for hash routines into
          OSD file.

	- Cleanups to make the code more readable, including comments
          and certain variable and structure names.

	- Fixed CM_BUSTED code so that it works again: very useful for
          new dapl ports where infrastructure is lacking. Also made
	  some fixes for IBHOSTS_NAMING conditional code.

	- Added DAPL_MERGE_CM_DTO as a compile time switch to support
	  EVD stream merging of CM and DTO events. Default is off.

	- 'Quit' test ported to kdapltest

	- uDAPL now builds on Linux 2.6 platform (SuSE 9.1).

	- kDAPL now builds for a larger range of Linux kernels, but
          still lacks 2.6 support.

	- Added shared memory ID to LMR structure. Shared memory is
          still not fully supported in the reference implementation, but
          the common code will appear soon.

	* Bug fixes
	  - Various Makefiles fixed to use the correct dat registry
	    library in its new location (as of Beta 2.03)
	  - Simple reorg of dat headers files to be consistent with
	    the spec.
	  - fixed bug in vapi_dto.h recv macro where we could have an
	    uninitialized pointer.
	  - Simple fix in dat_dr.c to initialize a variable early in the
	    routine before errors occur.
	  - Removed private data pointers from a CONNECTED event, as
	    there should be no private data here.
	  - dat_strerror no longer returns an uninitialized pointer if
	    the error code is not recognized.
	  - dat_dup_connect() will reject 0 timeout values, per the
	    spec.
	  - Removed unused internal_hca_names parameter from
	    ib_enum_hcas() interface. 
	  - Use a temporary DAT_EVENT for kdapl up-calls rather than
	    making assumptions about the current event queue.
	  - Relocated some platform dependent code to an OSD file.
	  - Eliminated several #ifdefs in .c files.
	  - Inserted a missing unlock() on an error path.
	  - Added bounds checking on size of private data to make sure
	    we don't overrun the buffer
	  - Fixed a kdapltest problem that caused a machine to panic if
	    the user hit ^C
	  - kdapltest now uses spin locks more appropriate for their
	    context, e.g. spin_lock_bh or spin_lock_irq. Under a
	    conditional. 
	  - Fixed kdapltest loops that drain EVDs so they don't go into
	    endless loops.
	  - Fixed bug in dapl_llist_add_entry link list code.
	  - Better error reporting from provider code.
	  - Handle case of user trying to reap DTO completions on an
	    EP that has been freed.
	  - No longer hold lock when ep_free() calls into provider layer
	  - Fixed cr_accept() to not have an extra copy of
	    private_data. 
	  - Verify private_data pointers before using them, avoid
	    panic. 
	  - Fixed memory leak in kdapltest where print buffers were not
	    getting reclaimed.



	DAPL BETA 2.03 RELEASE NOTES

	There are some  prominent features in this release:
	1) dapltest/kdapltest. The dapltest test program has been
	   rearchitected such that a kernel version is now available
	   to test with kdapl. The most obvious change is a new
	   directory structure that more closely matches other core
	   dapl software. But there are a large number of changes
	   throughout the source files to accommodate both the
	   differences in udapl/kdapl interfaces, but also more mundane
	   things such as printing.

	   The new dapltest is in the tree at ./test/dapltest, while the
	   old remains at ./test/udapl/dapltest. For this release, we
	   have maintained both versions. In a future release, perhaps
	   the next release, the old dapltest directory will be
	   removed. Ongoing development will only occur in the new tree.

	2) DAT 1.1 compliance. The DAT Collaborative has been busy
	   finalizing the 1.1 revision of the spec. The header files
	   have been reviewed and posted on the DAT Collaborative web
	   site, they are now in full compliance.

	   The reference implementation has been at a 1.1 level for a
	   while. The current implementation has some features that will
	   be part of the 1.2 DAT specification, but only in places
	   where full compatibility can be maintained.

	3) The DAT Registry has undergone some positive changes for
           robustness and support of more platforms. It now has the
           ability to support several identical provider names
           simultaneously, which enables the same dat.conf file to
           support multiple platforms. The registry will open each
           library and return when successful. For example, a dat.conf
           file may contain multiple provider names for ex0a, each
           pointing to a different library that may represent different
           platforms or vendors. This simplifies distribution into
           different environments by enabling the use of common
           dat.conf files.

	In addition, there are a large number of bug fixes throughout
	the code. Bug reports and fixes have come from a number of
	companies.

	Also note that the Release notes are cleaned up, no longer
	containing the complete text of previous releases.

	* EVDs no longer support DTO and CONNECTION event types on the
          same EVD. NOTE: The problem is maintaining the event ordering
          between two channels such that no DTO completes before a
          connection is received; and no DTO completes after a
          disconnect is received. For 90% of the cases this can be made
          to work, but the remaining 10% will cause serious performance
          degradation to get right.

	NEW SINCE Beta 2.2

	* DAT 1.1 spec compliance. This includes some new types, error
          codes, and moving structures around in the header files,
          among other things. Note the Class bits of dat_error.h have
	  returned to a #define (from an enum) to cover the broadest
	  range of platforms.

	* Several additions for robustness, including handle and
          pointer checking, better argument checking, state
          verification, etc. Better recovery from error conditions,
	  and some assert()s have been replaced with 'if' statements to
          handle the error.

	* EVDs now maintain the actual queue length, rather than the
	  requested amount. Both the DAT spec and IB (and other
	  transports) allow the underlying implementation to provide
	  more CQ entries than requested.

	  Requests for the same number of entries contained by an EVD
	  return immediate success.

	* kDAPL enhancements:
	  - module parameters & OS support calls updated to work with
            more recent Linux kernels.
	  - kDAPL build options changes to match the Linux kernel, vastly
	    reducing the size and making it more robust.
	  - kDAPL unload now works properly
	  - kDAPL takes a reference on the provider driver when it
	    obtains a verbs vector, to prevent an accidental unload
	  - Cleaned out all of the uDAPL cruft from the linux/osd files.

	* New dapltest (see above).

	* Added a new I/O trace facility, enabling a developer to debug
          all I/O that are in progress or recently completed. Default
          is OFF in the build.

	* 0 timeout connections now refused, per the spec.

	* Moved the remaining uDAPL specific files from the common/
          directory to udapl/. Also removed udapl files from the kdapl
	  build.

	* Bug fixes
	  - Better error reporting from provider layer  
	  - Fixed race condition on reference counts for posting DTO
	    ops.
	  - Use DAT_COMPLETION_SUPPRESS_FLAG to suppress successful
	    completion of dapl_rmr_bind  (instead of
	    DAT_COMPLEITON_UNSIGNALLED, which is for non-notification
	    completion). 
	  - Verify psp_flags value per the spec
	  - Bug in psp_create_any() checking psp_flags fixed
	  - Fixed type of flags in ib_disconnect from
	    DAT_COMPLETION_FLAGS to DAT_CLOSE_FLAGS
	  - Removed hard coded check for ASYNC_EVD. Placed all EVD
	    prevention in evd_stream_merging_supported array, and
	    prevent ASYNC_EVD from being created by an app.
	  - ep_free() fixed to comply with the spec
	  - Replaced various printfs with dbg_log statements
	  - Fixed kDAPL interaction with the Linux kernel
	  - Corrected phy_register protottype
	  - Corrected kDAPL wait/wakeup synchronization
	  - Fixed kDAPL evd_kcreate() such that it no longer depends
	    on uDAPL only code.
	  - dapl_provider.h had wrong guard #def: changed DAT_PROVIDER_H
	    to DAPL_PROVIDER_H
	  - removed extra (and bogus) call to dapls_ib_completion_notify()
	    in evd_kcreate.c
	  - Inserted missing error code assignment in
	    dapls_rbuf_realloc() 
	  - When a CONNECTED event arrives, make sure we are ready for
	    it, else something bad may have happened to the EP and we
	    just return; this replaces an explicit check for a single
	    error condition, replacing it with the general check for the
	    state capable of dealing with the request.
	  - Better context pointer verification. Removed locks around
	    call to ib_disconnect on an error path, which would result
	    in a deadlock. Added code for BROKEN events.
	  - Brought the vapi code more up to date: added conditional
	    compile switches, removed obsolete __ActivePort, deal
	    with 0 length DTO
	  - Several dapltest fixes to bring the code up to the 1.1
	    specification.
	  - Fixed mismatched dalp_os_dbg_print() #else dapl_Dbg_Print();
	    the latter was replaced with the former.
	  - ep_state_subtype() now includes UNCONNECTED.
	  - Added some missing ibapi error codes.
 


	NEW SINCE Beta 2.1

	* Changes for Erratta and 1.1 Spec
	  - Removed DAT_NAME_NOT_FOUND, per DAT erratta
	  - EVD's with DTO and CONNECTION flags set no longer valid.
	  - Removed DAT_IS_SUCCESS macro
	  - Moved provider attribute structures from vendor files to udat.h
	    and kdat.h
	  - kdapl UPCALL_OBJECT now passed by reference

	* Completed dat_strerr return strings

	* Now support interrupted system calls

	* dapltest now used dat_strerror for error reporting.

	* Large number of files were formatted to meet project standard,
	  very cosmetic changes but improves readability and
	  maintainability.  Also cleaned up a number of comments during
	  this effort.

	* dat_registry and RPM file changes (contributed by Steffen Persvold):
	  - Renamed the RPM name of the registry to be dat-registry 
	    (renamed the .spec file too, some cvs add/remove needed)
	  - Added the ability to create RPMs as normal user (using 
	    temporal paths), works on SuSE, Fedora, and RedHat.
	  - 'make rpm' now works even if you didn't build first.
	  - Changed to using the GNU __attribute__((constructor)) and
	    __attribute__((destructor)) on the dat_init functions, dat_init
	    and dat_fini. The old -init and -fini options to LD makes 
	    applications crash on some platforms (Fedora for example).
	  - Added support for 64 bit platforms.
	  - Added code to allow multiple provider names in the registry,
	    primarily to support ia32 and ia64 libraries simultaneously. 
	    Provider names are now kept in a list, the first successful
	    library open will be the provider.

	* Added initial infrastructure for DAPL_DCNTR, a feature that
	  will aid in debug and tuning of a dapl implementation. Partial
	  implementation only at this point.

	* Bug fixes
	- Prevent debug messages from crashing dapl in EVD completions by
	  verifying the error code to ensure data is valid.
	- Verify CNO before using it to clean up in evd_free()
	- CNO timeouts now return correct error codes, per the spec.
	- cr_accept now complies with the spec concerning connection 
	  requests that go away before the accept is invoked.
	- Verify valid EVD before posting connection evens on active side
	  of a connection. EP locking also corrected.
	- Clean up of dapltest Makefile, no longer need to declare
	  DAT_THREADSAFE
	- Fixed check of EP states to see if we need to disconnect an
	  IA is closed.
	- ep_free() code reworked such that we can properly close a 
	  connection pending EP.
	- Changed disconnect processing to comply with the spec: user will
	   see a BROKEN event, not DISCONNECTED.
	- If we get a DTO error, issue a disconnect to let the CM and
	  the user know the EP state changed to disconnect; checked IBA
	  spec to make sure we disconnect on correct error codes.
	- ep_disconnect now properly deals with abrupt disconnects on the
	  active side of a connection.
	- PSP now created in the correct state for psp_create_any(), making
	  it usable.
	- dapl_evd_resize() now returns correct status, instead of always
	  DAT_NOT_IMPLEMENTED.
	- dapl_evd_modify_cno() does better error checking before invoking
	  the provider layer, avoiding bugs.
	- Simple change to allow dapl_evd_modify_cno() to set the CNO to 
	  NULL, per the spec.
	- Added required locking around call to dapl_sp_remove_cr.

	- Fixed problems related to dapl_ep_free: the new
	  disconnect(abrupt) allows us to do a more immediate teardown of
	  connections, removing the need for the MAGIC_EP_EXIT magic
	  number/state, which has been removed. Mmuch cleanup of paths,
	  and made more robust.
	- Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is
	  triggered if there are waiters when the last EVD is removed
	  or when the IA is freed.
	- Added code to deal with the provider synchronously telling us
	   a connection is unreachable, and generate the appropriate
	   event.
	- Changed timer routine type from unsigned long to uintptr_t
	  to better fit with machine architectures.
	- ep.param data now initialized in ep_create, not ep_alloc.
	- Or Gerlitz provided updates to Mellanox files for evd_resize,
	  fw attributes, many others. Also implemented changes for correct
	  sizes on REP side of a connection request.



	NEW SINCE Beta 2.0

	* dat_echo now DAT 1.1 compliant. Various small enhancements.

	* Revamped atomic_inc/dec to be void, the return value was never
	  used. This allows kdapl to use Linux kernel equivalents, and
	  is a small performance advantage.

	* kDAPL: dapl_evd_modify_upcall implemented and tested.

	* kDAPL: physical memory registration implemented and tested.

	* uDAPL now builds cleanly for non-debug versions.

	* Default RDMA credits increased to 8.

	* Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
	  months).

	* Cleaned up dat_error.h, now 1.1 compliant in comments.

	* evd_resize initial implementation. Untested.

	* Bug fixes
	  - __KDAPL__ is defined in kdat_config.h, so apps don't need
	    to define it.
	  - Changed include file ordering in kdat.h to put kdat_config.h
	    first.
	  - resolved connection/tear-down race on the client side.
	  - kDAPL timeouts now scaled properly; fixed 3 orders of
	    magnitude difference.
	  - kDAPL EVD callbacks now get invoked for all completions; old
	    code would drop them in heavy utilization.
	  - Fixed error path in kDAPL evd creation, so we no longer
	    leak CNOs.
	  - create_psp_any returns correct error code if it can't create
	    a connection qualifier.
	  - lock fix in ibapi disconnect code.
	  - kDAPL INFINITE waits now work properly (non connection
	    waits) 
	  - kDAPL driver unload now works properly
	  - dapl_lmr_[k]create now returns 1.1 error codes
	  - ibapi routines now return DAT 1.1 error codes
	  


	NEW SINCE Beta 1.10

	* kDAPL is now part of the DAPL distribution. See the release
	  notes above.

	  The kDAPL 1.1 spec is now contained in the doc/ subdirectory.

	* Several files have been moved around as part of the kDAPL
	  checkin. Some files that were previously in udapl/ are now
	  in common/, some in common are now in udapl/. The goal was
	  to make sure files are properly located and make sense for
	  the build.

	* Source code formatting changes for consistency.

	* Bug fixes
	  - dapl_evd_create() was comparing the wrong bit combinations,
	    allowing bogus EVDs to be created.
	  - Removed code that swallowed zero length I/O requests, which
	    are allowed by the spec and are useful to applications.
	  - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
	    routine will take and release the lock. Cosmetic change.
	  - dapl_get_consuemr_context() will now verify the pointer
	    argument 'context' is not NULL.




	OBTAIN THE CODE

	To obtain the tree for your local machine you can check it
	out of the source repository using CVS tools. CVS is common
	on Unix systems and available as freeware on Windows machines.
	The command to anonymously obtain the source code from 
	Source Forge (with no password) is:
	
	cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
	cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co .

	When prompted for a password, simply  press the Enter key.

	Source Forge also contains explicit directions on how to become
	a developer, as well as how to use different CVS commands. You may
	also browse the source code using the URL:

	   http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/


	SYSTEM REQUIREMENTS

	This project has been implemented on Red Hat Linux 7.3, SuSE
	SLES 8, Windows 2000, RHEL 3.0, and a couple of other Linux
	distrubutions. The structure of the code is designed to allow
	other operating systems to easily be adapted.

	The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
	development, and continues with this platform. Our HCAs use the
	IB verbs API submitted by IBM. Mellanox has contributed an
	adapter layer using their VAPI verbs API. Either platform is
	available to any group considering DAPL work. The structure of
	the uDAPL source allows other provider API sets to be easily
	integrated.

	The development team uses any one of three topologies: two HCAs
	in a single machine; a single HCA in each of two machines; and
	most commonly, a switch. Machines connected to a switch may have
	more than one HCA.

	The DAPL Plugfest revealed that switches and HCAs available from
	most vendors will interoperate with little trouble, given the
	most recent releases of software. The dapl reference team makes
	no recommendation on HCA or switch vendors.

	Explicit machine configurations are available upon request.	


	IN THE TREE

	The DAPL tree contains source code for the uDAPL and kDAPL
	implementations, and also includes tests and documentation.

	Included documentation has the base level API of the
	providers: the IBM Access API and the Mellanox Verbs API. Also
	included are a growing number of DAPL design documents which
	lead the reader through specific DAPL subsystems. More
	design documents are in progress and will appear in the tree in
	the near future.

	A small number of test applications and a unit test framework
	are also included. dapltest is the primary testing application
	used by the DAPL team, it is capable of simulating a variety of
	loads and exercises a large number of interfaces. Full
	documentation is included for each of the tests.

	Recently, the dapl conformance test has been added to the source
	repository. The test provides coverage of the most common
	interfaces, doing both positive and negative testing. Vendors
	providing DAPL implementation are strongly encouraged to run
	this set of tests.


	MAKEFILE NOTES

	There are a number #ifdef's in the code that were necessary
	during early development. They are disappearing as we
	have time to take advantage of features and work available from
	newer releases of provider software. You may notice an #ifdef
	<something>_BUSTED, which indicates a particular feature was not
	working at the time the code was written and the DAPL team
	developed a work-around.

	These #ifdefs are not documented as the intent is to remove
	them as soon as possible.

	Of particular relevance are the following #defines:

	- CM_BUSTED

	  The DAPL team has been an early adopter of InfiniBand and has
	  had to improvise missing functionality while the vendors lag
	  our development. InfiniBand uses a Connection Manager (CM) to
	  establish a connection between nodes. This #define essentially
	  'fakes' a connection by moving a QP into the appropriate
	  state. Most of the IB vendors have a working CM now and this
	  is no longer the default, but the code remains as some
	  development groups are working to catch up.

	- NO_NAME_SERVICE

	  Naming is a thorny issue in InfiniBand; translating from a
	  hostname or an interface name to a GID that can be used to
	  establish a connection with a remote machine. The reference
	  implementation provides a simple name service under this
	  #define. The goal is to use IPoIB when it becomes
	  available. NO_NAME_SERVICE will probably remain in the code
	  long term in order to enable various implementations. A
	  description of how this works is found in the end_point_design
	  document in the doc/ directory.


	CONTRIBUTIONS

	As is common to Source Forge projects, there are a small number
	of developers directly associated with the source tree and having
	privileges to change the tree. Requested updates, changes, bug
	fixes, enhancements, or contributions should be sent to Steve
	Sears at sj...@ne... for review. We welcome your
	contributions and expect the quality of the project will
	improve thanks to your help.

	The core DAPL team is:

	  Steve Sears
	  Philip Christopher

	  ... with contributions from a number of excellent engineers in
	  various companies contributing to the open source effort.


	ONGOING WORK

	Not all of the DAPL spec is implemented at this time.
	Functionality such as shared memory will probably not be
	implemented by the reference implementation (there is a write up
	on this in the doc/ area), and there are yet various cases where
	work remains to be done.  And of course, not all of the
	implemented functionality has been tested yet.  The DAPL team
	continues to develop and test the tree with the intent of
	completing the specification and delivering a robust and useful
	implementation.


The DAPL Team

[Dapl-announce] DAT API for OpenIB stack

[James L. <jle...@ne...>]

An open source implementation of the DAT API for the OpenIB stack 
(gen2) is available in the OpenIB subversion repository at

https://openib.org/svn/gen2/users/jlentini/

If you are interested in learning more about the DAT API, please see

http://www.datcollaborative.org/

or

http://sourceforge.net/projects/dapl

The DAT kernel code on OpenIB is a native Linux implementation and the 
provider is only intended to be used on the OpenIB stack. This is in 
contrast to the SourceForge code that is intended for use on multiple 
operating systems and verbs APIs.

If you are interested in using the code, please see the README files 
at:

https://openib.org/svn/gen2/users/jlentini/linux-kernel/README
https://openib.org/svn/gen2/users/jlentini/userspace/dapl/openib/README

The code is still in the early stages of development. Currently we are 
testing the code with the "[k]dapltest" tool (available in the 
source). Our goal is to be able to run all 6 of dapltest invocations 
in the linux-kernel/test/dapltest/scripts/kregress.sh script reliably. 
After obtaining that level of stability, we will begin testing the 
code with an NFS/RDMA client and server.

A number of people have contributed to the DAT code in OpenIB. In 
particular I'd like to thank Arlin Davis from Intel, Tom Duffy from 
Sun, Itamar Rabenstein from Mellanox, and Hal Rosenstock from 
Voltaire.

Any help would be appreciated. If anyone would like to volunteer 
please let me know.

james

James Lentini                email: jle...@ne...
Network Appliance            phone: 781-768-5359
375 Totten Pond Rd.            fax: 781-895-1195
Waltham, MA 02451-2010        main: 781-768-5300

[Dapl-announce] new dapl patch process

[James L. <jle...@ne...>]

I've created a document describing a new process for submitting 
patches to the DAPL Reference Implementation. The goal is to 
decrease the time necessary to process feature improvements and bug 
fixes, and to publicly acknowledge the contributors.

The document is located at docs/dapl_patch.txt in CVS and at 
http://dapl.sourceforge.net/dapl_patch.txt on the web.

Please let me know if you have any questions,
james

James Lentini                email: jle...@ne...
Network Appliance            phone: 781-768-5359
375 Totten Pond Rd.            fax: 781-895-1195
Waltham, MA 02451-2010        main: 781-768-5300

[Dapl-announce] Re: Porting DAPL to OpenIB Gen 2

[James L. <jle...@ne...>]

As promised during the call today, here is a link to the patches Tom
Duffy sent:

http://sourceforge.net/mailarchive/forum.php?thread_id=6549863&forum_id=13070


On Mon, 28 Feb 2005, James Lentini wrote:

>
> Below are my meeting notes:
>
> Porting DAPL to OpenIB
> ======================
> Meeting Notes
> Date: 2/28/2005
> Time: 11:30 AM EST
>
> Attendees:
> James Lentini       jle...@ne...
> Dror Goldenberg     gd...@me...
> Itamar Rabenstein   it...@me...
> Arkady Kanevsky     ar...@ne...
> Hal Rosenstock      ha...@vo...
> Aviram Gutman       av...@me...
> Arlin Davis         arl...@in...
> Tom Duffy           td...@su...
> Yaron Haviv         ya...@vo...
> Or Gerlitz          oge...@vo...
> Steve Sears         sj...@ne...
> Bob Woodruff        rob...@in... Tom Talpey 
> ta...@ne...
>
> + Enumerate Tasks:
>  - uDAPL
>    who: Arlin Davis at Intel
>
>    No additional volunteers.
>
>    We agreed to keep SourceForge structure (provider and platform
>    abstraction layers) for uDAPL. Openib user space software lacks
>    a CM and name resolution APIs. uDAPL will need to work around
>    these deficiencies (by hard coding connections, etc.) at first.
>
>    uDAPL will require kernel support for mapping the cookies used
>    by shared registrations.
>
>  - kDAPL
>    who: Itamar Rabenstein at Mellanox
>         Hal Rosenstock at Voltaire
>
>   Initially we will work with current verbs abstraction layer. Once
>   OpenIB support is functional and stable, we will begin removing
>   the verbs and platform abstraction layers.
>
>   MTHCA does not support all of DAPL required features (memory
>   windows, virtual memory registration, etc.). When will these
>   be supported?
>
>   [AI] James Lentini: inquire about above
>
>  - name resolution interface
>    who: ?
>
>    The kernel APIs are evolving. User will be done second.
>
>  - Linux coding style
>    who: everyone
>
>    Documentation is in every linux kernel in
>    "Documentation/CodingStyle" and "Documentation/SubmittingPatches"
>    Tom Duffy gave James Lentini an overview of the types of changes
>    at the OpenIB workshop. No apparent interest in making a single
>    individual responsible for this. We expect to receive feedback
>    from the community once code is in OpenIB tree.
>
>  - kDAPL API changes for Linux coding style
>    who: James Lentini and Arkady Kanevsky at NetApp
>
>    First step is changing header files. We would like to keep
>    old code from breaking while updates are being made. One option is
>    to use a temporary set of typedefs while the headers are in flux.
>
>    [AI] James Lentini: send Tom Duffy's patches to attendees
>
>  - tools (dapltest, dat conformance test, etc.)
>    who: volunteers?
>
>    DAT Conformance test needs recode. No volunteers to do this work.
>
>  - documentation (user guide, design guides, etc.)
>    who: volunteers? everyone?
>
>    We only touched on this briefly as time was running out. James
>    would like written record of design decisions for posterity.
>
> + Appropriate location for DAPL source code in OpenIB tree.
>  Split kernel and user space code? Split registry and provider?
>
>  Do we want to split kernel and user space code? Majority are in
>  favor of split. License issues are simpler (i.e. if the OpenIB
>  community ever wanted to add the LGPL for user components this would be
>  feasible). Intel's experience with past Linux projects suggest that
>  this is the correct decision. Keeping them together would mean lots
>  of kernel changes affect user code.
>
>  We will keep the registry code in the same general location as the
>  provider code. If other RDMA transports pop up, we will adjust this.
>
>  Initially, we will put DAPL code on a branch and move to mainline
>  when it is ready. Initial OpenIB checkin will be based on the soon
>  to be created DAPL Gamma 3.1 drop (this drop will be Gamma 3.0 +
>  a few fixes + GPL option).
>
>  [AI] James Lentini: create tar of current CVS and send to Tom,
>       Arlin, Itamar, and Hal. They will get started with this while
>       Gamma 3.1 is being prepared.
>
> + Procedure for checkins
>
> Use the same procedure for other parts of the OpenIB tree. Submitter
> will send an email containing patch to ope...@op... and cc'ing 
> James Lenitni (jle...@ne...). Submitter should indicate that it is for 
> dapl by placing the text "[DAPL]" in the subject.
>
> NOTE: We didn't discuss this on the phone, but checkins should also
> have a "signed-off by" line (in accordance with OpenIB practices).
>
> + SourceForge synchronization
>
> Proposal would be to have patches sent to both OpenIB and SourceForge.
> The files in OpenIB will have 2 licenses and the SourceForge will have 3 
> licenses.
>
> Little enthusiasm for this on the phone. Two counter proposals
> made:
>
> 1) submit code with CPL license 2) re-license code with CPL
>
> [AI] Tom Duffy: At the next OpenIB board meeting, ask if it is a
>     violation of OpenIB bylaws to have dapl files licensed under
>     a third license (CPL)? [AI] James Lentini: try to attend board meeting 
> to answer questions.
> [AI] James Lentini: investigate option #2 with lawyers. Note: early
>     indications are that this is not feasible.
>
>
>
> On Mon, 28 Feb 2005, James Lentini wrote:
>
>> 
>> Below is an agenda for the call:
>> 
>> + Additional Items?
>> 
>> + Enumerate Tasks:
>> - uDAPL
>>   who: Arlin Davis at Intel
>> 
>>  - kDAPL
>>    who: Itamar Rabenstein at Mellanox
>>         Voltaire for CM related features
>> 
>>  - Linux coding style
>>    who: everyone
>> 
>>  - kDAPL API changes for Linux coding style
>>    who: James Lentini and Arkady Kanevsky at NetApp
>> 
>>  - tools (dapltest, dat conformance test, etc.)
>>    who: volunteers?
>> 
>>  - documentation (user guide, design guides, etc.)
>>    who: volunteers? everyone?
>> 
>> + Appropriate location for DAPL source code in OpenIB tree.
>>  Split kernel and user space code? Split registry and provider?
>> 
>> + Procedure for checkins
>> 
>> + SourceForge synchronization
>> 
>> 
>> On Thu, 24 Feb 2005, James Lentini wrote:
>> 
>>> 
>>> Yesterday the DAT Collaborative voted to add the GPL license to the DAPL 
>>> Source Forge reference implementation. We are now in a position to begin 
>>> porting the DAPL SourceForge project to OpenIB. I could like to hold a 
>>> conference call to help plan and divide up this work.
>>> 
>>> 
>>>         Date: Monday, February 28
>>>         Time: 11:30 AM EST
>>>     Domestic: 888-827-8686
>>> International: 303-928-2620
>>> Conference ID: 1125043
>>> 
>>> 
>>> James Lentini                email: jle...@ne...
>>> Network Appliance            phone: 781-768-5359
>>> 375 Totten Pond Rd.            fax: 781-895-1195
>>> Waltham, MA 02451-2010        main: 781-768-5300
>>> 
>>> 
>> 
>

[Dapl-announce] Re: Porting DAPL to OpenIB Gen 2

[James L. <jle...@ne...>]

Below are my meeting notes:

Porting DAPL to OpenIB
======================
Meeting Notes
Date: 2/28/2005
Time: 11:30 AM EST

Attendees:
James Lentini       jle...@ne...
Dror Goldenberg     gd...@me...
Itamar Rabenstein   it...@me...
Arkady Kanevsky     ar...@ne...
Hal Rosenstock      ha...@vo...
Aviram Gutman       av...@me...
Arlin Davis         arl...@in...
Tom Duffy           td...@su...
Yaron Haviv         ya...@vo...
Or Gerlitz          oge...@vo...
Steve Sears         sj...@ne...
Bob Woodruff        rob...@in... 
Tom Talpey          ta...@ne...

+ Enumerate Tasks:
   - uDAPL
     who: Arlin Davis at Intel

     No additional volunteers.

     We agreed to keep SourceForge structure (provider and platform
     abstraction layers) for uDAPL. Openib user space software lacks
     a CM and name resolution APIs. uDAPL will need to work around
     these deficiencies (by hard coding connections, etc.) at first.

     uDAPL will require kernel support for mapping the cookies used
     by shared registrations.

   - kDAPL
     who: Itamar Rabenstein at Mellanox
          Hal Rosenstock at Voltaire

    Initially we will work with current verbs abstraction layer. Once
    OpenIB support is functional and stable, we will begin removing
    the verbs and platform abstraction layers.

    MTHCA does not support all of DAPL required features (memory
    windows, virtual memory registration, etc.). When will these
    be supported?

    [AI] James Lentini: inquire about above

   - name resolution interface
     who: ?

     The kernel APIs are evolving. User will be done second.

   - Linux coding style
     who: everyone

     Documentation is in every linux kernel in
     "Documentation/CodingStyle" and "Documentation/SubmittingPatches"
     Tom Duffy gave James Lentini an overview of the types of changes
     at the OpenIB workshop. No apparent interest in making a single
     individual responsible for this. We expect to receive feedback
     from the community once code is in OpenIB tree.

   - kDAPL API changes for Linux coding style
     who: James Lentini and Arkady Kanevsky at NetApp

     First step is changing header files. We would like to keep
     old code from breaking while updates are being made. One option is
     to use a temporary set of typedefs while the headers are in flux.

     [AI] James Lentini: send Tom Duffy's patches to attendees

   - tools (dapltest, dat conformance test, etc.)
     who: volunteers?

     DAT Conformance test needs recode. No volunteers to do this work.

   - documentation (user guide, design guides, etc.)
     who: volunteers? everyone?

     We only touched on this briefly as time was running out. James
     would like written record of design decisions for posterity.

+ Appropriate location for DAPL source code in OpenIB tree.
   Split kernel and user space code? Split registry and provider?

   Do we want to split kernel and user space code? Majority are in
   favor of split. License issues are simpler (i.e. if the OpenIB
   community ever wanted to add the LGPL for user components this would be
   feasible). Intel's experience with past Linux projects suggest that
   this is the correct decision. Keeping them together would mean lots
   of kernel changes affect user code.

   We will keep the registry code in the same general location as the
   provider code. If other RDMA transports pop up, we will adjust this.

   Initially, we will put DAPL code on a branch and move to mainline
   when it is ready. Initial OpenIB checkin will be based on the soon
   to be created DAPL Gamma 3.1 drop (this drop will be Gamma 3.0 +
   a few fixes + GPL option).

   [AI] James Lentini: create tar of current CVS and send to Tom,
        Arlin, Itamar, and Hal. They will get started with this while
        Gamma 3.1 is being prepared.

+ Procedure for checkins

Use the same procedure for other parts of the OpenIB tree. Submitter
will send an email containing patch to ope...@op... and 
cc'ing James Lenitni (jle...@ne...). Submitter should indicate 
that it is for dapl by placing the text "[DAPL]" in the subject.

NOTE: We didn't discuss this on the phone, but checkins should also
have a "signed-off by" line (in accordance with OpenIB practices).

+ SourceForge synchronization

Proposal would be to have patches sent to both OpenIB and SourceForge.
The files in OpenIB will have 2 licenses and the SourceForge will have 
3 licenses.

Little enthusiasm for this on the phone. Two counter proposals
made:

1) submit code with CPL license 
2) re-license code with CPL

[AI] Tom Duffy: At the next OpenIB board meeting, ask if it is a
      violation of OpenIB bylaws to have dapl files licensed under
      a third license (CPL)? 
[AI] James Lentini: try to attend board meeting to answer questions.
[AI] James Lentini: investigate option #2 with lawyers. Note: early
      indications are that this is not feasible.



On Mon, 28 Feb 2005, James Lentini wrote:

>
> Below is an agenda for the call:
>
> + Additional Items?
>
> + Enumerate Tasks:
> - uDAPL
>   who: Arlin Davis at Intel
>
>  - kDAPL
>    who: Itamar Rabenstein at Mellanox
>         Voltaire for CM related features
>
>  - Linux coding style
>    who: everyone
>
>  - kDAPL API changes for Linux coding style
>    who: James Lentini and Arkady Kanevsky at NetApp
>
>  - tools (dapltest, dat conformance test, etc.)
>    who: volunteers?
>
>  - documentation (user guide, design guides, etc.)
>    who: volunteers? everyone?
>
> + Appropriate location for DAPL source code in OpenIB tree.
>  Split kernel and user space code? Split registry and provider?
>
> + Procedure for checkins
>
> + SourceForge synchronization
>
>
> On Thu, 24 Feb 2005, James Lentini wrote:
>
>> 
>> Yesterday the DAT Collaborative voted to add the GPL license to the DAPL 
>> Source Forge reference implementation. We are now in a position to begin 
>> porting the DAPL SourceForge project to OpenIB. I could like to hold a 
>> conference call to help plan and divide up this work.
>> 
>> 
>>         Date: Monday, February 28
>>         Time: 11:30 AM EST
>>     Domestic: 888-827-8686
>> International: 303-928-2620
>> Conference ID: 1125043
>> 
>> 
>> James Lentini                email: jle...@ne...
>> Network Appliance            phone: 781-768-5359
>> 375 Totten Pond Rd.            fax: 781-895-1195
>> Waltham, MA 02451-2010        main: 781-768-5300
>> 
>> 
>

[Dapl-announce] Re: Porting DAPL to OpenIB Gen 2

[James L. <jle...@ne...>]

Below is an agenda for the call:

+ Additional Items?

+ Enumerate Tasks:
  - uDAPL
    who: Arlin Davis at Intel

   - kDAPL
     who: Itamar Rabenstein at Mellanox
          Voltaire for CM related features

   - Linux coding style
     who: everyone

   - kDAPL API changes for Linux coding style
     who: James Lentini and Arkady Kanevsky at NetApp

   - tools (dapltest, dat conformance test, etc.)
     who: volunteers?

   - documentation (user guide, design guides, etc.)
     who: volunteers? everyone?

+ Appropriate location for DAPL source code in OpenIB tree.
   Split kernel and user space code? Split registry and provider?

+ Procedure for checkins

+ SourceForge synchronization


On Thu, 24 Feb 2005, James Lentini wrote:

>
> Yesterday the DAT Collaborative voted to add the GPL license to the DAPL 
> Source Forge reference implementation. We are now in a position to begin 
> porting the DAPL SourceForge project to OpenIB. I could like to hold a 
> conference call to help plan and divide up this work.
>
>
>         Date: Monday, February 28
>         Time: 11:30 AM EST
>     Domestic: 888-827-8686
> International: 303-928-2620
> Conference ID: 1125043
>
>
> James Lentini                email: jle...@ne...
> Network Appliance            phone: 781-768-5359
> 375 Totten Pond Rd.            fax: 781-895-1195
> Waltham, MA 02451-2010        main: 781-768-5300
>
>

Re: [Dapl-announce] Porting DAPL to OpenIB Gen 2

[Masanori I. <it...@nt...>]

Hi all,

If possible, I'm also going to join the conf-call.
# But, in Japan it's 1:30AM of March 1, and I'm not sure. :(

Regards,
Masanori

From: James Lentini <jle...@ne...>
Subject: [Dapl-announce] Porting DAPL to OpenIB Gen 2
Date: Thu, 24 Feb 2005 12:30:10 -0500 (EST)

> 
> Yesterday the DAT Collaborative voted to add the GPL license to the 
> DAPL Source Forge reference implementation. We are now in a position 
> to begin porting the DAPL SourceForge project to OpenIB. I could like 
> to hold a conference call to help plan and divide up this work.
> 
> 
>           Date: Monday, February 28
>           Time: 11:30 AM EST
>       Domestic: 888-827-8686
> International: 303-928-2620
> Conference ID: 1125043

---
Masanori ITOH  Open Source Software Development Center, NTT DATA CORPORATION
               e-mail: it...@nt...
               phone : +81-3-3523-8122 (ext. 172-7199)

Re: [Dapl-announce] Porting DAPL to OpenIB Gen 2

[James L. <jle...@ne...>]

> Hi,
>
> A couple of quick questions ...
>
> If you were not using a GPL license then ... which license were you using
> for the DAPL ref implementation?

Previously users had the choice of either BSD or CPL. The DAT 
collaborative has added the GPL as a third option.

> Some quotes from the OpenIB.org web site ...
>
> "The  OpenIB  server houses many of the development tools that users of the
> project  will require to meet the goals of an open source InfiniBand stack.
> These   resources  include  a  Subversion  code  repository,  Bugzilla  bug
> reporting,  and  mailing  lists  with  monthly archives. The website itself
> provides  links  and  information  to  the  various members of the project,
> downloads, IRC details, and the latest news on the open InfiniBand stack"
>
> Does  this  mean  the DAPL project still be hosted by the OpenIB server ...
> not Source Forge?

The DAPL project on SourceForge will continue. The OpenIB tree will 
contain a DAPL designed specifically for the OpenIB verbs on Linux.

> "OpenIB.org is the premier site for the latest information, updates, and
> downloads to the open-source InfiniBand (IB) effort. OpenIB is focused on
> delivering a high performance InfiniBand software stack for Linux. "
>
> Does this mean that the DAPL project will be a singular Linux effort.

The DAPL API will continue to support other platforms in addition to 
Linux.

> Porting the project to OpenIB ... I readily admit that I don't  totally
> understand the significance.   Could you clarify a bit.

The DAPL reference implementation can be run on either IBAPI, VAPI, or 
OpenIB Gen 1 verbs. We are now starting a project to add OpenIB Gen 2 
to the list. At the same time, we will re-format the kernel code to 
meet Linux kernel coding standards.

>
> Thanks!
>
> Don Mulvey
> IBM Austin
> (512) 838-1787

Re: [Dapl-announce] Porting DAPL to OpenIB Gen 2

[Don M. <dlm...@us...>]

Hi,

A couple of quick questions ...

If you were not using a GPL license then ... which license were you using
for the DAPL ref implementation?

Some quotes from the OpenIB.org web site ...

"The  OpenIB  server houses many of the development tools that users of the
project  will require to meet the goals of an open source InfiniBand stack.
These   resources  include  a  Subversion  code  repository,  Bugzilla  bug
reporting,  and  mailing  lists  with  monthly archives. The website itself
provides  links  and  information  to  the  various members of the project,
downloads, IRC details, and the latest news on the open InfiniBand stack"

Does  this  mean  the DAPL project still be hosted by the OpenIB server ...
not Source Forge?

"OpenIB.org is the premier site for the latest information, updates, and
downloads to the open-source InfiniBand (IB) effort. OpenIB is focused on
delivering a high performance InfiniBand software stack for Linux. "

Does this mean that the DAPL project will be a singular Linux effort.

Porting the project to OpenIB ... I readily admit that I don't  totally
understand the significance.   Could you clarify a bit.

Thanks!

Don Mulvey
IBM Austin
(512) 838-1787



                                                                           
             James Lentini                                                 
             <jlentini@netapp.                                             
             com>                                                       To 
             Sent by:                  ope...@op...,          
             dapl-announce-adm         dap...@li... 
             in...@li...                                          cc 
             rge.net                   David Ford <df...@ne...>,      
                                       Arkady Kanevsky                     
                                       <ar...@ne...>,                
             02/24/2005 11:30          ya...@vo...,                
             AM                        arl...@in...,            
                                       rob...@in...,        
                                       oge...@vo...,              
                                       it...@me...,              
                                       ha...@vo...                   
                                                                   Subject 
                                       [Dapl-announce] Porting DAPL to     
                                       OpenIB Gen 2                        
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           





Yesterday the DAT Collaborative voted to add the GPL license to the
DAPL Source Forge reference implementation. We are now in a position
to begin porting the DAPL SourceForge project to OpenIB. I could like
to hold a conference call to help plan and divide up this work.


          Date: Monday, February 28
          Time: 11:30 AM EST
      Domestic: 888-827-8686
International: 303-928-2620
Conference ID: 1125043


James Lentini                email: jle...@ne...
Network Appliance            phone: 781-768-5359
375 Totten Pond Rd.            fax: 781-895-1195
Waltham, MA 02451-2010        main: 781-768-5300


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Dapl-announce mailing list
Dap...@li...
https://lists.sourceforge.net/lists/listinfo/dapl-announce

[Dapl-announce] Porting DAPL to OpenIB Gen 2

[James L. <jle...@ne...>]

Yesterday the DAT Collaborative voted to add the GPL license to the 
DAPL Source Forge reference implementation. We are now in a position 
to begin porting the DAPL SourceForge project to OpenIB. I could like 
to hold a conference call to help plan and divide up this work.


          Date: Monday, February 28
          Time: 11:30 AM EST
      Domestic: 888-827-8686
International: 303-928-2620
Conference ID: 1125043


James Lentini                email: jle...@ne...
Network Appliance            phone: 781-768-5359
375 Totten Pond Rd.            fax: 781-895-1195
Waltham, MA 02451-2010        main: 781-768-5300

[Dapl-announce] uDAPL/kDAPL Gamma 3.0 now available

[James L. <jle...@ne...>]

 		   Release Notes for
 		Gamma 3.0 uDAPL/kDAPL Release
 		   February 15, 2005


         DAPL GAMMA 3.0 RELEASE NOTES

         This is the first release based on version 1.2 of the spec. There
         are some components, such a shared receive queues (SRQs), which
         are not implemented yet.

         Once again there were numerous bug fixes submitted by the
         DAPL community.

         NEW SINCE Beta 2.06

         - DAT 1.2 headers
         - DAT_IA_HANDLEs implemented as small integers
 	- Changed default device name to be "ia0a"
         - Initial support for Linux 2.6.X kernels
         - Updates to the OpenIB gen 1 provider

         * BUG FIXES

         + Updated Makefile for differentiation between OS releases.
         + Updated atomic routines to use appropriate API
         + Removed unnecessary assert from atomic_dec.
         + Fixed bugs when freeing a PSP.
         + Fixed error codes returned by the DAT static registry.
         + Kernel updates for dat_strerror.
         + Cleaned up the transport layer/adapter interface to use DAPL
           types rather than transport types.
         + Fixed ring buffer reallocation.
         + Removed old test/udapl/dapltest directory.
         + Fixed DAT_IA_HANDLE translation (from pointer to int and
           vice versa) on 64-bit platforms.

 	DAP BETA 2.06 RELEASE NOTES

 	We are not planning any further releases of the Beta series,
 	which are based  on the 1.1 version of the spec. There may be
 	further releases for bug fixes, but we anticipate the DAPL
 	community to move to the new 1.2 version of the spec and the
 	changes mandated in the reference implementation.

 	The biggest item in this release is the first inclusion of the
 	OpenIB Gen 1 provider, an item generating a lot of interest in
 	the IB community. This implementation has graciously been
 	provided by the Mellanox team. The kdapl implementation is in
 	progress, and we imagine work will soon begin on Gen 2.

 	There are also a handful of bug fixes available, as well as a long
 	awaited update to the endpoint design document.

 	NEW SINCE Beta 2.05

 	- OpenIB gen 1 provider support has been added
 	- Added dapls_evd_post_generic_event(), routine to post generic
 	  event types as requested by some providers. Also cleaned up
 	  error reporting.
 	- Updated the endpoint design document in the doc/ directory.

 	* BUG FIXES

 	+ Cleaned up memory leak on close by freeing the HCA structure;
 	+ Removed bogus #defs for rdtsc calls on IA64.
 	+ Changed daptest thread types to use internal types for
 	  portability & correctness
 	+ Various 64 bit enhancements & updates
 	+ Fixes to conformance test that were defining CONN_QUAL twice
 	  and using it in different ways
 	+ Cleaned up private data handling in ep_connect & provider
 	  support: we now avoid extra copy in connect code; reduced
 	  stack requirements by using private_data structure in the EP;
 	  removed provider variable.
 	+ Fixed problem in the dat conformance test where cno_wait would
 	  attempt to dereference a timer value and SEGV.
 	+ Removed old vestiges of depricated POLLING_COMPLETIONS
 	  conditionals.

 	DAPL BETA 2.05 RELEASE NOTES

 	This was to be a very minor release, the primary change was
 	going to be the new wording of the DAT license as contained in
 	the header for all source files. But the interest and
 	development occurring in DAPL provided some extra bug fixes, and
 	some new functionality that has been requested for a while.

 	First, you may notice that every single source file was
 	changed. If you read the release notes from DAPL BETA 2.04, you
 	were warned this would happen. There was a legal issue with the
 	wording in the header, the end result was that every source file
 	was required to change the word 'either of' to 'both'. We've
 	been putting this change off as long as possible, but we wanted
 	to do it in a clean drop before we start working on DAT 1.2
 	changes in the reference implementation, just to keep things
 	reasonably sane.

 	kdapltest has enabled three of the subtests supported by
 	dapltest. The Performance test in particular has been very
 	useful to dapltest in getting minima and maxima. The Limit test
 	pushes the limits by allocating the maximum number of specific
 	resources. And the FFT tests are also available.

 	Most vendors have supported shared memory regions for a while,
 	several of which have asked the reference implementation team to
 	provide a common implementation. Shared memory registration has
 	been tested on ibapi, and compiled into vapi. Both InfiniBand
 	providers have the restriction that a memory region must be
 	created before it can be shared; not all RDMA APIs are this way,
 	several allow you to declare a memory region shared when it is
 	registered. Hence, details of the implementation are hidden in
 	the provider layer, rather than forcing other APIs to do
 	something strange.

 	This release also contains some changes that will allow dapl to
 	work on Opteron processors, as well as some preliminary support
 	for Power PC architecture. These features are not well tested
 	and may be incomplete at this time.

 	Finally, we have been asked several times over the course of the
 	project for a canonical interface between the common and
 	provider layers. This release includes a dummy provider to meet
 	that need. Anyone should be able to download the release and do
 	a:
 	   make VERBS=DUMMY

 	And have a cleanly compiled dapl library. This will be useful
 	both to those porting new transport providers, as well as those
 	going to new machines.

 	The DUMMY provider has been compiled on both Linux and Windows
 	machines.


 	NEW SINCE Beta 2.4
 	- kdapltest enhancements:
 	  * Limit subtests now work
 	  * Performance subtests now work.
 	  * FFT tests now work.

 	- The VAPI headers have been refreshed by Mellanox

 	- Initial Opteron and PPC support.

 	- Atomic data types now have consistent treatment, allowing us to
 	  use native data types other than integers. The Linux kdapl
 	  uses atomic_t, allowing dapl to use the kernel macros and
 	  eliminate the assembly code in dapl_osd.h

 	- The license language was updated per the direction of the
 	  DAT Collaborative. This two word change affected the header
 	  of every file in the tree.

 	- SHARED memory regions are now supported.

 	- Initial support for the TOPSPIN provider.

 	- Added a dummy provider, essentially the NULL provider. It's
 	  purpose is to aid in porting and to clarify exactly what is
 	  expected in a provider implementation.

 	- Removed memory allocation from the DTO path for VAPI

 	- cq_resize will now allow the CQ to be resized smaller. Not all
 	  providers support this, but it's a provider problem, not a
 	  limitation of the common code.

 	* BUG FIXES

 	+ Removed spurious lock in dapl_evd_connection_callb.c that
 	  would have caused a deadlock.
 	+ The Async EVD was getting torn down too early, potentially
 	  causing lost errors. Has been moved later in the teardown
 	  process.
 	+ kDAPL replaced mem_map_reserve() with newer SetPageReserved()
 	  for better Linux integration.
 	+ kdapltest no longer allocate large print buffers on the stack,
 	  is more careful to ensure buffers don't overflow.
 	+ Put dapl_os_dbg_print() under DAPL_DBG conditional, it is
 	  supposed to go away in a production build.
 	+ dapltest protocol version has been bumped to reflect the
 	  change in the Service ID.
 	+ Corrected several instances of routines that did not adhere
 	  to the DAT 1.1 error code scheme.
 	+ Cleaned up vapi ib_reject_connection to pass DAT types rather
 	  than provider specific types. Also cleaned up naming interface
 	  declarations and their use in vapi_cm.c; fixed incorrect
 	  #ifdef for naming.
 	+ Initialize missing uDAPL provider attr, pz_support.
 	+ Changes for better layering: first, moved
 	  dapl_lmr_convert_privileges to the provider layer as memory
 	  permissions are clearly transport specific and are not always
 	  defined in an integer bitfield; removed common routines for
 	  lmr and rmr. Second, move init and release setup/teardown
 	  routines into adapter_util.h, which defined the provider
 	  interface.
 	+ Cleaned up the HCA name cruft that allowed different types
 	  of names such as strings or ints to be dealt with in common
 	  code; but all names are presented by the dat_registry as
 	  strings, so pushed conversions down to the provider
 	  level. Greatly simplifies names.
 	+ Changed deprecated true/false to DAT_TRUE/DAT_FALSE.
 	+ Removed old IB_HCA_NAME type in favor of char *.
 	+ Fixed race condition in kdapltest's use of dat_evd_dequeue.
 	+ Changed cast for SERVER_PORT_NUMBER to DAT_CONN_QUAL as it
 	  should be.
 	+ Small code reorg to put the CNO into the EVD when it is
 	  allocated, which simplifies things.
 	+ Removed gratuitous ib_hca_port_t and ib_send_op_type_t types,
 	  replaced with standard int.
 	+ Pass a pointer to cqe debug routine, not a structure. Some
 	  clean up of data types.
 	+ kdapl threads now invoke reparent_to_init() on exit to allow
 	  threads to get cleaned up.



 	DAPL BETA 2.04 RELEASE NOTES

 	The big changes for this release involve a more strict adherence
 	to the original dapl architecture. Originally, only InfiniBand
 	providers were available, so allowing various data types and
 	event codes to show through into common code wasn't a big deal.

 	But today, there are an increasing number of providers available
 	on a number of transports. Requiring an IP iWarp provider to
 	match up to InfiniBand events is silly, for example.

 	Restructuring the code allows more flexibility in providing an
 	implementation.

 	There are also a large number of bug fixes available in this
 	release, particularly in kdapl related code.

 	Be warned that the next release will change every file in the
 	tree as we move to the newly approved DAT license. This is a
 	small change, but all files are affected.

 	Future releases will also support to the soon to be ratified DAT
 	1.2 specification.

 	This release has benefited from many bug reports and fixes from
 	a number of individuals and companies. On behalf of the DAPL
 	community, thank you!


 	NEW SINCE Beta 2.3

 	- Made several changes to be more rigorous on the layering
 	  design of dapl. The intent is to make it easier for non
 	  InfiniBand transports to use dapl. These changes include:

 	  * Revamped the ib_hca_open/close code to use an hca_ptr
 	    rather than an ib_handle, giving the transport layer more
 	    flexibility in assigning transport handles and resources.

 	  * Removed the CQD calls, they are specific to the IBM API;
 	    folded this functionality into the provider open/close calls.

 	  * Moved VAPI, IBAPI transport specific items into a transport
 	    structure placed inside of the HCA structure. Also updated
 	    routines using these fields to use the new location. Cleaned
 	    up provider knobs that have been exposed for too long.

 	  * Changed a number of provider routines to use DAPL structure
 	    pointers rather than exposing provider handles & values. Moved
 	    provider specific items out of common code, including provider
 	    data types (e.g. ib_uint32_t).

 	  * Pushed provider completion codes and type back into the
             provider layer. We no longer use EVD or CM completion types at
             the common layer, instead we obtain the appropriate DAT type
             from the provider and process only DAT types.

 	  * Change private_data handling such that we can now accommodate
             variable length private data.

 	- Remove DAT 1.0 cruft from the DAT header files.

 	- Better spec compliance in headers and various routines.

 	- Major updates to the VAPI implementation from
           Mellanox. Includes initial kdapl implementation

 	- Move kdapl platform specific support for hash routines into
           OSD file.

 	- Cleanups to make the code more readable, including comments
           and certain variable and structure names.

 	- Fixed CM_BUSTED code so that it works again: very useful for
           new dapl ports where infrastructure is lacking. Also made
 	  some fixes for IBHOSTS_NAMING conditional code.

 	- Added DAPL_MERGE_CM_DTO as a compile time switch to support
 	  EVD stream merging of CM and DTO events. Default is off.

 	- 'Quit' test ported to kdapltest

 	- uDAPL now builds on Linux 2.6 platform (SuSE 9.1).

 	- kDAPL now builds for a larger range of Linux kernels, but
           still lacks 2.6 support.

 	- Added shared memory ID to LMR structure. Shared memory is
           still not fully supported in the reference implementation, but
           the common code will appear soon.

 	* Bug fixes
 	  - Various Makefiles fixed to use the correct dat registry
 	    library in its new location (as of Beta 2.03)
 	  - Simple reorg of dat headers files to be consistent with
 	    the spec.
 	  - fixed bug in vapi_dto.h recv macro where we could have an
 	    uninitialized pointer.
 	  - Simple fix in dat_dr.c to initialize a variable early in the
 	    routine before errors occur.
 	  - Removed private data pointers from a CONNECTED event, as
 	    there should be no private data here.
 	  - dat_strerror no longer returns an uninitialized pointer if
 	    the error code is not recognized.
 	  - dat_dup_connect() will reject 0 timeout values, per the
 	    spec.
 	  - Removed unused internal_hca_names parameter from
 	    ib_enum_hcas() interface.
 	  - Use a temporary DAT_EVENT for kdapl up-calls rather than
 	    making assumptions about the current event queue.
 	  - Relocated some platform dependent code to an OSD file.
 	  - Eliminated several #ifdefs in .c files.
 	  - Inserted a missing unlock() on an error path.
 	  - Added bounds checking on size of private data to make sure
 	    we don't overrun the buffer
 	  - Fixed a kdapltest problem that caused a machine to panic if
 	    the user hit ^C
 	  - kdapltest now uses spin locks more appropriate for their
 	    context, e.g. spin_lock_bh or spin_lock_irq. Under a
 	    conditional.
 	  - Fixed kdapltest loops that drain EVDs so they don't go into
 	    endless loops.
 	  - Fixed bug in dapl_llist_add_entry link list code.
 	  - Better error reporting from provider code.
 	  - Handle case of user trying to reap DTO completions on an
 	    EP that has been freed.
 	  - No longer hold lock when ep_free() calls into provider layer
 	  - Fixed cr_accept() to not have an extra copy of
 	    private_data.
 	  - Verify private_data pointers before using them, avoid
 	    panic.
 	  - Fixed memory leak in kdapltest where print buffers were not
 	    getting reclaimed.



 	DAPL BETA 2.03 RELEASE NOTES

 	There are some  prominent features in this release:
 	1) dapltest/kdapltest. The dapltest test program has been
 	   rearchitected such that a kernel version is now available
 	   to test with kdapl. The most obvious change is a new
 	   directory structure that more closely matches other core
 	   dapl software. But there are a large number of changes
 	   throughout the source files to accommodate both the
 	   differences in udapl/kdapl interfaces, but also more mundane
 	   things such as printing.

 	   The new dapltest is in the tree at ./test/dapltest, while the
 	   old remains at ./test/udapl/dapltest. For this release, we
 	   have maintained both versions. In a future release, perhaps
 	   the next release, the old dapltest directory will be
 	   removed. Ongoing development will only occur in the new tree.

 	2) DAT 1.1 compliance. The DAT Collaborative has been busy
 	   finalizing the 1.1 revision of the spec. The header files
 	   have been reviewed and posted on the DAT Collaborative web
 	   site, they are now in full compliance.

 	   The reference implementation has been at a 1.1 level for a
 	   while. The current implementation has some features that will
 	   be part of the 1.2 DAT specification, but only in places
 	   where full compatibility can be maintained.

 	3) The DAT Registry has undergone some positive changes for
            robustness and support of more platforms. It now has the
            ability to support several identical provider names
            simultaneously, which enables the same dat.conf file to
            support multiple platforms. The registry will open each
            library and return when successful. For example, a dat.conf
            file may contain multiple provider names for ex0a, each
            pointing to a different library that may represent different
            platforms or vendors. This simplifies distribution into
            different environments by enabling the use of common
            dat.conf files.

 	In addition, there are a large number of bug fixes throughout
 	the code. Bug reports and fixes have come from a number of
 	companies.

 	Also note that the Release notes are cleaned up, no longer
 	containing the complete text of previous releases.

 	* EVDs no longer support DTO and CONNECTION event types on the
           same EVD. NOTE: The problem is maintaining the event ordering
           between two channels such that no DTO completes before a
           connection is received; and no DTO completes after a
           disconnect is received. For 90% of the cases this can be made
           to work, but the remaining 10% will cause serious performance
           degradation to get right.

 	NEW SINCE Beta 2.2

 	* DAT 1.1 spec compliance. This includes some new types, error
           codes, and moving structures around in the header files,
           among other things. Note the Class bits of dat_error.h have
 	  returned to a #define (from an enum) to cover the broadest
 	  range of platforms.

 	* Several additions for robustness, including handle and
           pointer checking, better argument checking, state
           verification, etc. Better recovery from error conditions,
 	  and some assert()s have been replaced with 'if' statements to
           handle the error.

 	* EVDs now maintain the actual queue length, rather than the
 	  requested amount. Both the DAT spec and IB (and other
 	  transports) allow the underlying implementation to provide
 	  more CQ entries than requested.

 	  Requests for the same number of entries contained by an EVD
 	  return immediate success.

 	* kDAPL enhancements:
 	  - module parameters & OS support calls updated to work with
             more recent Linux kernels.
 	  - kDAPL build options changes to match the Linux kernel, vastly
 	    reducing the size and making it more robust.
 	  - kDAPL unload now works properly
 	  - kDAPL takes a reference on the provider driver when it
 	    obtains a verbs vector, to prevent an accidental unload
 	  - Cleaned out all of the uDAPL cruft from the linux/osd files.

 	* New dapltest (see above).

 	* Added a new I/O trace facility, enabling a developer to debug
           all I/O that are in progress or recently completed. Default
           is OFF in the build.

 	* 0 timeout connections now refused, per the spec.

 	* Moved the remaining uDAPL specific files from the common/
           directory to udapl/. Also removed udapl files from the kdapl
 	  build.

 	* Bug fixes
 	  - Better error reporting from provider layer
 	  - Fixed race condition on reference counts for posting DTO
 	    ops.
 	  - Use DAT_COMPLETION_SUPPRESS_FLAG to suppress successful
 	    completion of dapl_rmr_bind  (instead of
 	    DAT_COMPLEITON_UNSIGNALLED, which is for non-notification
 	    completion).
 	  - Verify psp_flags value per the spec
 	  - Bug in psp_create_any() checking psp_flags fixed
 	  - Fixed type of flags in ib_disconnect from
 	    DAT_COMPLETION_FLAGS to DAT_CLOSE_FLAGS
 	  - Removed hard coded check for ASYNC_EVD. Placed all EVD
 	    prevention in evd_stream_merging_supported array, and
 	    prevent ASYNC_EVD from being created by an app.
 	  - ep_free() fixed to comply with the spec
 	  - Replaced various printfs with dbg_log statements
 	  - Fixed kDAPL interaction with the Linux kernel
 	  - Corrected phy_register protottype
 	  - Corrected kDAPL wait/wakeup synchronization
 	  - Fixed kDAPL evd_kcreate() such that it no longer depends
 	    on uDAPL only code.
 	  - dapl_provider.h had wrong guard #def: changed DAT_PROVIDER_H
 	    to DAPL_PROVIDER_H
 	  - removed extra (and bogus) call to dapls_ib_completion_notify()
 	    in evd_kcreate.c
 	  - Inserted missing error code assignment in
 	    dapls_rbuf_realloc()
 	  - When a CONNECTED event arrives, make sure we are ready for
 	    it, else something bad may have happened to the EP and we
 	    just return; this replaces an explicit check for a single
 	    error condition, replacing it with the general check for the
 	    state capable of dealing with the request.
 	  - Better context pointer verification. Removed locks around
 	    call to ib_disconnect on an error path, which would result
 	    in a deadlock. Added code for BROKEN events.
 	  - Brought the vapi code more up to date: added conditional
 	    compile switches, removed obsolete __ActivePort, deal
 	    with 0 length DTO
 	  - Several dapltest fixes to bring the code up to the 1.1
 	    specification.
 	  - Fixed mismatched dalp_os_dbg_print() #else dapl_Dbg_Print();
 	    the latter was replaced with the former.
 	  - ep_state_subtype() now includes UNCONNECTED.
 	  - Added some missing ibapi error codes.



 	NEW SINCE Beta 2.1

 	* Changes for Erratta and 1.1 Spec
 	  - Removed DAT_NAME_NOT_FOUND, per DAT erratta
 	  - EVD's with DTO and CONNECTION flags set no longer valid.
 	  - Removed DAT_IS_SUCCESS macro
 	  - Moved provider attribute structures from vendor files to udat.h
 	    and kdat.h
 	  - kdapl UPCALL_OBJECT now passed by reference

 	* Completed dat_strerr return strings

 	* Now support interrupted system calls

 	* dapltest now used dat_strerror for error reporting.

 	* Large number of files were formatted to meet project standard,
 	  very cosmetic changes but improves readability and
 	  maintainability.  Also cleaned up a number of comments during
 	  this effort.

 	* dat_registry and RPM file changes (contributed by Steffen Persvold):
 	  - Renamed the RPM name of the registry to be dat-registry
 	    (renamed the .spec file too, some cvs add/remove needed)
 	  - Added the ability to create RPMs as normal user (using
 	    temporal paths), works on SuSE, Fedora, and RedHat.
 	  - 'make rpm' now works even if you didn't build first.
 	  - Changed to using the GNU __attribute__((constructor)) and
 	    __attribute__((destructor)) on the dat_init functions, dat_init
 	    and dat_fini. The old -init and -fini options to LD makes
 	    applications crash on some platforms (Fedora for example).
 	  - Added support for 64 bit platforms.
 	  - Added code to allow multiple provider names in the registry,
 	    primarily to support ia32 and ia64 libraries simultaneously.
 	    Provider names are now kept in a list, the first successful
 	    library open will be the provider.

 	* Added initial infrastructure for DAPL_DCNTR, a feature that
 	  will aid in debug and tuning of a dapl implementation. Partial
 	  implementation only at this point.

 	* Bug fixes
 	- Prevent debug messages from crashing dapl in EVD completions by
 	  verifying the error code to ensure data is valid.
 	- Verify CNO before using it to clean up in evd_free()
 	- CNO timeouts now return correct error codes, per the spec.
 	- cr_accept now complies with the spec concerning connection
 	  requests that go away before the accept is invoked.
 	- Verify valid EVD before posting connection evens on active side
 	  of a connection. EP locking also corrected.
 	- Clean up of dapltest Makefile, no longer need to declare
 	  DAT_THREADSAFE
 	- Fixed check of EP states to see if we need to disconnect an
 	  IA is closed.
 	- ep_free() code reworked such that we can properly close a
 	  connection pending EP.
 	- Changed disconnect processing to comply with the spec: user will
 	   see a BROKEN event, not DISCONNECTED.
 	- If we get a DTO error, issue a disconnect to let the CM and
 	  the user know the EP state changed to disconnect; checked IBA
 	  spec to make sure we disconnect on correct error codes.
 	- ep_disconnect now properly deals with abrupt disconnects on the
 	  active side of a connection.
 	- PSP now created in the correct state for psp_create_any(), making
 	  it usable.
 	- dapl_evd_resize() now returns correct status, instead of always
 	  DAT_NOT_IMPLEMENTED.
 	- dapl_evd_modify_cno() does better error checking before invoking
 	  the provider layer, avoiding bugs.
 	- Simple change to allow dapl_evd_modify_cno() to set the CNO to
 	  NULL, per the spec.
 	- Added required locking around call to dapl_sp_remove_cr.

 	- Fixed problems related to dapl_ep_free: the new
 	  disconnect(abrupt) allows us to do a more immediate teardown of
 	  connections, removing the need for the MAGIC_EP_EXIT magic
 	  number/state, which has been removed. Mmuch cleanup of paths,
 	  and made more robust.
 	- Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is
 	  triggered if there are waiters when the last EVD is removed
 	  or when the IA is freed.
 	- Added code to deal with the provider synchronously telling us
 	   a connection is unreachable, and generate the appropriate
 	   event.
 	- Changed timer routine type from unsigned long to uintptr_t
 	  to better fit with machine architectures.
 	- ep.param data now initialized in ep_create, not ep_alloc.
 	- Or Gerlitz provided updates to Mellanox files for evd_resize,
 	  fw attributes, many others. Also implemented changes for correct
 	  sizes on REP side of a connection request.



 	NEW SINCE Beta 2.0

 	* dat_echo now DAT 1.1 compliant. Various small enhancements.

 	* Revamped atomic_inc/dec to be void, the return value was never
 	  used. This allows kdapl to use Linux kernel equivalents, and
 	  is a small performance advantage.

 	* kDAPL: dapl_evd_modify_upcall implemented and tested.

 	* kDAPL: physical memory registration implemented and tested.

 	* uDAPL now builds cleanly for non-debug versions.

 	* Default RDMA credits increased to 8.

 	* Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
 	  months).

 	* Cleaned up dat_error.h, now 1.1 compliant in comments.

 	* evd_resize initial implementation. Untested.

 	* Bug fixes
 	  - __KDAPL__ is defined in kdat_config.h, so apps don't need
 	    to define it.
 	  - Changed include file ordering in kdat.h to put kdat_config.h
 	    first.
 	  - resolved connection/tear-down race on the client side.
 	  - kDAPL timeouts now scaled properly; fixed 3 orders of
 	    magnitude difference.
 	  - kDAPL EVD callbacks now get invoked for all completions; old
 	    code would drop them in heavy utilization.
 	  - Fixed error path in kDAPL evd creation, so we no longer
 	    leak CNOs.
 	  - create_psp_any returns correct error code if it can't create
 	    a connection qualifier.
 	  - lock fix in ibapi disconnect code.
 	  - kDAPL INFINITE waits now work properly (non connection
 	    waits)
 	  - kDAPL driver unload now works properly
 	  - dapl_lmr_[k]create now returns 1.1 error codes
 	  - ibapi routines now return DAT 1.1 error codes



 	NEW SINCE Beta 1.10

 	* kDAPL is now part of the DAPL distribution. See the release
 	  notes above.

 	  The kDAPL 1.1 spec is now contained in the doc/ subdirectory.

 	* Several files have been moved around as part of the kDAPL
 	  checkin. Some files that were previously in udapl/ are now
 	  in common/, some in common are now in udapl/. The goal was
 	  to make sure files are properly located and make sense for
 	  the build.

 	* Source code formatting changes for consistency.

 	* Bug fixes
 	  - dapl_evd_create() was comparing the wrong bit combinations,
 	    allowing bogus EVDs to be created.
 	  - Removed code that swallowed zero length I/O requests, which
 	    are allowed by the spec and are useful to applications.
 	  - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
 	    routine will take and release the lock. Cosmetic change.
 	  - dapl_get_consuemr_context() will now verify the pointer
 	    argument 'context' is not NULL.




 	OBTAIN THE CODE

 	To obtain the tree for your local machine you can check it
 	out of the source repository using CVS tools. CVS is common
 	on Unix systems and available as freeware on Windows machines.
 	The command to anonymously obtain the source code from
 	Source Forge (with no password) is:

 	cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 	cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co .

 	When prompted for a password, simply  press the Enter key.

 	Source Forge also contains explicit directions on how to become
 	a developer, as well as how to use different CVS commands. You may
 	also browse the source code using the URL:

 	   http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/


 	SYSTEM REQUIREMENTS

 	This project has been implemented on Red Hat Linux 7.3, SuSE
 	SLES 8, Windows 2000, RHEL 3.0, and a couple of other Linux
 	distrubutions. The structure of the code is designed to allow
 	other operating systems to easily be adapted.

 	The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
 	development, and continues with this platform. Our HCAs use the
 	IB verbs API submitted by IBM. Mellanox has contributed an
 	adapter layer using their VAPI verbs API. Either platform is
 	available to any group considering DAPL work. The structure of
 	the uDAPL source allows other provider API sets to be easily
 	integrated.

 	The development team uses any one of three topologies: two HCAs
 	in a single machine; a single HCA in each of two machines; and
 	most commonly, a switch. Machines connected to a switch may have
 	more than one HCA.

 	The DAPL Plugfest revealed that switches and HCAs available from
 	most vendors will interoperate with little trouble, given the
 	most recent releases of software. The dapl reference team makes
 	no recommendation on HCA or switch vendors.

 	Explicit machine configurations are available upon request.


 	IN THE TREE

 	The DAPL tree contains source code for the uDAPL and kDAPL
 	implementations, and also includes tests and documentation.

 	Included documentation has the base level API of the
 	providers: the IBM Access API and the Mellanox Verbs API. Also
 	included are a growing number of DAPL design documents which
 	lead the reader through specific DAPL subsystems. More
 	design documents are in progress and will appear in the tree in
 	the near future.

 	A small number of test applications and a unit test framework
 	are also included. dapltest is the primary testing application
 	used by the DAPL team, it is capable of simulating a variety of
 	loads and exercises a large number of interfaces. Full
 	documentation is included for each of the tests.

 	Recently, the dapl conformance test has been added to the source
 	repository. The test provides coverage of the most common
 	interfaces, doing both positive and negative testing. Vendors
 	providing DAPL implementation are strongly encouraged to run
 	this set of tests.


 	MAKEFILE NOTES

 	There are a number #ifdef's in the code that were necessary
 	during early development. They are disappearing as we
 	have time to take advantage of features and work available from
 	newer releases of provider software. You may notice an #ifdef
 	<something>_BUSTED, which indicates a particular feature was not
 	working at the time the code was written and the DAPL team
 	developed a work-around.

 	These #ifdefs are not documented as the intent is to remove
 	them as soon as possible.

 	Of particular relevance are the following #defines:

 	- CM_BUSTED

 	  The DAPL team has been an early adopter of InfiniBand and has
 	  had to improvise missing functionality while the vendors lag
 	  our development. InfiniBand uses a Connection Manager (CM) to
 	  establish a connection between nodes. This #define essentially
 	  'fakes' a connection by moving a QP into the appropriate
 	  state. Most of the IB vendors have a working CM now and this
 	  is no longer the default, but the code remains as some
 	  development groups are working to catch up.

 	- NO_NAME_SERVICE

 	  Naming is a thorny issue in InfiniBand; translating from a
 	  hostname or an interface name to a GID that can be used to
 	  establish a connection with a remote machine. The reference
 	  implementation provides a simple name service under this
 	  #define. The goal is to use IPoIB when it becomes
 	  available. NO_NAME_SERVICE will probably remain in the code
 	  long term in order to enable various implementations. A
 	  description of how this works is found in the end_point_design
 	  document in the doc/ directory.


 	CONTRIBUTIONS

 	As is common to Source Forge projects, there are a small number
 	of developers directly associated with the source tree and having
 	privileges to change the tree. Requested updates, changes, bug
 	fixes, enhancements, or contributions should be sent to Steve
 	Sears at sj...@ne... for review. We welcome your
 	contributions and expect the quality of the project will
 	improve thanks to your help.

 	The core DAPL team is:

 	  Steve Sears
 	  Philip Christopher

 	  ... with contributions from a number of excellent engineers in
 	  various companies contributing to the open source effort.


 	ONGOING WORK

 	Not all of the DAPL spec is implemented at this time.
 	Functionality such as shared memory will probably not be
 	implemented by the reference implementation (there is a write up
 	on this in the doc/ area), and there are yet various cases where
 	work remains to be done.  And of course, not all of the
 	implemented functionality has been tested yet.  The DAPL team
 	continues to develop and test the tree with the intent of
 	completing the specification and delivering a robust and useful
 	implementation.


The DAPL Team

[Dapl-announce] uDAPL/kDAPL Beta 2.06 now available

[Steve S. <sj...@ne...>]

           Release Notes for
        Beta 2.06 uDAPL/kDAPL Release
           October 5, 2004


    DAPL BETA 2.06 RELEASE NOTES

    We are not planning any further releases of the Beta series,
    which are based  on the 1.1 version of the spec. There may be
    further releases for bug fixes, but we anticipate the DAPL
    community to move to the new 1.2 version of the spec and the
    changes mandated in the reference implementation.

    The biggest item in this release is the first inclusion of the
    OpenIB Gen 1 provider, an item generating a lot of interest in
    the IB community. This implementation has graciously been
    provided by the Mellanox team. The kdapl implementation is in
    progress, and we imagine work will soon begin on Gen 2.

    There are also a handful of bug fixes available, as well as a long
    awaited update to the endpoint design document.

    NEW SINCE Beta 2.4

    - OpenIB gen 1 provider support has been added
    - Added dapls_evd_post_generic_event(), routine to post generic
      event types as requested by some providers. Also cleaned up
      error reporting.
    - Updated the endpoint design document in the doc/ directory.

    * BUG FIXES

    + Cleaned up memory leak on close by freeing the HCA structure;
    + Removed bogus #defs for rdtsc calls on IA64.
    + Changed daptest thread types to use internal types for
      portability & correctness
    + Various 64 bit enhancements & updates
    + Fixes to conformance test that were defining CONN_QUAL twice
      and using it in different ways
    + Cleaned up private data handling in ep_connect & provider
      support: we now avoid extra copy in connect code; reduced
      stack requirements by using private_data structure in the EP;
      removed provider variable.
    + Fixed problem in the dat conformance test where cno_wait would
      attempt to dereference a timer value and SEGV.
    + Removed old vestiges of depricated POLLING_COMPLETIONS
      conditionals.

    DAPL BETA 2.05 RELEASE NOTES

    This was to be a very minor release, the primary change was
    going to be the new wording of the DAT license as contained in
    the header for all source files. But the interest and
    development occurring in DAPL provided some extra bug fixes, and
    some new functionality that has been requested for a while.

    First, you may notice that every single source file was
    changed. If you read the release notes from DAPL BETA 2.04, you
    were warned this would happen. There was a legal issue with the
    wording in the header, the end result was that every source file
    was required to change the word 'either of' to 'both'. We've
    been putting this change off as long as possible, but we wanted
    to do it in a clean drop before we start working on DAT 1.2
    changes in the reference implementation, just to keep things
    reasonably sane.

    kdapltest has enabled three of the subtests supported by
    dapltest. The Performance test in particular has been very
    useful to dapltest in getting minima and maxima. The Limit test
    pushes the limits by allocating the maximum number of specific
    resources. And the FFT tests are also available.

    Most vendors have supported shared memory regions for a while,
    several of which have asked the reference implementation team to
    provide a common implementation. Shared memory registration has
    been tested on ibapi, and compiled into vapi. Both InfiniBand
    providers have the restriction that a memory region must be
    created before it can be shared; not all RDMA APIs are this way,
    several allow you to declare a memory region shared when it is
    registered. Hence, details of the implementation are hidden in
    the provider layer, rather than forcing other APIs to do
    something strange.

    This release also contains some changes that will allow dapl to
    work on Opteron processors, as well as some preliminary support
    for Power PC architecture. These features are not well tested
    and may be incomplete at this time.

    Finally, we have been asked several times over the course of the
    project for a canonical interface between the common and
    provider layers. This release includes a dummy provider to meet
    that need. Anyone should be able to download the release and do
    a:
       make VERBS=DUMMY

    And have a cleanly compiled dapl library. This will be useful
    both to those porting new transport providers, as well as those
    going to new machines.

    The DUMMY provider has been compiled on both Linux and Windows
    machines.


    NEW SINCE Beta 2.4
    - kdapltest enhancements:
      * Limit subtests now work
      * Performance subtests now work.
      * FFT tests now work.

    - The VAPI headers have been refreshed by Mellanox

    - Initial Opteron and PPC support.

    - Atomic data types now have consistent treatment, allowing us to
      use native data types other than integers. The Linux kdapl
      uses atomic_t, allowing dapl to use the kernel macros and
      eliminate the assembly code in dapl_osd.h

    - The license language was updated per the direction of the
      DAT Collaborative. This two word change affected the header
      of every file in the tree.

    - SHARED memory regions are now supported.

    - Initial support for the TOPSPIN provider.

    - Added a dummy provider, essentially the NULL provider. It's
      purpose is to aid in porting and to clarify exactly what is
      expected in a provider implementation.

    - Removed memory allocation from the DTO path for VAPI

    - cq_resize will now allow the CQ to be resized smaller. Not all
      providers support this, but it's a provider problem, not a
      limitation of the common code.

    * BUG FIXES

    + Removed spurious lock in dapl_evd_connection_callb.c that
      would have caused a deadlock.
    + The Async EVD was getting torn down too early, potentially
      causing lost errors. Has been moved later in the teardown
      process.
    + kDAPL replaced mem_map_reserve() with newer SetPageReserved()
      for better Linux integration.
    + kdapltest no longer allocate large print buffers on the stack,
      is more careful to ensure buffers don't overflow.
    + Put dapl_os_dbg_print() under DAPL_DBG conditional, it is
      supposed to go away in a production build.
    + dapltest protocol version has been bumped to reflect the
      change in the Service ID.
    + Corrected several instances of routines that did not adhere
      to the DAT 1.1 error code scheme.
    + Cleaned up vapi ib_reject_connection to pass DAT types rather
      than provider specific types. Also cleaned up naming interface
      declarations and their use in vapi_cm.c; fixed incorrect
      #ifdef for naming.
    + Initialize missing uDAPL provider attr, pz_support.
    + Changes for better layering: first, moved
      dapl_lmr_convert_privileges to the provider layer as memory
      permissions are clearly transport specific and are not always
      defined in an integer bitfield; removed common routines for
      lmr and rmr. Second, move init and release setup/teardown
      routines into adapter_util.h, which defined the provider
      interface.
    + Cleaned up the HCA name cruft that allowed different types
      of names such as strings or ints to be dealt with in common
      code; but all names are presented by the dat_registry as
      strings, so pushed conversions down to the provider
      level. Greatly simplifies names.
    + Changed deprecated true/false to DAT_TRUE/DAT_FALSE.
    + Removed old IB_HCA_NAME type in favor of char *.
    + Fixed race condition in kdapltest's use of dat_evd_dequeue.
    + Changed cast for SERVER_PORT_NUMBER to DAT_CONN_QUAL as it
      should be. 
    + Small code reorg to put the CNO into the EVD when it is
      allocated, which simplifies things.
    + Removed gratuitous ib_hca_port_t and ib_send_op_type_t types,
      replaced with standard int.
    + Pass a pointer to cqe debug routine, not a structure. Some
      clean up of data types.
    + kdapl threads now invoke reparent_to_init() on exit to allow
      threads to get cleaned up.



    DAPL BETA 2.04 RELEASE NOTES

    The big changes for this release involve a more strict adherence
    to the original dapl architecture. Originally, only InfiniBand
    providers were available, so allowing various data types and
    event codes to show through into common code wasn't a big deal.

    But today, there are an increasing number of providers available
    on a number of transports. Requiring an IP iWarp provider to
    match up to InfiniBand events is silly, for example.

    Restructuring the code allows more flexibility in providing an
    implementation.

    There are also a large number of bug fixes available in this
    release, particularly in kdapl related code.

    Be warned that the next release will change every file in the
    tree as we move to the newly approved DAT license. This is a
    small change, but all files are affected.

    Future releases will also support to the soon to be ratified DAT
    1.2 specification.

    This release has benefited from many bug reports and fixes from
    a number of individuals and companies. On behalf of the DAPL
    community, thank you!


    NEW SINCE Beta 2.3

    - Made several changes to be more rigorous on the layering
      design of dapl. The intent is to make it easier for non
      InfiniBand transports to use dapl. These changes include:
      
      * Revamped the ib_hca_open/close code to use an hca_ptr
        rather than an ib_handle, giving the transport layer more
        flexibility in assigning transport handles and resources.

      * Removed the CQD calls, they are specific to the IBM API;
        folded this functionality into the provider open/close calls.

      * Moved VAPI, IBAPI transport specific items into a transport
        structure placed inside of the HCA structure. Also updated
        routines using these fields to use the new location. Cleaned
        up provider knobs that have been exposed for too long.

      * Changed a number of provider routines to use DAPL structure
        pointers rather than exposing provider handles & values. Moved
        provider specific items out of common code, including provider
        data types (e.g. ib_uint32_t).

      * Pushed provider completion codes and type back into the
            provider layer. We no longer use EVD or CM completion types at
            the common layer, instead we obtain the appropriate DAT type
            from the provider and process only DAT types.

      * Change private_data handling such that we can now accommodate
            variable length private data.

    - Remove DAT 1.0 cruft from the DAT header files.

    - Better spec compliance in headers and various routines.

    - Major updates to the VAPI implementation from
          Mellanox. Includes initial kdapl implementation

    - Move kdapl platform specific support for hash routines into
          OSD file.

    - Cleanups to make the code more readable, including comments
          and certain variable and structure names.

    - Fixed CM_BUSTED code so that it works again: very useful for
          new dapl ports where infrastructure is lacking. Also made
      some fixes for IBHOSTS_NAMING conditional code.

    - Added DAPL_MERGE_CM_DTO as a compile time switch to support
      EVD stream merging of CM and DTO events. Default is off.

    - 'Quit' test ported to kdapltest

    - uDAPL now builds on Linux 2.6 platform (SuSE 9.1).

    - kDAPL now builds for a larger range of Linux kernels, but
          still lacks 2.6 support.

    - Added shared memory ID to LMR structure. Shared memory is
          still not fully supported in the reference implementation, but
          the common code will appear soon.

    * Bug fixes
      - Various Makefiles fixed to use the correct dat registry
        library in its new location (as of Beta 2.03)
      - Simple reorg of dat headers files to be consistent with
        the spec.
      - fixed bug in vapi_dto.h recv macro where we could have an
        uninitialized pointer.
      - Simple fix in dat_dr.c to initialize a variable early in the
        routine before errors occur.
      - Removed private data pointers from a CONNECTED event, as
        there should be no private data here.
      - dat_strerror no longer returns an uninitialized pointer if
        the error code is not recognized.
      - dat_dup_connect() will reject 0 timeout values, per the
        spec.
      - Removed unused internal_hca_names parameter from
        ib_enum_hcas() interface.
      - Use a temporary DAT_EVENT for kdapl up-calls rather than
        making assumptions about the current event queue.
      - Relocated some platform dependent code to an OSD file.
      - Eliminated several #ifdefs in .c files.
      - Inserted a missing unlock() on an error path.
      - Added bounds checking on size of private data to make sure
        we don't overrun the buffer
      - Fixed a kdapltest problem that caused a machine to panic if
        the user hit ^C
      - kdapltest now uses spin locks more appropriate for their
        context, e.g. spin_lock_bh or spin_lock_irq. Under a
        conditional.
      - Fixed kdapltest loops that drain EVDs so they don't go into
        endless loops.
      - Fixed bug in dapl_llist_add_entry link list code.
      - Better error reporting from provider code.
      - Handle case of user trying to reap DTO completions on an
        EP that has been freed.
      - No longer hold lock when ep_free() calls into provider layer
      - Fixed cr_accept() to not have an extra copy of
        private_data.
      - Verify private_data pointers before using them, avoid
        panic. 
      - Fixed memory leak in kdapltest where print buffers were not
        getting reclaimed.



    DAPL BETA 2.03 RELEASE NOTES

    There are some  prominent features in this release:
    1) dapltest/kdapltest. The dapltest test program has been
       rearchitected such that a kernel version is now available
       to test with kdapl. The most obvious change is a new
       directory structure that more closely matches other core
       dapl software. But there are a large number of changes
       throughout the source files to accommodate both the
       differences in udapl/kdapl interfaces, but also more mundane
       things such as printing.

       The new dapltest is in the tree at ./test/dapltest, while the
       old remains at ./test/udapl/dapltest. For this release, we
       have maintained both versions. In a future release, perhaps
       the next release, the old dapltest directory will be
       removed. Ongoing development will only occur in the new tree.

    2) DAT 1.1 compliance. The DAT Collaborative has been busy
       finalizing the 1.1 revision of the spec. The header files
       have been reviewed and posted on the DAT Collaborative web
       site, they are now in full compliance.

       The reference implementation has been at a 1.1 level for a
       while. The current implementation has some features that will
       be part of the 1.2 DAT specification, but only in places
       where full compatibility can be maintained.

    3) The DAT Registry has undergone some positive changes for
           robustness and support of more platforms. It now has the
           ability to support several identical provider names
           simultaneously, which enables the same dat.conf file to
           support multiple platforms. The registry will open each
           library and return when successful. For example, a dat.conf
           file may contain multiple provider names for ex0a, each
           pointing to a different library that may represent different
           platforms or vendors. This simplifies distribution into
           different environments by enabling the use of common
           dat.conf files.

    In addition, there are a large number of bug fixes throughout
    the code. Bug reports and fixes have come from a number of
    companies.

    Also note that the Release notes are cleaned up, no longer
    containing the complete text of previous releases.

    * EVDs no longer support DTO and CONNECTION event types on the
          same EVD. NOTE: The problem is maintaining the event ordering
          between two channels such that no DTO completes before a
          connection is received; and no DTO completes after a
          disconnect is received. For 90% of the cases this can be made
          to work, but the remaining 10% will cause serious performance
          degradation to get right.

    NEW SINCE Beta 2.2

    * DAT 1.1 spec compliance. This includes some new types, error
          codes, and moving structures around in the header files,
          among other things. Note the Class bits of dat_error.h have
      returned to a #define (from an enum) to cover the broadest
      range of platforms.

    * Several additions for robustness, including handle and
          pointer checking, better argument checking, state
          verification, etc. Better recovery from error conditions,
      and some assert()s have been replaced with 'if' statements to
          handle the error.

    * EVDs now maintain the actual queue length, rather than the
      requested amount. Both the DAT spec and IB (and other
      transports) allow the underlying implementation to provide
      more CQ entries than requested.

      Requests for the same number of entries contained by an EVD
      return immediate success.

    * kDAPL enhancements:
      - module parameters & OS support calls updated to work with
            more recent Linux kernels.
      - kDAPL build options changes to match the Linux kernel, vastly
        reducing the size and making it more robust.
      - kDAPL unload now works properly
      - kDAPL takes a reference on the provider driver when it
        obtains a verbs vector, to prevent an accidental unload
      - Cleaned out all of the uDAPL cruft from the linux/osd files.

    * New dapltest (see above).

    * Added a new I/O trace facility, enabling a developer to debug
          all I/O that are in progress or recently completed. Default
          is OFF in the build.

    * 0 timeout connections now refused, per the spec.

    * Moved the remaining uDAPL specific files from the common/
          directory to udapl/. Also removed udapl files from the kdapl
      build.

    * Bug fixes
      - Better error reporting from provider layer
      - Fixed race condition on reference counts for posting DTO
        ops.
      - Use DAT_COMPLETION_SUPPRESS_FLAG to suppress successful
        completion of dapl_rmr_bind  (instead of
        DAT_COMPLEITON_UNSIGNALLED, which is for non-notification
        completion).
      - Verify psp_flags value per the spec
      - Bug in psp_create_any() checking psp_flags fixed
      - Fixed type of flags in ib_disconnect from
        DAT_COMPLETION_FLAGS to DAT_CLOSE_FLAGS
      - Removed hard coded check for ASYNC_EVD. Placed all EVD
        prevention in evd_stream_merging_supported array, and
        prevent ASYNC_EVD from being created by an app.
      - ep_free() fixed to comply with the spec
      - Replaced various printfs with dbg_log statements
      - Fixed kDAPL interaction with the Linux kernel
      - Corrected phy_register protottype
      - Corrected kDAPL wait/wakeup synchronization
      - Fixed kDAPL evd_kcreate() such that it no longer depends
        on uDAPL only code.
      - dapl_provider.h had wrong guard #def: changed DAT_PROVIDER_H
        to DAPL_PROVIDER_H
      - removed extra (and bogus) call to dapls_ib_completion_notify()
        in evd_kcreate.c
      - Inserted missing error code assignment in
        dapls_rbuf_realloc()
      - When a CONNECTED event arrives, make sure we are ready for
        it, else something bad may have happened to the EP and we
        just return; this replaces an explicit check for a single
        error condition, replacing it with the general check for the
        state capable of dealing with the request.
      - Better context pointer verification. Removed locks around
        call to ib_disconnect on an error path, which would result
        in a deadlock. Added code for BROKEN events.
      - Brought the vapi code more up to date: added conditional
        compile switches, removed obsolete __ActivePort, deal
        with 0 length DTO
      - Several dapltest fixes to bring the code up to the 1.1
        specification.
      - Fixed mismatched dalp_os_dbg_print() #else dapl_Dbg_Print();
        the latter was replaced with the former.
      - ep_state_subtype() now includes UNCONNECTED.
      - Added some missing ibapi error codes.
 


    NEW SINCE Beta 2.1

    * Changes for Erratta and 1.1 Spec
      - Removed DAT_NAME_NOT_FOUND, per DAT erratta
      - EVD's with DTO and CONNECTION flags set no longer valid.
      - Removed DAT_IS_SUCCESS macro
      - Moved provider attribute structures from vendor files to udat.h
        and kdat.h
      - kdapl UPCALL_OBJECT now passed by reference

    * Completed dat_strerr return strings

    * Now support interrupted system calls

    * dapltest now used dat_strerror for error reporting.

    * Large number of files were formatted to meet project standard,
      very cosmetic changes but improves readability and
      maintainability.  Also cleaned up a number of comments during
      this effort.

    * dat_registry and RPM file changes (contributed by Steffen Persvold):
      - Renamed the RPM name of the registry to be dat-registry
        (renamed the .spec file too, some cvs add/remove needed)
      - Added the ability to create RPMs as normal user (using
        temporal paths), works on SuSE, Fedora, and RedHat.
      - 'make rpm' now works even if you didn't build first.
      - Changed to using the GNU __attribute__((constructor)) and
        __attribute__((destructor)) on the dat_init functions, dat_init
        and dat_fini. The old -init and -fini options to LD makes
        applications crash on some platforms (Fedora for example).
      - Added support for 64 bit platforms.
      - Added code to allow multiple provider names in the registry,
        primarily to support ia32 and ia64 libraries simultaneously.
        Provider names are now kept in a list, the first successful
        library open will be the provider.

    * Added initial infrastructure for DAPL_DCNTR, a feature that
      will aid in debug and tuning of a dapl implementation. Partial
      implementation only at this point.

    * Bug fixes
    - Prevent debug messages from crashing dapl in EVD completions by
      verifying the error code to ensure data is valid.
    - Verify CNO before using it to clean up in evd_free()
    - CNO timeouts now return correct error codes, per the spec.
    - cr_accept now complies with the spec concerning connection
      requests that go away before the accept is invoked.
    - Verify valid EVD before posting connection evens on active side
      of a connection. EP locking also corrected.
    - Clean up of dapltest Makefile, no longer need to declare
      DAT_THREADSAFE
    - Fixed check of EP states to see if we need to disconnect an
      IA is closed.
    - ep_free() code reworked such that we can properly close a
      connection pending EP.
    - Changed disconnect processing to comply with the spec: user will
       see a BROKEN event, not DISCONNECTED.
    - If we get a DTO error, issue a disconnect to let the CM and
      the user know the EP state changed to disconnect; checked IBA
      spec to make sure we disconnect on correct error codes.
    - ep_disconnect now properly deals with abrupt disconnects on the
      active side of a connection.
    - PSP now created in the correct state for psp_create_any(), making
      it usable.
    - dapl_evd_resize() now returns correct status, instead of always
      DAT_NOT_IMPLEMENTED.
    - dapl_evd_modify_cno() does better error checking before invoking
      the provider layer, avoiding bugs.
    - Simple change to allow dapl_evd_modify_cno() to set the CNO to
      NULL, per the spec.
    - Added required locking around call to dapl_sp_remove_cr.

    - Fixed problems related to dapl_ep_free: the new
      disconnect(abrupt) allows us to do a more immediate teardown of
      connections, removing the need for the MAGIC_EP_EXIT magic
      number/state, which has been removed. Mmuch cleanup of paths,
      and made more robust.
    - Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is
      triggered if there are waiters when the last EVD is removed
      or when the IA is freed.
    - Added code to deal with the provider synchronously telling us
       a connection is unreachable, and generate the appropriate
       event.
    - Changed timer routine type from unsigned long to uintptr_t
      to better fit with machine architectures.
    - ep.param data now initialized in ep_create, not ep_alloc.
    - Or Gerlitz provided updates to Mellanox files for evd_resize,
      fw attributes, many others. Also implemented changes for correct
      sizes on REP side of a connection request.



    NEW SINCE Beta 2.0

    * dat_echo now DAT 1.1 compliant. Various small enhancements.

    * Revamped atomic_inc/dec to be void, the return value was never
      used. This allows kdapl to use Linux kernel equivalents, and
      is a small performance advantage.

    * kDAPL: dapl_evd_modify_upcall implemented and tested.

    * kDAPL: physical memory registration implemented and tested.

    * uDAPL now builds cleanly for non-debug versions.

    * Default RDMA credits increased to 8.

    * Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
      months).

    * Cleaned up dat_error.h, now 1.1 compliant in comments.

    * evd_resize initial implementation. Untested.

    * Bug fixes
      - __KDAPL__ is defined in kdat_config.h, so apps don't need
        to define it.
      - Changed include file ordering in kdat.h to put kdat_config.h
        first.
      - resolved connection/tear-down race on the client side.
      - kDAPL timeouts now scaled properly; fixed 3 orders of
        magnitude difference.
      - kDAPL EVD callbacks now get invoked for all completions; old
        code would drop them in heavy utilization.
      - Fixed error path in kDAPL evd creation, so we no longer
        leak CNOs.
      - create_psp_any returns correct error code if it can't create
        a connection qualifier.
      - lock fix in ibapi disconnect code.
      - kDAPL INFINITE waits now work properly (non connection
        waits) 
      - kDAPL driver unload now works properly
      - dapl_lmr_[k]create now returns 1.1 error codes
      - ibapi routines now return DAT 1.1 error codes
      


    NEW SINCE Beta 1.10

    * kDAPL is now part of the DAPL distribution. See the release
      notes above.

      The kDAPL 1.1 spec is now contained in the doc/ subdirectory.

    * Several files have been moved around as part of the kDAPL
      checkin. Some files that were previously in udapl/ are now
      in common/, some in common are now in udapl/. The goal was
      to make sure files are properly located and make sense for
      the build.

    * Source code formatting changes for consistency.

    * Bug fixes
      - dapl_evd_create() was comparing the wrong bit combinations,
        allowing bogus EVDs to be created.
      - Removed code that swallowed zero length I/O requests, which
        are allowed by the spec and are useful to applications.
      - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
        routine will take and release the lock. Cosmetic change.
      - dapl_get_consuemr_context() will now verify the pointer
        argument 'context' is not NULL.




    OBTAIN THE CODE

    To obtain the tree for your local machine you can check it
    out of the source repository using CVS tools. CVS is common
    on Unix systems and available as freeware on Windows machines.
    The command to anonymously obtain the source code from
    Source Forge (with no password) is:
    
    cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
    cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co .

    When prompted for a password, simply  press the Enter key.

    Source Forge also contains explicit directions on how to become
    a developer, as well as how to use different CVS commands. You may
    also browse the source code using the URL:

       http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/


    SYSTEM REQUIREMENTS

    This project has been implemented on Red Hat Linux 7.3, SuSE
    SLES 8, Windows 2000, RHEL 3.0, and a couple of other Linux
    distrubutions. The structure of the code is designed to allow
    other operating systems to easily be adapted.

    The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
    development, and continues with this platform. Our HCAs use the
    IB verbs API submitted by IBM. Mellanox has contributed an
    adapter layer using their VAPI verbs API. Either platform is
    available to any group considering DAPL work. The structure of
    the uDAPL source allows other provider API sets to be easily
    integrated.

    The development team uses any one of three topologies: two HCAs
    in a single machine; a single HCA in each of two machines; and
    most commonly, a switch. Machines connected to a switch may have
    more than one HCA.

    The DAPL Plugfest revealed that switches and HCAs available from
    most vendors will interoperate with little trouble, given the
    most recent releases of software. The dapl reference team makes
    no recommendation on HCA or switch vendors.

    Explicit machine configurations are available upon request.


    IN THE TREE

    The DAPL tree contains source code for the uDAPL and kDAPL
    implementations, and also includes tests and documentation.

    Included documentation has the base level API of the
    providers: the IBM Access API and the Mellanox Verbs API. Also
    included are a growing number of DAPL design documents which
    lead the reader through specific DAPL subsystems. More
    design documents are in progress and will appear in the tree in
    the near future.

    A small number of test applications and a unit test framework
    are also included. dapltest is the primary testing application
    used by the DAPL team, it is capable of simulating a variety of
    loads and exercises a large number of interfaces. Full
    documentation is included for each of the tests.

    Recently, the dapl conformance test has been added to the source
    repository. The test provides coverage of the most common
    interfaces, doing both positive and negative testing. Vendors
    providing DAPL implementation are strongly encouraged to run
    this set of tests.


    MAKEFILE NOTES

    There are a number #ifdef's in the code that were necessary
    during early development. They are disappearing as we
    have time to take advantage of features and work available from
    newer releases of provider software. You may notice an #ifdef
    <something>_BUSTED, which indicates a particular feature was not
    working at the time the code was written and the DAPL team
    developed a work-around.

    These #ifdefs are not documented as the intent is to remove
    them as soon as possible.

    Of particular relevance are the following #defines:

    - CM_BUSTED

      The DAPL team has been an early adopter of InfiniBand and has
      had to improvise missing functionality while the vendors lag
      our development. InfiniBand uses a Connection Manager (CM) to
      establish a connection between nodes. This #define essentially
      'fakes' a connection by moving a QP into the appropriate
      state. Most of the IB vendors have a working CM now and this
      is no longer the default, but the code remains as some
      development groups are working to catch up.

    - NO_NAME_SERVICE

      Naming is a thorny issue in InfiniBand; translating from a
      hostname or an interface name to a GID that can be used to
      establish a connection with a remote machine. The reference
      implementation provides a simple name service under this
      #define. The goal is to use IPoIB when it becomes
      available. NO_NAME_SERVICE will probably remain in the code
      long term in order to enable various implementations. A
      description of how this works is found in the end_point_design
      document in the doc/ directory.


    CONTRIBUTIONS

    As is common to Source Forge projects, there are a small number
    of developers directly associated with the source tree and having
    privileges to change the tree. Requested updates, changes, bug
    fixes, enhancements, or contributions should be sent to Steve
    Sears at sj...@ne... for review. We welcome your
    contributions and expect the quality of the project will
    improve thanks to your help.

    The core DAPL team is:

      Steve Sears
      Philip Christopher

      ... with contributions from a number of excellent engineers in
      various companies contributing to the open source effort.


    ONGOING WORK

    Not all of the DAPL spec is implemented at this time.
    Functionality such as shared memory will probably not be
    implemented by the reference implementation (there is a write up
    on this in the doc/ area), and there are yet various cases where
    work remains to be done.  And of course, not all of the
    implemented functionality has been tested yet.  The DAPL team
    continues to develop and test the tree with the intent of
    completing the specification and delivering a robust and useful
    implementation.


The DAPL Team

[Dapl-announce] uDAPL/kDAPL Beta 2.05 now available

[Sears, S. <Ste...@ne...>]

		   Release Notes for=20
		Beta 2.05 uDAPL/kDAPL Release
		   August 4, 2004


	DAPL BETA 2.05 RELEASE NOTES

	This was to be a very minor release, the primary change was
	going to be the new wording of the DAT license as contained in
	the header for all source files. But the interest and
	development occurring in DAPL provided some extra bug fixes, and
	some new functionality that has been requested for a while.

	First, you may notice that every single source file was
	changed. If you read the release notes from DAPL BETA 2.04, you
	were warned this would happen. There was a legal issue with the
	wording in the header, the end result was that every source file
	was required to change the word 'either of' to 'both'. We've
	been putting this change off as long as possible, but we wanted
	to do it in a clean drop before we start working on DAT 1.2
	changes in the reference implementation, just to keep things
	reasonably sane.

	kdapltest has enabled three of the subtests supported by
	dapltest. The Performance test in particular has been very
	useful to dapltest in getting minima and maxima. The Limit test
	pushes the limits by allocating the maximum number of specific
	resources. And the FFT tests are also available.

	Most vendors have supported shared memory regions for a while,
	several of which have asked the reference implementation team to
	provide a common implementation. Shared memory registration has
	been tested on ibapi, and compiled into vapi. Both InfiniBand
	providers have the restriction that a memory region must be
	created before it can be shared; not all RDMA APIs are this way,
	several allow you to declare a memory region shared when it is
	registered. Hence, details of the implementation are hidden in
	the provider layer, rather than forcing other APIs to do
	something strange.

	This release also contains some changes that will allow dapl to
	work on Opteron processors, as well as some preliminary support
	for Power PC architecture. These features are not well tested
	and may be incomplete at this time.

	Finally, we have been asked several times over the course of the
	project for a canonical interface between the common and
	provider layers. This release includes a dummy provider to meet
	that need. Anyone should be able to download the release and do
	a:
	   make VERBS=3DDUMMY

	And have a cleanly compiled dapl library. This will be useful
	both to those porting new transport providers, as well as those
	going to new machines.

	The DUMMY provider has been compiled on both Linux and Windows
	machines.


	NEW SINCE Beta 2.4
	- kdapltest enhancements:
	  * Limit subtests now work
	  * Performance subtests now work.
	  * FFT tests now work.

	- The VAPI headers have been refreshed by Mellanox

	- Initial Opteron and PPC support.

	- Atomic data types now have consistent treatment, allowing us to
	  use native data types other than integers. The Linux kdapl
	  uses atomic_t, allowing dapl to use the kernel macros and
	  eliminate the assembly code in dapl_osd.h

	- The license language was updated per the direction of the
	  DAT Collaborative. This two word change affected the header
	  of every file in the tree.

	- SHARED memory regions are now supported.

	- Initial support for the TOPSPIN provider.

	- Added a dummy provider, essentially the NULL provider. It's
	  purpose is to aid in porting and to clarify exactly what is
	  expected in a provider implementation.

	- Removed memory allocation from the DTO path for VAPI

	- cq_resize will now allow the CQ to be resized smaller. Not all
	  providers support this, but it's a provider problem, not a
	  limitation of the common code.

	* BUG FIXES

	+ Removed spurious lock in dapl_evd_connection_callb.c that
	  would have caused a deadlock.
	+ The Async EVD was getting torn down too early, potentially
	  causing lost errors. Has been moved later in the teardown
	  process.
	+ kDAPL replaced mem_map_reserve() with newer SetPageReserved()
	  for better Linux integration.
	+ kdapltest no longer allocate large print buffers on the stack,
	  is more careful to ensure buffers don't overflow.
	+ Put dapl_os_dbg_print() under DAPL_DBG conditional, it is
	  supposed to go away in a production build.=20
	+ dapltest protocol version has been bumped to reflect the
	  change in the Service ID.
	+ Corrected several instances of routines that did not adhere
	  to the DAT 1.1 error code scheme.
	+ Cleaned up vapi ib_reject_connection to pass DAT types rather
	  than provider specific types. Also cleaned up naming interface
	  declarations and their use in vapi_cm.c; fixed incorrect
	  #ifdef for naming. =20
	+ Initialize missing uDAPL provider attr, pz_support.
	+ Changes for better layering: first, moved
	  dapl_lmr_convert_privileges to the provider layer as memory
	  permissions are clearly transport specific and are not always
	  defined in an integer bitfield; removed common routines for
	  lmr and rmr. Second, move init and release setup/teardown
	  routines into adapter_util.h, which defined the provider
	  interface.
	+ Cleaned up the HCA name cruft that allowed different types
	  of names such as strings or ints to be dealt with in common
	  code; but all names are presented by the dat_registry as
	  strings, so pushed conversions down to the provider
	  level. Greatly simplifies names.
	+ Changed deprecated true/false to DAT_TRUE/DAT_FALSE.
	+ Removed old IB_HCA_NAME type in favor of char *.
	+ Fixed race condition in kdapltest's use of dat_evd_dequeue.=20
	+ Changed cast for SERVER_PORT_NUMBER to DAT_CONN_QUAL as it
	  should be.=20
	+ Small code reorg to put the CNO into the EVD when it is
	  allocated, which simplifies things.=20
	+ Removed gratuitous ib_hca_port_t and ib_send_op_type_t types,
	  replaced with standard int.
	+ Pass a pointer to cqe debug routine, not a structure. Some
	  clean up of data types.
	+ kdapl threads now invoke reparent_to_init() on exit to allow
	  threads to get cleaned up.



	DAPL BETA 2.04 RELEASE NOTES

	The big changes for this release involve a more strict adherence
	to the original dapl architecture. Originally, only InfiniBand
	providers were available, so allowing various data types and
	event codes to show through into common code wasn't a big deal.

	But today, there are an increasing number of providers available
	on a number of transports. Requiring an IP iWarp provider to
	match up to InfiniBand events is silly, for example.

	Restructuring the code allows more flexibility in providing an
	implementation.

	There are also a large number of bug fixes available in this
	release, particularly in kdapl related code.

	Be warned that the next release will change every file in the
	tree as we move to the newly approved DAT license. This is a
	small change, but all files are affected.

	Future releases will also support to the soon to be ratified DAT
	1.2 specification.

	This release has benefited from many bug reports and fixes from
	a number of individuals and companies. On behalf of the DAPL
	community, thank you!


	NEW SINCE Beta 2.3

	- Made several changes to be more rigorous on the layering
	  design of dapl. The intent is to make it easier for non
	  InfiniBand transports to use dapl. These changes include:
	 =20
	  * Revamped the ib_hca_open/close code to use an hca_ptr
	    rather than an ib_handle, giving the transport layer more
	    flexibility in assigning transport handles and resources.

	  * Removed the CQD calls, they are specific to the IBM API;
	    folded this functionality into the provider open/close calls.

	  * Moved VAPI, IBAPI transport specific items into a transport
	    structure placed inside of the HCA structure. Also updated
	    routines using these fields to use the new location. Cleaned
	    up provider knobs that have been exposed for too long.

	  * Changed a number of provider routines to use DAPL structure
	    pointers rather than exposing provider handles & values. Moved
	    provider specific items out of common code, including provider
	    data types (e.g. ib_uint32_t).

	  * Pushed provider completion codes and type back into the
            provider layer. We no longer use EVD or CM completion types =
at
            the common layer, instead we obtain the appropriate DAT type
            from the provider and process only DAT types.

	  * Change private_data handling such that we can now accommodate
            variable length private data.

	- Remove DAT 1.0 cruft from the DAT header files.

	- Better spec compliance in headers and various routines.

	- Major updates to the VAPI implementation from
          Mellanox. Includes initial kdapl implementation

	- Move kdapl platform specific support for hash routines into
          OSD file.

	- Cleanups to make the code more readable, including comments
          and certain variable and structure names.

	- Fixed CM_BUSTED code so that it works again: very useful for
          new dapl ports where infrastructure is lacking. Also made
	  some fixes for IBHOSTS_NAMING conditional code.

	- Added DAPL_MERGE_CM_DTO as a compile time switch to support
	  EVD stream merging of CM and DTO events. Default is off.

	- 'Quit' test ported to kdapltest

	- uDAPL now builds on Linux 2.6 platform (SuSE 9.1).

	- kDAPL now builds for a larger range of Linux kernels, but
          still lacks 2.6 support.

	- Added shared memory ID to LMR structure. Shared memory is
          still not fully supported in the reference implementation, but
          the common code will appear soon.

	* Bug fixes
	  - Various Makefiles fixed to use the correct dat registry
	    library in its new location (as of Beta 2.03)
	  - Simple reorg of dat headers files to be consistent with
	    the spec.
	  - fixed bug in vapi_dto.h recv macro where we could have an
	    uninitialized pointer.
	  - Simple fix in dat_dr.c to initialize a variable early in the
	    routine before errors occur.
	  - Removed private data pointers from a CONNECTED event, as
	    there should be no private data here.
	  - dat_strerror no longer returns an uninitialized pointer if
	    the error code is not recognized.
	  - dat_dup_connect() will reject 0 timeout values, per the
	    spec.
	  - Removed unused internal_hca_names parameter from
	    ib_enum_hcas() interface.=20
	  - Use a temporary DAT_EVENT for kdapl up-calls rather than
	    making assumptions about the current event queue.
	  - Relocated some platform dependent code to an OSD file.
	  - Eliminated several #ifdefs in .c files.
	  - Inserted a missing unlock() on an error path.
	  - Added bounds checking on size of private data to make sure
	    we don't overrun the buffer
	  - Fixed a kdapltest problem that caused a machine to panic if
	    the user hit ^C
	  - kdapltest now uses spin locks more appropriate for their
	    context, e.g. spin_lock_bh or spin_lock_irq. Under a
	    conditional.=20
	  - Fixed kdapltest loops that drain EVDs so they don't go into
	    endless loops.
	  - Fixed bug in dapl_llist_add_entry link list code.
	  - Better error reporting from provider code.
	  - Handle case of user trying to reap DTO completions on an
	    EP that has been freed.
	  - No longer hold lock when ep_free() calls into provider layer
	  - Fixed cr_accept() to not have an extra copy of
	    private_data.=20
	  - Verify private_data pointers before using them, avoid
	    panic.=20
	  - Fixed memory leak in kdapltest where print buffers were not
	    getting reclaimed.



	DAPL BETA 2.03 RELEASE NOTES

	There are some  prominent features in this release:
	1) dapltest/kdapltest. The dapltest test program has been
	   rearchitected such that a kernel version is now available
	   to test with kdapl. The most obvious change is a new
	   directory structure that more closely matches other core
	   dapl software. But there are a large number of changes
	   throughout the source files to accommodate both the
	   differences in udapl/kdapl interfaces, but also more mundane
	   things such as printing.

	   The new dapltest is in the tree at ./test/dapltest, while the
	   old remains at ./test/udapl/dapltest. For this release, we
	   have maintained both versions. In a future release, perhaps
	   the next release, the old dapltest directory will be
	   removed. Ongoing development will only occur in the new tree.

	2) DAT 1.1 compliance. The DAT Collaborative has been busy
	   finalizing the 1.1 revision of the spec. The header files
	   have been reviewed and posted on the DAT Collaborative web
	   site, they are now in full compliance.

	   The reference implementation has been at a 1.1 level for a
	   while. The current implementation has some features that will
	   be part of the 1.2 DAT specification, but only in places
	   where full compatibility can be maintained.

	3) The DAT Registry has undergone some positive changes for
           robustness and support of more platforms. It now has the
           ability to support several identical provider names
           simultaneously, which enables the same dat.conf file to
           support multiple platforms. The registry will open each
           library and return when successful. For example, a dat.conf
           file may contain multiple provider names for ex0a, each
           pointing to a different library that may represent different
           platforms or vendors. This simplifies distribution into
           different environments by enabling the use of common
           dat.conf files.

	In addition, there are a large number of bug fixes throughout
	the code. Bug reports and fixes have come from a number of
	companies.

	Also note that the Release notes are cleaned up, no longer
	containing the complete text of previous releases.

	* EVDs no longer support DTO and CONNECTION event types on the
          same EVD. NOTE: The problem is maintaining the event ordering
          between two channels such that no DTO completes before a
          connection is received; and no DTO completes after a
          disconnect is received. For 90% of the cases this can be made
          to work, but the remaining 10% will cause serious performance
          degradation to get right.

	NEW SINCE Beta 2.2

	* DAT 1.1 spec compliance. This includes some new types, error
          codes, and moving structures around in the header files,
          among other things. Note the Class bits of dat_error.h have
	  returned to a #define (from an enum) to cover the broadest
	  range of platforms.

	* Several additions for robustness, including handle and
          pointer checking, better argument checking, state
          verification, etc. Better recovery from error conditions,
	  and some assert()s have been replaced with 'if' statements to
          handle the error.

	* EVDs now maintain the actual queue length, rather than the
	  requested amount. Both the DAT spec and IB (and other
	  transports) allow the underlying implementation to provide
	  more CQ entries than requested.

	  Requests for the same number of entries contained by an EVD
	  return immediate success.

	* kDAPL enhancements:
	  - module parameters & OS support calls updated to work with
            more recent Linux kernels.
	  - kDAPL build options changes to match the Linux kernel, vastly
	    reducing the size and making it more robust.
	  - kDAPL unload now works properly
	  - kDAPL takes a reference on the provider driver when it
	    obtains a verbs vector, to prevent an accidental unload
	  - Cleaned out all of the uDAPL cruft from the linux/osd files.

	* New dapltest (see above).

	* Added a new I/O trace facility, enabling a developer to debug
          all I/O that are in progress or recently completed. Default
          is OFF in the build.

	* 0 timeout connections now refused, per the spec.

	* Moved the remaining uDAPL specific files from the common/
          directory to udapl/. Also removed udapl files from the kdapl
	  build.

	* Bug fixes
	  - Better error reporting from provider layer =20
	  - Fixed race condition on reference counts for posting DTO
	    ops.
	  - Use DAT_COMPLETION_SUPPRESS_FLAG to suppress successful
	    completion of dapl_rmr_bind  (instead of
	    DAT_COMPLEITON_UNSIGNALLED, which is for non-notification
	    completion).=20
	  - Verify psp_flags value per the spec
	  - Bug in psp_create_any() checking psp_flags fixed
	  - Fixed type of flags in ib_disconnect from
	    DAT_COMPLETION_FLAGS to DAT_CLOSE_FLAGS
	  - Removed hard coded check for ASYNC_EVD. Placed all EVD
	    prevention in evd_stream_merging_supported array, and
	    prevent ASYNC_EVD from being created by an app.
	  - ep_free() fixed to comply with the spec
	  - Replaced various printfs with dbg_log statements
	  - Fixed kDAPL interaction with the Linux kernel
	  - Corrected phy_register protottype
	  - Corrected kDAPL wait/wakeup synchronization
	  - Fixed kDAPL evd_kcreate() such that it no longer depends
	    on uDAPL only code.
	  - dapl_provider.h had wrong guard #def: changed DAT_PROVIDER_H
	    to DAPL_PROVIDER_H
	  - removed extra (and bogus) call to dapls_ib_completion_notify()
	    in evd_kcreate.c
	  - Inserted missing error code assignment in
	    dapls_rbuf_realloc()=20
	  - When a CONNECTED event arrives, make sure we are ready for
	    it, else something bad may have happened to the EP and we
	    just return; this replaces an explicit check for a single
	    error condition, replacing it with the general check for the
	    state capable of dealing with the request.
	  - Better context pointer verification. Removed locks around
	    call to ib_disconnect on an error path, which would result
	    in a deadlock. Added code for BROKEN events.
	  - Brought the vapi code more up to date: added conditional
	    compile switches, removed obsolete __ActivePort, deal
	    with 0 length DTO
	  - Several dapltest fixes to bring the code up to the 1.1
	    specification.
	  - Fixed mismatched dalp_os_dbg_print() #else dapl_Dbg_Print();
	    the latter was replaced with the former.
	  - ep_state_subtype() now includes UNCONNECTED.
	  - Added some missing ibapi error codes.
=20


	NEW SINCE Beta 2.1

	* Changes for Erratta and 1.1 Spec
	  - Removed DAT_NAME_NOT_FOUND, per DAT erratta
	  - EVD's with DTO and CONNECTION flags set no longer valid.
	  - Removed DAT_IS_SUCCESS macro
	  - Moved provider attribute structures from vendor files to udat.h
	    and kdat.h
	  - kdapl UPCALL_OBJECT now passed by reference

	* Completed dat_strerr return strings

	* Now support interrupted system calls

	* dapltest now used dat_strerror for error reporting.

	* Large number of files were formatted to meet project standard,
	  very cosmetic changes but improves readability and
	  maintainability.  Also cleaned up a number of comments during
	  this effort.

	* dat_registry and RPM file changes (contributed by Steffen Persvold):
	  - Renamed the RPM name of the registry to be dat-registry=20
	    (renamed the .spec file too, some cvs add/remove needed)
	  - Added the ability to create RPMs as normal user (using=20
	    temporal paths), works on SuSE, Fedora, and RedHat.
	  - 'make rpm' now works even if you didn't build first.
	  - Changed to using the GNU __attribute__((constructor)) and
	    __attribute__((destructor)) on the dat_init functions, dat_init
	    and dat_fini. The old -init and -fini options to LD makes=20
	    applications crash on some platforms (Fedora for example).
	  - Added support for 64 bit platforms.
	  - Added code to allow multiple provider names in the registry,
	    primarily to support ia32 and ia64 libraries simultaneously.=20
	    Provider names are now kept in a list, the first successful
	    library open will be the provider.

	* Added initial infrastructure for DAPL_DCNTR, a feature that
	  will aid in debug and tuning of a dapl implementation. Partial
	  implementation only at this point.

	* Bug fixes
	- Prevent debug messages from crashing dapl in EVD completions by
	  verifying the error code to ensure data is valid.
	- Verify CNO before using it to clean up in evd_free()
	- CNO timeouts now return correct error codes, per the spec.
	- cr_accept now complies with the spec concerning connection=20
	  requests that go away before the accept is invoked.
	- Verify valid EVD before posting connection evens on active side
	  of a connection. EP locking also corrected.
	- Clean up of dapltest Makefile, no longer need to declare
	  DAT_THREADSAFE
	- Fixed check of EP states to see if we need to disconnect an
	  IA is closed.
	- ep_free() code reworked such that we can properly close a=20
	  connection pending EP.
	- Changed disconnect processing to comply with the spec: user will
	   see a BROKEN event, not DISCONNECTED.
	- If we get a DTO error, issue a disconnect to let the CM and
	  the user know the EP state changed to disconnect; checked IBA
	  spec to make sure we disconnect on correct error codes.
	- ep_disconnect now properly deals with abrupt disconnects on the
	  active side of a connection.
	- PSP now created in the correct state for psp_create_any(), making
	  it usable.
	- dapl_evd_resize() now returns correct status, instead of always
	  DAT_NOT_IMPLEMENTED.
	- dapl_evd_modify_cno() does better error checking before invoking
	  the provider layer, avoiding bugs.
	- Simple change to allow dapl_evd_modify_cno() to set the CNO to=20
	  NULL, per the spec.
	- Added required locking around call to dapl_sp_remove_cr.

	- Fixed problems related to dapl_ep_free: the new
	  disconnect(abrupt) allows us to do a more immediate teardown of
	  connections, removing the need for the MAGIC_EP_EXIT magic
	  number/state, which has been removed. Mmuch cleanup of paths,
	  and made more robust.
	- Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is
	  triggered if there are waiters when the last EVD is removed
	  or when the IA is freed.
	- Added code to deal with the provider synchronously telling us
	   a connection is unreachable, and generate the appropriate
	   event.
	- Changed timer routine type from unsigned long to uintptr_t
	  to better fit with machine architectures.
	- ep.param data now initialized in ep_create, not ep_alloc.
	- Or Gerlitz provided updates to Mellanox files for evd_resize,
	  fw attributes, many others. Also implemented changes for correct
	  sizes on REP side of a connection request.



	NEW SINCE Beta 2.0

	* dat_echo now DAT 1.1 compliant. Various small enhancements.

	* Revamped atomic_inc/dec to be void, the return value was never
	  used. This allows kdapl to use Linux kernel equivalents, and
	  is a small performance advantage.

	* kDAPL: dapl_evd_modify_upcall implemented and tested.

	* kDAPL: physical memory registration implemented and tested.

	* uDAPL now builds cleanly for non-debug versions.

	* Default RDMA credits increased to 8.

	* Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
	  months).

	* Cleaned up dat_error.h, now 1.1 compliant in comments.

	* evd_resize initial implementation. Untested.

	* Bug fixes
	  - __KDAPL__ is defined in kdat_config.h, so apps don't need
	    to define it.
	  - Changed include file ordering in kdat.h to put kdat_config.h
	    first.
	  - resolved connection/tear-down race on the client side.
	  - kDAPL timeouts now scaled properly; fixed 3 orders of
	    magnitude difference.
	  - kDAPL EVD callbacks now get invoked for all completions; old
	    code would drop them in heavy utilization.
	  - Fixed error path in kDAPL evd creation, so we no longer
	    leak CNOs.
	  - create_psp_any returns correct error code if it can't create
	    a connection qualifier.
	  - lock fix in ibapi disconnect code.
	  - kDAPL INFINITE waits now work properly (non connection
	    waits)=20
	  - kDAPL driver unload now works properly
	  - dapl_lmr_[k]create now returns 1.1 error codes
	  - ibapi routines now return DAT 1.1 error codes
	 =20


	NEW SINCE Beta 1.10

	* kDAPL is now part of the DAPL distribution. See the release
	  notes above.

	  The kDAPL 1.1 spec is now contained in the doc/ subdirectory.

	* Several files have been moved around as part of the kDAPL
	  checkin. Some files that were previously in udapl/ are now
	  in common/, some in common are now in udapl/. The goal was
	  to make sure files are properly located and make sense for
	  the build.

	* Source code formatting changes for consistency.

	* Bug fixes
	  - dapl_evd_create() was comparing the wrong bit combinations,
	    allowing bogus EVDs to be created.
	  - Removed code that swallowed zero length I/O requests, which
	    are allowed by the spec and are useful to applications.
	  - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
	    routine will take and release the lock. Cosmetic change.
	  - dapl_get_consuemr_context() will now verify the pointer
	    argument 'context' is not NULL.




	OBTAIN THE CODE

	To obtain the tree for your local machine you can check it
	out of the source repository using CVS tools. CVS is common
	on Unix systems and available as freeware on Windows machines.
	The command to anonymously obtain the source code from=20
	Source Forge (with no password) is:
=09
	cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
	cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.

	When prompted for a password, simply  press the Enter key.

	Source Forge also contains explicit directions on how to become
	a developer, as well as how to use different CVS commands. You may
	also browse the source code using the URL:

	   http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/


	SYSTEM REQUIREMENTS

	This project has been implemented on Red Hat Linux 7.3, SuSE
	SLES 8, Windows 2000, RHEL 3.0, and a couple of other Linux
	distrubutions. The structure of the code is designed to allow
	other operating systems to easily be adapted.

	The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
	development, and continues with this platform. Our HCAs use the
	IB verbs API submitted by IBM. Mellanox has contributed an
	adapter layer using their VAPI verbs API. Either platform is
	available to any group considering DAPL work. The structure of
	the uDAPL source allows other provider API sets to be easily
	integrated.

	The development team uses any one of three topologies: two HCAs
	in a single machine; a single HCA in each of two machines; and
	most commonly, a switch. Machines connected to a switch may have
	more than one HCA.

	The DAPL Plugfest revealed that switches and HCAs available from
	most vendors will interoperate with little trouble, given the
	most recent releases of software. The dapl reference team makes
	no recommendation on HCA or switch vendors.

	Explicit machine configurations are available upon request.=09


	IN THE TREE

	The DAPL tree contains source code for the uDAPL and kDAPL
	implementations, and also includes tests and documentation.

	Included documentation has the base level API of the
	providers: the IBM Access API and the Mellanox Verbs API. Also
	included are a growing number of DAPL design documents which
	lead the reader through specific DAPL subsystems. More
	design documents are in progress and will appear in the tree in
	the near future.

	A small number of test applications and a unit test framework
	are also included. dapltest is the primary testing application
	used by the DAPL team, it is capable of simulating a variety of
	loads and exercises a large number of interfaces. Full
	documentation is included for each of the tests.

	Recently, the dapl conformance test has been added to the source
	repository. The test provides coverage of the most common
	interfaces, doing both positive and negative testing. Vendors
	providing DAPL implementation are strongly encouraged to run
	this set of tests.


	MAKEFILE NOTES

	There are a number #ifdef's in the code that were necessary
	during early development. They are disappearing as we
	have time to take advantage of features and work available from
	newer releases of provider software. You may notice an #ifdef
	<something>_BUSTED, which indicates a particular feature was not
	working at the time the code was written and the DAPL team
	developed a work-around.

	These #ifdefs are not documented as the intent is to remove
	them as soon as possible.

	Of particular relevance are the following #defines:

	- CM_BUSTED

	  The DAPL team has been an early adopter of InfiniBand and has
	  had to improvise missing functionality while the vendors lag
	  our development. InfiniBand uses a Connection Manager (CM) to
	  establish a connection between nodes. This #define essentially
	  'fakes' a connection by moving a QP into the appropriate
	  state. Most of the IB vendors have a working CM now and this
	  is no longer the default, but the code remains as some
	  development groups are working to catch up.

	- NO_NAME_SERVICE

	  Naming is a thorny issue in InfiniBand; translating from a
	  hostname or an interface name to a GID that can be used to
	  establish a connection with a remote machine. The reference
	  implementation provides a simple name service under this
	  #define. The goal is to use IPoIB when it becomes
	  available. NO_NAME_SERVICE will probably remain in the code
	  long term in order to enable various implementations. A
	  description of how this works is found in the end_point_design
	  document in the doc/ directory.


	CONTRIBUTIONS

	As is common to Source Forge projects, there are a small number
	of developers directly associated with the source tree and having
	privileges to change the tree. Requested updates, changes, bug
	fixes, enhancements, or contributions should be sent to Steve
	Sears at sj...@ne... for review. We welcome your
	contributions and expect the quality of the project will
	improve thanks to your help.

	The core DAPL team is:

	  Steve Sears
	  Philip Christopher

	  ... with contributions from a number of excellent engineers in
	  various companies contributing to the open source effort.


	ONGOING WORK

	Not all of the DAPL spec is implemented at this time.
	Functionality such as shared memory will probably not be
	implemented by the reference implementation (there is a write up
	on this in the doc/ area), and there are yet various cases where
	work remains to be done.  And of course, not all of the
	implemented functionality has been tested yet.  The DAPL team
	continues to develop and test the tree with the intent of
	completing the specification and delivering a robust and useful
	implementation.


The DAPL Team

[Dapl-announce] uDAPL/kDAPL Beta 2.04 now available

[Sears, S. <Ste...@ne...>]

           Release Notes for
        Beta 2.04 uDAPL/kDAPL Release
           June 15, 2004


    DAPL BETA 2.04 RELEASE NOTES

    The big changes for this release involve a more strict adherence
    to the original dapl architecture. Originally, only InfiniBand
    providers were available, so allowing various data types and
    event codes to show through into common code wasn't a big deal.

    But today, there are an increasing number of providers available
    on a number of transports. Requiring an IP iWarp provider to
    match up to InfiniBand events is silly, for example.

    Restructuring the code allows more flexibility in providing an
    implementation.

    There are also a large number of bug fixes available in this
    release, particularly in kdapl related code.

    Be warned that the next release will change every file in the
    tree as we move to the newly approved DAT license. This is a
    small change, but all files are affected.

    Future releases will also support to the soon to be ratified DAT
    1.2 specification.

    This release has benefited from many bug reports and fixes from
    a number of individuals and companies. On behalf of the DAPL
    community, thank you!


    NEW SINCE Beta 2.3

    - Made several changes to be more rigorous on the layering
      design of dapl. The intent is to make it easier for non
      InfiniBand transports to use dapl. These changes include:
     =20
      * Revamped the ib_hca_open/close code to use an hca_ptr
        rather than an ib_handle, giving the transport layer more
        flexibility in assigning transport handles and resources.

      * Removed the CQD calls, they are specific to the IBM API;
        folded this functionality into the provider open/close calls.

      * Moved VAPI, IBAPI transport specific items into a transport
        structure placed inside of the HCA structure. Also updated
        routines using these fields to use the new location. Cleaned
        up provider knobs that have been exposed for too long.

      * Changed a number of provider routines to use DAPL structure
        pointers rather than exposing provider handles & values. Moved
        provider specific items out of common code, including provider
        data types (e.g. ib_uint32_t).

      * Pushed provider completion codes and type back into the
            provider layer. We no longer use EVD or CM completion types =
at
            the common layer, instead we obtain the appropriate DAT type
            from the provider and process only DAT types.

      * Change private_data handling such that we can now accommodate
            variable length private data.

    - Remove DAT 1.0 cruft from the DAT header files.

    - Better spec compliance in headers and various routines.

    - Major updates to the VAPI implementation from
          Mellanox. Includes initial kdapl implementation

    - Move kdapl platform specific support for hash routines into
          OSD file.

    - Cleanups to make the code more readable, including comments
          and certain variable and structure names.

    - Fixed CM_BUSTED code so that it works again: very useful for
          new dapl ports where infrastructure is lacking. Also made
      some fixes for IBHOSTS_NAMING conditional code.

    - Added DAPL_MERGE_CM_DTO as a compile time switch to support
      EVD stream merging of CM and DTO events. Default is off.

    - 'Quit' test ported to kdapltest

    - uDAPL now builds on Linux 2.6 platform (SuSE 9.1).

    - kDAPL now builds for a larger range of Linux kernels, but
          still lacks 2.6 support.

    - Added shared memory ID to LMR structure. Shared memory is
          still not fully supported in the reference implementation, but
          the common code will appear soon.

    * Bug fixes
      - Various Makefiles fixed to use the correct dat registry
        library in its new location (as of Beta 2.03)
      - Simple reorg of dat headers files to be consistent with
        the spec.
      - fixed bug in vapi_dto.h recv macro where we could have an
        uninitialized pointer.
      - Simple fix in dat_dr.c to initialize a variable early in the
        routine before errors occur.
      - Removed private data pointers from a CONNECTED event, as
        there should be no private data here.
      - dat_strerror no longer returns an uninitialized pointer if
        the error code is not recognized.
      - dat_dup_connect() will reject 0 timeout values, per the
        spec.
      - Removed unused internal_hca_names parameter from
        ib_enum_hcas() interface.
      - Use a temporary DAT_EVENT for kdapl up-calls rather than
        making assumptions about the current event queue.
      - Relocated some platform dependent code to an OSD file.
      - Eliminated several #ifdefs in .c files.
      - Inserted a missing unlock() on an error path.
      - Added bounds checking on size of private data to make sure
        we don't overrun the buffer
      - Fixed a kdapltest problem that caused a machine to panic if
        the user hit ^C
      - kdapltest now uses spin locks more appropriate for their
        context, e.g. spin_lock_bh or spin_lock_irq. Under a
        conditional.
      - Fixed kdapltest loops that drain EVDs so they don't go into
        endless loops.
      - Fixed bug in dapl_llist_add_entry link list code.
      - Better error reporting from provider code.
      - Handle case of user trying to reap DTO completions on an
        EP that has been freed.
      - No longer hold lock when ep_free() calls into provider layer
      - Fixed cr_accept() to not have an extra copy of
        private_data.
      - Verify private_data pointers before using them, avoid
        panic.=20
      - Fixed memory leak in kdapltest where print buffers were not
        getting reclaimed.



    DAPL BETA 2.03 RELEASE NOTES

    There are some  prominent features in this release:
    1) dapltest/kdapltest. The dapltest test program has been
       rearchitected such that a kernel version is now available
       to test with kdapl. The most obvious change is a new
       directory structure that more closely matches other core
       dapl software. But there are a large number of changes
       throughout the source files to accommodate both the
       differences in udapl/kdapl interfaces, but also more mundane
       things such as printing.

       The new dapltest is in the tree at ./test/dapltest, while the
       old remains at ./test/udapl/dapltest. For this release, we
       have maintained both versions. In a future release, perhaps
       the next release, the old dapltest directory will be
       removed. Ongoing development will only occur in the new tree.

    2) DAT 1.1 compliance. The DAT Collaborative has been busy
       finalizing the 1.1 revision of the spec. The header files
       have been reviewed and posted on the DAT Collaborative web
       site, they are now in full compliance.

       The reference implementation has been at a 1.1 level for a
       while. The current implementation has some features that will
       be part of the 1.2 DAT specification, but only in places
       where full compatibility can be maintained.

    3) The DAT Registry has undergone some positive changes for
           robustness and support of more platforms. It now has the
           ability to support several identical provider names
           simultaneously, which enables the same dat.conf file to
           support multiple platforms. The registry will open each
           library and return when successful. For example, a dat.conf
           file may contain multiple provider names for ex0a, each
           pointing to a different library that may represent different
           platforms or vendors. This simplifies distribution into
           different environments by enabling the use of common
           dat.conf files.

    In addition, there are a large number of bug fixes throughout
    the code. Bug reports and fixes have come from a number of
    companies.

    Also note that the Release notes are cleaned up, no longer
    containing the complete text of previous releases.

    * EVDs no longer support DTO and CONNECTION event types on the
          same EVD. NOTE: The problem is maintaining the event ordering
          between two channels such that no DTO completes before a
          connection is received; and no DTO completes after a
          disconnect is received. For 90% of the cases this can be made
          to work, but the remaining 10% will cause serious performance
          degradation to get right.

    NEW SINCE Beta 2.2

    * DAT 1.1 spec compliance. This includes some new types, error
          codes, and moving structures around in the header files,
          among other things. Note the Class bits of dat_error.h have
      returned to a #define (from an enum) to cover the broadest
      range of platforms.

    * Several additions for robustness, including handle and
          pointer checking, better argument checking, state
          verification, etc. Better recovery from error conditions,
      and some assert()s have been replaced with 'if' statements to
          handle the error.

    * EVDs now maintain the actual queue length, rather than the
      requested amount. Both the DAT spec and IB (and other
      transports) allow the underlying implementation to provide
      more CQ entries than requested.

      Requests for the same number of entries contained by an EVD
      return immediate success.

    * kDAPL enhancements:
      - module parameters & OS support calls updated to work with
            more recent Linux kernels.
      - kDAPL build options changes to match the Linux kernel, vastly
        reducing the size and making it more robust.
      - kDAPL unload now works properly
      - kDAPL takes a reference on the provider driver when it
        obtains a verbs vector, to prevent an accidental unload
      - Cleaned out all of the uDAPL cruft from the linux/osd files.

    * New dapltest (see above).

    * Added a new I/O trace facility, enabling a developer to debug
          all I/O that are in progress or recently completed. Default
          is OFF in the build.

    * 0 timeout connections now refused, per the spec.

    * Moved the remaining uDAPL specific files from the common/
          directory to udapl/. Also removed udapl files from the kdapl
      build.

    * Bug fixes
      - Better error reporting from provider layer
      - Fixed race condition on reference counts for posting DTO
        ops.
      - Use DAT_COMPLETION_SUPPRESS_FLAG to suppress successful
        completion of dapl_rmr_bind  (instead of
        DAT_COMPLEITON_UNSIGNALLED, which is for non-notification
        completion).
      - Verify psp_flags value per the spec
      - Bug in psp_create_any() checking psp_flags fixed
      - Fixed type of flags in ib_disconnect from
        DAT_COMPLETION_FLAGS to DAT_CLOSE_FLAGS
      - Removed hard coded check for ASYNC_EVD. Placed all EVD
        prevention in evd_stream_merging_supported array, and
        prevent ASYNC_EVD from being created by an app.
      - ep_free() fixed to comply with the spec
      - Replaced various printfs with dbg_log statements
      - Fixed kDAPL interaction with the Linux kernel
      - Corrected phy_register protottype
      - Corrected kDAPL wait/wakeup synchronization
      - Fixed kDAPL evd_kcreate() such that it no longer depends
        on uDAPL only code.
      - dapl_provider.h had wrong guard #def: changed DAT_PROVIDER_H
        to DAPL_PROVIDER_H
      - removed extra (and bogus) call to dapls_ib_completion_notify()
        in evd_kcreate.c
      - Inserted missing error code assignment in
        dapls_rbuf_realloc()
      - When a CONNECTED event arrives, make sure we are ready for
        it, else something bad may have happened to the EP and we
        just return; this replaces an explicit check for a single
        error condition, replacing it with the general check for the
        state capable of dealing with the request.
      - Better context pointer verification. Removed locks around
        call to ib_disconnect on an error path, which would result
        in a deadlock. Added code for BROKEN events.
      - Brought the vapi code more up to date: added conditional
        compile switches, removed obsolete __ActivePort, deal
        with 0 length DTO
      - Several dapltest fixes to bring the code up to the 1.1
        specification.
      - Fixed mismatched dalp_os_dbg_print() #else dapl_Dbg_Print();
        the latter was replaced with the former.
      - ep_state_subtype() now includes UNCONNECTED.
      - Added some missing ibapi error codes.
=20


    NEW SINCE Beta 2.1

    * Changes for Erratta and 1.1 Spec
      - Removed DAT_NAME_NOT_FOUND, per DAT erratta
      - EVD's with DTO and CONNECTION flags set no longer valid.
      - Removed DAT_IS_SUCCESS macro
      - Moved provider attribute structures from vendor files to udat.h
        and kdat.h
      - kdapl UPCALL_OBJECT now passed by reference

    * Completed dat_strerr return strings

    * Now support interrupted system calls

    * dapltest now used dat_strerror for error reporting.

    * Large number of files were formatted to meet project standard,
      very cosmetic changes but improves readability and
      maintainability.  Also cleaned up a number of comments during
      this effort.

    * dat_registry and RPM file changes (contributed by Steffen =
Persvold):
      - Renamed the RPM name of the registry to be dat-registry
        (renamed the .spec file too, some cvs add/remove needed)
      - Added the ability to create RPMs as normal user (using
        temporal paths), works on SuSE, Fedora, and RedHat.
      - 'make rpm' now works even if you didn't build first.
      - Changed to using the GNU __attribute__((constructor)) and
        __attribute__((destructor)) on the dat_init functions, dat_init
        and dat_fini. The old -init and -fini options to LD makes
        applications crash on some platforms (Fedora for example).
      - Added support for 64 bit platforms.
      - Added code to allow multiple provider names in the registry,
        primarily to support ia32 and ia64 libraries simultaneously.
        Provider names are now kept in a list, the first successful
        library open will be the provider.

    * Added initial infrastructure for DAPL_DCNTR, a feature that
      will aid in debug and tuning of a dapl implementation. Partial
      implementation only at this point.

    * Bug fixes
    - Prevent debug messages from crashing dapl in EVD completions by
      verifying the error code to ensure data is valid.
    - Verify CNO before using it to clean up in evd_free()
    - CNO timeouts now return correct error codes, per the spec.
    - cr_accept now complies with the spec concerning connection
      requests that go away before the accept is invoked.
    - Verify valid EVD before posting connection evens on active side
      of a connection. EP locking also corrected.
    - Clean up of dapltest Makefile, no longer need to declare
      DAT_THREADSAFE
    - Fixed check of EP states to see if we need to disconnect an
      IA is closed.
    - ep_free() code reworked such that we can properly close a
      connection pending EP.
    - Changed disconnect processing to comply with the spec: user will
       see a BROKEN event, not DISCONNECTED.
    - If we get a DTO error, issue a disconnect to let the CM and
      the user know the EP state changed to disconnect; checked IBA
      spec to make sure we disconnect on correct error codes.
    - ep_disconnect now properly deals with abrupt disconnects on the
      active side of a connection.
    - PSP now created in the correct state for psp_create_any(), making
      it usable.
    - dapl_evd_resize() now returns correct status, instead of always
      DAT_NOT_IMPLEMENTED.
    - dapl_evd_modify_cno() does better error checking before invoking
      the provider layer, avoiding bugs.
    - Simple change to allow dapl_evd_modify_cno() to set the CNO to
      NULL, per the spec.
    - Added required locking around call to dapl_sp_remove_cr.

    - Fixed problems related to dapl_ep_free: the new
      disconnect(abrupt) allows us to do a more immediate teardown of
      connections, removing the need for the MAGIC_EP_EXIT magic
      number/state, which has been removed. Mmuch cleanup of paths,
      and made more robust.
    - Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is
      triggered if there are waiters when the last EVD is removed
      or when the IA is freed.
    - Added code to deal with the provider synchronously telling us
       a connection is unreachable, and generate the appropriate
       event.
    - Changed timer routine type from unsigned long to uintptr_t
      to better fit with machine architectures.
    - ep.param data now initialized in ep_create, not ep_alloc.
    - Or Gerlitz provided updates to Mellanox files for evd_resize,
      fw attributes, many others. Also implemented changes for correct
      sizes on REP side of a connection request.



    NEW SINCE Beta 2.0

    * dat_echo now DAT 1.1 compliant. Various small enhancements.

    * Revamped atomic_inc/dec to be void, the return value was never
      used. This allows kdapl to use Linux kernel equivalents, and
      is a small performance advantage.

    * kDAPL: dapl_evd_modify_upcall implemented and tested.

    * kDAPL: physical memory registration implemented and tested.

    * uDAPL now builds cleanly for non-debug versions.

    * Default RDMA credits increased to 8.

    * Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
      months).

    * Cleaned up dat_error.h, now 1.1 compliant in comments.

    * evd_resize initial implementation. Untested.

    * Bug fixes
      - __KDAPL__ is defined in kdat_config.h, so apps don't need
        to define it.
      - Changed include file ordering in kdat.h to put kdat_config.h
        first.
      - resolved connection/tear-down race on the client side.
      - kDAPL timeouts now scaled properly; fixed 3 orders of
        magnitude difference.
      - kDAPL EVD callbacks now get invoked for all completions; old
        code would drop them in heavy utilization.
      - Fixed error path in kDAPL evd creation, so we no longer
        leak CNOs.
      - create_psp_any returns correct error code if it can't create
        a connection qualifier.
      - lock fix in ibapi disconnect code.
      - kDAPL INFINITE waits now work properly (non connection
        waits)=20
      - kDAPL driver unload now works properly
      - dapl_lmr_[k]create now returns 1.1 error codes
      - ibapi routines now return DAT 1.1 error codes
     =20


    NEW SINCE Beta 1.10

    * kDAPL is now part of the DAPL distribution. See the release
      notes above.

      The kDAPL 1.1 spec is now contained in the doc/ subdirectory.

    * Several files have been moved around as part of the kDAPL
      checkin. Some files that were previously in udapl/ are now
      in common/, some in common are now in udapl/. The goal was
      to make sure files are properly located and make sense for
      the build.

    * Source code formatting changes for consistency.

    * Bug fixes
      - dapl_evd_create() was comparing the wrong bit combinations,
        allowing bogus EVDs to be created.
      - Removed code that swallowed zero length I/O requests, which
        are allowed by the spec and are useful to applications.
      - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
        routine will take and release the lock. Cosmetic change.
      - dapl_get_consuemr_context() will now verify the pointer
        argument 'context' is not NULL.




    OBTAIN THE CODE

    To obtain the tree for your local machine you can check it
    out of the source repository using CVS tools. CVS is common
    on Unix systems and available as freeware on Windows machines.
    The command to anonymously obtain the source code from
    Source Forge (with no password) is:
   =20
    cvs -d:pserver:ano...@cv...:/cvsroot/dapl =
login
    cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl =
co .

    When prompted for a password, simply  press the Enter key.

    Source Forge also contains explicit directions on how to become
    a developer, as well as how to use different CVS commands. You may
    also browse the source code using the URL:

       http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/


    SYSTEM REQUIREMENTS

    This project has been implemented on Red Hat Linux 7.3, SuSE
    SLES 8, Windows 2000, RHEL 3.0, and a couple of other Linux
    distrubutions. The structure of the code is designed to allow
    other operating systems to easily be adapted.

    The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
    development, and continues with this platform. Our HCAs use the
    IB verbs API submitted by IBM. Mellanox has contributed an
    adapter layer using their VAPI verbs API. Either platform is
    available to any group considering DAPL work. The structure of
    the uDAPL source allows other provider API sets to be easily
    integrated.

    The development team uses any one of three topologies: two HCAs
    in a single machine; a single HCA in each of two machines; and
    most commonly, a switch. Machines connected to a switch may have
    more than one HCA.

    The DAPL Plugfest revealed that switches and HCAs available from
    most vendors will interoperate with little trouble, given the
    most recent releases of software. The dapl reference team makes
    no recommendation on HCA or switch vendors.

    Explicit machine configurations are available upon request.


    IN THE TREE

    The DAPL tree contains source code for the uDAPL and kDAPL
    implementations, and also includes tests and documentation.

    Included documentation has the base level API of the
    providers: the IBM Access API and the Mellanox Verbs API. Also
    included are a growing number of DAPL design documents which
    lead the reader through specific DAPL subsystems. More
    design documents are in progress and will appear in the tree in
    the near future.

    A small number of test applications and a unit test framework
    are also included. dapltest is the primary testing application
    used by the DAPL team, it is capable of simulating a variety of
    loads and exercises a large number of interfaces. Full
    documentation is included for each of the tests.

    Recently, the dapl conformance test has been added to the source
    repository. The test provides coverage of the most common
    interfaces, doing both positive and negative testing. Vendors
    providing DAPL implementation are strongly encouraged to run
    this set of tests.


    MAKEFILE NOTES

    There are a number #ifdef's in the code that were necessary
    during early development. They are disappearing as we
    have time to take advantage of features and work available from
    newer releases of provider software. You may notice an #ifdef
    <something>_BUSTED, which indicates a particular feature was not
    working at the time the code was written and the DAPL team
    developed a work-around.

    These #ifdefs are not documented as the intent is to remove
    them as soon as possible.

    Of particular relevance are the following #defines:

    - CM_BUSTED

      The DAPL team has been an early adopter of InfiniBand and has
      had to improvise missing functionality while the vendors lag
      our development. InfiniBand uses a Connection Manager (CM) to
      establish a connection between nodes. This #define essentially
      'fakes' a connection by moving a QP into the appropriate
      state. Most of the IB vendors have a working CM now and this
      is no longer the default, but the code remains as some
      development groups are working to catch up.

    - NO_NAME_SERVICE

      Naming is a thorny issue in InfiniBand; translating from a
      hostname or an interface name to a GID that can be used to
      establish a connection with a remote machine. The reference
      implementation provides a simple name service under this
      #define. The goal is to use IPoIB when it becomes
      available. NO_NAME_SERVICE will probably remain in the code
      long term in order to enable various implementations. A
      description of how this works is found in the end_point_design
      document in the doc/ directory.


    CONTRIBUTIONS

    As is common to Source Forge projects, there are a small number
    of developers directly associated with the source tree and having
    privileges to change the tree. Requested updates, changes, bug
    fixes, enhancements, or contributions should be sent to Steve
    Sears at sj...@ne... for review. We welcome your
    contributions and expect the quality of the project will
    improve thanks to your help.

    The core DAPL team is:

      Steve Sears
      Philip Christopher

      ... with contributions from a number of excellent engineers in
      various companies contributing to the open source effort.


    ONGOING WORK

    Not all of the DAPL spec is implemented at this time.
    Functionality such as shared memory will probably not be
    implemented by the reference implementation (there is a write up
    on this in the doc/ area), and there are yet various cases where
    work remains to be done.  And of course, not all of the
    implemented functionality has been tested yet.  The DAPL team
    continues to develop and test the tree with the intent of
    completing the specification and delivering a robust and useful
    implementation.


The DAPL Team

[Dapl-announce] uDAPL/kDAPL Beta 2.03 now available

[Sears, S. <Ste...@ne...>]

           Release Notes for
        Beta 2.03 uDAPL/kDAPL Release
           April 9, 2004


    DAPL BETA 2.03 RELEASE NOTES

    There are some  prominent features in this release:
    1) dapltest/kdapltest. The dapltest test program has been
       rearchitected such that a kernel version is now available
       to test with kdapl. The most obvious change is a new
       directory structure that more closely matches other core
       dapl software. But there are a large number of changes
       throughout the source files to accommodate both the
       differences in udapl/kdapl interfaces, and also more mundane
       things such as printing.

       The new dapltest is in the tree at ./test/dapltest, while the
       old remains at ./test/udapl/dapltest. For this release, we
       have maintained both versions. In a future release, perhaps
       the next release, the old dapltest directory will be
       removed. Ongoing development will only occur in the new tree.

    2) DAT 1.1 compliance. The DAT Collaborative has been busy
       finalizing the 1.1 revision of the spec. The header files
       have been reviewed and posted on the DAT Collaborative web
       site, they are now in full compliance.

       The reference implementation has been at a 1.1 level for a
       while. The current implementation has some features that will
       be part of the 1.2 DAT specification, but only in places
       where full compatibility can be maintained.

    3) The DAT Registry has undergone some positive changes for
           robustness and support of more platforms. It now has the
           ability to support several identical provider names
           simultaneously, which enables the same dat.conf file to
           support multiple platforms. The registry will open each
           library and return when successful. For example, a dat.conf
           file may contain multiple provider names for ex0a, each
           pointing to a different library that may represent different
           platforms or vendors. This simplifies distribution into
           different environments by enabling the use of common
           dat.conf files.

    In addition, there are a large number of bug fixes throughout
    the code. Bug reports and fixes have come from a number of
    companies.

    Also note that the Release notes are cleaned up, no longer
    containing the complete text of previous releases.

    * EVDs no longer support DTO and CONNECTION event types on the
          same EVD. NOTE: The problem is maintaining the event ordering
          between two channels such that no DTO completes before a
          connection is received; and no DTO completes after a
          disconnect is received. For 90% of the cases this can be made
          to work, but the remaining 10% will cause serious performance
          degradation to get right.

    NEW SINCE Beta 2.2

    * DAT 1.1 spec compliance. This includes some new types, error
          codes, and moving structures around in the header files,
          among other things. Note the Class bits of dat_error.h have
      returned to a #define (from an enum) to cover the broadest
      range of platforms.

    * Several additions for robustness, including handle and
          pointer checking, better argument checking, state
          verification, etc. Better recovery from error conditions,
      and some assert()s have been replaced with 'if' statements to
          handle the error.

    * EVDs now maintain the actual queue length, rather than the
      requested amount. Both the DAT spec and IB (and other
      transports) allow the underlying implementation to provide
      more CQ entries than requested.

      Requests for the same number of entries contained by an EVD
      return immediate success.

    * kDAPL enhancements:
      - module parameters & OS support calls updated to work with
            more recent Linux kernels.
      - kDAPL build options changes to match the Linux kernel, vastly
        reducing the size and making it more robust.
      - kDAPL unload now works properly
      - kDAPL takes a reference on the provider driver when it
        obtains a verbs vector, to prevent an accidental unload
      - Cleaned out all of the uDAPL cruft from the linux/osd files.

    * New dapltest (see above).

    * Added a new I/O trace facility, enabling a developer to debug
          all I/O that are in progress or recently completed. Default
          is OFF in the build.

    * 0 timeout connections now refused, per the spec.

    * Moved the remaining uDAPL specific files from the common/
          directory to udapl/. Also removed udapl files from the kdapl
      build.

    * Bug fixes
      - Better error reporting from provider layer
      - Fixed race condition on reference counts for posting DTO
        ops.
      - Use DAT_COMPLETION_SUPPRESS_FLAG to suppress successful
        completion of dapl_rmr_bind  (instead of
        DAT_COMPLEITON_UNSIGNALLED, which is for non-notification
        completion).
      - Verify psp_flags value per the spec
      - Bug in psp_create_any() checking psp_flags fixed
      - Fixed type of flags in ib_disconnect from
        DAT_COMPLETION_FLAGS to DAT_CLOSE_FLAGS
      - Removed hard coded check for ASYNC_EVD. Placed all EVD
        prevention in evd_stream_merging_supported array, and
        prevent ASYNC_EVD from being created by an app.
      - ep_free() fixed to comply with the spec
      - Replaced various printfs with dbg_log statements
      - Fixed kDAPL interaction with the Linux kernel
      - Corrected phy_register protottype
      - Corrected kDAPL wait/wakeup synchronization
      - Fixed kDAPL evd_kcreate() such that it no longer depends
        on uDAPL only code.
      - dapl_provider.h had wrong guard #def: changed DAT_PROVIDER_H
        to DAPL_PROVIDER_H
      - removed extra (and bogus) call to dapls_ib_completion_notify()
        in evd_kcreate.c
      - Inserted missing error code assignment in
        dapls_rbuf_realloc()
      - When a CONNECTED event arrives, make sure we are ready for
        it, else something bad may have happened to the EP and we
        just return; this replaces an explicit check for a single
        error condition, replacing it with the general check for the
        state capable of dealing with the request.
      - Better context pointer verification. Removed locks around
        call to ib_disconnect on an error path, which would result
        in a deadlock. Added code for BROKEN events.
      - Brought the vapi code more up to date: added conditional
        compile switches, removed obsolete __ActivePort, deal
        with 0 length DTO
      - Several dapltest fixes to bring the code up to the 1.1
        specification.
      - Fixed mismatched dalp_os_dbg_print() #else dapl_Dbg_Print();
        the latter was replaced with the former.
      - ep_state_subtype() now includes UNCONNECTED.
      - Added some missing ibapi error codes.
=20


    NEW SINCE Beta 2.1

    * Changes for Erratta and 1.1 Spec
      - Removed DAT_NAME_NOT_FOUND, per DAT erratta
      - EVD's with DTO and CONNECTION flags set no longer valid.
      - Removed DAT_IS_SUCCESS macro
      - Moved provider attribute structures from vendor files to udat.h
        and kdat.h
      - kdapl UPCALL_OBJECT now passed by reference

    * Completed dat_strerr return strings

    * Now support interrupted system calls

    * dapltest now used dat_strerror for error reporting.

    * Large number of files were formatted to meet project standard,
      very cosmetic changes but improves readability and
      maintainability.  Also cleaned up a number of comments during
      this effort.

    * dat_registry and RPM file changes (contributed by Steffen =
Persvold):
      - Renamed the RPM name of the registry to be dat-registry
        (renamed the .spec file too, some cvs add/remove needed)
      - Added the ability to create RPMs as normal user (using
        temporal paths), works on SuSE, Fedora, and RedHat.
      - 'make rpm' now works even if you didn't build first.
      - Changed to using the GNU __attribute__((constructor)) and
        __attribute__((destructor)) on the dat_init functions, dat_init
        and dat_fini. The old -init and -fini options to LD makes
        applications crash on some platforms (Fedora for example).
      - Added support for 64 bit platforms.
      - Added code to allow multiple provider names in the registry,
        primarily to support ia32 and ia64 libraries simultaneously.
        Provider names are now kept in a list, the first successful
        library open will be the provider.

    * Added initial infrastructure for DAPL_DCNTR, a feature that
      will aid in debug and tuning of a dapl implementation. Partial
      implementation only at this point.

    * Bug fixes
    - Prevent debug messages from crashing dapl in EVD completions by
      verifying the error code to ensure data is valid.
    - Verify CNO before using it to clean up in evd_free()
    - CNO timeouts now return correct error codes, per the spec.
    - cr_accept now complies with the spec concerning connection
      requests that go away before the accept is invoked.
    - Verify valid EVD before posting connection evens on active side
      of a connection. EP locking also corrected.
    - Clean up of dapltest Makefile, no longer need to declare
      DAT_THREADSAFE
    - Fixed check of EP states to see if we need to disconnect an
      IA is closed.
    - ep_free() code reworked such that we can properly close a
      connection pending EP.
    - Changed disconnect processing to comply with the spec: user will
       see a BROKEN event, not DISCONNECTED.
    - If we get a DTO error, issue a disconnect to let the CM and
      the user know the EP state changed to disconnect; checked IBA
      spec to make sure we disconnect on correct error codes.
    - ep_disconnect now properly deals with abrupt disconnects on the
      active side of a connection.
    - PSP now created in the correct state for psp_create_any(), making
      it usable.
    - dapl_evd_resize() now returns correct status, instead of always
      DAT_NOT_IMPLEMENTED.
    - dapl_evd_modify_cno() does better error checking before invoking
      the provider layer, avoiding bugs.
    - Simple change to allow dapl_evd_modify_cno() to set the CNO to
      NULL, per the spec.
    - Added required locking around call to dapl_sp_remove_cr.

    - Fixed problems related to dapl_ep_free: the new
      disconnect(abrupt) allows us to do a more immediate teardown of
      connections, removing the need for the MAGIC_EP_EXIT magic
      number/state, which has been removed. Mmuch cleanup of paths,
      and made more robust.
    - Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is
      triggered if there are waiters when the last EVD is removed
      or when the IA is freed.
    - Added code to deal with the provider synchronously telling us
       a connection is unreachable, and generate the appropriate
       event.
    - Changed timer routine type from unsigned long to uintptr_t
      to better fit with machine architectures.
    - ep.param data now initialized in ep_create, not ep_alloc.
    - Or Gerlitz provided updates to Mellanox files for evd_resize,
      fw attributes, many others. Also implemented changes for correct
      sizes on REP side of a connection request.



    NEW SINCE Beta 2.0

    * dat_echo now DAT 1.1 compliant. Various small enhancements.

    * Revamped atomic_inc/dec to be void, the return value was never
      used. This allows kdapl to use Linux kernel equivalents, and
      is a small performance advantage.

    * kDAPL: dapl_evd_modify_upcall implemented and tested.

    * kDAPL: physical memory registration implemented and tested.

    * uDAPL now builds cleanly for non-debug versions.

    * Default RDMA credits increased to 8.

    * Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
      months).

    * Cleaned up dat_error.h, now 1.1 compliant in comments.

    * evd_resize initial implementation. Untested.

    * Bug fixes
      - __KDAPL__ is defined in kdat_config.h, so apps don't need
        to define it.
      - Changed include file ordering in kdat.h to put kdat_config.h
        first.
      - resolved connection/tear-down race on the client side.
      - kDAPL timeouts now scaled properly; fixed 3 orders of
        magnitude difference.
      - kDAPL EVD callbacks now get invoked for all completions; old
        code would drop them in heavy utilization.
      - Fixed error path in kDAPL evd creation, so we no longer
        leak CNOs.
      - create_psp_any returns correct error code if it can't create
        a connection qualifier.
      - lock fix in ibapi disconnect code.
      - kDAPL INFINITE waits now work properly (non connection
        waits)=20
      - kDAPL driver unload now works properly
      - dapl_lmr_[k]create now returns 1.1 error codes
      - ibapi routines now return DAT 1.1 error codes
     =20


    NEW SINCE Beta 1.10

    * kDAPL is now part of the DAPL distribution. See the release
      notes above.

      The kDAPL 1.1 spec is now contained in the doc/ subdirectory.

    * Several files have been moved around as part of the kDAPL
      checkin. Some files that were previously in udapl/ are now
      in common/, some in common are now in udapl/. The goal was
      to make sure files are properly located and make sense for
      the build.

    * Source code formatting changes for consistency.

    * Bug fixes
      - dapl_evd_create() was comparing the wrong bit combinations,
        allowing bogus EVDs to be created.
      - Removed code that swallowed zero length I/O requests, which
        are allowed by the spec and are useful to applications.
      - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
        routine will take and release the lock. Cosmetic change.
      - dapl_get_consuemr_context() will now verify the pointer
        argument 'context' is not NULL.




    OBTAIN THE CODE

    To obtain the tree for your local machine you can check it
    out of the source repository using CVS tools. CVS is common
    on Unix systems and available as freeware on Windows machines.
    The command to anonymously obtain the source code from
    Source Forge (with no password) is:
   =20
    cvs -d:pserver:ano...@cv...:/cvsroot/dapl =
login
    cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl =
co .

    When prompted for a password, simply  press the Enter key.

    Source Forge also contains explicit directions on how to become
    a developer, as well as how to use different CVS commands. You may
    also browse the source code using the URL:

       http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/


    SYSTEM REQUIREMENTS

    This project has been implemented on Red Hat Linux 7.3, SuSE
    SLES 8, Windows 2000, RHEL 3.0, and a couple of other Linux
    distrubutions. The structure of the code is designed to allow
    other operating systems to easily be adapted.

    The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
    development, and continues with this platform. Our HCAs use the
    IB verbs API submitted by IBM. Mellanox has contributed an
    adapter layer using their VAPI verbs API. Either platform is
    available to any group considering DAPL work. The structure of
    the uDAPL source allows other provider API sets to be easily
    integrated.

    The development team uses any one of three topologies: two HCAs
    in a single machine; a single HCA in each of two machines; and
    most commonly, a switch. Machines connected to a switch may have
    more than one HCA.

    The DAPL Plugfest revealed that switches and HCAs available from
    most vendors will interoperate with little trouble, given the
    most recent releases of software. The dapl reference team makes
    no recommendation on HCA or switch vendors.

    Explicit machine configurations are available upon request.


    IN THE TREE

    The DAPL tree contains source code for the uDAPL and kDAPL
    implementations, and also includes tests and documentation.

    Included documentation has the base level API of the
    providers: the IBM Access API and the Mellanox Verbs API. Also
    included are a growing number of DAPL design documents which
    lead the reader through specific DAPL subsystems. More
    design documents are in progress and will appear in the tree in
    the near future.

    A small number of test applications and a unit test framework
    are also included. dapltest is the primary testing application
    used by the DAPL team, it is capable of simulating a variety of
    loads and exercises a large number of interfaces. Full
    documentation is included for each of the tests.

    Recently, the dapl conformance test has been added to the source
    repository. The test provides coverage of the most common
    interfaces, doing both positive and negative testing. Vendors
    providing DAPL implementation are strongly encouraged to run
    this set of tests.


    MAKEFILE NOTES

    There are a number #ifdef's in the code that were necessary
    during early development. They are disappearing as we
    have time to take advantage of features and work available from
    newer releases of provider software. You may notice an #ifdef
    <something>_BUSTED, which indicates a particular feature was not
    working at the time the code was written and the DAPL team
    developed a work-around.

    These #ifdefs are not documented as the intent is to remove
    them as soon as possible.

    Of particular relevance are the following #defines:

    - CM_BUSTED

      The DAPL team has been an early adopter of InfiniBand and has
      had to improvise missing functionality while the vendors lag
      our development. InfiniBand uses a Connection Manager (CM) to
      establish a connection between nodes. This #define essentially
      'fakes' a connection by moving a QP into the appropriate
      state. Most of the IB vendors have a working CM now and this
      is no longer the default, but the code remains as some
      development groups are working to catch up.

    - NO_NAME_SERVICE

      Naming is a thorny issue in InfiniBand; translating from a
      hostname or an interface name to a GID that can be used to
      establish a connection with a remote machine. The reference
      implementation provides a simple name service under this
      #define. The goal is to use IPoIB when it becomes
      available. NO_NAME_SERVICE will probably remain in the code
      long term in order to enable various implementations. A
      description of how this works is found in the end_point_design
      document in the doc/ directory.


    CONTRIBUTIONS

    As is common to Source Forge projects, there are a small number
    of developers directly associated with the source tree and having
    privileges to change the tree. Requested updates, changes, bug
    fixes, enhancements, or contributions should be sent to Steve
    Sears at sj...@ne... for review. We welcome your
    contributions and expect the quality of the project will
    improve thanks to your help.

    The core DAPL team is:

      Steve Sears
      Philip Christopher

      ... with contributions from a number of excellent engineers in
      various companies contributing to the open source effort.


    ONGOING WORK

    Not all of the DAPL spec is implemented at this time.
    Functionality such as shared memory will probably not be
    implemented by the reference implementation (there is a write up
    on this in the doc/ area), and there are yet various cases where
    work remains to be done.  And of course, not all of the
    implemented functionality has been tested yet.  The DAPL team
    continues to develop and test the tree with the intent of
    completing the specification and delivering a robust and useful
    implementation.


The DAPL Team
uDAPL/kDAPL Beta 2.02 now available

[Dapl-announce] uDAPL/kDAPL Beta 2.02 now available

[Sears, S. <Ste...@ne...>]

     Release Notes for=20
  Beta 2.02 uDAPL/kDAPL Release
     January 7, 2004
=20
=20
=20
 DAPL BETA 2.02 RELEASE NOTES
=20
 There have been a large number of bug fixes and further spec
 compliance since the last release. The DAPL team expects more
 bug fixing to occur over the next month or two as the
 implementation continues to become more robust and mature.
=20
 Driving these changes are the adoption of DAPL by several
 commercial products found in different industries, and on
 different transport technologies.
=20
 The reference team continues to develop on InfiniBand, but
 observe that DAPL implementations now also exist on GigE,
 Myrinet, MPI, and at least one proprietary interconnect. If
 license terms change such that the API of any of these
 transports is put into the public domain, we encourage
 contributions to the reference implementation; we'd love to have
 the provider layer files and make it available to the DAPL
 community.
=20
 We also express our appreciation to the many people who have
 contributed to this release; this release, even more than
 others, has really demonstrated the value of Open Source
 collaboration. Some of the contributors are identified in the
 notes below, but many others contributed bug fixes and analysis
 that is extremely valuable.
=20
 And just to forestall some email, the answer is no, we still
 haven't gotten around to updating the design notes in the doc/
 directory. Hopefully it'll happen soon.
=20
 DAPL BETA 2.0 RELEASE NOTES
=20
 The DAPL team is pleased to announce the addition of kDAPL to
 the dapl repository. This code has been contributed by Fujitsu
 Prime Software Technologies, Limited, the DAPL project is very
 appreciative of this excellent work!
=20
 The implementation is believed to be complete, but has not been
 thoroughly tested. At least one large application uses it, as
 well as a number of test programs in use by the dapl team,
 providing some confidence that the core implementation is
 working. There are several routines that have explicitly NOT
 been tested, please see the comments next to the provider jump
 vector in ./common/dapl_provider.c.
=20
 kDAPL has a large source code overlap with uDAPL, so they are
 tightly integrated into the same tree. There are a number of
 changes to the location of certain files as we separated kDAPL
 from uDAPL from common into their appropriate directories. The
 files have not been renamed, but may be in different directories
 than you expect if you have been working with previous versions
 of DAPL.
=20
 Some of the files in ./kDAPL contain conditional code under
 #if defined(__KDAPL__); this code is currently similar to the=20
 uDAPL version so conditionals exist to make it easier to merge
 bug fixes to either tree.
=20
 There are a handful if conditional statements in the common and
 adapter code as well. For the most part, we have avoided
 conditionals as they make the code unreadable, but they are
 justified in a few cases.
=20
 This code has been tested on the JNI InfiniBand HCA, used by the
 DAPL team, which is based the IB API verbs. This API supports
 the same verbs in user and kernel space, the difference being
 the kernel version of the verbs is accessed through a jump
 vector obtained from the provider driver. A sample of this
 vector is found in ./ibapi/verbslist.h. THE verbslist.h FILE IN
 THE DISTRIBUTION IS FAKE, IT IS ONLY PROVIDED AS AN EXAMPLE AND
 TO ALLOW THE CODE TO COMPILE. Your HCA provider vendor must
 provide you with the proper verbs structure and must implement
 the Linux inter_module interfaces to obtain it. Note that this
 sample file may be incomplete for a product.
=20
 At this time, the equivalent VAPI work has not occurred. If you
 can contribute the necessary pieces to support kDAPL on that
 interface it would be greatly appreciated.
=20
 The top level build does not compile the kDAPL trees, they must
 be done manually. In order to use kdapl:
=20
 > cd dat/kdat
 > make
 > cd Target
 > insmod dat_registry.o
 > cd ../../../dapl/kdapl
 > make
 > cd Target
 > insmod dapl.o
=20
 You can now do an lsmod and see your drivers running. To state
 the obvious, if your provider driver is not running then you may
 run into trouble.
=20
 The kdapl driver also supports a limited set of debugging
 options, using the same format as provided by the uDAPL
 environment variable DAPL_DBG_TYPE. To get debug messages
 printed on the console, use
=20
 > insmod dapl.i DbgLvl=3D0xff
=20
 ... or whatever your favorite set of debug bits is. kdapl runs
 as a driver, so printing debug messages to the console has a
 serious impact on system performance.
=20
 As always, the DAPL community appreciates the contribution of
 bug fixes and code that will complete the implementation.
=20
 There may yet be spec violations and/or unimplemented options in
 this code, it continues as a work in progress.
=20

 SPECIAL NOTE ON BETA 1.0
=20
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable.=20
=20
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
=20
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
=20
 RELEASE NOTES
=20
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
=20
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
=20
    <http://sourceforge.net/projects/dapl> =
http://sourceforge.net/projects/dapl
=20
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
=20
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
=20
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
=20

 NEW SINCE Beta 2.1
=20
 * Changes for Erratta and 1.1 Spec
   - Removed DAT_NAME_NOT_FOUND, per DAT erratta
   - EVD's with DTO and CONNECTION flags set no longer valid.
   - Removed DAT_IS_SUCCESS macro
   - Moved provider attribute structures from vendor files to udat.h
     and kdat.h
   - kdapl UPCALL_OBJECT now passed by reference(!!)
=20
 * Completed dat_strerr return strings
=20
 * Now support interrupted system calls
=20
 * dapltest now used dat_strerror for error reporting.
=20
 * Large number of files were formatted to meet project standard,
   very cosmetic changes but improves readability and
   maintainability.  Also cleaned up a number of comments during
   this effort.
=20
 * dat_registry and RPM file changes (contributed by Steffen Persvold):
   - Renamed the RPM name of the registry to be dat-registry=20
     (renamed the .spec file too, some cvs add/remove needed)
   - Added the ability to create RPMs as normal user (using=20
     temporal paths), works on SuSE, Fedora, and RedHat.
   - 'make rpm' now works even if you didn't build first.
   - Changed to using the GNU __attribute__((constructor)) and
     __attribute__((destructor)) on the dat_init functions, dat_init
     and dat_fini. The old -init and -fini options to LD makes=20
     applications crash on some platforms (Fedora for example).
   - Added support for 64 bit platforms.
   - Added code to allow multiple provider names in the registry,
     primarily to support ia32 and ia64 libraries simultaneously.=20
     Provider names are now kept in a list, the first successful
     library open will be the provider.
=20
 * Added initial infrastructure for DAPL_DCNTR, a feature that
   will aid in debug and tuning of a dapl implementation. Partial
   implementation only at this point.
=20
 * Bug fixes
 - Prevent debug messages from crashing dapl in EVD completions by
   verifying the error code to ensure data is valid.
 - Verify CNO before using it to clean up in evd_free()
 - CNO timeouts now return correct error codes, per the spec.
 - cr_accept now complies with the spec concerning connection=20
   requests that go away before the accept is invoked.
 - Verify valid EVD before posting connection evens on active side
   of a connection. EP locking also corrected.
 - Clean up of dapltest Makefile, no longer need to declare
   DAT_THREADSAFE
 - Fixed check of EP states to see if we need to disconnect an
   IA is closed.
 - ep_free() code reworked such that we can properly close a=20
   connection pending EP.
 - Changed disconnect processing to comply with the spec: user will
    see a BROKEN event, not DISCONNECTED.
 - If we get a DTO error, issue a disconnect to let the CM and
   the user know the EP state changed to disconnect; checked IBA
   spec to make sure we disconnect on correct error codes.
 - ep_disconnect now properly deals with abrupt disconnects on the
   active side of a connection.
 - PSP now created in the correct state for psp_create_any(), making
   it usable.
 - dapl_evd_resize() now returns correct status, instead of always
   DAT_NOT_IMPLEMENTED.
 - dapl_evd_modify_cno() does better error checking before invoking
   the provider layer, avoiding bugs.
 - Simple change to allow dapl_evd_modify_cno() to set the CNO to=20
   NULL, per the spec.
 - Added required locking around call to dapl_sp_remove_cr.
=20
 - Fixed problems related to dapl_ep_free: the new
   disconnect(abrupt) allows us to do a more immediate teardown of
   connections, removing the need for the MAGIC_EP_EXIT magic
   number/state, which has been removed. Mmuch cleanup of paths,
   and made more robust.
 - Made changes to meet the spec, uDAPL 1.1 6.3.2.3: CNO is
   triggered if there are waiters when the last EVD is removed
   or when the IA is freed.
 - Added code to deal with the provider synchronously telling us
    a connection is unreachable, and generate the appropriate
    event.
 - Changed timer routine type from unsigned long to uintptr_t
   to better fit with machine architectures.
 - ep.param data now initialized in ep_create, not ep_alloc.
 - Or Gerlitz provided updates to Mellanox files for evd_resize,
   fw attributes, many others. Also implemented changes for correct
   sizes on REP side of a connection request.
=20
=20
=20
 NEW SINCE Beta 2.0
=20
 * dat_echo now DAT 1.1 compliant. Various small enhancements.
=20
 * Revamped atomic_inc/dec to be void, the return value was never
   used. This allows kdapl to use Linux kernel equivalents, and
   is a small performance advantage.
=20
 * kDAPL: dapl_evd_modify_upcall implemented and tested.
=20
 * kDAPL: physical memory registration implemented and tested.
=20
 * uDAPL now builds cleanly for non-debug versions.
=20
 * Default RDMA credits increased to 8.
=20
 * Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
   months).
=20
 * Cleaned up dat_error.h, now 1.1 compliant in comments.
=20
 * evd_resize initial implementation. Untested.
=20
 * Bug fixes
   - __KDAPL__ is defined in kdat_config.h, so apps don't need
     to define it.
   - Changed include file ordering in kdat.h to put kdat_config.h
     first.
   - resolved connection/tear-down race on the client side.
   - kDAPL timeouts now scaled properly; fixed 3 orders of
     magnitude difference.
   - kDAPL EVD callbacks now get invoked for all completions; old
     code would drop them in heavy utilization.
   - Fixed error path in kDAPL evd creation, so we no longer
     leak CNOs.
   - create_psp_any returns correct error code if it can't create
     a connection qualifier.
   - lock fix in ibapi disconnect code.
   - kDAPL INFINITE waits now work properly (non connection
     waits)=20
   - kDAPL driver unload now works properly
   - dapl_lmr_[k]create now returns 1.1 error codes
   - ibapi routines now return DAT 1.1 error codes
  =20
=20

 NEW SINCE Beta 1.10
=20
 * kDAPL is now part of the DAPL distribution. See the release
   notes above.
=20
   The kDAPL 1.1 spec is now contained in the doc/ subdirectory.
=20
 * Several files have been moved around as part of the kDAPL
   checkin. Some files that were previously in udapl/ are now
   in common/, some in common are now in udapl/. The goal was
   to make sure files are properly located and make sense for
   the build.
=20
 * Source code formatting changes for consistency.
=20
 * Bug fixes
   - dapl_evd_create() was comparing the wrong bit combinations,
     allowing bogus EVDs to be created.
   - Removed code that swallowed zero length I/O requests, which
     are allowed by the spec and are useful to applications.
   - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
     routine will take and release the lock. Cosmetic change.
   - dapl_get_consuemr_context() will now verify the pointer
     argument 'context' is not NULL.
=20
=20
=20
 NEW SINCE Beta 1.09
=20
 * We now take advantage of cm_disconnect(abrupt), which has the
   semantics to disconnect the connection but will not generate
   a callback. DAPL does generate a disconnect event for the app,
   so there is some care needed. This also introduces some
   interesting race conditions for the number of disconnect
   scenarios, e.g. remote disconnect results in asynchronous
   local disconnect callback; disconnect(graceful) followed by
   disconnect(abrupt) may still get a callback, etc. The code
   now properly deals with these race conditions.
=20
 * Increased the number of RDMA_READ credits (ibapi only) to allow
   better performance for overlapped I/O.
=20
 * Bug fixes
   - Typos and formatting changes to clean up and help make the code
     base more consistent. Added a couple new debug statements.
   - CR records now removed from the PSP/RSP when doing a
     close(abrupt)
   - Added casts for 64 bit values to prevent inadvertent sign
     propagation.
   - Do a better job of cleaning up EP fields if a cr_accept fails,
     which prevents false information from being presented by the
     next ep_query.
   - Fixed ia_close_abrupt to check for empty queues before trying
     to dequeue, which was causing an assert.
   - Changed the ACK_TIMEOUT value to be within the range specified
     by the IBTA spec. Noticed various DTO errors when going through
     a switch with high traffic, traced down to insufficient ACK
     timeout value. ibapi only.
       =20
=20
 NEW SINCE Beta 1.08
=20
 * Support for DAT RPM is now complete.
=20
 * Minor documentation updates
=20
 * Fixed dat_error.h to have a DAT_NO_SUBTYPE as the first (zeroth)
   element of the subtype enum. If there is no subtype, dat_strerr
   should not report something bogus.
=20
 * dapltest fixes:
   - fixed FFT memory test
   - fft_queryinfo now uses correct set of DAT enums for query
     ops
   - Cleanup of fft initializers
   - Thread and EP synchronization has been revamped and should be
     correct. Two major problems here:
     1) All EP's used the same connection EVD, making it very likely
        that a specific EP would lose a race and harvest the
        wrong event.
     2) Threads were not synchronized on start/stop, so they could
        easily miss connections or get rejected when they were
        confused.
=20
 * Modest number of simple cleanups
=20
 * Bug fixes
   - Several SMP synchronization fixes:
     a) RSP/PSP locking and synchronization
     b) ep_free and CM thread synchronization
     c) disconnect_clean called with locks held
     d) EP locking corrected for disconnects
     e) connection timer setup/teardown
   - Added support to dapl_ia_close() force a QP into the ERROR=20
     state in order to force posted ops to flush; used by
     disconnect ABRUPT at present.
   - Fixed missing data structure when calling the provider to
     modify the QP.
   - Fixed bug where we copied the IPv6 address twice when querying
     the QP.
   - Fixed ep_disconnect to be a no-op if the EP is disconnected.
   - EVD's are now initialized to enabled state per the spec
   - Corrected notion that UNSIGNALLED means the same thing in
     IB and DAPL, which is not the case.
=20
=20
=20
=20
=20
 NEW SINCE Beta 1.07
 * Red Hat Package Management (RPM) specification file and
   makefile updates to allow manual RPM creation of dat library.
   Automated support for creating a DAT RPM will come in a future
   release.
=20
 * Bug fixes
   - cr_query will return correct status and the remote IP
     address of the connecting node.
   - 'threshold' value in evd_wait is tested against the queue
     length and rejected if too large.
   - Fixed synchronization between RSP/PSP free and the CM
     callback thread.
   - Fixed synchronization between EP free and the CM callback=20
     thread.
   - Spec compliance for ep_disconnect() to be a no-op if the
     EP is DISCONNECTED.
  =20
=20
 NEW SINCE Beta 1.06
 * All DAPL ALPHA release notes removed from this file.
=20
 * For IBM Access providers, ib_types.h has been updated such
   that max_block_bytes is now 64 bits (instead of 32) to allow
   large memories. THIS CHANGE IMPLIES ia_query IS
    INCOMPATIBLE WITH OLD VERSIONS.
=20
 * local and remote address formats reworked to present as
   AF_INET (IPv4) addresses if possible.
=20
 * dapltest now displays local HCA IP address as part of config
   info (use -d to see it).
=20
 * Removed compile time references to PSC, no longer in who is
   business.=20
=20
 * dapltest changed to use service_id instead of pid when
   printing debug messages. Now both sides of the wire print
   the same value for connected threads.
=20
   Also added simple test for private_data on connections; errors
   are non fatal, but will cause a warning message to be printed.
=20
 * Changed the dat headers to allow MAJOR, MINOR, and THREADSAFE
   #defines to be in config files; per DAT Collaborative
   revision. Reference implementation default is now
   DAT_THREADSAFE=3DFALSE.
=20
 * Cleanded up various debug messages that occasionally spew
   out.=20
=20
 * Deleted unused fields in dapl structures, & bogus references
   to them in code.
=20
 * VAPI files updated and building cleanly again.
=20
 * Framework for SOLICITED_WAIT support now in place. Still not
   fully implemented.
=20

 * Conformance test updates:
   - depends_on results field more consistent & correct
   - Send  & recv DTO tests added
   - Updated RDMA tests
   - Enhanced recv tests
   - Connection management tests added
   - CNO tests added
  =20
=20
 * Bug fixes
   - Provider allocated EP cleaned up on cr_reject
   - dapl_ia_open fixed to prevent memory leaks on error paths
   - Using a reject reason of CONSUMER_REJECT (defined by
     IBTA), we can get a CM callback type of
     IB_CME_DESTINATION_REJECT_PRIVATE_DATA, even if there is no
     private data; all other reject types result in a callback
     of IB_CME_DESTINATION_REJECT. This allows us to generate
     the correct events on the connection EVD and to distinguish
     app level rejects vs. system level rejects or no listener
     present.
   - dapl_hca structure provides storage for local EP ip
     addresses.=20
   - In dapl_ep_create.c, cleaned up request_evd error checking
     to return correct error values.=20
   - Fixed bug when providing an EP for a CR, link it onto the
     IA.
   - Various small changes to keep compilers happy.
   - Fixed race condition between a CM thread and an app thread
     when releasing an SP object.
=20

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented.=20
=20
 * ib_enum_hca_if() moved below the IB abstraction layer.
=20
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
=20
 * Dynamic library loading support added for Windows
=20
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
=20
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
=20
=20
=20
 NEW SINCE Beta 1.04
=20
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
=20
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
=20
 * DAT registry is now fully 1.1 compliant.
=20
 * evd_stream_merging_supported enabled and used.
=20
 * DAT headers updated for DAT errata 99 and 100.
=20
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
=20
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
   =20
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making=20
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
=20

=20
 NEW SINCE Beta 1.03
=20
 * strerror() implemented
=20
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
=20
 * dat.conf file example updated for 1.1 spec
=20
 * Now obtain firmware information from the hca_attrib structure
   for ibapi.=20
=20
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
=20
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
=20
 * DTO/RMR completion status returns now comply with 1.1 spec.
=20
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
=20
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
=20
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
=20
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
=20

 NEW SINCE Beta 1.02
=20
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
=20
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
=20
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
=20
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added=20
=20
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to=20
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
=20

 NEW SINCE Beta 1.01
=20
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
=20
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
=20
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
=20
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!).=20
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
=20

 NEW SINCE Beta 1.0
=20
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming.=20
=20
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
=20
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
=20
 * Memory tests and echo tests updated to 1.1
=20
 * Assertion for 'over decremented' reference counts
=20
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups;=20
=20
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h.=20
=20
 * Ignore 0 length IOVs (1.1 compliance)
=20
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
=20
 * memory test now has ability to request invalid memry regions
   for negative testing.
=20
 * Enhanced error reporting (more subtypes added to returns)
=20
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
=20
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take=20
     corrective action.
   - ep_create will verify ep_attributes=20
=20
 * rmr_context now correctly returned from lmr_create().
    =20
=20

 OBTAIN THE CODE
=20
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from=20
 Source Forge (with no password) is:
=20
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.
=20
 When prompted for a password, simply  press the Enter key.
=20
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
=20
     <http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/> =
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
=20

 SYSTEM REQUIREMENTS
=20
 This project has been implemented on Red Hat Linux 7.3, and SuSE
 SLES 8. The structure of the code is designed to allow other
 operating systems to easily be adapted, but no work has been
 done in that direction.
=20
 The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
 development, and continue with this platform. Our HCAs use the
 IB verbs API submitted by IBM. Mellanox has contributed an
 adapter layer using their VAPI verbs API. Either platform is
 available to any group considering DAPL work. The structure of
 the uDAPL source allows other provider API sets to be easily
 integrated.
=20
 The development team uses any one of four topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines; and most
 commonly, a switch.
=20
 The DAPL Plugfest revealed that switches and HCAs available from
 most vendors will interoperate with little trouble, given the
 most recent releases of software. The dapl reference team makes
 no recommendation on HCA or switch vendors.
=20
 Explicit machine configurations are available upon request.=20
=20

 IN THE TREE
=20
 The DAPL tree contains source code for the uDAPL and kDAPL
 implementations, and also includes tests and documentation.
=20
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
=20
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
=20
 Recently, the dapl conformance test has been added to the source
 repository. The test provides coverage of the most common
 interfaces, doing both positive and negative testing. Vendors
 providing DAPL implementation are strongly encouraged to run
 this set of tests.
=20

 MAKEFILE NOTES
=20
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
=20
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
=20
 Of particular relevance are the following #defines:
=20
 - CM_BUSTED
=20
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
=20
 - NO_NAME_SERVICE
=20
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
=20

 CONTRIBUTIONS
=20
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at  <mailto:sj...@ne...> sj...@ne... for review. We welcome =
your
 contributions and expect the quality of the project will
 improve thanks to your help.
=20
 The core DAPL team is:
=20
   Steve Sears
   Randy Smith
=20
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
=20

 ONGOING WORK
=20
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
=20

The DAPL Team

[Dapl-announce] uDAPL/kDAPL Beta 2.01 now available

[Sears, S. <Ste...@ne...>]

     Release Notes for=20
  Beta 2.01 uDAPL/kDAPL Release
     October 28, 2003
=20
=20
=20
 DAPL BETA 2.0 RELEASE NOTES
=20
 The DAPL team is pleased to announce the addition of kDAPL to
 the dapl repository. This code has been contributed by Fujitsu
 Prime Software Technologies, Limited, the DAPL project is very
 appreciative of this excellent work!
=20
 The implementation is believed to be complete, but has not been
 thoroughly tested. At least one large application uses it, as
 well as a number of test programs in use by the dapl team,
 providing some confidence that the core implementation is
 working. There are several routines that have explicitly NOT
 been tested, please see the comments next to the provider jump
 vector in ./common/dapl_provider.c.
=20
 kDAPL has a large source code overlap with uDAPL, so they are
 tightly integrated into the same tree. There are a number of
 changes to the location of certain files as we separated kDAPL
 from uDAPL from common into their appropriate directories. The
 files have not been renamed, but may be in different directories
 than you expect if you have been working with previous versions
 of DAPL.
=20
 Some of the files in ./kDAPL contain conditional code under
 #if defined(__KDAPL__); this code is currently similar to the=20
 uDAPL version so conditionals exist to make it easier to merge
 bug fixes to either tree.
=20
 There are a handful if conditional statements in the common and
 adapter code as well. For the most part, we have avoided
 conditionals as they make the code unreadable, but they are
 justified in a few cases.
=20
 This code has been tested on the JNI InfiniBand HCA, used by the
 DAPL team, which is based the IB API verbs. This API supports
 the same verbs in user and kernel space, the difference being
 the kernel version of the verbs is accessed through a jump
 vector obtained from the provider driver. A sample of this
 vector is found in ./ibapi/verbslist.h. THE verbslist.h FILE IN
 THE DISTRIBUTION IS FAKE, IT IS ONLY PROVIDED AS AN EXAMPLE AND
 TO ALLOW THE CODE TO COMPILE. Your HCA provider vendor must
 provide you with the proper verbs structure and must implement
 the Linux inter_module interfaces to obtain it. Note that this
 sample file may be incomplete for a product.
=20
 At this time, the equivalent VAPI work has not occurred. If you
 can contribute the necessary pieces to support kDAPL on that
 interface it would be greatly appreciated.
=20
 The top level build does not compile the kDAPL trees, they must
 be done manually. In order to use kdapl:
=20
 > cd dat/kdat
 > make
 > cd Target
 > insmod dat_registry.o
 > cd ../../../dapl/kdapl
 > make
 > cd Target
 > insmod dapl.o
=20
 You can now do an lsmod and see your drivers running. To state
 the obvious, if your provider driver is not running then you may
 run into trouble.
=20
 The kdapl driver also supports a limited set of debugging
 options, using the same format as provided by the uDAPL
 environment variable DAPL_DBG_TYPE. To get debug messages
 printed on the console, use
=20
 > insmod dapl.i DbgLvl=3D0xff
=20
 ... or whatever your favorite set of debug bits is. kdapl runs
 as a driver, so printing debug messages to the console has a
 serious impact on system performance.
=20
 As always, the DAPL community appreciates the contribution of
 bug fixes and code that will complete the implementation.
=20
 There may yet be spec violations and/or unimplemented options in
 this code, it continues as a work in progress.
=20

 SPECIAL NOTE ON BETA 1.0
=20
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable.=20
=20
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
=20
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
=20
 RELEASE NOTES
=20
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
=20
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
=20
   http://sourceforge.net/projects/dapl
=20
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
=20
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
=20
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
=20

 NEW SINCE Beta 2.0
=20
 * dat_echo now DAT 1.1 compliant. Various small enhancements.
=20
 * Revamped atomic_inc/dec to be void, the return value was never
   used. This allows kdapl to use Linux kernel equivalents, and
   is a small performance advantage.
=20
 * kDAPL: dapl_evd_modify_upcall implemented and tested.
=20
 * kDAPL: physical memory registration implemented and tested.
=20
 * uDAPL now builds cleanly for non-debug versions.
=20
 * Default RDMA credits increased to 8.
=20
 * Default ACK_TIMEOUT now a reasonable value (2 sec vs old 2
   months).
=20
 * Cleaned up dat_error.h, now 1.1 compliant in comments.
=20
 * evd_resize initial implementation. Untested.
=20
 * Bug fixes
   - __KDAPL__ is defined in kdat_config.h, so apps don't need
     to define it.
   - Changed include file ordering in kdat.h to put kdat_config.h
     first.
   - resolved connection/tear-down race on the client side.
   - kDAPL timeouts now scaled properly; fixed 3 orders of
     magnitude difference.
   - kDAPL EVD callbacks now get invoked for all completions; old
     code would drop them in heavy utilization.
   - Fixed error path in kDAPL evd creation, so we no longer
     leak CNOs.
   - create_psp_any returns correct error code if it can't create
     a connection qualifier.
   - lock fix in ibapi disconnect code.
   - kDAPL INFINITE waits now work properly (non connection
     waits)=20
   - kDAPL driver unload now works properly
   - dapl_lmr_[k]create now returns 1.1 error codes
   - ibapi routines now return DAT 1.1 error codes
  =20
=20

 NEW SINCE Beta 1.10
=20
 * kDAPL is now part of the DAPL distribution. See the release
   notes above.
=20
   The kDAPL 1.1 spec is now contained in the doc/ subdirectory.
=20
 * Several files have been moved around as part of the kDAPL
   checkin. Some files that were previously in udapl/ are now
   in common/, some in common are now in udapl/. The goal was
   to make sure files are properly located and make sense for
   the build.
=20
 * Source code formatting changes for consistency.
=20
 * Bug fixes
   - dapl_evd_create() was comparing the wrong bit combinations,
     allowing bogus EVDs to be created.
   - Removed code that swallowed zero length I/O requests, which
     are allowed by the spec and are useful to applications.
   - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
     routine will take and release the lock. Cosmetic change.
   - dapl_get_consuemr_context() will now verify the pointer
     argument 'context' is not NULL.
=20
=20
=20
 NEW SINCE Beta 1.09
=20
 * We now take advantage of cm_disconnect(abrupt), which has the
   semantics to disconnect the connection but will not generate
   a callback. DAPL does generate a disconnect event for the app,
   so there is some care needed. This also introduces some
   interesting race conditions for the number of disconnect
   scenarios, e.g. remote disconnect results in asynchronous
   local disconnect callback; disconnect(graceful) followed by
   disconnect(abrupt) may still get a callback, etc. The code
   now properly deals with these race conditions.
=20
 * Increased the number of RDMA_READ credits (ibapi only) to allow
   better performance for overlapped I/O.
=20
 * Bug fixes
   - Typos and formatting changes to clean up and help make the code
     base more consistent. Added a couple new debug statements.
   - CR records now removed from the PSP/RSP when doing a
     close(abrupt)
   - Added casts for 64 bit values to prevent inadvertent sign
     propagation.
   - Do a better job of cleaning up EP fields if a cr_accept fails,
     which prevents false information from being presented by the
     next ep_query.
   - Fixed ia_close_abrupt to check for empty queues before trying
     to dequeue, which was causing an assert.
   - Changed the ACK_TIMEOUT value to be within the range specified
     by the IBTA spec. Noticed various DTO errors when going through
     a switch with high traffic, traced down to insufficient ACK
     timeout value. ibapi only.
       =20
=20
 NEW SINCE Beta 1.08
=20
 * Support for DAT RPM is now complete.
=20
 * Minor documentation updates
=20
 * Fixed dat_error.h to have a DAT_NO_SUBTYPE as the first (zeroth)
   element of the subtype enum. If there is no subtype, dat_strerr
   should not report something bogus.
=20
 * dapltest fixes:
   - fixed FFT memory test
   - fft_queryinfo now uses correct set of DAT enums for query
     ops
   - Cleanup of fft initializers
   - Thread and EP synchronization has been revamped and should be
     correct. Two major problems here:
     1) All EP's used the same connection EVD, making it very likely
        that a specific EP would lose a race and harvest the
        wrong event.
     2) Threads were not synchronized on start/stop, so they could
        easily miss connections or get rejected when they were
        confused.
=20
 * Modest number of simple cleanups
=20
 * Bug fixes
   - Several SMP synchronization fixes:
     a) RSP/PSP locking and synchronization
     b) ep_free and CM thread synchronization
     c) disconnect_clean called with locks held
     d) EP locking corrected for disconnects
     e) connection timer setup/teardown
   - Added support to dapl_ia_close() force a QP into the ERROR=20
     state in order to force posted ops to flush; used by
     disconnect ABRUPT at present.
   - Fixed missing data structure when calling the provider to
     modify the QP.
   - Fixed bug where we copied the IPv6 address twice when querying
     the QP.
   - Fixed ep_disconnect to be a no-op if the EP is disconnected.
   - EVD's are now initialized to enabled state per the spec
   - Corrected notion that UNSIGNALLED means the same thing in
     IB and DAPL, which is not the case.
=20
=20
=20
=20
=20
 NEW SINCE Beta 1.07
 * Red Hat Package Management (RPM) specification file and
   makefile updates to allow manual RPM creation of dat library.
   Automated support for creating a DAT RPM will come in a future
   release.
=20
 * Bug fixes
   - cr_query will return correct status and the remote IP
     address of the connecting node.
   - 'threshold' value in evd_wait is tested against the queue
     length and rejected if too large.
   - Fixed synchronization between RSP/PSP free and the CM
     callback thread.
   - Fixed synchronization between EP free and the CM callback=20
     thread.
   - Spec compliance for ep_disconnect() to be a no-op if the
     EP is DISCONNECTED.
  =20
=20
 NEW SINCE Beta 1.06
 * All DAPL ALPHA release notes removed from this file.
=20
 * For IBM Access providers, ib_types.h has been updated such
   that max_block_bytes is now 64 bits (instead of 32) to allow
   large memories. THIS CHANGE IMPLIES ia_query IS
    INCOMPATIBLE WITH OLD VERSIONS.
=20
 * local and remote address formats reworked to present as
   AF_INET (IPv4) addresses if possible.
=20
 * dapltest now displays local HCA IP address as part of config
   info (use -d to see it).
=20
 * Removed compile time references to PSC, no longer in who is
   business.=20
=20
 * dapltest changed to use service_id instead of pid when
   printing debug messages. Now both sides of the wire print
   the same value for connected threads.
=20
   Also added simple test for private_data on connections; errors
   are non fatal, but will cause a warning message to be printed.
=20
 * Changed the dat headers to allow MAJOR, MINOR, and THREADSAFE
   #defines to be in config files; per DAT Collaborative
   revision. Reference implementation default is now
   DAT_THREADSAFE=3DFALSE.
=20
 * Cleanded up various debug messages that occasionally spew
   out.=20
=20
 * Deleted unused fields in dapl structures, & bogus references
   to them in code.
=20
 * VAPI files updated and building cleanly again.
=20
 * Framework for SOLICITED_WAIT support now in place. Still not
   fully implemented.
=20

 * Conformance test updates:
   - depends_on results field more consistent & correct
   - Send  & recv DTO tests added
   - Updated RDMA tests
   - Enhanced recv tests
   - Connection management tests added
   - CNO tests added
  =20
=20
 * Bug fixes
   - Provider allocated EP cleaned up on cr_reject
   - dapl_ia_open fixed to prevent memory leaks on error paths
   - Using a reject reason of CONSUMER_REJECT (defined by
     IBTA), we can get a CM callback type of
     IB_CME_DESTINATION_REJECT_PRIVATE_DATA, even if there is no
     private data; all other reject types result in a callback
     of IB_CME_DESTINATION_REJECT. This allows us to generate
     the correct events on the connection EVD and to distinguish
     app level rejects vs. system level rejects or no listener
     present.
   - dapl_hca structure provides storage for local EP ip
     addresses.=20
   - In dapl_ep_create.c, cleaned up request_evd error checking
     to return correct error values.=20
   - Fixed bug when providing an EP for a CR, link it onto the
     IA.
   - Various small changes to keep compilers happy.
   - Fixed race condition between a CM thread and an app thread
     when releasing an SP object.
=20

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented.=20
=20
 * ib_enum_hca_if() moved below the IB abstraction layer.
=20
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
=20
 * Dynamic library loading support added for Windows
=20
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
=20
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
=20
=20
=20
 NEW SINCE Beta 1.04
=20
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
=20
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
=20
 * DAT registry is now fully 1.1 compliant.
=20
 * evd_stream_merging_supported enabled and used.
=20
 * DAT headers updated for DAT errata 99 and 100.
=20
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
=20
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
   =20
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making=20
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
=20

=20
 NEW SINCE Beta 1.03
=20
 * strerror() implemented
=20
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
=20
 * dat.conf file example updated for 1.1 spec
=20
 * Now obtain firmware information from the hca_attrib structure
   for ibapi.=20
=20
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
=20
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
=20
 * DTO/RMR completion status returns now comply with 1.1 spec.
=20
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
=20
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
=20
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
=20
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
=20

 NEW SINCE Beta 1.02
=20
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
=20
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
=20
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
=20
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added=20
=20
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to=20
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
=20

 NEW SINCE Beta 1.01
=20
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
=20
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
=20
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
=20
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!).=20
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
=20

 NEW SINCE Beta 1.0
=20
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming.=20
=20
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
=20
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
=20
 * Memory tests and echo tests updated to 1.1
=20
 * Assertion for 'over decremented' reference counts
=20
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups;=20
=20
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h.=20
=20
 * Ignore 0 length IOVs (1.1 compliance)
=20
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
=20
 * memory test now has ability to request invalid memry regions
   for negative testing.
=20
 * Enhanced error reporting (more subtypes added to returns)
=20
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
=20
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take=20
     corrective action.
   - ep_create will verify ep_attributes=20
=20
 * rmr_context now correctly returned from lmr_create().
    =20
=20

 OBTAIN THE CODE
=20
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from=20
 Source Forge (with no password) is:
=20
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.
=20
 When prompted for a password, simply  press the Enter key.
=20
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
=20
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
=20

 SYSTEM REQUIREMENTS
=20
 This project has been implemented on Red Hat Linux 7.3, and SuSE
 SLES 8. The structure of the code is designed to allow other
 operating systems to easily be adapted, but no work has been
 done in that direction.
=20
 The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
 development, and continue with this platform. Our HCAs use the
 IB verbs API submitted by IBM. Mellanox has contributed an
 adapter layer using their VAPI verbs API. Either platform is
 available to any group considering DAPL work. The structure of
 the uDAPL source allows other provider API sets to be easily
 integrated.
=20
 The development team uses any one of four topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines; and most
 commonly, a switch.
=20
 The DAPL Plugfest revealed that switches and HCAs available from
 most vendors will interoperate with little trouble, given the
 most recent releases of software. The dapl reference team makes
 no recommendation on HCA or switch vendors.
=20
 Explicit machine configurations are available upon request.=20
=20

 IN THE TREE
=20
 The DAPL tree contains source code for the uDAPL and kDAPL
 implementations, and also includes tests and documentation.
=20
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
=20
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
=20
 Recently, the dapl conformance test has been added to the source
 repository. The test provides coverage of the most common
 interfaces, doing both positive and negative testing. Vendors
 providing DAPL implementation are strongly encouraged to run
 this set of tests.
=20

 MAKEFILE NOTES
=20
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
=20
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
=20
 Of particular relevance are the following #defines:
=20
 - CM_BUSTED
=20
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
=20
 - NO_NAME_SERVICE
=20
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
=20

 CONTRIBUTIONS
=20
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at sj...@ne... for review. We welcome your
 contributions and expect the quality of the project will
 improve thanks to your help.
=20
 The core DAPL team is:
=20
   Steve Sears
   Randy Smith
=20
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
=20

 ONGOING WORK
=20
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
=20

The DAPL Team

[Dapl-announce] uDAPL/kDAPL Beta 2.0 now available

[Sears, S. <Ste...@ne...>]

     Release Notes for=20
  Beta 2.0 uDAPL/kDAPL Release
     October 28, 2003
=20
=20
=20
 DAPL BETA 2.0 RELEASE NOTES
=20
 The DAPL team is pleased to announce the addition of kDAPL to
 the dapl repository. This code has been contributed by Fujitsu
 Prime Software Technologies, Limited, the DAPL project is very
 appreciative of this excellent work!
=20
 The implementation is believed to be complete, but has not been
 thoroughly tested. At least one large application uses it, as
 well as a number of test programs in use by the dapl team,
 providing some confidence that the core implementation is
 working. There are several routines that have explicitly NOT
 been tested, please see the comments next to the provider jump
 vector in ./common/dapl_provider.c.
=20
 kDAPL has a large source code overlap with uDAPL, so they are
 tightly integrated into the same tree. There are a number of
 changes to the location of certain files as we separated kDAPL
 from uDAPL from common into their appropriate directories. The
 files have not been renamed, but may be in different directories
 than you expect if you have been working with previous versions
 of DAPL.
=20
 Some of the files in ./kDAPL contain conditional code under
 #if defined(__KDAPL__); this code is currently similar to the=20
 uDAPL version so conditionals exist to make it easier to merge
 bug fixes to either tree.
=20
 There are a handful if conditional statements in the common and
 adapter code as well. For the most part, we have avoided
 conditionals as they make the code unreadable, but they are
 justified in a few cases.
=20
 This code has been tested on the JNI InfiniBand HCA, used by the
 DAPL team, which is based the IB API verbs. This API supports
 the same verbs in user and kernel space, the difference being
 the kernel version of the verbs is accessed through a jump
 vector obtained from the provider driver. A sample of this
 vector is found in ./ibapi/verbslist.h. THE verbslist.h FILE IN
 THE DISTRIBUTION IS FAKE, IT IS ONLY PROVIDED AS AN EXAMPLE AND
 TO ALLOW THE CODE TO COMPILE. Your HCA provider vendor must
 provide you with the proper verbs structure and must implement
 the Linux inter_module interfaces to obtain it. Note that this
 sample file may be incomplete for a product.
=20
 At this time, the equivalent VAPI work has not occurred. If you
 can contribute the necessary pieces to support kDAPL on that
 interface it would be greatly appreciated.
=20
 The top level build does not compile the kDAPL trees, they must
 be done manually. In order to use kdapl:
=20
 > cd dat/kdat
 > make
 > cd Target
 > insmod dat_registry.o
 > cd ../../../dapl/kdapl
 > make
 > cd Target
 > insmod dapl.o
=20
 You can now do an lsmod and see your drivers running. To state
 the obvious, if your provider driver is not running then you may
 run into trouble.
=20
 The kdapl driver also supports a limited set of debugging
 options, using the same format as provided by the uDAPL
 environment variable DAPL_DBG_TYPE. To get debug messages
 printed on the console, use
=20
 > insmod dapl.i DbgLvl=3D0xff
=20
 ... or whatever your favorite set of debug bits is. kdapl runs
 as a driver, so printing debug messages to the console has a
 serious impact on system performance.
=20
 As always, the DAPL community appreciates the contribution of
 bug fixes and code that will complete the implementation.
=20
 There may yet be spec violations and/or unimplemented options in
 this code, it continues as a work in progress.
=20

 SPECIAL NOTE ON BETA 1.0
=20
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable.=20
=20
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
=20
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
=20
 RELEASE NOTES
=20
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
=20
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
=20
   http://sourceforge.net/projects/dapl
=20
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
=20
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
=20
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
=20

 NEW SINCE Beta 1.10
=20
 * kDAPL is now part of the DAPL distribution. See the release
   notes above.
=20
   The kDAPL 1.1 spec is now contained in the doc/ subdirectory.
=20
 * Several files have been moved around as part of the kDAPL
   checkin. Some files that were previously in udapl/ are now
   in common/, some in common are now in udapl/. The goal was
   to make sure files are properly located and make sense for
   the build.
=20
 * Source code formatting changes for consistency.
=20
 * Bug fixes
   - dapl_evd_create() was comparing the wrong bit combinations,
     allowing bogus EVDs to be created.
   - Removed code that swallowed zero length I/O requests, which
     are allowed by the spec and are useful to applications.
   - Locking in dapli_get_sp_ep was asymmetric; fixed it so the
     routine will take and release the lock. Cosmetic change.
   - dapl_get_consuemr_context() will now verify the pointer
     argument 'context' is not NULL.
=20
=20
=20
 NEW SINCE Beta 1.09
=20
 * We now take advantage of cm_disconnect(abrupt), which has the
   semantics to disconnect the connection but will not generate
   a callback. DAPL does generate a disconnect event for the app,
   so there is some care needed. This also introduces some
   interesting race conditions for the number of disconnect
   scenarios, e.g. remote disconnect results in asynchronous
   local disconnect callback; disconnect(graceful) followed by
   disconnect(abrupt) may still get a callback, etc. The code
   now properly deals with these race conditions.
=20
 * Increased the number of RDMA_READ credits (ibapi only) to allow
   better performance for overlapped I/O.
=20
 * Bug fixes
   - Typos and formatting changes to clean up and help make the code
     base more consistent. Added a couple new debug statements.
   - CR records now removed from the PSP/RSP when doing a
     close(abrupt)
   - Added casts for 64 bit values to prevent inadvertent sign
     propagation.
   - Do a better job of cleaning up EP fields if a cr_accept fails,
     which prevents false information from being presented by the
     next ep_query.
   - Fixed ia_close_abrupt to check for empty queues before trying
     to dequeue, which was causing an assert.
   - Changed the ACK_TIMEOUT value to be within the range specified
     by the IBTA spec. Noticed various DTO errors when going through
     a switch with high traffic, traced down to insufficient ACK
     timeout value. ibapi only.
       =20
=20
 NEW SINCE Beta 1.08
=20
 * Support for DAT RPM is now complete.
=20
 * Minor documentation updates
=20
 * Fixed dat_error.h to have a DAT_NO_SUBTYPE as the first (zeroth)
   element of the subtype enum. If there is no subtype, dat_strerr
   should not report something bogus.
=20
 * dapltest fixes:
   - fixed FFT memory test
   - fft_queryinfo now uses correct set of DAT enums for query
     ops
   - Cleanup of fft initializers
   - Thread and EP synchronization has been revamped and should be
     correct. Two major problems here:
     1) All EP's used the same connection EVD, making it very likely
        that a specific EP would lose a race and harvest the
        wrong event.
     2) Threads were not synchronized on start/stop, so they could
        easily miss connections or get rejected when they were
        confused.
=20
 * Modest number of simple cleanups
=20
 * Bug fixes
   - Several SMP synchronization fixes:
     a) RSP/PSP locking and synchronization
     b) ep_free and CM thread synchronization
     c) disconnect_clean called with locks held
     d) EP locking corrected for disconnects
     e) connection timer setup/teardown
   - Added support to dapl_ia_close() force a QP into the ERROR=20
     state in order to force posted ops to flush; used by
     disconnect ABRUPT at present.
   - Fixed missing data structure when calling the provider to
     modify the QP.
   - Fixed bug where we copied the IPv6 address twice when querying
     the QP.
   - Fixed ep_disconnect to be a no-op if the EP is disconnected.
   - EVD's are now initialized to enabled state per the spec
   - Corrected notion that UNSIGNALLED means the same thing in
     IB and DAPL, which is not the case.
=20
=20
=20
=20
=20
 NEW SINCE Beta 1.07
 * Red Hat Package Management (RPM) specification file and
   makefile updates to allow manual RPM creation of dat library.
   Automated support for creating a DAT RPM will come in a future
   release.
=20
 * Bug fixes
   - cr_query will return correct status and the remote IP
     address of the connecting node.
   - 'threshold' value in evd_wait is tested against the queue
     length and rejected if too large.
   - Fixed synchronization between RSP/PSP free and the CM
     callback thread.
   - Fixed synchronization between EP free and the CM callback=20
     thread.
   - Spec compliance for ep_disconnect() to be a no-op if the
     EP is DISCONNECTED.
  =20
=20
 NEW SINCE Beta 1.06
 * All DAPL ALPHA release notes removed from this file.
=20
 * For IBM Access providers, ib_types.h has been updated such
   that max_block_bytes is now 64 bits (instead of 32) to allow
   large memories. THIS CHANGE IMPLIES ia_query IS
    INCOMPATIBLE WITH OLD VERSIONS.
=20
 * local and remote address formats reworked to present as
   AF_INET (IPv4) addresses if possible.
=20
 * dapltest now displays local HCA IP address as part of config
   info (use -d to see it).
=20
 * Removed compile time references to PSC, no longer in who is
   business.=20
=20
 * dapltest changed to use service_id instead of pid when
   printing debug messages. Now both sides of the wire print
   the same value for connected threads.
=20
   Also added simple test for private_data on connections; errors
   are non fatal, but will cause a warning message to be printed.
=20
 * Changed the dat headers to allow MAJOR, MINOR, and THREADSAFE
   #defines to be in config files; per DAT Collaborative
   revision. Reference implementation default is now
   DAT_THREADSAFE=3DFALSE.
=20
 * Cleanded up various debug messages that occasionally spew
   out.=20
=20
 * Deleted unused fields in dapl structures, & bogus references
   to them in code.
=20
 * VAPI files updated and building cleanly again.
=20
 * Framework for SOLICITED_WAIT support now in place. Still not
   fully implemented.
=20

 * Conformance test updates:
   - depends_on results field more consistent & correct
   - Send  & recv DTO tests added
   - Updated RDMA tests
   - Enhanced recv tests
   - Connection management tests added
   - CNO tests added
  =20
=20
 * Bug fixes
   - Provider allocated EP cleaned up on cr_reject
   - dapl_ia_open fixed to prevent memory leaks on error paths
   - Using a reject reason of CONSUMER_REJECT (defined by
     IBTA), we can get a CM callback type of
     IB_CME_DESTINATION_REJECT_PRIVATE_DATA, even if there is no
     private data; all other reject types result in a callback
     of IB_CME_DESTINATION_REJECT. This allows us to generate
     the correct events on the connection EVD and to distinguish
     app level rejects vs. system level rejects or no listener
     present.
   - dapl_hca structure provides storage for local EP ip
     addresses.=20
   - In dapl_ep_create.c, cleaned up request_evd error checking
     to return correct error values.=20
   - Fixed bug when providing an EP for a CR, link it onto the
     IA.
   - Various small changes to keep compilers happy.
   - Fixed race condition between a CM thread and an app thread
     when releasing an SP object.
=20

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented.=20
=20
 * ib_enum_hca_if() moved below the IB abstraction layer.
=20
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
=20
 * Dynamic library loading support added for Windows
=20
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
=20
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
=20
=20
=20
 NEW SINCE Beta 1.04
=20
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
=20
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
=20
 * DAT registry is now fully 1.1 compliant.
=20
 * evd_stream_merging_supported enabled and used.
=20
 * DAT headers updated for DAT errata 99 and 100.
=20
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
=20
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
   =20
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making=20
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
=20

=20
 NEW SINCE Beta 1.03
=20
 * strerror() implemented
=20
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
=20
 * dat.conf file example updated for 1.1 spec
=20
 * Now obtain firmware information from the hca_attrib structure
   for ibapi.=20
=20
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
=20
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
=20
 * DTO/RMR completion status returns now comply with 1.1 spec.
=20
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
=20
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
=20
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
=20
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
=20

 NEW SINCE Beta 1.02
=20
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
=20
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
=20
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
=20
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added=20
=20
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to=20
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
=20

 NEW SINCE Beta 1.01
=20
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
=20
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
=20
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
=20
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!).=20
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
=20

 NEW SINCE Beta 1.0
=20
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming.=20
=20
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
=20
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
=20
 * Memory tests and echo tests updated to 1.1
=20
 * Assertion for 'over decremented' reference counts
=20
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups;=20
=20
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h.=20
=20
 * Ignore 0 length IOVs (1.1 compliance)
=20
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
=20
 * memory test now has ability to request invalid memry regions
   for negative testing.
=20
 * Enhanced error reporting (more subtypes added to returns)
=20
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
=20
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take=20
     corrective action.
   - ep_create will verify ep_attributes=20
=20
 * rmr_context now correctly returned from lmr_create().
    =20
=20

 OBTAIN THE CODE
=20
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from=20
 Source Forge (with no password) is:
=20
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.
=20
 When prompted for a password, simply  press the Enter key.
=20
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
=20
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
=20

 SYSTEM REQUIREMENTS
=20
 This project has been implemented on Red Hat Linux 7.3, and SuSE
 SLES 8. The structure of the code is designed to allow other
 operating systems to easily be adapted, but no work has been
 done in that direction.
=20
 The DAPL team has used Mellanox Tavor based InfiniBand HCAs for
 development, and continue with this platform. Our HCAs use the
 IB verbs API submitted by IBM. Mellanox has contributed an
 adapter layer using their VAPI verbs API. Either platform is
 available to any group considering DAPL work. The structure of
 the uDAPL source allows other provider API sets to be easily
 integrated.
=20
 The development team uses any one of four topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines; and most
 commonly, a switch.
=20
 The DAPL Plugfest revealed that switches and HCAs available from
 most vendors will interoperate with little trouble, given the
 most recent releases of software. The dapl reference team makes
 no recommendation on HCA or switch vendors.
=20
 Explicit machine configurations are available upon request.=20
=20

 IN THE TREE
=20
 The DAPL tree contains source code for the uDAPL and kDAPL
 implementations, and also includes tests and documentation.
=20
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
=20
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
=20
 Recently, the dapl conformance test has been added to the source
 repository. The test provides coverage of the most common
 interfaces, doing both positive and negative testing. Vendors
 providing DAPL implementation are strongly encouraged to run
 this set of tests.
=20

 MAKEFILE NOTES
=20
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
=20
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
=20
 Of particular relevance are the following #defines:
=20
 - CM_BUSTED
=20
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
=20
 - NO_NAME_SERVICE
=20
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
=20

 CONTRIBUTIONS
=20
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at sj...@ne... for review. We welcome your
 contributions and expect the quality of the project will
 improve thanks to your help.
=20
 The core DAPL team is:
=20
   Steve Sears
   Randy Smith
=20
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
=20

 ONGOING WORK
=20
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
=20

The DAPL Team

[Dapl-announce] uDAPL Beta 1.10 now available

[Sears, S. <Ste...@ne...>]

     Release Notes for=20
  Beta 1.10 uDAPL Release
     October 9, 2003
=20

 SPECIAL NOTE ON BETA 1.0
=20
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable.=20
=20
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
=20
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
=20
 RELEASE NOTES
=20
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
=20
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
=20
   http://sourceforge.net/projects/dapl
=20
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
=20
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
=20
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
=20

 NEW SINCE Beta 1.09
=20
 * We now take advantage of cm_disconnect(abrupt), which has the
   semantics to disconnect the connection but will not generate
   a callback. DAPL does generate a disconnect event for the app,
   so there is some care needed. This also introduces some
   interesting race conditions for the number of disconnect
   scenarios, e.g. remote disconnect results in asynchronous
   local disconnect callback; disconnect(graceful) followed by
   disconnect(abrupt) may still get a callback, etc. The code
   now properly deals with these race conditions.
=20
 * Increased the number of RDMA_READ credits (ibapi only) to allow
   better performance for overlapped I/O.
=20
 * Bug fixes
   - Typos and formatting changes to clean up and help make the code
     base more consistant. Added a couple new debug statements.
   - CR records now removed from the PSP/RSP when doing a
     close(abrupt)
   - Added casts for 64 bit values to prevent inadvertant sign
     propogation.
   - Do a better job of cleaning up EP fields if a cr_accept fails,
     which prevents false information from being presented by the
     next ep_query.
   - Fixed ia_close_abrupt to check for empty queues before trying
     to dequeue, which was causing an assert.
   - Changed the ACK_TIMEOUT value to be within the range specified
     by the IBTA spec. Noticed various DTO errors when going through
     a switch with high traffic, traced down to insufficient ACK
     timeout value. ibapi only.
=20

 NEW SINCE Beta 1.08
=20
 * Support for DAT RPM is now complete.
=20
 * Minor documentation updates
=20
 * Fixed dat_error.h to have a DAT_NO_SUBTYPE as the first (zeroth)
   element of the subtype enum. If there is no subtype, dat_strerr
   should not report something bogus.
=20
 * dapltest fixes:
   - fixed FFT memory test
   - fft_queryinfo now uses correct set of DAT enums for query
     ops
   - Cleanup of fft initializers
   - Thread and EP synchronization has been revamped and should be
     correct. Two major problems here:
     1) All EP's used the same connection EVD, making it very likely
        that a specific EP would lose a race and harvest the
        wrong event.
     2) Threads were not synchronized on start/stop, so they could
        easily miss connections or get rejected when they were
        confused.
=20
 * Modest number of simple cleanups
=20
 * Bug fixes
   - Several SMP synchronization fixes:
     a) RSP/PSP locking and synchronization
     b) ep_free and CM thread synchronization
     c) disconnect_clean called with locks held
     d) EP locking corrected for disconnects
     e) connection timer setup/teardown
   - Added support to dapl_ia_close() force a QP into the ERROR=20
     state in order to force posted ops to flush; used by
     disconnect ABRUPT at present.
   - Fixed missing data structure when calling the provider to
     modify the QP.
   - Fixed bug where we copied the IPv6 address twice when querying
     the QP.
   - Fixed ep_disconnect to be a no-op if the EP is disconnected.
   - EVD's are now initialized to enabled state per the spec
   - Corrected notion that UNSIGNALLED means the same thing in
     IB and DAPL, which is not the case.
=20
=20
=20
=20
=20
 NEW SINCE Beta 1.07
 * Red Hat Package Management (RPM) specification file and
   makefile updates to allow manual RPM creation of dat library.
   Automated support for creating a DAT RPM will come in a future
   release.
=20
 * Bug fixes
   - cr_query will return correct status and the remote IP
     address of the connecting node.
   - 'threshold' value in evd_wait is tested against the queue
     length and rejected if too large.
   - Fixed synchronization between RSP/PSP free and the CM
     callback thread.
   - Fixed synchronization between EP free and the CM callback=20
     thread.
   - Spec compliance for ep_disconnect() to be a no-op if the
     EP is DISCONNECTED.
  =20
=20
 NEW SINCE Beta 1.06
 * All DAPL ALPHA release notes removed from this file.
=20
 * For IBM Access providers, ib_types.h has been updated such
   that max_block_bytes is now 64 bits (instead of 32) to allow
   large memories. THIS CHANGE IMPLIES ia_query IS
    INCOMPATIBLE WITH OLD VERSIONS.
=20
 * local and remote address formats reworked to present as
   AF_INET (IPv4) addresses if possible.
=20
 * dapltest now displays local HCA IP address as part of config
   info (use -d to see it).
=20
 * Removed compile time references to PSC, no longer in who is
   business.=20
=20
 * dapltest changed to use service_id instead of pid when
   printing debug messages. Now both sides of the wire print
   the same value for connected threads.
=20
   Also added simple test for private_data on connections; errors
   are non fatal, but will cause a warning message to be printed.
=20
 * Changed the dat headers to allow MAJOR, MINOR, and THREADSAFE
   #defines to be in config files; per DAT Collaborative
   revision. Reference implementation default is now
   DAT_THREADSAFE=3DFALSE.
=20
 * Cleanded up various debug messages that occasionally spew
   out.=20
=20
 * Deleted unused fields in dapl structures, & bogus references
   to them in code.
=20
 * VAPI files updated and building cleanly again.
=20
 * Framework for SOLICITED_WAIT support now in place. Still not
   fully implemented.
=20

 * Conformance test updates:
   - depends_on results field more consistent & correct
   - Send  & recv DTO tests added
   - Updated RDMA tests
   - Enhanced recv tests
   - Connection management tests added
   - CNO tests added
  =20
=20
 * Bug fixes
   - Provider allocated EP cleaned up on cr_reject
   - dapl_ia_open fixed to prevent memory leaks on error paths
   - Using a reject reason of CONSUMER_REJECT (defined by
     IBTA), we can get a CM callback type of
     IB_CME_DESTINATION_REJECT_PRIVATE_DATA, even if there is no
     private data; all other reject types result in a callback
     of IB_CME_DESTINATION_REJECT. This allows us to generate
     the correct events on the connection EVD and to distinguish
     app level rejects vs. system level rejects or no listener
     present.
   - dapl_hca structure provides storage for local EP ip
     addresses.=20
   - In dapl_ep_create.c, cleaned up request_evd error checking
     to return correct error values.=20
   - Fixed bug when providing an EP for a CR, link it onto the
     IA.
   - Various small changes to keep compilers happy.
   - Fixed race condition between a CM thread and an app thread
     when releasing an SP object.
=20

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented.=20
=20
 * ib_enum_hca_if() moved below the IB abstraction layer.
=20
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
=20
 * Dynamic library loading support added for Windows
=20
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
=20
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
=20
=20
=20
 NEW SINCE Beta 1.04
=20
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
=20
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
=20
 * DAT registry is now fully 1.1 compliant.
=20
 * evd_stream_merging_supported enabled and used.
=20
 * DAT headers updated for DAT errata 99 and 100.
=20
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
=20
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
   =20
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making=20
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
=20

=20
 NEW SINCE Beta 1.03
=20
 * strerror() implemented
=20
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
=20
 * dat.conf file example updated for 1.1 spec
=20
 * Now obtain firmware information from the hca_attrib structure
   for ibapi.=20
=20
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
=20
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
=20
 * DTO/RMR completion status returns now comply with 1.1 spec.
=20
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
=20
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
=20
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
=20
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
=20

 NEW SINCE Beta 1.02
=20
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
=20
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
=20
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
=20
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added=20
=20
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to=20
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
=20

 NEW SINCE Beta 1.01
=20
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
=20
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
=20
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
=20
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!).=20
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
=20

 NEW SINCE Beta 1.0
=20
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming.=20
=20
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
=20
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
=20
 * Memory tests and echo tests updated to 1.1
=20
 * Assertion for 'over decremented' reference counts
=20
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups;=20
=20
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h.=20
=20
 * Ignore 0 length IOVs (1.1 compliance)
=20
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
=20
 * memory test now has ability to request invalid memry regions
   for negative testing.
=20
 * Enhanced error reporting (more subtypes added to returns)
=20
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
=20
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take=20
     corrective action.
   - ep_create will verify ep_attributes=20
=20
 * rmr_context now correctly returned from lmr_create().
    =20
=20

 OBTAIN THE CODE
=20
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from=20
 Source Forge (with no password) is:
=20
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.
=20
 When prompted for a password, simply  press the Enter key.
=20
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
=20
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
=20

 SYSTEM REQUIREMENTS
=20
 This project has been implemented on Red Hat Linux 7.2. The
 structure of the code is designed to allow other operating
 systems to easily be adapted, but no work has been done in that
 direction.
=20
 The DAPL team has used IBM based InfiniBand HCAs for development,
 and continue with this platform. Concurrently, Mellanox has
 ported this work to their silicon and are successfully running
 DAPL as well. Either platform is available to any group
 considering DAPL work. The structure of the uDAPL source=20
 allows other provider API sets to be easily integrated.
=20
 The development team uses any one of three topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines. We have not
 used a switch as of this writing. All connections are assumed
 to be point-to-point (P2P) using the same hardware and software
 on both sides of the 4X wire. Using switches and multiple
 source/destinations is under development and will be available
 soon.
=20
 Explicit machine configurations are available upon request.=20
=20

 IN THE TREE
=20
 The DAPL tree contains source code for the uDAPL implementation,
 and also includes tests and documentation.=20
=20
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
=20
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
=20

 MAKEFILE NOTES
=20
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
=20
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
=20
 Of particular relevance are the following #defines:
=20
 - CM_BUSTED
=20
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
=20
 - NO_NAME_SERVICE
=20
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
=20

 CONTRIBUTIONS
=20
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at sj...@ne... for review. We welcome your
 contributions and expect the quality of the project will
 improve thanks to your help.
=20
 The core DAPL team is:
=20
   James Lentini
   Steve Sears
   Randy Smith
=20
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
=20

 ONGOING WORK
=20
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
=20
 We expect work on a kDAPL implementation will begin soon,
 we welcome those who are interested in contributing to this
 project.
=20

The DAPL Team

[Dapl-announce] uDAPL Beta 1.09 now available

[Sears, S. <Ste...@ne...>]

     Release Notes for=20
  Beta 1.09 uDAPL Release
     September 24, 2003
=20

 SPECIAL NOTE ON BETA 1.0
=20
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable.=20
=20
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
=20
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
=20
 RELEASE NOTES
=20
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
=20
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
=20
   http://sourceforge.net/projects/dapl
=20
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
=20
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
=20
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
=20

 NEW SINCE Beta 1.08
=20
 * Support for DAT RPM is now complete.
=20
 * Minor documentation updates
=20
 * Fixed dat_error.h to have a DAT_NO_SUBTYPE as the first (zero-th)
   element of the subtype enum. If there is no subtype, dat_strerr
   should not report something bogus.
=20
 * dapltest fixes:
   - fixed FFT memory test
   - fft_queryinfo now uses correct set of DAT enums for query
     ops
   - Cleanup of fft initializers
   - Thread and EP synchronization has been revamped and should be
     correct. Two major problems here:
     1) All EP's used the same connection EVD, making it very likely
        that a specific EP would lose a race and harvest the
        wrong event.
     2) Threads were not synchronized on start/stop, so they could
        easily miss connections or get rejected when they were
        confused.
=20
 * Modest number of simple cleanups
=20
 * Bug fixes
   - Several SMP synchronization fixes:
     a) RSP/PSP locking and synchronization
     b) ep_free and CM thread synchronization
     c) disconnect_clean called with locks held
     d) EP locking corrected for disconnects
     e) connection timer setup/teardown
   - Added support to dapl_ia_close() force a QP into the ERROR=20
     state in order to force posted ops to flush; used by
     disconnect ABRUPT at present.
   - Fixed missing data structure when calling the provider to
     modify the QP.
   - Fixed bug where we copied the IPv6 address twice when querying
     the QP.
   - Fixed ep_disconnect to be a no-op if the EP is disconnected.
   - EVD's are now initialized to enabled state per the spec
   - Corrected notion that UNSIGNALLED means the same thing in
     IB and DAPL, which is not the case.
=20
=20
=20
=20
=20
 NEW SINCE Beta 1.07
 * Red Hat Package Management (RPM) specification file and
   makefile updates to allow manual RPM creation of dat library.
   Automated support for creating a DAT RPM will come in a future
   release.
=20
 * Bug fixes
   - cr_query will return correct status and the remote IP
     address of the connecting node.
   - 'threshold' value in evd_wait is tested against the queue
     length and rejected if too large.
   - Fixed synchronization between RSP/PSP free and the CM
     callback thread.
   - Fixed synchronization between EP free and the CM callback=20
     thread.
   - Spec compliance for ep_disconnect() to be a no-op if the
     EP is DISCONNECTED.
  =20
=20
 NEW SINCE Beta 1.06
 * All DAPL ALPHA release notes removed from this file.
=20
 * For IBM Access providers, ib_types.h has been updated such
   that max_block_bytes is now 64 bits (instead of 32) to allow
   large memories. THIS CHANGE IMPLIES ia_query IS
    INCOMPATIBLE WITH OLD VERSIONS.
=20
 * local and remote address formats reworked to present as
   AF_INET (IPv4) addresses if possible.
=20
 * dapltest now displays local HCA IP address as part of config
   info (use -d to see it).
=20
 * Removed compile time references to PSC, no longer in who is
   business.=20
=20
 * dapltest changed to use service_id instead of pid when
   printing debug messages. Now both sides of the wire print
   the same value for connected threads.
=20
   Also added simple test for private_data on connections; errors
   are non fatal, but will cause a warning message to be printed.
=20
 * Changed the dat headers to allow MAJOR, MINOR, and THREADSAFE
   #defines to be in config files; per DAT Collaborative
   revision. Reference implementation default is now
   DAT_THREADSAFE=3DFALSE.
=20
 * Cleanded up various debug messages that occasionally spew
   out.=20
=20
 * Deleted unused fields in dapl structures, & bogus references
   to them in code.
=20
 * VAPI files updated and building cleanly again.
=20
 * Framework for SOLICITED_WAIT support now in place. Still not
   fully implemented.
=20

 * Conformance test updates:
   - depends_on results field more consistent & correct
   - Send  & recv DTO tests added
   - Updated RDMA tests
   - Enhanced recv tests
   - Connection management tests added
   - CNO tests added
  =20
=20
 * Bug fixes
   - Provider allocated EP cleaned up on cr_reject
   - dapl_ia_open fixed to prevent memory leaks on error paths
   - Using a reject reason of CONSUMER_REJECT (defined by
     IBTA), we can get a CM callback type of
     IB_CME_DESTINATION_REJECT_PRIVATE_DATA, even if there is no
     private data; all other reject types result in a callback
     of IB_CME_DESTINATION_REJECT. This allows us to generate
     the correct events on the connection EVD and to distinguish
     app level rejects vs. system level rejects or no listener
     present.
   - dapl_hca structure provides storage for local EP ip
     addresses.=20
   - In dapl_ep_create.c, cleaned up request_evd error checking
     to return correct error values.=20
   - Fixed bug when providing an EP for a CR, link it onto the
     IA.
   - Various small changes to keep compilers happy.
   - Fixed race condition between a CM thread and an app thread
     when releasing an SP object.
=20

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented.=20
=20
 * ib_enum_hca_if() moved below the IB abstraction layer.
=20
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
=20
 * Dynamic library loading support added for Windows
=20
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
=20
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
=20
=20
=20
 NEW SINCE Beta 1.04
=20
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
=20
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
=20
 * DAT registry is now fully 1.1 compliant.
=20
 * evd_stream_merging_supported enabled and used.
=20
 * DAT headers updated for DAT errata 99 and 100.
=20
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
=20
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
   =20
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making=20
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
=20

=20
 NEW SINCE Beta 1.03
=20
 * strerror() implemented
=20
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
=20
 * dat.conf file example updated for 1.1 spec
=20
 * Now obtain firmware information from the hca_attrib structure
   for ibapi.=20
=20
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
=20
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
=20
 * DTO/RMR completion status returns now comply with 1.1 spec.
=20
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
=20
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
=20
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
=20
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
=20

 NEW SINCE Beta 1.02
=20
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
=20
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
=20
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
=20
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added=20
=20
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to=20
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
=20

 NEW SINCE Beta 1.01
=20
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
=20
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
=20
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
=20
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!).=20
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
=20

 NEW SINCE Beta 1.0
=20
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming.=20
=20
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
=20
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
=20
 * Memory tests and echo tests updated to 1.1
=20
 * Assertion for 'over decremented' reference counts
=20
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups;=20
=20
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h.=20
=20
 * Ignore 0 length IOVs (1.1 compliance)
=20
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
=20
 * memory test now has ability to request invalid memry regions
   for negative testing.
=20
 * Enhanced error reporting (more subtypes added to returns)
=20
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
=20
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take=20
     corrective action.
   - ep_create will verify ep_attributes=20
=20
 * rmr_context now correctly returned from lmr_create().
    =20
=20

 OBTAIN THE CODE
=20
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from=20
 Source Forge (with no password) is:
=20
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.
=20
 When prompted for a password, simply  press the Enter key.
=20
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
=20
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
=20

 SYSTEM REQUIREMENTS
=20
 This project has been implemented on Red Hat Linux 7.2. The
 structure of the code is designed to allow other operating
 systems to easily be adapted, but no work has been done in that
 direction.
=20
 The DAPL team has used IBM based InfiniBand HCAs for development,
 and continue with this platform. Concurrently, Mellanox has
 ported this work to their silicon and are successfully running
 DAPL as well. Either platform is available to any group
 considering DAPL work. The structure of the uDAPL source=20
 allows other provider API sets to be easily integrated.
=20
 The development team uses any one of three topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines. We have not
 used a switch as of this writing. All connections are assumed
 to be point-to-point (P2P) using the same hardware and software
 on both sides of the 4X wire. Using switches and multiple
 source/destinations is under development and will be available
 soon.
=20
 Explicit machine configurations are available upon request.=20
=20

 IN THE TREE
=20
 The DAPL tree contains source code for the uDAPL implementation,
 and also includes tests and documentation.=20
=20
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
=20
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
=20

 MAKEFILE NOTES
=20
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
=20
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
=20
 Of particular relevance are the following #defines:
=20
 - CM_BUSTED
=20
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
=20
 - NO_NAME_SERVICE
=20
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
=20

 CONTRIBUTIONS
=20
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at sj...@ne... for review. We welcome your
 contributions and expect the quality of the project will
 improve thanks to your help.
=20
 The core DAPL team is:
=20
   James Lentini
   Steve Sears
   Randy Smith
=20
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
=20

 ONGOING WORK
=20
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
=20
 We expect work on a kDAPL implementation will begin soon,
 we welcome those who are interested in contributing to this
 project.
=20

The DAPL Team

RE: [Dapl-announce] uDAPL Beta 1.08 now available

[Sears, S. <Ste...@ne...>]

There was a problem in the tar.gz file that was originally posted. If =
you downloaded before 3:10 PM EDT, please download again.
=20
Sorry for this inconvenience.
=20
=20
 -Steve
=20

-----Original Message-----
From: Sears, Steven=20
Sent: Wednesday, September 10, 2003 2:45 PM
To: dap...@li...
Subject: [Dapl-announce] uDAPL Beta 1.08 now available


     Release Notes for=20
  Beta 1.08 uDAPL Release
     September 10, 2003
=20
=20
=20
 SPECIAL NOTE ON BETA 1.0
=20
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable.=20
=20
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
=20
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
=20
 RELEASE NOTES
=20
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
=20
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
=20
   http://sourceforge.net/projects/dapl
=20
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
=20
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
=20
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
=20


 NEW SINCE Beta 1.07
 * Red Hat Package Management (RPM) specification file and
   makefile updates to allow manual RPM creation of dat library.
   Automated support for creating a DAT RPM will come in a future
   release.
=20
 * Bug fixes
   - cr_query will return correct status and the remote IP
     address of the connecting node.
   - 'threshold' value in evd_wait is tested against the queue
     length and rejected if too large.
   - Fixed synchronization between RSP/PSP free and the CM
     callback thread.
   - Fixed synchronization between EP free and the CM callback=20
     thread.
   - Spec compliance for ep_disconnect() to be a no-op if the
     EP is DISCONNECTED.
  =20
=20
 NEW SINCE Beta 1.06
 * All DAPL ALPHA release notes removed from this file.
=20
 * For IBM Access providers, ib_types.h has been updated such
   that max_block_bytes is now 64 bits (instead of 32) to allow
   large memories. THIS CHANGE IMPLIES ia_query IS
    INCOMPATIBLE WITH OLD VERSIONS.
=20
 * local and remote address formats reworked to present as
   AF_INET (IPv4) addresses if possible.
=20
 * dapltest now displays local HCA IP address as part of config
   info (use -d to see it).
=20
 * Removed compile time references to PSC, no longer in who is
   business.=20
=20
 * dapltest changed to use service_id instead of pid when
   printing debug messages. Now both sides of the wire print
   the same value for connected threads.
=20
   Also added simple test for private_data on connections; errors
   are non fatal, but will cause a warning message to be printed.
=20
 * Changed the dat headers to allow MAJOR, MINOR, and THREADSAFE
   #defines to be in config files; per DAT Collaborative
   revision. Reference implementation default is now
   DAT_THREADSAFE=3DFALSE.
=20
 * Cleanded up various debug messages that occasionally spew
   out.=20
=20
 * Deleted unused fields in dapl structures, & bogus references
   to them in code.
=20
 * VAPI files updated and building cleanly again.
=20
 * Framework for SOLICITED_WAIT support now in place. Still not
   fully implemented.
=20

 * Conformance test updates:
   - depends_on results field more consistent & correct
   - Send  & recv DTO tests added
   - Updated RDMA tests
   - Enhanced recv tests
   - Connection management tests added
   - CNO tests added
  =20
=20
 * Bug fixes
   - Provider allocated EP cleaned up on cr_reject
   - dapl_ia_open fixed to prevent memory leaks on error paths
   - Using a reject reason of CONSUMER_REJECT (defined by
     IBTA), we can get a CM callback type of
     IB_CME_DESTINATION_REJECT_PRIVATE_DATA, even if there is no
     private data; all other reject types result in a callback
     of IB_CME_DESTINATION_REJECT. This allows us to generate
     the correct events on the connection EVD and to distinguish
     app level rejects vs. system level rejects or no listener
     present.
   - dapl_hca structure provides storage for local EP ip
     addresses.=20
   - In dapl_ep_create.c, cleaned up request_evd error checking
     to return correct error values.=20
   - Fixed bug when providing an EP for a CR, link it onto the
     IA.
   - Various small changes to keep compilers happy.
   - Fixed race condition between a CM thread and an app thread
     when releasing an SP object.
=20

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented.=20
=20
 * ib_enum_hca_if() moved below the IB abstraction layer.
=20
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
=20
 * Dynamic library loading support added for Windows
=20
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
=20
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
=20
=20
=20
 NEW SINCE Beta 1.04
=20
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
=20
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
=20
 * DAT registry is now fully 1.1 compliant.
=20
 * evd_stream_merging_supported enabled and used.
=20
 * DAT headers updated for DAT errata 99 and 100.
=20
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
=20
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
   =20
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making=20
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
=20

=20
 NEW SINCE Beta 1.03
=20
 * strerror() implemented
=20
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
=20
 * dat.conf file example updated for 1.1 spec
=20
 * Now obtain firmware information from the hca_attrib structure
   for ibapi.=20
=20
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
=20
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
=20
 * DTO/RMR completion status returns now comply with 1.1 spec.
=20
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
=20
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
=20
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
=20
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
=20

 NEW SINCE Beta 1.02
=20
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
=20
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
=20
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
=20
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added=20
=20
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to=20
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
=20

 NEW SINCE Beta 1.01
=20
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
=20
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
=20
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
=20
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!).=20
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
=20

 NEW SINCE Beta 1.0
=20
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming.=20
=20
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
=20
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
=20
 * Memory tests and echo tests updated to 1.1
=20
 * Assertion for 'over decremented' reference counts
=20
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups;=20
=20
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h.=20
=20
 * Ignore 0 length IOVs (1.1 compliance)
=20
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
=20
 * memory test now has ability to request invalid memry regions
   for negative testing.
=20
 * Enhanced error reporting (more subtypes added to returns)
=20
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
=20
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take=20
     corrective action.
   - ep_create will verify ep_attributes=20
=20
 * rmr_context now correctly returned from lmr_create().
    =20
=20

 OBTAIN THE CODE
=20
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from=20
 Source Forge (with no password) is:
=20
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.
=20
 When prompted for a password, simply  press the Enter key.
=20
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
=20
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
=20

 SYSTEM REQUIREMENTS
=20
 This project has been implemented on Red Hat Linux 7.2. The
 structure of the code is designed to allow other operating
 systems to easily be adapted, but no work has been done in that
 direction.
=20
 The DAPL team has used IBM based InfiniBand HCAs for development,
 and continue with this platform. Concurrently, Mellanox has
 ported this work to their silicon and are successfully running
 DAPL as well. Either platform is available to any group
 considering DAPL work. The structure of the uDAPL source=20
 allows other provider API sets to be easily integrated.
=20
 The development team uses any one of three topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines. We have not
 used a switch as of this writing. All connections are assumed
 to be point-to-point (P2P) using the same hardware and software
 on both sides of the 4X wire. Using switches and multiple
 source/destinations is under development and will be available
 soon.
=20
 Explicit machine configurations are available upon request.=20
=20

 IN THE TREE
=20
 The DAPL tree contains source code for the uDAPL implementation,
 and also includes tests and documentation.=20
=20
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
=20
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
=20

 MAKEFILE NOTES
=20
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
=20
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
=20
 Of particular relevance are the following #defines:
=20
 - CM_BUSTED
=20
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
=20
 - NO_NAME_SERVICE
=20
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
=20

 CONTRIBUTIONS
=20
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at sj...@ne... for review. We welcome your
 contributions and expect the quality of the project will
 improve thanks to your help.
=20
 The core DAPL team is:
=20
   James Lentini
   Steve Sears
   Randy Smith
=20
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
=20

 ONGOING WORK
=20
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
=20
 We expect work on a kDAPL implementation will begin soon,
 we welcome those who are interested in contributing to this
 project.
=20

The DAPL Team

=20

[Dapl-announce] uDAPL Beta 1.08 now available

[Sears, S. <Ste...@ne...>]

     Release Notes for=20
  Beta 1.08 uDAPL Release
     September 10, 2003
=20
=20
=20
 SPECIAL NOTE ON BETA 1.0
=20
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable.=20
=20
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
=20
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
=20
 RELEASE NOTES
=20
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
=20
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
=20
   http://sourceforge.net/projects/dapl
=20
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
=20
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
=20
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
=20

 NEW SINCE Beta 1.07
 * Red Hat Package Management (RPM) specification file and
   makefile updates to allow manual RPM creation of dat library.
   Automated support for creating a DAT RPM will come in a future
   release.
=20
 * Bug fixes
   - cr_query will return correct status and the remote IP
     address of the connecting node.
   - 'threshold' value in evd_wait is tested against the queue
     length and rejected if too large.
   - Fixed synchronization between RSP/PSP free and the CM
     callback thread.
   - Fixed synchronization between EP free and the CM callback=20
     thread.
   - Spec compliance for ep_disconnect() to be a no-op if the
     EP is DISCONNECTED.
  =20
=20
 NEW SINCE Beta 1.06
 * All DAPL ALPHA release notes removed from this file.
=20
 * For IBM Access providers, ib_types.h has been updated such
   that max_block_bytes is now 64 bits (instead of 32) to allow
   large memories. THIS CHANGE IMPLIES ia_query IS
    INCOMPATIBLE WITH OLD VERSIONS.
=20
 * local and remote address formats reworked to present as
   AF_INET (IPv4) addresses if possible.
=20
 * dapltest now displays local HCA IP address as part of config
   info (use -d to see it).
=20
 * Removed compile time references to PSC, no longer in who is
   business.=20
=20
 * dapltest changed to use service_id instead of pid when
   printing debug messages. Now both sides of the wire print
   the same value for connected threads.
=20
   Also added simple test for private_data on connections; errors
   are non fatal, but will cause a warning message to be printed.
=20
 * Changed the dat headers to allow MAJOR, MINOR, and THREADSAFE
   #defines to be in config files; per DAT Collaborative
   revision. Reference implementation default is now
   DAT_THREADSAFE=3DFALSE.
=20
 * Cleanded up various debug messages that occasionally spew
   out.=20
=20
 * Deleted unused fields in dapl structures, & bogus references
   to them in code.
=20
 * VAPI files updated and building cleanly again.
=20
 * Framework for SOLICITED_WAIT support now in place. Still not
   fully implemented.
=20

 * Conformance test updates:
   - depends_on results field more consistent & correct
   - Send  & recv DTO tests added
   - Updated RDMA tests
   - Enhanced recv tests
   - Connection management tests added
   - CNO tests added
  =20
=20
 * Bug fixes
   - Provider allocated EP cleaned up on cr_reject
   - dapl_ia_open fixed to prevent memory leaks on error paths
   - Using a reject reason of CONSUMER_REJECT (defined by
     IBTA), we can get a CM callback type of
     IB_CME_DESTINATION_REJECT_PRIVATE_DATA, even if there is no
     private data; all other reject types result in a callback
     of IB_CME_DESTINATION_REJECT. This allows us to generate
     the correct events on the connection EVD and to distinguish
     app level rejects vs. system level rejects or no listener
     present.
   - dapl_hca structure provides storage for local EP ip
     addresses.=20
   - In dapl_ep_create.c, cleaned up request_evd error checking
     to return correct error values.=20
   - Fixed bug when providing an EP for a CR, link it onto the
     IA.
   - Various small changes to keep compilers happy.
   - Fixed race condition between a CM thread and an app thread
     when releasing an SP object.
=20

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented.=20
=20
 * ib_enum_hca_if() moved below the IB abstraction layer.
=20
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
=20
 * Dynamic library loading support added for Windows
=20
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
=20
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
=20
=20
=20
 NEW SINCE Beta 1.04
=20
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
=20
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
=20
 * DAT registry is now fully 1.1 compliant.
=20
 * evd_stream_merging_supported enabled and used.
=20
 * DAT headers updated for DAT errata 99 and 100.
=20
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
=20
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
   =20
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making=20
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
=20

=20
 NEW SINCE Beta 1.03
=20
 * strerror() implemented
=20
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
=20
 * dat.conf file example updated for 1.1 spec
=20
 * Now obtain firmware information from the hca_attrib structure
   for ibapi.=20
=20
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
=20
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
=20
 * DTO/RMR completion status returns now comply with 1.1 spec.
=20
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
=20
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
=20
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
=20
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
=20

 NEW SINCE Beta 1.02
=20
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
=20
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
=20
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
=20
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added=20
=20
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to=20
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
=20

 NEW SINCE Beta 1.01
=20
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
=20
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
=20
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
=20
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!).=20
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
=20

 NEW SINCE Beta 1.0
=20
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming.=20
=20
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
=20
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
=20
 * Memory tests and echo tests updated to 1.1
=20
 * Assertion for 'over decremented' reference counts
=20
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups;=20
=20
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h.=20
=20
 * Ignore 0 length IOVs (1.1 compliance)
=20
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
=20
 * memory test now has ability to request invalid memry regions
   for negative testing.
=20
 * Enhanced error reporting (more subtypes added to returns)
=20
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
=20
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take=20
     corrective action.
   - ep_create will verify ep_attributes=20
=20
 * rmr_context now correctly returned from lmr_create().
    =20
=20

 OBTAIN THE CODE
=20
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from=20
 Source Forge (with no password) is:
=20
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.
=20
 When prompted for a password, simply  press the Enter key.
=20
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
=20
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
=20

 SYSTEM REQUIREMENTS
=20
 This project has been implemented on Red Hat Linux 7.2. The
 structure of the code is designed to allow other operating
 systems to easily be adapted, but no work has been done in that
 direction.
=20
 The DAPL team has used IBM based InfiniBand HCAs for development,
 and continue with this platform. Concurrently, Mellanox has
 ported this work to their silicon and are successfully running
 DAPL as well. Either platform is available to any group
 considering DAPL work. The structure of the uDAPL source=20
 allows other provider API sets to be easily integrated.
=20
 The development team uses any one of three topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines. We have not
 used a switch as of this writing. All connections are assumed
 to be point-to-point (P2P) using the same hardware and software
 on both sides of the 4X wire. Using switches and multiple
 source/destinations is under development and will be available
 soon.
=20
 Explicit machine configurations are available upon request.=20
=20

 IN THE TREE
=20
 The DAPL tree contains source code for the uDAPL implementation,
 and also includes tests and documentation.=20
=20
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
=20
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
=20

 MAKEFILE NOTES
=20
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
=20
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
=20
 Of particular relevance are the following #defines:
=20
 - CM_BUSTED
=20
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
=20
 - NO_NAME_SERVICE
=20
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
=20

 CONTRIBUTIONS
=20
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at sj...@ne... for review. We welcome your
 contributions and expect the quality of the project will
 improve thanks to your help.
=20
 The core DAPL team is:
=20
   James Lentini
   Steve Sears
   Randy Smith
=20
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
=20

 ONGOING WORK
=20
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
=20
 We expect work on a kDAPL implementation will begin soon,
 we welcome those who are interested in contributing to this
 project.
=20

The DAPL Team

=20

[Dapl-announce] uDAPL Beta 1.07 now available

[Sears, S. <Ste...@ne...>]

     Release Notes for 
  Beta 1.07 uDAPL Release
     September 3, 2003
 
 
 
 SPECIAL NOTE ON BETA 1.0
 
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable. 
 
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
 
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
 
 RELEASE NOTES
 
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
 
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
 
   http://sourceforge.net/projects/dapl
<http://sourceforge.net/projects/dapl> 
 
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
 
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
 
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
 

 NEW SINCE Beta 1.06
 * All DAPL ALPHA release notes removed from this file.
 
 * For IBM Access providers, ib_types.h has been updated such
   that max_block_bytes is now 64 bits (instead of 32) to allow
   large memories. THIS CHANGE IMPLIES ia_query IS
    INCOMPATIBLE WITH OLD VERSIONS.
 
 * local and remote address formats reworked to present as
   AF_INET (IPv4) addresses if possible.
 
 * dapltest now displays local HCA IP address as part of config
   info (use -d to see it).
 
 * Removed compile time references to PSC, no longer in who is
   business. 
 
 * dapltest changed to use service_id instead of pid when
   printing debug messages. Now both sides of the wire print
   the same value for connected threads.
 
   Also added simple test for private_data on connections; errors
   are non fatal, but will cause a warning message to be printed.
 
 * Changed the dat headers to allow MAJOR, MINOR, and THREADSAFE
   #defines to be in config files; per DAT Collaborative
   revision. Reference implementation default is now
   DAT_THREADSAFE=FALSE.
 
 * Cleanded up various debug messages that occasionally spew
   out. 
 
 * Deleted unused fields in dapl structures, & bogus references
   to them in code.
 
 * VAPI files updated and building cleanly again.
 
 * Framework for SOLICITED_WAIT support now in place. Still not
   fully implemented.
 

 * Conformance test updates:
   - depends_on results field more consistent & correct
   - Send  & recv DTO tests added
   - Updated RDMA tests
   - Enhanced recv tests
   - Connection management tests added
   - CNO tests added
   
 
 * Bug fixes
   - Provider allocated EP cleaned up on cr_reject
   - dapl_ia_open fixed to prevent memory leaks on error paths
   - Using a reject reason of CONSUMER_REJECT (defined by
     IBTA), we can get a CM callback type of
     IB_CME_DESTINATION_REJECT_PRIVATE_DATA, even if there is no
     private data; all other reject types result in a callback
     of IB_CME_DESTINATION_REJECT. This allows us to generate
     the correct events on the connection EVD and to distinguish
     app level rejects vs. system level rejects or no listener
     present.
   - dapl_hca structure provides storage for local EP ip
     addresses. 
   - In dapl_ep_create.c, cleaned up request_evd error checking
     to return correct error values. 
   - Fixed bug when providing an EP for a CR, link it onto the
     IA.
   - Various small changes to keep compilers happy.
   - Fixed race condition between a CM thread and an app thread
     when releasing an SP object.
 

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented. 
 
 * ib_enum_hca_if() moved below the IB abstraction layer.
 
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
 
 * Dynamic library loading support added for Windows
 
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
 
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
 
 
 
 NEW SINCE Beta 1.04
 
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
 
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
 
 * DAT registry is now fully 1.1 compliant.
 
 * evd_stream_merging_supported enabled and used.
 
 * DAT headers updated for DAT errata 99 and 100.
 
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
 
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
    
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making 
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
 

 
 NEW SINCE Beta 1.03
 
 * strerror() implemented
 
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
 
 * dat.conf file example updated for 1.1 spec
 
 * Now obtain firmware information from the hca_attrib structure
   for ibapi. 
 
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
 
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
 
 * DTO/RMR completion status returns now comply with 1.1 spec.
 
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
 
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
 
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
 
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
 

 NEW SINCE Beta 1.02
 
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
 
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
 
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
 
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added 
 
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to 
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
 

 NEW SINCE Beta 1.01
 
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
 
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
 
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
 
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!). 
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
 

 NEW SINCE Beta 1.0
 
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming. 
 
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
 
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
 
 * Memory tests and echo tests updated to 1.1
 
 * Assertion for 'over decremented' reference counts
 
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups; 
 
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h. 
 
 * Ignore 0 length IOVs (1.1 compliance)
 
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
 
 * memory test now has ability to request invalid memry regions
   for negative testing.
 
 * Enhanced error reporting (more subtypes added to returns)
 
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
 
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take 
     corrective action.
   - ep_create will verify ep_attributes 
 
 * rmr_context now correctly returned from lmr_create().
     
 

 OBTAIN THE CODE
 
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from 
 Source Forge (with no password) is:
 
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co .
 
 When prompted for a password, simply  press the Enter key.
 
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
 
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
<http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/> 
 

 SYSTEM REQUIREMENTS
 
 This project has been implemented on Red Hat Linux 7.2. The
 structure of the code is designed to allow other operating
 systems to easily be adapted, but no work has been done in that
 direction.
 
 The DAPL team has used IBM based InfiniBand HCAs for development,
 and continue with this platform. Concurrently, Mellanox has
 ported this work to their silicon and are successfully running
 DAPL as well. Either platform is available to any group
 considering DAPL work. The structure of the uDAPL source 
 allows other provider API sets to be easily integrated.
 
 The development team uses any one of three topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines. We have not
 used a switch as of this writing. All connections are assumed
 to be point-to-point (P2P) using the same hardware and software
 on both sides of the 4X wire. Using switches and multiple
 source/destinations is under development and will be available
 soon.
 
 Explicit machine configurations are available upon request. 
 

 IN THE TREE
 
 The DAPL tree contains source code for the uDAPL implementation,
 and also includes tests and documentation. 
 
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
 
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
 

 MAKEFILE NOTES
 
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
 
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
 
 Of particular relevance are the following #defines:
 
 - CM_BUSTED
 
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
 
 - NO_NAME_SERVICE
 
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
 

 CONTRIBUTIONS
 
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at sj...@ne... <mailto:sj...@ne...>  for review. We welcome
your
 contributions and expect the quality of the project will
 improve thanks to your help.
 
 The core DAPL team is:
 
   James Lentini
   Steve Sears
   Randy Smith
 
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
 

 ONGOING WORK
 
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
 
 We expect work on a kDAPL implementation will begin soon,
 we welcome those who are interested in contributing to this
 project.
 

The DAPL Team

[Dapl-announce] uDAPL Beta 1.06 now available

[Sears, S. <Ste...@ne...>]

     Release Notes for=20
  Beta 1.06 uDAPL Release
     August 20, 2003
=20
=20
=20
 SPECIAL NOTE ON BETA 1.0
=20
 Perhaps it is hubris on the part of the DAPL team, but now that
 we are nearly DAT Spec 1.1 compliant and the code base is being
 used by many companies, we have decided to rev our release
 signature to Beta releases and discontinue Alpha releases. This
 is a stake in the ground both for spec compliance and the
 recognition that things are generally stable.=20
=20
 If you have DAPL applications, they are going to require changes
 in order to be compliant with this code base, so do not pick it
 up lightly: you have been warned!
=20
 In changing to the new level of the spec, some of the documents
 found in the ./doc directory may be out of date. We have made an
 effort to keep them current, but it is time for a wholesale
 review by the development team to see if they still reflect
 the implementation.
=20
 RELEASE NOTES
=20
 We would like to officially announce the availability of a
 public source implementation of uDAPL. This implementation
 has been developed by Network Appliance with significant
 contributions from JNI, IBM, Mellanox, and a number of
 companies who wish to remain anonymous at this time.
=20
 The uDAPL source code is available on Source Forge under the
 new DAPL foundry, see
=20
   http://sourceforge.net/projects/dapl
=20
 Both kDAPL and uDAPL exist in this source tree, although only
 a small amount of work has been done on kDAPL to date.
=20
 Source Forge is the world's largest Open Source development
 website and provides a number of services to developers and
 the Open Source community.
=20
 The DAPL source code is being provided free to all interested
 parties. The license provisions for this project is designed to
 encourage commercial development and products based upon this
 source code. There are no intellectual property claims attached
 to this code. See the project web page above for hyperlinks
 pointing to the license.
=20

 NEW SINCE Beta 1.05
 * dat_evd_set_unwaitable and dat_evd_clear_unwaitable
   implemented.=20
=20
 * ib_enum_hca_if() moved below the IB abstraction layer.
=20
 * Added rpath to LD_FLAGS in Makefile to find vendor libraries
=20
 * Dynamic library loading support added for Windows
=20
 * Conformance test updates:
   - CNO tests added
   - Function Records now have depends_on support
   - More connection related tests
   - RDMA tests
=20
 * Bug fixes
   - Only request the remote IP address from the switch (ATS
     naming) when the app requests it. Don't be aggressive.
   - Removed duplicate dapl_evd_wait prototype in dapl.h
   - Fixed uninitialized handle bug in dapltest performance tests
   - cr_accept and cr_reject properly back out changes if the
     provider fails.
   - dapltest fft test fixes will no longer SEGV on non existent
     device names. Cleaned up output.
   - dapltest pthreads create in DETACHED state to avoid resource
     problems.
   - Added locking for SP structure in critical places.
   - Register fixes for correctness, and to report the number
     available when the user asks for 0 entries (instead of
     SEGV).
   - Use More portable member of in6 structure
   - Clean ups for 64 bit compilers
=20
=20
=20
 NEW SINCE Beta 1.04
=20
 * More cleanup of debug and logging messages. More debug and
   trace messages added.
=20
 * Now support 1.1 semantics for PSP and RSPs fully, per CTN 60.
=20
 * DAT registry is now fully 1.1 compliant.
=20
 * evd_stream_merging_supported enabled and used.
=20
 * DAT headers updated for DAT errata 99 and 100.
=20
 * ep_free still not sorted out totally; if you fail to disconnect
   or fail to wait for a disconnect event to arrive, the
   underlying QP will not be freed and you cannot dispose of
   the associated PZ.
=20
 * Conformance test updates:
   - connection tests
   - RDMA tests
   - various bug fixes
   =20
 * Bug fixes
   - ep_free will allow UNCONNECTED EPs to be freed.
   - Cleanup of some comments
   - cr_accept finishes verifying parameters before making=20
     assignments
   - DAT locking fixed to not lock around malloc/free/library
     calls.
   - evd_enable will verify a valid CQ is present before asking
     the provider for callbacks (not all EVDs have CQs).
   - Fixed up support for obtaining remote IP address on a
     connection, and cleaned up associated files.
=20

=20
 NEW SINCE Beta 1.03
=20
 * strerror() implemented
=20
 * Initial pass at verifying the extended attributes of
   ia_openv(). More changes next drop as we get this sorted
   out. This may mandate that you change your current dat.conf
   file: it should be MAJOR 1, MINOR 1, and nonthreadsafe. dat.h
   specifies 'threadsafe' so you need to adjust your makefile
   to override this.
=20
 * dat.conf file example updated for 1.1 spec
=20
 * Now obtain firmware information from the hca_attrib structure
   for ibapi.=20
=20
 * The reference implementation is not thread_safe, so we updated
   the makefiles to reflect this.
=20
 * New 1.1 Provider Attribute optimal_alignment field now
   accurately set.
=20
 * DTO/RMR completion status returns now comply with 1.1 spec.
=20
 * DAT_NOT_IMPLEMENTED changed from 0xffff0000 to 0x0fff0000 per
   1.1 erratta.
=20
 * dat_ep_free() now conforms with the 1.1 spec; will do a
   disconnect if the EP is connected.
=20
 * Conformance test updates:
   - Now supports client/server tests
   - Supports multi-node tests
   - EVD tests
   - Updated IA tests
   - EP tests
   - Improved Makefile
=20
 * Bug fixes
   - Verify connection EVD on EP create
   - DAT_RETURN_SUBTYPE duplicate DAT_INVALID_STATE_EVD_IN_USE
     changed
   - Bug where DISCONNECT_LINK_DOWN was different from DISCONNECT
   - EVD now accurately reflects the number of CQ entries
     allocated by the underlying provider
   - Locking fixes for SMP
=20

 NEW SINCE Beta 1.02
=20
 * Reworked the debug print/log routine and support. This change
   is largely cosmetic, but affects almost every file.
=20
 * Updated error return from dapls_evd_post_software_event() to
   comply with DAPL 1.1
=20
 * Updated ep_free to disconnect if the EP is connected, per the
   1.1 spec.
=20
 * Conformance test updates:
   - Fixes for OS dependent layer
   - Better IA tests
   - Sample/examples directory added
   - depends_on support added=20
=20
 * Bug fixes
   - LMR_PARAM now properly initialized in lmr_create
   - Code clean up: Removed extra assert, better compliance
     with dapl internal naming, better formatting in a few
     source files, removed C++ style comments.
   - Fixed incorrect comparison in dapl_lmr_bind.c
   - dat library now preserves the order of registrations
   - ep_disconnect now uses the correct DAT_CLOSE_FLAGS
   - Fixed code under IBHOSTS_NAMING to work with new SOCKADDR
     definitions.
   - removed extra disconnect_clean() call in callback code
   - dapls_ep_state_subtype() now returns DAT_RETURN_SUBTYPE
   - Corrected simple typos in dat include files
   - ibapi code now properly obtains the IP address from the
     correct port_static_info structures according to the port
     number (instead of using the first one for all ports)
   - Moved lock in dat/common/dat_dr.c to not include malloc;
     this is insufficient, need to go through locking in the
     dictionary routines for next drop.
   - dat_cr_accept now transitions the EP state to=20
     DAT_EP_STATE_COMPLETION_PENDING, in compliance with the 1.1
     spec.
=20

 NEW SINCE Beta 1.01
=20
 * Updated dat_ia_close to not unload the provider library when
   provider's close function fails.
=20
 * Added compile time option to support real-time/embedded
   systems that don't have/use a file system, making the
   DAT static registry problematic (DAT_NO_STATIC_REGISTRY).
=20
 * Conformance test updates:
   - README file with usage instructions
   - Better support for dynamic value tables, removed specific
     types that limited chains development
   - Simple test case for ia_open/query/close
   - Fixes to be 1.1 compliant
   - Reworked much of the code to support an OS dependent layer,
     for portability.
=20
 * Bug fixes
   - rmr_context now correctly returned from lmr_create()
     (really!).=20
   - Fixed bugs on error paths of dat library
   - Fixed casts in various error programs to use an unsigned
     rather than signed value, avoiding sign bit propogation.
   - Cleaned up problems with exit codes from ep_free.
   - Cleaned up error codes in various dapl files
   - Use more portable DAT_SOCK_ADDR dtypes in ibapi connection
     code.
   - dapl_ep_disconnect now complies with 1.1 spec, several
     changes made. GRACEFUL flag now supported in ibapi code.
   - RDMA read limits now supported in EP_ATTRIBUTES on EP
     creation
   - Check async_evd_qlen for bogus values in dapl_ia_open
=20

 NEW SINCE Beta 1.0
=20
 * DAPL conformance test has been checked in. Uses the chain
   technology described in a not to the DAT Collaborative mail
   list. More tests and documentation will be coming.=20
=20
   NOTE:
   This is a work in progress, there will be several changes
   in the upcoming weeks.
=20
 * Updates to document: dapl_vendor_specific_changes.txt. New
   doc: dapl_ibm_api_variations.txt
=20
 * Memory tests and echo tests updated to 1.1
=20
 * Assertion for 'over decremented' reference counts
=20
 * Numerous cleanups, including but not limited to removing
   unnecessary casts; formatting changes; replacing true/false
   with more portable DAT_TRUE/DAT_FALSE; removed dead ifdefs;
   removed unnecessary locks; makefile cleanups;=20
=20
 * DAPL version numbers, as reported in ia_attributes, are now
   in sync with nuemonics in dat.h.=20
=20
 * Ignore 0 length IOVs (1.1 compliance)
=20
 * DAPL_ATS now obtains the IP address (DAT_IA_ADDRESS) from
   the provider. Note that the IA_ADDRESS returned by ep_query
   is a sockaddr_in6 of AF_INET6 family.
=20
 * memory test now has ability to request invalid memry regions
   for negative testing.
=20
 * Enhanced error reporting (more subtypes added to returns)
=20
 * dapl_vendor.h has better documentation and removed confusing
   compile time switch.
=20
 * Bug fixes
   - Don't decrement the EVD refcount in psp_create error path,
     as it will happen again in psp_free. rsp_create had the
     same problem and fix.
   - evd_create will return errors if the queue length is bogus
     or too large for the underlying provider to support.
   - dapltest working better with 1.1 API thanks to some judiciuos
     dat_ep_reset() calls.
   - EP states now correspond exactly to the DAT spec.
   - Added missing locks
   - rsp_create failure will now clean up the EP.
   - rsp_create will now return error codes if it fails.
   - Correctly handle close failures
   - Various *_free routines now return an error code and do not
     clean up on provider failure, allowing the app to recover or take=20
     corrective action.
   - ep_create will verify ep_attributes=20
=20
 * rmr_context now correctly returned from lmr_create().
    =20
=20
 NEW SINCE Alpha 17.0
=20
 * Conformance with the DAT 1.1 specification:
   - Header files
   - API signatures
   - Two level error scheme
   - Dual license implemented: now can license the code under the
     Common Public License or the BSD license.
=20
   Not all 1.1 functionality is implemented in this release,=20
   specifically:
   - dat_lmr_create does not return a valid rmr_context
   - Not all of the EP state transitions are correct
   - 1.1 RSP semantics not fully implemented
   - 1.1 EVD semantics not fully implemented
   - 1.1 registry changes not yet implemented
   - 1.1 provider changes not yet implemented
   - other smaller issues to be resolved
=20
 * dapltest fixes to allow recovery from a REJECT that puts the
   EP into the DISCONNECTED state (and the QP into the ERROR
   state, per the IBTA spec).
=20
 * Implementation of dat_ep_reset.
=20
 * Many, many code cleanups and formatting to comply with the
   DAPL project coding standards (see
   ./dapl/doc/dapl_coding_style.txt)
=20
 * Bug fixes
   - Corrected bug in close(ABRUPT) where evd list was not checked
     for empty.
   - post routines now return sane error codes instead of always
     returning INSUFFICIENT_RESOURCES
=20

 NEW SINCE Alpha 16.0
=20
 * DAPL 1.1 thread safety semantics
=20
 * Full support for ATS name resolution. This is now the=20
   default build for the reference implementation.
=20
 * Put the old style /etc/dapl/ibhosts naming scheme under
   IBHOSTS_NAMING conditional.
=20
 * Private data now complies with errata for the 1.1 spec.
=20
 * Bug fixes
   - Make sure EP state is correct in cr_accept
   - more __func__ compiler cleanups
   - Eliminated calls to ib_hca_query_us with=20
     HCA_QUERY_PORT_INFO_STATIC as it is no longer needed, and
     it avoids a memory trashing bug. ibapi only.
=20
 NEW SINCE Alpha 15.0
=20
 * Improvements to provide all event information regardless of
   how the corresponding operation was posted.
=20
 * dat_mem_test now has the ability to test shared memory
   registration.
=20
 * Shared memory design doc updated.
=20
 * Enhanced dapltest to deal with DTOs getting flushed when
   a connection REJECT is received.
=20
 * Bug fixes
   - Fixed race condition between CQ and EVD destruction
   - Guarantee IB objects are deleted before corresponding
     dapl objects are disabled.
   - Fixed mtu_size specification for EIP and IA; should be
     maximum message size rather than the underlying network mtu
   - CQD was not getting destroyed, but it is now.
   - Abrupt close now always returns DAT_SUCCESS
=20

 NEW SINCE Alpha 14.0
=20
 * All of the Vendor specific values are now collected into a
   single include file, ./include/dapl_vendor.h. Vendors should
   update values in this file with their information.
=20
 * Basic performance metrics added to memory tests.
=20
 * Put a copy of the license governing the reference implementation.
   NOTE: The reference implementation will now include the BSD
   license as an option, but the changes are not reflected in=20
   this copy of the source code.
=20
 * Removed malloc from RMR bind path
=20
 * Added code to support Address Translation Services for the
   Access API provider code. Currently untested, added for prototype
   and demonstration purposes at this time.
=20
 * Enabled DAPL_EXPOSE_HCA_PORTS as the default build. Provider
   names now include the port designator (e.g. ib0a and ib0b
   instead of ib0).
=20
 * Bug fixes
   - CM callback type for DISCONNECT_LINK_DOWN
   - Multiple ports on an HCA now work
   - Added IA name printout to DAT memory test output
   - simple data type fixes to satisfy picky compilers
   - Fixed CM completion types for the Access API
=20
 NEW SINCE Alpha 13.0
=20
 * Error reporting revamped somewhat for both DAT and DAPL.
   Better output, more useful messages.
=20
 * Removed IPOIB_NAMING code. IPoIB is inadequate to do reverse
   name lookups as it is currently defined.
=20
 * Added a new interface to dapl_ipoib_names.h to do reverse name
   lookup. This is a placeholder for the future.
=20
 * Added debug trace flag for DTO error completions. This gets
   rid of the DTO ERROR code 13 op 5 warning message that
   results from closing an EP that has RECV DTO's queued.
=20
 * Are now able to register memory on multiple HCAs
   simultaneously.
=20
 * Made global LMR context hash table into a per-HCA data structure.
=20
 * Bug fixes
   - dapl_init fixed to handle multiple HCAs; the previous code
     failed to increment a pointer which prevented this.
   - Fixed some comments up that were misleading or just plain
     wrong.=20
   - Fixed rmr_bind bug.
   - Fixed Mellanox files to it will build again.
   - Lazy binding of a QP to an EP has been fixed.
   - Moved DAPL_OS_DBG_TYPE to dapl.h; they are global defs
   - max_private_data_size in provider attributes now has the
     right size.
   - ep_free() now implemented according to the 1.1 spec; more
     aggressive on freeing resources.
   - ia_close(ABRUPT) now implemented according to the 1.1 spec;
     much more aggressive on freeing resources.
   - dapltest fft endpoint test, case 2 fixed to follow the spec.
   - EVDs now properly take a reference on a CNO if present.
   - Corrected the use of internal __func__ in the echo_test,
     which makes it portable to more platforms.
=20
 NEW SINCE Alpha 12.0
=20
 * By defining a compile time switch, both ports of an HCA are
   available. They are named by putting an 'a' or 'b' suffix
   on the end of the provider name. The default is to disable
   this switch. For example, on the jni provider:
   If you enable DAPL_EXPOSE_HCA_PORTS, your provider names=20
   are
  jni0a  port 1
  jni0b  port 2
   Otherwise, the standard default is
  jni0  port 1
=20
 * Moved hca_open out of the init code and into ia_open; by
   popular request. If a user doesn't open the IA, the hca is
   not opened and resources are not committed, this is a
   much cleaner implementation. Obtaining an IP address for
   the HCA is also in the ia_open code.
=20
 * Design documents updated
=20
 * dapltest Makefile now supports EXPLICIT_LINK option to change
   static/dynamic linking.
=20
 * Added queryinfo to dapltest fft tests.
=20
 * Improved debugging.
=20
 * DAPL 1.1 spec is now part of the documents tree.
=20
 * Bug fixes
   - Clean up and addition of many comments
   - Fixed handling of unaffiliated errors
   - Corrected several places where the wrong return code was
     being generated.
   - Corrected the events generated by connection callbacks
   - Correctly install the local IP address into the appropriate
     data structures: DAT_IA_ATTR, DAT_EP_PARAM
   - dapltest performance test latency measurements are now correct.
   - Can no longer free an Async EVD allocated by DAPL
=20

 NEW SINCE Alpha 11.0
=20
 * Now support NULL DTO EVDs on ep_create().
=20
 * Improvements to DAT registry to handle arbitrary
   deinitilization order.
=20
 * Removed dead code from dat_mem_test.
=20
 * Several formatting and changes to comply with the DAPL coding
   standard.
=20
 * Bug fixes
   - IA_ATTR rdma_size returned from dat_ia_query() is now set
     to the IBTA spec defined 2^31.
   - dapltest will print out a message upon completion if there
     wasn't enough data to print statistics, rather than just
     exiting.
   - dapltest fft subtests now report if they are not yet
     implemented.
   - dapltest fft connmgt tests fixed to release resources
   - Fix for IPOIB_NAMING.
   - Private data is now properly provided to the user on connect
     events.
   - PSP created EPs (PSP with flag DAT_PSP_PROVIDER) are now
     put into the correct state to allow ep_modify() to properly
     change EP parameters, including the PZ.
   - evd_wait() has been modified to prevent gratuitous notify
     invocations.
=20

 NEW SINCE Alpha 10.0
=20
 * The dapltest limit tests and fft tests have been updated and
   work now. These provide good tests of resource exhaustion and
   common error paths.
=20
 * dat_psp_create_any implemented
=20
 * IPoIB naming API now fully operational. Still not the default
   build, however.
=20
 * dapl_ep_dup_connect reimplemented and tested.
=20
 * disconnecting a disconnected EP now forces a flush, per the
   spec. We also deal with connect_pending disconnects, and
   properly manage state in error paths.
=20
 * No longer allocate CQ's for EVDs that can't use them, e.g.
   connection EVDs.
=20
 * Windows makefiles have been simplified.
=20
 * Improvements to the dat static registry handling.
=20
 * Improved explanations in various comment fields.
=20
 * Various code clean up for implementation consistency
=20
 * Improved debugging
=20
 * Bug fixes
   - Fixed invalid ordering assumptions on multiple EPs in
     dalptest.=20
   - RSP creation now verifies the EP state
   - rsp_free updates the EP state
   - dat_registry_list_providers returns correct size
   - Limit tests fixed: limit_rsp
   - dapltest now cleans up outstanding DTOs after a disconnect,
     avoiding nasty error messages
   - dapl create threads now block all signals (Linux only).
     Library threads should not intercept signals, and now they
     don't.=20
=20

 NEW SINCE Alpha 9.0
=20
 * Asynchronous errors implemented properly.
=20
 * dat_registry now fully functional and cleaned up
=20
 * IB CQD domains are now per HCA, not per IA. On the ibapi
   implementation a CQD results in a thread. Having one per
   HCA is more efficient of resources as well as latency, avoiding
   unnecessary context switches.
=20
 * Implemented missing code for dat_registry_list_providers
=20
 * dapl debugging now supports directing messages to the system
   log. See description below.
=20
 * Better use of 'static'
=20
 * Bug fixes
   - Make sure we reset global variables to prevent racing fini's.
   - Reset IB_INVALID_HANDLE for ibapi to be non-NULL
   - Windows builds with $(SYSTEMROOT) instead of C:
   - ep_free() now fails if there are queued DTOs (per the spec).
   - Removed various bogus tests from dapltest fft suite.
   - Corrected ep_create handle checking
   - Correct locking problem in timeout thread, introduced when
     we changed the lock implementation
   - Verify we don't free a free cm_handle
   - More cleanups to dapltest
   - Changed dapltest device name default from /dev/jni0 to jni0
   -=20
=20
 ** New syslogging notes:
=20
 DAT and DAPL debug libraries have been improved to support
 logging debugging messages to multiple destinations. With this
 change, we've introduced/modified some of the DAT and DAPL
 environment variables.  The environment variables are:
=20
 DAT_CONF : value used as the static registry configuration file,
 overriding the default location, /etc/dat.conf
=20
 DAT_DBG_LEVEL : value sets which parts of the registry you want
 to debug, valid values are
=20
     DAT_OS_DBG_TYPE_SR                  =3D 0x1,=20
     DAT_OS_DBG_TYPE_DR                  =3D 0x2,=20
     DAT_OS_DBG_TYPE_PROVIDER_API        =3D 0x4,=20
     DAT_OS_DBG_TYPE_CONSUMER_API        =3D 0x8,=20
     DAT_OS_DBG_TYPE_ALL                 =3D 0xf=20
=20
 or any combination of these. For example you can use 0x3 to get
 both static and dynamic registry output ("setenv DAT_DBG_LEVEL
 0x3" in csh).
=20
 DAT_DBG_DEST : value sets the output destination, valid values are=20
=20
   DAT_OS_DBG_DEST_STDOUT                =3D 0x1,=20
   DAT_OS_DBG_DEST_SYSLOG                =3D 0x2,=20
   DAT_OS_DBG_DEST_ALL                   =3D 0x3=20
=20
 For example, 0x3 will output to both stdout and the syslog.=20
=20
 DAPL_DBG_LEVEL : value sets which parts of the registry you want
 to debug, valid values are
=20
       DAPL_OS_DBG_LEVEL_ERR       =3D 0x0001,=20
       DAPL_OS_DBG_LEVEL_CALLBACK  =3D 0x0002,=20
       DAPL_OS_DBG_LEVEL_EVD       =3D 0x0004,=20
       DAPL_OS_DBG_LEVEL_CM        =3D 0x0008,=20
       DAPL_OS_DBG_LEVEL_EP        =3D 0x0010,=20
       DAPL_OS_DBG_LEVEL_UTIL      =3D 0x0020=20
=20
 or any combination of these.=20
=20
 DAPL_DBG_DEST : value sets the output destination, valid values
 are
=20
     DAPL_OS_DBG_DEST_STDOUT     =3D 0x0001,=20
     DAPL_OS_DBG_DEST_SYSLOG     =3D 0x0002,=20
     DAPL_OS_DBG_DEST_ALL        =3D 0x0003=20
=20
 The syslog(3) priority is currently=20
=20
 LOG_USER | LOG_DEBUG=20
=20
 so on Linux, the following /etc/syslog.conf entry will put
 messages into the /var/log/messages file:
=20
 ----------------------------------------------------------------=20
 *.debug                   /var/log/messages=20
 ----------------------------------------------------------------=20
=20
=20
=20
 NEW SINCE Alpha 8.0
=20
 * Static registry implemented. This is now the default for=20
   dapltest, but under Makefile conditional.
=20
   See the DAPL spec, Section 7.2, for a description of the=20
   registry. The Unix static registry format is defined in
   7.3.4.=20
=20
   A sample file is located in ./doc/dat.conf. Modify this
   and copy to /etc/dat.conf for it to work.
=20
   See ./doc/dapl_registry_design.txt for an overview.
=20
 * Limit test enhancements
=20
 * Enhanced debugging (but not a lot...)
=20
 * Bug fixes
   - State of RSP EPs now correctly set as RESERVED
   - Connection timeouts now correctly update EP state.
   - Fixed provider layer violation introduced in Alpha 8
     (can't reference qp_state from common!).
   - dapltest deals with multiple recv completions from multiple
     EPs.
   - Replaced various magic numbers ~0 with real typdef constants
   - Corrected typos around RSP code.
   - fixed dapltest racing disconnects: client disconnects are
     racing with events generated by server disconnects, must
     manage events and cleanup without regard to ordering of
     EPs.
   - dapl_ep_free now returns INVALID_HANDLE instead of incorrect
     INVALID_PARAMETER.
   - If disconnect is pending, do not allow the EP state to be
     diverted to ERROR if an async event occurs. The EP must
     transition to UNCONNECTED to be cleaned up.
   - Put QP allocation printf under debug conditional
   - Now returns correct error number if PZ allocation fails
   - If can't allocate an IA, things get properly cleaned up
   - Various small cleanups.
=20

 NEW SINCE Alpha 7.0
=20
 * Now support DAT_COMPLETION_SUPRESS_FLAG
=20
 * Given that the Torrent chip is no longer being sold, and that
   the IB Access API developed originally for Torrent is now
   implemented on Mellanox Tavor chips, the old separation of
   provider directories has become misnamed. They are now
   renamed to follow the interface names, i.e.
=20
    torrent -> ibapi
    tavor   -> vapi
=20
   This is strictly a name change, but allows the tree to make
   more sense.
=20
   The old files have been deleted. Unfortunately, CVS does not
   allow the logs to follow a file rename, so the old logs
   are only available by looking at the old files.
=20
 * Implemented support for DAT_CONNECTION_EVENT_UNREACHABLE.
   This is part of the 1.1 spec, and has been demanded by
   application developers.
=20
 * Implemented dat_ep_dup_connect(). Has not been fully tested.
=20
 * Bug Fixes
   - Fixed cqd not being released if handle was NULL, which is
     a valid handle value for this.
   - Removed some debugging prints
   - Added asserts in critical places.
   - No longer release the cm_handle on the passive side of
     a connection; the provider owns the handle and will manage
     it.
   - Changed the dat calls to make sure things are initialized
     even if dat_init has not been called by the linker/loader.
     This fixes the issue that some systems call library init
     routines in nondeterministic order; a late dat_init destroys
     data placed by registrations.
=20
 NEW SINCE Alpha 6.0
 * Improved diagnostics for the dat_mem_test.
=20
 * Connection timers now enabled. The Timeout parameter of
   dat_ep_connect() now works properly.
=20
 * Enhancements for improved debugging & metrics (small number,
   more to follow).
=20
 * Changed dapl wait objects from being patterned after condition
   variables to being patterned after semaphores. This changes
   lock requirements and makes for cleaner and more efficient
   code.
=20
 * Changed QP state management after a disconnect. Rather than
   immediateley forcing it to RESET and INIT, leave it in the
   ERROR state until the app needs to use the EP again. This
   avoids the unfortunate side effect of losing pending
   completions when the QP moves to RESET.
=20
 * Bug Fixes
   - QP handle is now correctly noted as cleaned up.
   - dat_echo code and Makefile fixes.
   - various small fixes.
=20
 NEW SINCE Alpha 5.0
=20
 * Added support to post SEND and RECV DTOs at the same time on
   the same EP.
=20
 * Improved DAT memory test utility to perform RMR bind
   operations.
=20
 * New FFT (Feature Function )tests for dalptest. Functional
   tests to verify the implementation.
=20
 * Better Windows support. Support for gcc __BASE_FILE__. Changed
   all "%llx" format statements to be platform generic.
=20
 * Various cleanups and changes to enable debugging, and=20
   removed gratuitous printfs on startup.
=20
 * Bug Fixes
   - Maintain private data in the EP structure on the active
     side of a connection. Prevents data from going to the
     free pool after the upcall.
   - Update from Mellanox for cm.h file: alpha5 build was broken.
   - DAPL_CR now properly unlinked from queue on SP before=20
     deallocation
   - Changed 'dat_success' variable names to 'dat_status' to
     conform with coding standard.
   - Fixed multiple #defines that were the same, not all
     compilers support this.
   - Transaction test synchronization message algorithm updated
     to deal with out of order messages.
   - evd_alloc initializes the wait_obj earlier to deal with
     error conditions.
   - Removed linear link walk from llist_remove_entry, which
     was only for debug. The Limit test creates huge numbers
     of resources that took too long to clean up.
   - Added llist element head pointer to entry structures for
     queue removal verification purposes.
   - Cleaned up PZ, LMR, and RMR alloc/dealloc.
   - Removed setting the QP MTU in ep_modify, bogus thing to do.
=20

 NEW SINCE Alpha 4.0
=20
 * Small changes to the IPoIB API. The main code has still not
   been adequately tested to check in, should happen soon.
=20
 * Cleaned up files to be more consistent.
=20
 * First implementation of fork support, Linux only. The pthread
   package supports pthread_at_fork(), which allows handlers to
   be registered and invoked in either the parent or the child
   process. For dapl, the child handler should close handles and
   release resources to avoid any attempt at sharing; the child
   process really can't use the IB resources of the parent.
=20
   This code is complete, but is not active in this release due
   to some weirdness surrounding the fork call. Vendors may
   chose to debug it faster than the dapl team will get back to
   it, so it is provided in the release. The code is fully
   tested as self consistent.
=20
 * Various enhancements to the llist code, general double linked
   list routines.
=20
 * Changed the way we set up for asynchronous errors such that
   the EP pointer is provided in the error callback. This
   provides two important properties: 1) the EP state can always
   be set properly, and 2) we always report on the correct async
   error EVD.
=20
 * Numerous bug fixes and changes to more fully comply with
   the specification.
=20
 * dapl_ep_get_status now properly reports with DTOs are in
   progress.
=20
 * New unit tests for CNO's and OS Wait Proxy Agents.
=20
 * dapl_ep_modify now atomically succeeds or fails.
=20
 * Performance subtest of dapltest now supports: blocking mode,
   polling mode, and CPU utilization measurements.
=20
 * Better vendor support.
=20
 * Windows support is more complete, now includes DLLMain
   entry point, makefile is cleaned up, compilation not longer
   defines single byte alignment (/Zp1), support for Explorer
   Properties and versioning, many fixes.
=20
 * New platform added: Windows Cygwin. Mostly a makefile effort,
   but some supporting code too.
=20
 * Initial support for IA64 (itanium). This is largely untested.
=20
 * Now support timers. The timer code is not used by the
   implementation at this time, but is intended for use by
   connection timeouts. The code has been tested.
=20
 * Added thread support. Only the timer is on a separate thread.
=20
 * Better error detection and reporting.
=20
 * Changed the license from the IBM Common Public License 0.5
   to the IBM Common Public License 1.0. The changes between
   them are strictly cosmetic, but the 1.0 license is now the
   Source Forge approved CPL.
=20
 * Please see the check-in logs since Alpha 4 (December 20)
   for details.
=20
 NEW SINCE Alpha 3.0
 * Removed all instances of OLD_IB_ACCESS_API (IBM only).
=20
 * Added vendor name prefixing for dapl devices. This directly
   applies to the IB Access API, but could affect other Verb
   implementations as well. Consider two vendors that each
   return a list of devices from an hca_enum call, and the
   names are identical. For most applications you only link
   to a single library, so it doesn't matter. However, DAT
   is designed to allow several libraries to be linked at
   once, so we have the possibility of a name space collision.
=20
   The dapl solution is to provide a dapl version of the name
   by prefixing a simple vendor string to the device name. The
   string may be zero length if you have a unique name, and the
   prefix string will be removed when opening the device.
=20
   A simple example involves two companies, Foo and Bar. Both return
   HCA names that are ordinal numbers: 0, 1, etc. An application
   linking with libFooVerbs would hca_open (0). An application
   linking with uDAPL will dat_ia_open("foo0"), and for
   libBarVerbs is would be dat_ia_open("bar0").
=20
   This is a simple scheme to prevent name space collisions.
=20
   Vendors should update their prefix string in ./udapl/dapl_init.c
=20
 * Documentation updates: new IBM_access_api spec and DAT coding
   standards document.
=20
 * Bug Fixes:
   - If ia_open gets an early error it will not be linked on to
     the hca chain, so watch for this case.
   - Torrent fix for hca_query: was trying to use an invalid
     pointer.
   - Better initialize of dynamic structures
   - Clarification of Max RDMA size in various places
   - Replaced malloc/free with platform independent functions in
     a few files.
   - Simple fixes for picky compilers
   - Cleaned up ifdef's in dapltest mdep.h file for consistency.
   - Fixed order of bindings in pointer-array
=20
 * Enhancements to the unit test infrastructure
=20
 * Unit tests for CNOs
=20
 * Source code comments and makefile fixes for Windows
=20
 * Memory allocation has been removed from the DTO path
=20
 * Unused DAT_EVENT_GROUP removed from the spec and the code.
=20
 * Implemented dapl_ep_modify completion flags,
   dapls_ib_qp_modify max_mtu_size and max_rdma_size.
=20
=20
 NEW SINCE Alpha 2.0
 * Bugs fixes:
   - EP now set up properly in Events
   - Event status on DTO properly updated on failure
   - Better pointer validation
   - RMR Bind EVD implemented correctly
   - EP_CREATE deals with errors from underlying provider
   - EP_CONNECT deals with errors from underlying provider
   - CQ initialization corrected
   - No longer try to wait on EVDs unassociated with a CQ
   - Retain the SP until it is really free; were releasing it
     when the last EP was disconnected.
   - misc other fixes.
=20
 * Post disconnect processing introduced
=20
 * Now using the new IBM Access API on IBM builds
=20
 * dapltest performance test enhancements
=20
 * Small correction to dat file to comply with the spec
=20
 * Windows fixes/enhancements. At least one vendor is running
   dapltest on Windows now.
=20
 * The new IPoIB naming API header file is included in
   ./dapl/include/dapl_ipoib_names.h
   The file is self documenting (really!) and should provide
   insights into the direction we are taking. The implementation
   using this API is finished but has not been adequately tested,
   it will be in the next drop.
=20
=20
=20
 NEW SINCE Alpha 1.0
 * Event System rewrite
=20
   Disables InfiniBand callbacks unless there is a thread in
   dat_evd_wait() or CNOs or OS Proxy Wait Objects are in use.
   This removes the cost of the completion path for
   applications that can effectively use polling
   (dat_evd_dequeue()) rather than blocking (dat_evd_wait())
   for their completions.
=20
   Does not copy CQEs from the CQ until required, saving a copy
   along some completion paths.
=20
 * Various updates from Mellanox and IBM. Better compatibility
   in functions and provider support.
=20
 * Change dat/include structure to dat/include/dat, mandating
   changes in dat include files. E.g. now use <dat/udat.h>
   instead of <dat.h>. This allows us to port to more OS
   platforms more easily.
=20
 * Changes to be more compliant with the spec, especially
   acknowledging that any dat function may return
   DAT_INTERNAL_ERROR. More to be done here.
=20
 * Updates and good changes for the dapltest performance module,
   including the ability to set the depth of the pipeline for
   DTOs and fixes to support multiple EPs.
=20
 * Initial Windows port. Compiles but has not been
   tested. Makefile is a bit crude. Only set up to build static
   libraries, need to add support for a udapl DLL. Also need to
   prefix EXPORT to all externally exported functions. Built on
   Win2K using MSVC++ 6.0.
=20
   This software is a little premature, but enough external
   projects are asking for Windows support that we are releasing
   it early. We will certainly appreciate any fixes sent back
   our way!
=20
 * Numerous bug fixes.
=20

 OBTAIN THE CODE
=20
 To obtain the tree for your local machine you can check it
 out of the source repository using CVS tools. CVS is common
 on Unix systems and available as freeware on Windows machines.
 The command to anonymously obtain the source code from=20
 Source Forge (with no password) is:
=20
 cvs -d:pserver:ano...@cv...:/cvsroot/dapl login
 cvs -z3 -d:pserver:ano...@cv...:/cvsroot/dapl co =
.
=20
 When prompted for a password, simply  press the Enter key.
=20
 Source Forge also contains explicit directions on how to become
 a developer, as well as how to use different CVS commands. You may
 also browse the source code using the URL:
=20
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/dapl/
=20

 SYSTEM REQUIREMENTS
=20
 This project has been implemented on Red Hat Linux 7.2. The
 structure of the code is designed to allow other operating
 systems to easily be adapted, but no work has been done in that
 direction.
=20
 The DAPL team has used IBM based InfiniBand HCAs for development,
 and continue with this platform. Concurrently, Mellanox has
 ported this work to their silicon and are successfully running
 DAPL as well. Either platform is available to any group
 considering DAPL work. The structure of the uDAPL source=20
 allows other provider API sets to be easily integrated.
=20
 The development team uses any one of three topologies: a single
 HCA with a wrap plug (loopback) device; two HCAs in a single
 machine; a single HCA in each of two machines. We have not
 used a switch as of this writing. All connections are assumed
 to be point-to-point (P2P) using the same hardware and software
 on both sides of the 4X wire. Using switches and multiple
 source/destinations is under development and will be available
 soon.
=20
 Explicit machine configurations are available upon request.=20
=20

 IN THE TREE
=20
 The DAPL tree contains source code for the uDAPL implementation,
 and also includes tests and documentation.=20
=20
 Included documentation has the base level API of the
 providers: the IBM Access API and the Mellanox Verbs API. Also
 included are a growing number of DAPL design documents which
 lead the reader through specific DAPL subsystems. More
 design documents are in progress and will appear in the tree in
 the near future.
=20
 A small number of test applications and a unit test framework
 are also included. dapltest is the primary testing application
 used by the DAPL team, it is capable of simulating a variety of
 loads and exercises a large number of interfaces. Full
 documentation is included for each of the tests.
=20

 MAKEFILE NOTES
=20
 There are a number #ifdef's in the code that were necessary
 during early development. They are disappearing as we
 have time to take advantage of features and work available from
 newer releases of provider software. You may notice an #ifdef
 <something>_BUSTED, which indicates a particular feature was not
 working at the time the code was written and the DAPL team
 developed a work-around.
=20
 These #ifdefs are not documented as the intent is to remove
 them as soon as possible.
=20
 Of particular relevance are the following #defines:
=20
 - CM_BUSTED
=20
   The DAPL team has been an early adopter of InfiniBand and has
   had to improvise missing functionality while the vendors lag
   our development. InfiniBand uses a Connection Manager (CM) to
   establish a connection between nodes. This #define essentially
   'fakes' a connection by moving a QP into the appropriate
   state. Most of the IB vendors have a working CM now and this
   is no longer the default, but the code remains as some
   development groups are working to catch up.
=20
 - NO_NAME_SERVICE
=20
   Naming is a thorny issue in InfiniBand; translating from a
   hostname or an interface name to a GID that can be used to
   establish a connection with a remote machine. The reference
   implementation provides a simple name service under this
   #define. The goal is to use IPoIB when it becomes
   available. NO_NAME_SERVICE will probably remain in the code
   long term in order to enable various implementations. A
   description of how this works is found in the end_point_design
   document in the doc/ directory.
=20

 CONTRIBUTIONS
=20
 As is common to Source Forge projects, there are a small number
 of developers directly associated with the source tree and having
 privileges to change the tree. Requested updates, changes, bug
 fixes, enhancements, or contributions should be sent to Steve
 Sears at sj...@ne... for review. We welcome your
 contributions and expect the quality of the project will
 improve thanks to your help.
=20
 The core DAPL team is:
=20
   James Lentini
   Steve Sears
   Randy Smith
=20
   ... with contributions from a number of excellent engineers in
   various companies contributing to the open source effort.
=20

 ONGOING WORK
=20
 Not all of the DAPL spec is implemented at this time. Some
 functionality is missing from CNO's and events, shared memory
 will probably not be implemented by the reference implementation
 (there is a write up on this in the doc/ area), and there are
 yet various cases where work remains to be done.  And of course,
 not all of the implemented functionality has been tested yet.
 The DAPL team continues to develop and test the tree with the
 intent of completing the specification and delivering a robust
 and useful implementation.
=20
 We expect work on a kDAPL implementation will begin soon,
 we welcome those who are interested in contributing to this
 project.
=20

The DAPL Team