Menu
▾
▴
sandiaportals-devel — development and design discussions
You can subscribe to this list here.
| 2002 |
Jan
|
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
|
Jul
|
Aug
(3) |
Sep
(31) |
Oct
(9) |
Nov
(23) |
Dec
(9) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2003 |
Jan
(3) |
Feb
(11) |
Mar
(36) |
Apr
(4) |
May
(5) |
Jun
(4) |
Jul
(1) |
Aug
(4) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Andreas D. <ad...@cl...> - 2003-03-13 19:31:48
|
On Mar 13, 2003 12:00 -0700, Peter Braam wrote: > That's a good idea. Another one is "drop once" followed by resuming > normal service. > > I'm not quite sure why we are doing this at such a low level btw, > given that our OST e.g. already has /proc/sys controllable macros to > drop requests and replies. Can someone refresh my memory? For some of the acceptance tests we need to simulate total network failure. For TCP/socknal we do this by just "ifconfig <iface> down", but we can't do that for Elan. This gives us a consistent method of failing the network. It also ensures (I think?) that the OST sees the network failure at the same time as the clients do, so the OST is unable to continue sending packets at the same time the clients' packets are being dropped. Eric, are we also able to block sending of packets with this mechanism? Cheers, Andreas -- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://www-mddsp.enel.ucalgary.ca/People/adilger/ |
|
From: Peter B. <br...@cl...> - 2003-03-13 19:13:09
|
That's a good idea. Another one is "drop once" followed by resuming normal service. I'm not quite sure why we are doing this at such a low level btw, given that our OST e.g. already has /proc/sys controllable macros to drop requests and replies. Can someone refresh my memory? - Peter - On Thu, Mar 13, 2003 at 01:19:25PM -0500, Lee Ward wrote: > Would it be useful to extend your API so that the network fails > intermittently? It might be nice if the simulated failures occurred > unpredictably as well. > > On Thu, 2003-03-13 at 10:45, Eric Barton wrote: > > > > Guys, > > > > I've added a new API in the b_devel branch of the sandiaportals CVS > > repository. We needed something that could simulate network failures for > > lustre recovery testing and I thought this would be the best way to > > implement it. > > > > * int PtlFailNid (ptl_handle_ni_t ni, ptl_nid_t nid, > > unsigned int threshold); > > > > This is a *new* portals test/debug API. It's not in the portals spec. > > It is used to simulate communications failure (incoming and outgoing) > > with arbitrary sets of peers on the given portals interface. > > > > ni which interface to test > > nid PTL_NID_ANY or a specific NID to match > > threshold 0 - restore communications with matching peers > > -1 - fail communications with matching peers indefinitely > > n - fail the next 'n' communications (incoming or outgoing) > > with matching peers. > > The interface might be changed by adding a new argument to be used as > the denominator in: > > failure-probability = 1/n > > You'll still want a threshold. It changes to: > > -1 - simulate failure-rate indefinitely > [0 - n] - simulate failure-rate for only the > next `threshold' communications > > > > > Repeated calls have the obvious effect (i.e. restore does restore, > > additional failures add). > > > > Obviously I'm _not_ arguing this should go into the standard portals spec, > > although I'm sure it will be useful for testing any system that uses > > portals and has to survive network failures. > > > > Hope this is in keeping with the spirit of the API... > > > > -- > > > > Cheers, > > Eric > > > > ---------------------------------------------------- > > |Eric Barton Barton Software | > > |9 York Gardens Tel: +44 (117) 330 1575 | > > |Clifton Mobile: +44 (7909) 680 356 | > > |Bristol BS8 4LL Fax: call first | > > |United Kingdom E-Mail: er...@ba...| > > ---------------------------------------------------- > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by:Crypto Challenge is now open! > > Get cracking and register here for some mind boggling fun and > > the chance of winning an Apple iPod: > > http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en > > _______________________________________________ > > Sandiaportals-devel mailing list > > San...@li... > > https://lists.sourceforge.net/lists/listinfo/sandiaportals-devel > > > > > ------------------------------------------------------- > This SF.net email is sponsored by:Crypto Challenge is now open! > Get cracking and register here for some mind boggling fun and > the chance of winning an Apple iPod: > http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en > _______________________________________________ > Sandiaportals-devel mailing list > San...@li... > https://lists.sourceforge.net/lists/listinfo/sandiaportals-devel - Peter - |
|
From: Eric B. <er...@ba...> - 2003-03-13 19:05:42
|
Sounds reasonable; although my initial brief was just "turn it on/turn it off".
Maybe an further 'probability' parameter of type double and range (0.0 - 1.0)?
Cheers,
Eric
----------------------------------------------------
|Eric Barton Barton Software |
|9 York Gardens Tel: +44 (117) 330 1575 |
|Clifton Mobile: +44 (7909) 680 356 |
|Bristol BS8 4LL Fax: call first |
|United Kingdom E-Mail: er...@ba...|
----------------------------------------------------
> -----Original Message-----
> From: san...@li...
> [mailto:san...@li...]On
> Behalf Of Lee
> Ward
> Sent: Thursday, March 13, 2003 6:19 PM
> To: Eric Barton
> Cc: san...@li...
> Subject: Re: [Sandiaportals-devel] Simulation of Portals
> communicationsfailures.
>
>
> Would it be useful to extend your API so that the network fails
> intermittently? It might be nice if the simulated failures occurred
> unpredictably as well.
>
> On Thu, 2003-03-13 at 10:45, Eric Barton wrote:
> >
> > Guys,
> >
> > I've added a new API in the b_devel branch of the sandiaportals CVS
> > repository. We needed something that could simulate
> network failures for
> > lustre recovery testing and I thought this would be the best way to
> > implement it.
> >
> > * int PtlFailNid (ptl_handle_ni_t ni, ptl_nid_t nid,
> > unsigned int threshold);
> >
> > This is a *new* portals test/debug API. It's not in the
> portals spec.
> > It is used to simulate communications failure (incoming
> and outgoing)
> > with arbitrary sets of peers on the given portals interface.
> >
> > ni which interface to test
> > nid PTL_NID_ANY or a specific NID to match
> > threshold 0 - restore communications with matching peers
> > -1 - fail communications with matching peers
> indefinitely
> > n - fail the next 'n' communications
> (incoming or outgoing)
> > with matching peers.
>
> The interface might be changed by adding a new argument to be used as
> the denominator in:
>
> failure-probability = 1/n
>
> You'll still want a threshold. It changes to:
>
> -1 - simulate failure-rate indefinitely
> [0 - n] - simulate failure-rate for only the
> next `threshold' communications
>
> >
> > Repeated calls have the obvious effect (i.e. restore
> does restore,
> > additional failures add).
> >
> > Obviously I'm _not_ arguing this should go into the
> standard portals spec,
> > although I'm sure it will be useful for testing any system that uses
> > portals and has to survive network failures.
> >
> > Hope this is in keeping with the spirit of the API...
> >
> > --
> >
> > Cheers,
> > Eric
> >
> > ----------------------------------------------------
> > |Eric Barton Barton Software |
> > |9 York Gardens Tel: +44 (117) 330 1575 |
> > |Clifton Mobile: +44 (7909) 680 356 |
> > |Bristol BS8 4LL Fax: call first |
> > |United Kingdom E-Mail: er...@ba...|
> > ----------------------------------------------------
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by:Crypto Challenge is now open!
> > Get cracking and register here for some mind boggling fun and
> > the chance of winning an Apple iPod:
> > http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en
> > _______________________________________________
> > Sandiaportals-devel mailing list
> > San...@li...
> > https://lists.sourceforge.net/lists/listinfo/sandiaportals-devel
>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by:Crypto Challenge is now open!
> Get cracking and register here for some mind boggling fun and
> the chance of winning an Apple iPod:
> http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en
> _______________________________________________
> Sandiaportals-devel mailing list
> San...@li...
> https://lists.sourceforge.net/lists/listinfo/sandiaportals-devel
>
|
|
From: Lee W. <le...@sa...> - 2003-03-13 18:20:00
|
Would it be useful to extend your API so that the network fails intermittently? It might be nice if the simulated failures occurred unpredictably as well. On Thu, 2003-03-13 at 10:45, Eric Barton wrote: > > Guys, > > I've added a new API in the b_devel branch of the sandiaportals CVS > repository. We needed something that could simulate network failures for > lustre recovery testing and I thought this would be the best way to > implement it. > > * int PtlFailNid (ptl_handle_ni_t ni, ptl_nid_t nid, > unsigned int threshold); > > This is a *new* portals test/debug API. It's not in the portals spec. > It is used to simulate communications failure (incoming and outgoing) > with arbitrary sets of peers on the given portals interface. > > ni which interface to test > nid PTL_NID_ANY or a specific NID to match > threshold 0 - restore communications with matching peers > -1 - fail communications with matching peers indefinitely > n - fail the next 'n' communications (incoming or outgoing) > with matching peers. The interface might be changed by adding a new argument to be used as the denominator in: failure-probability = 1/n You'll still want a threshold. It changes to: -1 - simulate failure-rate indefinitely [0 - n] - simulate failure-rate for only the next `threshold' communications > > Repeated calls have the obvious effect (i.e. restore does restore, > additional failures add). > > Obviously I'm _not_ arguing this should go into the standard portals spec, > although I'm sure it will be useful for testing any system that uses > portals and has to survive network failures. > > Hope this is in keeping with the spirit of the API... > > -- > > Cheers, > Eric > > ---------------------------------------------------- > |Eric Barton Barton Software | > |9 York Gardens Tel: +44 (117) 330 1575 | > |Clifton Mobile: +44 (7909) 680 356 | > |Bristol BS8 4LL Fax: call first | > |United Kingdom E-Mail: er...@ba...| > ---------------------------------------------------- > > > ------------------------------------------------------- > This SF.net email is sponsored by:Crypto Challenge is now open! > Get cracking and register here for some mind boggling fun and > the chance of winning an Apple iPod: > http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en > _______________________________________________ > Sandiaportals-devel mailing list > San...@li... > https://lists.sourceforge.net/lists/listinfo/sandiaportals-devel |
|
From: Eric B. <er...@ba...> - 2003-03-13 16:09:13
|
Guys,
I've added a new API in the b_devel branch of the sandiaportals CVS
repository. We needed something that could simulate network failures for
lustre recovery testing and I thought this would be the best way to
implement it.
* int PtlFailNid (ptl_handle_ni_t ni, ptl_nid_t nid,
unsigned int threshold);
This is a *new* portals test/debug API. It's not in the portals spec.
It is used to simulate communications failure (incoming and outgoing)
with arbitrary sets of peers on the given portals interface.
ni which interface to test
nid PTL_NID_ANY or a specific NID to match
threshold 0 - restore communications with matching peers
-1 - fail communications with matching peers indefinitely
n - fail the next 'n' communications (incoming or outgoing)
with matching peers.
Repeated calls have the obvious effect (i.e. restore does restore,
additional failures add).
Obviously I'm _not_ arguing this should go into the standard portals spec,
although I'm sure it will be useful for testing any system that uses
portals and has to survive network failures.
Hope this is in keeping with the spirit of the API...
--
Cheers,
Eric
----------------------------------------------------
|Eric Barton Barton Software |
|9 York Gardens Tel: +44 (117) 330 1575 |
|Clifton Mobile: +44 (7909) 680 356 |
|Bristol BS8 4LL Fax: call first |
|United Kingdom E-Mail: er...@ba...|
----------------------------------------------------
|
|
From: Gerard B. <gb...@cs...> - 2003-02-25 22:28:12
|
Hey! I've got the Portals mpich 1.2.0 library. I think this library is from Cplant Portals and I don't know how to compile it under the Sourceforge Portals. (There's a CPLANT_ARCH variable in the Makefiles) Has anyone any experience with this? I'm thankfull for any help! G=E9rard |
|
From: Eric B. <er...@ba...> - 2003-02-03 22:48:24
|
Monish,
> I just talked to our hardware team and the change in the length offset is
> not an option for them (at least in the near future). It would require a lot
> of effort and testing to make this change. Could we stick to the fix that
> Andreas put in for 0.5.19.1 to handle the lengths correctly? I know that it
> is not the most elegant solution but it will be the easiest one to proceed
> with at this point.
>
> I will start tests to verify if the connection message changes will work
> with the BA OST. Is there a reason why we cannot have the source nid's
> exchanged during the lustre CONNECT message? It would be very easy for us to
> refine the lustre connect protocol and exchange the nid's along with the
> uuid's.
Since expedience seems to predominate at this time, we should go with
Andreas's patch and shelve the connection handshake until later.
Please note however that the portals connection handshake is concerned
purely with the portals protocol and not higher levels (e.g. lustre is to
portals as TCP is to IP). We need it to implement a simple and general
method for looking up connected socket by peer NID, so that a single node
can have a single NID but multiple IP addresses. It is not even possible
to deliver the lustre CONNECT message to a multi-NIC lustre server until
this exchange has taken place.
We _will_ need to implement the connection handshake before lustre 1.0, and
we really should have moved all common message header fields to the common
part of the header by then too.
With that in mind, I'd appreciate it if you could give me an indication of
timescales required.
--
Cheers,
Eric
----------------------------------------------------
|Eric Barton Barton Software |
|9 York Gardens Tel: +44 (117) 330 1575 |
|Clifton Mobile: +44 (7909) 680 356 |
|Bristol BS8 4LL Fax: call first |
|United Kingdom E-Mail: er...@ba...|
----------------------------------------------------
|
|
From: Andreas D. <ad...@cl...> - 2003-02-03 20:43:21
|
On Feb 03, 2003 12:33 -0800, Monish Suvarna wrote: > I just talked to our hardware team and the change in the length offset is > not an option for them (at least in the near future). It would require a lot > of effort and testing to make this change. Could we stick to the fix that > Andreas put in for 0.5.19.1 to handle the lengths correctly? I know that it > is not the most elegant solution but it will be the easiest one to proceed > with at this point. Monish, I think in light of LLNL's desire to have a release of the BA OST code which will work with more recent Lustre code, we should proceed with the use-current-length-offset code that we have now. You can start the process to have the hardware folks make the "length" offset change for some later release. > I will start tests to verify if the connection message changes will work > with the BA OST. Is there a reason why we cannot have the source nid's > exchanged during the lustre CONNECT message? It would be very easy for us to > refine the lustre connect protocol and exchange the nid's along with the > uuid's. Again, I don't think we should start making additional changes for the pending release - I think Eric's comments are more for implementation in say the following BA OST release. Cheers, Andreas -- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://www-mddsp.enel.ucalgary.ca/People/adilger/ |
|
From: Monish S. <msu...@bl...> - 2003-02-03 20:34:30
|
Eric,
I just talked to our hardware team and the change in the length offset is
not an option for them (at least in the near future). It would require a lot
of effort and testing to make this change. Could we stick to the fix that
Andreas put in for 0.5.19.1 to handle the lengths correctly? I know that it
is not the most elegant solution but it will be the easiest one to proceed
with at this point.
I will start tests to verify if the connection message changes will work
with the BA OST. Is there a reason why we cannot have the source nid's
exchanged during the lustre CONNECT message? It would be very easy for us to
refine the lustre connect protocol and exchange the nid's along with the
uuid's.
Thanks,
Monish
-----Original Message-----
From: Eric Barton [mailto:er...@ba...]
Sent: Monday, February 03, 2003 6:25 AM
To: msu...@bl...
Cc: za...@za...; Andreas Dilger; Peter Braam;
san...@li...
Subject: RE: [Sandiaportals-devel] RE: ptl_get_t "length" field
Monish,
Before I dive in, I should mention that portals is actually an API,
rather than a wire protocol standard, so we are free to define different
wire formats on different interconnects. However, since we envisage
portals over TCP/IP as inter-operating with many different hardware
platforms, we do need to take some care in the protocol we define
and agree here.
We have actually been waiting for a suitable opportunity to introduce
a couple of portals/TCP/IP wire protocol changes...
Firstly, we need to implement a NID exchange when a new connection is
established. The reason we need this exchange is that both parties need to
determine the NID of the peer node. Until now, we've been "getting away"
with using the IP address of the peer as its NID, thus avoiding the need;
however as we build more complex portals networks out of nodes with
multiple NICs, this leads to problems.
BTW, I understand that this might seem totally irrelevent to the BA
implementation, since BAs are always servers; i.e. they are always on the
"edge" of the network and only accept connections and never initiate them
so they never need to interpret the dest_nid field, and always reply on the
same connection the request came in on. However in the interests of
tractible network configuration, we really want to make the NID exchange
ubiquitous.
Secondly, we can't assume that the portals/TCP/IP wire protocol will remain
fixed for all time. Including a 'magic' number to identify the TCP/IP
stream as a series of portals messages, along with a wire protocol version
number seems sensible.
With this in mind, I'd like to suggest that when a portals/TCP/IP
connection is established (irrespective of whether it was initiated or
accepted), both peers send and receive a connection message that includes
the portals 'magic' number, the portals/TCP/IP wire protocol version and
the sender's NID.
Please note that the portals protocol version says _nothing_ about the
version of the lustre protocol it encapsulates, which is passed in the
lustre request message.
Now back to the ptl_get_t 'length' field.
Making 'length' a common field for all portals message types seems
very appropriate to me, and actually simplifies some of the portals
'lib' code. However it seems to me that we really should move this
field into the common part of the portals header.
These proposals are summarised below. Please call me (+44 117 330 1575) to
discuss...
1/ All low level datatypes remain as now. However all types that
require interpretation are little-endian on the wire; i.e...
ptl_nid_t 64 bit little-endian unsigned integer
ptl_pid_t 32 bit little-endian unsigned integer
ptl_pt_index_t 32 bit little-endian unsigned integer
ptl_match_bits_t 64 bit little-endian unsigned integer
ptl_hdr_data_t 64 bit opaque. Only interpreted by higher levels.
ptl_size_t 32 bit little-endian unsigned integer
ptl_handle_wire_t 128 bit opaque, except for the reserved value
0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF which
is used in PTL_MSG_PUT to mean no ACK
is required
2/ Define the connection message...
#define PORTALS_MAGIC 0x01020304c0d0e0f0
#define PORTALS_TCPIP_VERSION_MAJOR 0
#define PORTALS_TCPIP_VERISON_MINOR 1
typedef struct {
__u64 magic;
ptl_nid_t src_nid;
__u32 version_minor;
__u32 version_major;
__u32 reserved_zero;
ptl_size_t reserved_zero;
} ptl_connection_msg_t;
All fields are little-endian. Note that this overlays fairly sensibly on
the portals message header described below; if it makes life easier, this
message could be padded to match the size of portals header (with implicit
zero length message body).
2/ Portals messages consist of a fixed sized message header, followed by
a variable sized message body...
typedef enum {
PTL_MSG_ACK = 0,
PTL_MSG_PUT,
PTL_MSG_GET,
PTL_MSG_REPLY,
} ptl_msg_type_t;
typedef struct ptl_ack {
ptl_handle_wire_t dst_wmd;
ptl_match_bits_t match_bits;
ptl_size_t mlength;
} ptl_ack_t;
typedef struct ptl_put {
ptl_handle_wire_t ack_wmd;
ptl_match_bits_t match_bits;
ptl_pt_index_t ptl_index;
ptl_size_t offset;
ptl_hdr_data_t hdr_data;
} ptl_put_t;
typedef struct ptl_get {
ptl_handle_wire_t return_wmd;
ptl_match_bits_t match_bits;
ptl_pt_index_t ptl_index;
ptl_size_t reply_length;
ptl_size_t src_offset;
ptl_size_t return_offset;
} ptl_get_t;
typedef struct ptl_reply {
ptl_handle_wire_t dst_wmd;
ptl_size_t dst_offset;
} ptl_reply_t;
typedef struct {
ptl_nid_t dest_nid;
ptl_nid_t src_nid;
ptl_pid_t dest_pid;
ptl_pid_t src_pid;
__u32 type;
ptl_size_t length;
union {
ptl_ack_t ack;
ptl_put_t put;
ptl_get_t get;
ptl_reply_t reply;
} msg;
} ptl_hdr_t;
--
Cheers,
Eric
----------------------------------------------------
|Eric Barton Barton Software |
|9 York Gardens Tel: +44 (117) 330 1575 |
|Clifton Mobile: +44 (7909) 680 356 |
|Bristol BS8 4LL Fax: call first |
|United Kingdom E-Mail: er...@ba...|
----------------------------------------------------
*********************************************************************
This e-mail and any attachment is confidential. It may only be read, copied and used by the intended recipient(s). If you are not the intended recipient(s), you may not copy, use, distribute, forward, store or disclose this e-mail or any attachment. If you are not the intended recipient(s) or have otherwise received this e-mail in error, you should destroy it and any attachment and notify the sender by reply e-mail or send a message to sys...@bl...
*********************************************************************
|
|
From: Eric B. <er...@ba...> - 2003-02-03 14:25:08
|
Monish,
Before I dive in, I should mention that portals is actually an API,
rather than a wire protocol standard, so we are free to define different
wire formats on different interconnects. However, since we envisage
portals over TCP/IP as inter-operating with many different hardware
platforms, we do need to take some care in the protocol we define
and agree here.
We have actually been waiting for a suitable opportunity to introduce
a couple of portals/TCP/IP wire protocol changes...
Firstly, we need to implement a NID exchange when a new connection is
established. The reason we need this exchange is that both parties need to
determine the NID of the peer node. Until now, we've been "getting away"
with using the IP address of the peer as its NID, thus avoiding the need;
however as we build more complex portals networks out of nodes with
multiple NICs, this leads to problems.
BTW, I understand that this might seem totally irrelevent to the BA
implementation, since BAs are always servers; i.e. they are always on the
"edge" of the network and only accept connections and never initiate them
so they never need to interpret the dest_nid field, and always reply on the
same connection the request came in on. However in the interests of
tractible network configuration, we really want to make the NID exchange
ubiquitous.
Secondly, we can't assume that the portals/TCP/IP wire protocol will remain
fixed for all time. Including a 'magic' number to identify the TCP/IP
stream as a series of portals messages, along with a wire protocol version
number seems sensible.
With this in mind, I'd like to suggest that when a portals/TCP/IP
connection is established (irrespective of whether it was initiated or
accepted), both peers send and receive a connection message that includes
the portals 'magic' number, the portals/TCP/IP wire protocol version and
the sender's NID.
Please note that the portals protocol version says _nothing_ about the
version of the lustre protocol it encapsulates, which is passed in the
lustre request message.
Now back to the ptl_get_t 'length' field.
Making 'length' a common field for all portals message types seems
very appropriate to me, and actually simplifies some of the portals
'lib' code. However it seems to me that we really should move this
field into the common part of the portals header.
These proposals are summarised below. Please call me (+44 117 330 1575) to
discuss...
1/ All low level datatypes remain as now. However all types that
require interpretation are little-endian on the wire; i.e...
ptl_nid_t 64 bit little-endian unsigned integer
ptl_pid_t 32 bit little-endian unsigned integer
ptl_pt_index_t 32 bit little-endian unsigned integer
ptl_match_bits_t 64 bit little-endian unsigned integer
ptl_hdr_data_t 64 bit opaque. Only interpreted by higher levels.
ptl_size_t 32 bit little-endian unsigned integer
ptl_handle_wire_t 128 bit opaque, except for the reserved value
0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF which
is used in PTL_MSG_PUT to mean no ACK
is required
2/ Define the connection message...
#define PORTALS_MAGIC 0x01020304c0d0e0f0
#define PORTALS_TCPIP_VERSION_MAJOR 0
#define PORTALS_TCPIP_VERISON_MINOR 1
typedef struct {
__u64 magic;
ptl_nid_t src_nid;
__u32 version_minor;
__u32 version_major;
__u32 reserved_zero;
ptl_size_t reserved_zero;
} ptl_connection_msg_t;
All fields are little-endian. Note that this overlays fairly sensibly on
the portals message header described below; if it makes life easier, this
message could be padded to match the size of portals header (with implicit
zero length message body).
2/ Portals messages consist of a fixed sized message header, followed by
a variable sized message body...
typedef enum {
PTL_MSG_ACK = 0,
PTL_MSG_PUT,
PTL_MSG_GET,
PTL_MSG_REPLY,
} ptl_msg_type_t;
typedef struct ptl_ack {
ptl_handle_wire_t dst_wmd;
ptl_match_bits_t match_bits;
ptl_size_t mlength;
} ptl_ack_t;
typedef struct ptl_put {
ptl_handle_wire_t ack_wmd;
ptl_match_bits_t match_bits;
ptl_pt_index_t ptl_index;
ptl_size_t offset;
ptl_hdr_data_t hdr_data;
} ptl_put_t;
typedef struct ptl_get {
ptl_handle_wire_t return_wmd;
ptl_match_bits_t match_bits;
ptl_pt_index_t ptl_index;
ptl_size_t reply_length;
ptl_size_t src_offset;
ptl_size_t return_offset;
} ptl_get_t;
typedef struct ptl_reply {
ptl_handle_wire_t dst_wmd;
ptl_size_t dst_offset;
} ptl_reply_t;
typedef struct {
ptl_nid_t dest_nid;
ptl_nid_t src_nid;
ptl_pid_t dest_pid;
ptl_pid_t src_pid;
__u32 type;
ptl_size_t length;
union {
ptl_ack_t ack;
ptl_put_t put;
ptl_get_t get;
ptl_reply_t reply;
} msg;
} ptl_hdr_t;
--
Cheers,
Eric
----------------------------------------------------
|Eric Barton Barton Software |
|9 York Gardens Tel: +44 (117) 330 1575 |
|Clifton Mobile: +44 (7909) 680 356 |
|Bristol BS8 4LL Fax: call first |
|United Kingdom E-Mail: er...@ba...|
----------------------------------------------------
|
|
From: Peter J. B. <br...@cl...> - 2003-02-01 06:30:40
|
While you are at it, would you mind adding NTOH__u32 etc to each of these? Thanks. There is a bug filed for that. - Peter - On Fri, Jan 31, 2003 at 10:30:18PM -0700, Andreas Dilger wrote: > On Jan 31, 2003 20:38 -0500, Zach Brown wrote: > > > The following patch (which appears to work fine for me) moves our usage of > > > the "length" field in the ptl_get_t to data_get_length, and sets "length" > > > to zero to match other uses of that field as the extra request data. > > > > cool, thanks. > > > > > + __u32 unused2; /* for "length" alignment */ > > > > do we want to be sure to explicitly set this to 0 with the other reply > > fields in parse_get? > > Probably yes - we should zero all unused fields so that we may be able > to use them in the future. > > Cheers, Andreas > -- > Andreas Dilger > http://sourceforge.net/projects/ext2resize/ > http://www-mddsp.enel.ucalgary.ca/People/adilger/ - Peter - |
|
From: Andreas D. <ad...@cl...> - 2003-02-01 05:30:36
|
On Jan 31, 2003 20:38 -0500, Zach Brown wrote: > > The following patch (which appears to work fine for me) moves our usage of > > the "length" field in the ptl_get_t to data_get_length, and sets "length" > > to zero to match other uses of that field as the extra request data. > > cool, thanks. > > > + __u32 unused2; /* for "length" alignment */ > > do we want to be sure to explicitly set this to 0 with the other reply > fields in parse_get? Probably yes - we should zero all unused fields so that we may be able to use them in the future. Cheers, Andreas -- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://www-mddsp.enel.ucalgary.ca/People/adilger/ |
|
From: Monish S. <msu...@bl...> - 2003-02-01 01:48:14
|
Looks great to me too. I am currently testing with a similar hack to ptl_reply on my clients. Monday is fine for the patch as long as we do not change directions or make additional changes. I will test bulk writes over the weekend using Andreas's patch. Andreas, thanks a lot of the quick turnaround. Monish -----Original Message----- From: Zach Brown [mailto:za...@cl...] Sent: Friday, January 31, 2003 5:38 PM To: ad...@cl...; Monish Suvarna; Peter J. Braam; Eric Barton; san...@li... Subject: Re: ptl_get_t "length" field > The following patch (which appears to work fine for me) moves our usage of > the "length" field in the ptl_get_t to data_get_length, and sets "length" > to zero to match other uses of that field as the extra request data. cool, thanks. > + __u32 unused2; /* for "length" alignment */ do we want to be sure to explicitly set this to 0 with the other reply fields in parse_get? > Zach, Eric, does this look OK? if Monish signs off on it, I'm happy. he cooked a very similar patch during his testing so it shouldn't be too bad.. - z ********************************************************************* This e-mail and any attachment is confidential. It may only be read, copied and used by the intended recipient(s). If you are not the intended recipient(s), you may not copy, use, distribute, forward, store or disclose this e-mail or any attachment. If you are not the intended recipient(s) or have otherwise received this e-mail in error, you should destroy it and any attachment and notify the sender by reply e-mail or send a message to sys...@bl... ********************************************************************* |
|
From: Zach B. <za...@cl...> - 2003-02-01 01:38:30
|
> The following patch (which appears to work fine for me) moves our usage of > the "length" field in the ptl_get_t to data_get_length, and sets "length" > to zero to match other uses of that field as the extra request data. cool, thanks. > + __u32 unused2; /* for "length" alignment */ do we want to be sure to explicitly set this to 0 with the other reply fields in parse_get? > Zach, Eric, does this look OK? if Monish signs off on it, I'm happy. he cooked a very similar patch during his testing so it shouldn't be too bad.. - z |
|
From: Eric B. <er...@ba...> - 2003-02-01 00:55:01
|
Adilger, Monish,
I need a little time to collect my thoughts. If Monday is too late
please let me know ASAP.
Cheers,
Eric
----------------------------------------------------
|Eric Barton Barton Software |
|9 York Gardens Tel: +44 (117) 330 1575 |
|Clifton Mobile: +44 (7909) 680 356 |
|Bristol BS8 4LL Fax: call first |
|United Kingdom E-Mail: er...@ba...|
----------------------------------------------------
> -----Original Message-----
> From: 'ad...@cl...' [mailto:ad...@cl...]
> Sent: Saturday, February 01, 2003 12:33 AM
> To: Monish Suvarna
> Cc: 'Zach Brown'; Peter J. Braam; Eric Barton;
> san...@li...
> Subject: Re: ptl_get_t "length" field
>
>
> On Jan 31, 2003 15:18 -0800, Monish Suvarna wrote:
> > The changes with adding extra 4 byes before the plt_replt_t
> structure works.
> > The server no longer drops the connection when it gets a
> ptl_reply. I am now
> > fixing the ost_bulk_handler code to use the ptl_reply
> cookie instead of the
> > xid to match up the bulk structures sent by the original ptl_put.
>
> Monish,
> I will commit the change to move the "length" field in the
> ptl_reply_t field
> to the same offset as the other messages. This fixes the
> immediate concern
> of working with your next release.
>
> The following patch (which appears to work fine for me) moves
> our usage of
> the "length" field in the ptl_get_t to data_get_length, and
> sets "length"
> to zero to match other uses of that field as the extra request data.
>
> Zach, Eric, does this look OK? Monish, since I'm not a
> Portals person and
> I haven't done extensive testing of this patch (although I
> think any error
> would be evident immediately), I'm hesitant to say that you
> should use this
> change to implement PtlGet on the BA OSTs for this release
> until I get some
> confirmation from Zach and/or Eric that it is correct.
>
> Cheers, Andreas
> ==============================================================
> ================
> diff -u -p -r1.31 lib-types.h
> --- include/portals/lib-types.h 13 Nov 2002 21:37:03
> -0000 1.31
> +++ include/portals/lib-types.h 1 Feb 2003 00:16:42 -0000
> @@ -40,7 +40,10 @@ typedef enum {
>
> /* Each of these structs should start with an odd number of
> * __u32, or the compiler could add its own padding and confuse
> - * everyone. */
> + * everyone.
> + *
> + * Also, "length" needs to be at offset 28 of each struct.
> + */
> typedef struct ptl_ack {
> ptl_size_t mlength;
> ptl_handle_wire_t dst_wmd;
> @@ -61,15 +64,17 @@ typedef struct ptl_get {
> ptl_pt_index_t ptl_index;
> ptl_handle_wire_t return_wmd;
> ptl_match_bits_t match_bits;
> - ptl_size_t length;
> + ptl_size_t length; /* length of extra
> message data */
> ptl_size_t src_offset;
> ptl_size_t return_offset;
> + ptl_size_t data_req_length; /* length of
> requested bulk data */
> } ptl_get_t;
>
> typedef struct ptl_reply {
> - __u32 unused;
> + __u32 unused1; /* for 64-bit alignment */
> ptl_handle_wire_t dst_wmd;
> ptl_size_t dst_offset;
> + __u32 unused2; /* for "length" alignment */
> ptl_size_t length;
> } ptl_reply_t;
>
> diff -u -p -r1.46 lib-move.c
> --- lib/lib-move.c 20 Dec 2002 11:22:35 -0000 1.46
> +++ lib/lib-move.c 1 Feb 2003 00:16:42 -0000
> @@ -341,25 +341,26 @@ static int parse_get(nal_cb_t * nal, ptl
>
> me = lib_find_me(nal, hdr->msg.get.ptl_index, PTL_MD_OP_GET,
> hdr->src_nid, hdr->src_pid,
> - hdr->msg.get.length,
> hdr->msg.get.src_offset,
> - hdr->msg.get.match_bits,
> - &mlength, &offset, &unlink);
> + hdr->msg.get.data_req_length,
> hdr->msg.get.src_offset,
> + hdr->msg.get.match_bits, &mlength,
> &offset, &unlink);
>
> if (me == NULL) {
> - CERROR("Dropping get from portal %d offset
> %d, length %d, "
> - "match "LPX64", from "LPU64".%d\n",
> hdr->msg.get.ptl_index,
> - hdr->msg.get.src_offset, hdr->msg.get.length,
> + CERROR("Dropping get from portal %u offset
> %u, data length %u, "
> + "match "LPX64", from "LPU64".%d\n",
> + hdr->msg.get.ptl_index,
> hdr->msg.get.src_offset,
> + hdr->msg.get.data_req_length,
> hdr->msg.get.match_bits,
> hdr->src_nid, hdr->src_pid);
> goto drop;
> }
>
> md = me->md;
> ptl_md2handle(&handle, md, nal);
> - CDEBUG(D_NET, "Incoming get index %d from "LPU64".%u
> of length %d/%d from "
> - "md %ld [%d](%p:%d)... + %d\n",
> hdr->msg.get.ptl_index,
> - hdr->src_nid, hdr->src_pid,
> - mlength, hdr->msg.get.length,
> handle.handle_idx, md->md_niov,
> - md->md_iov[0].mdiov_base,
> md->md_iov[0].mdiov_len, offset);
> + CDEBUG(D_NET, "Incoming get index %d from "LPU64".%u of "
> + "length %u/%u from md %lu [%d](%p:%d)... + %d\n",
> + hdr->msg.get.ptl_index, hdr->src_nid, hdr->src_pid,
> + mlength, hdr->msg.get.data_req_length,
> handle.handle_idx,
> + md->md_niov, md->md_iov[0].mdiov_base,
> md->md_iov[0].mdiov_len,
> + offset);
>
> msg = get_new_msg (nal);
> if (msg == NULL) {
> @@ -375,7 +376,7 @@ static int parse_get(nal_cb_t * nal, ptl
> msg->ev.initiator.pid = hdr->src_pid;
> msg->ev.portal = hdr->msg.get.ptl_index;
> msg->ev.match_bits = hdr->msg.get.match_bits;
> - msg->ev.rlength = hdr->msg.get.length;
> + msg->ev.rlength = hdr->msg.get.data_req_length;
> msg->ev.mlength = mlength;
> msg->ev.offset = offset;
> msg->ev.hdr_data = 0;
> @@ -640,15 +641,16 @@ void print_hdr(nal_cb_t * nal, ptl_hdr_t
>
> if (hdr->type == PTL_MSG_GET) {
> nal->cb_printf(nal,
> - " Ptl index %d, return md
> 0x%08llx, match "
> - "bits 0x%0lx\n",
> hdr->msg.get.ptl_index,
> - (long
> long)hdr->msg.get.return_wmd.handle_idx,
> + " Ptl index %u, return md
> "LPX64", match "
> + "bits 0x%0x\n",
> hdr->msg.get.ptl_index,
> + hdr->msg.get.return_wmd.handle_idx,
> hdr->msg.get.match_bits);
> nal->cb_printf(nal,
> - " Length %d, src offset
> %d, return offset "
> - "%d\n", hdr->msg.get.length,
> + " Length %u, src offset
> %u, return offset "
> + "%u, data length %u\n",
> hdr->msg.get.length,
> hdr->msg.get.src_offset,
> - hdr->msg.get.return_offset);
> + hdr->msg.get.return_offset,
> + hdr->msg.get.data_req_length);
> }
>
> if (hdr->type == PTL_MSG_ACK) {
> @@ -848,7 +847,8 @@ int do_PtlGet(nal_cb_t * nal, void *priv
> hdr.src_pid = ni->pid;
> hdr.msg.get.ptl_index = args->portal_in;
> hdr.msg.get.match_bits = args->match_bits_in;
> - hdr.msg.get.length = md->length;
> + hdr.msg.get.length = 0;
> + hdr.msg.get.data_req_length = md->length;
> hdr.msg.get.src_offset = args->offset_in;
> hdr.msg.get.return_offset = md->offset;
> #warning FIXME: convert this to proper cookie checking of handle
> diff -u -p -r1.6 debug.c
> --- linux/p3mod/debug.c 9 Jan 2003 01:03:36 -0000 1.6
> +++ linux/p3mod/debug.c 1 Feb 2003 00:16:42 -0000
> @@ -173,7 +173,7 @@ void p3_debug_proc(char **pb_ptr, char *
> LPX64", len %d, src off %d,
> ret off %d\n",
> hdr->msg.get.ptl_index,
> hdr->msg.get.match_bits,
> - hdr->msg.get.length,
> + hdr->msg.get.data_req_length,
> hdr->msg.get.src_offset,
> hdr->msg.get.return_offset);
> } else if (hdr->type == PTL_MSG_REPLY) {
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
|
|
From: '<ad...@cl...> - 2003-02-01 00:33:08
|
On Jan 31, 2003 15:18 -0800, Monish Suvarna wrote:
> The changes with adding extra 4 byes before the plt_replt_t structure works.
> The server no longer drops the connection when it gets a ptl_reply. I am now
> fixing the ost_bulk_handler code to use the ptl_reply cookie instead of the
> xid to match up the bulk structures sent by the original ptl_put.
Monish,
I will commit the change to move the "length" field in the ptl_reply_t field
to the same offset as the other messages. This fixes the immediate concern
of working with your next release.
The following patch (which appears to work fine for me) moves our usage of
the "length" field in the ptl_get_t to data_get_length, and sets "length"
to zero to match other uses of that field as the extra request data.
Zach, Eric, does this look OK? Monish, since I'm not a Portals person and
I haven't done extensive testing of this patch (although I think any error
would be evident immediately), I'm hesitant to say that you should use this
change to implement PtlGet on the BA OSTs for this release until I get some
confirmation from Zach and/or Eric that it is correct.
Cheers, Andreas
==============================================================================
diff -u -p -r1.31 lib-types.h
--- include/portals/lib-types.h 13 Nov 2002 21:37:03 -0000 1.31
+++ include/portals/lib-types.h 1 Feb 2003 00:16:42 -0000
@@ -40,7 +40,10 @@ typedef enum {
/* Each of these structs should start with an odd number of
* __u32, or the compiler could add its own padding and confuse
- * everyone. */
+ * everyone.
+ *
+ * Also, "length" needs to be at offset 28 of each struct.
+ */
typedef struct ptl_ack {
ptl_size_t mlength;
ptl_handle_wire_t dst_wmd;
@@ -61,15 +64,17 @@ typedef struct ptl_get {
ptl_pt_index_t ptl_index;
ptl_handle_wire_t return_wmd;
ptl_match_bits_t match_bits;
- ptl_size_t length;
+ ptl_size_t length; /* length of extra message data */
ptl_size_t src_offset;
ptl_size_t return_offset;
+ ptl_size_t data_req_length; /* length of requested bulk data */
} ptl_get_t;
typedef struct ptl_reply {
- __u32 unused;
+ __u32 unused1; /* for 64-bit alignment */
ptl_handle_wire_t dst_wmd;
ptl_size_t dst_offset;
+ __u32 unused2; /* for "length" alignment */
ptl_size_t length;
} ptl_reply_t;
diff -u -p -r1.46 lib-move.c
--- lib/lib-move.c 20 Dec 2002 11:22:35 -0000 1.46
+++ lib/lib-move.c 1 Feb 2003 00:16:42 -0000
@@ -341,25 +341,26 @@ static int parse_get(nal_cb_t * nal, ptl
me = lib_find_me(nal, hdr->msg.get.ptl_index, PTL_MD_OP_GET,
hdr->src_nid, hdr->src_pid,
- hdr->msg.get.length, hdr->msg.get.src_offset,
- hdr->msg.get.match_bits,
- &mlength, &offset, &unlink);
+ hdr->msg.get.data_req_length, hdr->msg.get.src_offset,
+ hdr->msg.get.match_bits, &mlength, &offset, &unlink);
if (me == NULL) {
- CERROR("Dropping get from portal %d offset %d, length %d, "
- "match "LPX64", from "LPU64".%d\n", hdr->msg.get.ptl_index,
- hdr->msg.get.src_offset, hdr->msg.get.length,
+ CERROR("Dropping get from portal %u offset %u, data length %u, "
+ "match "LPX64", from "LPU64".%d\n",
+ hdr->msg.get.ptl_index, hdr->msg.get.src_offset,
+ hdr->msg.get.data_req_length,
hdr->msg.get.match_bits, hdr->src_nid, hdr->src_pid);
goto drop;
}
md = me->md;
ptl_md2handle(&handle, md, nal);
- CDEBUG(D_NET, "Incoming get index %d from "LPU64".%u of length %d/%d from "
- "md %ld [%d](%p:%d)... + %d\n", hdr->msg.get.ptl_index,
- hdr->src_nid, hdr->src_pid,
- mlength, hdr->msg.get.length, handle.handle_idx, md->md_niov,
- md->md_iov[0].mdiov_base, md->md_iov[0].mdiov_len, offset);
+ CDEBUG(D_NET, "Incoming get index %d from "LPU64".%u of "
+ "length %u/%u from md %lu [%d](%p:%d)... + %d\n",
+ hdr->msg.get.ptl_index, hdr->src_nid, hdr->src_pid,
+ mlength, hdr->msg.get.data_req_length, handle.handle_idx,
+ md->md_niov, md->md_iov[0].mdiov_base, md->md_iov[0].mdiov_len,
+ offset);
msg = get_new_msg (nal);
if (msg == NULL) {
@@ -375,7 +376,7 @@ static int parse_get(nal_cb_t * nal, ptl
msg->ev.initiator.pid = hdr->src_pid;
msg->ev.portal = hdr->msg.get.ptl_index;
msg->ev.match_bits = hdr->msg.get.match_bits;
- msg->ev.rlength = hdr->msg.get.length;
+ msg->ev.rlength = hdr->msg.get.data_req_length;
msg->ev.mlength = mlength;
msg->ev.offset = offset;
msg->ev.hdr_data = 0;
@@ -640,15 +641,16 @@ void print_hdr(nal_cb_t * nal, ptl_hdr_t
if (hdr->type == PTL_MSG_GET) {
nal->cb_printf(nal,
- " Ptl index %d, return md 0x%08llx, match "
- "bits 0x%0lx\n", hdr->msg.get.ptl_index,
- (long long)hdr->msg.get.return_wmd.handle_idx,
+ " Ptl index %u, return md "LPX64", match "
+ "bits 0x%0x\n", hdr->msg.get.ptl_index,
+ hdr->msg.get.return_wmd.handle_idx,
hdr->msg.get.match_bits);
nal->cb_printf(nal,
- " Length %d, src offset %d, return offset "
- "%d\n", hdr->msg.get.length,
+ " Length %u, src offset %u, return offset "
+ "%u, data length %u\n", hdr->msg.get.length,
hdr->msg.get.src_offset,
- hdr->msg.get.return_offset);
+ hdr->msg.get.return_offset,
+ hdr->msg.get.data_req_length);
}
if (hdr->type == PTL_MSG_ACK) {
@@ -848,7 +847,8 @@ int do_PtlGet(nal_cb_t * nal, void *priv
hdr.src_pid = ni->pid;
hdr.msg.get.ptl_index = args->portal_in;
hdr.msg.get.match_bits = args->match_bits_in;
- hdr.msg.get.length = md->length;
+ hdr.msg.get.length = 0;
+ hdr.msg.get.data_req_length = md->length;
hdr.msg.get.src_offset = args->offset_in;
hdr.msg.get.return_offset = md->offset;
#warning FIXME: convert this to proper cookie checking of handle
diff -u -p -r1.6 debug.c
--- linux/p3mod/debug.c 9 Jan 2003 01:03:36 -0000 1.6
+++ linux/p3mod/debug.c 1 Feb 2003 00:16:42 -0000
@@ -173,7 +173,7 @@ void p3_debug_proc(char **pb_ptr, char *
LPX64", len %d, src off %d, ret off %d\n",
hdr->msg.get.ptl_index,
hdr->msg.get.match_bits,
- hdr->msg.get.length,
+ hdr->msg.get.data_req_length,
hdr->msg.get.src_offset,
hdr->msg.get.return_offset);
} else if (hdr->type == PTL_MSG_REPLY) {
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
|
|
From: '<ad...@cl...> - 2003-01-31 19:52:01
|
Forwarding to sandiaportals-devel so they can chime in on the issue.
To recap - BlueArc uses a hardware TCP offload engine to parse the
portals header (ptl_hdr_t), and assumes that the "length" field in the
"msg" field is always at a constant offset so that the hardware can do
the message copy directly.
Lustre recently began using the PTL_MSG_GET and PTL_MSG_REPLY message types
(previously we were only using PTL_MSG_PUT and PTL_MSG_ACK messages, which
have the "length" field at the same offset 56 bytes into ptl_hdr_t).
The question is whether it would cause anyone grief if we changed the
alignment of the ptl_reply_t struct to have "length" at the same offset?
Secondly, ptl_get_t is currently using the "length" field to specify the
length of the data requested instead of the length of the message itself.
It is possible to add a new field to ptl_get_t which holds the data request
length without increasing the size of the ptl_hdr_t struct.
Comments? Preferences of putting "length" into the header instead of into
each of the messages?
Cheers, Andreas
On Jan 31, 2003 11:45 -0700, 'ad...@cl...' wrote:
> On Jan 31, 2003 12:20 -0500, Zach Brown wrote:
> > > For that matter, does the ptl_get_t actually need to send the length
> > > back to the client, since we assume? it is the same as the original
> > > length that was sent in the OST_WRITE in the first place. Sending it
> > > again is safer, I suppose, and doesn't really affect us negatively.
> >
> > its nice to be able to have a specific length also specify the source
> > buffers.. sure, in our protocol the match bits are enough to find the
> > source buffers all the time, but one can imagine a protocol with long
> > lived match bits and length attributes that really select the proper
> > source buffers.
> >
> > so we shouldn't cripple the concept of a get specifying length. but we
> > could move it to the last 4 bytes of the ptl_get structure and shuffle
> > all the other structures so that they have a 'length' member that lines
> > up with put's.
>
> Actually, I don't _think_ we need to do any shuffling of any of the
> structs, as they already all have "length" at a 28-byte offset. Hmm,
> except ptl_reply_t it seems (which has it as the 3rd parameter but at
> a 24-byte offset). Since the usage of ptl_reply_t is also new with the
> PtlGet change it should be possible to add a padding word before length
> to ensure length is also at the same offset there.
>
> Yeah, it's kind of yucky to have a common fixed-offset member in the
> middle of the struct... We _could_ move it into ptl_hdr_t and at the
> same time remove the need for a 32-bit member as the first element of
> "msg" for the next BA OST release, but that is a bit of a bigger change
> than I am comfortable making (I'm not a Portals person, Peter would know
> if that is OK).
>
> So, in the minimally intrusive case we have:
>
> typedef struct ptl_get {
> ptl_pt_index_t ptl_index;
> ptl_handle_wire_t return_wmd;
> ptl_match_bits_t match_bits;
> ptl_size_t length;
> ptl_size_t src_offset;
> ptl_size_t return_offset;
> ptl_size_t data_request_length;
> } ptl_get_t;
>
> typedef struct ptl_reply {
> __u32 unused1; /* to keep dst_wmd 64-bit aligned */
> ptl_handle_wire_t dst_wmd;
> ptl_size_t dst_offset;
> __u32 unused2; /* to keep "length" at same offset */
> ptl_size_t length;
> } ptl_reply_t;
>
> Monish, now is "length" the length of the whole message? Is it 72 bytes in
> all cases, or is that only true for e.g. ptl_put_t and it is less for
> ptl_reply_t?
>
> The "cleaner" but more intrusive approach moves "length" out of the union
> and puts it into ptl_hdr_t, and requires that the other fields in the union
> struct be shuffled to remove the 32-bit struct at the beginning, so the 64-bit
> elements are 64-bit aligned:
>
> typedef struct {
> ptl_nid_t dest_nid;
> ptl_nid_t src_nid;
> ptl_pid_t dest_pid;
> ptl_pid_t src_pid;
> __u32 type; /* ptl_msg_type_t */
> ptl_size_t length;
> union { ... } msg;
> } ptl_hdr_t;
>
> and then we have "ptl_handle_wire_t *_wmd" as the first struct in the union
> and can remove the "unused" field in ptl_reply.
>
> > it would make the protocol look inconsistent, though.. "why isn't the
> > length up in the shared part of the ptl_hdr_t?" "why, if length is
> > always 0 for a get, can't it be derived from the type?" etc.
>
> Part of the problem with having length be dependent upon type is that it
> requires a table of all possible message types (currently only 4, granted)
> to look up the length, instead of extracting it from the header directly.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
|
|
From: Parag S. <pa...@cr...> - 2003-01-03 20:53:56
|
Hi, I recently downloaded the Portals source and compiled with 2.4.19-ac4 kernel. I encountered unresolved symbols when trying to load the Portals module. Is there a way to know the Linux kernel version with which the downloaded Portals source is guaranteed to work? I see that the patches in the patches directory are for 2.4.18 kernel. Are there patches available for 2.4.19? If not is something in the works? Thanks Parag |
|
From: Kevin P. <ped...@en...> - 2003-01-02 17:11:05
|
|
From: 'Peter B. <br...@cl...> - 2002-12-19 17:43:43
|
David, I'm interested in doing this, but it will likely only happen in Q2 of 2003 after we have completed Lustre 1.0. I will get in touch with you then. - peter - On Thu, Dec 19, 2002 at 09:21:59AM -0600, David Du wrote: > Dear Peter: > I would like to invite you to visit us and give a talk at Digital > Technology Center Seminar. We have established an Intelligent Storage > Consortium to work on OSD based intelligent storage devices as well as > systems. As Jim indicated StorageTek is the first industrial member of > the Consortium. Currently Seagate, Intel, IBM, Cisco and EMC have > expressed some interests in the Consortium. We are, of course, very > interested in Lustre Project and have installed Lustre software here. > Our semester is winding down now and the next semester will start on > January 21st. Any day after January 21st will be good for us. Please let > me know your availability. I am looking forward to seeing you in > Minnesota. > > David Du > Professor > Computer Science and Engineering > University of Minnesota > 612-6252560 > du...@cs... > > > -----Original Message----- > From: Jim Hughes [mailto:ji...@ne...] > Sent: Saturday, December 14, 2002 12:49 AM > To: Peter Braam > Cc: Ravi Kavuri; David Du > Subject: Re: Peter --- more stuff.. > > > > The University of MN Digital Technology Center Intelligent Storage > Consortium (which STK is the first member of with more coming soon) is > focusing on OSD and your code in particular and has a significant number > of students working on this effort. > > The DTC sponsors seminars and Prof. David Du (lead of Intelligent > Storage Consortium) expressed interest in trying to get you to come to > MN to give a talk. > > I fully expect that the STK team (10 seasoned engineers) and the DTC > team (6 RA) are going to be working in this area and collaboration may > be the right thing to do. > > Given this introduction, I believe I can let David take it from here. > > Thanks! > > jim > > > > On Tue, 2002-12-10 at 02:25, Peter Braam wrote: > > Jim, > > > > Send the people to the Lustre course - they will learn a ton and it > > will be fun. We do all development on laptops under user mode Linux > > (then you can run the kernel in GDB). > > > > The Lustre HOWTO should be in reasonable shape to build some code and > > we have a pre-patched kernel on the ftp site nowadays. > > > > - Peter - > > > > > > On Mon, Dec 09, 2002 at 09:19:17PM -0600, Jim Hughes wrote: > > > I need to recreate your environment to parallel what you are doing. > > > > > > What do I need so that I am closest to your development environment? > > How > > > many boxes, network, kernel, etc, > > > > > > Should I be able to do this on my own? Send people to your place, > > > etc. > > > > > > > > Thanks . > > > > > > jim > > - Peter - > -- > Jim Hughes <ji...@ne...> - Peter - |
|
From: Gerard B. <gb...@cs...> - 2002-12-12 18:29:12
|
Hi Kevin, It is in my interest as well that the code works on SMP machines. If I can help you with something then let me know. ~G=E9rard On Thu, 12 Dec 2002, Kevin Pedretti wrote: > I haven't tested the SourceForge versoin of the RTSCTS NAL yet on SMP > boxes. I know that there are problems in p3mod dealing with the > cb_table. I'm also not sure if the configure scripts setup the right > flags when compiling the RTSCTS NAL. I will look into it and get back > to you. Unfortunately, I don't have any SMP boxes to test on. We > should be able to get it going though in a few iterations if your > willing to try out my changes. > > > Kevin |
|
From: Kevin P. <kt...@sa...> - 2002-12-12 18:15:16
|
I haven't tested the SourceForge versoin of the RTSCTS NAL yet on SMP=20 boxes. I know that there are problems in p3mod dealing with the=20 cb_table. I'm also not sure if the configure scripts setup the right=20 flags when compiling the RTSCTS NAL. I will look into it and get back=20 to you. Unfortunately, I don't have any SMP boxes to test on. We=20 should be able to get it going though in a few iterations if your=20 willing to try out my changes. =20 Kevin Gerard Basler wrote: >Is Portals SMP capable? > >I'm using a dual P4 / Redhat 8.0 / 2.4.18-14 > >If I compile a SMP Kernel then I get the following error when I start >Portals over the rtscts nal (via etc/portals.eth start 0 eth0): > >'test_kmem_cache_validate: FAIL: freed pointer validates' > >The 'hello_world_svr' program makes my machine crash if I try to run the >server and the client program on different nodes. It works if I start bo= th >on the same node. > >If I compile a Kernel without SMP support then the >'test_kmem_cache_validate' error message doesn't appear and the >hello_world program runs on two nodes. > >I made the same experience with an unpatched 2.4.18 Kernel from >kernel.org. > >Please help! > >~G=E9rard > > > >------------------------------------------------------- >This sf.net email is sponsored by: >With Great Power, Comes Great Responsibility=20 >Learn to use your power at OSDN's High Performance Computing Channel >http://hpc.devchannel.org/ >_______________________________________________ >Sandiaportals-devel mailing list >San...@li... >https://lists.sourceforge.net/lists/listinfo/sandiaportals-devel > =20 > |
|
From: Kevin P. <kt...@sa...> - 2002-12-12 18:11:35
|
Hi Gerard,
The RTSCTS NAL currently relies on the system_call_table being=20
exported so that it can provide an ioctl interface to mlockall(). This=20
is a hack, but there was a good reason for it. We have supported in the=20
past compiling Linux Cplant applications on True64 using some sort of=20
compatibility library (I don't know the details). Evidently, True64=20
doesn't have the equivalant of a mlockall() system call and a stub isn't=20
in the compatiblity library either. Since an app needs to lock all of=20
its memory before using the Portals RTSCTS nal, there needed to be some=20
way to call the Linux mlockall system call. That way was through an=20
ioctl in p3mod. The best and most portable way to do it IMO would have=20
been to call do_mlockall() directly. Unfortunately, that isn't exported=20
to modules either.
Ron or Rolf -- are we supporting the True64 compilers anymore on Cplant?=20
If not, I'll take out the mlockall ioctl from p3mod (in Source Forge).
Kevin
switch (myrnal_forward.p3cmd) {
case PTL_MLOCKALL:
#ifdef __ia64__
/* TODO: this looks hackish. is there a better=20
way?? */ rc =3D __ia64_syscall((int) sys_arg, 0, 0=
,=20
0, 0,
__NR_mlockall);
#else
rc =3D syscalls[__NR_mlockall] (sys_arg);
#endif
nalstat.ioctlMlockall++;
Gerard Basler wrote:
>Hey!
>
>I just wanted to mention that the Redhat 8.0 / 2.4.18-14 patch is not
>complete.
>In order to compile the rtscts NAL you should add the line:
>EXPORT(sys_call_table);
>in kernel/ksyms.c because the Redhat people removed it.
>
>~G=E9rard
>
>
>
>-------------------------------------------------------
>This sf.net email is sponsored by:
>With Great Power, Comes Great Responsibility=20
>Learn to use your power at OSDN's High Performance Computing Channel
>http://hpc.devchannel.org/
>_______________________________________________
>Sandiaportals-devel mailing list
>San...@li...
>https://lists.sourceforge.net/lists/listinfo/sandiaportals-devel
> =20
>
|
|
From: Gerard B. <gb...@cs...> - 2002-12-11 19:13:53
|
Is Portals SMP capable? I'm using a dual P4 / Redhat 8.0 / 2.4.18-14 If I compile a SMP Kernel then I get the following error when I start Portals over the rtscts nal (via etc/portals.eth start 0 eth0): 'test_kmem_cache_validate: FAIL: freed pointer validates' The 'hello_world_svr' program makes my machine crash if I try to run the server and the client program on different nodes. It works if I start both on the same node. If I compile a Kernel without SMP support then the 'test_kmem_cache_validate' error message doesn't appear and the hello_world program runs on two nodes. I made the same experience with an unpatched 2.4.18 Kernel from kernel.org. Please help! ~G=E9rard |
|
From: Gerard B. <gb...@cs...> - 2002-12-11 18:44:23
|
Hey! I just wanted to mention that the Redhat 8.0 / 2.4.18-14 patch is not complete. In order to compile the rtscts NAL you should add the line: EXPORT(sys_call_table); in kernel/ksyms.c because the Redhat people removed it. ~G=E9rard |