#112 SRPC protocol version negotiation
Currently, SRPC uses a very simplistic method for
handling different protocol versions. The client tells
the server which version of the protocol it wants to
use, and the server either handles it or rejects it.
If the server rejects it (i.e. the client asked for a
newer version of the protocol than the server knows how
to handle), it's a fatal error for the client.
This means that running servers must always be upgraded
before clients. This can be somewhat problematic for
upgrading production environments, especially because
in some cases the client/server relationship is
counter-intuitive. Two examples are:
- The repository server acts as client to the
evaluator's ToolDirectoryServer for getting volatile
directory contents
- When replicating, the destination repository acts as
a client to the source repository
While it would complicate the client-side code
somewhat, it would be nice if the protocol negotiation
could be bi-directional, allowing newer clients to talk
to older servers. This would probably mean that the
client would send a range of protocol versions it is
capable of handling, and the server would respond with
the protocol version to use.
Currently, the protocol version is sent along with the
procedure number at the start of each call. This
negotiation would require an additional response from
the server, and thus would have to happen before the
first procedure call. Unfortunately, this changes the
basic SRPC protocol, which is even more disruptive.
(See the use of "SRPC_VERSION" in src/SRPC_impl.C in
/vesta/vestasys.org/vesta/srpc.)
It's worth mentioning that generally we don't change
the protocol version number for adding new procedures.
We only change it when changing the calling/return
conventions of a particular call.
I know this sounds like a lot of work for something of
questionable value, but having a lack of flexibility to
make protocol changes tends to lead to poor choices
(like adding a new procedure rather than changing an
existing one).
Logged In: YES
user_id=291223
Another possibility would be to have each RPC have a
seperate version number. This could be implimented with a
fairly minor change to the code since every RPC already has
the interface version as a parameter. I've accidentially
written something like this in
http://pub.vestasys.org/cgi-bin/vestaweb?path=/vesta/vestasys.org/vesta/repos/checkout/75.venier_vestasys.org.1/5
when I was trying to change the VestaSourceSRPC::version and
yet still allow replication from other repositories.
There might be an interesting issue in the
VestaSourceReceptionist() which currently checks to make
sure the SRPC version is inside the supported range of
versions. This would have to change to a per-RPC check, or
simply the lowest and highest of all the RPC versions.
Logged In: YES
user_id=95236
Would it be good enough (at least in most cases) to add code
on the client side that retries with a lower protocol
version number if it gets an error from the server with the
number it wants to use?
Also note that the protocol version number doesn't have to
change often. You can add new RPCs to an SRPC protocol
without bumping the version number, though of course in that
case a client that gets an error from using an unknown RPC
has to retry with older RPCs that do the same job in another
way.
The protocol version really only has to change when you want
to change existing RPCs to have different parameters or
different semantics, and you aren't happy to do that by
adding new RPCs with the new semantics and deprecating the
old ones.
Logged In: YES
user_id=304837
> There might be an interesting issue in the
> VestaSourceReceptionist() which currently checks to make
> sure the SRPC version is inside the supported range of
> versions.
I think it's actually more interesting on the client end.
Since the protocol version is currently a unilateral
declaration by the client, you'd want to have the client
send the lowest possible protocol number that's valid for
the RPC in question.
> Would it be good enough (at least in most cases) to add
> code on the client side that retries with a lower protocol
> version number if it gets an error from the server with
> the number it wants to use?
These days we're dealing with very remote servers (the
opposite side of the globe, with round-trip times up above
500ms). We're trying to reduce the number of network
round-trips in general, so I would hesitate to solve this
problem in a way which would potentially add many additional
round-trips. Of course a single client could probably keep
it down to a small number of additional failed round-trips,
as long as it remembered the need to lower the protocol for
all subsequent RPCs. However that doesn't seem like any
less work than implementing real protocol version
negotiation, and it doesn't help short-lived clients (like
the repos_ui tools) very much.
I can't really think of an argument against per-RPC protocol
numbers, and there are some attractive things about it. I
think we could even hide most the complexity of it inside
the SRPC library.
How might that work with protocol negotiation? The client
could send a range of protocol versions for each procedure
it knows about, and the server would respond with the
protocol version to use for each procedure. Ideally that
information would be stored in the SRPC object allowing the
client code to retrieve it when needed. Perhaps even
SRPC::start_call would just take the procedure number and
return the protocol version.
Logged In: YES
user_id=95236
I suggested the retry idea since it will work (unless I'm
missing something) with existing servers -- there's no need
for a flag-day protocol change. However it sounds like
there are enough drawbacks that you'll want to opt for the
bigger change.
Logged In: YES
user_id=304837
Originator: YES
We held a detailed discussion on how to implement this last
Friday. Here are some decisions we made:
- We'll do protocol version negotiation in the intial
exchange. The client's "hello" message will include a range
of protocol verisons that the client knows. The server will
respond with a specific protocol version number to use
(which must be in the range that the client specified).
This adds a single round-trip during connection start-up.
If the server doesn't suppot any of the protocol versions
that the client asked for, it will respond with an error.
- Contrary to the approach Scott mentioned, we're going to
stick with just a single protocol version. (We considered
having a major protocol version number and then a minor
protocol version bump for each procedure, but that would
require an additional round-trips for each procedure with a
change over the major protocol version and doesn't seem to
have any added benefits.)
- Contrary to the way we have operated in the past, we will
increment the protocol version number when we add new
procedures. This will allow the client to know whether new
procedures are available from the server without a
round-trip. (This avoids the kind of kludge we used when we
added the "read whole file compressed" procedure to the
repository or retrying with a different procedure when the
preferred procedure fails.)
- Currently the client waits until the first RPC to send its
"hello" message. Since it will now include protocol version
negotiation it will need to be done sooner (i.e. on
construction of the SRPC instance). We can't wait until the
first item code is sent for starting a procedure, as we may
need to check the negotiated protocol version to determine
if the procedure is available from the server.
Here are some specifics on how different parts of the code
will change:
- SRPC instances will store the negotiated protocol version
in a member variable and provide a function to retrieve it.
- SRPC::start_call will no longer have the intf_version
argument. (This means changing every start_call in all the
client-side code.)
- SRPC constructors will need to change. Here are the new
ones:
- caller:
// Opens connection to specified host:port
SRPC(const Text &port, const Text &host,
uint32 min_intf_vers, uint32 max_intf_vers);
// "connected_sock" is an already connected socket
SRPC(TCP_sock *connected_sock,
uint32 min_intf_vers, uint32 max_intf_vers);
- callee:
// Starts listening on "port". "select_intf_vers" is
// called to pick the protocol version to use.
SRPC(const Text &port,
uint32 (*select_intf_vers)(uint32, uint32));
// "connected_sock" is an already connected socket.
SRPC(TCP_sock *connected_sock,
uint32 (*select_intf_vers)(uint32, uint32));
// "listener_sock" is a listening socket which can be
// used to accept connections. Maybe we can get rid of
// this one if it's not used any more? If not we need
// some way to differentiate it from the connected
// socket case (e.g. add a "bool listener" argument).
SRPC(TCP_sock *listener_sock,
uint32 (*select_intf_vers)(uint32, uint32));
- LimService will need to be changed to use the new
constructors for SRPC.
- LimService::Handler::call can keep the same parameters
(i.e. including the protocol version in the arguments), so
the application-level server side code shouldn't need to be
changed.
- The MultiSRPC constructors should take the protocol
version range (uint32 min_intf_vers, uint32 max_intf_vers)
as parameters. MultiSRPC::Start doesn't need to change.
- After calling MultiSRPC::Start, the client should get the
negotiated protocol version from the SRPC instance before
making a call.
- We should support older clients without protocol version
negotiation:
- The string following the "hello" item code can be used
to distinguish between clients with protocol negotiation
and those without protocol negotiation. We'll need to
remember this in a member variable.
- If we're not doing protocol version negotiation, each
RPC will come with a protocol version number.
- SRPC::await_call still needs the "intf_version" argument
to support old clients. With new clients, we'll set it to
the negotiated value.
- The server SRPC_impl::read_item_code should set the
"version_sent" member variable to true when it sends the
protocol version and the client SRPC_impl::send_item_code
should set the "version_checked" member variable to true
when it recevies the protocol version so that we don't try
to do bi-directional negotiation.