[Geotools-devel] ArcSDE status update

[Gabriel R. <gr...@op...>]

Hi,

here's the deal with arcsde after a week of hacking:

Saul, if you don't want to go through the whole content check item 4- at the 
end.

- command queue strategy for connection handling successfully implemented for 
vector data. I even set the connection's concurrency policy to 
SE_ONE_THREAD_POLICY so it fails fast if tried to be used from another thread 
than the one it was created with. This helped to spot a couple places where 
command should have been used but were not being.

- all vector data tests pass, except those three that were not relevant to 
this work, that still need to be fixed. There are a difference on the level 
of transaction isolation ArcSDE uses depending on if the backend is mssql or 
not, see <http://jira.codehaus.org/browse/GEOT-1831>

- By using SessionTransactionState, I've turned Session into an interface, and 
am providing a decorator for non auto commit transactions that just avoids 
the session to be returned to the pool if its on transaction. This provides 
for a lot of code clean up, since now all the code can follow the same usage 
pattern when it needs a Session: acquire/use/dispose. The actual session will 
take care of returning itself to the pool or not but the client code does not 
care.

- Sessions are no longer kept as instance variables except in 
ArcSDEAttributeReader and ArcSDEFeatureWriter, the only two that need to hold 
on a Session until they're exhausted/closed. The only one that knows about 
the pool is ArcSDEDataStore. Sessions are always dynamically acquired from 
the pool in a per transaction basis. Read only code leads to 
getSession(Transaction.AUTO_COMMIT).

- Care has been taken for a single connection to be enough for any single 
common usage. Moreover, thanks to the command queue stuff, sessions (and 
hence SeConnections) have a much higher probability of serving multiple 
requests "at a time", since the "execution units" needing a session are much 
finely grained.

All in all, performance through WMS seems good, even using a large OL preview 
with tiles and maxConnections set to 2 with a quite large layer.

What's still pending:
 1- fixing the polygon with hole creation bug
 2- testing hard with versioned tables, a user reported a lot of deadlocks in 
the database
 3- backporting to 2.4.x
 4- and very importantly: go through all the gce code and fix/adapt to 
commands.
 5- Test ArcSDEConnectionReference for the single connection configuration. 
Jody I would try caching the connection once and just overriding getSession() 
to return it over and over.

Saul, I would be willing to help on this last one, just need to know how to 
run the tests, if I need to set up something special or the tests create 
their own data, etc

Best regards,

Gabriel

On Wednesday 21 May 2008 09:58:35 pm Gabriel Roldán wrote:
> On Wednesday 21 May 2008 09:39:34 pm Jody Garnett wrote:
> > Gabriel Roldán wrote:
> > > Hey Jody,
> > >
> > > so this is what I did:
> > > - got rid of connection.commit/rollback in ArcTransactionState
> > > - added ArcSDEDataStore.getSession(Transaction). For read only use just
> > > call it with Transaction.AUTO_COMMIT, and that's what FeatureSource
> > > does, whilst FetureStore uses the transaction it was set with
> >
> > Sounds good.
> >
> > > There's something strange with the command execution though, may be you
> > > have some time to test it, since its almost bed time for me now.
> >
> > I will update now.
> >
> > > The strange is that running ArcSDEFeatureStoreTest against my server
> > > (which is close so it runs fast) I get a lot of "THIS CONNECTION IS
> > > BEING USED AT THE MOMENT BY ANOTHER THREAD" errors, while executing the
> > > same test against your server ( which is far away so most of the time
> > > is spent in network io) only the last two test cases fail.
> >
> > That is confusing to me; I don't understand it either.
> >
> > > My hope is that if you run it you'll get a lot of broken cases as your
> > > server is fast for you, and you may be able to get a clue why some
> > > command execution is breaking that way...
> > >
> > > Another thing: using issueReadCommand like you did in fetchSchema
> > > implies needing two connections at a given time for a single thing.
> > > I've commented it by now, since I guess when using
> > > ArcSDEConnectionReference you just want to return the same session over
> > > and over, so there shouldn't be a need to do anything special? or
> > > rather I'm not
> > > understanding why you want the issueReadOnly method. The idea being to
> > > pick up any available connection, but you have only one, so why not
> > > just overriding getSession to return the same one? not sure..
> >
> > This is why I need the code review:
> > - ArcSDEConnectionReference.issueReadCommand will try and reuse the
> > "last" Session (ie the only session), if it is not available it will
> > grab a Session from the pool like normal
> > -ArcSDEConnectionPool.issueReadCommand will grab a session from the pool
> > like normal - I did not want to change any functionality on you until
> > you had done a code review. So this method behaves exactly the same as
> > before (ie we always needed 2 connections in the past)
>
> it did in past, then I got with working with a single one, then introducing
> this required two again. right now you can run the ArcSDEFeatureSourceTest
> with a single connection, though I won't do any statement that a single
> connection is enough for all and every code path, yet..
> The big difference is that now we have a much finer granularity on what an
> execution unit is, connection wise, so there're higher probabilities a
> single connection serves more read only operations.
>
> > > Also, you'll note a lot of verbosity on the console, it was me freaking
> > > out while trying to understand about what's causing the connection to
> > > complain about threading.
> >
> > So where does the exception about threading appear?
>
> ArcSDEFeatureStoreTest.testUpdateAutocommit first, then everything gets
> screwed up.
>
> but I couldnt understand for the last long hours why the heck the command
> sent at ArcSdeFeatureWriter.getUpdatableColumnNames() (line 694) leads to
> this error and not any other command.
>
> any clue will be much appreciated, I seem too burned right now for a single
> day, and am starting to stop beleaving in resolving hard problems in dreams
> or shower...  :)
>
> cheers,
>
> Gabriel
>
> > Jody
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Geotools-devel mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
> !DSPAM:4045,48347f5e26452090977483!