geotools-devel

[Geotools-devel] ArcSDE Single Session mode

[Jody G. <jga...@re...>]

Hi Gabriel; I see you put the command queue in - nice work.

I have broken out the Transaction.State use into two:
- ArcTransactionState has the feature event notification and versioned 
flag and is stored against the DataStore
- SessionTransactionState holds the Session and is stored against the 
connectionPool

ArcSDEConnectionPool has the following methods:
- getSession(); used to lease a Session from the pool
- getSession( Transaction ); used to access the Session for the provided 
Transaction
- issueReadOnlyCommand( Command )

The last method is why I signed up for this work; the 
ArcSDEConnectionReference subclass of ConnectionPool remembers the 
single Session in use and will use it to issue read-only commands. If 
you can review; I have changed a single bit of functionality over to use 
issueReadOnlyCommand; the ArcSDEAdapter.fetSchema method can serve as an 
example of what needs to be done.

Jody

Re: [Geotools-devel] ArcSDE Single Session mode

[Jody G. <jga...@re...>]

Gabriel I ended up modifying your Session contact with the taskExecutor; 
I needed to store Thread being used so I could detect command recursion.

Cheers,
Jody

> Hi Gabriel; I see you put the command queue in - nice work.
>
> I have broken out the Transaction.State use into two:
> - ArcTransactionState has the feature event notification and versioned 
> flag and is stored against the DataStore
> - SessionTransactionState holds the Session and is stored against the 
> connectionPool
>
> ArcSDEConnectionPool has the following methods:
> - getSession(); used to lease a Session from the pool
> - getSession( Transaction ); used to access the Session for the provided 
> Transaction
> - issueReadOnlyCommand( Command )
>
> The last method is why I signed up for this work; the 
> ArcSDEConnectionReference subclass of ConnectionPool remembers the 
> single Session in use and will use it to issue read-only commands. If 
> you can review; I have changed a single bit of functionality over to use 
> issueReadOnlyCommand; the ArcSDEAdapter.fetSchema method can serve as an 
> example of what needs to be done.
>
> Jody
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft 
> Defy all challenges. Microsoft(R) Visual Studio 2008. 
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Geotools-devel mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>   

Re: [Geotools-devel] ArcSDE Single Session mode

[Gabriel R. <gr...@op...>]

On Wednesday 21 May 2008 06:51:31 am Jody Garnett wrote:
> Gabriel I ended up modifying your Session contact with the taskExecutor;
> I needed to store Thread being used so I could detect command recursion.
yeah I saw it and liked it.
I've been trying to avoid command recursion explicitly but like better this 
approach, so I changed back all the static issueXXX utility methods by 
instance methods that just issue a command and let the issue(Command) method 
decide whether it needs to be scheduled or not. Good work

Gabriel
>
> Cheers,
> Jody
>
> > Hi Gabriel; I see you put the command queue in - nice work.
> >
> > I have broken out the Transaction.State use into two:
> > - ArcTransactionState has the feature event notification and versioned
> > flag and is stored against the DataStore
> > - SessionTransactionState holds the Session and is stored against the
> > connectionPool
> >
> > ArcSDEConnectionPool has the following methods:
> > - getSession(); used to lease a Session from the pool
> > - getSession( Transaction ); used to access the Session for the provided
> > Transaction
> > - issueReadOnlyCommand( Command )
> >
> > The last method is why I signed up for this work; the
> > ArcSDEConnectionReference subclass of ConnectionPool remembers the
> > single Session in use and will use it to issue read-only commands. If
> > you can review; I have changed a single bit of functionality over to use
> > issueReadOnlyCommand; the ArcSDEAdapter.fetSchema method can serve as an
> > example of what needs to be done.
> >
> > Jody
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Microsoft
> > Defy all challenges. Microsoft(R) Visual Studio 2008.
> > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > _______________________________________________
> > Geotools-devel mailing list
> > Geo...@li...
> > https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
> !DSPAM:4045,4833aa52303071637810514!

Re: [Geotools-devel] ArcSDE Single Session mode

[Gabriel R. <gr...@op...>]

Okay I took my time to review... yet I'm still confused.

besides getting a lot of test breakages due to NPE when asking for the Session 
from the state, my concerns are as follow:

I understand the responsibility breakdown you made between ArcTransactionState 
and SessionTransactionState, but right now we have:
- both Transaction.State implementations commiting/rolling back the connection
- The transaction commit and the version handling shall be handled atomically. 
Having them handled at different places (the two states) gives us no way of 
knowing the order they will be executed in, they'll be called by 
Transaction.commit

Still, ArcTransactionState having too much responsibility is annoying. For 
instance:
- commiting/rolling back the transaction over the Session
- issueing edit events
- handling version states

but I don't see a clear escape route.

What I do see is access to Session and State quite convoluted, so my plan is 
as follows:
- only ArcSDEDataStore knows of ArcSDEConnectionPool, holds it as an instance 
variable
- Add ArcSDEDataStore.getSession(Transaction) as the only way of getting at a 
session from the objects it creates (FeatureSource and store)
- Short lived objects (ArcSDEAttributeReader and ArcSDEFeatureWriter) receive 
the Session they work on as well as the Transaction. When they're done (ie 
close()) calls ArcSDEDataStore.dispose(Session, Transaction), which knows 
whether to return the session to the pool or not.

This is meant to avoid some logic duplication, hiding ArcTransactionState and 
avoid ArcTransactionState from creating itself.

I may be missing something though, but all in all it seems to me those actions 
will provide for some more clean up.

thoughts?

Cheers,

Gabriel

On Wednesday 21 May 2008 06:19:05 am Jody Garnett wrote:
> Hi Gabriel; I see you put the command queue in - nice work.
>
> I have broken out the Transaction.State use into two:
> - ArcTransactionState has the feature event notification and versioned
> flag and is stored against the DataStore
> - SessionTransactionState holds the Session and is stored against the
> connectionPool
>
> ArcSDEConnectionPool has the following methods:
> - getSession(); used to lease a Session from the pool
> - getSession( Transaction ); used to access the Session for the provided
> Transaction
> - issueReadOnlyCommand( Command )
>
> The last method is why I signed up for this work; the
> ArcSDEConnectionReference subclass of ConnectionPool remembers the
> single Session in use and will use it to issue read-only commands. If
> you can review; I have changed a single bit of functionality over to use
> issueReadOnlyCommand; the ArcSDEAdapter.fetSchema method can serve as an
> example of what needs to be done.
>
> Jody
>
> !DSPAM:4045,4833a2ba293332090977483!

Re: [Geotools-devel] ArcSDE Single Session mode

[Jody G. <jga...@re...>]

Gabriel Roldán wrote:
> Okay I took my time to review... yet I'm still confused.
>
> besides getting a lot of test breakages due to NPE when asking for the Session 
> from the state, my concerns are as follow:
>
> I understand the responsibility breakdown you made between ArcTransactionState 
> and SessionTransactionState, but right now we have:
> - both Transaction.State implementations commiting/rolling back the connection
> - The transaction commit and the version handling shall be handled atomically. 
> Having them handled at different places (the two states) gives us no way of 
> knowing the order they will be executed in, they'll be called by 
> Transaction.commit
>   
Only the SessionTransactionState should be handling commit/rollback, 
lets fix this. It seems I did not get the responsibilities correct.
> Still, ArcTransactionState having too much responsibility is annoying. For 
> instance:
> x commiting/rolling back the transaction over the Session
> - issueing edit events
> - handling version states
>   
Can we just remove the one marked "x" above?
> but I don't see a clear escape route.
>
> What I do see is access to Session and State quite convoluted, so my plan is 
> as follows:
> - only ArcSDEDataStore knows of ArcSDEConnectionPool, holds it as an instance variable
>   
Okay; we also have some raster classes that use ArcSDEConnectionPool right?
> - Add ArcSDEDataStore.getSession(Transaction) as the only way of getting at a session from the objects it creates (FeatureSource and store)
>   
What about a FeatureSource on Transaction.AUTO_COMMIT ?
> - Short lived objects (ArcSDEAttributeReader and ArcSDEFeatureWriter) receive the Session they work on as well as the Transaction. When they're done (ie 
> close()) calls ArcSDEDataStore.dispose(Session, Transaction), which knows whether to return the session to the pool or not.
>   
We can also make a Session wrapper; that calls dispose( Session, 
Transaction ) so that ArcSDEAttributeReader / ArcSDEFeatureWriter do not 
have to be complicated further.
> This is meant to avoid some logic duplication, hiding ArcTransactionState and avoid ArcTransactionState from creating itself.
> I may be missing something though, but all in all it seems to me those actions will provide for some more clean up.
> thoughts?
>   
I am still wanting feedback on my problem / solution; namely can you 
review how getSchema( typeName is handled) - that is the point of the 
changes you saw yesterday :-)
Jody

[Geotools-devel] ArcSDE state of play [was Re: ArcSDE Single Session mode]

[Gabriel R. <gr...@op...>]

Hey Jody,

so this is what I did:
- got rid of connection.commit/rollback in ArcTransactionState
- added ArcSDEDataStore.getSession(Transaction). For read only use just call 
it with Transaction.AUTO_COMMIT, and that's what FeatureSource does, whilst 
FetureStore uses the transaction it was set with

There's something strange with the command execution though, may be you have 
some time to test it, since its almost bed time for me now.
The strange is that running ArcSDEFeatureStoreTest against my server (which is 
close so it runs fast) I get a lot of "THIS CONNECTION IS BEING USED AT THE 
MOMENT BY ANOTHER THREAD" errors, while executing the same test against your 
server ( which is far away so most of the time is spent in network io) only 
the last two test cases fail.

My hope is that if you run it you'll get a lot of broken cases as your server 
is fast for you, and you may be able to get a clue why some command execution 
is breaking that way...

Another thing: using issueReadCommand like you did in fetchSchema implies 
needing two connections at a given time for a single thing. I've commented it 
by now, since I guess when using ArcSDEConnectionReference you just want to 
return the same session over and over, so there shouldn't be a need to do 
anything special? or rather I'm not understanding why you want the 
issueReadOnly method. The idea being to pick up any available connection, but 
you have only one, so why not just overriding getSession to return the same 
one? not sure..

Also, you'll note a lot of verbosity on the console, it was me freaking out 
while trying to understand about what's causing the connection to complain 
about threading.

On Wednesday 21 May 2008 07:28:20 pm Jody Garnett wrote:
> Gabriel Roldán wrote:
> > Okay I took my time to review... yet I'm still confused.
> >
> > besides getting a lot of test breakages due to NPE when asking for the
> > Session from the state, my concerns are as follow:
> >
> > I understand the responsibility breakdown you made between
> > ArcTransactionState and SessionTransactionState, but right now we have:
> > - both Transaction.State implementations commiting/rolling back the
> > connection - The transaction commit and the version handling shall be
> > handled atomically. Having them handled at different places (the two
> > states) gives us no way of knowing the order they will be executed in,
> > they'll be called by Transaction.commit
>
> Only the SessionTransactionState should be handling commit/rollback,
> lets fix this. It seems I did not get the responsibilities correct.
>
> > Still, ArcTransactionState having too much responsibility is annoying.
> > For instance:
> > x commiting/rolling back the transaction over the Session
> > - issueing edit events
> > - handling version states
>
> Can we just remove the one marked "x" above?
>
> > but I don't see a clear escape route.
> >
> > What I do see is access to Session and State quite convoluted, so my plan
> > is as follows:
> > - only ArcSDEDataStore knows of ArcSDEConnectionPool, holds it as an
> > instance variable
>
> Okay; we also have some raster classes that use ArcSDEConnectionPool right?
>
> > - Add ArcSDEDataStore.getSession(Transaction) as the only way of getting
> > at a session from the objects it creates (FeatureSource and store)
>
> What about a FeatureSource on Transaction.AUTO_COMMIT ?
>
> > - Short lived objects (ArcSDEAttributeReader and ArcSDEFeatureWriter)
> > receive the Session they work on as well as the Transaction. When they're
> > done (ie close()) calls ArcSDEDataStore.dispose(Session, Transaction),
> > which knows whether to return the session to the pool or not.
>
> We can also make a Session wrapper; that calls dispose( Session,
> Transaction ) so that ArcSDEAttributeReader / ArcSDEFeatureWriter do not
> have to be complicated further.
>
> > This is meant to avoid some logic duplication, hiding ArcTransactionState
> > and avoid ArcTransactionState from creating itself. I may be missing
> > something though, but all in all it seems to me those actions will
> > provide for some more clean up. thoughts?
>
> I am still wanting feedback on my problem / solution; namely can you
> review how getSchema( typeName is handled) - that is the point of the
> changes you saw yesterday :-)
> Jody
>
> !DSPAM:4045,48345bb4257492090977483!

Re: [Geotools-devel] ArcSDE state of play [was Re: ArcSDE Single Session mode]

[Jody G. <jga...@re...>]

Gabriel Roldán wrote:
> Hey Jody,
>
> so this is what I did:
> - got rid of connection.commit/rollback in ArcTransactionState
> - added ArcSDEDataStore.getSession(Transaction). For read only use just call 
> it with Transaction.AUTO_COMMIT, and that's what FeatureSource does, whilst 
> FetureStore uses the transaction it was set with
>   
Sounds good.
> There's something strange with the command execution though, may be you have 
> some time to test it, since its almost bed time for me now.
>   
I will update now.
> The strange is that running ArcSDEFeatureStoreTest against my server (which is 
> close so it runs fast) I get a lot of "THIS CONNECTION IS BEING USED AT THE 
> MOMENT BY ANOTHER THREAD" errors, while executing the same test against your 
> server ( which is far away so most of the time is spent in network io) only 
> the last two test cases fail.
>   
That is confusing to me; I don't understand it either.
> My hope is that if you run it you'll get a lot of broken cases as your server 
> is fast for you, and you may be able to get a clue why some command execution 
> is breaking that way...
>
> Another thing: using issueReadCommand like you did in fetchSchema implies 
> needing two connections at a given time for a single thing. I've commented it 
> by now, since I guess when using ArcSDEConnectionReference you just want to 
> return the same session over and over, so there shouldn't be a need to do 
> anything special? or rather I'm not understanding why you want the 
> issueReadOnly method. The idea being to pick up any available connection, but 
> you have only one, so why not just overriding getSession to return the same 
> one? not sure..
>   
This is why I need the code review:
- ArcSDEConnectionReference.issueReadCommand will try and reuse the 
"last" Session (ie the only session), if it is not available it will 
grab a Session from the pool like normal
-ArcSDEConnectionPool.issueReadCommand will grab a session from the pool 
like normal - I did not want to change any functionality on you until 
you had done a code review. So this method behaves exactly the same as 
before (ie we always needed 2 connections in the past)
> Also, you'll note a lot of verbosity on the console, it was me freaking out while trying to understand about what's causing the connection to complain 
> about threading.
>   
So where does the exception about threading appear?
Jody

Re: [Geotools-devel] ArcSDE state of play [was Re: ArcSDE Single Session mode]

[Gabriel R. <gr...@op...>]

On Wednesday 21 May 2008 09:39:34 pm Jody Garnett wrote:
> Gabriel Roldán wrote:
> > Hey Jody,
> >
> > so this is what I did:
> > - got rid of connection.commit/rollback in ArcTransactionState
> > - added ArcSDEDataStore.getSession(Transaction). For read only use just
> > call it with Transaction.AUTO_COMMIT, and that's what FeatureSource does,
> > whilst FetureStore uses the transaction it was set with
>
> Sounds good.
>
> > There's something strange with the command execution though, may be you
> > have some time to test it, since its almost bed time for me now.
>
> I will update now.
>
> > The strange is that running ArcSDEFeatureStoreTest against my server
> > (which is close so it runs fast) I get a lot of "THIS CONNECTION IS BEING
> > USED AT THE MOMENT BY ANOTHER THREAD" errors, while executing the same
> > test against your server ( which is far away so most of the time is spent
> > in network io) only the last two test cases fail.
>
> That is confusing to me; I don't understand it either.
>
> > My hope is that if you run it you'll get a lot of broken cases as your
> > server is fast for you, and you may be able to get a clue why some
> > command execution is breaking that way...
> >
> > Another thing: using issueReadCommand like you did in fetchSchema implies
> > needing two connections at a given time for a single thing. I've
> > commented it by now, since I guess when using ArcSDEConnectionReference
> > you just want to return the same session over and over, so there
> > shouldn't be a need to do anything special? or rather I'm not
> > understanding why you want the issueReadOnly method. The idea being to
> > pick up any available connection, but you have only one, so why not just
> > overriding getSession to return the same one? not sure..
>
> This is why I need the code review:
> - ArcSDEConnectionReference.issueReadCommand will try and reuse the
> "last" Session (ie the only session), if it is not available it will
> grab a Session from the pool like normal
> -ArcSDEConnectionPool.issueReadCommand will grab a session from the pool
> like normal - I did not want to change any functionality on you until
> you had done a code review. So this method behaves exactly the same as
> before (ie we always needed 2 connections in the past)
it did in past, then I got with working with a single one, then introducing 
this required two again. right now you can run the ArcSDEFeatureSourceTest 
with a single connection, though I won't do any statement that a single 
connection is enough for all and every code path, yet..
The big difference is that now we have a much finer granularity on what an 
execution unit is, connection wise, so there're higher probabilities a single 
connection serves more read only operations.
>
> > Also, you'll note a lot of verbosity on the console, it was me freaking
> > out while trying to understand about what's causing the connection to
> > complain about threading.
>
> So where does the exception about threading appear?

ArcSDEFeatureStoreTest.testUpdateAutocommit first, then everything gets 
screwed up.

but I couldnt understand for the last long hours why the heck the command sent 
at ArcSdeFeatureWriter.getUpdatableColumnNames() (line 694) leads to this 
error and not any other command.

any clue will be much appreciated, I seem too burned right now for a single 
day, and am starting to stop beleaving in resolving hard problems in dreams 
or shower...  :)

cheers,

Gabriel

> Jody
>
> !DSPAM:4045,48347a7910434901796417!

[Geotools-devel] ArcSDE status update

[Gabriel R. <gr...@op...>]

Hi,

here's the deal with arcsde after a week of hacking:

Saul, if you don't want to go through the whole content check item 4- at the 
end.

- command queue strategy for connection handling successfully implemented for 
vector data. I even set the connection's concurrency policy to 
SE_ONE_THREAD_POLICY so it fails fast if tried to be used from another thread 
than the one it was created with. This helped to spot a couple places where 
command should have been used but were not being.

- all vector data tests pass, except those three that were not relevant to 
this work, that still need to be fixed. There are a difference on the level 
of transaction isolation ArcSDE uses depending on if the backend is mssql or 
not, see <http://jira.codehaus.org/browse/GEOT-1831>

- By using SessionTransactionState, I've turned Session into an interface, and 
am providing a decorator for non auto commit transactions that just avoids 
the session to be returned to the pool if its on transaction. This provides 
for a lot of code clean up, since now all the code can follow the same usage 
pattern when it needs a Session: acquire/use/dispose. The actual session will 
take care of returning itself to the pool or not but the client code does not 
care.

- Sessions are no longer kept as instance variables except in 
ArcSDEAttributeReader and ArcSDEFeatureWriter, the only two that need to hold 
on a Session until they're exhausted/closed. The only one that knows about 
the pool is ArcSDEDataStore. Sessions are always dynamically acquired from 
the pool in a per transaction basis. Read only code leads to 
getSession(Transaction.AUTO_COMMIT).

- Care has been taken for a single connection to be enough for any single 
common usage. Moreover, thanks to the command queue stuff, sessions (and 
hence SeConnections) have a much higher probability of serving multiple 
requests "at a time", since the "execution units" needing a session are much 
finely grained.

All in all, performance through WMS seems good, even using a large OL preview 
with tiles and maxConnections set to 2 with a quite large layer.

What's still pending:
 1- fixing the polygon with hole creation bug
 2- testing hard with versioned tables, a user reported a lot of deadlocks in 
the database
 3- backporting to 2.4.x
 4- and very importantly: go through all the gce code and fix/adapt to 
commands.
 5- Test ArcSDEConnectionReference for the single connection configuration. 
Jody I would try caching the connection once and just overriding getSession() 
to return it over and over.

Saul, I would be willing to help on this last one, just need to know how to 
run the tests, if I need to set up something special or the tests create 
their own data, etc

Best regards,

Gabriel

On Wednesday 21 May 2008 09:58:35 pm Gabriel Roldán wrote:
> On Wednesday 21 May 2008 09:39:34 pm Jody Garnett wrote:
> > Gabriel Roldán wrote:
> > > Hey Jody,
> > >
> > > so this is what I did:
> > > - got rid of connection.commit/rollback in ArcTransactionState
> > > - added ArcSDEDataStore.getSession(Transaction). For read only use just
> > > call it with Transaction.AUTO_COMMIT, and that's what FeatureSource
> > > does, whilst FetureStore uses the transaction it was set with
> >
> > Sounds good.
> >
> > > There's something strange with the command execution though, may be you
> > > have some time to test it, since its almost bed time for me now.
> >
> > I will update now.
> >
> > > The strange is that running ArcSDEFeatureStoreTest against my server
> > > (which is close so it runs fast) I get a lot of "THIS CONNECTION IS
> > > BEING USED AT THE MOMENT BY ANOTHER THREAD" errors, while executing the
> > > same test against your server ( which is far away so most of the time
> > > is spent in network io) only the last two test cases fail.
> >
> > That is confusing to me; I don't understand it either.
> >
> > > My hope is that if you run it you'll get a lot of broken cases as your
> > > server is fast for you, and you may be able to get a clue why some
> > > command execution is breaking that way...
> > >
> > > Another thing: using issueReadCommand like you did in fetchSchema
> > > implies needing two connections at a given time for a single thing.
> > > I've commented it by now, since I guess when using
> > > ArcSDEConnectionReference you just want to return the same session over
> > > and over, so there shouldn't be a need to do anything special? or
> > > rather I'm not
> > > understanding why you want the issueReadOnly method. The idea being to
> > > pick up any available connection, but you have only one, so why not
> > > just overriding getSession to return the same one? not sure..
> >
> > This is why I need the code review:
> > - ArcSDEConnectionReference.issueReadCommand will try and reuse the
> > "last" Session (ie the only session), if it is not available it will
> > grab a Session from the pool like normal
> > -ArcSDEConnectionPool.issueReadCommand will grab a session from the pool
> > like normal - I did not want to change any functionality on you until
> > you had done a code review. So this method behaves exactly the same as
> > before (ie we always needed 2 connections in the past)
>
> it did in past, then I got with working with a single one, then introducing
> this required two again. right now you can run the ArcSDEFeatureSourceTest
> with a single connection, though I won't do any statement that a single
> connection is enough for all and every code path, yet..
> The big difference is that now we have a much finer granularity on what an
> execution unit is, connection wise, so there're higher probabilities a
> single connection serves more read only operations.
>
> > > Also, you'll note a lot of verbosity on the console, it was me freaking
> > > out while trying to understand about what's causing the connection to
> > > complain about threading.
> >
> > So where does the exception about threading appear?
>
> ArcSDEFeatureStoreTest.testUpdateAutocommit first, then everything gets
> screwed up.
>
> but I couldnt understand for the last long hours why the heck the command
> sent at ArcSdeFeatureWriter.getUpdatableColumnNames() (line 694) leads to
> this error and not any other command.
>
> any clue will be much appreciated, I seem too burned right now for a single
> day, and am starting to stop beleaving in resolving hard problems in dreams
> or shower...  :)
>
> cheers,
>
> Gabriel
>
> > Jody
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Geotools-devel mailing list
> Geo...@li...
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
> !DSPAM:4045,48347f5e26452090977483!

[Geotools-devel] ArcSDE is now great

[Jody G. <jga...@re...>]

Hi Gabriel; just getting back to you so you know I am alive.

Changing over the the queue has made a *vast* difference in how well the 
datastore performs in uDig; everything is much better to work with and 
my customer is happy. I know we have some more issue to work through; 
but I wanted you to let you know the approach is a resounding success.

I am updating now and will try and respond to the items you listed below.
Jody

Gabriel Roldán wrote:
> Hi,
>
> here's the deal with arcsde after a week of hacking:
>
> Saul, if you don't want to go through the whole content check item 4- at the 
> end.
>
> - command queue strategy for connection handling successfully implemented for 
> vector data. I even set the connection's concurrency policy to 
> SE_ONE_THREAD_POLICY so it fails fast if tried to be used from another thread 
> than the one it was created with. This helped to spot a couple places where 
> command should have been used but were not being.
>
> - all vector data tests pass, except those three that were not relevant to 
> this work, that still need to be fixed. There are a difference on the level 
> of transaction isolation ArcSDE uses depending on if the backend is mssql or 
> not, see <http://jira.codehaus.org/browse/GEOT-1831>
>
> - By using SessionTransactionState, I've turned Session into an interface, and 
> am providing a decorator for non auto commit transactions that just avoids 
> the session to be returned to the pool if its on transaction. This provides 
> for a lot of code clean up, since now all the code can follow the same usage 
> pattern when it needs a Session: acquire/use/dispose. The actual session will 
> take care of returning itself to the pool or not but the client code does not 
> care.
>
> - Sessions are no longer kept as instance variables except in 
> ArcSDEAttributeReader and ArcSDEFeatureWriter, the only two that need to hold 
> on a Session until they're exhausted/closed. The only one that knows about 
> the pool is ArcSDEDataStore. Sessions are always dynamically acquired from 
> the pool in a per transaction basis. Read only code leads to 
> getSession(Transaction.AUTO_COMMIT).
>
> - Care has been taken for a single connection to be enough for any single 
> common usage. Moreover, thanks to the command queue stuff, sessions (and 
> hence SeConnections) have a much higher probability of serving multiple 
> requests "at a time", since the "execution units" needing a session are much 
> finely grained.
>
> All in all, performance through WMS seems good, even using a large OL preview 
> with tiles and maxConnections set to 2 with a quite large layer.
>
> What's still pending:
>  1- fixing the polygon with hole creation bug
>  2- testing hard with versioned tables, a user reported a lot of deadlocks in 
> the database
>  3- backporting to 2.4.x
>  4- and very importantly: go through all the gce code and fix/adapt to 
> commands.
>  5- Test ArcSDEConnectionReference for the single connection configuration. 
> Jody I would try caching the connection once and just overriding getSession() 
> to return it over and over.
>
> Saul, I would be willing to help on this last one, just need to know how to 
> run the tests, if I need to set up something special or the tests create 
> their own data, etc
>
> Best regards,
>
> Gabriel
>
> On Wednesday 21 May 2008 09:58:35 pm Gabriel Roldán wrote:
>   
>> On Wednesday 21 May 2008 09:39:34 pm Jody Garnett wrote:
>>     
>>> Gabriel Roldán wrote:
>>>       
>>>> Hey Jody,
>>>>
>>>> so this is what I did:
>>>> - got rid of connection.commit/rollback in ArcTransactionState
>>>> - added ArcSDEDataStore.getSession(Transaction). For read only use just
>>>> call it with Transaction.AUTO_COMMIT, and that's what FeatureSource
>>>> does, whilst FetureStore uses the transaction it was set with
>>>>         
>>> Sounds good.
>>>
>>>       
>>>> There's something strange with the command execution though, may be you
>>>> have some time to test it, since its almost bed time for me now.
>>>>         
>>> I will update now.
>>>
>>>       
>>>> The strange is that running ArcSDEFeatureStoreTest against my server
>>>> (which is close so it runs fast) I get a lot of "THIS CONNECTION IS
>>>> BEING USED AT THE MOMENT BY ANOTHER THREAD" errors, while executing the
>>>> same test against your server ( which is far away so most of the time
>>>> is spent in network io) only the last two test cases fail.
>>>>         
>>> That is confusing to me; I don't understand it either.
>>>
>>>       
>>>> My hope is that if you run it you'll get a lot of broken cases as your
>>>> server is fast for you, and you may be able to get a clue why some
>>>> command execution is breaking that way...
>>>>
>>>> Another thing: using issueReadCommand like you did in fetchSchema
>>>> implies needing two connections at a given time for a single thing.
>>>> I've commented it by now, since I guess when using
>>>> ArcSDEConnectionReference you just want to return the same session over
>>>> and over, so there shouldn't be a need to do anything special? or
>>>> rather I'm not
>>>> understanding why you want the issueReadOnly method. The idea being to
>>>> pick up any available connection, but you have only one, so why not
>>>> just overriding getSession to return the same one? not sure..
>>>>         
>>> This is why I need the code review:
>>> - ArcSDEConnectionReference.issueReadCommand will try and reuse the
>>> "last" Session (ie the only session), if it is not available it will
>>> grab a Session from the pool like normal
>>> -ArcSDEConnectionPool.issueReadCommand will grab a session from the pool
>>> like normal - I did not want to change any functionality on you until
>>> you had done a code review. So this method behaves exactly the same as
>>> before (ie we always needed 2 connections in the past)
>>>       
>> it did in past, then I got with working with a single one, then introducing
>> this required two again. right now you can run the ArcSDEFeatureSourceTest
>> with a single connection, though I won't do any statement that a single
>> connection is enough for all and every code path, yet..
>> The big difference is that now we have a much finer granularity on what an
>> execution unit is, connection wise, so there're higher probabilities a
>> single connection serves more read only operations.
>>
>>     
>>>> Also, you'll note a lot of verbosity on the console, it was me freaking
>>>> out while trying to understand about what's causing the connection to
>>>> complain about threading.
>>>>         
>>> So where does the exception about threading appear?
>>>       
>> ArcSDEFeatureStoreTest.testUpdateAutocommit first, then everything gets
>> screwed up.
>>
>> but I couldnt understand for the last long hours why the heck the command
>> sent at ArcSdeFeatureWriter.getUpdatableColumnNames() (line 694) leads to
>> this error and not any other command.
>>
>> any clue will be much appreciated, I seem too burned right now for a single
>> day, and am starting to stop beleaving in resolving hard problems in dreams
>> or shower...  :)
>>
>> cheers,
>>
>> Gabriel
>>
>>     
>>> Jody
>>>       
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> Geotools-devel mailing list
>> Geo...@li...
>> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>>
>> !DSPAM:4045,48347f5e26452090977483!
>>     
>
>
>   

Re: [Geotools-devel] ArcSDE status update

[Gabriel R. <gr...@op...>]

>  5- Test ArcSDEConnectionReference for the single connection configuration.
> Jody I would try caching the connection once and just overriding
> getSession() to return it over and over.
>
Jody, just tried this and everything worked but a single test: 
testEditVersionedTableTransaction

Just check out my latest commit and tell me what you think. Note I've removed 
ArcSDEMinimalDataStore

Cheers,

Gabriel

Re: [Geotools-devel] ArcSDE status update

[Jody G. <jga...@re...>]

Gabriel Roldán wrote:
>>  5- Test ArcSDEConnectionReference for the single connection configuration.
>> Jody I would try caching the connection once and just overriding
>> getSession() to return it over and over.
>>
>>     
> Jody, just tried this and everything worked but a single test: 
> testEditVersionedTableTransaction
>
> Just check out my latest commit and tell me what you think. Note I've removed 
> ArcSDEMinimalDataStore
>   
That is fine; I think the functionality moved into the 
ArcSDEConnectReference class at the end of the day right.
Jody