pythomnic3k-questions — General questions about Pythomnic3k framework

Re: [Pythomnic3k-questions] [ANN] Pythomnic3k 1.2 released

[Eric L. <Er...@Th...>]

Hmmm, well, I guess my question is, as a best practice, how to I easily
partition my source files?

My main module .py file is now up to about 2,000 lines, and typically I
don't like to do that - I'd prefer to have each of my classes broken out
into separate files, have a "header" file with my constants in it (I guess
revealing my roots in C and C++), etc.

I haven't worked out how to easily break that file down into say 10 or more
"import"able modules that I can call, etc.

How would you do such a thing? I'm not trying to make each class a "module"
with separate pmnc call overhead, etc - just trying to simply break up the
source files for my one main "cage" into workable pieces.

-----Original Message-----
From: Dmitry Dvoinikov [mailto:dm...@ta...] 
Sent: Monday, December 13, 2010 1:38 AM
To: Eric Livingston
Cc: pyt...@li...
Subject: Re: [Pythomnic3k-questions] [ANN] Pythomnic3k 1.2 released

 > import *" fails due to it not being found.
 > What syntax should I use to allow
 > the import to locate other .py files in my cage  > directory for
importing?

In Pythomnic cage modules are effectively components, with public methods
accessible through pmnc calls, they cannot be import'ed in a regular way.

If you need definitions, put the module containing them to lib and do your
regular import statement, but then it won't be reloaded upon change, because
the framework has no control over what has been imported where. For example
if some application module did

from foo import bar

and module foo has been changed since, it is hard to find that application
module's namespace needs to be patched. There also are problems with
concurrency and such.

This is why modules in Pythomnic have to access each other through

pmnc.module.method(...)

to allow pmnc proxy to intercept the call and do the magic.

Sincerely,
Dmitry Dvoinikov

On 12.12.2010 22:14, Eric Livingston wrote:
> Another quick question, can't find anything on the site:
>
> How do I reference a module from my cage? I created a utility file, 
> "constants.py" with a whole bunch of constant definitions in it, so 
> I'd not have all of them on top of my module code. But simply using 
> "from constants import *" fails due to it not being found. What syntax 
> should I use to allow the import to locate other .py files in my cage
directory for importing?
>
> -----Original Message-----
> From: Dmitry Dvoinikov [mailto:dm...@ta...]
> Sent: Monday, December 06, 2010 8:49 AM
> To: pyt...@li...
> Subject: [Pythomnic3k-questions] [ANN] Pythomnic3k 1.2 released
>
> Hello all,
>
> I'm happy to announce release of Pythomnic3k version 1.2.
>
> See http://www.pythomnic3k.org/download.html
>
> Here, a list of changes, in order of decreasing practical usefulness:
>
> ------------------------------------------------
>
> * Logging priority can be changed at runtime,
>     no need to restart a cage:
>
>     log_level = "LEVEL"
>
>     in config_interfaces.py
>
>     Command-line parameter to startup.py is now ignored and
>     should be removed from your cage starting scripts if you
>     have any.
>
> ------------------------------------------------
>
> * Request timeouts can be specified on per-interface basis.
>     This is useful if a cage uses retried calls as deferred
>     processing facility. In this case a request arriving
>     via synchronous interface requiring fast response may
>     have timeout of ex. 10 seconds, and the enqueued retried
>     call arriving later via retry interface may have
>     timeout of ex. 60 seconds.
>
>     request_timeout = 10.0
>
>     optionally in config_interface_NAME.py
>
> ------------------------------------------------
>
> * Changed the way logging works. Now each request has a readable
>     description which is included in its logging messages.
>     Request's remaining time to live is also included.
>
> ------------------------------------------------
>
> * Resource pools can be configured to be kept warm,
>     i.e. having a few free instances always connected.
>     Note that a resource needs to be accessed before its
>     pool is being created at all.
>
>     pool__standby = N
>
>     optionally in config_resource_NAME.py
>
> ------------------------------------------------
>
> * Added support for custom acceptance methods for transactions.
>     Acceptance method receives intermediate results of the resources
>     participating in the transaction and decides what to do with them.
>     It can perform tricks such as:
>     - Pick the fastest result and return transaction result early
>       without waiting for all participants to return.
>     - Deduce the result by voting despite occasional failures.
>     - Mask irrelevant failures.
>     etc.
>
>     def custom_accept(...):
>       ...
>
>     xa = pmnc.transaction.create(accept = custom_accept)
>     ...
>
> ------------------------------------------------
>
> * Changed the way transactions and resources report errors.
>     Now if a transaction fails, it throws ResourceError, which
>     contains failure information in a uniform way. For instance,
>     ResourceError has "participant_index" property which contains
>     index of the failed participant, and "recoverable" property
>     which is True if the failing resource did not make any
>     irreversible changes before it failed:
>
>     xa = pmnc.transaction.create()
>     xa.good.do_stuff(...)
>     xa.bad.do_stuff_and_fail(...)
>     try:
>       xa.execute()
>     except ResourceError as e:
>       assert e.participant_index == 1
>       if e.recoverable:
>         # phew... that was close...
>
>     ResourceError also has protocol specific properties "code"
>     and "description". Database resources throw SQLResourceError,
>     a subclass with "state" property containing error SQL state.
>
> ------------------------------------------------
>
> * Retried call queues can be configured to maintain
>     FIFO order, a failed call will remain in front of
>     the queue and retried.
>
>     keep_order = True
>
>     in config_interface_retry.py
>
> ------------------------------------------------
>
> * Added configurable file permissions when files are being
>     saved to a shared directory using file resource.
>
>     file_permissions = "rw-r--r--"
>
>     in config_resource_file.py
>
> ------------------------------------------------
>
> * Email protocol now supports sending (SMTP) and receiving (POP3)
>     messages over SSL.
>
>     ssl_key_cert_file = "/cert.pem",
>     ssl_ca_cert_file = "/ca.pem",
>
>     in config_interface/resource_email.py
>
> ------------------------------------------------
>
> * Oracle SQL resources return all numbers as Decimals, never ints,
>     making no attempt to deduce type information from the values.
>
> ------------------------------------------------
>
> * Unknown SMPP optional tags on incoming PDUs are ignored,
>     rather than cause a parsing failure. One awkward SMPP SMSC kept
>     sending optional tags from SMPP 5.0 to SMPP 3.4 client which
>     Pythomnic3k is.
>
> ------------------------------------------------
>
> * GREATLY simplified the request creation logic. This is of use
>     for anyone who is going to implement some protocol support for
>     Pythomnic3k.
>
> ------------------------------------------------
>
> * Added CPU, memory statistics to cage performance web page.
>
> ------------------------------------------------
>
> * Request context is preserved between cages verbatim,
>     not modified across RPC calls.
>
> ------------------------------------------------
>
> * Persistent state is disabled if free disk space falls
>     below configured threshold. Note that this affects
>     retried calls too.
>
>     minimum_free_space = 50
>
>     in config_state.py.
>
> ------------------------------------------------
>
> * Python 3.2 compatibility fixes.
>
> ------------------------------------------------
>
> Upgrade instructions:
>
> 1. Remove lib, cages/.shared and startup.py and replace it
>      with new version. As you should have never modified any file
>      there, you are not losing any work.
>
> 2. In each of your working directories cages/cage modify
>      configuration files:
>
> 2.1. In all interface configuration files config_interface_NAME.py
>        remove self_test_config dicts at all. Replace
>        self_test_config with empty dicts in get/copy lambdas.
>        Use any config_interface_NAME.py from .shared as reference.
>
> 2.2. In all resource configuration files config_resource_NAME.py
>        remove self_test_config dicts unless *your* self-tests are using
>        them. Insert a NameError catch before get/copy lambdas.
>        Use any config_resource_NAME.py from .shared as reference.
>
> 2.3. In config_interfaces.py add log_level = "LEVEL".
>
> 2.4. In config_interface_retry.py add keep_order = False
>
> 2.5. In config_resource_oracle.py remove rounding_precision = ...
>
> 2.6. In config_interface_email.py add
>
>     ssl_key_cert_file = None,
>     ssl_ca_cert_file = None,
>
> Sincerely,
> Dmitry Dvoinikov
>
> ----------------------------------------------------------------------
> ------
> --
> What happens now with your Lotus Notes apps - do you make another 
> costly upgrade, or settle for being marooned without product support? 
> Time to move off Lotus Notes and onto the cloud with Force.com, apps 
> are easier to build, use, and manage than apps on traditional 
> platforms. Sign up for the Lotus Notes Migration Kit to learn more. 
> http://p.sf.net/sfu/salesforce-d2d
> _______________________________________________
> Pythomnic3k-questions mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pythomnic3k-questions
>
>

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Dmitry D. <dm...@ta...>]

Hello Eric,

Use

pmnc.state.delete(k)

In fact, get/set/delete are just shortcuts. /.shared/state.py also
provides facilities to have as many BerkeleyDB databases as you like
and access them in any way you like. This of course has BerkeleyDB
specific semantics, but it allows to have useful things like
blazingly fast persistent queues (used for example in retried calls).

Here, an example of a transactional persistent queue:

---------------------------------------------------------

# actual data access method, db is a native BerkeleyDB database

def _push(txn, db, value):
     return db.append(pickle(value), txn)

# a method to wrap the first one in transaction

def push(db, value):
     pmnc.state.implicit_transaction(_push, db, value)

# same thing with pop

def _pop(txn, db, default):
     value = db.consume(txn)
     if value is not None:
         return unpickle(value[1])
     else:
         return default

def pop(db, default = None):
     return pmnc.state.implicit_transaction(_pop, db, default)

# allocate a named BerkeleyDB database of type queue

q = pmnc.state.get_queue("some_queue", re_len = 1024)

# and get busy

push(q, "foo")
assert pop(q) == "foo"

---------------------------------------------------------

See self-tests in module state.py for more examples.

There is no shortcut to clear a state database but you
can do it manually using the above technique.

 > Also, I tried using pmnc3ksvc to set up my cage as a service, but it
 > silently fails. I'm presuming that's another 32 vs. 64 bit issue, but 
wasn't
 > sure. Do you have any tips for getting that working?

I don't think 64 bit is a problem, I'm using it here and it works.
Does it say anything ? Do you follow the command line hint precisely ?
Do all the files you mention (ex. python.exe, startup.py) exist in
the locations ? Do you have administrative rights ?

Dmitry

On 13.12.2010 21:29, Eric Livingston wrote:
> Thanks for the tip! Is there a way to delete out entries from the state
> database so as to maintain it over time, or reset it?
>
> Also, I tried using pmnc3ksvc to set up my cage as a service, but it
> silently fails. I'm presuming that's another 32 vs. 64 bit issue, but wasn't
> sure. Do you have any tips for getting that working?
>
> Thanks!
>
> Eric

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <eri...@gm...>]

Thanks for the tip! Is there a way to delete out entries from the state
database so as to maintain it over time, or reset it?

Also, I tried using pmnc3ksvc to set up my cage as a service, but it
silently fails. I'm presuming that's another 32 vs. 64 bit issue, but wasn't
sure. Do you have any tips for getting that working?

Thanks!

Eric

-----Original Message-----
From: Dmitry Dvoinikov [mailto:dm...@ta...] 
Sent: Monday, December 13, 2010 2:43 AM
To: Eric Livingston
Cc: pyt...@li...
Subject: Re: [Pythomnic3k-questions] Changing the server address in
Resource_http

A value is anything of any size, not necessarily string.
Serialization of data is done transparently, which is a benefit compared to
other database, although calling pickle manually is only a minor nuisance.

Speed-wise, embedded BerkeleyDB should generally be at least as fast as an
external PostgreSQL, but YMMV.

If you want freedom to choose, you can make a thunk module, say mystate.py
of this kind

--------------------------------------
__all__ = [ "get", "set" ]

def get(key):
     return pmnc.state.get(key)

def set(key, value):
     return pmnc.state.set(key, value)

# EOF
--------------------------------------

and then you may replace it with database version (simplified)

--------------------------------------
__all__ = [ "get", "set" ]

def get(key):
     xa = pmnc.transaction.create()
     xa.db.execute("SELECT v FROM t WHERE k = {key}", key = key)
     v = xa.execute()[0][0]["v"]
     return unpickle(v)

def set(key, value):
     xa = pmnc.transaction.create()
     xa.db.execute("INSERT OR UPDATE t SET v = {value} WHERE k = {key}",
                   key = key, value = pickle(value))
     xa.execute()

# EOF
--------------------------------------

without touching the rest of the application, which would simply be using

pmnc.mystate.get(k, v) and pmnc.mystate.set(k, v)

Sincerely,
Dmitry Dvoinikov

On 13.12.2010 12:14, Eric Livingston wrote:
> No, that's good enough - assuming v is an arbitrary-length value, I 
> can use pickle to marshal and unmarshal some key data structures that 
> will save me several database lookups. Should go much faster, as I'd 
> assume the overhead to set and get from the State db is much less than 
> for postgresql? If not, then I'll just keep saving everything to the 
> database, but this might be a good thing to try.
>
> -----Original Message-----
> From: Dmitry Dvoinikov [mailto:dm...@ta...]
> Sent: Monday, December 13, 2010 1:41 AM
> To: Eric Livingston
> Cc: pyt...@li...
> Subject: Re: [Pythomnic3k-questions] Changing the server address in 
> Resource_http
>
> I think you are misled. State is just a key-value storage, provided by 
> the framework for your convenience. Each module has its private 
> database it accesses using simply
>
> pmnc.state.set(k, v)
> pmnc.state.get(k)
>
> It is not like there is a mechanism for making a persistent snapshot 
> of the application. Such wonder is up to you as a developer.
>
> Sincerely,
> Dmitry Dvoinikov
>
> On 11.12.2010 1:22, Eric Livingston wrote:
>> Wow, the State module appears to have loaded correctly after 
>> downloading all your updates and the right 64-bit build of bsddb3 
>> from your site! At least the log shows no errors when I load up my cage!
>>
>> Now, is there any documentation regarding how to use it? Lol - if I 
>> could use a key of some sort to re-load a cage on a re-request from a 
>> client, to pick up where it left off, it would save me about 15 
>> database reads that I have to execute each time a request comes in to 
>> re-initialize the cage from scratch each time.
>
>

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Dmitry D. <dm...@ta...>]

A value is anything of any size, not necessarily string.
Serialization of data is done transparently, which is a
benefit compared to other database, although calling pickle
manually is only a minor nuisance.

Speed-wise, embedded BerkeleyDB should generally be at
least as fast as an external PostgreSQL, but YMMV.

If you want freedom to choose, you can make a thunk module,
say mystate.py of this kind

--------------------------------------
__all__ = [ "get", "set" ]

def get(key):
     return pmnc.state.get(key)

def set(key, value):
     return pmnc.state.set(key, value)

# EOF
--------------------------------------

and then you may replace it with database version (simplified)

--------------------------------------
__all__ = [ "get", "set" ]

def get(key):
     xa = pmnc.transaction.create()
     xa.db.execute("SELECT v FROM t WHERE k = {key}", key = key)
     v = xa.execute()[0][0]["v"]
     return unpickle(v)

def set(key, value):
     xa = pmnc.transaction.create()
     xa.db.execute("INSERT OR UPDATE t SET v = {value} WHERE k = {key}",
                   key = key, value = pickle(value))
     xa.execute()

# EOF
--------------------------------------

without touching the rest of the application, which would simply
be using

pmnc.mystate.get(k, v) and pmnc.mystate.set(k, v)

Sincerely,
Dmitry Dvoinikov

On 13.12.2010 12:14, Eric Livingston wrote:
> No, that's good enough - assuming v is an arbitrary-length value, I can use
> pickle to marshal and unmarshal some key data structures that will save me
> several database lookups. Should go much faster, as I'd assume the overhead
> to set and get from the State db is much less than for postgresql? If not,
> then I'll just keep saving everything to the database, but this might be a
> good thing to try.
>
> -----Original Message-----
> From: Dmitry Dvoinikov [mailto:dm...@ta...]
> Sent: Monday, December 13, 2010 1:41 AM
> To: Eric Livingston
> Cc: pyt...@li...
> Subject: Re: [Pythomnic3k-questions] Changing the server address in
> Resource_http
>
> I think you are misled. State is just a key-value storage, provided by the
> framework for your convenience. Each module has its private database it
> accesses using simply
>
> pmnc.state.set(k, v)
> pmnc.state.get(k)
>
> It is not like there is a mechanism for making a persistent snapshot of the
> application. Such wonder is up to you as a developer.
>
> Sincerely,
> Dmitry Dvoinikov
>
> On 11.12.2010 1:22, Eric Livingston wrote:
>> Wow, the State module appears to have loaded correctly after
>> downloading all your updates and the right 64-bit build of bsddb3 from
>> your site! At least the log shows no errors when I load up my cage!
>>
>> Now, is there any documentation regarding how to use it? Lol - if I
>> could use a key of some sort to re-load a cage on a re-request from a
>> client, to pick up where it left off, it would save me about 15
>> database reads that I have to execute each time a request comes in to
>> re-initialize the cage from scratch each time.
>
>

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <Er...@Th...>]

No, that's good enough - assuming v is an arbitrary-length value, I can use
pickle to marshal and unmarshal some key data structures that will save me
several database lookups. Should go much faster, as I'd assume the overhead
to set and get from the State db is much less than for postgresql? If not,
then I'll just keep saving everything to the database, but this might be a
good thing to try.

-----Original Message-----
From: Dmitry Dvoinikov [mailto:dm...@ta...] 
Sent: Monday, December 13, 2010 1:41 AM
To: Eric Livingston
Cc: pyt...@li...
Subject: Re: [Pythomnic3k-questions] Changing the server address in
Resource_http

I think you are misled. State is just a key-value storage, provided by the
framework for your convenience. Each module has its private database it
accesses using simply

pmnc.state.set(k, v)
pmnc.state.get(k)

It is not like there is a mechanism for making a persistent snapshot of the
application. Such wonder is up to you as a developer.

Sincerely,
Dmitry Dvoinikov

On 11.12.2010 1:22, Eric Livingston wrote:
> Wow, the State module appears to have loaded correctly after 
> downloading all your updates and the right 64-bit build of bsddb3 from 
> your site! At least the log shows no errors when I load up my cage!
>
> Now, is there any documentation regarding how to use it? Lol - if I 
> could use a key of some sort to re-load a cage on a re-request from a 
> client, to pick up where it left off, it would save me about 15 
> database reads that I have to execute each time a request comes in to 
> re-initialize the cage from scratch each time.

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Dmitry D. <dm...@ta...>]

I think you are misled. State is just a key-value storage,
provided by the framework for your convenience. Each module
has its private database it accesses using simply

pmnc.state.set(k, v)
pmnc.state.get(k)

It is not like there is a mechanism for making a persistent
snapshot of the application. Such wonder is up to you as
a developer.

Sincerely,
Dmitry Dvoinikov

On 11.12.2010 1:22, Eric Livingston wrote:
> Wow, the State module appears to have loaded correctly after downloading all
> your updates and the right 64-bit build of bsddb3 from your site! At least
> the log shows no errors when I load up my cage!
>
> Now, is there any documentation regarding how to use it? Lol - if I could
> use a key of some sort to re-load a cage on a re-request from a client, to
> pick up where it left off, it would save me about 15 database reads that I
> have to execute each time a request comes in to re-initialize the cage from
> scratch each time.

Re: [Pythomnic3k-questions] [ANN] Pythomnic3k 1.2 released

[Dmitry D. <dm...@ta...>]

 > import *" fails due to it not being found.
 > What syntax should I use to allow
 > the import to locate other .py files in my cage
 > directory for importing?

In Pythomnic cage modules are effectively components,
with public methods accessible through pmnc calls, they
cannot be import'ed in a regular way.

If you need definitions, put the module containing them
to lib and do your regular import statement, but then it won't
be reloaded upon change, because the framework has no control
over what has been imported where. For example if some
application module did

from foo import bar

and module foo has been changed since, it is hard to find that
application module's namespace needs to be patched. There also
are problems with concurrency and such.

This is why modules in Pythomnic have to access each other through

pmnc.module.method(...)

to allow pmnc proxy to intercept the call and do the magic.

Sincerely,
Dmitry Dvoinikov

On 12.12.2010 22:14, Eric Livingston wrote:
> Another quick question, can't find anything on the site:
>
> How do I reference a module from my cage? I created a utility file,
> "constants.py" with a whole bunch of constant definitions in it, so I'd not
> have all of them on top of my module code. But simply using "from constants
> import *" fails due to it not being found. What syntax should I use to allow
> the import to locate other .py files in my cage directory for importing?
>
> -----Original Message-----
> From: Dmitry Dvoinikov [mailto:dm...@ta...]
> Sent: Monday, December 06, 2010 8:49 AM
> To: pyt...@li...
> Subject: [Pythomnic3k-questions] [ANN] Pythomnic3k 1.2 released
>
> Hello all,
>
> I'm happy to announce release of Pythomnic3k version 1.2.
>
> See http://www.pythomnic3k.org/download.html
>
> Here, a list of changes, in order of decreasing practical usefulness:
>
> ------------------------------------------------
>
> * Logging priority can be changed at runtime,
>     no need to restart a cage:
>
>     log_level = "LEVEL"
>
>     in config_interfaces.py
>
>     Command-line parameter to startup.py is now ignored and
>     should be removed from your cage starting scripts if you
>     have any.
>
> ------------------------------------------------
>
> * Request timeouts can be specified on per-interface basis.
>     This is useful if a cage uses retried calls as deferred
>     processing facility. In this case a request arriving
>     via synchronous interface requiring fast response may
>     have timeout of ex. 10 seconds, and the enqueued retried
>     call arriving later via retry interface may have
>     timeout of ex. 60 seconds.
>
>     request_timeout = 10.0
>
>     optionally in config_interface_NAME.py
>
> ------------------------------------------------
>
> * Changed the way logging works. Now each request has a readable
>     description which is included in its logging messages.
>     Request's remaining time to live is also included.
>
> ------------------------------------------------
>
> * Resource pools can be configured to be kept warm,
>     i.e. having a few free instances always connected.
>     Note that a resource needs to be accessed before its
>     pool is being created at all.
>
>     pool__standby = N
>
>     optionally in config_resource_NAME.py
>
> ------------------------------------------------
>
> * Added support for custom acceptance methods for transactions.
>     Acceptance method receives intermediate results of the resources
>     participating in the transaction and decides what to do with them.
>     It can perform tricks such as:
>     - Pick the fastest result and return transaction result early
>       without waiting for all participants to return.
>     - Deduce the result by voting despite occasional failures.
>     - Mask irrelevant failures.
>     etc.
>
>     def custom_accept(...):
>       ...
>
>     xa = pmnc.transaction.create(accept = custom_accept)
>     ...
>
> ------------------------------------------------
>
> * Changed the way transactions and resources report errors.
>     Now if a transaction fails, it throws ResourceError, which
>     contains failure information in a uniform way. For instance,
>     ResourceError has "participant_index" property which contains
>     index of the failed participant, and "recoverable" property
>     which is True if the failing resource did not make any
>     irreversible changes before it failed:
>
>     xa = pmnc.transaction.create()
>     xa.good.do_stuff(...)
>     xa.bad.do_stuff_and_fail(...)
>     try:
>       xa.execute()
>     except ResourceError as e:
>       assert e.participant_index == 1
>       if e.recoverable:
>         # phew... that was close...
>
>     ResourceError also has protocol specific properties "code"
>     and "description". Database resources throw SQLResourceError,
>     a subclass with "state" property containing error SQL state.
>
> ------------------------------------------------
>
> * Retried call queues can be configured to maintain
>     FIFO order, a failed call will remain in front of
>     the queue and retried.
>
>     keep_order = True
>
>     in config_interface_retry.py
>
> ------------------------------------------------
>
> * Added configurable file permissions when files are being
>     saved to a shared directory using file resource.
>
>     file_permissions = "rw-r--r--"
>
>     in config_resource_file.py
>
> ------------------------------------------------
>
> * Email protocol now supports sending (SMTP) and receiving (POP3)
>     messages over SSL.
>
>     ssl_key_cert_file = "/cert.pem",
>     ssl_ca_cert_file = "/ca.pem",
>
>     in config_interface/resource_email.py
>
> ------------------------------------------------
>
> * Oracle SQL resources return all numbers as Decimals, never ints,
>     making no attempt to deduce type information from the values.
>
> ------------------------------------------------
>
> * Unknown SMPP optional tags on incoming PDUs are ignored,
>     rather than cause a parsing failure. One awkward SMPP SMSC kept
>     sending optional tags from SMPP 5.0 to SMPP 3.4 client which
>     Pythomnic3k is.
>
> ------------------------------------------------
>
> * GREATLY simplified the request creation logic. This is of use
>     for anyone who is going to implement some protocol support for
>     Pythomnic3k.
>
> ------------------------------------------------
>
> * Added CPU, memory statistics to cage performance web page.
>
> ------------------------------------------------
>
> * Request context is preserved between cages verbatim,
>     not modified across RPC calls.
>
> ------------------------------------------------
>
> * Persistent state is disabled if free disk space falls
>     below configured threshold. Note that this affects
>     retried calls too.
>
>     minimum_free_space = 50
>
>     in config_state.py.
>
> ------------------------------------------------
>
> * Python 3.2 compatibility fixes.
>
> ------------------------------------------------
>
> Upgrade instructions:
>
> 1. Remove lib, cages/.shared and startup.py and replace it
>      with new version. As you should have never modified any file
>      there, you are not losing any work.
>
> 2. In each of your working directories cages/cage modify
>      configuration files:
>
> 2.1. In all interface configuration files config_interface_NAME.py
>        remove self_test_config dicts at all. Replace
>        self_test_config with empty dicts in get/copy lambdas.
>        Use any config_interface_NAME.py from .shared as reference.
>
> 2.2. In all resource configuration files config_resource_NAME.py
>        remove self_test_config dicts unless *your* self-tests are using
>        them. Insert a NameError catch before get/copy lambdas.
>        Use any config_resource_NAME.py from .shared as reference.
>
> 2.3. In config_interfaces.py add log_level = "LEVEL".
>
> 2.4. In config_interface_retry.py add keep_order = False
>
> 2.5. In config_resource_oracle.py remove rounding_precision = ...
>
> 2.6. In config_interface_email.py add
>
>     ssl_key_cert_file = None,
>     ssl_ca_cert_file = None,
>
> Sincerely,
> Dmitry Dvoinikov
>
> ----------------------------------------------------------------------------
> --
> What happens now with your Lotus Notes apps - do you make another costly
> upgrade, or settle for being marooned without product support? Time to move
> off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
> use, and manage than apps on traditional platforms. Sign up for the Lotus
> Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
> _______________________________________________
> Pythomnic3k-questions mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pythomnic3k-questions
>
>

Re: [Pythomnic3k-questions] [ANN] Pythomnic3k 1.2 released

[Eric L. <Er...@Th...>]

Another quick question, can't find anything on the site:

How do I reference a module from my cage? I created a utility file,
"constants.py" with a whole bunch of constant definitions in it, so I'd not
have all of them on top of my module code. But simply using "from constants
import *" fails due to it not being found. What syntax should I use to allow
the import to locate other .py files in my cage directory for importing?

-----Original Message-----
From: Dmitry Dvoinikov [mailto:dm...@ta...] 
Sent: Monday, December 06, 2010 8:49 AM
To: pyt...@li...
Subject: [Pythomnic3k-questions] [ANN] Pythomnic3k 1.2 released

Hello all,

I'm happy to announce release of Pythomnic3k version 1.2.

See http://www.pythomnic3k.org/download.html

Here, a list of changes, in order of decreasing practical usefulness:

------------------------------------------------

* Logging priority can be changed at runtime,
   no need to restart a cage:

   log_level = "LEVEL"

   in config_interfaces.py

   Command-line parameter to startup.py is now ignored and
   should be removed from your cage starting scripts if you
   have any.

------------------------------------------------

* Request timeouts can be specified on per-interface basis.
   This is useful if a cage uses retried calls as deferred
   processing facility. In this case a request arriving
   via synchronous interface requiring fast response may
   have timeout of ex. 10 seconds, and the enqueued retried
   call arriving later via retry interface may have
   timeout of ex. 60 seconds.

   request_timeout = 10.0

   optionally in config_interface_NAME.py

------------------------------------------------

* Changed the way logging works. Now each request has a readable
   description which is included in its logging messages.
   Request's remaining time to live is also included.

------------------------------------------------

* Resource pools can be configured to be kept warm,
   i.e. having a few free instances always connected.
   Note that a resource needs to be accessed before its
   pool is being created at all.

   pool__standby = N

   optionally in config_resource_NAME.py

------------------------------------------------

* Added support for custom acceptance methods for transactions.
   Acceptance method receives intermediate results of the resources
   participating in the transaction and decides what to do with them.
   It can perform tricks such as:
   - Pick the fastest result and return transaction result early
     without waiting for all participants to return.
   - Deduce the result by voting despite occasional failures.
   - Mask irrelevant failures.
   etc.

   def custom_accept(...):
     ...

   xa = pmnc.transaction.create(accept = custom_accept)
   ...

------------------------------------------------

* Changed the way transactions and resources report errors.
   Now if a transaction fails, it throws ResourceError, which
   contains failure information in a uniform way. For instance,
   ResourceError has "participant_index" property which contains
   index of the failed participant, and "recoverable" property
   which is True if the failing resource did not make any
   irreversible changes before it failed:

   xa = pmnc.transaction.create()
   xa.good.do_stuff(...)
   xa.bad.do_stuff_and_fail(...)
   try:
     xa.execute()
   except ResourceError as e:
     assert e.participant_index == 1
     if e.recoverable:
       # phew... that was close...

   ResourceError also has protocol specific properties "code"
   and "description". Database resources throw SQLResourceError,
   a subclass with "state" property containing error SQL state.

------------------------------------------------

* Retried call queues can be configured to maintain
   FIFO order, a failed call will remain in front of
   the queue and retried.

   keep_order = True

   in config_interface_retry.py

------------------------------------------------

* Added configurable file permissions when files are being
   saved to a shared directory using file resource.

   file_permissions = "rw-r--r--"

   in config_resource_file.py

------------------------------------------------

* Email protocol now supports sending (SMTP) and receiving (POP3)
   messages over SSL.

   ssl_key_cert_file = "/cert.pem",
   ssl_ca_cert_file = "/ca.pem",

   in config_interface/resource_email.py

------------------------------------------------

* Oracle SQL resources return all numbers as Decimals, never ints,
   making no attempt to deduce type information from the values.

------------------------------------------------

* Unknown SMPP optional tags on incoming PDUs are ignored,
   rather than cause a parsing failure. One awkward SMPP SMSC kept
   sending optional tags from SMPP 5.0 to SMPP 3.4 client which
   Pythomnic3k is.

------------------------------------------------

* GREATLY simplified the request creation logic. This is of use
   for anyone who is going to implement some protocol support for
   Pythomnic3k.

------------------------------------------------

* Added CPU, memory statistics to cage performance web page.

------------------------------------------------

* Request context is preserved between cages verbatim,
   not modified across RPC calls.

------------------------------------------------

* Persistent state is disabled if free disk space falls
   below configured threshold. Note that this affects
   retried calls too.

   minimum_free_space = 50

   in config_state.py.

------------------------------------------------

* Python 3.2 compatibility fixes.

------------------------------------------------

Upgrade instructions:

1. Remove lib, cages/.shared and startup.py and replace it
    with new version. As you should have never modified any file
    there, you are not losing any work.

2. In each of your working directories cages/cage modify
    configuration files:

2.1. In all interface configuration files config_interface_NAME.py
      remove self_test_config dicts at all. Replace
      self_test_config with empty dicts in get/copy lambdas.
      Use any config_interface_NAME.py from .shared as reference.

2.2. In all resource configuration files config_resource_NAME.py
      remove self_test_config dicts unless *your* self-tests are using
      them. Insert a NameError catch before get/copy lambdas.
      Use any config_resource_NAME.py from .shared as reference.

2.3. In config_interfaces.py add log_level = "LEVEL".

2.4. In config_interface_retry.py add keep_order = False

2.5. In config_resource_oracle.py remove rounding_precision = ...

2.6. In config_interface_email.py add

   ssl_key_cert_file = None,
   ssl_ca_cert_file = None,

Sincerely,
Dmitry Dvoinikov

----------------------------------------------------------------------------
--
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
_______________________________________________
Pythomnic3k-questions mailing list
Pyt...@li...
https://lists.sourceforge.net/lists/listinfo/pythomnic3k-questions

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <Er...@Th...>]

Wow, the State module appears to have loaded correctly after downloading all
your updates and the right 64-bit build of bsddb3 from your site! At least
the log shows no errors when I load up my cage!

Now, is there any documentation regarding how to use it? Lol - if I could
use a key of some sort to re-load a cage on a re-request from a client, to
pick up where it left off, it would save me about 15 database reads that I
have to execute each time a request comes in to re-initialize the cage from
scratch each time.

-----Original Message-----
From: Dmitry Dvoinikov [mailto:dm...@ta...] 
Sent: Monday, December 06, 2010 8:52 AM
To: Eric Livingston
Cc: pyt...@li...
Subject: Re: [Pythomnic3k-questions] Changing the server address in
Resource_http

Hello Eric,

> Just to let you know, this appears to have worked just fine; I create 
> the pmnc.protocol_http.Resource directly, as you described below, and 
> that stitches my outbound requests into the framework just as the 
> inbound requests are, which also helps consolidate logging operations 
> and consistent handling of syntax, headers and other information.

Ok, good.

> The persistence layer would be nice to have - we hypothesized that the 
> problem I have loading up the binary you built was a 32/64 bit issue. 
> At this point all I can do is wait until a compatible build is 
> produced somewhere, as I don't at the moment have a build environment 
> that would let me create one.

If you still need it, bsddb3 win32 builds for various architectures are
available here: http://www.pythomnic3k.org/download.html

Sincerely,
Dmitry Dvoinikov

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Dmitry D. <dm...@ta...>]

Hello Eric,

> Just to let you know, this appears to have worked just fine; I create the
> pmnc.protocol_http.Resource directly, as you described below, and that
> stitches my outbound requests into the framework just as the inbound
> requests are, which also helps consolidate logging operations and consistent
> handling of syntax, headers and other information.

Ok, good.

> The persistence layer would be nice to have - we hypothesized that the
> problem I have loading up the binary you built was a 32/64 bit issue. At
> this point all I can do is wait until a compatible build is produced
> somewhere, as I don't at the moment have a build environment that would let
> me create one.

If you still need it, bsddb3 win32 builds for various architectures
are available here: http://www.pythomnic3k.org/download.html

Sincerely,
Dmitry Dvoinikov

[Pythomnic3k-questions] [ANN] Pythomnic3k 1.2 released

[Dmitry D. <dm...@ta...>]

Hello all,

I'm happy to announce release of Pythomnic3k version 1.2.

See http://www.pythomnic3k.org/download.html

Here, a list of changes, in order of decreasing
practical usefulness:

------------------------------------------------

* Logging priority can be changed at runtime,
   no need to restart a cage:

   log_level = "LEVEL"

   in config_interfaces.py

   Command-line parameter to startup.py is now ignored and
   should be removed from your cage starting scripts if you
   have any.

------------------------------------------------

* Request timeouts can be specified on per-interface basis.
   This is useful if a cage uses retried calls as deferred
   processing facility. In this case a request arriving
   via synchronous interface requiring fast response may
   have timeout of ex. 10 seconds, and the enqueued retried
   call arriving later via retry interface may have
   timeout of ex. 60 seconds.

   request_timeout = 10.0

   optionally in config_interface_NAME.py

------------------------------------------------

* Changed the way logging works. Now each request has a readable
   description which is included in its logging messages.
   Request's remaining time to live is also included.

------------------------------------------------

* Resource pools can be configured to be kept warm,
   i.e. having a few free instances always connected.
   Note that a resource needs to be accessed before its
   pool is being created at all.

   pool__standby = N

   optionally in config_resource_NAME.py

------------------------------------------------

* Added support for custom acceptance methods for transactions.
   Acceptance method receives intermediate results of the resources
   participating in the transaction and decides what to do with them.
   It can perform tricks such as:
   - Pick the fastest result and return transaction result early
     without waiting for all participants to return.
   - Deduce the result by voting despite occasional failures.
   - Mask irrelevant failures.
   etc.

   def custom_accept(...):
     ...

   xa = pmnc.transaction.create(accept = custom_accept)
   ...

------------------------------------------------

* Changed the way transactions and resources report errors.
   Now if a transaction fails, it throws ResourceError, which
   contains failure information in a uniform way. For instance,
   ResourceError has "participant_index" property which contains
   index of the failed participant, and "recoverable" property
   which is True if the failing resource did not make any
   irreversible changes before it failed:

   xa = pmnc.transaction.create()
   xa.good.do_stuff(...)
   xa.bad.do_stuff_and_fail(...)
   try:
     xa.execute()
   except ResourceError as e:
     assert e.participant_index == 1
     if e.recoverable:
       # phew... that was close...

   ResourceError also has protocol specific properties "code"
   and "description". Database resources throw SQLResourceError,
   a subclass with "state" property containing error SQL state.

------------------------------------------------

* Retried call queues can be configured to maintain
   FIFO order, a failed call will remain in front of
   the queue and retried.

   keep_order = True

   in config_interface_retry.py

------------------------------------------------

* Added configurable file permissions when files are being
   saved to a shared directory using file resource.

   file_permissions = "rw-r--r--"

   in config_resource_file.py

------------------------------------------------

* Email protocol now supports sending (SMTP) and receiving (POP3)
   messages over SSL.

   ssl_key_cert_file = "/cert.pem",
   ssl_ca_cert_file = "/ca.pem",

   in config_interface/resource_email.py

------------------------------------------------

* Oracle SQL resources return all numbers as Decimals, never ints,
   making no attempt to deduce type information from the values.

------------------------------------------------

* Unknown SMPP optional tags on incoming PDUs are ignored,
   rather than cause a parsing failure. One awkward SMPP SMSC kept
   sending optional tags from SMPP 5.0 to SMPP 3.4 client which
   Pythomnic3k is.

------------------------------------------------

* GREATLY simplified the request creation logic. This is of use
   for anyone who is going to implement some protocol support for
   Pythomnic3k.

------------------------------------------------

* Added CPU, memory statistics to cage performance web page.

------------------------------------------------

* Request context is preserved between cages verbatim,
   not modified across RPC calls.

------------------------------------------------

* Persistent state is disabled if free disk space falls
   below configured threshold. Note that this affects
   retried calls too.

   minimum_free_space = 50

   in config_state.py.

------------------------------------------------

* Python 3.2 compatibility fixes.

------------------------------------------------

Upgrade instructions:

1. Remove lib, cages/.shared and startup.py and replace it
    with new version. As you should have never modified any file
    there, you are not losing any work.

2. In each of your working directories cages/cage modify
    configuration files:

2.1. In all interface configuration files config_interface_NAME.py
      remove self_test_config dicts at all. Replace
      self_test_config with empty dicts in get/copy lambdas.
      Use any config_interface_NAME.py from .shared as reference.

2.2. In all resource configuration files config_resource_NAME.py
      remove self_test_config dicts unless *your* self-tests are using
      them. Insert a NameError catch before get/copy lambdas.
      Use any config_resource_NAME.py from .shared as reference.

2.3. In config_interfaces.py add log_level = "LEVEL".

2.4. In config_interface_retry.py add keep_order = False

2.5. In config_resource_oracle.py remove rounding_precision = ...

2.6. In config_interface_email.py add

   ssl_key_cert_file = None,
   ssl_ca_cert_file = None,

Sincerely,
Dmitry Dvoinikov

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <Er...@Th...>]

Just to let you know, this appears to have worked just fine; I create the
pmnc.protocol_http.Resource directly, as you described below, and that
stitches my outbound requests into the framework just as the inbound
requests are, which also helps consolidate logging operations and consistent
handling of syntax, headers and other information.

-----Original Message-----
From: Дмитрий Двойников [mailto:dvo...@in...] 
Sent: Thursday, October 14, 2010 12:36 AM
Cc: pyt...@li...; Eric Livingston
Subject: Re: [Pythomnic3k-questions] Changing the server address in
Resource_http

Hello Eric,

That's a good question.

In Pythomnic3k resources are named instances of remote servers, and access
to each is decoupled and protected by a separate connection pool of limited
size. This works well if there is one fixed server known upfront, not an
arbitrary server, just like you correctly pointed out.

You have two options here.

First is a hack, but is kind of ok if you have a handful of servers you ever
need to access, not thousands of them.
Even though you may not know the target address beforehand, you can create
configuration files at runtime and use them immediately. Something like
this:

-------------------------------------------

config_filename = os.path.join(__cage_dir__, "config_resource_foo.py")

with open(config_filename, "w") as f:
   f.write("""\
config = dict
(
protocol = "http",
server_address = ({addr}, {port}),
...
)
...
""".format(addr = "foo.com", port = 80))

xa = pmnc.transaction.create()
xa.foo.get("/")
xa.execute()

-------------------------------------------

But this won't work if you have to access thousands of servers, possibly
over time. For one, the cage directory will be cluttered with one-time
files. For two, the created connection pools will hang around in memory,
even though they would be empty.

So, the second option is to give up connection pools altogether and use the
HTTP client code from protocol_http.py directly.
Something like this:

-------------------------------------------

r = pmnc.protocol_http.Resource("foo",
          server_address = ("foo.com", 80),
          connect_timeout = 3.0,
          ssl_key_cert_file = None, # or filename
          ssl_ca_cert_file = None,  # or filename
          extra_headers = { "connection": "close" },
          http_version = "HTTP/1.1")

r.connect()
try:
	result = r.get("/")
finally:
     r.disconnect()

-------------------------------------------

See health_monitor.py (line 199 in version 1.1) for an example.
See, health_monitor is in exactly that position, because it doesn't know the
locations of cages it will be probing, and so it creates resources manually.

But now there is a catch. The example works well for one-time requests, when
TCP connection is closed after each request.
But if you need persistent connections, for example you need HTTPS and you
don't want to renegotiate SSL session for each request, you need to keep the
created resources and manage their lifetime manually. Essentially, you will
need to reimplement part of connection pools code that you threw away.

Something like this:

-------------------------------------------

rs = {} # I'm not bothering with interlocking
         # thread access in this example

...

r = rs.get("foo")
if not r:
   r = pmnc.protocol_http.Resource("foo", ...)
   rs["foo"] = r
		
... use r ...

if r.expired():
   r.disconnect()      	
   del rs["foo"]
	
-------------------------------------------

And so on and so forth.

If you have any further questions, please ask.

Sincerely,
Dmitry Dvoinikov

On 13.10.2010 9:58, Eric Livingston wrote:
> I have an application that needs to use dynamically-generated URLs 
> (including server address) as a resource. I can't seem to figure out a 
> way to do that, however, as the server address and port seem to be 
> hard-coded into the config file up front.
>
> Is there any way I can work things so that upon creating a transaction 
> using the http resource, I can specific at that time what server 
> address to use?
>
> Thanks,
>
> Eric
>
> P.S. - Sorry if this posts twice; I initially sent from a 
> non-subscribed email address
>
>
>
> ----------------------------------------------------------------------
> -------- Beautiful is writing same markup. Internet Explorer 9 
> supports standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2&  
> L3.
> Spend less time writing and  rewriting code and more time creating 
> great experiences on the web. Be a part of the beta today.
> http://p.sf.net/sfu/beautyoftheweb
>
>
>
> _______________________________________________
> Pythomnic3k-questions mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pythomnic3k-questions

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Dmitry D. <dm...@ta...>]

Hello Eric,

> How's the new release coming along? I'm looking forward to it! Will you be
> notifying this mailing list when you have a release ready?

The code is frozen I hope, as soon as I have it deployed and tested
more around here it is released. You can check out current svn trunk
from https://pythomnic3k.svn.sourceforge.net/svnroot/pythomnic3k/trunk/
if you want to try it early.

> I've been having great success with Pythomnic - it's really nice to have the
> services the framework delivers for transaction processing; just what I need
> for my application, wherein I have multiple requests winging in from clients
> at all times, and have to juggle and handle them all.

Yes, it is exactly what it is for. Nice to know it fits you too.

> The persistence layer would be nice to have - we hypothesized that the
> problem I have loading up the binary you built was a 32/64 bit issue. At
> this point all I can do is wait until a compatible build is produced
> somewhere, as I don't at the moment have a build environment that would let
> me create one.

I'll see to it.

> One question I have is: Can I alter the level of logging that Pythomnic
> does? My log files are growing large quickly, and ideally at this point I'd
> rather remove the "info" level of logging - that would clean it up a great
> deal and focus down on things I have to pay attention to.

In version 1.1 that you have now the only way of setting logging level
is to pass it from command line at cage startup:

# startup.py cage LOGLEVEL

where LOGLEVEL is one of DEBUG, INFO, LOG, WARNING or ERROR. INFO is
the default and is really noisy, I agree. You better go with LOG.

In new version, one of the most useful changes is the capability
to change logging level at runtime without having the cage restarted.
Also, the logging structure is reworked so that log messages contain
more useful information and less noise. You'll see.

> I'm going to try your idea regarding dropping down a level to access the
> HTML transactions directly, so as to facilitate dynamically-generated URLs
> on a case-by-case basis. Will see how that goes!

Sure, let me know how it goes.

> Thanks for everything! Great platform :)

Thank you and good luck !

Sincerely,
Dmitry Dvoinikov

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <Er...@Th...>]

How's the new release coming along? I'm looking forward to it! Will you be
notifying this mailing list when you have a release ready?

I've been having great success with Pythomnic - it's really nice to have the
services the framework delivers for transaction processing; just what I need
for my application, wherein I have multiple requests winging in from clients
at all times, and have to juggle and handle them all.

The persistence layer would be nice to have - we hypothesized that the
problem I have loading up the binary you built was a 32/64 bit issue. At
this point all I can do is wait until a compatible build is produced
somewhere, as I don't at the moment have a build environment that would let
me create one.

One question I have is: Can I alter the level of logging that Pythomnic
does? My log files are growing large quickly, and ideally at this point I'd
rather remove the "info" level of logging - that would clean it up a great
deal and focus down on things I have to pay attention to.

I'm going to try your idea regarding dropping down a level to access the
HTML transactions directly, so as to facilitate dynamically-generated URLs
on a case-by-case basis. Will see how that goes!

Thanks for everything! Great platform :)

Eric

-----Original Message-----
From: dm...@te... [mailto:dm...@te...] 
Sent: Saturday, October 16, 2010 10:29 AM
To: Eric Livingston
Cc: pyt...@li...
Subject: RE: [Pythomnic3k-questions] Changing the server address in
Resource_http

> Hmmm, I'm thinking it's a 32 vs. 64-bit issue, perhaps?

Yes I think you are right. You may switch to 32-bit Python 3.2, but I was
unable to build BerkeleyDB 64 bit quickly so I stuck with 32-bit version,
makes no difference to me in alpha release testing.

> would have to really turn into something like this:
>
> 	try:
> 		xa = pmnc.transaction.create()
> 		xa.pg.execute("SELECT id FROM users WHERE
> name='{name}'",name=Myname)
> 		pg_result = xa.execute()[0]
> 	except:
> 		pmnc.log (some stuff)
> 		{maybe do other stuff}

A few notes here:

1. You do NOT need to put keyword parameters in single quotes, because
they are not substituted literally. They are rather binding points. When
you execute

--------------------------------
xa.pg.execute("SELECT id FROM users WHERE name = {name}", name = "Eric")
--------------------------------

the database actually sees this:

"SELECT id FROM users WHERE name = ?"
params: (varchar("Eric"), )

and NOT this:

"SELECT id FROM users WHERE name = Eric"

Do not treat keyword parameters as literals. If you need to do so, do this:

--------------------------------
xa.pg.execute("SELECT id FROM users WHERE "
              name = '{name}'".format(name = "Eric"))
--------------------------------

then the SQL substitution is done in str.format before the resource uses it.

2. You really need to wrap only xa.execute() into exception handler
because everything else is preparatory work, it does very little and
shouldn't throw:

--------------------------------
xa = pmnc.transaction.create()
xa.resource1.method1(args1)
xa.resource2.method2(args2)
...
xa.resourceN.methodN(argsN)
try:
    xa.execute()
except:
    ...
--------------------------------

3. You should be using pmnc.log.error for reporting errors:
--------------------------------
try:
    xa.execute()
except:
    pmnc.log.error("foo")
--------------------------------
There also are (in decreasing level order): pmnc.log.message(),
pmnc.log.error(), pmnc.log.warning(), pmnc.log(), pmnc.log.info(),
pmnc.log.debug()

> be used without try/catch blocks, and exceptions were instead routed
> through
> the callback function? Just a thought.

I think it is a bad idea. A failing transaction throws, just like any
other executable statement does. You still may have it your way if you
write a simple exception handler.

> If I wanted to accept parameters as you do with the execute() function,
> then
> turn around and pass them to you, is there a way to do that?

You must be talking about kwargs syntax. For a function to accept any
number of positional and/or keyword arguments and be able to pass them
along in a bunch you may do this:

--------------------------------
def foo(a, b, *, k):
    print(a, b, k)
def bar(*args, **kwargs): <<<<<<<<<<<< here
    print(args, kwargs)
    foo(*args, **kwargs)
bar(1, 2, k = "v")
--------------------------------

this prints

--------------------------------
(1, 2) {'k': 'v'}
1 2 v
--------------------------------

If you have any further questions please ask.

Sincerely,
Dmitry Dvoinikov

Re: [Pythomnic3k-questions] asynchronous loop event with pyqt

[Dmitry D. <dm...@ta...>]

Hello Olivier,

It is somewhat awkward to use middleware framework for GUI applications,
but there is no inherent problem with that. In Pythomnic terms whatever
it is that throws in requests is an interface. Therefore you are likely
looking at implementing a special interface protocol "pyqt", which
handles all the GUI magic such as event loop internally, and then,
whenever for example a button is pushed, or another appropriate
event happens, throws in a request in regular Pythomnic manner.
For a framework there is no difference from where the request arrived.

Now, I have no idea about threading architecture of pyqt, but you can
try launching a separate thread owned by the "pyqt" interface just
like many other protocols do (for a simple example see
protocol_file.py::Interface) and in that thread spin up pyqt.
Alternatively, you can modify startup.py to add code executed
by the main thread at service startup. Let me know if you have any
questions with Pythomnic side of that puzzle.

Sincerely,
Dmitry Dvoinikov

On 25.10.2010 23:21, ol...@le... wrote:
> Hi,
> I would use pythomnic3k with pyqt4, but is it possible to run
> asynchronous loop event with pyqt?
>
> thanks
>
>
>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America contest
> Create new apps&  games for the Nokia N8 for consumers in  U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
> _______________________________________________
> Pythomnic3k-questions mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pythomnic3k-questions
>

[Pythomnic3k-questions] asynchronous loop event with pyqt

[<ol...@le...>]

Hi,
I would use pythomnic3k with pyqt4, but is it possible to run 
asynchronous loop event with pyqt?

thanks

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[<dm...@te...>]

> What is the functional difference between these two?
>
> xa = pmnc.transaction.create()
> xa.pg.execute(sql1,sql2,param1 = value1)
> pg_result = xa.execute()[0]
>
> and
>
> xa = pmnc.transaction.create()
> xa.pg.execute(sql1,param1 = value1)
> xa.pg.execute(sql2,param1 = value1)
> pg_result = xa.execute()[0]

In the first case both SQL queries are executed through the same database
connection within the same PostgreSQL transaction. In the second case the
two SQL queries are each executed in its own separate database connection
in separate PostgreSQL transactions and therefore they don't correlate in
any way from PostgreSQL point of view.

In Pythomnic transactions participants are independent. It is just like as
if you did

-------------------------------------
xa = pmnc.transaction.create()
xa.pg1.execute(sql1)
xa.pg2.execute(sql2)
xa.execute()
-------------------------------------

So it may as well be separate databases.

Sincerely,
Dmitry Dvoinikov

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[<dm...@te...>]

> Hmmm, I'm thinking it's a 32 vs. 64-bit issue, perhaps?

Yes I think you are right. You may switch to 32-bit Python 3.2, but I was
unable to build BerkeleyDB 64 bit quickly so I stuck with 32-bit version,
makes no difference to me in alpha release testing.

> would have to really turn into something like this:
>
> 	try:
> 		xa = pmnc.transaction.create()
> 		xa.pg.execute("SELECT id FROM users WHERE
> name='{name}'",name=Myname)
> 		pg_result = xa.execute()[0]
> 	except:
> 		pmnc.log (some stuff)
> 		{maybe do other stuff}

A few notes here:

1. You do NOT need to put keyword parameters in single quotes, because
they are not substituted literally. They are rather binding points. When
you execute

--------------------------------
xa.pg.execute("SELECT id FROM users WHERE name = {name}", name = "Eric")
--------------------------------

the database actually sees this:

"SELECT id FROM users WHERE name = ?"
params: (varchar("Eric"), )

and NOT this:

"SELECT id FROM users WHERE name = Eric"

Do not treat keyword parameters as literals. If you need to do so, do this:

--------------------------------
xa.pg.execute("SELECT id FROM users WHERE "
              name = '{name}'".format(name = "Eric"))
--------------------------------

then the SQL substitution is done in str.format before the resource uses it.

2. You really need to wrap only xa.execute() into exception handler
because everything else is preparatory work, it does very little and
shouldn't throw:

--------------------------------
xa = pmnc.transaction.create()
xa.resource1.method1(args1)
xa.resource2.method2(args2)
...
xa.resourceN.methodN(argsN)
try:
    xa.execute()
except:
    ...
--------------------------------

3. You should be using pmnc.log.error for reporting errors:
--------------------------------
try:
    xa.execute()
except:
    pmnc.log.error("foo")
--------------------------------
There also are (in decreasing level order): pmnc.log.message(),
pmnc.log.error(), pmnc.log.warning(), pmnc.log(), pmnc.log.info(),
pmnc.log.debug()

> be used without try/catch blocks, and exceptions were instead routed
> through
> the callback function? Just a thought.

I think it is a bad idea. A failing transaction throws, just like any
other executable statement does. You still may have it your way if you
write a simple exception handler.

> If I wanted to accept parameters as you do with the execute() function,
> then
> turn around and pass them to you, is there a way to do that?

You must be talking about kwargs syntax. For a function to accept any
number of positional and/or keyword arguments and be able to pass them
along in a bunch you may do this:

--------------------------------
def foo(a, b, *, k):
    print(a, b, k)
def bar(*args, **kwargs): <<<<<<<<<<<< here
    print(args, kwargs)
    foo(*args, **kwargs)
bar(1, 2, k = "v")
--------------------------------

this prints

--------------------------------
(1, 2) {'k': 'v'}
1 2 v
--------------------------------

If you have any further questions please ask.

Sincerely,
Dmitry Dvoinikov

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <Er...@Th...>]

I remember the question I had on the transactions:

What is the functional difference between these two?

xa = pmnc.transaction.create()
xa.pg.execute(sql1,sql2,param1 = value1)
pg_result = xa.execute()[0]

and

xa = pmnc.transaction.create()
xa.pg.execute(sql1,param1 = value1)
xa.pg.execute(sql2,param1 = value1)
pg_result = xa.execute()[0]

Is the second form even allowed, or is only one execute call allowed per
resource, per transaction?

When I tried a 2-part transaction, where sql1 was an INSERT and sql2 was a
SELECT on the items just inserted, it worked when passed in the first form
(both SQL queries in one execute() call), but failed when I tried the second
form (each query in its own execute() call). In the second case, the INSERT
was not recognized by the SELECT (came up with nothing found).

Eric

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <Er...@Th...>]

Hmmm, I'm thinking it's a 32 vs. 64-bit issue, perhaps? My test results in:

Python 3.2a2 (r32a2:84522, Sep  5 2010, 22:03:10) [MSC v.1500 64 bit
(AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import bsddb3
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    import bsddb3
  File "C:\Python\lib\site-packages\bsddb3\__init__.py", line 59, in
<module>
    exec("from . import _pybsddb")
  File "<string>", line 1, in <module>
ImportError: DLL load failed: %1 is not a valid Win32 application.

Regarding the transaction code, your interface is quite easy and clean and I
went back and forth on just using it straight vs. "wrapping" it. In the end,
it was the error processing that swung me, because in order to use the xa
stuff "raw" I found it was getting to where I'd have many lines of code
where one or two would do, because this:

   q=sql("SELECT id FROM users WHERE name='{name}'".format(name=Myname))
   records=q.execute()[0]

would have to really turn into something like this:

	try:
		xa = pmnc.transaction.create()
		xa.pg.execute("SELECT id FROM users WHERE
name='{name}'",name=Myname)
		pg_result = xa.execute()[0]
	except:
		pmnc.log (some stuff)
		{maybe do other stuff}

I find it's helpful for me to encapsulate the error handling of the
transactions to a single function that can deal with the basic DB errors and
log them consistently, rather than have those try/catch blocks everywhere.
It might be interesting if your transaction code could accept a callback
function that was raised on an exception, so that the transaction code could
be used without try/catch blocks, and exceptions were instead routed through
the callback function? Just a thought.

If I wanted to accept parameters as you do with the execute() function, then
turn around and pass them to you, is there a way to do that?

-----Original Message-----
From: dm...@te... [mailto:dm...@te...] 
Sent: Saturday, October 16, 2010 5:00 AM
To: Eric Livingston
Cc: pyt...@li...
Subject: Re: [Pythomnic3k-questions] Changing the server address in
Resource_http

> Maybe it's the DB version? I have the following installed:
> Berkely DB 11gR2 Version 5.1.19

It doesn't matter. I link the pybsddb library statically so it doesn't use
any local DLLs. Once you have it unpacked to
C:\Python32\Lib\site-packages, here's what should be working, try it:

------------------------------------------
C:> python
Python 3.2a2 (r32a2:84522, Sep  5 2010, 22:35:34) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb3
>>> bsddb3.__version__
'5.0.0'
>>> bsddb3.db.version()
(5, 0, 21)
>>> ^Z
------------------------------------------

> I can do this:
>         q=sql("SELECT id FROM users WHERE
> name='{name}'".format(name=Myname))
>         records=q.execute()[0]
> or
>         q=sql("INSERT INTO users (name) VALUES
> ('{ownername}')".format(name=Myname))
>         q.add("SELECT id FROM users WHERE
> name='{name}'".format(name=Myname))
>         records=q.execute()[1]
>
> For 99% of my db accesses, that's all I need, so hiding the rest of the
> transaction stuff cleans it up a lot for me. I'd have preferred passing
> the
> query parameters as you've done, but I can't work out how to do that!

The syntax for any Pythomnic transaction is this:

------------------------------------------
xa = pmnc.transaction.create()
xa.resource1.method1(args1)
xa.resource2.method2(args2)
...
xa.resourceN.methodN(argsN)
result1, result2, ..., resultN = xa.execute()
------------------------------------------

where N resources execute their private transactions in parallel and then
perform a group commit. The results are returned in N-tuple.

If you have just one participant - SQL database, it becomes this:

------------------------------------------
xa = pmnc.transaction.create()
xa.pg.execute(sql1, sql2, ..., sqlP,
              param1 = value1, ..., paramQ = valueQ)
pg_result = xa.execute()[0]
------------------------------------------

You can pass more than one SQL query to the execute() call, each is a
positional argument, they are all executed in one transaction and in that
order, for example:

------------------------------------------
xa = pmnc.transaction.create()
xa.pg.execute("SELECT f FROM tableA",
              "SELECT g FROM tableB")
pg_result = xa.execute()[0]
recordsA = pg_result[0]
recordsB = pg_result[1]
for r in recordsA:
    print(r["f"])
for r in recordsB:
    print(r["g"])
------------------------------------------

The individual result from the SQL participant is therefore a list of
lists of records.

Now, to parameter passing. This is the same with standard str.format
syntax (which you are using too !), where the parameters are passed as
keyword arguments:

"key = {value}".format(value = 123) yields "key = 123"

Here is an example:

------------------------------------------
xa = pmnc.transaction.create()
xa.pg.execute("SELECT f FROM tableA WHERE key = {value}",
              "SELECT g FROM tableB WHERE id > {value} - 1",
              value = 123)
xa.execute()
------------------------------------------

results in the following two SQL queries being executed in one transaction:

SELECT f FROM tableA WHERE key = 123
SELECT g FROM tableB WHERE id > 123 - 1

So even if you make your own wrapper similar to the class you mention, all
you can hide is xa = pmnc.transaction.create() and xa.execute(),
everything else will still be there, perhaps in a slightly different form.
I therefore suggest not to bother.

> So, I tried to insert this:
>     def add(s,query):
> 	  s.xa.commit() #Try to queue up a commit in the query list for the
> transaction
>         s.xa.pg.execute(query) #queue up another query in this transaction
>

You should never call methods such as commit or rollback manually, this is
done by the transaction code. I'd also advise using self instead of s by
convention.

If you have any further questions please ask.

Sincerely,
Dmitry Dvoinikov

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[<dm...@te...>]

> Maybe it's the DB version? I have the following installed:
> Berkely DB 11gR2 Version 5.1.19

It doesn't matter. I link the pybsddb library statically so it doesn't use
any local DLLs. Once you have it unpacked to
C:\Python32\Lib\site-packages, here's what should be working, try it:

------------------------------------------
C:> python
Python 3.2a2 (r32a2:84522, Sep  5 2010, 22:35:34) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb3
>>> bsddb3.__version__
'5.0.0'
>>> bsddb3.db.version()
(5, 0, 21)
>>> ^Z
------------------------------------------

> I can do this:
>         q=sql("SELECT id FROM users WHERE
> name='{name}'".format(name=Myname))
>         records=q.execute()[0]
> or
>         q=sql("INSERT INTO users (name) VALUES
> ('{ownername}')".format(name=Myname))
>         q.add("SELECT id FROM users WHERE
> name='{name}'".format(name=Myname))
>         records=q.execute()[1]
>
> For 99% of my db accesses, that's all I need, so hiding the rest of the
> transaction stuff cleans it up a lot for me. I'd have preferred passing
> the
> query parameters as you've done, but I can't work out how to do that!

The syntax for any Pythomnic transaction is this:

------------------------------------------
xa = pmnc.transaction.create()
xa.resource1.method1(args1)
xa.resource2.method2(args2)
...
xa.resourceN.methodN(argsN)
result1, result2, ..., resultN = xa.execute()
------------------------------------------

where N resources execute their private transactions in parallel and then
perform a group commit. The results are returned in N-tuple.

If you have just one participant - SQL database, it becomes this:

------------------------------------------
xa = pmnc.transaction.create()
xa.pg.execute(sql1, sql2, ..., sqlP,
              param1 = value1, ..., paramQ = valueQ)
pg_result = xa.execute()[0]
------------------------------------------

You can pass more than one SQL query to the execute() call, each is a
positional argument, they are all executed in one transaction and in that
order, for example:

------------------------------------------
xa = pmnc.transaction.create()
xa.pg.execute("SELECT f FROM tableA",
              "SELECT g FROM tableB")
pg_result = xa.execute()[0]
recordsA = pg_result[0]
recordsB = pg_result[1]
for r in recordsA:
    print(r["f"])
for r in recordsB:
    print(r["g"])
------------------------------------------

The individual result from the SQL participant is therefore a list of
lists of records.

Now, to parameter passing. This is the same with standard str.format
syntax (which you are using too !), where the parameters are passed as
keyword arguments:

"key = {value}".format(value = 123) yields "key = 123"

Here is an example:

------------------------------------------
xa = pmnc.transaction.create()
xa.pg.execute("SELECT f FROM tableA WHERE key = {value}",
              "SELECT g FROM tableB WHERE id > {value} - 1",
              value = 123)
xa.execute()
------------------------------------------

results in the following two SQL queries being executed in one transaction:

SELECT f FROM tableA WHERE key = 123
SELECT g FROM tableB WHERE id > 123 - 1

So even if you make your own wrapper similar to the class you mention, all
you can hide is xa = pmnc.transaction.create() and xa.execute(),
everything else will still be there, perhaps in a slightly different form.
I therefore suggest not to bother.

> So, I tried to insert this:
>     def add(s,query):
> 	  s.xa.commit() #Try to queue up a commit in the query list for the
> transaction
>         s.xa.pg.execute(query) #queue up another query in this transaction
>

You should never call methods such as commit or rollback manually, this is
done by the transaction code. I'd also advise using self instead of s by
convention.

If you have any further questions please ask.

Sincerely,
Dmitry Dvoinikov

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <Er...@Th...>]

Hmm, I tried the bsddb3 you had down there, but it still won't load. Not
much to go on:
18:26:21.35 MSG [cage] module state has been loaded (not reloadable)
18:26:21.37 MSG [cage] module bsddb3 could not be imported, persistent state
is disabled # state.py:268 in start()
Maybe it's the DB version? I have the following installed:
Berkely DB 11gR2 Version 5.1.19

This has been my experience with the postgresql interface. Right off the
bat, I decided I wanted to encapsulate all that transaction stuff, and the
error catching and such, so the interface could be cleaner for most of what
I had to do. As this is my first python project, I was unable to work out
how to take a variable parameters list and pass it on to your execute
command, so I had to alter it to something I could figure out. Here's what
I've wound up with (minus any error checking/handling for now):

class sql:
    def __init__(s,query):
        s.query=[query]

    def add(s,query):
        s.query.append(query)

    def execute(s)->list:
        s.xa = pmnc.transaction.create()
        if len(s.query)==1:
            s.xa.pg.execute(s.query[0])
        elif len(s.query)==2:
            s.xa.pg.execute(s.query[0],s.query[1])
        raw = s.xa.execute()
        records = raw[0]
        return records

As you can see, it's quite simple, and it can only handle two queries
"stacked up" so far, as I'm not sure how to dynamically pass an arbitrary
list of them to your execute function, as I was saying. However, with this
simple wrapper, I can do this:
        q=sql("SELECT id FROM users WHERE
name='{name}'".format(name=Myname))
        records=q.execute()[0]
or
        q=sql("INSERT INTO users (name) VALUES
('{ownername}')".format(name=Myname))
        q.add("SELECT id FROM users WHERE
name='{name}'".format(name=Myname))
        records=q.execute()[1]

For 99% of my db accesses, that's all I need, so hiding the rest of the
transaction stuff cleans it up a lot for me. I'd have preferred passing the
query parameters as you've done, but I can't work out how to do that!

One thing I had tried to do was this type of thing:
class sql:
    def __init__(s,query):
        s.xa = pmnc.transaction.create()
	  s.xa.pg.execute(query) #queue up underlying Postgres query

    def add(s,query):
        s.xa.pg.execute(query) #queue up another query in this transaction

    def execute(s)->list:
        return s.xa.execute()[0]

This *almost* worked - was simpler and could accommodate as many queries as
I wanted to queue up in the transaction, but unfortunately the results of
the first query were not committed before executing the second. So, in the
case of my "insert, then select" two-query thing, it always failed, as the
insert wasn't committed prior to the select.

So, I tried to insert this:
    def add(s,query):
	  s.xa.commit() #Try to queue up a commit in the query list for the
transaction
        s.xa.pg.execute(query) #queue up another query in this transaction

That did something, as I noticed a 'None' list was introduced into the
result coming back from the execute, but it didn't do anything interesting,
as the same problem persisted: the first query wasn't committed prior to the
second being done.

So, that's where I am so far!

-----Original Message-----
From: Дмитрий Двойников [mailto:dvo...@in...] 
Sent: Friday, October 15, 2010 1:25 AM
To: Eric Livingston
Cc: pyt...@li...
Subject: Re: [Pythomnic3k-questions] Changing the server address in
Resource_http

Eric,

 > I didn't seen a lot of activity in this list for many months;
 > are you still actively developing and supporting Pythomnic?

Yes, in fact I do. At the moment, I'm done committing a bunch
of changes for what's to become version 1.2. It passes all the
tests and I have upgraded a couple of development servers. I will
be upgrading customers' servers for a month and probably some
minor bugs will spring up. After that when it's all running
fine, I will release version 1.2. Late November I assume. If you
like, you can check out current trunk and see for yourself.

Not that the list has a lot of activity indeed. :)

 > So far it's been great - my greatest disappointment has been
 > that it doesn't seem to like any version of Berkeley DB on Windows

Thank you ! As far as Berkeley DB goes, you can download the version
that I have just built for testing with Python 3.2a3, here:

http://www.pythomnic3k.org/files/bsddb3-py32-i386.zip

One other similar problem is that pywin32 is not available
for Python 3.2 yet.

 > My "main" store is PostgreSQL, and pythomnic is certainly
 > helping me with that, which is great.

Same here. BTW, do you find Pythomnic's syntax for transactions
with SQL databases usable/intuitive ? Are there any problems ?

 > do I just run a single "cage" and rely on pythomnic
 > to generate threads when necessary of that cage and the resources

Note that maximum number of worker threads is configured
in config_interfaces.py, and will not be exceeded. Therefore
if you have thread_count = 10, at most 10 requests will be
processed by this cage simultaneously. All the outstanding
requests will be accepted, queued up and waiting.

 > or should I be running multiple instances of the http interface
 > cage I created as the "front end"?

You should really perform tests with _your_ workload. I have
tested a single http interface with http_load and such, and it
maxed out at ~150 req/sec., but request processing was just
a stub. On the other hand, if your requests take a second to run
on average, then with 10 parallel requests you will get 10 req/sec.
You may increase thread_count, test again and see. YMMV.

Anyway, although performance of Pythomnic under Python 3.2
is significantly better compared to that under Python 3.1,
Python is still largely single-threaded. If single cage becomes
a bottleneck, and not all cores of your CPU are loaded, you
should run two or more identical cages, assign them to
different listening ports and distribute load across using
some sort of external balancer - pound, haproxy, varnish etc.

If you have any further questions please ask.

Sincerely,
Dmitry Dvoinikov

On 14.10.2010 17:50, Eric Livingston wrote:
> Dmitry,
>
> Luckily the connections I'm making are one-shot, disposable queries and
are
> short and single-purpose, though they are to a server pool of arbitrary
> size. It seems the second approach is the way to go, as there's no need
for
> SSL or for state persistence; the connections are created on-the-fly from
> incoming requests to the cage, and are thrown out at the end of the
> transaction. I had been falling back to the standard urllib to quickly
> generate the POSTs, but I'll give this a shot, as in general I'd like to
> keep things "in the framework" as much as possible!
>
> Thanks for the reply - I'll let you know how it goes to drop down a level
> and query that way.
>
> I didn't seen a lot of activity in this list for many months; are you
still
> actively developing and supporting Pythomnic? I stumbled across it looking
> for exactly this kind of framework, and frankly this is my first Python
> project - I picked the language largely due to pythomnic!
>
> So far it's been great - my greatest disappointment has been that it
doesn't
> seem to like any version of Berkeley DB I can get a hold of (I'm on
Windows,
> and don't have a build environment so I'm stuck with pre-compiled
> libraries.) I'm using Python 3.2 and your latest 1.1. So, I get an error
on
> that and am giving up any persistence I believe, and I'm not sure what
else.
> My "main" store is PostgreSQL, and pythomnic is certainly helping me with
> that, which is great.
>
> One final quick question: in terms of scaling to increasing traffic
volume,
> do I just run a single "cage" for my http interface, and rely on pythomnic
> to generate threads when necessary of that cage and the resources, or
should
> I be running multiple instances of the http interface cage I created as
the
> "front end"? I believe you handle all that, but I was able to start up two
> instances of the cage with no apparent error, so I'm not sure what I
should
> be doing.
>
> Thanks!
> Eric

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Дмитрий Д. <dvo...@in...>]

Eric,

 > I didn't seen a lot of activity in this list for many months;
 > are you still actively developing and supporting Pythomnic?

Yes, in fact I do. At the moment, I'm done committing a bunch
of changes for what's to become version 1.2. It passes all the
tests and I have upgraded a couple of development servers. I will
be upgrading customers' servers for a month and probably some
minor bugs will spring up. After that when it's all running
fine, I will release version 1.2. Late November I assume. If you
like, you can check out current trunk and see for yourself.

Not that the list has a lot of activity indeed. :)

 > So far it's been great - my greatest disappointment has been
 > that it doesn't seem to like any version of Berkeley DB on Windows

Thank you ! As far as Berkeley DB goes, you can download the version
that I have just built for testing with Python 3.2a3, here:

http://www.pythomnic3k.org/files/bsddb3-py32-i386.zip

One other similar problem is that pywin32 is not available
for Python 3.2 yet.

 > My "main" store is PostgreSQL, and pythomnic is certainly
 > helping me with that, which is great.

Same here. BTW, do you find Pythomnic's syntax for transactions
with SQL databases usable/intuitive ? Are there any problems ?

 > do I just run a single "cage" and rely on pythomnic
 > to generate threads when necessary of that cage and the resources

Note that maximum number of worker threads is configured
in config_interfaces.py, and will not be exceeded. Therefore
if you have thread_count = 10, at most 10 requests will be
processed by this cage simultaneously. All the outstanding
requests will be accepted, queued up and waiting.

 > or should I be running multiple instances of the http interface
 > cage I created as the "front end"?

You should really perform tests with _your_ workload. I have
tested a single http interface with http_load and such, and it
maxed out at ~150 req/sec., but request processing was just
a stub. On the other hand, if your requests take a second to run
on average, then with 10 parallel requests you will get 10 req/sec.
You may increase thread_count, test again and see. YMMV.

Anyway, although performance of Pythomnic under Python 3.2
is significantly better compared to that under Python 3.1,
Python is still largely single-threaded. If single cage becomes
a bottleneck, and not all cores of your CPU are loaded, you
should run two or more identical cages, assign them to
different listening ports and distribute load across using
some sort of external balancer - pound, haproxy, varnish etc.

If you have any further questions please ask.

Sincerely,
Dmitry Dvoinikov

On 14.10.2010 17:50, Eric Livingston wrote:
> Dmitry,
>
> Luckily the connections I'm making are one-shot, disposable queries and are
> short and single-purpose, though they are to a server pool of arbitrary
> size. It seems the second approach is the way to go, as there's no need for
> SSL or for state persistence; the connections are created on-the-fly from
> incoming requests to the cage, and are thrown out at the end of the
> transaction. I had been falling back to the standard urllib to quickly
> generate the POSTs, but I'll give this a shot, as in general I'd like to
> keep things "in the framework" as much as possible!
>
> Thanks for the reply - I'll let you know how it goes to drop down a level
> and query that way.
>
> I didn't seen a lot of activity in this list for many months; are you still
> actively developing and supporting Pythomnic? I stumbled across it looking
> for exactly this kind of framework, and frankly this is my first Python
> project - I picked the language largely due to pythomnic!
>
> So far it's been great - my greatest disappointment has been that it doesn't
> seem to like any version of Berkeley DB I can get a hold of (I'm on Windows,
> and don't have a build environment so I'm stuck with pre-compiled
> libraries.) I'm using Python 3.2 and your latest 1.1. So, I get an error on
> that and am giving up any persistence I believe, and I'm not sure what else.
> My "main" store is PostgreSQL, and pythomnic is certainly helping me with
> that, which is great.
>
> One final quick question: in terms of scaling to increasing traffic volume,
> do I just run a single "cage" for my http interface, and rely on pythomnic
> to generate threads when necessary of that cage and the resources, or should
> I be running multiple instances of the http interface cage I created as the
> "front end"? I believe you handle all that, but I was able to start up two
> instances of the cage with no apparent error, so I'm not sure what I should
> be doing.
>
> Thanks!
> Eric

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Eric L. <Er...@Th...>]

Dmitry,

Luckily the connections I'm making are one-shot, disposable queries and are
short and single-purpose, though they are to a server pool of arbitrary
size. It seems the second approach is the way to go, as there's no need for
SSL or for state persistence; the connections are created on-the-fly from
incoming requests to the cage, and are thrown out at the end of the
transaction. I had been falling back to the standard urllib to quickly
generate the POSTs, but I'll give this a shot, as in general I'd like to
keep things "in the framework" as much as possible!

Thanks for the reply - I'll let you know how it goes to drop down a level
and query that way.

I didn't seen a lot of activity in this list for many months; are you still
actively developing and supporting Pythomnic? I stumbled across it looking
for exactly this kind of framework, and frankly this is my first Python
project - I picked the language largely due to pythomnic!

So far it's been great - my greatest disappointment has been that it doesn't
seem to like any version of Berkeley DB I can get a hold of (I'm on Windows,
and don't have a build environment so I'm stuck with pre-compiled
libraries.) I'm using Python 3.2 and your latest 1.1. So, I get an error on
that and am giving up any persistence I believe, and I'm not sure what else.
My "main" store is PostgreSQL, and pythomnic is certainly helping me with
that, which is great.

One final quick question: in terms of scaling to increasing traffic volume,
do I just run a single "cage" for my http interface, and rely on pythomnic
to generate threads when necessary of that cage and the resources, or should
I be running multiple instances of the http interface cage I created as the
"front end"? I believe you handle all that, but I was able to start up two
instances of the cage with no apparent error, so I'm not sure what I should
be doing. 

Thanks!
Eric

-----Original Message-----
From: Дмитрий Двойников [mailto:dvo...@in...] 
Sent: Thursday, October 14, 2010 12:36 AM
Cc: pyt...@li...; Eric Livingston
Subject: Re: [Pythomnic3k-questions] Changing the server address in
Resource_http

Hello Eric,

That's a good question.

In Pythomnic3k resources are named instances of remote servers,
and access to each is decoupled and protected by a separate
connection pool of limited size. This works well if there is
one fixed server known upfront, not an arbitrary server, just
like you correctly pointed out.

You have two options here.

First is a hack, but is kind of ok if you have a handful
of servers you ever need to access, not thousands of them.
Even though you may not know the target address beforehand,
you can create configuration files at runtime and use them
immediately. Something like this:

-------------------------------------------

config_filename = os.path.join(__cage_dir__, "config_resource_foo.py")

with open(config_filename, "w") as f:
   f.write("""\
config = dict
(
protocol = "http",
server_address = ({addr}, {port}),
...
)
...
""".format(addr = "foo.com", port = 80))

xa = pmnc.transaction.create()
xa.foo.get("/")
xa.execute()

-------------------------------------------

But this won't work if you have to access thousands of servers,
possibly over time. For one, the cage directory will be cluttered
with one-time files. For two, the created connection pools will
hang around in memory, even though they would be empty.

So, the second option is to give up connection pools altogether
and use the HTTP client code from protocol_http.py directly.
Something like this:

-------------------------------------------

r = pmnc.protocol_http.Resource("foo",
          server_address = ("foo.com", 80),
          connect_timeout = 3.0,
          ssl_key_cert_file = None, # or filename
          ssl_ca_cert_file = None,  # or filename
          extra_headers = { "connection": "close" },
          http_version = "HTTP/1.1")

r.connect()
try:
	result = r.get("/")
finally:
     r.disconnect()

-------------------------------------------

See health_monitor.py (line 199 in version 1.1) for an example.
See, health_monitor is in exactly that position, because it
doesn't know the locations of cages it will be probing,
and so it creates resources manually.

But now there is a catch. The example works well for one-time
requests, when TCP connection is closed after each request.
But if you need persistent connections, for example you need
HTTPS and you don't want to renegotiate SSL session for each
request, you need to keep the created resources and manage
their lifetime manually. Essentially, you will need to
reimplement part of connection pools code that you threw away.

Something like this:

-------------------------------------------

rs = {} # I'm not bothering with interlocking
         # thread access in this example

...

r = rs.get("foo")
if not r:
   r = pmnc.protocol_http.Resource("foo", ...)
   rs["foo"] = r
		
... use r ...

if r.expired():
   r.disconnect()      	
   del rs["foo"]
	
-------------------------------------------

And so on and so forth.

If you have any further questions, please ask.

Sincerely,
Dmitry Dvoinikov

On 13.10.2010 9:58, Eric Livingston wrote:
> I have an application that needs to use dynamically-generated URLs
> (including server address) as a resource. I can't seem to figure out a
> way to do that, however, as the server address and port seem to be
> hard-coded into the config file up front.
>
> Is there any way I can work things so that upon creating a transaction
> using the http resource, I can specific at that time what server address
> to use?
>
> Thanks,
>
> Eric
>
> P.S. - Sorry if this posts twice; I initially sent from a non-subscribed
> email address
>
>
>
>
----------------------------------------------------------------------------
--
> Beautiful is writing same markup. Internet Explorer 9 supports
> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2&  L3.
> Spend less time writing and  rewriting code and more time creating great
> experiences on the web. Be a part of the beta today.
> http://p.sf.net/sfu/beautyoftheweb
>
>
>
> _______________________________________________
> Pythomnic3k-questions mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pythomnic3k-questions

Re: [Pythomnic3k-questions] Changing the server address in Resource_http

[Дмитрий Д. <dvo...@in...>]

Hello Eric,

That's a good question.

In Pythomnic3k resources are named instances of remote servers,
and access to each is decoupled and protected by a separate
connection pool of limited size. This works well if there is
one fixed server known upfront, not an arbitrary server, just
like you correctly pointed out.

You have two options here.

First is a hack, but is kind of ok if you have a handful
of servers you ever need to access, not thousands of them.
Even though you may not know the target address beforehand,
you can create configuration files at runtime and use them
immediately. Something like this:

-------------------------------------------

config_filename = os.path.join(__cage_dir__, "config_resource_foo.py")

with open(config_filename, "w") as f:
   f.write("""\
config = dict
(
protocol = "http",
server_address = ({addr}, {port}),
...
)
...
""".format(addr = "foo.com", port = 80))

xa = pmnc.transaction.create()
xa.foo.get("/")
xa.execute()

-------------------------------------------

But this won't work if you have to access thousands of servers,
possibly over time. For one, the cage directory will be cluttered
with one-time files. For two, the created connection pools will
hang around in memory, even though they would be empty.

So, the second option is to give up connection pools altogether
and use the HTTP client code from protocol_http.py directly.
Something like this:

-------------------------------------------

r = pmnc.protocol_http.Resource("foo",
          server_address = ("foo.com", 80),
          connect_timeout = 3.0,
          ssl_key_cert_file = None, # or filename
          ssl_ca_cert_file = None,  # or filename
          extra_headers = { "connection": "close" },
          http_version = "HTTP/1.1")

r.connect()
try:
	result = r.get("/")
finally:
     r.disconnect()

-------------------------------------------

See health_monitor.py (line 199 in version 1.1) for an example.
See, health_monitor is in exactly that position, because it
doesn't know the locations of cages it will be probing,
and so it creates resources manually.

But now there is a catch. The example works well for one-time
requests, when TCP connection is closed after each request.
But if you need persistent connections, for example you need
HTTPS and you don't want to renegotiate SSL session for each
request, you need to keep the created resources and manage
their lifetime manually. Essentially, you will need to
reimplement part of connection pools code that you threw away.

Something like this:

-------------------------------------------

rs = {} # I'm not bothering with interlocking
         # thread access in this example

...

r = rs.get("foo")
if not r:
   r = pmnc.protocol_http.Resource("foo", ...)
   rs["foo"] = r
		
... use r ...

if r.expired():
   r.disconnect()      	
   del rs["foo"]
	
-------------------------------------------

And so on and so forth.

If you have any further questions, please ask.

Sincerely,
Dmitry Dvoinikov

On 13.10.2010 9:58, Eric Livingston wrote:
> I have an application that needs to use dynamically-generated URLs
> (including server address) as a resource. I can’t seem to figure out a
> way to do that, however, as the server address and port seem to be
> hard-coded into the config file up front.
>
> Is there any way I can work things so that upon creating a transaction
> using the http resource, I can specific at that time what server address
> to use?
>
> Thanks,
>
> Eric
>
> P.S. – Sorry if this posts twice; I initially sent from a non-subscribed
> email address
>
>
>
> ------------------------------------------------------------------------------
> Beautiful is writing same markup. Internet Explorer 9 supports
> standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2&  L3.
> Spend less time writing and  rewriting code and more time creating great
> experiences on the web. Be a part of the beta today.
> http://p.sf.net/sfu/beautyoftheweb
>
>
>
> _______________________________________________
> Pythomnic3k-questions mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pythomnic3k-questions