> I'm definitely available for this and I have my head fairly deep in the
> ZTC/PTC/zope.testing internals so I think I can be helpful.
So, I was thinking that over this week, I may create a skeleton
plone.testing in svn and add some "science fiction" documentation to it,
outlining how I envisage it might work. Then we can see if we can make
any of that reality, and also get feedback on how the "new" approach
will look to end users.
> If there's interest, I can morph collective.testcaselayer into these
I think that'd be a good idea. I'd maybe rename some of the c.tcl
classes a little to fit with a "new" world, but the basic concept is
sound and makes sense in this context.
> Of course since collective.testcaselayer stitches ZTC/PTC
> into layer classes, whatever bits of convenience come from that
> integration will have to be copied from ZTC/PTC into these new
> I don't think this would be a problem since the convenience
> implementations are the simple part and the complicated bits are the
> bits we *want* to ditch here, ages of testing cruft.
>> For each of these, I see the following layers being available:
>> - No layer - just a plain unit test
>> - No layer, but full teardown of the ZCA after each test. This makes
>> it safe to use e.g. provideAdapter() in a test.
>> - A minimal ZCA layer that hooks up things like the object event
>> - A basic ZCA layer that also registers the core ZCML directives
>> (those in the zope and browser namespaces)
>> - A Zope 2 layer that wires up Zope 2's configuration, but not to the
>> point of setting up users or a ZODB root.
>> - A basic ZODB layer that makes a ZODB root available.
>> - A combination of the minimal Zope 2 layer + basic ZODB layer, with a
>> default user installed. This would be more or less like a vanilla Zope 2
>> - A Plone layer that configures the minimum possible to get a full
>> Plone site. In particular, this should *not* auto-load site.zcml or all
>> of Products.* or anything included with z3c.autoinclude's plugin
>> mechanism. It also should not install any content or create any users.
>> As far as possible, this should be isolated from the possibly-changing
>> Plone default policies, e.g. it should use a well-defined (or not)
>> workflow. This would probably also need before-test and after-test hooks
>> so that we could do things like set the site hook.
>> - A ZServer layer that starts up a ZServer for things like Selenium or
>> Windmill testing. This would probably be used in combination with either
>> the "vanilla" Zope 2 layer or the Plone site layer.
> I very much agree with this list of layers/fixtures. I do have concern
> about creating testing only factoring that is so separate from the
> internals of the deployed app that we too often develop issues where
> passing tests don't actually reflect working deployed code. For myself,
> a test that will actually reflect the workings of the deployed app but
> is more expensive/less isolated will always win over a test that is less
> expensive/more isolated but I can't count on passing meaning working
> deployed code.
Maybe we need two things here. Let me answer below.
> My dream for cleaning up testing has always been to define "layers" (not
> in the technical zope.testing sense) in the *application* that our
> testing layers just happen to setup, where much of the factoring/code is
> shared between the tests and the deployed app. I'm not sure how much of
> this we can meaningfully do since we can't afford to target Plone 5 only
> for this and the time is past for making changes to Zope2 for Plone 4.
I think we should target Zope 2.12 + Plone 4 at this stage.
> The ZCML loading could be handled this way, we refactor our ZCML into
> discreet files representing the different layers where those files are
> loaded nearly identically in both the testing environment and deployment
> environments. But non-ZCML layering would be difficult to do without
> making changes to Zope2 or changes to Plone which would be inappropriate
> for Plone 4 at this point in the release cycle.
Yes. I still think we can have sensible test layers, though. It may just
be a little bit harder.
> Another underlying problem with this is that Zope2/Plone themselves are
> too monolithic and entangled. We have a rich set of functionality grown
> by a vibrant community at the expense of the kind of factoring and
> isolation that can give a degree of confidence that what works in one
> "layer" will not be interfered with by other functionality when
> deployed. A possible approach here is to define minimum testing
> dependencies, "This suite needs this layer to run *at all*", but allow
> the test runner to run suites with other/all layers installed when
> looking for surprising interactions. This would be *wonderful* in my
> daily work as it would allow me to address the pain of how long my tests
> take in the inner testing loop without sacrificing the option to be able
> to run tests which I *can* count on to expose surprising interactions.
Okay, so to your earlier point, let me explain why I want a "minimal"
Plone setup. We had a test in our project last week that had been
passing for weeks, and then suddenly started breaking with weird setup
errors. Why? Because our test used PloneTestCase, it loaded Plone's
ZCML. That in turn loaded all installed packages using z3c.autoinclude's
plugin mechanism. We'd just added a new package that used this loading.
That package depended on a Products.* package that had to be initialised
in the Zope 2 product sense. Since the test case in our other package
hadn't done ztc.installProduct(...) for that one, completely unrelated
product, its tests broke.
So, whilst I agree that it'd be useful to have a layer that is as close
to runtime as possible (e.g. one where *all* packages are installed and
configured just like Zope does on startup), I think that for many
packages, that adds implicit dependencies that are harmful. If I were
writing and releasing a re-usable Plone add-on, I'd want to test it with
a minimal, isolated profile. If I'm working on a project-specific
package (e.g. a theme or policy installation product), I'd want to work
on the full-stack layer.
>> In addition, I think we should have collective.testcaselayer-style
>> infrastructure for defining and adding layers, including some layer
>> helpers for doing common things like adding users or installing
>> products. I've made some suggestions below. These will obviously only
>> make sense in some layers, and sometimes be more useful in tests than in
>> layers (though I think they should be available in both).
>> - Load the ZCML for a package
>> - Install a Zope 2-style product (is that still necessary?) - no
>> difference between "products" and "packages"!
> I believe there are still differences. For one, a package needs to have
> it's ZCML loaded so it can be registered as a Zope2 package before it
> can be Zope2-installed.
True. And I don't think loading Product.Five's ZCML (and thus site.zcml
and thus all of Products.*'s ZCML) is a good idea. I think we need
helpers that only load individual packages and their configuration in
> I'm not sure if there are cases where this
> would ever make a difference. Also, last I looked, installPackage and
> installProduct don't have shared code, so it might not be best to assume
> they're the same. Of course, *best* would be to *make* them the same in
> Zope2. :)
I think we need to make a helper that picks the right one to call. It's
incredibly arbitrary to someone who's not been doing Zope for 5 years
that these should be different. But it's also weird that
"installPackage" is not the same as "load ZCML" and is not the same as
"install into Plone site". I'd much rather we had a helper that let the
user list dependencies and have them configured appropriately for each
layer, e.g. a ZCML layer could include those packages, whilst a Zope 2
layer could initialise them as products.
>> - Add a user to the root user folder
>> - Add a user to the user folder in a Plone site.
>> - Apply an extension profile (maybe easier to just use portal_setup
>> - Set roles for a user
>> - Set a current authenticated user
>> - Clear the current authenticated user
> I agree with the list of helpers. Most of these helpers could be copied
> nearly verbatim from ZTC/PTC. I'm in favor of maintaining the existing
> names as it greatly increases the ease with which the add-on community
> can convert their existing tests as appropriate.
>> Right now, there's a lot of confusion around how to properly create
>> layers, how to register test suites (think ZopeDocFileSuite vs.
>> DocFileSuite ), etc. I think we want a few basic patterns. This may be
>> more about documentation that code, though.
>> - Create a suite of all TestCase-style tests in a module
>> (unittest.defaultTestLoader helps here)
>> - Create a suite of all doctests in another module, with a layer (in
>> one call)
>> - Create a doctest with a layer (in one call)
> Can you give some pseudo code demonstrating the semantics you'd like to
I'll do that in the "science fiction" documentation, but basically:
Right now, you can't pass a layer to the doctest constructors, so you
have to create them and then set a .layer attribute, or pass a
test_class, which is weird.
>> - ZODB sandboxing for functional tests. It needs to be quite clear
>> where transaction rollback and other forms of sandboxing is happening.
>> We have to support both the normal case, where transactions are rolled
>> back for each test, and the more complex scenario where we have
>> functional tests using things like testbrowser, Selenium, Windmill that
>> execute several requests that may result in commits.
> Of all the things mentioned, I'd expect this to be the worst single
> rat's nest, but very important to do. c.tcl provides what I think is a
> very good layer based sandboxing, so if that's acceptable (after ripping
> out the ZTC dependencies), then what's left is the ZTC functional
> testing bits that patch the publisher and testbrowser.
I was hoping you'd say that. ;-)
>> Okay, I think that's enough for now. I'm quite interested in helping
>> with this effort (obviously), in particular around quality control -
>> making sure the approaches are consistent, obvious and well documented
>> enough to be useful. I'm also willing to help with code, although I
>> think others (like Stefan and Ross) are better able to interpret the
>> basic infrastructure.
>> We're all affected by this, and I'm sure that if we added up the time we
>> all spend looking up code samples and hunting down poorly constrained
>> tests, we could easily justify spending a bit of time making a better
>> toolset for ourselves. :)
> Thanks for kicking this off!
And thanks for volunteering. ;)
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book