Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Automated testing - Sahi

Developers
Mike Mckay
2011-09-06
2013-03-08
  • Mike Mckay
    Mike Mckay
    2011-09-06

    First, I want to say what a great pleasure it was to meet everyone at the world conference on the weekend.  I had a great time and thoroughly enjoyed the proceedings and the people.  The meetings were very positive and I've come away feeling energized and excited about the future.

    On the flights home (3 legs this time), the time change was in my favor and I spent the time working out the details of a sahi test architecture based on our discussions.  I've committed some of the code to the contribution_mjmckay repository and we can move it to the main test repository once we agree on the approach.  Please have a look and let me know if I'm on the right track.

    One note: you will need to revert change 6679 from the 3.70 release branch in order to make these tests work with that release.  6679 removed all the readable text from the zk IDs which makes sahi hard to use.

    The test architecture is based on the following:
    * references to zk implementation of any element should only be made in one place.  (See fText() below.)
    * the high level test scripts should be written largely using wrappers that do not refer to sahi code.  (eg. fDocAction("Complete"))
    * high level wrappers should be writable with a minimum knowledge of sahi and detailed knowledge about the application dictionary and the column names.  (To set the Price List to standard, the field is a list and the column name is M_PriceList_ID.  The resultant call is fSetList("M_PriceList_ID","Standard");)
    * Strict visibility is assumed to be on at all times, unless specifically turned off when needed.

    Here are some examples of the high-level tests:

    // A simple test to show the use of the field functions to abstract
    // the sahi scripts.
    // 
    // Start from the webui login
    _include("./miscellaneous/fields.sah");
    _include("./miscellaneous/common_functions.sah");
    _setStrictVisibilityCheck(true);
    loginDefault("GardenAdmin","GardenAdmin");
    open_window("Purchase Order");
    new_record();
    fSetList("AD_Org_ID", "HQ");
    fSetList("M_Warehouse_ID", "HQ Warehouse");
    fSetSearch("C_BPartner_ID","Tree Farm Inc.");
    fSetList("M_PriceList_ID", "Purchase");
    fSetText("POReference","Test Order");
    fSetText("DateOrdered","09/04/2011");  //Date fields are a problem - the format is system dependent
    var $LongString = "This is an example of a long string.  It can go on and on for a long";
    $LongString = $LongString + " long time.";
    fSetTextArea("Description",$LongString);
    fSetList("SalesRep_ID","GardenAdmin");
    fDocAction("Prepare");  // Will fail - no lines created.
    save_record();
    close_window("Purchase Order");
    logout();
    _setStrictVisibilityCheck(false);
    // End of test.
    

    Here is an example of the low level functions for use with List type fields:

    // From fields.sah
    /********************************************************************
     *
     * fTextArea($FieldName)
     * 
     * Returns the element for the specified field
     *
     *******************************************************************/
    <browser>
    function fTextArea($FieldName){
        /* Double underscores required to prevent scheduling */
        __assertExists(__textarea("/^Field_"+ $FieldName + "/"), "Error: fTextArea() can't find field " + $FieldName);
        return __textarea("/^Field_"+ $FieldName + "/");
    }
    </browser>
    /********************************************************************
     *
     * fList($FieldName)
     * 
     * Returns the element for the specified field
     *
     *******************************************************************/
    <browser>
    function fList($FieldName){
        return fText($FieldName); /* Uses the same id as the text field */
    }
    </browser>
    /********************************************************************
     *
     * fGetList($FieldName)
     * 
     * Get the value/contents of a list field.
     *
     *******************************************************************/
    <browser>
    function fGetList($FieldName) {
        return __getText(fList($FieldName));
    }
    </browser>
    /********************************************************************
     *
     * fSetList($FieldName, $Value)
     * 
     * Set a list field to a value. No Requery.
     *
     *******************************************************************/
    function fSetList($FieldName, $Value){
    
        fSetListRq($FieldName, $Value, "N");
    }
    /********************************************************************
     *
     * fSetListRq($FieldName, $Value, $Requery)
     * 
     * Set a list field to a value. Option to Requery if $Requery = "Y"
     *
     *******************************************************************/
    function fSetListRq($FieldName, $Value, $Requery){
    
        if($Requery = "Y") {
            fRequeryList($FieldName);
        }
        _setValue(fList($FieldName), $Value);
        _removeFocus(fList($FieldName));
        _assertEqual($Value, fGetList($FieldName), "Error: fSetList() failed to set field " + $FieldName + " to value = " + $Value);
    }
    
    /********************************************************************
     *
     * fRequeryList($FieldName)
     * 
     * Requery a record
     *
     *******************************************************************/
    function fRequeryList($FieldName){
        _rightClick(fList($FieldName));
        _click(_link("Re&Query")); //Requery
    }
    
     
  • Mike Mckay
    Mike Mckay
    2011-09-06

    Sorry, I included fTextArea() instead of fText() in the last example.  Here is the correct snippet.

    /********************************************************************
     *
     * fText($FieldName)
     * 
     * Returns the element for the specified field
     *
     *******************************************************************/
    <browser>
    function fText($FieldName){
        /* Double underscores required to prevent scheduling */
        __assertExists(__textbox("/^Field_"+ $FieldName + "/"), "Error: f_text() can't find field " + $FieldName);
        return __textbox("/^Field_"+ $FieldName + "/");
    }
    </browser>
    
     
  • Hi Mike,
    thanks for the samples an clarifications, I think that they help a lot in starting to play with own tests.

    One note: you will need to revert change 6679 from the 3.70 release branch in order to make these tests work with that release. 6679 removed all the readable text from the zk IDs which makes sahi hard to use.

    Let me take little detour here to try and help ommiting some future problems (and because i like to be a smartass sometimes :-P):

    As mercurial is a distributed repository system, it is kind of dangerous to refer to a change by it's number, because the number is local and is most likely to be different among local clones.
    In this case, however it seems that both you and me have checked out a new local clone without any individual changes (at least not prior to the change set in question).
    Only therefore, I can be quite sure that we talk about this change:

    (01919bf50f30) The current class not generate the ID zk component, the new class is based on the ZK reference
    

    The important thing (which is the same among different clones) is the "global revision number": 01919bf50f30.
    You can find details here: http://hgbook.red-bean.com/read/a-tour-of-mercurial-the-basics.html (look for the section called "Changesets, revisions, and talking to other people").

    OK, now about the actual topic:

    @Victor: what is your opinion about this? Is it a bug or feature ;-)? Do you think we should revert this change in the development branch? It is a matter of "programming/scripting" sahi tests vs. "recording" them, right?
    If this is the case, and we can't (or don't want!) decide on one paradigma yet, maybe we can add an AD_System paramater for this?

    WDYT?

    Best Regards
    Tobi

     
  • Hi Mike , Tobi!

    About of revision :

    >(01919bf50f30) The current class not generate the ID zk component, the new class is based on the ZK reference

    Before the change ZK ID is not generated rightly, so that the identification was a random number. so that when you try to run the Sahi script again,  the script not work, the reason the fail is the random identification,  the bugfix allow to generate ZK ID with static reference  so that can be execute in any moment without the need the modification.

    I think that is possible the change of prefix of as the ZK references are generate to take the new structure ZK ID reference, so that  we can unifique two approach without some issue.

    kind regards
    Victor Perez
    www.e-evolution.com

     
  • Hi,

    if I understand the changes right, now the components (like textfields) just get a number as an id. The old version gave something like _textbox("Field_C_BPartner_Location_ID_186_1_1852!real"), the new one something like _textbox(13) which is hard to use in sahi scripts. That is because - as I understand - the number is still dependent on the layout and changes if you change the sequence of the shown elements but we no longer have the fields column name to identify it.

    I think the old way is better from the point of sahi tests. If we could find a way to use a combination of column name and the AD_Field_ID it would even be better.

    Best regards,
    Karsten

     
  • Daniel Prado
    Daniel Prado
    2011-11-01

    Hello,

    Excuse me for reply to an existant thread instead of creating a new one, but i can't find the option to do it. I'm an Ms on Computer Science student from Spain and i'm now working on my MS dissertation, which is intended to be a Functional Test Plan for Adempiere's HR module.

    I've ben reading the PMC:QA section from adempiere wiki, but it is a bit messy (or i'm a little dissoriented : ) )and  i'm a bit confused about the testing infraestructure that you are using now. I will be pleased if the job i'm doing would help to yours, so i want to do it using the same testing approach you are working on now.

    If you can tell me where would i found the test guidelines and testing framework you are using now, i will be very grateful.

    Thanks for your Time¡

                                                Daniel Prado

     
  • Mike Mckay
    Mike Mckay
    2011-11-02

    Hi Daniel,

    The QA stuff is all over the place at the moment and we don't have a test guide or framework in place.  The PMC team did a lot of work back in 2009/2010 to research solutions.  Red1 worked with Fitnesse, UISpec4J and Selenium and a number of us have used Sahi recently.  A full plan should probably use a combination of these tools.  Most of the efforts have been focused on trying to figure out what works for us and we have a ways to go yet.

    The goals are simple: we need to test the functions defined in http://www.adempiere.com/Functional_Tests in both the swing and zk interfaces.  We need the tests automated as part of the nightly build and they have to be maintainable.

    I've done some work to get a structure in place with Sahi which is starting to bear some fruit.  See http://adempiere.hg.sourceforge.net/hgweb/adempiere/contribution_mjmckay/ in the mercurial repository.  We'll move this into the main repository at some point.  If you can get a local install of ADempiere running, you can use the code in this repository to run some simple tests.  The file scenario/scenario_main_functional_tests.sah is the starting point.  So far, it will create new clients and test the creation process, but that is just the beginning.  The rest of the code provides more examples and supporting wrappers for the Sahi functions.  I'm trying to build a structure that allows the test scripts to be written simply and (if I can find the holy grail) that can be used on both interfaces with minimal modification.

    For the documentation, we could really use your academic approach.  Would you be willing to share your research with us?  I'm sure it would help us a lot.

    Mike.

     
  • Daniel Prado
    Daniel Prado
    2011-11-02

    Hi,

    Thanks for the quick answers. Now i´ve a local Adempiere install from source  and i´m exploring possible framework options for functional testing, so i will take a look to your sahi and selenium aproach. And, of course, i will be pleased to share my  research with you.

    Daniel

     
  • Suman
    Suman
    2012-01-04

    Hi Mike,
                   I have downloaded the scripts and tried to execute. But my observation is it is taking of time to execute each command. One example line is
                                     " _click(_div("Sales Order"));"

                  I think script will take lot of time as it has to seach big tree. is there a way that I will improve this.

                 One More thing, in AdempiereIDGenerator.java, I have the following code
    ######################################################################

    public class AdempiereIdGenerator implements IdGenerator {

    private static final String DEFAULT_ZK_COMP_PREFIX = "zk_comp_";
    private static final String DESKTOP_ID_ATTRIBUTE = "org.adempiere.comp.id";
    public static final String ZK_COMPONENT_PREFIX_ATTRIBUTE = "zk_component_prefix";

    @Override
    public String nextComponentUuid(Desktop desktop, Component comp) {
    String prefix = (String) comp.getAttribute(ZK_COMPONENT_PREFIX_ATTRIBUTE);
    if (prefix == null || prefix.length() == 0){
    prefix = DEFAULT_ZK_COMP_PREFIX;
    }
    else {
    Pattern pattern = Pattern.compile("");
    Matcher matcher = pattern.matcher(prefix);
    StringBuffer sb = new StringBuffer();
    while(matcher.find()) {
    matcher.appendReplacement(sb, "_");
    }
    matcher.appendTail(sb);
    prefix = sb.toString();
    }
    int i = 0;
    try {
    String number = null;
    if (desktop.getAttribute(DESKTOP_ID_ATTRIBUTE) != null) {
    number = desktop.getAttribute(DESKTOP_ID_ATTRIBUTE).toString();
    i = Integer.parseInt(number);
    i++;// Start from 1
    }
    } catch (Throwable t) {
    i = 1;
    }
    desktop.setAttribute(DESKTOP_ID_ATTRIBUTE, String.valueOf(i));
    if (!prefix.endsWith("_"))
    prefix = prefix + "_";
    return prefix +  i;

    }
    #########################################################################################

    is this code correct?

    Thanks,
    Ravuri

     
  • Mike Mckay
    Mike Mckay
    2012-01-04

    Hi Ravuri,

    Thanks for helping out.

    The speed of execution is dependent on the settings in your Sahi installation.  I use the following settings in userdata.properties and the execution is pretty fast except for number fields and the initial action after logging in, both of which remain painfully slow.  I've tried to reduce these settings further but run into errors when the test script is faster than the web site.

    # Script execution params
    #Time (in milliseconds) delay between steps
    script.time_between_steps=50
    #Time (in milliseconds) delay between retries once an error occurs
    script.time_between_steps_on_error=75
    #Number of retries once an error occurs
    script.max_reattempts_on_error=5
    #Number of cycles Sahi will wait for the page to load before it proceeds with execution
    #Time spent is (script.max_cycles_for_page_load x script.time_between_steps) milliseconds
    script.max_cycles_for_page_load=20
    #No of times Sahi should wait for stability of AJAX and page loads before continuing. min value is 1
    script.stability_index=1
    

    For the ID generator, I reverted the code in 3.7 to the previous version here http://adempiere.hg.sourceforge.net/hgweb/adempiere/adempiere/file/64800ef650d5/zkwebui/WEB-INF/src/org/adempiere/webui/AdempiereIdGenerator.java.

    The key is to generate IDs in a predictable way that are human readable.  I like being able to use a regular expression to find fields by the AD column name.  See the file fields.sah for examples.  In dialogs and in several other cases where there is no relation to a AD column name, the identification of fields is much harder.  The lookup and payment term dialogs are examples that where difficult to test.

    In the functional tests (http://www.adempiere.com/Functional_Tests) I'm at the point of establishing price lists and testing the requisition, PO, Invoice processes.  Up to now, the scenario only establishes the initial conditions for the tests - kind of like a mini implementation.  The code in scenario/scenario_main_functional_tests.sah shows the high level process.

    Let us know how it goes.

    Mike.

     
  • Hi,

    For the ID generator, I reverted the code in 3.7 to the previous version here

    Teo implemented a little fix to make the ID generator pluggable.

    Please refer to http://www.adempiere.com/FR3456545:_Pluggable_IdGenerator for further infos.
    I think Teo would be glad about your evaluation and feedback. Also, feel free to extend the FR wiki page.

    My understanding is that currently there is only a default implementaion (the one from the zk-Forum).
    Teo, can you confirm?

    Mike, maybe you can include that reverterted version of the ID generator which is required for your tests?
    You migh include it as class "org.adempiere.webui.DeterministicIdGenerator" (just a suggestion, feel free to to pick a better name).
    into the source folder \zkwebui\WEB-INF\src\.

    Best regards
    Tobi

     
  • Hi,

    it would be good if the id of the html element contains the ad_field_id.

    Regards
    Karsten

     
  • Teo Sarca
    Teo Sarca
    2012-01-06

    Hi guys,

    Long story told short: i just introduced flexibility to plug and ZK ID generator. As Tobi pointed out (thanks for creating the wiki page) the default one is SahiIdGenerator which is the old AdempiereIdGenerator from 3.7.
    Because I don't have experience with Sahi, i will need you guys to confirm if everything is ok but technical speaking everything shall work as it worked in the past. Nothing changed, just some renamings and ID generators are pluggable.

    Hope it helps.

    If you have any questions or issues, i am happy to support.

    Best regards,
    Teo Sarca

     
  • Mike Mckay
    Mike Mckay
    2012-01-09

    Hi Guys,

    The ID generator code is called at a pretty low level as the webui window is being created.  In the previous version of the generator, the code used a property ("zk_component_prefix") of the html component to identify what field the component was referring to.  This property was set in the WEditor class (org.adempiere.webui.editor)  that applied to the field.  Specifically,

        public WEditor(Component comp, GridField gridField)
        {
    ...
            this.setComponent(comp);
            comp.setAttribute("zk_component_prefix", "Field_" + gridField.getColumnName() + "_" + gridField.getAD_Tab_ID() + "_" + gridField.getWindowNo() + "_");
            this.gridField = gridField;
    ...
            init();
        }
    

    This info would be used by the ID generator to create a tag in the HTML.  Adding the AD_Field_ID to this would be simple.

    Mike.

     
  • Hi everyone,

    I am Narayan from the Sahi team. If you would like to discuss issues/solutions while automating with Sahi, please reach out to me, and I would be happy to help.

    Regards,
    Narayan

     
  • Mike Mckay
    Mike Mckay
    2012-01-10

    Hi Narayan,

    Thanks for the offer!  I have two questions:

    1.  What determines the order of execution of the test scripts when you execute _runTests()

    2. Why do calls to fSetAmount() in the following code run so slowly in comparison to nearly everything else.

    /********************************************************************
     *
     * fAmount($FieldName)
     * 
     * Returns the element for the specified field
     *
     *******************************************************************/
    <browser>
    function fAmount($FieldName){
        /* Double underscores required to prevent scheduling */
        __assertExists(__textbox(/^zk/, __in(__div("/^Field_" + $FieldName + "/"))), "Error: fAmount() can't find field " + $FieldName);
        return __textbox(/^zk/, __in(__div("/^Field_" + $FieldName + "/")));
    }
    </browser>
    /********************************************************************
     *
     * fGetAmount($FieldName)
     * 
     * Get the value/contents of an amount field.
     *
     *******************************************************************/
    <browser>
    function fGetAmount($FieldName) {
        return __getText(fAmount($FieldName));
    }
    </browser>
    /********************************************************************
     *
     * fSetAmount($FieldName, $Value)
     * 
     * Set a amount field to a value.
     *
     *******************************************************************/
    function fSetAmount($FieldName, $Value){
    
        _setValue(fAmount($FieldName), $Value);
        _removeFocus(fAmount($FieldName));
        _assertEqual($Value, fGetAmount($FieldName), "Error: fSetAmount() failed to set field " + $FieldName + " to value = " + $Value);
    }
    
     
  • Hi Mike,

    1) _runUnitTests does not guarantee an order. Sahi queries the javascript runtime for all loaded functions, and if they happen to start with "test" Sahi executes it. It is dependent on how Rhino returns the properties of the "global" object, which is, again, not  guaranteed to be ordered.

    2) The code does not seem to suggest anything specifically wrong. We may need to reproduce it here. Is there a test system (available online) and a test script which we can use to reproduce the problem? (I am sorry that we are hard pressed for time and would not be able to checkout and set up the environment here.)

    Regards,
    Narayan

     
  • Mike Mckay
    Mike Mckay
    2012-01-11

    Narayan,

    Thanks for the kind offer.  I'll set something up and get back to you in a week or so.

    Mike