When running a test using the Selenium plugin, 'element not found' exceptions are classified as 'Exceptions' (ie. adding the expectException="true" attribute will interpret such a situation as a successful test). My question is, shouldn't such an event be classified as a 'failure' instead?
For example: I'm testing a webapplication's search function. Steps are:
1> input search query in input field
2> click on searchbutton
3> check the value of a certain HTML element.
These steps are executed using data from a CSV file. One of the values of the CSV file is used as value for the attribute expectFailure. This way I can test input queries which shouldn't return search results.
The problem with this approach is that step 3 can 'fail' in two different ways. When there are search results, the targeted element could have an incorrect value resulting in a Fail. But when there are no search results, the element does not exists which results in an Exception.
This means that I should have 2 values in my CSV file, one for expectFailure (when the element's value is supposed to be wrong or not) and one for expectException (when the element should or should not be on the page). However, in this situation the separation of these events seem a bit redundant. Because if the HTML element doesn't exist, the non-existant value cannot be correct as well.
However, when both expectException and expectFailure are set to "true" when no search results are found, Jameleon reports an expected failure did not happen. So it is required that both expectFailure and expectException are to be explicitly set in order to define this test successfully, which needlessly complicates the definition and data of the test.
That brings back the question: shouldn't 'exceptions' like these be handled as Failures? Alternatively, is it a good idea to add a catch-all attribute which covers both Fails and Exceptions (for instance, a expectExceptionOrFailure attribute)?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The mentioned redundancy isn't a real issue for 'click' and 'type', as the only failure coming from those actions can be an Exception. Although having them return Failures is maybe a bit more consistent. But for select and assert-field-value-equals there are two possible outcomes.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Use the tag below on a page that *doesn't* have an element that corresponds with the locator attribute value.
<selenium-assert-field-value-equals locator="bogus_element" value="asdf" expectFailure="true" functionId="assert-field-value "/>
This will generate a SeleniumException. Notice that when you run this test, the testreport displays: 'Expected a failure that did not happen'.
I've (temporarily) fixed this in a local build, by adding a try/catch around the session.getValue call in SeleniumAssertFieldValueEqualsTag.java.
Here's the stacktrace:
com.thoughtworks.selenium.SeleniumException: ERROR: Element bogus_element not found
at com.thoughtworks.selenium.HttpCommandProcessor.doCommand(HttpCommandProcessor.java:73)
at com.thoughtworks.selenium.HttpCommandProcessor.getString(HttpCommandProcessor.java:150)
at com.thoughtworks.selenium.DefaultSelenium.getValue(DefaultSelenium.java:302)
at net.sf.jameleon.plugin.selenium.tags.SeleniumAssertFieldValueEqualsTag.testBlock(SeleniumAssertFieldValueEqualsTag.java:65)
at net.sf.jameleon.function.FunctionTag.doTag(FunctionTag.java:276)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
at net.sf.jameleon.data.AbstractDataDrivableTag.executeDrivableRow(AbstractDataDrivableTag.java:274)
at net.sf.jameleon.data.DataExecuter.executeData(DataExecuter.java:79)
at net.sf.jameleon.data.AbstractDataDrivableTag.doTag(AbstractDataDrivableTag.java:388)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
at net.sf.jameleon.SessionTag.doTag(SessionTag.java:190)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
at net.sf.jameleon.TestCaseTag.invokeChildren(TestCaseTag.java:643)
at net.sf.jameleon.TestCaseTag$1.run(TestCaseTag.java:843)
at net.sf.jameleon.TestCaseTag.executeBody(TestCaseTag.java:896)
at net.sf.jameleon.TestCaseTag.executeNoCSV(TestCaseTag.java:841)
at net.sf.jameleon.TestCaseTag.doTag(TestCaseTag.java:789)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:706)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:670)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:579)
at net.sf.jameleon.ExecuteTestCase.runScript(ExecuteTestCase.java:273)
at net.sf.jameleon.ExecuteTestCase.executeJellyScript(ExecuteTestCase.java:238)
at net.sf.jameleon.ExecuteTestCase.execute(ExecuteTestCase.java:211)
at net.sf.jameleon.TestScriptTag.doTag(TestScriptTag.java:79)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
at net.sf.jameleon.TestSuiteTag.doTag(TestSuiteTag.java:102)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:706)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:670)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:579)
at net.sf.jameleon.ExecuteTestCase.runScript(ExecuteTestCase.java:273)
at net.sf.jameleon.ExecuteTestCase.executeJellyScript(ExecuteTestCase.java:238)
at net.sf.jameleon.ExecuteTestCase.execute(ExecuteTestCase.java:205)
at net.sf.jameleon.ExecuteTestCase.executeFiles(ExecuteTestCase.java:223)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thank you for the stack trace and the test case. can you please submit a bug against the Selenium plug-in so I don't forget to fix it. I am pretty busy this week so I won't be able to take a look at it until sometime this weekend.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Posted bug 'selenium-assert-field-value-equals throws unjust exception', ID 1685796. As the title says, this bug is mainly for assert-field-value-equals.
But what about the exceptions thrown by 'select', 'type' and 'click'? When these tags are pointed to non-existing elements, are they supposed to throw Exceptions or should they return Failures as well?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well the other ones you listed are action tags not validation tags. If you tell an action tag to do something and the application isn't in the correct state, then shouldn't an exception be thrown?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
When running a test using the Selenium plugin, 'element not found' exceptions are classified as 'Exceptions' (ie. adding the expectException="true" attribute will interpret such a situation as a successful test). My question is, shouldn't such an event be classified as a 'failure' instead?
For example: I'm testing a webapplication's search function. Steps are:
1> input search query in input field
2> click on searchbutton
3> check the value of a certain HTML element.
These steps are executed using data from a CSV file. One of the values of the CSV file is used as value for the attribute expectFailure. This way I can test input queries which shouldn't return search results.
The problem with this approach is that step 3 can 'fail' in two different ways. When there are search results, the targeted element could have an incorrect value resulting in a Fail. But when there are no search results, the element does not exists which results in an Exception.
This means that I should have 2 values in my CSV file, one for expectFailure (when the element's value is supposed to be wrong or not) and one for expectException (when the element should or should not be on the page). However, in this situation the separation of these events seem a bit redundant. Because if the HTML element doesn't exist, the non-existant value cannot be correct as well.
However, when both expectException and expectFailure are set to "true" when no search results are found, Jameleon reports an expected failure did not happen. So it is required that both expectFailure and expectException are to be explicitly set in order to define this test successfully, which needlessly complicates the definition and data of the test.
That brings back the question: shouldn't 'exceptions' like these be handled as Failures? Alternatively, is it a good idea to add a catch-all attribute which covers both Fails and Exceptions (for instance, a expectExceptionOrFailure attribute)?
This sounds more like a problem with the tag you are using.
Which tag is acting this way? For the most part, we should be doing an assertNotNull(element) on the element before doing anything on it.
This would then turn the exception into a failure.
Although I have not tested all of them, I can confirm this for the following tags when pointing them to non-existing elements:
selenium-select
selenium-click
selenium-type
selenium-assert-field-value-equals
The mentioned redundancy isn't a real issue for 'click' and 'type', as the only failure coming from those actions can be an Exception. Although having them return Failures is maybe a bit more consistent. But for select and assert-field-value-equals there are two possible outcomes.
correction: selenium-select can only have an exception as well. The main problem therefor is with assert-field-value-equals
I don't see how assert-field-value-equals can throw an Exception. What type of exception is it throwing?
Use the tag below on a page that *doesn't* have an element that corresponds with the locator attribute value.
<selenium-assert-field-value-equals locator="bogus_element" value="asdf" expectFailure="true" functionId="assert-field-value "/>
This will generate a SeleniumException. Notice that when you run this test, the testreport displays: 'Expected a failure that did not happen'.
I've (temporarily) fixed this in a local build, by adding a try/catch around the session.getValue call in SeleniumAssertFieldValueEqualsTag.java.
Here's the stacktrace:
com.thoughtworks.selenium.SeleniumException: ERROR: Element bogus_element not found
at com.thoughtworks.selenium.HttpCommandProcessor.doCommand(HttpCommandProcessor.java:73)
at com.thoughtworks.selenium.HttpCommandProcessor.getString(HttpCommandProcessor.java:150)
at com.thoughtworks.selenium.DefaultSelenium.getValue(DefaultSelenium.java:302)
at net.sf.jameleon.plugin.selenium.tags.SeleniumAssertFieldValueEqualsTag.testBlock(SeleniumAssertFieldValueEqualsTag.java:65)
at net.sf.jameleon.function.FunctionTag.doTag(FunctionTag.java:276)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
at net.sf.jameleon.data.AbstractDataDrivableTag.executeDrivableRow(AbstractDataDrivableTag.java:274)
at net.sf.jameleon.data.DataExecuter.executeData(DataExecuter.java:79)
at net.sf.jameleon.data.AbstractDataDrivableTag.doTag(AbstractDataDrivableTag.java:388)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
at net.sf.jameleon.SessionTag.doTag(SessionTag.java:190)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
at net.sf.jameleon.TestCaseTag.invokeChildren(TestCaseTag.java:643)
at net.sf.jameleon.TestCaseTag$1.run(TestCaseTag.java:843)
at net.sf.jameleon.TestCaseTag.executeBody(TestCaseTag.java:896)
at net.sf.jameleon.TestCaseTag.executeNoCSV(TestCaseTag.java:841)
at net.sf.jameleon.TestCaseTag.doTag(TestCaseTag.java:789)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:706)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:670)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:579)
at net.sf.jameleon.ExecuteTestCase.runScript(ExecuteTestCase.java:273)
at net.sf.jameleon.ExecuteTestCase.executeJellyScript(ExecuteTestCase.java:238)
at net.sf.jameleon.ExecuteTestCase.execute(ExecuteTestCase.java:211)
at net.sf.jameleon.TestScriptTag.doTag(TestScriptTag.java:79)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
at net.sf.jameleon.TestSuiteTag.doTag(TestSuiteTag.java:102)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:706)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:670)
at org.apache.commons.jelly.JellyContext.runScript(JellyContext.java:579)
at net.sf.jameleon.ExecuteTestCase.runScript(ExecuteTestCase.java:273)
at net.sf.jameleon.ExecuteTestCase.executeJellyScript(ExecuteTestCase.java:238)
at net.sf.jameleon.ExecuteTestCase.execute(ExecuteTestCase.java:205)
at net.sf.jameleon.ExecuteTestCase.executeFiles(ExecuteTestCase.java:223)
thank you for the stack trace and the test case. can you please submit a bug against the Selenium plug-in so I don't forget to fix it. I am pretty busy this week so I won't be able to take a look at it until sometime this weekend.
Posted bug 'selenium-assert-field-value-equals throws unjust exception', ID 1685796. As the title says, this bug is mainly for assert-field-value-equals.
But what about the exceptions thrown by 'select', 'type' and 'click'? When these tags are pointed to non-existing elements, are they supposed to throw Exceptions or should they return Failures as well?
Well the other ones you listed are action tags not validation tags. If you tell an action tag to do something and the application isn't in the correct state, then shouldn't an exception be thrown?