[dojo-contributors] selenium vs. windmill

Mark Hays haysmark at us.ibm.com
Wed May 28 11:48:13 EDT 2008


Mikeal, that Controller API page is one of my biggest concerns as far as 
documentation goes: the API as given there is actually very sparse in 
comparison to the Selenium docs. Have you seen the Selenium docs lately?
http://selenium-core.openqa.org/reference.html

Everything you claim Selenium does not have, is documented there. Open 
your browser search and search for: waitForElementPresent, 
waitForNotTitle, etc. It's all there. Selenium supports these very 
important API calls for testing dynamic content and then some. For 
example, you can do regexp matches against values you wait for using 
"regexp:" in the value field. Very useful for rapidly asserting that the 
autocompleted value in an asynchronous ComboBox is correct. 

Now where in the Windmill docs are:
- CSS locators (like dojo.query; we are used to it)
- mouseDown/UpAt (emphasis At; we need to be able to specify an xy offset 
from the locator, which we need for Slider to move the slider to just 
below a ruler marker)
- control/shift/altKeyDown/Up
- keyPress/keyUp/keyDown
- fireEvent
- typeKeys

These Selenium commands simply do have corresponding Windmill commands 
documented in the Windmill Controller API page. I most certainly can't 
select them in the Windmill IDE. But they are all there and documented on 
the Selenium website and even better documented in the Selenium IDE. Do 
you see what I mean by documentation now?

We need mouseDown/UpAt in particular if we want our dojo Slider tests to 
work. We will need keyPress and control/shift/ltKeyDown/Up for a11y 
testing; I already have spoken with the a11y people and they have said 
they aren't going to accept selecting "complex" from the Windmill IDE for 
every single Tab and keyboard action.  So I hope you will surprise me by 
saying, "mouseDownAt (emphasis At), keyPress, and waitForValue regexp 
matching are in the Windmill IDE's list of actions and documented at this 
Windmill URL," because I can't find it.

As for the bug, here is the page I am testing type and triggerKeyEvent in 
IE against:
http://archive.dojotoolkit.org/nightly/checkout/dojox/data/tests/QueryReadStore.html
When I "type" in the ComboBox, the menu doesn't drop down, indicating no 
event fired. When I "windmill.event.triggerKeyEvent" I get the menu, but 
don't get the text I typed in IE.

And here is where I am testing Slider:
http://archive.dojotoolkit.org/nightly/checkout/dijit/tests/form/test_Slider.html
Give me a Windmill script you tested to move the Slider with a mouseDown 
then mouseUp to the 80% marker, then assert that the value in the textbox 
is 80%. My attempt in Windmill failed, but worked great in Selenium since 
I had mouseUpAt to move the mouse down from the 80% ruler marker to the 
slider area. If we can overcome this in Windmill at least as easily as we 
did with Selenium, I think it would make a stronger case for Windmill.




Dustin Machi <dmachi at sitepen.com> 
Sent by: dojo-contributors-bounces at dojotoolkit.org
05/28/2008 09:57 AM
Please respond to
"dojo dev." <dojo-contributors at dojotoolkit.org>


To
"dojo dev." <dojo-contributors at dojotoolkit.org>
cc

Subject
Re: [dojo-contributors] selenium vs. windmill






Forwarding this email from Mikeal Rogers (also added him to the list for 
this discussion)
- Dustin

---------------------------------------------------------------------------------------------------------------
>
> David, I thought that Windmill code seemed strangely familiar. It is
> just ripped from Selenium's htmlutils.js! The resemblance is uncanny:
> 
http://svn.openqa.org/fisheye/browse/~raw,r=2075/selenium/trunk/src/main/resources/core/scripts/htmlutils.js 

>

Initially we thought windmill would end up being a fork of Selenium with 
better communication between the front end and a newly written back end. 
After we started we realized that it was going to take a rework of 
almost everything in the javascript end, and the Python backend is 
obviously it's own code base.

>
>
>
> So I was not surprised when I fired up Windmill in IE and discovered
> that BOTH Windmill and Selenium have the exact same keyboard event bug I
> showed Becky. When you call
> windmill.events.triggerKeyEvent/selenium.typeKeys, Firefox actually
> types the keys, but IE only sends the event; the text doesn't appear.
> This means that neither Windmill nor Selenium have x-browser keyboard
> support. To resolve this, I have a java.awt.Robot applet proof of
> concept to do proper keyboard support for us that works with either
> Selenium or Windmill.

This surprises me. I know that Adam Christian did a large rework of the 
way our "type" method works and shouldn't be that close to Selenium. In 
fact a bug in trying to test a YUI widget that had bindings to keypress 
events is what triggered the rework of that code and this issue sounds 
very similar, if you would come on to #windmill on freenode and show us 
this issue I'm sure Adam will fix it, we're on IRC during most 
non-sleeping PDT hours.

>
>
> I've used both Selenium and Windmill, and for me, here's what it comes
> down to:
> - Selenium has a Grid, Windmill does not.
>
> - Selenium Grid has demos+documentation,

This is interesting, one of the comments we get from people coming to 
the project tends to be that the documentation is very good in 
comparison to Selenium, although it's never quite as good as we think it 
should be. Is there something more you wanted in the WindmillBook docs 
that you couldn't find?

http://windmill.osafoundation.org/trac/wiki/WindmillBook

> Windmill is still a WIP


Every open source project is a work in progress. Feature wise I would 
say we have parity with Selenium, we have a lot they don't and they have 
SSL and Grid. In terms of stability Selenium has a big advantage as it's 
the older project, but that said we have incredibly fast turnaround 
times for bugs that people bring us and our community support is much 
better, we're on IRC and we're quick to answer mail to the list.

>
> - Selenium Grid comes pre-compiled, Windmill is python source, and
> Windmill python dependencies in turn have VS2003 dependencies in 
Windows.

That's not accurate. There is one dependency, simplejson, that requires 
a compiler for it's _default_ install. Because of the speed improvements 
associated with the compiled version we encourage people to use it. It 
does not require VS2003, it requires a compiler, any compiler, and we 
link to documentation on using minGW for windows.

It might also be worth mentioning that simplejson will be included in 
the Python standard library as of Python 2.6 and this problem will 
slowly fade away.

Our install docs have some pretty comprehensive instructions on 
installing Python and getting everything up and running, if they are 
lacking I would love to know where so that we can improve them.

http://windmill.osafoundation.org/trac/wiki/BookChapter-1-Install

>
>
> Now, maybe promises from the Windmill folks for: a Grid, more
> documentation, binary Windows builds, and fixing browser security
> restrictions so htmlutils works, will drive us to choose Windmill. And
> it would be great if they delivered all of that for us right now. But it
> seems to me that we needed these things yesterday and Selenium already
> had them.

I think I dealt with the prebuilt binary issue already. I would love to 
hear some clarification on what exactly needs better documentation, 
we're constantly improving the documentation but I'm shocked to hear 
that doc improvement would be a barrier to adoption.

Here is the issue with Grid from our perspective; it imposes a 
particular ideology about test structure and uses Java for it's load 
distribution.

In windmill we have a very different way of organizing large test 
structures with dependencies which is a bit much to delve in to in this 
email, but at a high level dependencies enforce a test hierarchy and 
that hierarchy could be easily exploited for test parallelism, we just 
don't provide an out of the box strategy for that, and we probably never 
will. There are just too many requirements from too many different camps 
to provide an out of the box package for test parallelism, and since our 
test structure isn't preventing anyone from writing a solution that fits 
their needs I don't see why the project itself should attempt to 
facilitate it. We take the same stance on additional language libraries, 
we have a complete XMLRPC interface that anyone can use to write a 
windmill client library in their language but we aren't going to burden 
the larger project with a multitude of requirements or bless any 
particular client library as "the library" for their language.

To be honest, I'm very surprised to hear that these are the barriers one 
would have to adoption of windmill in favor of Selenium. We certainly 
don't claim that windmill is for everyone, one of the first thing we 
tell people when they come along to the project is that we don't support 
SSL and most likely won't for a very long time. If someone has a very 
Java heavy infrastructure there are niceties that come with Selenium you 
won't find in any other project, but in general we steer people towards 
Sahi if they require SSL support.

What we tend to emphasize as Windmill's strength is it's niceties for 
testing dynamic content.

http://windmill.osafoundation.org/trac/wiki/BookChapter-4-1-TheControllerAPI 



If you look through the Controller API documentation you'll notice a lot 
of additions over Selenium, it's been a long time since I actually 
brought up Selenium up and wrote a test but you'll notice that the API 
is a little more clean in terms of unifying the identification methods 
in to "locator" arguments that are applicable to every method in the 
controller.

Some other things I don't believe Selenium has implemented yet are 
conditional waits;

waits.forElement
    Waits for a DOM element to exist, or times out. Takes all locator 
params and an optional param; timeout.
waits.forPageLoad
    Waits for a new page to load in the testing window. Takes all an 
optional param; timeout.
waits.forNotTitle
    Waits for the testing window to NOT have the provided title. 
Requires a param; title, and has an optional param; timeout.

These are incredibly useful when testing dynamic content and using them 
in place of static sleep calls and tend to speed up test times by a lot.

Overall I don't think one tool in this space shines so brightly over 
another that it should be considered the easy choice, this is a 
relatively new area of technology ( cross-browser functional testing ) 
and as such all of the current works are very opinionated software.

-Mikeal

_______________________________________________
dojo-contributors mailing list
dojo-contributors at dojotoolkit.org
http://turtle.dojotoolkit.org/mailman/listinfo/dojo-contributors

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.dojotoolkit.org/pipermail/dojo-contributors/attachments/20080528/a8229dff/attachment.htm 


More information about the dojo-contributors mailing list