[dojo-contributors] selenium vs. windmill

Dustin Machi dmachi at sitepen.com
Wed May 28 09:57:57 EDT 2008


Forwarding this email from Mikeal Rogers (also added him to the list for 
this discussion)
- Dustin

---------------------------------------------------------------------------------------------------------------
>
> David, I thought that Windmill code seemed strangely familiar. It is
> just ripped from Selenium's htmlutils.js! The resemblance is uncanny:
> http://svn.openqa.org/fisheye/browse/~raw,r=2075/selenium/trunk/src/main/resources/core/scripts/htmlutils.js 
>

Initially we thought windmill would end up being a fork of Selenium with 
better communication between the front end and a newly written back end. 
After we started we realized that it was going to take a rework of 
almost everything in the javascript end, and the Python backend is 
obviously it's own code base.

>
>
>
> So I was not surprised when I fired up Windmill in IE and discovered
> that BOTH Windmill and Selenium have the exact same keyboard event bug I
> showed Becky. When you call
> windmill.events.triggerKeyEvent/selenium.typeKeys, Firefox actually
> types the keys, but IE only sends the event; the text doesn't appear.
> This means that neither Windmill nor Selenium have x-browser keyboard
> support. To resolve this, I have a java.awt.Robot applet proof of
> concept to do proper keyboard support for us that works with either
> Selenium or Windmill.

This surprises me. I know that Adam Christian did a large rework of the 
way our "type" method works and shouldn't be that close to Selenium. In 
fact a bug in trying to test a YUI widget that had bindings to keypress 
events is what triggered the rework of that code and this issue sounds 
very similar, if you would come on to #windmill on freenode and show us 
this issue I'm sure Adam will fix it, we're on IRC during most 
non-sleeping PDT hours.

>
>
> I've used both Selenium and Windmill, and for me, here's what it comes
> down to:
> - Selenium has a Grid, Windmill does not.
>
> - Selenium Grid has demos+documentation,

This is interesting, one of the comments we get from people coming to 
the project tends to be that the documentation is very good in 
comparison to Selenium, although it's never quite as good as we think it 
should be. Is there something more you wanted in the WindmillBook docs 
that you couldn't find?

http://windmill.osafoundation.org/trac/wiki/WindmillBook

> Windmill is still a WIP


Every open source project is a work in progress. Feature wise I would 
say we have parity with Selenium, we have a lot they don't and they have 
SSL and Grid. In terms of stability Selenium has a big advantage as it's 
the older project, but that said we have incredibly fast turnaround 
times for bugs that people bring us and our community support is much 
better, we're on IRC and we're quick to answer mail to the list.

>
> - Selenium Grid comes pre-compiled, Windmill is python source, and
> Windmill python dependencies in turn have VS2003 dependencies in Windows.

That's not accurate. There is one dependency, simplejson, that requires 
a compiler for it's _default_ install. Because of the speed improvements 
associated with the compiled version we encourage people to use it. It 
does not require VS2003, it requires a compiler, any compiler, and we 
link to documentation on using minGW for windows.

It might also be worth mentioning that simplejson will be included in 
the Python standard library as of Python 2.6 and this problem will 
slowly fade away.

Our install docs have some pretty comprehensive instructions on 
installing Python and getting everything up and running, if they are 
lacking I would love to know where so that we can improve them.

http://windmill.osafoundation.org/trac/wiki/BookChapter-1-Install

>
>
> Now, maybe promises from the Windmill folks for: a Grid, more
> documentation, binary Windows builds, and fixing browser security
> restrictions so htmlutils works, will drive us to choose Windmill. And
> it would be great if they delivered all of that for us right now. But it
> seems to me that we needed these things yesterday and Selenium already
> had them.

I think I dealt with the prebuilt binary issue already. I would love to 
hear some clarification on what exactly needs better documentation, 
we're constantly improving the documentation but I'm shocked to hear 
that doc improvement would be a barrier to adoption.

Here is the issue with Grid from our perspective; it imposes a 
particular ideology about test structure and uses Java for it's load 
distribution.

In windmill we have a very different way of organizing large test 
structures with dependencies which is a bit much to delve in to in this 
email, but at a high level dependencies enforce a test hierarchy and 
that hierarchy could be easily exploited for test parallelism, we just 
don't provide an out of the box strategy for that, and we probably never 
will. There are just too many requirements from too many different camps 
to provide an out of the box package for test parallelism, and since our 
test structure isn't preventing anyone from writing a solution that fits 
their needs I don't see why the project itself should attempt to 
facilitate it. We take the same stance on additional language libraries, 
we have a complete XMLRPC interface that anyone can use to write a 
windmill client library in their language but we aren't going to burden 
the larger project with a multitude of requirements or bless any 
particular client library as "the library" for their language.

To be honest, I'm very surprised to hear that these are the barriers one 
would have to adoption of windmill in favor of Selenium. We certainly 
don't claim that windmill is for everyone, one of the first thing we 
tell people when they come along to the project is that we don't support 
SSL and most likely won't for a very long time. If someone has a very 
Java heavy infrastructure there are niceties that come with Selenium you 
won't find in any other project, but in general we steer people towards 
Sahi if they require SSL support.

What we tend to emphasize as Windmill's strength is it's niceties for 
testing dynamic content.

http://windmill.osafoundation.org/trac/wiki/BookChapter-4-1-TheControllerAPI 


If you look through the Controller API documentation you'll notice a lot 
of additions over Selenium, it's been a long time since I actually 
brought up Selenium up and wrote a test but you'll notice that the API 
is a little more clean in terms of unifying the identification methods 
in to "locator" arguments that are applicable to every method in the 
controller.

Some other things I don't believe Selenium has implemented yet are 
conditional waits;

waits.forElement
    Waits for a DOM element to exist, or times out. Takes all locator 
params and an optional param; timeout.
waits.forPageLoad
    Waits for a new page to load in the testing window. Takes all an 
optional param; timeout.
waits.forNotTitle
    Waits for the testing window to NOT have the provided title. 
Requires a param; title, and has an optional param; timeout.

These are incredibly useful when testing dynamic content and using them 
in place of static sleep calls and tend to speed up test times by a lot.

Overall I don't think one tool in this space shines so brightly over 
another that it should be considered the easy choice, this is a 
relatively new area of technology ( cross-browser functional testing ) 
and as such all of the current works are very opinionated software.

-Mikeal




More information about the dojo-contributors mailing list