[dojo-contributors] selenium vs. windmill
haysmark at us.ibm.com
Wed May 28 11:48:13 EDT 2008
Mikeal, that Controller API page is one of my biggest concerns as far as
documentation goes: the API as given there is actually very sparse in
comparison to the Selenium docs. Have you seen the Selenium docs lately?
Everything you claim Selenium does not have, is documented there. Open
your browser search and search for: waitForElementPresent,
waitForNotTitle, etc. It's all there. Selenium supports these very
important API calls for testing dynamic content and then some. For
example, you can do regexp matches against values you wait for using
"regexp:" in the value field. Very useful for rapidly asserting that the
autocompleted value in an asynchronous ComboBox is correct.
Now where in the Windmill docs are:
- CSS locators (like dojo.query; we are used to it)
- mouseDown/UpAt (emphasis At; we need to be able to specify an xy offset
from the locator, which we need for Slider to move the slider to just
below a ruler marker)
These Selenium commands simply do have corresponding Windmill commands
documented in the Windmill Controller API page. I most certainly can't
select them in the Windmill IDE. But they are all there and documented on
the Selenium website and even better documented in the Selenium IDE. Do
you see what I mean by documentation now?
We need mouseDown/UpAt in particular if we want our dojo Slider tests to
work. We will need keyPress and control/shift/ltKeyDown/Up for a11y
testing; I already have spoken with the a11y people and they have said
they aren't going to accept selecting "complex" from the Windmill IDE for
every single Tab and keyboard action. So I hope you will surprise me by
saying, "mouseDownAt (emphasis At), keyPress, and waitForValue regexp
matching are in the Windmill IDE's list of actions and documented at this
Windmill URL," because I can't find it.
As for the bug, here is the page I am testing type and triggerKeyEvent in
When I "type" in the ComboBox, the menu doesn't drop down, indicating no
event fired. When I "windmill.event.triggerKeyEvent" I get the menu, but
don't get the text I typed in IE.
And here is where I am testing Slider:
Give me a Windmill script you tested to move the Slider with a mouseDown
then mouseUp to the 80% marker, then assert that the value in the textbox
is 80%. My attempt in Windmill failed, but worked great in Selenium since
I had mouseUpAt to move the mouse down from the 80% ruler marker to the
slider area. If we can overcome this in Windmill at least as easily as we
did with Selenium, I think it would make a stronger case for Windmill.
Dustin Machi <dmachi at sitepen.com>
Sent by: dojo-contributors-bounces at dojotoolkit.org
05/28/2008 09:57 AM
Please respond to
"dojo dev." <dojo-contributors at dojotoolkit.org>
"dojo dev." <dojo-contributors at dojotoolkit.org>
Re: [dojo-contributors] selenium vs. windmill
Forwarding this email from Mikeal Rogers (also added him to the list for
> David, I thought that Windmill code seemed strangely familiar. It is
> just ripped from Selenium's htmlutils.js! The resemblance is uncanny:
Initially we thought windmill would end up being a fork of Selenium with
better communication between the front end and a newly written back end.
After we started we realized that it was going to take a rework of
obviously it's own code base.
> So I was not surprised when I fired up Windmill in IE and discovered
> that BOTH Windmill and Selenium have the exact same keyboard event bug I
> showed Becky. When you call
> windmill.events.triggerKeyEvent/selenium.typeKeys, Firefox actually
> types the keys, but IE only sends the event; the text doesn't appear.
> This means that neither Windmill nor Selenium have x-browser keyboard
> support. To resolve this, I have a java.awt.Robot applet proof of
> concept to do proper keyboard support for us that works with either
> Selenium or Windmill.
This surprises me. I know that Adam Christian did a large rework of the
way our "type" method works and shouldn't be that close to Selenium. In
fact a bug in trying to test a YUI widget that had bindings to keypress
events is what triggered the rework of that code and this issue sounds
very similar, if you would come on to #windmill on freenode and show us
this issue I'm sure Adam will fix it, we're on IRC during most
non-sleeping PDT hours.
> I've used both Selenium and Windmill, and for me, here's what it comes
> down to:
> - Selenium has a Grid, Windmill does not.
> - Selenium Grid has demos+documentation,
This is interesting, one of the comments we get from people coming to
the project tends to be that the documentation is very good in
comparison to Selenium, although it's never quite as good as we think it
should be. Is there something more you wanted in the WindmillBook docs
that you couldn't find?
> Windmill is still a WIP
Every open source project is a work in progress. Feature wise I would
say we have parity with Selenium, we have a lot they don't and they have
SSL and Grid. In terms of stability Selenium has a big advantage as it's
the older project, but that said we have incredibly fast turnaround
times for bugs that people bring us and our community support is much
better, we're on IRC and we're quick to answer mail to the list.
> - Selenium Grid comes pre-compiled, Windmill is python source, and
> Windmill python dependencies in turn have VS2003 dependencies in
That's not accurate. There is one dependency, simplejson, that requires
a compiler for it's _default_ install. Because of the speed improvements
associated with the compiled version we encourage people to use it. It
does not require VS2003, it requires a compiler, any compiler, and we
link to documentation on using minGW for windows.
It might also be worth mentioning that simplejson will be included in
the Python standard library as of Python 2.6 and this problem will
slowly fade away.
Our install docs have some pretty comprehensive instructions on
installing Python and getting everything up and running, if they are
lacking I would love to know where so that we can improve them.
> Now, maybe promises from the Windmill folks for: a Grid, more
> documentation, binary Windows builds, and fixing browser security
> restrictions so htmlutils works, will drive us to choose Windmill. And
> it would be great if they delivered all of that for us right now. But it
> seems to me that we needed these things yesterday and Selenium already
> had them.
I think I dealt with the prebuilt binary issue already. I would love to
hear some clarification on what exactly needs better documentation,
we're constantly improving the documentation but I'm shocked to hear
that doc improvement would be a barrier to adoption.
Here is the issue with Grid from our perspective; it imposes a
particular ideology about test structure and uses Java for it's load
In windmill we have a very different way of organizing large test
structures with dependencies which is a bit much to delve in to in this
email, but at a high level dependencies enforce a test hierarchy and
that hierarchy could be easily exploited for test parallelism, we just
don't provide an out of the box strategy for that, and we probably never
will. There are just too many requirements from too many different camps
to provide an out of the box package for test parallelism, and since our
test structure isn't preventing anyone from writing a solution that fits
their needs I don't see why the project itself should attempt to
facilitate it. We take the same stance on additional language libraries,
we have a complete XMLRPC interface that anyone can use to write a
windmill client library in their language but we aren't going to burden
the larger project with a multitude of requirements or bless any
particular client library as "the library" for their language.
To be honest, I'm very surprised to hear that these are the barriers one
would have to adoption of windmill in favor of Selenium. We certainly
don't claim that windmill is for everyone, one of the first thing we
tell people when they come along to the project is that we don't support
SSL and most likely won't for a very long time. If someone has a very
Java heavy infrastructure there are niceties that come with Selenium you
won't find in any other project, but in general we steer people towards
Sahi if they require SSL support.
What we tend to emphasize as Windmill's strength is it's niceties for
testing dynamic content.
If you look through the Controller API documentation you'll notice a lot
of additions over Selenium, it's been a long time since I actually
brought up Selenium up and wrote a test but you'll notice that the API
is a little more clean in terms of unifying the identification methods
in to "locator" arguments that are applicable to every method in the
Some other things I don't believe Selenium has implemented yet are
Waits for a DOM element to exist, or times out. Takes all locator
params and an optional param; timeout.
Waits for a new page to load in the testing window. Takes all an
optional param; timeout.
Waits for the testing window to NOT have the provided title.
Requires a param; title, and has an optional param; timeout.
These are incredibly useful when testing dynamic content and using them
in place of static sleep calls and tend to speed up test times by a lot.
Overall I don't think one tool in this space shines so brightly over
another that it should be considered the easy choice, this is a
relatively new area of technology ( cross-browser functional testing )
and as such all of the current works are very opinionated software.
dojo-contributors mailing list
dojo-contributors at dojotoolkit.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the dojo-contributors