[dojo-contributors] selenium vs. windmill
dmachi at sitepen.com
Wed May 28 09:57:57 EDT 2008
Forwarding this email from Mikeal Rogers (also added him to the list for
> David, I thought that Windmill code seemed strangely familiar. It is
> just ripped from Selenium's htmlutils.js! The resemblance is uncanny:
Initially we thought windmill would end up being a fork of Selenium with
better communication between the front end and a newly written back end.
After we started we realized that it was going to take a rework of
obviously it's own code base.
> So I was not surprised when I fired up Windmill in IE and discovered
> that BOTH Windmill and Selenium have the exact same keyboard event bug I
> showed Becky. When you call
> windmill.events.triggerKeyEvent/selenium.typeKeys, Firefox actually
> types the keys, but IE only sends the event; the text doesn't appear.
> This means that neither Windmill nor Selenium have x-browser keyboard
> support. To resolve this, I have a java.awt.Robot applet proof of
> concept to do proper keyboard support for us that works with either
> Selenium or Windmill.
This surprises me. I know that Adam Christian did a large rework of the
way our "type" method works and shouldn't be that close to Selenium. In
fact a bug in trying to test a YUI widget that had bindings to keypress
events is what triggered the rework of that code and this issue sounds
very similar, if you would come on to #windmill on freenode and show us
this issue I'm sure Adam will fix it, we're on IRC during most
non-sleeping PDT hours.
> I've used both Selenium and Windmill, and for me, here's what it comes
> down to:
> - Selenium has a Grid, Windmill does not.
> - Selenium Grid has demos+documentation,
This is interesting, one of the comments we get from people coming to
the project tends to be that the documentation is very good in
comparison to Selenium, although it's never quite as good as we think it
should be. Is there something more you wanted in the WindmillBook docs
that you couldn't find?
> Windmill is still a WIP
Every open source project is a work in progress. Feature wise I would
say we have parity with Selenium, we have a lot they don't and they have
SSL and Grid. In terms of stability Selenium has a big advantage as it's
the older project, but that said we have incredibly fast turnaround
times for bugs that people bring us and our community support is much
better, we're on IRC and we're quick to answer mail to the list.
> - Selenium Grid comes pre-compiled, Windmill is python source, and
> Windmill python dependencies in turn have VS2003 dependencies in Windows.
That's not accurate. There is one dependency, simplejson, that requires
a compiler for it's _default_ install. Because of the speed improvements
associated with the compiled version we encourage people to use it. It
does not require VS2003, it requires a compiler, any compiler, and we
link to documentation on using minGW for windows.
It might also be worth mentioning that simplejson will be included in
the Python standard library as of Python 2.6 and this problem will
slowly fade away.
Our install docs have some pretty comprehensive instructions on
installing Python and getting everything up and running, if they are
lacking I would love to know where so that we can improve them.
> Now, maybe promises from the Windmill folks for: a Grid, more
> documentation, binary Windows builds, and fixing browser security
> restrictions so htmlutils works, will drive us to choose Windmill. And
> it would be great if they delivered all of that for us right now. But it
> seems to me that we needed these things yesterday and Selenium already
> had them.
I think I dealt with the prebuilt binary issue already. I would love to
hear some clarification on what exactly needs better documentation,
we're constantly improving the documentation but I'm shocked to hear
that doc improvement would be a barrier to adoption.
Here is the issue with Grid from our perspective; it imposes a
particular ideology about test structure and uses Java for it's load
In windmill we have a very different way of organizing large test
structures with dependencies which is a bit much to delve in to in this
email, but at a high level dependencies enforce a test hierarchy and
that hierarchy could be easily exploited for test parallelism, we just
don't provide an out of the box strategy for that, and we probably never
will. There are just too many requirements from too many different camps
to provide an out of the box package for test parallelism, and since our
test structure isn't preventing anyone from writing a solution that fits
their needs I don't see why the project itself should attempt to
facilitate it. We take the same stance on additional language libraries,
we have a complete XMLRPC interface that anyone can use to write a
windmill client library in their language but we aren't going to burden
the larger project with a multitude of requirements or bless any
particular client library as "the library" for their language.
To be honest, I'm very surprised to hear that these are the barriers one
would have to adoption of windmill in favor of Selenium. We certainly
don't claim that windmill is for everyone, one of the first thing we
tell people when they come along to the project is that we don't support
SSL and most likely won't for a very long time. If someone has a very
Java heavy infrastructure there are niceties that come with Selenium you
won't find in any other project, but in general we steer people towards
Sahi if they require SSL support.
What we tend to emphasize as Windmill's strength is it's niceties for
testing dynamic content.
If you look through the Controller API documentation you'll notice a lot
of additions over Selenium, it's been a long time since I actually
brought up Selenium up and wrote a test but you'll notice that the API
is a little more clean in terms of unifying the identification methods
in to "locator" arguments that are applicable to every method in the
Some other things I don't believe Selenium has implemented yet are
Waits for a DOM element to exist, or times out. Takes all locator
params and an optional param; timeout.
Waits for a new page to load in the testing window. Takes all an
optional param; timeout.
Waits for the testing window to NOT have the provided title.
Requires a param; title, and has an optional param; timeout.
These are incredibly useful when testing dynamic content and using them
in place of static sleep calls and tend to speed up test times by a lot.
Overall I don't think one tool in this space shines so brightly over
another that it should be considered the easy choice, this is a
relatively new area of technology ( cross-browser functional testing )
and as such all of the current works are very opinionated software.
More information about the dojo-contributors