[dojo-contributors] web-based build tool

[James Burke]

On 12/14/06, Owen Williams <owen at smartsoul.com> wrote:
>
>
> Howdy!  To make it easier for people to do a custom build of dojo if they
> don't have Java/Rhino/etc set up, I'm interested in making a page similar to
> this one:
>
>  http://mootools.net/download/release
>
> (The UI will be different because we have so many more files, but the idea
> is the same).  I'm willing to design and code the front end, and am looking
> for someone who wants to build the back-end.  This would involve setting up
> some sort of CGI-ish thing (in your language of choice) on Mastadon (or
> wherever) which would have the following client/server API:
>
>  1) Send the resource file dependency tree down to the client.  This can
> come from a static file at first, we should be able to pull this from the
> doc parser output eventually (which is not part of what I'm asking for at
> this time).
>
>  2) Receive a profile.js file (see enclosed) with some additional parameters
> if necessary and return a jammed-together and optimized for download .js
> file.  Bonus points for returning it in a way that the browser will download
> it rather than display it (probably some mime-type fanciness is all that's
> involved).
>
>  3) (Not necessary but cool) have some sort of mechanism where folks can
> store/retreive builds that they've done before.  (Eg: after the next version
> of Dojo comes out, they go to the page and can pull up a build they did on
> the last version and run it again.)  I don't think we need user
> authentication on this -- just naming the builds should be sufficient.  We
> could also do this with dojo.storage for that particular user on that
> particular machine/browser if storing it on the server doesn't make sense.
>
>
> Eventually, it would be awesome if this front end could work on a local dojo
> source install -- not sure exactly how this would work, but it would be
> neat!  Ideas?
>
>
> Anyone interested in working on this with me?
>
> cheers
> Owen

Hi Owen,

Sorry for not responding sooner. I wanted to be sure I could give some
time to the proposal. Warning: this is a long post.

I think it is great if you want to work on the front-end (or even the
back-end) of this. Here is what I was thinking on how the build would
occur and some implications. But it is only my opinion, and I'm fine
with alternatives.

To me, this sort of web-based tool is only useful for generating
dojo.js. This tool is aimed at folks that want to use Dojo, want
something that performs well, but don't want to deal with all the dojo
files. This would also help with our perception of being big and
unweildy.

To accomplish that task, then it seems best to use an xdomain build
for this sort of thing. It means they can download the widget graphics
(and as Brad mentioned) the Flash files that are needed for Dojo
without having to have them locally. Less things to track. It makes
Dojo seem more approachable (I only have to deal with dojo.js, and the
rest is handled for me).

However, using an xdomain build means abiding by xdomain rules, mainly
using dojo.addOnLoad() for any code you want to execute immediately
after doing some dojo.require() calls (since xdomain loading is async
loading). The upside to this though, is that even regular dojo builds
and usage can use the same pattern (follow dojo.require() statements
with a dojo.addOnLoad() call for things you want to execute
immediately after the modules load).

But if we feel this sort of build process (and the ability to just use
an xdomain CDN Dojo without a build) will be the primary newbie
interaction with Dojo, then I suggest we convert our test code any any
tutorials we see to use this dojo.require/dojo.addOnLoad pattern. I
can handle the modifications to our test files for this, but in
general, we all need to be sure we are aware of this code pattern when
discussing Dojo with others.

This might seem like a small inconvenience, but using an xdomain build
gives us something even greater: the ability to use other modules on
other servers in a xdomain way -- the user does not need to download
those "third party" modules to use them. They just need to know the
URL to the modules. Now a paranoid user may want to download the
modules locally to avoid a network dependency and scrub the code for
good behavior, but for first impressions/trying out code, it really
opens up a way to share Dojo-compatible code by using just an URL.
That is wicked awesome. (Note that the "third party" code would have
to do an xdomain build to make this work)

So, assuming using an xdomain build from a CDN is OK, and we are
comfortable with what I described above, then the next piece is how to
actually build dojo.js.

You can get a list of available module resources by looking at the
file paths, since the file paths have to match the dojo.require()
statements (src/undo/browser.js maps to dojo.undo.browser). So the
list of available module resources can be known ahead of time, just do
a file search and convert the file paths to resource names. The UI can
use this list to offer the user the ability to choose the resources
they want in the custom build by checking off the modules that are
needed.  We can also give the user a textarea to just paste in a list
of module resources.

By converting at least the dojo.js part of the build to JavaScript
means that we can use that same JavaScript in the user's browser to do
the build. This means we would not need a back-end CGI to to do the
work. I'm still not sure if it will work yet, but I think there is a
good chance it can.

When the user selects the module resources they want to build, their
browser can download the build script that collects the resources,
concatenates them into one file and do the final modifications to make
dojo.js. When Tom and I talked about this before on an older email
thread, it seems like if we can do something with the
Content-Disposition header, we can get the browser to download the
file instead of display it (trick is how to do this on the client
side, and not use a server CGI).

The last trick would be to post the resulting dojo.js to the
JavaScript Compressor, since that is in a Java jar file, and not
exportable to JavaScript at this time (at least very easily). However,
we can leverage the ShrinkSafe web page to do the compression. Post
the dojo.js built in the browser to ShrinkSafe for compression. Maybe
we will need to refresh the interface and UI to ShrinkSafe for this.

However, maybe the majority of contributores would prefer to user
server resources to generate the dojo.js, instead of having the
browser request the individual files and do the work in the browser.
Using server resources will use less server bandwidth, but then it is
a server resource, and something that is also vulnerable to
attack/something that needs to be monitored so that it does not
consume too much server resources. But I'm fine with that approach if
that is the majority feel and someone wants to do the CGI work.

And there may be issues with the "doing it all in the browser" either
from a usability or workflow perspective that doesn't make it usable.
But I would like to try it. Even if it doesn't work out, I think
converting the build process as much as possible to JavaScript reduces
build dependencies, and also proves the usefulness of JavaScript
outside of the browser. It is also a common language all contributors
know, so more people can help out with the build system if need be.

So my plan of attack is the following:
1) Make the xdomain build truly usable, without needing any local
files. I have a bunch of bugs I filed in trac for me (jburke) if you
want to track the progress.

2) Convert the parts of the build system that make dojo.js to
JavaScript, so they can also be used by the browser for a web tool.

3) As time permits, convert more of the build process to JavaScript,
and eventually get rid of the Ant and Jython dependencies.

So only #1 is truly necessary for the web-based build if someone would
rather use CGI to run the build on the server to generate dojo.js. But
I'll be steadily working on all three items. I'm happy to support
someone who wants to use CGI, and if we all feel like that is a better
way to go for the actual build, I'm fine with that. I just don't want
to do the CGI work, someone else can do it. I will only do it if the
browser based method fails.

Owen, if there are things I can do to help support your front-end
efforts (or if you decide you want a CGI), then please let me know.
I'll be mostly focused on #1 above though, since I consider it a
blocker to the build process working.

It also could be that most contributors would prefer not to go the
xdomain route for this. That is fine, but I need to fix the xdomain
issues anyway, so I'll be working on that. I'm happy to help support
anyone via code reviews if the consensus is we should use another
path.

I don't really care what the UI looks like for the front-end, besides
offering a list of modules resources to choose from and giving the
option for a textarea for those users that know what they want. So
don't feel like you need to consult with me on UI layout or design.

Sorry for the long post.

James