Grid computing - the idea of distributing work among multiple computers so that
it gets done faster - has been around for a few years now. What The Digipede Network
brings to the table is a polished way to set up a .NET-based grid on your own network,
ready to undertake distributed computing jobs at a moment's notice. When you install
their product, you get a well-designed layer of management infrastructure together
with code hooks that make it easy to submit piece of work to the grid.
Installing Digipede starts with picking one machine to be the Digipede Server and
running a master setup application there. This sets up a Web-based management console
and a database that Digipede uses to track what's going on. Other computers can
then log on to the internal Web site and install Digipede Agents - bits of code
that communicate with the server to download and run distributed pieces of your
application. You can submit jobs from the Digipede Workbench, a GUI application,
or using an API that talks directly to the server. There are also extensive help
files and samples to get you started on the development side of things. As you send
jobs to the Server, it makes them available to Agents, sends out the pieces, collects the results, and sends them back to where
they belong. The Agents make their own
determination of which jobs to run, since they know more about their local resources
and load, while the Server monitors to make sure that nothing gets lost (for example,
it can recover from an Agent losing power after it takes a job).
The developer SDK documents a variety of patterns for distributing work to Agents.
On the simple end of things, you can send out entire command-line executables, feeding
them input and parsing their output or grabbing output files on the agent computers.
This requires around 20 to 100 lines of setup code to handle all the plumbing if you do
it entirely in code (depending on how fancy you want to get)- or no lines at all if you do it using the Workbench. Perhaps
more interesting for most existing applications is their Worker Library Forms pattern,
in which the only thing sent to Agents are objects in particular .NET libraries.
Here, you end up creating Digipede Job objects, adding Tasks to those Jobs, and
then assigning your own objects to the tasks:
task.Worker = new MyWorker(...)
Where you give the worker object whatever initialization it requires for an individual
agent, of course. Thus, if you have time-consuming work in an inner loop you can
easily split that loop off into a worker object, initialize it via the loop index,
wrap it in the Digipede scaffolding, and you're off to the distributed computing
races. Another well-developed sample shows you how to use this same technique behind
Excel worksheets via the VSTO bits, which should be attractive in the financial
sector. Any good .NET developer should be able to get Digipede up and running, and
have a piece of their own key application distributed across multiple machines,
in less than a day.
Of course, not every huge application is appropriate for grid computing. The key
question is whether you can identify some bottleneck that can be effectively parallelized
into independent tasks. But assuming that you have a .NET application with such
a parallelizable bottleneck, Digipede will help you concentrate on the logical effort
of picking the work to distribute, rather than the near-endless tedium of writing
a distributed computing platform. It's the same sort of time and effort savings
you get from a good GUI library, on a different level of abstraction.
If you've got a problem in mind, you can get started by signing up for a free copy
of their Developer Edition.
This has all the bits of the full editions, and the full SDK, but is limited to
two computing agents. From there, you move up to the Team Edition for jobs wanting
up to 20 agents or the Professional Edition (with some fancier options, like job
priorities and multiple agent pools) for unlimited numbers of agents, when you're
ready to deploy.