Any good distributed agent/service models for .NET

Question

I'm looking for tools that implement the distributed agent/service model ... I guess we could also call this grid or cloud computing, but I'm not sure the term is exactly analagous. A distributed agent model would be one where developers build agents that are dispatched to a framework and the framework executes the agent somewhere in the distributed environment.

Specifically, I'm looking for a framework that support dynamic distribution across the grid/cloud, but replaces the transient agent with a more persistent service. The goal here is to minimize the amount of time and effort it requires to deploy a service into the grid/cloud container.

FWIW, I think Digipede gets pretty close to this, but I'm interested in what else is out there. I'm under the impression that assembly distribution would have to be considered in order for the solution to be viable.

Answer 1

Qizmt is used by MySpace for distributed MapReduce operations. It's analogous to Hadoop from the Java world. Both projects are based on MapReduce by google. It runs on normal Windows boxes, unlike dryad which requires Windows Server High Performacne Computing (HPC)

Plus it's open source. Which is nice. http://code.google.com/p/qizmt/

Answer 2

I've been looking into a similar thing for my own work. So far I've found Digipede, Alchemi and NGrid, but they all need a SQL database as a grid manager except for NGrid which isn't as mature.

Answer 3

There's a project at Microsoft called Dryad. Here's a good podcast with more info: http://www.dotnetrocks.com/default.aspx?showNum=378 It uses LINQ and is supposed to be easy to use. Microsoft has it doing some production stuff for Microsoft's AdCenter. Here's the research page. http://research.microsoft.com/research/sv/Dryad/

Dryad is quite expressive. It completely subsumes other computation frameworks, such as Google's map-reduce, or the relational algebra. Moreover, Dryad handles job creation and management, resource management, job monitoring and visualization, fault tolerance, re-execution, scheduling, and accounting.

P.S. I don't see a download, but if you contact them and you are serious you could probably get it.

Update: it's now available for download here: http://research.microsoft.com/en-us/downloads/03960cab-bb92-4c5c-be23-ce51aee0792c/default.aspx

Answer 4

Voyager from Recursion Software provides a mature framework for many styles of distributed architectures.

Answer 5

This may be a little late to the party - but we're the developers of Alchemi. We've developed a better version of the framework to address its limitations.

You might want to check out the Utilify Platform (www.utilify.com) - which is conceptually similar to Alchemi - with a more flexible model.

You do need a grid manager, and a database - however, you could use the free SQL Server Express version and it is tested to work on Windows Vista, XP, 2003.

We're happy to help you in using Utilify for your application.

Regards, Krishna