Framework recommendation needed on .NET or JAVA fo

Question

I want client machines on internet who subscribe to my server to donate their idle cpu cycles.(Like SETI@Home)

They would take jobs(work-units) from server to process, and send back results to the server. (This is the most simple description). The framework i need should allow me to define a job/task. Rest of things like communication, job execution/tracking, client binaries update etc. should be managed by framework.

• I evaluated Alchemi.NET a bit, but its not actively maintained, seems half-baked.

• BOINC has API in C, but i want a .NET or JAVA framework.

• I am looking at Manjrasoft's ANEKA , but it seems to work only for LAN clouds.

There must be some such frameworks available. I need expert recommendations!

Answer 1

BOINC is the framework that most naturally meets your volunteer computing requirements, and it's stable and highly scalable -- I'd make sure you've considered it completely before ruling it out.

If you need to deliver something to a short deadline then I'd consider working up a simple supervisor (or scheduler) - worker pattern, just to get the application off the ground. The supervisor would be responsible for chunking up the data and making it available over http. Workers (your client app) would connect to a supervisor server; download a chunk of work; complete the chunk; and upload the results to the supervisor.

The main trick is to get the state machine thrashed out properly, so that you can accurately track what state each work chunk is in. I'd have the supervisor persist state in a database in the background.

Does your first release need to be internal, or is it for public consumption?

While that's working I'd get started on looking at getting up to speed with BOINC and planning a migration.

Answer 2

My recommendation

Work dist:

1. Have a receiver of requests, that places messages in a message queue, like rabbit mq
2. Have a host of workers, listening to the same queue, taking work from it and acking it when done.
3. When done, send a message on another queue, containing an URI to a known location, such as your network drive. The target is your parsed data.
4. The receiver listens to these "completed" messages. Fetches the data from the URI.

Done.

RabbitMQ comes with great CLR APIs.

The same reasoning works well with Microsoft Azure and their AbbFabric Queue. A plus is that is scales extremely well.

Hot Versioning

http://topshelf-project.com/

It gives a folder where you can drop binaries, which are then run. It manages versioning of these as well as running them as windows services.

Deployment

You can deploy the binaries with robocopy/xcopy and "net use Q: pwd \server\share", "net delete Q:"

Continuous Integration

Teamcity

After working with MsBuild extensively, I would recommend scripting it with psake and running the build with PowerShell. If you get advanced with PowerShell you also have WinRM available to you from your build scripts, which is really smooth.

Use the git/subversion commit number as the 0.0.0.x, x in the previous version number, and you will have automatic versioning that is "shared" across "Debug"/"Production" builds.

The Azure way

Work dist:

Same as above but with AppFabric Queue instead of RabbitMQ.

Hot Versioning

By swapping "Staging" and "Production" instances around, you avoid the downtime.

Deployment

You can either tap into the Azure Tools for Visual Studio's MsBuild tasks as can be read about here or you could use the PowerShell AzureSnapIns with a similar setup as above for Continuous Integration.

Continuous Integration

Same as above.

Answer 3

I'm hardly an expert but I do have a little experience with distributed computing using MPI (with C). What you're talking about does not sound like grid computing, rather a master/slave system. That is your server is the master and it directs all the clients (slaves).

I know very little about .net programming so I'll speak in general.

There are a lot of web frame works out there and probably most have the facilities you'll need. That is your client will be able to upload files with the content they have gathered (or they could just use http get/post), because you don't have to worry about UI issues you can handle everything probably through one action (assuming an action based web framework). Then the server can return a response via JSON or XML which the client can use to take further direction on. JSON is the right choice if the system is very simple and probably a good choice for prototyping.

For the application upgrade I would consider this a separate issue (although it should be a simple matter for the server to return this to the client).

Answer 4

How about .net's ClickOnce installer to manage auto updated client binaries. http://msdn.microsoft.com/en-us/library/t71a733d.aspx

I'm not sure of a "jobs framework" per-say, but Microsoft's Sync framework to support rolling your own jobs syncing with clients?

http://msdn.microsoft.com/en-us/sync/default