Working with file compression/encryption in Azure

Question

Is Azure an appropriate/viable platform for a process that receives various encrypted and/or compressed files, extracts them, and manipulates them?

For example, I currently have a process that uses SSIS to download flat files as described above and loads them to a database. New files are then extracted and gzipped or PGP encrypted and FTP'd out.

If I was to migrate this process to Azure + SQL Azure, what is the best way to go about this? I have a front end that allows files to be posted to blob storage - where do I go from here? VM role? Worker role?

Answer 1

One additional tidbit to add to knightpfhor's answer: A worker role will cover nearly all of what you need, as you can have startup tasks that run with administrative privileges. There are very few cases that require a VM role:

• Startup / setup takes a really long time. This is a bit subjective, but a good rule-of-thumb is around 5 minutes
• Startup / setup tasks are unreliable and don't always work the first time you run them
• Human interaction is required

Answer 2

I think that Azure is a good fit for this situation.

As you already have a front end, I would make a minor change to it so that when it has finished storing the file in blob storage it writes a message to a queue with the the path to the file to be processed and information about how it's been encrypted (if you have this information and it would make latter processing easier).

I would write a class that is able to monitors this queue for incoming work. It gets the message off the queue (you'll want to ensure that the visibility timeout that you set when you get the message off the queue is sufficient so that you can finish this processing before it expires), does the work that you've already specified, then deletes the queue message to say that it's finished that piece of work.

In a basic setup, write a worker role that uses the class above. If you're able to process multiple files at once on an instance, have the worker role create multiple instances of this class on its own thread on startup.

If you need more processing power, deploy more worker instances.