Run .EXE and Powershell tasks with Airflow

问题:

our systems are basically just Windows Servers running C# and Powershell applications in conjunction with MS SQL Server. We have a in-house WorkflowManagement solution that is able to run tasks that execute EXE/BAT/PS1 and even call DLL-Functions.

Now I am evaluating if Apache Airflow is a better solution for us. My naive plan so far is to run airflow scheduler on a Linux-machine and then let the consumers run on Windows machines. But how would I setup the consumer to run a .exe task for instance?

Do I need to create a Wrapper-Service that takes HTTP-Calls and then executes the .Exe files?

回答1:

I think the consumers you are mentioning are called "workers" in airflow. The typical workers run an executor which doesn't run on Windows.

In theory you could install an ssh server on your windows boxes, and then use SSHExecuteOperator to run the exe commands and etc. there.

This would probably work better than an HTTP impl because your tasks in airflow would stay "running" while the ssh command was running and automatically go to success or fail accordingly. With an HTTP impl you would need to implement this functionality yourself if you wanted to integrate nicely with airflow.