Execute python scripts in Azure DataFactory

2020-07-17 05:56发布

I have my data stored in blobs and I have written a python script to do some computations and create another csv. How can I execute this in Azure Data Factory ?

2条回答
Summer. ? 凉城
2楼-- · 2020-07-17 06:16

Another option is using a DatabricksSparkPython Activity. This makes sense if you want to scale out, but could require some code modifications for PySpark support. Prerequisite of cause is an Azure Databricks workspace. You have to upload your script to DBFS and can trigger it via Azure Data Factory. The following example triggers the script pi.py:

{
    "activity": {
        "name": "MyActivity",
        "description": "MyActivity description",
        "type": "DatabricksSparkPython",
        "linkedServiceName": {
            "referenceName": "MyDatabricksLinkedservice",
             "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "pythonFile": "dbfs:/docs/pi.py",
            "parameters": [
                "10"
            ],
            "libraries": [
                {
                    "pypi": {
                        "package": "tensorflow"
                    }
                }
            ]
        }
    }
}

See the Documentation for more details.

查看更多
看我几分像从前
3楼-- · 2020-07-17 06:23

Mighty. You could use Azure Data Factory V2 custom activity for your requirements. You can directly execute a command to invoke python script using Custom Activity.

Please refer to this sample on the github.

Hope it helps you.

查看更多
登录 后发表回答