首页 文章

U-SQL活动未在Azure数据工厂上运行

提问于
浏览
1

我使用Azure Data Factory将本地数据传输到Azure Data Lake存储 . 复制数据后,我在上传的文件上运行U-SQL脚本,将其转换为新的.csv文件 . 如果从Visual Studio或直接从Data Lake Analytics运行,我的U-SQL作业运行正常 .

但是如果在Azure Data Factory中添加和活动 . 复制数据后,此U-SQL脚本活动立即失败 . 我尝试了很多方法但无法解决问题 . 它给了我以下错误 .

我的U-SQL活动的JSON定义是

{
    "name": "Transform Data",
    "description": "This will transform work space data.",
    "type": "DataLakeAnalyticsU-SQL",
    "dependsOn": [
        {
            "activity": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
            "dependencyConditions": [
                "Completed"
            ]
        }
    ],
    "policy": {
        "timeout": "7.00:00:00",
        "retry": 0,
        "retryIntervalInSeconds": 30,
        "secureOutput": false
    },
    "typeProperties": {
        "scriptPath": "Scripts/Script.usql",
        "scriptLinkedService": {
            "referenceName": "Destination_DataLakeStore_lc0",
            "type": "LinkedServiceReference"
        }
    },
    "linkedServiceName": {
        "referenceName": "AzureDataLakeAnalyticsForDF",
        "type": "LinkedServiceReference"
    }
}

JSON of entire pipeline is

{
    "name": "CopyPipeline_d26",
    "properties": {
        "activities": [
            {
                "name": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
                "type": "Copy",
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false
                },
                "typeProperties": {
                    "source": {
                        "type": "FileSystemSource",
                        "recursive": false
                    },
                    "sink": {
                        "type": "AzureDataLakeStoreSink",
                        "copyBehavior": "MergeFiles"
                    },
                    "enableStaging": false,
                    "cloudDataMovementUnits": 0,
                    "enableSkipIncompatibleRow": true
                },
                "inputs": [
                    {
                        "referenceName": "workplace_groups_info_2018_03_19_09_32_csv_i_lc0",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "workplace_groups_info_2018_03_19_09_32_csv_o_lc0",
                        "type": "DatasetReference"
                    }
                ]
            },
            {
                "name": "Transform Data",
                "description": "This will transform work space data.",
                "type": "DataLakeAnalyticsU-SQL",
                "dependsOn": [
                    {
                        "activity": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
                        "dependencyConditions": [
                            "Completed"
                        ]
                    }
                ],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false
                },
                "typeProperties": {
                    "scriptPath": "Scripts/Script.usql",
                    "scriptLinkedService": {
                        "referenceName": "Destination_DataLakeStore_lc0",
                        "type": "LinkedServiceReference"
                    }
                },
                "linkedServiceName": {
                    "referenceName": "AzureDataLakeAnalyticsForDF",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "parameters": {
            "windowStart": {
                "type": "String"
            },
            "windowEnd": {
                "type": "String"
            }
        }
    }
}

1 回答

  • 1

    我通过使用AppService创建运行时解决了这个问题 . 我按照以下步骤操作 .

    • 我在Active Directory中创建了一个WebApp .

    • 也将Azure Data Lake权限分配给该WebApp .

    • 在该应用程序中创建一个公钥并记下它 . 它永远不会再显示出来 .

    • 注意该WebApp的应用程序ID .

    • 打开Azure Data Lake Analytics并分配参与者角色,以在Active Directory中创建WebApp .

    • 在创建运行时时,将Application ID用作服务主体ID和公钥作为WebApp的服务主键 .

    它工作正常 . :)

相关问题