我使用Azure Data Factory将本地数据传输到Azure Data Lake存储 . 复制数据后,我在上传的文件上运行U-SQL脚本,将其转换为新的.csv文件 . 如果从Visual Studio或直接从Data Lake Analytics运行,我的U-SQL作业运行正常 .
但是如果在Azure Data Factory中添加和活动 . 复制数据后,此U-SQL脚本活动立即失败 . 我尝试了很多方法但无法解决问题 . 它给了我以下错误 .
我的U-SQL活动的JSON定义是
{
"name": "Transform Data",
"description": "This will transform work space data.",
"type": "DataLakeAnalyticsU-SQL",
"dependsOn": [
{
"activity": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
"dependencyConditions": [
"Completed"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false
},
"typeProperties": {
"scriptPath": "Scripts/Script.usql",
"scriptLinkedService": {
"referenceName": "Destination_DataLakeStore_lc0",
"type": "LinkedServiceReference"
}
},
"linkedServiceName": {
"referenceName": "AzureDataLakeAnalyticsForDF",
"type": "LinkedServiceReference"
}
}
JSON of entire pipeline is
{
"name": "CopyPipeline_d26",
"properties": {
"activities": [
{
"name": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
"type": "Copy",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false
},
"typeProperties": {
"source": {
"type": "FileSystemSource",
"recursive": false
},
"sink": {
"type": "AzureDataLakeStoreSink",
"copyBehavior": "MergeFiles"
},
"enableStaging": false,
"cloudDataMovementUnits": 0,
"enableSkipIncompatibleRow": true
},
"inputs": [
{
"referenceName": "workplace_groups_info_2018_03_19_09_32_csv_i_lc0",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "workplace_groups_info_2018_03_19_09_32_csv_o_lc0",
"type": "DatasetReference"
}
]
},
{
"name": "Transform Data",
"description": "This will transform work space data.",
"type": "DataLakeAnalyticsU-SQL",
"dependsOn": [
{
"activity": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
"dependencyConditions": [
"Completed"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false
},
"typeProperties": {
"scriptPath": "Scripts/Script.usql",
"scriptLinkedService": {
"referenceName": "Destination_DataLakeStore_lc0",
"type": "LinkedServiceReference"
}
},
"linkedServiceName": {
"referenceName": "AzureDataLakeAnalyticsForDF",
"type": "LinkedServiceReference"
}
}
],
"parameters": {
"windowStart": {
"type": "String"
},
"windowEnd": {
"type": "String"
}
}
}
}
1 回答
我通过使用AppService创建运行时解决了这个问题 . 我按照以下步骤操作 .
我在Active Directory中创建了一个WebApp .
也将Azure Data Lake权限分配给该WebApp .
在该应用程序中创建一个公钥并记下它 . 它永远不会再显示出来 .
注意该WebApp的应用程序ID .
打开Azure Data Lake Analytics并分配参与者角色,以在Active Directory中创建WebApp .
在创建运行时时,将Application ID用作服务主体ID和公钥作为WebApp的服务主键 .
它工作正常 . :)