首页 文章

HDinsight配置单元活动管道不在Azure数据湖中创建输出

提问于
浏览
0

尝试使用数据工厂运行配置单元活动,管道完成正常,并且在集群内部创建数据表,但输出数据集不是在Azure数据湖存储中创建文件,这是故意的吗?

只是想学习如此温柔 .

Input dataset:

包含数据的标准输入csv文件

{
"name": "dlsinput",
"properties": {
    "published": false,
    "type": "AzureDataLakeStore",
    "linkedServiceName": "dls",
    "typeProperties": {
        "fileName": "output.csv",
        "folderPath": "data/output/",
        "format": {
            "type": "TextFormat",
            "columnDelimiter": ","
        }
    },
    "availability": {
        "frequency": "Day",
        "interval": 1
    },
    "external": true,
    "policy": {}
}
}

Pipeline:

管道指的是hdinsight集群

{
"name": "HiveActivitySamplePipeline",
"properties": {
    "activities": [
        {
            "type": "HDInsightHive",
            "typeProperties": {
                "scriptPath": "scripts/hive.hql",
                "scriptLinkedService": "sta"
            },
            "inputs": [
                {
                    "name": "dlsinput"
                }
            ],
            "outputs": [
                {
                    "name": "dlsoutput"
                }
            ],
            "scheduler": {
                "frequency": "Day",
                "interval": 1
            },
            "name": "HiveActivitySample",
            "linkedServiceName": "hdi"
        }
    ],
    "start": "2018-04-05T12:20:00Z",
    "end": "2018-04-10T23:59:59Z",
    "isPaused": false,
    "hubName": "adf",
    "pipelineMode": "Scheduled"
}
}

Output:

我希望创建的文件输出

{
"name": "dlsoutput",
"properties": {
    "published": false,
    "type": "AzureDataLakeStore",
    "linkedServiceName": "dls",
    "typeProperties": {
        "fileName": "myfile.csv",
        "folderPath": "data/output/",
        "format": {
            "type": "TextFormat",
            "rowDelimiter": "\n",
            "columnDelimiter": ","
        }
    },
    "availability": {
        "frequency": "Day",
        "interval": 1
    }
}
}

Hive.hql

DROP TABLE IF EXISTS temp;
CREATE EXTERNAL TABLE IF NOT EXISTS temp (
Name STRING,
Road STRING,
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
 LOCATION 'adl://dls.azuredatalakestore.net/data/output/';

1 回答

  • 0

    看起来是因为Hive.hql只包含创建表的命令,并且没有数据插入到表中,因此您没有看到生成的任何数据文件 .

相关问题