我正在构建一个 Cloud 功能,具有存储触发器并尝试从内部运行数据流 . 我创建了一个数据流作业"jsonbq-1",使用现有的 GCS_Text_BigQuery
模板创建了一个简单的uDF来获取传入的CSV数据并放入表中 . 执行Cloud函数并调用Dataflow但没有响应,并且 log:Error
:模板参数中显示的错误无效 .
我不确定错误在哪里 .
Index.js looks like this.
`const google = require('googleapis');
//const { auth } = require('google-auth-library');
exports.goWithTheDataFlow = (event, callback) => {
const file = event.data;
const context = event.context;
console.log(`Event ${context.eventId}`);
console.log(` Event Type: ${context.eventType}`);
console.log(` Bucket: ${file.bucket}`);
console.log(` File: ${file.name}`);
console.log(` Metageneration: ${file.metageneration}`);
console.log(` Created: ${file.timeCreated}`);
console.log(` Updated: ${file.updated}`);
// console.log(err);
//console.log(projectId);
//console.log(authClient);
google.auth.getApplicationDefault(function (err, authClient, projectId) {
if (err) {
throw err;
}
console.log(err)
console.log(projectId);
const dataflow = google.dataflow({ version: 'v1b3', auth: authClient });
console.log(`gs://${file.bucket}/${file.name}`);
dataflow.projects.templates.create({
projectId: projectId,
resource: {
parameters: {
inputFile: `gs://${file.bucket}/${file.name}`,
output_Table: 'titanium-gamma-212906:np_cf_1.cf1',
//bigQueryLoadingTemporaryDirectory: 'gs://test-bucket-np'
},
jobName: 'json-bq1',
gcsPath: 'gs://dataflow-templates/latest/GCS_Text_to_BigQuery',
//jobName: 'json-bq1',
//gcsPath: 'gs://dataflow-templates/latest/GCS_Text_to_BigQuery',
//output_Table: 'titanium-gamma-212906:np_cf_1.cf1',
//javascriptTextTransformFunctionName: 'transform',
//bigQueryLoadingTemporaryDirectory: 'gs://test-bucket-np'
}
}, function(err, response) {
if (err) {
console.error("problem running dataflow template, error was: ", err);
}
console.log("Dataflow template response: ", response);
callback();
});
});
callback();
};`
package.json
"name": "sample-cloud-storage",
"version": "0.0.1",
"dependencies": {
"googleapis": "24.0.0"
}
}
The LOG:
在此先感谢大家 .
1 回答
GCS_Text_to_Bigquery模板在调用中似乎缺少几个必需的参数:
“您要处理的文本的GCS位置”inputFilePattern
“带有BigQuery架构描述的JSON文件”JSONPath
“要写入的输出主题”outputTable
“用于转换输出的javascript fn的gCS路径”javascriptTextTransformGcsPath
“UDF Javascript函数名称”javascriptTextTransformFunctionName
“BigQuery加载过程的临时目录”bigQueryLoadingTemporaryDirectory