我有一个Angular 6应用程序,它从AWS Lambda请求数据 . 数据本身存储在Glue数据库中,并使用AWS Athena进行查询 . AWS Glue数据库设置了 skip.header.line.count=1
选项,当我在控制台中运行Athena查询时,我得到一个没有 Headers 的响应 . 当我尝试使用 boto3
检索数据时,会发生此问题 . 我有一个运行查询的函数,然后在结果中分页:
def run_query_paged(self, query, page_token=None, page_size=10):
"""
Run query.
"""
request = self.athena_client.start_query_execution(
QueryString=query,
QueryExecutionContext={
'Database': self.database
},
ResultConfiguration={
'OutputLocation': self.s3_output,
}
)
execution_id = request['QueryExecutionId']
if execution_id:
while True:
stats = self.athena_client.get_query_execution(QueryExecutionId=execution_id)
status = stats['QueryExecution']['Status']['State']
if status in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
break
time.sleep(0.2) # 200ms
if status == 'SUCCEEDED':
paginator = self.athena_client.get_paginator('get_query_results')
pagination_config = {
'MaxItems': page_size,
'PageSize': page_size,
}
if page_token:
pagination_config['StartingToken'] = page_token
response_iterator = paginator.paginate(
QueryExecutionId=execution_id,
PaginationConfig=pagination_config
)
for page in response_iterator:
next_token = page.get('NextToken', '')
results = page
break
return {
'rows': process_results(results),
'nextToken': next_token
}
if status == 'FAILED':
raise Exception(stats['QueryExecution']['Status']['StateChangeReason'])
return None
process_results
函数将响应转换为考虑列类型的列表:
def process_results(response):
"""
Processes the result of get_query_results function
"""
rows = response['ResultSet']['Rows']
meta = response['ResultSet']['ResultSetMetadata']['ColumnInfo']
result = []
for row in rows:
parsed_row = {}
for idx, val in enumerate(row['Data']):
field = val
column_info = meta[idx]
if 'VarCharValue' in val:
value = val['VarCharValue']
else:
value = ''
parsed_row[column_info['Name']] = process_row_value(value, column_info)
result.append(parsed_row)
return result
问题是分页响应的第一页具有列名称的 Headers ,如下所示:
{
"foo": "foo",
"bar": "bar"
},
{
"foo": 1,
"bar": 2
},
...
而所有其他页面都没有 . 当我从客户端应用程序请求第一页时,我得到一个 Headers 加上9行(页面大小为10),当我使用 NextToken
请求下一页时,我得到10行没有 Headers . 在第一页显示9个项目并且在所有结果中显示10个项目是非常尴尬的 .
如何跳过 Headers 跳过结果?