telegraf - exec插件 - aws ec2 ebs volumen info - 度量解析错误，原因：[缺少字段]或遇到错误：[无效数字]

Machine - CentOS 7.2或Ubuntu 14.04 / 16.xx

Telegraf 版本：1.0.1

Python 版本：2.7.5

Telegraf支持名为：exec的INPUT插件 . 首先请参见 README doc中的 EXAMPLE 2 . 我不能使用JSON格式，因为它只使用指标的数字值 . 根据文档：

If using JSON, only numeric values are parsed and turned into floats. Booleans and strings will be ignored.

所以，这个想法很简单，你在exec插件部分指定一个脚本，它应该吐出一些有意义的信息（以JSON - 或 - 流入数据格式 in my case ，因为我有一些包含非数字值的指标）你想要的在一个很酷的仪表板中捕获/显示，例如 Wavefront Dashboard ，如下所示：

：

基本上，人们可以使用这些指标，标签，来自这些指标来源的来源，找出有关内存，CPU，磁盘，网络，其他有意义信息的各种信息，并在发生不需要的事情时使用这些信息创建警报 .

好的，我想出了这个python脚本：

#!/usr/bin/python

# sudo pip install boto3 if you don't have it on your machine.
import boto3


def generate(key, value):
    """
    Creates a nicely formatted Key(Value) item for output
    """
    return '{}="{}"'.format(key, value)
    #return '{}={}'.format(key, value)


def main():
    ec2 = boto3.resource('ec2', region_name="us-west-2")
    volumes = ec2.volumes.all()

    for vol in volumes:
        # You don't need to wrap everything in `str` unless it is not a string
        # By default most things will come back as a string 
        # unless they are very obviously not (complex, date time, etc)
        # but since we are printing these (and formatting them into strings)
        # the cast to string will be implicit and we don't need to make it 
        # explicit


        # vol is already a fully returned volume you are essentially DOUBLING
        # your API calls when you do this
        #iv = ec2.Volume(vol.id)
        output_parts = [
            # Volume level details
            generate('create_time', vol.create_time),
            generate('availability_zone', vol.availability_zone),
            generate('volume_id', vol.volume_id),
            generate('volume_type', vol.volume_type),
            generate('state', vol.state),
            generate('size', vol.size),
            generate('iops', vol.iops),
            generate('encrypted', vol.encrypted),
            generate('snapshot_id', vol.snapshot_id),
            generate('kms_key_id', vol.kms_key_id),
        ]

        for _ in vol.attachments:
            # Will get any attachments and since it is a list
            # we should write this to handle MULTIPLE attachments
            output_parts.extend([
                generate('InstanceId', _.get('InstanceId')),
                generate('InstanceVolumeState', _.get('State')),
                generate('DeleteOnTermination', _.get('DeleteOnTermination')),
                generate('Device', _.get('Device')),
            ])

        # only process when there are tags to process        
        if vol.tags:
            for _ in vol.tags:
                # Get all of the tags
                output_parts.extend([
                    generate(_.get('Key'), _.get('Value')),
                ])

        # output everything at once.. 
        print ','.join(output_parts)


if __name__ == '__main__':
    main()

此脚本将与AWS EC2 EBS卷对话并输出它可以找到的所有值（通常是您在AWS EC2 EBS卷控制台中看到的），并将该信息格式化为有意义的CSV格式，我将其重定向到.csv日志文件 . 我们不希望一直运行python脚本（AWS API限制/成本因素） .

因此，一旦创建了.csv文件，我就创建了这个小的shell脚本，我将在Telegraf 's exec plugin'部分中设置它 .

Shell script /tmp/aws-vol-info.sh 在Telegraf exec插件中设置为：

#!/bin/bash

cat /tmp/aws-vol-info.csv

使用exec插件（ /etc/telegraf/telegraf.d/exec-plugin-aws-info.conf ）创建的Telegraf配置文件：

#--- https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec

[[inputs.exec]]
  commands = ["/tmp/aws-vol-info.sh"]

  ## Timeout for each command to complete.
  timeout = "5s"

  # Data format to consume.
  # NOTE json only reads numerical measurements, strings and booleans are ignored.
  data_format = "influx"

  name_suffix = "_telegraf_execplugin"

我 tweaked .py（用于生成函数的Python脚本）生成以下三种类型的输出格式（.csv文件），并希望 test 在启用配置文件（/ etc / telegraf / telegraf）之前，telegraf将如何处理这些数据 . d / catch-aws-ebs-info.conf）并重启 telegraf service .

Format 1: （每个值都包含双引号 " ）

create_time="2017-01-09 23:24:29.428000+00:00",availability_zone="us-east-2b",volume_id="vol-058e1d47dgh721121",volume_type="gp2",state="in-use",size="8",iops="100",encrypted="False",snapshot_id="snap-06h1h1b91bh662avn",kms_key_id="None",InstanceId="i-0jjb1boop26f42f50",InstanceVolumeState="attached",DeleteOnTermination="True",Device="/dev/sda1",Name="[company-2b-app90] secondary",hostname="company-2b-app90-i-0jjb1boop26f42f50",high_availability="1",mirror="secondary",cluster="company",autoscale="true",role="app"

在telegraf目录上测试 telegraf 配置会给我以下错误 .

Command ： $ telegraf --config-directory=/etc/telegraf --test --input-filter=exec

[vagrant@myvagrant ~] $ telegraf --config-directory=/etc/telegraf --test --input-filter=exec
2017/03/10 00:37:48 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.exec, Collection 1
2017-03-10T00:37:48Z E! Errors encountered: [ metric parsing error, reason: [invalid field format], buffer: [create_time="2017-01-09 23:24:29.428000+00:00",availability_zone="us-east-2b",volume_id="vol-058e1d47dgh721121",volume_type="gp2",state="in-use",size="8",iops="100",encrypted="False",snapshot_id="snap-06h1h1b91bh662avn",kms_key_id="None",InstanceId="i-0jjb1boop26f42f50",InstanceVolumeState="attached",DeleteOnTermination="True",Device="/dev/sda1",Name="[company-2b-app90] secondary",hostname="company-2b-app90-i-0jjb1boop26f42f50",high_availability="1",mirror="secondary",cluster="company",autoscale="true",role="app"], index: [372]]
[vagrant@myvagrant ~] $

Format 2: （没有任何 " 双引号）

create_time=2017-01-09 23:24:29.428000+00:00,availability_zone=us-east-2b,volume_id=vol-058e1d47dgh721121,volume_type=gp2,state=in-use,size=8,iops=100,encrypted=False,snapshot_id=snap-06h1h1b91bh662avn,kms_key_id=None,InstanceId=i-0jjb1boop26f42f50,InstanceVolumeState=attached,DeleteOnTermination=True,Device=/dev/sda1,Name=[company-2b-app90] secondary,hostname=company-2b-app90-i-0jjb1boop26f42f50,high_availability=1,mirror=secondary,cluster=company,autoscale=true,role=app

Getting same error 在测试Telegraf的exec插件配置时：

2017/03/10 00:45:01 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.exec, Collection 1
2017-03-10T00:45:01Z E! Errors encountered: [ metric parsing error, reason: [invalid value], buffer: [create_time=2017-01-09 23:24:29.428000+00:00,availability_zone=us-east-2b,volume_id=vol-058e1d47dgh721121,volume_type=gp2,state=in-use,size=8,iops=100,encrypted=False,snapshot_id=snap-06h1h1b91bh662avn,kms_key_id=None,InstanceId=i-0jjb1boop26f42f50,InstanceVolumeState=attached,DeleteOnTermination=True,Device=/dev/sda1,Name=[company-2b-app90] secondary,hostname=company-2b-app90-i-0jjb1boop26f42f50,high_availability=1,mirror=secondary,cluster=company,autoscale=true,role=app], index: [63]]

Format 3: （此格式在值中没有任何 " 双引号和空格 `` 字符） . 具有 _ 字符的替换空格 .

create_time=2017-01-09_23:24:29.428000+00:00,availability_zone=us-east-2b,volume_id=vol-058e1d47dgh721121,volume_type=gp2,state=in-use,size=8,iops=100,encrypted=False,snapshot_id=snap-06h1h1b91bh662avn,kms_key_id=None,InstanceId=i-0jjb1boop26f42f50,InstanceVolumeState=attached,DeleteOnTermination=True,Device=/dev/sda1,Name=[company-2b-app90]_secondary,hostname=company-2b-app90-i-0jjb1boop26f42f50,high_availability=1,mirror=secondary,cluster=company,autoscale=true,role=app

Still didn't work ，得到同样的错误：

[vagrant@myvagrant ~] $ telegraf --config-directory=/etc/telegraf --test --input-filter=exec
2017/03/10 00:50:30 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.exec, Collection 1
2017-03-10T00:50:30Z E! Errors encountered: [ metric parsing error, reason: [missing fields], buffer: [create_time=2017-01-09_23:24:29.428000+00:00,availability_zone=us-east-2b,volume_id=vol-058e1d47dgh721121,volume_type=gp2,state=in-use,size=8,iops=100,encrypted=False,snapshot_id=snap-06h1h1b91bh662avn,kms_key_id=None,InstanceId=i-0jjb1boop26f42f50,InstanceVolumeState=attached,DeleteOnTermination=True,Device=/dev/sda1,Name=[company-2b-app90]_secondary,hostname=company-2b-app90-i-0jjb1boop26f42f50,high_availability=1,mirror=secondary,cluster=company,autoscale=true,role=app], index: [476]]

Format 4 ：如果我按照此页面关注 influx line protocol ：https://docs.influxdata.com/influxdb/v1.2/write_protocols/line_protocol_tutorial/

awsebs,Name=[company-2b-app90]_secondary,hostname=company-2b-app90-i-0jjb1boop26f42f50,high_availability=1,mirror=secondary,cluster=company,autoscale=true,role=app create_time=2017-01-09_23:24:29.428000+00:00,availability_zone=us-east-2b,volume_id=vol-058e1d47dgh721121,volume_type=gp2,state=in-use,size=8,iops=100,encrypted=False,snapshot_id=snap-06h1h1b91bh662avn,kms_key_id=None,InstanceId=i-0jjb1boop26f42f50,InstanceVolumeState=attached,DeleteOnTermination=True,Device=/dev/sda1

I'm getting this error ：

[vagrant@myvagrant ~] $ telegraf --config-directory=/etc/telegraf --test --input-filter=exec
2017/03/10 02:34:30 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.exec, Collection 1
2017-03-10T02:34:30Z E! Errors encountered: [ invalid number]

HOW 我可以摆脱这个错误并使用telegraf与exec插件（运行.sh脚本）一起工作吗？

Other Info ：

Python脚本每天运行一次/两次（通过cron），telegraf将每1分钟运行一次（运行exec插件 - 运行.sh脚本 - 这将捕获.csv文件，以便telegraf可以以流入数据格式使用它） .

https://galaxy.ansible.com/wavefrontHQ/wavefront-ansible/

https://github.com/influxdata/telegraf/issues/2525