ruby – 在Logstash中转换时间戳时区以获取输出索引名称

发布时间：2020-12-17 04:02:12 所属栏目：百科来源：网络整理

导读：在我的场景中,Logstash接收的syslog行的“timestamp”是UTC,我们在Elasticsearch输出中使用事件“timestamp”： output { elasticsearch { embedded = false host = localhost port = 9200 protocol = http cluster = 'elasticsearch' index = "syslog-%{+YY

在我的场景中,Logstash接收的syslog行的“timestamp”是UTC,我们在Elasticsearch输出中使用事件“timestamp”：

output {
    elasticsearch {
        embedded => false
        host => localhost
        port => 9200
        protocol => http
        cluster => 'elasticsearch'
        index => "syslog-%{+YYYY.MM.dd}"
    }
}

我的问题是,在UTC午夜,Logstash在一天结束之前在时区(GMT-4 => America / Montreal)发送日志到不同的索引,并且索引在20h(晚上8点)之后没有日志,因为“时间戳“是UTC.

我们已经完成了转换时区的工作,但我们遇到了显着的性能下降：

filter {
    mutate {
        add_field => {
            # Create a new field with string value of the UTC event date
            "timestamp_zoned" => "%{@timestamp}"
        }
    }

    date {
        # Parse UTC string value and convert it to my timezone into a new field
        match => [ "timestamp_zoned","yyyy-MM-dd HH:mm:ss Z" ]
        timezone => "America/Montreal"
        locale => "en"
        remove_field => [ "timestamp_zoned" ]
        target => "timestamp_zoned_obj"
    }

    ruby {
        # Output the zoned date to a new field
        code => "event['index_day'] = event['timestamp_zoned_obj'].strftime('%Y.%m.%d')"
        remove_field => [ "timestamp_zoned_obj" ]
    }
}

output {
    elasticsearch {
        embedded => false
        host => localhost
        port => 9200
        protocol => http
        cluster => 'elasticsearch'
        # Use of the string value
        index => "syslog-%{index_day}"
    }
}

有没有办法优化此配置？

解决方法

这是优化配置,请试一试并测试性能.

您不需要使用mutate和date插件.直接使用ruby插件.

input {
    stdin {
    }
}

filter {
    ruby {
            code => "
                    event['index_day'] = event['@timestamp'].localtime.strftime('%Y.%m.%d')
            "
    }
}

output {
    stdout { codec => rubydebug }
}

示例输出：

{
       "message" => "test","@version" => "1","@timestamp" => "2015-03-30T05:27:06.310Z","host" => "BEN_LIM","index_day" => "2015.03.29"
}

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!