How to create separate Indices in elasticsearch for each filebeat host using logstash configuration

Bydefault the data from all hosts will be inserted to the common index filebeat. If separate index required for each host, we can configure it in logstash configuration.

This article explains the steps to configure logstash for creating separate indices in elasticsearch corresponding to filebeat host.

Step1: Login to the filebeat instance via ssh as root.

Step2: Open the filebeat configuration file.

#vi /etc/filebeat/filebeat.yml

Step3: Add the server name in General configuration block.

name: SERRVER

Where SERVER need to be replaced with the required name of the server.

Step4: Restart filebeat service.

#systemctl restart filebeat

Step5: Login to the logstash server via ssh as root.

Step6: Open the logstash configuration file.

#vi /etc/logstash/logstash.conf

Step7: Modify the output section with the following if condition.

output {
    if "SERVER" in [host][name] {
      elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "SERVER-%{+YYYY.MM.dd}"

      }
    }
    else
    {
    elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
            }
    }
}

Where SERVER need to be replaced with the required name of the server. This if condition need to be repeated for all those hosts that requires separate indices.

Step8: Restart logstash service.

#systemctl restart logstash

That's all…