Elastic-Stack on Centos7

What is Elastic-Stack?

Elasticsearch

A distributed, fast, highly scalable document database

 

Logstash

Aggregates, filters and supplements log data

Forwards altered/formatted logs to Elasticsearch

Sending data directly to Elastic Search without Logstash can lead to inconsistent data

  • Configuration
    • input {} Where is the data coming from? Logs? Beats?
    • filter {} How should we parse the data? Ignore some? Modify any?
    • output {} Where should we store the logs? Backend? Elasticsearch?

 

Kibana

Web-based front end

Works easily with Elasticsearch for charts, graphs and visualising data

 

Beats

Small, lightweight utilities for reading logs from a variety of sources, which usually send them on to Logstash.

These utilities are written in golang, which compiles to a static binary, so no JavaRTE is required for them to run. Go applications can be cross-compiled to run on Linux, BSD, Mac OS X or Windows and are usually small and lightweight, making go a great language for system utilities.

Filebeat: Text log files

Metribeat: OS and applications

  • Apache
  • HAProxy
  • MongoDB
  • MySQL
  • NginX
  • PostgreSQL
  • Redis
  • Zookeeper
  • System Logs

Packetbeat: Network monitoring

These utilities can send their data to:-

  • Elasticsearch
  • Logstash
  • Redis

 

Installation

Usually, I would do this with Chef, but here’s how to do it by hand!

Before we start to install the software we need to create a few RPM repositories. By adding the repositories rather than just installing the RPMs, it will make it much easier to update and maintain later.

Change directory to where the RPM repos are.

cd /etc/yum.repos.d

Create a repo definition for elasticsearch

vi elasticsearch.repo

[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Create a repo definition for logstash

vi logstash.repo

[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Create a repo definition for kibana

vi kibaba.repo

[kibana-5.x]
name=Kibana repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Create a repo definition for grafana

vi grafana.repo

[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/6/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Now install the software.

We also want to install the beat utilities, but luckily for us, they are also contained in the elastic search repository.

yum -y install elasticsearch filebeat metricbeat heartbeat packetbeat logstash kibana grafana

Configure logstash give it’s output to elasticsearch

logstash -e 'input { stdin { } } output { elasticsearch { hosts =>  ["a.b.c.d:9200"] } }'

But where does it keep its configuration?

/etc/logstash/conf.d