Monday, January 19, 2015

Logstash Examples with Data Generators

Logstash is a pretty cool application that generically takes some input, does something with the data, and spew it out somewhere else.

It's commonly used in conjunction with Elasticsearch, a Lucene based search service, and Kibana, a dashboard UI for Elasticsearch.  You can easily consider this as a "Splunk without super awesome ad hoc query capabilities" but the software itself is FREE so there's that.

The documentation for Logstash is pretty straightfoward, but I thought it would be nice to have some hard examples to work off of that involved the whole ELK stack:

Right now it just has two examples: file and log4j.

  • Write to a file
  • Logstash monitors the specific file and does some grokking before passing it to Elasticsearch
  • Load the provided kibana dashboard.
  • Spew log4j stuff out with SocketAppender
  • Set up Logstash to monitor a port for log4j messages and grok before passing it to Elasticsearch
  • load the provided Kibana dashboard
So yeah I hope this serves as a quick and easy way for people to observe the awesomeness that is Logstash.  Feel free to add your own examples to this guy as well.