James Harr, Univ. Nebraska
ELK stack – ElasticSearch, Logstash, Kibana, (+redis)
ElasticSearch indexes and analyzes JSON – no foreign keys or transactions, scalable, fast, I/O friendly. Needs lots of RAM
KIbana – WebUI to query ElasticSearch and visualize data.
Log stash – “a unix pipe on steroids” – start with input and output, but can add conditional filters (e.g. regex). Add-on tools like mutate and grok. Can have multiple inputs and outputs.
GROK – has a set of prebuilt regular expressions. Makes it easy to grab things and stuff them into fields. Have to do it on the way in not after the fact (it’s a pipe tool). 306 built-in patterns.
Grok GeoIP – includes built in database, breaks out geo data into fields.
LogStash – statsd – sums things up – give it key and values, adds values and once a minute sends to another tool.
Graphite – a graphing tool, easy to use. Three pieces of info per line: key you want logged to, time, value. Will create a new metric if it’s not in the database.
Can listen to twitter data with LogStash.
Redis – Message queue server
Queue – like a mailbox, can have multiple senders and receivers, but each message goes to one receiver. No receiver, messages pile up.
Channel (pub/sub) – like the radio, each message goes to all subscribers. No subscriber? message is lost, publisher is not held up. Useful for debugging.
Composing a log system: Logstash is not a single service: split up concerns, use queues to deal with bursts, errors. use channels to troubleshoot.
General architecture – start simple:
Collector -> queue -> analyzer -> ElasticSearch -> Kibana
Keep collectors simple – reliability and speed are the goal. A single collector can listen to multiple things.
Queue goes into Redis. Most work done in analyzer – groking, passing things to statsd, etc.Can run multiple instances.
Channels can be used to also send data to other receivers.
Composing a Log System – Archiving
collector -> queue -> analyzer -> archive -> archiver -> log file
JSON compresses very well. Do archiving after analyzer so all the fields are broken out.
Split out indices so you can have different retention policies, dashboards, etc. e.g. firewall data different than log stash.
Can use logstash to read syslog data from a device, filter out what you want to send to Splunk to get your data volume down.
Lessons (technical): clear query cache regularly (cron job every morning); more RAM is better, but the JVM doesn’t behave well after 32GB; Split unrelated data into indices (e.g. syslog messages vs. firewall logs); part simple; use channels to try new things.
Lessons: It’s not about full text search, though that’s nice. It’s about having data analytics. ElasticSearch, LogStash, and Kibana are just tools in your toolbox. If you don’t have enough resources to keep everything, prune what you don’t need.
Oren Sreebny 