Prudent Devs

Analysing Twitter With ELK Tools

Logstash, ElasticSearch & Kibana makes it easy to analyse any logs. In this post, I analyse twitter stream for AppSec tweets with ELK tools.

Logstash, ElasticSearch & Kibana makes it easy to analyse logs. To a programmer’s mind, logs can only mean application logs. But Logstash considers anything with a timestamp as a log. One such log input is twitter. As I’m getting more and more interested in appsec, I thought it will be cool to analyse twitter stream for #appsec tweets.

Refer my earlier post to know how to install ELK tools and connect each of them.

1. Create a Twitter app

To fetch twitter stream into logstash, you need twitter API keys. You need to create a twitter app for that. Goto Twitter Apps Page and create a new app. Once the app is created, it will show API Key & API secret. Then, you need to generate access keys. There is a button at the bottom page to generate these access keys. Generate them.

2. Configure Logstash via a config file

In the config file, Twitter API Keys & API secret goes under consumer_key & consumer_secret and access token and access token secret go as oauth_token and oauth_token_secret. For further reference refer to Logstash documentation

# connect twitter to ELK
input {
  twitter {
    consumer_key => "xxxxx"
    consumer_secret => "xxxxx"
    oauth_token => "xxxxx"
    oauth_token_secret => "xxxxx"
    keywords => ["appsec", "#appsec", "AppSec", "#AppSec"]
    tags => ["appsec"]
    type => "appsec"

output {
  stdout {
    codec => rubydebug
  elasticsearch_http {
    host => "localhost"

Ensure that the config file params (like, consumer_key) are in lower case. If you use elasticsearch_http as an output, then you are not limited by the ElasticSearch version that is currently compatible with Logstash.

3. Fetching Twitter stream

Run logstash with the above configuration file. -f indicates logstash should use the configuration file. I also run with --debug the first time to identify issues and troubleshoot them.

/opt/logstash/bin/logstash -f twitter-logstash.conf  --debug

Since appsec is a slow traffic topic, I had to run it over the night to get 14 entries :-(

4. Deal with the incoming tweets

All tweets that Logstash fetches will be stored in ES. ES has a REST API through which you can access / delete the individual tweets. As an example, let us say you can to delete all the tweets created by logstash, you would issue the below command (the last parameter is the index name which you can know from http://localhost:9200/_search?pretty):

curl -XDELETE 'http://localhost:9200/logstash-2014.07.25'

These tweets can be also seen from Kibana dashboard at: http://localhost:8080/index.html#/dashboard/file/logstash.json.

Kibana Dashboard For AppSec

Published On:
Under: #code , #data , #elk
Sign up for my newsletter