Monitoring your rails application with ActiveSupport Notifications and Elasticsearch

Here I explain how i set up monitoring for my rails application using ActiveSupport::Notifications and ElasticSearch.

The ActiveSupport::Notifications API is very useful for monitoring your rails application. It fires events for every SQL Query as well as for http requests/controller actions and rendered templates. This information can be gathered and used for new-relic-like monitoring of your application. You first have to subscribe to the ActiveSupport::Notifications:

module EventMonitoring

  if Rails.env.test?
    ADDRESS = 'http://localhost:10500'
    OPTIONS = {:log_level => :debug}
  else
    ADDRESS = 'http://localhost:9500'
    OPTIONS = {}
  end

  def self.initialize_dispatcher
    return nil if Rails.env.test?
    msg = "Establishing connection to MonitoringSystem (#{ADDRESS})"
    server = Stretcher::Server.new(ADDRESS, OPTIONS)
    dispatcher = BufferedDispatcher.new(ElasticEventDispatcher.new(server))
    server.up?
  rescue Faraday::Error::ConnectionFailed => e
    puts "#{msg}: #{Rainbow('FAIL').red}"
    return nil
  else
    puts "#{msg}: #{Rainbow('SUCCESS').green}"
    return dispatcher
  end

  DISPATCHER = initialize_dispatcher

  def self.initialized?
    dispatcher.present?
  end

  def self.dispatcher
    DISPATCHER
  end

  def self.flush
    dispatcher.flush
  end

  def self.fire(event)
    dispatcher.dispatch(event)
  end

  def self.fire_notification(name, start, finish, tx_id, payload)
    event = {
      _type:    'event',
      type:     name,
      start:    start,
      finish:   finish,
      context:  tx_id,
      data:     payload
    }

    fire event
  end

  def self.start!
    if initialized?
      ActiveSupport::Notifications.subscribe // do |*args|
        fire_notification(*args)
      end
    else
      Rails.logger.warn { "Monitoring not initialized" }
    end

  end

end

I use a separate ES instance for the monitoring data. The separate ES instance has its own port and configuration file. Also, I create one index per day which stores the ActiveSupport::Notifications of that day. The start!-method subscribes to all ActiveSupport::Notifications by passing the match-all regex //. The events are then passed to the fire_notification-method which does some simple conversion and passes the data to a dispatcher. The dispatcher is responsible for indexing the data in an elasticsearch-instance. In this case the dispatcher is a BufferedDispatcher, which means it does not index one event at a time but instead puts the events in a buffer and flushes them after a while to index them as a bulk of events.

And then i have a CoffeeScript-frontend that queries the ElasticSearch-instance directly to display graphs based on the events. ES 1.0 has aggregations that can calculate the average, minimum and maximum request times from the process_action.action_controller events. That query looks like this:

"""
  {
    "query": {
      "filtered": {
        "query" : { "match_all": {} },
        "filter" : {
          "and" : [
            {"terms" : { "type": ["process_action.action_controller"] }},
            {"range" : {
              "start": {
                "gte" : "#{min}",
                "lt" : "#{max}"
              }
            }}
          ]
        }
      }
    },
    "aggregations" : {
      "list" : {
        "date_histogram" : {
          "field" : "start",
          "interval" : "#{@interval}s"
        },
        "aggregations" : {
          "stats" : {
            "stats" : { "script" : "doc['finish'].value - doc['start'].value" }
          }
        }
      }
    }
  }
"""

By using ActiveSupport::Notifications and ElasticSearch it is very easy to add custom events and graphs for them. E.g. you can fire an event for every login attempt (including whether it succeeded or failed) or for each comment that is posted on your platform.