Installing Graphite

Background

Graphite is a statistics collection framework for time series data. While there isn't a canonical Graphite installation, the framework consists of three main components (as they are packaged in EPEL):

• python-carbon: Data caching and persistence daemon. Listens on a socket, writes out time series data to disk.
• python-whisper: Library that provides a time-series database format. Replaces RRDTool in typical use cases.
• graphite-web: Django-based web frontend for graphing data.

The design is also quite modular, and there have been a number of additional statistics collection services and alternative front-ends developed. One such collector daemon is the nodejs-based "statsd", written by Etsy. Another is a more generic service known as "collectd". For our purposes, we'll deploy collectd and a meta-aggregator / graphite adapter known as Bucky. Both are available in EPEL.

Installing the Carbon collector

To setup the Whisper library and the Carbon collector daemon, yum install the following:

# yum -y install python-whisper python-carbon nc

(We additionally install netcat to test writing data into Carbon)

Once Carbon has been installed, two files are created in /etc/carbon:

• /etc/carbon/carbon.conf - Daemon configuration options, includes options for relaying/aggregation for scalability.
• /etc/carbon/storage-schemas.conf - Data retention time options.

We won't touch either of these right now, but it's good to be aware of them.

Finally, let's go ahead and start the Carbon daemon:

# /etc/init.d/carbon-cache start

Installing the Graphite front-end

The Graphite Django application, while not our end-goal, is nice for demonstration purposes. Here's what need to install it:

# yum install -y graphite-web python-memcached httpd memcached 

Create the initial database and superuser account:

# python /usr/lib/python2.6/site-packages/graphite/manage.py syncdb

Start the daemons:

# service carbon-cache start
# service memcached start
# service httpd start

At this point, you can try sending data to Graphite using netcat:

echo "my_arbitrary_metric 1024 $(date +%s)" | nc webapps.uc.mwt2.org 2003

Installing Bucky

We want to be able to pull data from collectd, but the version of collectd with a direct-to-graphite plugin doesn't exist in EPEL (at the time of writing). Instead, we'll need to use bucky as an adapter for collectd data. To setup the Bucky meta-aggregator, first install it from EPEL:

# yum -y install python-bucky 

I don't think there is any need to touch the Bucky configuration files, but they exist in /etc/bucky just in case.

At the time of this writing, the Bucky package in EPEL doesn't support init.d scripts. Fortunately, someone wrote one! (https://github.com/electrical/puppet-bucky/blob/master/files/etc/init.d/bucky.RedHat), so create this script as /etc/init.d/buckyd:

#!/bin/bash
# bucky Init script for running the bucky daemon
#
#
# chkconfig: - 98 02
#
# description: some description
# processname: bucky

PATH=/usr/bin:/sbin:/bin:/usr/sbin
export PATH

lockfile='/var/lock/subsys/bucky'
pidfile='/var/run/bucky.pid'
bucky='/usr/bin/bucky'
config='/etc/bucky/bucky.conf'
logfile='/var/log/bucky.log'

RETVAL=0

# Source function library.
. /etc/rc.d/init.d/functions

# Determine if we can use the -p option to daemon, killproc, and status.
# RHEL < 5 can't.
if status | grep -q -- '-p' 2>/dev/null; then
    pidopts="-p $pidfile"
fi

start() {
    echo -n $"Starting bucky daemon: "
    $bucky $config >> $logfile 2>&1 &
    RETVAL=$?

    local PID=`pgrep -f "${bucky} ${config}"`
    echo $PID > ${pidfile}

    [ $RETVAL -eq 0 ] && (touch ${lockfile}; echo_success) || echo_failure
    echo

    return $RETVAL
}

stop() {
    echo -n $"Stopping bucky daemon: "
    killproc $pidopts $bucky
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f ${lockfile} ${pidfile}
}

restart() {
    stop
    start
}

rh_status() {
    status $pidopts $bucky
    RETVAL=$?
    return $RETVAL
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        start
    ;;
    stop)
        stop
    ;;
    restart)
        restart
    ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
    ;;
    status)
        rh_status
    ;;
    *)
        echo $"Usage: $0 {start|stop|restart|condrestart|try-restart|status}"
        exit 1
esac

exit $RETVAL

Installing Collectd

Collectd should have been pulled in as a python-bucky dependency, but we also need the collect-snmp plugin. To install:

# yum install -y collectd collect-snmp

We need to then configure the Network and SNMP plugins to get collectd to send data. To enable network plugin, add these lines to /etc/collectd.conf:

LoadPlugin network
<Plugin "network">
      Server "127.0.0.1" "25826"
</Plugin>

Change the server address if needed (almost certainly for the non-trivial case).

To setup SNMP, you'll need to load the module and know what OID to probe. Here's an example for Cisco 6500 temperature:

LoadPlugin snmp
<Plugin snmp>
   <Data "cisco_module1_temp_inlet">
       Type "temperature"
       Table false
       Instance ""
       Values ".1.3.6.1.4.1.9.9.13.1.3.1.3.93"
   </Data>
   <Data "cisco_module1_temp_outlet">
       Type "temperature"
       Table false
       Instance ""
       Values ".1.3.6.1.4.1.9.9.13.1.3.1.3.92"
   </Data>
   <Host "uct2-6509">
       Address "128.135.158.241"
       Version 2
       Community "mwt2"
       Collect "cisco_module1_temp_outlet" "cisco_module1_temp_inlet"
       Interval 60
   </Host>
</Plugin>

(Optional) Collectd CSV plugin

I found that loading the CSV plugin is useful for debugging: Add these lines to your /etc/collectd.conf

LoadPlugin csv
<Plugin csv>
       DataDir "/usr/var/lib/collectd/csv"
       StoreRates false
</Plugin>

Your output will all go to the datadir.

References

• http://kevinmccarthy.org/blog/2013/07/18/10-things-i-learned-deploying-graphite/
• http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
• https://gist.github.com/LincolnBryant/8250385
• https://github.com/etsy/statsd/
• https://github.com/kenhub/giraffe
• https://pypi.python.org/pypi/bucky
• http://dev.nuclearrooster.com/2011/05/11/sending-metrics-to-statsd-from-bash/

-- LincolnBryant - 07 Jan 2014