Getting your metrics into Graphite couldn't be easier. With support for a simple plaintext format, anyone can send metrics with a minimum of fuss. Check this out.
$ echo "foo.bar 1 `date +%s`" | nc localhost 2003
Congratulations, you just submitted your first measurement to Graphite!
This metric would be stored in a Whisper database on your Graphite server's filesystem at <prefix>/storage/whisper/foo/bar.wsp (in most cases, your Graphite prefix should be /opt/graphite). If you're building a new graph in the Graphite Composer, you can find your metric bar nested under the foo folder.
Note that Graphite uses a dot-delimited namespace structure for metrics, graphs, and even dashboards. This can be handy for grouping and organizing your metrics according to the application, server, datacenter, or even the person submitting the metric.
Because you probably don't want to sit around firing off metrics by hand, I'd like to suggest looking into some of these common scenarios for submitting metrics to Graphite.
Sometimes you just need a quick and dirty way to record some metrics. No need to be embarrassed, we've all hacked today a cron job to measure something adhoc. Let's say we needed to track the number of instances of a particular process. This could be a handy way to alert when something is awry, but could just as easily be used to correlate problems elsewhere. Let's whip up something basic.
#!/bin/bash
CARBON_HOST="10.10.10.5"
CARBON_PORT="2003"
METRIC_NAME="debug.jdixon.procs.bad_java_app"
TIMESTAMP=`date +%s`
NUMBER_OF_WAYWARD_PROCS=`ps auwx | grep MY_BAD_JAVA_APP | wc -l`
echo ${METRIC_NAME} ${NUMBER_OF_WAYWARD_PROCS} ${TIMESTAMP} | nc ${CARBON_HOST} ${CARBON_PORT}
Make the script executable and add it to your crontab to run at routine intervals. Last but not least, make sure that your interval matches the precision for your metric schema (e.g. every 60 seconds) or you may end up with gaps in your graphs.
Tracking the performance of your application is a must. Without built-in telemetry, we're left to rely on low-level tracing tools. Thanks to tools like StatsD, we can fire off asynchronous measurements from our apps without concern for blocking behavior or having to instrument our own collection and aggregation mechanisms.
Etsy released the original StatsD implementation back in 2011 as an open source metrics aggregator. Since then, a huge ecosystem of language bindings and competing server implementations have grown up around it. With a few lines of code, anyone can instrument their application to fire off regular, non-blocking metrics to Graphite.
gem "statsd-ruby"
# Set up a global Statsd client for a server on localhost:8125
$statsd = Statsd.new 'localhost', 8125
# Send some stats
$statsd.increment 'app.foo.my_counter'
$statsd.timing 'app.foo.my_timer', 320
$statsd.gauge 'app.foo.my_gauge', 100
But wait... there's more! Depending on the metric type you're submitting, StatsD will also track and report statistics such as percentiles and histograms.
Monitoring servers (or these days, Docker containers) may be the most common use of Graphite as a time-series datastore. Collectd is a very popular collection agent thanks to its efficient performance (it's written in C), ease of configuration, and large library of read and write plugins. Most Linux distributions offer up-to-date binary packages. Installation is a breeze.
$ sudo apt-get update
$ sudo apt-get install -y collectd
A minor configuration change will be needed to direct Collectd to report metrics to your Graphite server. Edit the configuration file (typically found at /etc/collectd/collectd.conf) and then restart the collectd service.
LoadPlugin write_graphite
<Plugin write_graphite>
<Node "example">
Host "localhost"
Port "2003"
Prefix "collectd"
</Node>
</Plugin>
You should begin seeing metrics under the collectd metrics namespace for each server running Collectd. Some of the more common metric classes include CPU, disk activity and usage, network traffic, system load, memory usage, and more.
Carbon supports additional transport mechanisms besides plaintext, such as AMQP and Python's Pickle format. If you haven't already, I encourage you to read up on these options in the official Graphite documentation.