How to graph data with Munin
Using something like munin it’s trivial to create a graph plotting trends – and assists in answering questions like :
- Why is the website on server X feeling slow today?
- Are we receiving significantly more hits/email/whatever than normal?
- How effective is the new mail server?
- When are we likely to run out of disk space?
- Are people having problems logging into our new web application?
- We had reports that the server was slow yesterday, can you check?
- How well is X working?
We use an extensive number of munin graphs – covering everything from disk usage to a hits on a remote API. It is far easier to look at a graph and realise a queue is far larger than normal and so on.
Generally quite easy – on Debian it’s equivalent to “apt-get install munin-node munin” and then making sure Apache is setup to allow you to access http://server/munin. You can arrange one ‘munin’ install that receives data from multiple servers (running munin-node) if you so wish.
Munin will attempt to auto-configure itself to a large part – i.e. enabling plugins for the specific database/mta/whatever you’re using. If you’ve recently installed something extra, you can run :
munin-node-configure –suggest –shell
For the shell code you could use to enable more plugins.
‘Enabled’ plugins are visible as files in /etc/munin/plugins
Manual Plugin Installation
ln -s /usr/share/munin/plugins/whatever /etc/munin/plugins/whatever
(/usr/share/munin/plugins is where the packaged/bundled plugins are placed)
(Generally have a name ending in _)
Allow you to re-use the same plugin for a different graph. The plugin uses it’s own name (i.e. what it is in /etc/munin/plugins) to determine what to do
e.g. if_ which plots traffic over a network interface can be used to create multiple graphs – like :
ln -s /usr/share/munin/plugins/if_ /etc/munin/plugins/if_eth0
ln -s /usr/share/munin/plugins/if_ /etc/munin/plugins/if_eth1 etc.
Testing a Plugin
Plugins are run through/via ‘munin-node’ – and may not be running as ‘root’ so permissions can matter.
To help testing them, try using ‘munin-run’ which will ensure you’re running the plugin in a similar environment to the munin-node daemon. e.g. ‘munin-run mysql-queries‘
Configuration of plugins is handled by /etc/munin/plugin-conf.d/munin-node.conf (create additional .conf files in this dir if you want to keep things split up). The format is a simple ‘ini’ file style – within which you can specify a different user (non-root perhaps) to run a specific plugin as, along with any other necessary parameters.
A simple custom plugin
A plugin has two main requirements – one that when run with the ‘config’ argument it tells munin metadata – how to graph the data etc. Secondly, when run without arguments it needs to output values to be graphed.
As a simple example, if we create a fail2ban plugin which initially just plots the number of bans with time, we might have something that looks like :
#!/bin/bash if [ "$1" = "config" ]; then echo 'graph_title SSH Fail2ban' echo 'graph_args --base 1000 -l 0' echo 'graph_vlabel ban count' echo 'graph_scale no' echo 'graph_category system' echo 'graph_info Count of IPs blocked by fail2ban' echo 'bancount.label SSH Ban Count' echo 'bancount.type GAUGE' echo 'bancount.min 0' exit 0 fi # Print some data. echo -n 'bancount.value ' /sbin/iptables -nL fail2ban-ssh | grep -c DROP
And usage like :
root@server:/etc/munin/plugins# ./fail2ban config graph_title SSH Fail2ban graph_args --base 1000 -l 0 graph_vlabel ban count graph_scale no graph_category system graph_info Count of IPs blocked by fail2ban bancount.label SSH Ban Count bancount.type GAUGE bancount.min 0 root@server:/etc/munin/plugins# ./fail2ban bancount.value 2
This is a trivial graph, but they can easily be expanded – perhaps to allow you to dynamically plot different jail bans with time — resulting in something more like :
root@server:/etc/munin/plugins# ./fail2ban config graph_title Fail2ban stats graph_args --base 1000 -l 0 graph_vlabel ban count graph_scale no graph_category security graph_info Count of IPs blocked by fail2ban apache.label Apache Ban Count apache.type GAUGE apache.min 0 ssh.label SSH Ban Count ssh.type GAUGE ssh.min 0 root@server:/etc/munin/plugins# ./fail2ban apache.value 5 ssh.value 2
Note, Munin plugins don’t take kindly to having hyphens (‘-‘) in their label names.
And the resultant graph –
Finally, as a note – obviously the data you graph may not be an absolute number that is the current value to plot – perhaps it’s always incrementing (like total hits to a website) so you’d need to use DERIVE or COUNTER line types where munin effectively tracks the difference between consecutive readings.. Or perhaps the act of reading resets the counter (ABSOLUTE). See the fieldname.type documentation page for more information.
You can also control the colours of the lines, and how they’re drawn (area, line etc).