diff --git a/cookbooks/collectd-custom/README.md b/cookbooks/collectd-custom/README.md new file mode 100644 index 00000000..8805b5be --- /dev/null +++ b/cookbooks/collectd-custom/README.md @@ -0,0 +1,448 @@ +collectd-custom Cookbook +======================== + +This cookbook will install a secondary collectd process on your instances so that it does not interfere with Engine Yard's primary collectd process. This way you can install your own plugins, alerting components, etc without having to worry what Engine Yard has set up. + +So that it does not interfere at all, it will compile collectd from source rather than using `emerge` and install it to `/var/lib/collectd-custom` (or your own directory of choice). + +Requirements +------------ + +### Platforms + +* Engine Yard Gentoo 12.11 stack + +Other Gentoo platforms should work fine as well. + +TL;DR +----- + +If you don't want to read the rest of this README, you can just paste this in your `main::default` recipe and get most of the way there: + +```ruby +service "monit" do + supports [:start, :restart, :reload, :stop] + action :nothing +end + +include_recipe "collectd-custom::credis" +include_recipe "collectd-custom" +include_recipe "collectd-custom::simplercpu" +# include_recipe "collectd-custom::graphite" +# include_recipe "collectd-custom::librato" +include_recipe "collectd-custom::monit" +``` + +You'll now need to include either [`collectd-custom::librato`](#collectd-customlibrato) or [`collectd-custom::graphite`](#collectd-customgraphite) as your backend of choice. + +Attributes +---------- + +The following are the default attributes used when running chef. You can modify them in your own repo, or override them in your `main/recipes/default.rb` recipe as such: + +```ruby +node.override['collectd-custom']['service_name'] = 'my-collectd-service' +``` + +#### default['collectd-custom'] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KeyTypeDescriptionDefault
service_nameStringName of the custom collectd service.collectd-custom
versionStringVersion of collectd to install.5.4.0
source_urlStringURL from which to download the collectd source.http://fossies.org/linux/privat/collectd-5.4.0.tar.gz
checksumStringSHA checksum of the source archive for validation.a90fe6cc53b76b7bdd56dc57950d90787cb9c96e
dirStringAbsolute directory to install collectd to./var/lib/collectd-custom
pid_fileStringAbsolute path to the pid file to use./var/run/collectd-custom.pid
intervalIntegerInterval for use in collectd.conf.30
read_threadsIntegerNumber of read threads for use in collectd.conf.5
hostnameStringHostname for the collectd process for use in collectd.conf.nil
packagesArrayPackages to install with collectd. Usually just the dependencies.['dev-libs/libgcrypt','sys-devel/libtool']
userStringUser to run collectd as. NOTE: This user will not be setup as part of the recipe, you must create it elsewhere.root
+ +#### default['collectd-custom']['credis'] + +If you want to install the redis plugin, you must include the `credis` recipe. These attributes are used for that. + + + + + + + + + + + + + + + + + + + + + + + + + + +
KeyTypeDescriptionDefault
versionStringVersion of credis to install.0.2.3
source_urlStringURL from which to download the credis source.http://credis.googlecode.com/files/credis-0.2.3.tar.gz
checksumStringSHA checksum of the credis archvice for validation.052ad7ebedf86ef3825a3863cf802baf289a624b
+ +#### default['collectd-custom']['graphite'] + +If you want to have collectd report to graphite, you must include the `graphite` recipe. These attributes are used for that. + + + + + + + + + + + + + + + + + + + + + + + + + + +
KeyTypeDescriptionDefault
hostStringHostname or IP address of your graphite installation.nil
portIntegerPort that graphite will listen on.2003
extra_configHashExtra configuration values to be passed to graphite's config.See attributes/default.rb
+ +#### default['collectd-custom']['librato'] + +If you want to have collectd report to librato, you must include the `librato` recipe. These attributes are used for that. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KeyTypeDescriptionDefault
src_dirStringAbsolute directory to install `collectd-librato` to./var/lib/collectd-librato
repoStringGit repository address for `collectd-librato`.https://github.com/librato/collectd-librato.git
versionStringGit version tag used when checking out the repo.0.0.10
emailStringYour librato account's email.nil
api_tokenStringYour librato account's API token.nil
extra_configHashExtra configuration values to be passed to collectd-librato's config. See README for all options.See attributes/default.rb
+ +#### default['collectd-custom']['plugins'] + +You can specify all the plugins you want and their configuration values directly from attributes if you please. A set number of defaults are installed automatically, but you can remove them as you see fit. + +```ruby + 'logfile' => { 'config' => { + 'LogLevel' => 'info', + 'File' => '/var/log/collectd-custom.log', + 'Timestamp' => true + } }, + 'cpu' => {}, + 'load' => {}, + 'swap' => {}, + 'memory' => {}, + 'disk' => { 'config' => { + 'Disk' => '/^xv/', + 'IgnoreSelected' => false + } }, + 'interface' => { 'config' => { + 'Interface' => 'eth0', + 'IgnoreSelected' => false + } } +``` + +Alternatively, you can install them via the [`collectd_plugin`](#definitions) or [`collectd_python_plugin`](#definitions) definitions. + +Recipes +------- + +#### collectd-custom::default + +This recipe will install all dependencies (except for credis), compile collectd from source, create configuration files, install plugins, and setup collectd as a service. + +To use, just add it to your node's `run_list`, or in the case of Engine Yard, reference it from another cookbook: + +```ruby +include_recipe "collectd-custom" +``` + +#### collectd-custom::plugins + +This recipe is automatically included via `collectd-custom::default` so you don't need to specify it yourself. + +This recipe will install any plugins configured via the node attributes and delete any plugins that were automatically generated and no longer in use. + +#### collectd-custom::librato + +This recipe is optional. + +This recipe will install collectd-librato and configure your custom collectd process to report to it. You must provide `node['collectd-custom']['librato']['email']` and `node['collectd-custom']['librato']['api_token']` for this to work properly. + +To use, just add it to your node's `run_list`, or in the case of Engine Yard, reference it from another cookbook after the `default` recipe: + +```ruby +include_recipe "collectd-custom" +include_recipe "collectd-custom::librato" +``` + +#### collectd-custom::graphite + +This recipe is optional. + +This recipe will enable the collectd graphite write plugin for reporting its metrics. You must provide `node['collectd-custom']['graphite']['host']` for this to work properly. Even if you don't host Graphite yourself, it may be useful to enable this while testing on EY Local or Vagrant. + +To use, just add it to your node's `run_list`, or in the case of Engine Yard, reference it from another cookbook after the `default` recipe: + +```ruby +include_recipe "collectd-custom" +include_recipe "collectd-custom::graphite" +``` + +#### collectd-custom::monit + +This recipe is optional. + +This recipe will setup monit to track your custom collectd process. Otherwise, it will not restart if it crashes or gets killed. You can also set it up in `inittab` if you prefer, but that recipe is not provided. + +To use, just add it to your node's `run_list`, or in the case of Engine Yard, reference it from another cookbook after the `default` recipe: + +```ruby +include_recipe "collectd-custom" +include_recipe "collectd-custom::monit" +``` + +You may also need to define your monit service in another of your cookbooks if you haven't already. This is for the proper reload notifications to work. To do that, include the following in another of your recipes, maybe `monit::service`: + +```ruby +service "monit" do + supports [:start, :restart, :reload, :stop] + action :nothing +end +``` + +#### collectd-custom::credis + +This recipe is required if you intend to use the redis plugin. + +This recipe will download and compile [credis](https://code.google.com/p/credis/) for use when compiling collectd. Failure to include this recipe will result in no redis plugin being compiled. + +To use, just add it to your node's `run_list`, or in the case of Engine Yard, reference it from another cookbook. It **must** come before the `default` recipe: + +```ruby +include_recipe "collectd-custom::credis" +include_recipe "collectd-custom" +``` + +If you previously had collectd compiled before including the redis plugin, you will need to recompile. That recipe is provided. + +#### collectd-custom::recompile + +This recipe is optional and advisable to use only after making drastric changes. + +This recipe will remove collectd and recompile it from source. Useful if you added plugin depencides after collectd has already been compiled but needs recompiling to get those plugins enabled. + +**NOTE:** This recipe should not be necessary for version upgrades. + +To use, just add it to your node's `run_list`, or in the case of Engine Yard, reference it from another cookbook replacing the `default` recipe: + +```ruby +include_recipe "collectd-custom::recompile" +``` + +#### collectd-custom::simplercpu + +This recipe is optional but recommended. + +This recipe will enable an aggregate plugin to aggregate CPU measurements per [Librato's recommendations](https://github.com/librato/collectd-librato/blob/master/README.md#simpler-cpu-metrics). Even if you aren't using Librato, consoliding your CPU measurements may be helpful. + +To use, just add it to your node's `run_list`, or in the case of Engine Yard, reference it from another cookbook replacing the `default` recipe: + +```ruby +include_recipe "collectd-custom::simplercpu" +``` + +Definitions +----------- + +You can install collectd plugins from your own recipes too. + +```ruby +collectd_plugin "postgresql" do + cookbook "main" # default: collectd-custom + template "postgresql.plugin.conf.erb" # default: plugin.conf.erb + options({ + 'Database' => { + 'myapp' => { 'Host' => 'localhost', 'Port' => '5432', 'User' => 'postgres', 'Password' => node[:owner_pass] } + } + }) + only_if { node[:instance_role] == 'db_master' } +end +``` + +Or for python plugins: + +```ruby +collectd_python_plugin "collectd-librato" do + path "/usr/local/bin/some-plugin.py" + module "some-module" + options({ + 'ApiToken' => 'some-token' + }) +end +``` + +So it may be useful for you to end up with your own `collectd-plugins` cookbook that generates your plugin configurations. For example, in my `main::default` recipe, I would include the plugins I defined: + +```ruby +include_recipe "collectd-custom::librato" unless vagrant? +include_recipe "collectd-custom::graphite" if vagrant? + +include_recipe "collectd-plugins::df" +include_recipe "collectd-plugins::filecount" +include_recipe "collectd-plugins::memcached" if util?(:memcached) +include_recipe "collectd-plugins::nginx" if app_server? +include_recipe "collectd-plugins::postgresql" if db_server? +include_recipe "collectd-plugins::redis" if util?(:redis) +include_recipe "collectd-plugins::resque" if util?(:redis) +``` + +See also +-------- + +This cookbook has been refactored and adapted for the EY Gentoo platform, inspired by these fine recipes: + +* [hectcastro/chef-collectd](https://github.com/hectcastro/chef-collectd) +* [coderanger/chef-collectd](https://github.com/coderanger/chef-collectd) + +Check them out if you have needs on other platforms. + +License and Authors +------------------- +Author:: Stephen Craton (scraton@gmail.com) + +The MIT License (MIT) + +Copyright (c) 2014 Stephen Craton + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/cookbooks/collectd-custom/attributes/default.rb b/cookbooks/collectd-custom/attributes/default.rb new file mode 100644 index 00000000..123695f5 --- /dev/null +++ b/cookbooks/collectd-custom/attributes/default.rb @@ -0,0 +1,77 @@ +# +# Cookbook Name:: collectd-custom +# Attributes:: default +# + +# default attributes for all platforms +default['collectd-custom']['service_name'] = 'collectd-custom' +default['collectd-custom']['version'] = "5.4.1" +default['collectd-custom']['source_url'] = "http://fossies.org/linux/privat/collectd-#{node['collectd-custom']['version']}.tar.gz" +default['collectd-custom']['checksum'] = "a90fe6cc53b76b7bdd56dc57950d90787cb9c96e" +default['collectd-custom']['dir'] = "/var/lib/collectd-custom" +default['collectd-custom']['pid_file'] = "/var/run/collectd/custom.pid" +default['collectd-custom']['interval'] = 30 +default['collectd-custom']['read_threads'] = 5 +default['collectd-custom']['hostname'] = nil +default['collectd-custom']['packages'] = ['dev-libs/libgcrypt','sys-devel/libtool'] +default['collectd-custom']['user'] = 'root' + +# redis plugin requires credis, so provide those details +default['collectd-custom']['credis']['version'] = '0.2.3' +default['collectd-custom']['credis']['source_url'] = "http://credis.googlecode.com/files/credis-#{node['collectd-custom']['credis']['version']}.tar.gz" +default['collectd-custom']['credis']['checksum'] = "052ad7ebedf86ef3825a3863cf802baf289a624b" + +# graphite configuration if you want to report there +default['collectd-custom']['graphite']['host'] = nil +default['collectd-custom']['graphite']['port'] = 2003 +default['collectd-custom']['graphite']['extra_config'] = { + 'Protocol' => 'tcp', + 'Prefix' => 'collectd.' +} + +# librato configuration if you want to report there +default['collectd-custom']['librato']['src_dir'] = '/var/lib/collectd-librato' +default['collectd-custom']['librato']['repo'] = 'https://github.com/librato/collectd-librato.git' +default['collectd-custom']['librato']['version'] = '0.0.10' + +# provide your librato credentials: +default['collectd-custom']['librato']['email'] = nil +default['collectd-custom']['librato']['api_token'] = nil +default['collectd-custom']['librato']['extra_config'] = { + # Config for the librato collection plugin + # For full reference see https://github.com/librato/collectd-librato/blob/master/README.md + 'TypesDB' => "#{node['collectd-custom']['dir']}/share/collectd/types.db", + 'LowercaseMetricNames' => true +} + +# plugins to automatically include +default['collectd-custom']['plugins'] = { + 'logfile' => { 'config' => { + 'LogLevel' => 'info', + 'File' => '/var/log/collectd-custom.log', + 'Timestamp' => true + } }, + 'cpu' => {}, + 'load' => {}, + 'swap' => {}, + 'memory' => {}, + 'disk' => { 'config' => { + 'Disk' => '/^xv/', + 'IgnoreSelected' => false + } }, + 'interface' => { 'config' => { + 'Interface' => 'eth0', + 'IgnoreSelected' => false + } } + + # Maybe you want to log to syslog instead? + # 'syslog' => { 'config' => { + # 'LogLevel' => 'info' + # } }, + + # You can track all memcached instances if you want, or override + # on your memcached instance if you have one. + # 'memcached' => { 'config' => { + # 'Host' => '127.0.0.1', 'Port' => '11211' + # } }, +} diff --git a/cookbooks/collectd-custom/definitions/collectd_plugin.rb b/cookbooks/collectd-custom/definitions/collectd_plugin.rb new file mode 100644 index 00000000..77e90c14 --- /dev/null +++ b/cookbooks/collectd-custom/definitions/collectd_plugin.rb @@ -0,0 +1,39 @@ +# +# Cookbook Name:: collectd-custom +# Definitions:: collectd_plugin +# + +define :collectd_plugin, :options => {}, :template => nil, :cookbook => nil do + template "/data/collectd.d/plugin-#{params[:name]}.conf" do + owner node['collectd-custom']['user'] + group node['collectd-custom']['user'] + mode 0644 + if params[:template].nil? + source "plugin.conf.erb" + cookbook params[:cookbook] || 'collectd-custom' + else + source params[:template] + cookbook params[:cookbook] + end + variables({ + :name => params[:name], + :options => params[:options] + }) + notifies :restart, "service[#{node['collectd-custom']['service_name']}]" + end +end + +define :collectd_python_plugin, :options => {}, :module => nil, :path => nil do + begin + t = resources(:template => '/data/collectd.d/plugin-python.conf') + rescue ArgumentError,Chef::Exceptions::ResourceNotFound + collectd_plugin "python" do + options :paths => ["#{node['collectd-custom']['dir']}/lib/collectd"], :modules => {} + template 'python_plugin.conf.erb' + cookbook 'collectd-custom' + end + retry + end + t.variables[:options][:paths] << params[:path] unless params[:path].nil? + t.variables[:options][:modules][params[:module] || params[:name]] = params[:options] +end diff --git a/cookbooks/collectd-custom/libraries/default.rb b/cookbooks/collectd-custom/libraries/default.rb new file mode 100644 index 00000000..10bc86b8 --- /dev/null +++ b/cookbooks/collectd-custom/libraries/default.rb @@ -0,0 +1,29 @@ +# +# Cookbook Name:: collectd-custom +# Library:: default +# + +def collectd_plugin_option_value(option) + return option if option.instance_of?(Fixnum) || option == true || option == false + "\"#{option}\"" +end + +def collectd_plugin_conf(config, level=0) + conf = [] + config.to_hash.each_pair do |key, value| + if value.is_a? Array + value.each do |subvalue| + conf << "#{key} #{collectd_plugin_option_value(subvalue)}" + end + elsif value.is_a? Hash + value.each_pair do |name, suboptions| + conf << "<#{key} \"#{name}\">" + conf << "#{collectd_plugin_conf(suboptions, level+2)}" + conf << "" + end + else + conf << "#{key} #{collectd_plugin_option_value(value)}" + end + end + conf.join("\n" + (" "*level)) +end diff --git a/cookbooks/collectd-custom/metadata.rb b/cookbooks/collectd-custom/metadata.rb new file mode 100644 index 00000000..a9e1ff24 --- /dev/null +++ b/cookbooks/collectd-custom/metadata.rb @@ -0,0 +1,9 @@ +name 'collectd-custom' +maintainer 'Stephen Craton' +maintainer_email 'scraton@gmail.com' +license 'MIT License' +description 'Installs/Configures collectd-custom' +long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) +version '0.1.0' + +supports 'gentoo' diff --git a/cookbooks/collectd-custom/recipes/credis.rb b/cookbooks/collectd-custom/recipes/credis.rb new file mode 100644 index 00000000..d7be4e85 --- /dev/null +++ b/cookbooks/collectd-custom/recipes/credis.rb @@ -0,0 +1,30 @@ +# +# Cookbook Name:: collectd-custom +# Recipe:: credis +# +# Installs credis library for use by the redis collectd plugin +# + +attributes = node['collectd-custom']['credis'] + +# Download source +remote_file "#{Chef::Config[:file_cache_path]}/credis-#{attributes['version']}.tar.gz" do + source attributes['source_url'] + checksum attributes['checksum'] + action :create_if_missing +end + +# Compile +bash "install-credis" do + cwd Chef::Config[:file_cache_path] + code <<-EOH + [ -d credis-#{attributes['version']}/ ] || mkdir credis-#{attributes['version']}/ + tar -xzf credis-#{attributes['version']}.tar.gz -C credis-#{attributes['version']}/ --strip 1 + cd credis-#{attributes['version']} + make + cp -f libcredis.so /usr/lib + cp -f libcredis.a /usr/include + cp -f credis.h /usr/include + EOH + not_if "[ -f /usr/lib/libcredis.so ]" +end diff --git a/cookbooks/collectd-custom/recipes/default.rb b/cookbooks/collectd-custom/recipes/default.rb new file mode 100644 index 00000000..630d6515 --- /dev/null +++ b/cookbooks/collectd-custom/recipes/default.rb @@ -0,0 +1,129 @@ +# +# Cookbook Name:: collectd-custom +# Recipe:: default +# + +attributes = node['collectd-custom'] +service = "service[#{attributes['service_name']}]" +config_opts = [] + +# Emerge required packages +attributes['packages'].each do |pkg| + package pkg do + action :install + end +end + +# Download source +remote_file "#{Chef::Config[:file_cache_path]}/collectd-#{attributes['version']}.tar.gz" do + source attributes['source_url'] + checksum attributes['checksum'] + action :create_if_missing +end + +# prefix +config_opts << "--prefix=#{attributes['dir']}" + +# platform specific +case node['platform_family'] +when 'gentoo' + config_opts << "--without-included-ltdl" + config_opts << "--with-libiptc=no" unless attributes['plugins'].include? 'iptables' +end + +# Compile +bash "install-collectd" do + cwd Chef::Config[:file_cache_path] + code <<-EOH + [ -d collectd-#{attributes['version']}/ ] || mkdir collectd-#{attributes['version']}/ + tar -xzf collectd-#{attributes['version']}.tar.gz -C collectd-#{attributes['version']}/ --strip 1 + cd collectd-#{attributes['version']} + export EPYTHON="python2.7" + ./configure #{config_opts.join(" ")} + make && make install + EOH + not_if "#{attributes['dir']}/sbin/collectd -h 2>&1 | grep #{attributes['version']}" +end + +# Create init.d +template "/etc/init.d/#{attributes['service_name']}" do + owner 'root' + group 'root' + mode 0766 + source 'collectd.init.erb' + variables({ + :dir => attributes['dir'], + :pid_file => attributes['pid_file'], + :pid_dir => File.dirname(attributes['pid_file']), + :user => attributes['user'] + }) + notifies :restart, "service[#{attributes['service_name']}]" +end + +# Create custom config directory under EY mount +directory "/data/collectd.d" do + owner attributes['user'] + group attributes['user'] + mode 0755 + action :create +end + +link "#{attributes['dir']}/etc/conf.d" do + to "/data/collectd.d" +end + +# Create the logfile if that's configured +if attributes['plugins'].keys.include?('logfile') + logfile_config = attributes['plugins']['logfile']['config'] + logfile_dir = File.dirname(logfile_config['File']) + + directory logfile_dir do + owner attributes['user'] + group attributes['user'] + mode 0755 + recursive true + not_if "[ -d #{logfile_dir} ]" + end + + file logfile_config['File'] do + owner attributes['user'] + group attributes['user'] + mode 0644 + action :create_if_missing + end + + template "/etc/logrotate.d/#{attributes['service_name']}" do + owner 'root' + group 'root' + mode 0644 + source 'logrotate.conf.erb' + variables({ + :log_file => logfile_config['File'], + :rotate => 7 + }) + end +end + +# Create the base configuration +template "#{attributes['dir']}/etc/collectd.conf" do + mode "0644" + source "collectd.conf.erb" + variables({ + :hostname => attributes['hostname'], + :pid_file => attributes['pid_file'], + :dir => attributes['dir'], + :interval => attributes['interval'], + :read_threads => attributes['read_threads'], + :plugins => attributes['plugins'] + }) + notifies :restart, "service[#{attributes['service_name']}]" +end + +# Configure plugins provided by DNA +include_recipe "collectd-custom::plugins" + +# Start collectd-custom +service attributes['service_name'] do + supports :status => true, :restart => true + action [ :enable, :start ] +end diff --git a/cookbooks/collectd-custom/recipes/graphite.rb b/cookbooks/collectd-custom/recipes/graphite.rb new file mode 100644 index 00000000..3d278b75 --- /dev/null +++ b/cookbooks/collectd-custom/recipes/graphite.rb @@ -0,0 +1,20 @@ +# +# Cookbook Name:: collectd-custom +# Recipe:: graphite +# +# Installs and configures the WriteGraphite plugin for the collectd-custom process +# + +graphite = node['collectd-custom']['graphite'] + +collectd_plugin "write_graphite" do + opts = { + 'Node' => { + 'carbon' => { 'Host' => graphite['host'], 'Port' => graphite['port'] } + } + } + opts['Node']['carbon'].merge!(graphite['extra_config']) if graphite['extra_config'] + options(opts) + + not_if { graphite['host'].nil? } +end diff --git a/cookbooks/collectd-custom/recipes/librato.rb b/cookbooks/collectd-custom/recipes/librato.rb new file mode 100644 index 00000000..3bc40ff5 --- /dev/null +++ b/cookbooks/collectd-custom/recipes/librato.rb @@ -0,0 +1,54 @@ +# +# Cookbook Name:: collectd-custom +# Recipe:: librato +# +# Installs the librato plugin for the collectd-custom process +# + +attributes = node['collectd-custom'] +librato = attributes['librato'] +plugin_path = "#{attributes['dir']}/lib/collectd/collectd-librato.py" + +directory librato['src_dir'] do + user attributes['user'] + group attributes['user'] + recursive true + action :create +end + +execute "install collectd-librato" do + user "root" + cwd librato['src_dir'] + command "git clone #{librato['repo']} ." + not_if { File.exists?("#{librato['src_dir']}/.git/")} +end + +execute "checkout specific collectd-librato version" do + cwd librato['src_dir'] + command "git checkout v#{librato['version']}" + not_if "git describe | grep v#{librato['version']}" +end + +execute "install collectd-librato plugin" do + cwd librato['src_dir'] + command "cp -f lib/collectd-librato.py #{plugin_path}" +end + +collectd_python_plugin "collectd-librato" do + path plugin_path + opts = { + 'APIToken' => librato['api_token'], + 'Email' => librato['email'] + } + + opts['Api'] = librato['api'] if librato['api'] + opts.merge!(librato['extra_config']) if librato['extra_config'] + + options(opts) +end + +# Delete legacy file, was moved to simplercpu recipe instead +file "/data/collectd.d/librato-aggregate-cpu.conf" do + action :delete + notifies :restart, "service[#{attributes['service_name']}]" +end diff --git a/cookbooks/collectd-custom/recipes/monit.rb b/cookbooks/collectd-custom/recipes/monit.rb new file mode 100644 index 00000000..42d8b893 --- /dev/null +++ b/cookbooks/collectd-custom/recipes/monit.rb @@ -0,0 +1,20 @@ +# +# Cookbook Name:: collectd-custom +# Recipe:: monit +# +# Installs a monitrc file to ensure collectd stays running (optional) +# + +attributes = node['collectd-custom'] + +template "/etc/monit.d/#{attributes['service_name']}.monitrc" do + owner 'root' + group 'root' + mode 0644 + source "collectd.monitrc.erb" + variables({ + :pid_file => attributes['pid_file'], + :service_name => attributes['service_name'] + }) + notifies :reload, "service[monit]" +end diff --git a/cookbooks/collectd-custom/recipes/plugins.rb b/cookbooks/collectd-custom/recipes/plugins.rb new file mode 100644 index 00000000..1a1e178a --- /dev/null +++ b/cookbooks/collectd-custom/recipes/plugins.rb @@ -0,0 +1,40 @@ +# +# Cookbook Name:: collectd-custom +# Recipe:: plugins +# +# Installs the base plugins defined in the node attributes and deletes the old ones +# + +plugins = node['collectd-custom']['plugins'] +plugins.each do |name, plugin| + next if %w(syslog logfile).include?(name) # don't generate conf files for loggers, we did that in default + + collectd_plugin name do + template plugin['template'] || nil + cookbook plugin['cookbook'] || nil + options plugin['config'] + end +end + +ruby_block "delete_old_plugins" do + block do + Dir['/data/collectd.d/*.conf'].each do |path| + autogen = false + File.open(path).each_line do |line| + if line.start_with?('#') and line.include?('autogenerated') + autogen = true + break + end + end + if autogen + begin + resources(:template => path) + rescue ArgumentError, Chef::Exceptions::ResourceNotFound + # If the file is autogenerated and has no template it has likely been removed from the run list + Chef::Log.info("Deleting old plugin config at #{path}") + File.unlink(path) + end + end + end + end +end diff --git a/cookbooks/collectd-custom/recipes/recompile.rb b/cookbooks/collectd-custom/recipes/recompile.rb new file mode 100644 index 00000000..2fba0f38 --- /dev/null +++ b/cookbooks/collectd-custom/recipes/recompile.rb @@ -0,0 +1,21 @@ +# +# Cookbook Name:: collectd-custom +# Recipe:: recompile +# +# Deletes collectd-custom and recompiles it. Useful if you added a package that's a plugin dependency. +# + +service node['collectd-custom']['service_name'] do + action [ :disable, :stop ] +end + +execute "remove_collectd_custom" do + command "rm -rf #{node['collectd-custom']['dir']}" + action :run +end + +include_recipe "collectd-custom" + +ruby_block "remove_recipe_collectd_recompile" do + block { node.run_list.remove("recipe[collectd-custom::recompile]") } +end diff --git a/cookbooks/collectd-custom/recipes/simplercpu.rb b/cookbooks/collectd-custom/recipes/simplercpu.rb new file mode 100644 index 00000000..89ce0b3e --- /dev/null +++ b/cookbooks/collectd-custom/recipes/simplercpu.rb @@ -0,0 +1,14 @@ +# +# Cookbook Name:: collectd-custom +# Recipe:: simplercpu +# +# Setup config to aggregate CPU +# + +template "/data/collectd.d/simpler-cpu.conf" do + owner node['collectd-custom']['user'] + group node['collectd-custom']['user'] + mode 0644 + source "simpler_cpu.conf.erb" + notifies :restart, "service[#{node['collectd-custom']['service_name']}]" +end diff --git a/cookbooks/collectd-custom/templates/default/collectd.conf.erb b/cookbooks/collectd-custom/templates/default/collectd.conf.erb new file mode 100644 index 00000000..7627f1ba --- /dev/null +++ b/cookbooks/collectd-custom/templates/default/collectd.conf.erb @@ -0,0 +1,53 @@ +# +# This file is generated by Chef +# Do not edit, changes will be overwritten +# + +############################################################################## +# Global # +#----------------------------------------------------------------------------# +# Global settings for the daemon. # +############################################################################## + +<% unless @hostname.nil? || @hostname == '' %> +Hostname "<%= @hostname %>" +FQDNLookup true +<% else %> +FQDNLookup false +<% end %> +BaseDir "<%= @dir %>" +PIDFile "<%= @pid_file %>" +PluginDir "<%= @dir %>/lib/collectd" +TypesDB "<%= @dir %>/share/collectd/types.db" +Interval <%= @interval %> +ReadThreads <%= @read_threads %> + +############################################################################## +# Logging # +#----------------------------------------------------------------------------# +# Plugins which provide logging functions should be loaded first, so log # +# messages generated when loading or configuring other plugins can be # +# accessed. # +############################################################################## + +<% if @plugins.keys.include?('syslog') %> +LoadPlugin syslog + + <%= collectd_plugin_conf(@plugins['syslog']['config']) %> + +<% end %> + +<% if @plugins.keys.include?('logfile') %> +LoadPlugin logfile + + <%= collectd_plugin_conf(@plugins['logfile']['config']) %> + +<% end %> + +############################################################################## +# Includes section # +#----------------------------------------------------------------------------# +# Load in extra config, usually for loading additional plugins. # +############################################################################## + +Include "<%= @dir %>/etc/conf.d/*.conf" diff --git a/cookbooks/collectd-custom/templates/default/collectd.init.erb b/cookbooks/collectd-custom/templates/default/collectd.init.erb new file mode 100644 index 00000000..0337ef27 --- /dev/null +++ b/cookbooks/collectd-custom/templates/default/collectd.init.erb @@ -0,0 +1,66 @@ +#!/sbin/runscript +# Copyright 1999-2012 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 +# $Header: /var/cvsroot/gentoo-x86/app-admin/collectd/files/collectd.initd,v 1.6 2012/02/07 20:12:03 dilfridge Exp $ + +: ${COLLECTD_PIDFILE:='<%= @pid_file %>'} +: ${COLLECTD_CFGFILE:='<%= @dir %>/etc/collectd.conf'} +: ${COLLECTD_NICELVL:='5'} +: ${COLLECTD_USER:='<%= @user %>'} +: ${COLLECTD_BIN:='<%= @dir %>/sbin/collectd'} + +extra_commands="configtest" + +depend() { + use net +} + +configtest() { + ebegin "Checking ${SVCNAME} configuration" + checkconfig + eend $? +} + +checkconfig() { + if [[ $(sed '/^$\|^#/d' ${COLLECTD_CFGFILE} | grep -c 'LoadPlugin\w\+oracle') != 0 ]] ; then + if [[ -e /etc/env.d/50oracle-instantclient-basic ]] ; then + source /etc/env.d/50oracle-instantclient-basic + export ORACLE_HOME + export TNS_ADMIN + else + ewarn "Unable to set Oracle environment, Oracle plugin wont work" + fi + fi + + ${COLLECTD_BIN} -t -C "${COLLECTD_CFGFILE}" 1>/dev/null 2>&1 + ret=$? + if [ $ret -ne 0 ]; then + eerror "${SVCNAME} has detected an error in your setup:" + ${COLLECTD_BIN} -t -C "${COLLECTD_CFGFILE}" + fi + + return $ret +} + +start() { + checkconfig || return 1 + [ -d <%= @pid_dir %> ] || mkdir <%= @pid_dir %> && chown <%= @user %>:<%= @user %> <%= @pid_dir %> + + ebegin "Starting collectd" + start-stop-daemon --start --user "${COLLECTD_USER}" \ + --nicelevel "${COLLECTD_NICELVL}" --exec "${COLLECTD_BIN}" -- \ + -P "${COLLECTD_PIDFILE}" -C "${COLLECTD_CFGFILE}" + eend $? "Failed to start collectd" +} + +stop() { + ebegin "Stopping collectd" + start-stop-daemon --stop \ + --pidfile "${COLLECTD_PIDFILE}" + ret=$? + if [ $ret -ne 0 ]; then + ewarn "Failed to stop collectd gracefully, forcing stop." + kill -9 `cat "${COLLECTD_PIDFILE}"` + fi + eend $? "Failed to stop collectd" +} diff --git a/cookbooks/collectd-custom/templates/default/collectd.monitrc.erb b/cookbooks/collectd-custom/templates/default/collectd.monitrc.erb new file mode 100644 index 00000000..68927bc3 --- /dev/null +++ b/cookbooks/collectd-custom/templates/default/collectd.monitrc.erb @@ -0,0 +1,5 @@ +check process <%= @service_name %> + with pidfile "<%= @pid_file %>" + start = "/etc/init.d/<%= @service_name %> restart" + stop = "/etc/init.d/<%= @service_name %> stop" + group <%= @service_name %> diff --git a/cookbooks/collectd-custom/templates/default/logrotate.conf.erb b/cookbooks/collectd-custom/templates/default/logrotate.conf.erb new file mode 100644 index 00000000..3f96b9f8 --- /dev/null +++ b/cookbooks/collectd-custom/templates/default/logrotate.conf.erb @@ -0,0 +1,8 @@ +<%= @log_file %> { + daily + dateext + rotate <%= @rotate %> + notifempty + missingok + copytruncate +} diff --git a/cookbooks/collectd-custom/templates/default/plugin.conf.erb b/cookbooks/collectd-custom/templates/default/plugin.conf.erb new file mode 100644 index 00000000..0cc8f905 --- /dev/null +++ b/cookbooks/collectd-custom/templates/default/plugin.conf.erb @@ -0,0 +1,10 @@ +# This file is autogenerated by Chef +# Do not edit, changes will be overwritten + +LoadPlugin "<%= @name %>" + +<% unless @options.nil? || @options.empty? %> +"> + <%= collectd_plugin_conf(@options, 2) %> + +<% end %> diff --git a/cookbooks/collectd-custom/templates/default/python_plugin.conf.erb b/cookbooks/collectd-custom/templates/default/python_plugin.conf.erb new file mode 100644 index 00000000..251757dc --- /dev/null +++ b/cookbooks/collectd-custom/templates/default/python_plugin.conf.erb @@ -0,0 +1,21 @@ +# This file is autogenerated by Chef +# Do not edit, changes will be overwritten + + + Globals true + + + + <% @options[:paths].each do |path| %> + ModulePath "<%= path %>" + <% end %> + <% @options[:modules].each_key do |mod| %> + Import "<%= mod %>" + <% end %> + + <% @options[:modules].each_pair do |mod, config| %> + "> + <%= collectd_plugin_conf(config, 4) %> + + <% end %> + diff --git a/cookbooks/collectd-custom/templates/default/simpler_cpu.conf.erb b/cookbooks/collectd-custom/templates/default/simpler_cpu.conf.erb new file mode 100644 index 00000000..cbaad2f7 --- /dev/null +++ b/cookbooks/collectd-custom/templates/default/simpler_cpu.conf.erb @@ -0,0 +1,31 @@ +# This file is autogenerated by Chef +# Do not edit, changes will be overwritten + +LoadPlugin aggregation +LoadPlugin match_regex + + + + Plugin "cpu" + Type "cpu" + + GroupBy "Host" + GroupBy "TypeInstance" + + CalculateSum true + CalculateAverage true + + + + + # Send "cpu" values to the aggregation plugin. + + Plugin "^cpu$" + + + Plugin "aggregation" + + Target stop + + Target "write" +