A Blog

An occasional data tap into Peter Burkholder’s brain

Fixing #sensuapp OpenSSL Peer Cert Validation Issues

| Comments

Today I used Chef to configure a test sensu-server, but my Hipchat notifications were failing with this snippet in the logs:

/opt/sensu/embedded/lib/ruby/2.0.0/net/http.rb:917:in `connect’‘: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)\n”

I soon determined that the httparty gem was at 0.11.0 on the prod sensu servers, and at 0.12.0 on the new one. Further, that httparty had (wisely) been changed to verify peer certs. No problem, but where to put the CA (Certificate Authority) bundle?

Tracking this down took more of the afternoon than ideal, but eventually I determined that the default SSL cert path can be determined with:

irb

irb(main):001:0> require ‘’openssl’‘ => true irb(main):002:0> File.dirname OpenSSL::Config::DEFAULT_CONFIG_FILE => “/opt/sensu/embedded/ssl”

To get the CA certs into embedded ruby we can update the default sensu install with a bit of Chefery

cookbook_file ‘’/opt/sensu/embedded/ssl/cert.pem’‘ do source “cert.pem” mode 0755 end

Where cert.pem contents are pulled from ‘http://curl.haxx.se/ca/cacert.pem%E2%80%99 so we have a complete list of acceptable Certificate Authorites.

Ideally, would submit a PR to https://github.com/sensu/sensu-build/pulls, but for now I’ll have to content myself with an issue report.

References:

http://www.rdoc.info/stdlib/openssl/OpenSSL/X509/Store:set_default_paths http://www.rubyinside.com/nethttp-cheat-sheet-2940.html https://github.com/emboss/ruby-openssl/blob/282912788da2247d10281988a2c35818ee14912f/ext/openssl/lib/openssl/ssl-internal.rb Update: - https://github.com/sensu/sensu-build/pull/79 has a PR to sensu Omnibus to fix this.

JMX - Collectd - Graphite

| Comments

I finally started sending some key JMX stats into Graphite via our collectd setup. A few notes since I’ll probably forget all about this until I next need to configure this.

JMX

JMX listens on a random port. Ended up adding all of the following to JAVA_OPTS

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=19876

What objects and attributes are available to monitor? Enable jmxproxy in tomcat with the following in /etc/tomcat7/tomcat-users.xml

and then peruse http://localhost:8080/manager/jmxproxy/

collectd

Configured the plugin with Miah’s chef-collectd cookbook. See my recipe and the template at:

https://gist.github.com/pburkholder/8341458

The main changes to the plugin configuration is a change to prefix for the thread_pools and the ‘Type’ for class loading.

carbon-writer

We use the carbon-writer plugin from Gregory Szorc. The plugin didn’t sanitize out double-quotes, which pretty much horked the Graphite browser. This pull request fixes that.

debugging collectd

The Ubuntu build doesn’t include debugging so turning up the log level to ‘’debug’ does nothing. And the ’‘info’ level gives you almost nothing. The most useful steps for tracking down my issues (which came down to the aforementioned double-quote) was a) running in the foreground:

/usr/sbin/collectd -f -C /etc/collectd/collectd.conf

and enabling the CSV plugin to see what was getting written before going to carbon/graphite.

LoadPlugin csv DataDir “/var/lib/collectd/csv” StoreRates false

It’s Been Real, Tumblr

| Comments

So I tried Tumblr as a blogging platform. Since Twitter has worked out so well, I thought Tumblr might have some appeal that wouldn’t be apparent until I dove in and tried it.

But I have a hard time taking myself seriously here, so I’m moving to Jekyll (at GitHub, but I can take it anywhere). The preview is at http://blog.pburkholder.com. I need to get a Disqus account set up and clean up the old posts. I hope it doesn’t take long, as I have some real content (sensu + chef, Puppet/Chef lessons) that deserve my real attention.

Update

Er, back on Tumblr again. Why, well, as cool as Jekyll is, I can’t quite justify the time to get it ‘just so’ when I can come here and just write.

Meanwhile, I can use http://import.jekyllrb.com/docs/tumblr/ to export/backup my content here, just in case.

Create Chef Client via Api With Validation Key

| Comments

So, suppose you have a Chef validator, how do you create a node client?

Like this:

ORG=“my_org” CHEF=“https://api.opscode.com/organizations/$ORG” CLIENT=userhack VALIDATION_KEY=“validation.pem” VALIDATION_USER=“$ORG-validator” # default open-source value is ‘’chef-validator’’

export PATH=$PATH:/opt/chef/bin

knife exec \ -E ‘’client_desc = { “name” => “’‘$CLIENT’‘”, “admin” => false}; n=api.post “/clients”, client_desc; puts n[“private_key”]’‘ \ -u $VALIDATION_USER -k $VALIDATION_KEY > client.pem

Now you can use the client.pem in a knife.rb:

cat << END > knife.rb log_level :info log_location STDOUT node_name “$CLIENT” client_key “client.pem” chef_server_url “$CHEF” END

No Title

| Comments

LifeOps sounds like just what the Doctor ordered to me. I’ve started with some goals 6 months ago, and had varying degrees of success in sticking with them. Sounding off some ideas, getting feedback and having some structure around it via regular meetings all sounds good to sustain the motivation. Anchoring with DevOpsDC meetups and augmenting with hangout fits nicely for me as well. So count me in!

Chef Pain Point 1: Multiple Repos

| Comments

At $WORK we’ve been migrating from Puppet to Chef. I was in the minority in voting to stay with Puppet, since we were already ½ way through refactoring our initial Puppet implementation. I have nothing against Chef as such, but there are some pain points that others considering a Puppet to Chef migration should consider. I need to write a full analysis of this migration, but with time short, I’ll start by just sharing some pain points which I’ll later pull into that magnum opus. Multiple repos Our Puppet code was in one repository. Our Chef code is in 24 repos so we can use Berkshelf that’s tied to versions and branches. When a Puppet module failed due to some odd dependency, I could ‘ack’ through the repo to find the a clue for what I was missing. Try that with 15 repos. Ugh.

LifeOps

| Comments

LifeHacks + DevOps + MeetUp: LifeOps When I worked at NCAR, in Boulder, CO, I fell into a group of three other professionals who met every three weeks to keep each other on track on our big-picture life goals. Although this was before I used Scrum, it was not unlike scrum for life: What have I done, what am I doing next, what are my roadblocks, and how does this fit into the big picture? I’m now looking for something along the same lines, since I found that that structure helped me see the progress I was making on what mattered to me, and to stay on track when roadblocks came up, as they inevitably did. Our NCAR’‘Goals Group’ comprised an atmospheric scientist, a science writer, a grants administrator, and a system administrator (me). Since none of us worked together or had friends in common, there was no baggage brought to the table. We could just focus on getting things done. What I’d like to see a similar group cover: Big picture professional goals: such as which free-lance writing project to pursue, certification to obtain, or degree program to research. Financial goals: Finding a new tax accountant, setting a new home budget target, tackling a home renovation project Personal goals: If you want to have more music in your life, are you going to join a choir or start taking guitar lessons? The goals are yours to choose, or to change your mind about. The role of the group is to challenge you to keep on track with them, and to help you step back and see the forest for the trees on a regular basis. Listening is the most important thing, but feedback should come in as well: “You finished the Mongo DBA 102 course; excellent!” “Maybe you’re not going to find the right degree course and you’ll just need to train yourself.” “GTD didn’t work for me; if you want to try out personal Kanban. I can give you some resources” “The last three meetings you said you were going to update your will. How about next time you just come back with a list of three lawyers to call?” “Jumping from open-source project to open-source project is just alienating people. Maybe you should back off for a bit and commit to just one in a month or two.” (maybe that one is a little harsh) What LifeOps is not: Group therapy: I’m all for you tackling your emotional issues, so set “Find group therapy” as a concrete goal with an actionable first step, and get back to us next meeting on how it went Bitch and moan: None of that either. Things are tough all over, so tough get going. If you need to whine, take it to your neighborhood bar. Monitorama was the impetus for me taking this goals group thing on again. I’d like to work on the documentation for Sensu, but I don’t want to start in unless I can assure myself I’ll stay with it for at least six months; that’ll it’ll stay a top priority amidst the other priorities posed by work, family, home, and various competing facets of modern life. An external sounding board is just the thing I need, and I hope you’ll join me in seeking the same. Proposed format: Every two months in person before DevOpsDC; every three weeks as a Google Hangout (after/before #hangops? or time TBD by  Doodle poll. Members: 3 min, 5 max.  Comment or reach me @pburkholder