A Blog

An occasional data tap into Peter Burkholder’s brain

That Thing to Add Packaged Gems Into Your GEM PATH

| Comments

It’s Gem::Specification.reset For example, in a Chef cookbook I’m installing the conjur package from Conjur, and need those Gems available to me during Chef’s compilation. So:

# Install from a downloaded .deb (or .rpm)
dpkg_package "conjur" do
  source target_path

# Append those embedded gems in my path:
Gem.path << "/opt/conjur/embedded/lib/ruby/gems/2.1.0"

# And reload the Gem specifications that are in your .path

It may also be sufficient to do Gem.clear_paths, but that also sets your Gem.paths to nil so that doesn’t seem right. Haven’t tested.

Maybe I should make a PR to alias that to Gem.reload_paths

Emulating Cookbook Semver Build Numbers With Chef Policyfiles

| Comments

This is a posting of [https://github.com/pburkholder/buildnumber/blob/master/README.md]

policyfiles and builds

Demonstrate use of policyfiles to emulate build number increments.


We assume some build process that bundles a Chef cookbook as the generated artifact, and then includes that cookbook in the run_list of nodes in the same env:

run_list: 'recipe[base@0.1.0], recipe[myapp@0.1.0]'

However, the myapp recipe configure index.html via a template to render the build number as well:

<h1>Welcome to myapp</h1>
 <li>Version: 0.1.0</li>
 <li>Build: 1001</li>

The ‘base’ recipe is unchanged in each build, but the ‘myapp’ recipe has an data_file that is incremented with each build.

Builds are generated and tested on ephemeral nodes. The build are pushed as the policy for the role ‘myapp’, and are labelled with policy_groups that correspond to each build, e.g.:

  • myapp-0.1.0-0
  • myapp-0.1.0-1

When build myapp-0.1.0-1 passes, then it’s promoted to the ‘prod’ policy_group and then the ‘prod’ node converges with that new policy based on the bundled cookbooks.

Chef Server setup

No recipe for this yet. DNS is chefserver.cheffian.com. To do:

  • install chef-server 12.0.7 per https://www.chef.io/blog/2015/03/27/chef-server-12-0-7-released/
  • /etc/opscode/chef-server.rb (you need to run chef-server-ctl reconfigure to make it take effect):

    lb["xdl_defaults"]["policies"] = true
    api_fqdn 'chefserver.cheffian.com'
  • create a user and an organization with that user associated (-a): sudo chef-server-ctl user-create pdb Peter Burkholder pburkholder@chef.io TestPassword -f pdb.pem sudo chef-server-ctl org-create pdb_org pdb_org -f pdb_org.pem -a pdb

  • make current_dir/.chef/knife.rb and copy the above .pem file into the .chef directory: cd .chef scp ubuntu@chefserver.cheffian.com:pdb.pem . scp ubuntu@chefserver.cheffian.com:pdb_org.pem .
  • fetch the ssl cert: knife ssl fetch https://chefserver.cheffian.com
  • test with knife user list:

Cookbooks and Clients:

First lets get the simplest case down

Commit pdb/policyfile 0211fbbis where I have base cookbook installing httpd, and the myapp cookbook installing the index.html with build number from libraries/build.json. The kitchen verify should pass.

Onwards and local setup

Install 0.5.0 of ChefDK:

curl "http://www.chef.io/chef/metadata-chefdk?p=mac_os_x&pv=10.10&m=x86_64&prerelease=true"

Update knife.rb with add’l configuration parameters:

use_policyfile true
policy_document_native_api true

#policy_name 'jenkins'
#policy_group 'dev'

Now I create cookbooks/myapp/Policyfile.rb and run chef install -D:

chef install -D
Building policy myapp
Expanded run list: recipe[base], recipe[myapp]
Caching Cookbooks...
Installing base  >= 0.0.0 from path
Installing myapp >= 0.0.0 from path
Installing apt   2.7.0
Installing httpd 0.2.11

and push that as ‘myapp-0.1.0-0’ using the ‘policygroup’ feature

chef push myapp-0.1.0-0
Uploading policy to policy group myapp-0.1.0-0
WARN: Using native policy API preview mode. You may be required to delete and
re-upload this data when upgrading to the final release version of the feature.
Uploaded base  0.1.0  (f5cdaad1)
Uploaded myapp 0.1.0  (81a87a95)
Uploaded apt   2.7.0  (16c57abb)
Uploaded httpd 0.2.11 (3c562c6a)

Try it on a node

I have set up in aws the nodes ‘p0.cheffian.com’, ‘p1….’, and ‘p2….’

Not setting any runlist the first time, then going back and editing client.rb to use the Policyfile endpoints and features.

knife bootstrap p0.cheffian.com -x ubuntu -r '' -N p0 --sudo

# Note that my pdb_org.pem didn't work so I rm the validation lines
# from knife.rb to use my personal creds

then on p0.cheffian.com set policy_name and policy_group:

log_location     STDOUT
chef_server_url  "https://chefserver.cheffian.com/organizations/pdb_org"
validation_client_name "chef-validator"
node_name "p0"
trusted_certs_dir "/etc/chef/trusted_certs"

use_policyfile true
policy_document_native_api true
policy_name  'myapp'
policy_group 'myapp-0.1.0-0'

and that works.

Now I tag build 0.1.0-0 and push that to our origin git repo.

Now build 1

on the workstation:

We change myapp/files/default/build.json to build 1, and:

rm Policyfile.lock.json

and re-run:

chef install -D

to update the Policyfile.lock.json. Then:

chef push myapp-0.1.0-1

git commit -am "0.1.0-1"
git tag -m 0.1.0-1 0.1.0-1
git push origin master
git push origin 0.1.0-1

OR: just run rake bump build tag

Now lets set up the target node:

bootstrap the node:

knife bootstrap p1.cheffian.com -x ubuntu -r '' -N p1 --sudo


ssh p1.cheffian.com
sudo bash
cd /etc/chef

cat >> client.rb
use_policyfile true
policy_document_native_api true
policy_name  'myapp'
policy_group 'myapp-0.1.0-1'

Lastly run the client and confirm

sudo chef-client
curl localhost

Lastly we promote this build of myapp to prod

Here we go:

git checkout 0.1.0-1
chef push prod
chef push prod
  Uploading policy to policy group prod
  WARN: Using native policy API preview mode. You may be required to delete and
  re-upload this data when upgrading to the final release version of the feature.
  Using    base  0.1.0  (fc79b25d)
  Using    apt   2.7.0  (16c57abb)
  Using    httpd 0.2.11 (3c562c6a)
  Uploaded myapp 0.1.0  (90a39ad0)

Check with knife raw the policy for ‘myapp’ in the ‘prod’ policy_group

knife raw /policy_groups/prod/policies/myapp | egrep -B2 \"identifier
    "base": {
      "version": "0.1.0",
      "identifier": "fc79b25dc1ac842bdf342a65a2dda0d83d929c12",
    "myapp": {
      "version": "0.1.0",
      "identifier": "90a39ad0ad73ff3b245e1c5ede2a60a437349a73",
    "apt": {
      "version": "2.7.0",
      "identifier": "16c57abbd056543f7d5a15dabbb03261024a9c5e",
    "httpd": {
      "version": "0.2.11",
      "identifier": "3c562c6ac6ac554b4a11a0ad4c522fab246bf8b3",

The prod node always has the ‘prod policy group’ as in the this ‘/etc/chef/client.rb’:

log_location     STDOUT
chef_server_url  "https://chefserver.cheffian.com/organizations/pdb_org"
validation_client_name "chef-validator"
node_name "prod"
trusted_certs_dir "/etc/chef/trusted_certs"

use_policyfile true
policy_document_native_api true
policy_name  'myapp'
policy_group 'prod'

And now that we’ve promoted to ‘prod’ policygroup the chef-client run produces this html:

<h1>Welcome to myapp</h1>
  <li>Version: 0.1.0</li>
  <li>Build: 1 </li>


I Was Never This Cool in Middle School

| Comments

Great drama at D&D this afternoon: The DM revealed that the players had been fighting for the wrong side. The fifth graders were beside themselves with how they were duped by the humans back in the village, who are all cultists. Turns out the guys with tails and fins are the ones trying to stop the Kraken from being resurrected.

Last spring the other DM had all the PCs as acting as undocumented immigrants, having to fight evil while dodging the immigration authorities.

Which makes my middle-school D&D experience even more lame, since there was definitely no greater social context coming into play.

Chef: Running Your ‘Security’ Resource Last Doesn’t Save You

| Comments

I’ve come across the suggestion that corporate requirements come last in the run_list. Why that won’t work if you don’t trust your other Chef devs, with /etc/motd as an example.

Evil developer early in run_list:

file “bwahahahaha” do action :nothing path “/etc/motd” content “All your Appliance are owned by us\n” subscribes :create, “file[/etc/motd]” end

Security team puts MOTD last in run_list:

file “/etc/motd” do content “Property of Awesome Appliance, Inc.\n” end


Recipe: (chef-apply cookbook)::(chef-apply recipe) * file[bwahahahaha] action nothing (skipped due to action :nothing) * file[/etc/motd] action create - create new file /etc/motd - update content in file /etc/motd from none to cd5731 — /etc/motd 2014-11-04 10:19:15.000000000 -0500 +++ /tmp/.motd20141104-86735-2lu4fs 2014-11-04 10:19:15.000000000 -0500 @@ -1 +1,2 @@ +Property of Awesome Appliance, Inc. * file[bwahahahaha] action create - update content in file /etc/motd from cd5731 to 6b34be — /etc/motd 2014-11-04 10:19:15.000000000 -0500 +++ /tmp/.bwahahahaha20141104-86735-qq9idq 2014-11-04 10:19:15.000000000 -0500 @@ -1,2 +1,2 @@ -Property of Awesome Appliance, Inc. +All your Appliance are owned by us

What Happened to Cloud Particle? Did You Present at DevOpsDays? Is It Open Source? I’m Really Interested, as I’m on the Path of Using CloudFormation, but Looking for Something Better.

| Comments

Thanks for asking. The crux of the issue is that I’ve moved on from that employer and no longer have any say in when CloudParticle is open-sourced. I’ve pinged some of my team members, but one is at Burning Man and the other may have mostly dropped of Twitter.

I do hope they open-source CloudParticle soon. It’s definitely superior to CloudFormation, and the syntax is much cleaner than Terraform. The only feature lacking, really, are elastic IPs, because every node resource is realized with auto-scale groups, and EIPs don’t map well to that. (and you want even single nodes to be a 1-node ASG so they get re-instantiated upon failure).

Most likely you can’t wait, so Terraform is definitely worth looking at. Good luck, Peter

Clearing the Counter Pt II: Knife Cleanup Tweaks #chef

| Comments

Part Two: knife cleanup versions

A colleague at $WORK discovered a plugin by Marius Ducea which would remove unused cookbook versions from a chef server. The plugin provided you the option of dumping all cookbooks locally before deleting them. The problem we saw was that ‘unused’ meant ‘’not explicitly pinned’ with equality; it disregarded ~> and >=, etc. So we could not use the plugin without trashing many of our cookbooks in use.

I extended the plugin to take a –runlist parameter, so it would query the chef server for the cookbook versions needed to satisfy that runlist, and do so for each of the environments present on the server. At the time we were not using Chef’s built-in roles, as some of us had mis-read/mis-understood work such as The Berkshelf Way or A Year with Chef. We had one cookbook work-roles, which in turn had recipes such as workroles::mongo or workroles::api. To clean up cookbooks, I’d run:

knife cleanup versions –runlist ‘’workroles::default’’

and it would list all the cookbooks eligible for deletion. Then I’d run that again with the -D option to actually do the deletion. A few score cookbooks would poof go away and our depsolver issues would be gone.

I have a PR open with Marius but I’ve not heard back since Christmas.

Meanwhile, you can use it from my GitHub runlist_feature branch.

What about roles?

Turns out that one master roles cookbook is an anti-pattern. That’s a story for another time. Chef roles are great if they’re kept lightweight; they were unfairly maligned by abuses heaped upon them.

My branch of knife-cleanup doesn’t handle roles well, since it expects a runlist of recipes, so I’ll need to address that.

Unless cookbook clean up has been built into Chef via some other avenue, I’ll fork the current knife-cleanup plugin into a knife-scrub plugin which will build up a runlist for all roles, and keep the versions in use in any environment. Or if Marius has time then we can work on this issue together.

I have a short Part Three forthcoming with some other musings around depsolver and cookbook cleanup. Stay tuned.


Marius Ducea’s blog post on knife cleanup Useful, to me, Chef api reference One of the posts lashing out at Chef roles Jira ticket Role Cookbooks are a Chef AntiPattern

Clearing the Counter: Cookbook Clutter and Knife Cleanup

| Comments

Part One

In a Chef development group with a high rate of cookbook churn, you may eventually find that your Chef server is timing out as the dependency solver (depsolver) works out the correct cookbook version to send to clients based on environment constraints and cookbook constraints. This gets ugly pretty quickly, since the Chef workers tied up doing depsolving aren’t available for servicing other clients. At $WORK, we’d see lots of failed chef client runs, and usually at the most inconvenient of times.

How did things go wrong? Well, we’d have a number of internal cookbooks with metadata.rb include constraints such as:

depends ‘’java’‘, ’‘< 1.14.0’‘ depends ’‘apt’‘, ’‘>= 1.8.2’‘ depends ’‘yum’‘, ’‘>= 3.0’‘ depends ’‘python’‘ depends ’‘runit’‘, ’‘>= 1.5.0’‘ depends ’‘bar’‘, ’‘~> 1.1’‘ depends ’‘baz’‘, ’‘~> 2.0’’

And when bar has versions 1.1.0, 1.1.1, 1.1.2, and baz has all its versions, and the upstream cookbooks have all their version iterations, all with their own constraints, the depsolver problem space grows exponentially. Eventually, the chef server will kill the long-running depsolvers, where long-running means about five seconds.

Buying yourself time with longer depsolver timeout.

In a pinch, you can throw more resources at the problem, increasing the timeouts and the threads available to the chef server. This is only a short-term stop-gap. As you update more cookbooks, you’ll soon be back to your earlier pain.

Good luck finding where to change those settings, as the omnibus-chef-server project (https://github.com/opscode/omnibus-chef-server) has an attribute defined for default[‘’chef_server’‘][’‘erchef’‘][’‘depsolver_timeout’‘], but that attibute isn’t used anywhere *.

What you need to do is edit /var/opt/chef-server/erchef/etc/app.config to change the depsolver_timeout (under the chef_objects key) and the max_count under the pooler key, as show in this diff where the timeout goes to 10,000 ms, and the worker count is bumped to 12:

— app.config 2014-07-23 20:57:06.714838003 +0000 +++ app.config~ 2014-07-23 21:24:09.674838002 +0000 @@ -114,7 +114,8 @@ {s3_external_url, host_header}, {s3_url_ttl, 900}, {s3_parallel_ops_timeout, 5000}, - {s3_parallel_ops_fanout, 20} + {s3_parallel_ops_fanout, 20}, + {depsolver_timeout, 10000} ]}, {stats_hero, [ {udp_socket_pool_size, 1 }, @@ -148,7 +149,7 @@ {init_count, 20}, {start_mfa, {sqerl_client, start_link, []}} ], [{name, chef_depsolver}, - {max_count, 5}, + {max_count, 12}, {init_count, 5}, {start_mfa, {chef_depsolver_worker, start_link, []}}] ]},

Then restart the chef-server.

In the next part, I’ll cover how to fix this with a safe cookbook clean-up.

#DNDwKids: Back to HeroLab After DndInsider #dnd

| Comments

Last spring and fall I ran a Pathfinder game with 4th, 5th, and 6th graders at my sons’ school. Each term I had to turn away kids ‘cause I had to cap participation at six players, and even that was too many for the attention span of some of the kids.

Now that I’ve gotten two kids (6th grade and 7th grade) interested in DM’‘ing, I’m going have them run two games while I provide support, feedback and guidance.

But we’re switching to DnD 4e since Jesse and Harry each have stacks of 4e books and know the rules much better than for Pathfinder.

At our first session we just worked on converting our existing characters, and rolling up new ones for those who have just joined. I know there are a lot errors but that I would resolve them all by moving from paper to …, well from paper to DnD Insider Character Builder (CB) is what I thought but, jeez, what a beast that is.

I’d used HeroLab last fall and thought it pretty good, but got the sense that the WOTC online tools would be better suited for 4e. So I thought I would be ready to roll once I signed up for 3 months. But the tools require Silverlight, burn through my CPU, and really, really laggy, turning a data entry task into what seems like 15s of waiting between advancing between fields. Creating just one character took about an hour.

Also, it’s clear we’ll be using 4e for at least of couple of years, and there’s rumors that 4e support will be coming to an end.

So, I’m going to give 4e a whirl on HeroLab. The starting process is a bit arcane. First, since I’d already had a fully licensed HeroLab with Pathfinder, I didn’t seem to be able to use the 4e support in demo mode (I may be mistaken on that) unless I purchased the 4e license. Since LoneWolf offers a money-back option within 60 days, I thought I would lay down my $20 and hope for the best.

The 4e package for HeroLab has no content, since LoneWolf doesn’t have a license from WOTC, but instead has a downloader tool. Since I have a DnDInsider subscription it uses those credentials to d/l the data files.

I’m hoping once this completes I’ll have a faster, more usable character management system. Let’s see…

Chef Shell Attribute Example

| Comments

I find the dearth of chef-shell examples on the web really frustrating, so starting my own.

Getting started

root@logstash-i-ab57c6d1:~# export PATH=/opt/chef/embedded/bin:$PATH root@logstash-i-ab57c6d1:~# chef-shell -z

Querying node attributes

Get the logstash server outputs: “”

attributes_mode chef:attributes > node[‘logstash’] … chef:attributes > node[:logstash][:server][:outputs].length => 5 chef:attributes > node[:logstash][:server][:outputs][0] => {“elasticsearch_http”=>{“host”=>“logstash-elasticsearch.infra.example.in”}} “”