A Blog

An occasional data tap into Peter Burkholder’s brain

Reaching Out to My Network

| Comments

I wasn’t sure where to start for reaching out to my network on my job hunt, found some great advice from The Muse

Here’s the message that I’m getting out to my world:

As you may know, I’ve spent most of the last two years with Chef Software, working with major Chef customers on successful infrastructure automation rollouts and the adoption of DevOps tools and practices.

I am on the hunt for new opportunities in DevOps implementation and leadership, which could span infrastructure automation, continuous delivery, cloud migration, and compliance/security. I’m reaching out to you to ask for your help with any leads or contacts. As I was let go as part of a recent corporate re-organization, I’d like to secure a new position by mid-June.

If you know of any job opportunities or leads that you might be able to share with me, please send them my way. Below is a summary of my recent positions, sample job titles, some parameters that are guiding my job search, and links to more detailed background material.

Thanks in advance for your help! I hope you are doing well and hope to catch up with you personally soon.



Recent Experience:

Customer Engineer, Chef Software (2014-2016): Worked with enterprise Chef customers to ensure a successful automation strategy and implementation, and advocated for adoption of DevOps culture. Wrote code for core Chef bugfixes/features and cookbooks.

Cloud Engineer, Rally Health (2012-2014): Operated a HIPAA-compliant social network platform comprising hundreds of nodes in AWS EC2 with team of three other engineers. Supported software engineering team in feature development and roll-out.

Senior Web Engineer, AARP (2008-2012): Managed web, application, database and development machines to serve 3 million pages/day for AARP.org, the United State’s largest membership organization. Mentored team of four system administrators in Scrum-style approach to work management.

Sample Job Titles:

  • Senior DevOps Engineer
  • Director of DevOps
  • Manager Cloud Operations
  • Web Operations Manager
  • Senior DevOps Consultant
  • Cloud/DevOps Architect

Ideal Institutions or Companies:

  • Forward-looking public service, such as 18F or U.S. Digital Service
  • Scientific research support, such as NOAA or NASA
  • Established NGOs/Non-Profits, such as World Bank or NPR
  • Newer companies with established revenue stream
  • Established companies with a commitment to modern IT practices


Washington, DC; Arlington, VA; Maryland suburbs; or 100% remote


Reviving My SSL Man-in-the-Middle Paper

| Comments

Hi - Republishing this artifact from 2002, an overview of how SSL Man-in-the-Middle (MITM) attacks happen, and how to conduct one.


Used to be the top hit in Google when you searched for ‘SSL MITM’ or ‘SSL man-in-the-middle.’ Ah, yes, those short-lived days of relative Internet fame.

I’ve also used this as the impetus to retire from Tumblr and move entirely over to Octopress/GitHub Pages. So all my sundry entries since 2013 are now in one place.

Thanks to Yassine Chaouche ‏\@ychaouche for prompting me to get this back out in the world. And also to Parker Moore https://byparker.com/ for easing the import: http://import.jekyllrb.com/docs/tumblr/


A “Thank You” From My Sons’ School

| Comments

My son’s teacher, Lisa Woodward, dropped this into our school’s weekly newsletter.

Geophysicist Visits 5th Graders

Today 5th graders were in for a treat. Peter Burkholder, Linus’s Dad spent three Summers working as a Geophysicist in Antarctica some years ago. He gave a riveting presentation about the continent, answered questions, and shared stories about his work. A huge map of Antarctica became the central focus. His powerpoint presentation consisted of many questions. Students chose questions they wanted answered and we were led on a journey of discovery using artifacts and photographs. Students examined fossils, rocks, and a rare piece of calcium carbonate that Peter bought back from his expeditions.

Student questions were insightful and thoughtful. One such question led Peter on a story about using an outhouse while on a field study. His elaborate illustrations illustrated how he used boiling water and a drill to dig a deep hole and how one would build an outhouse on top of it. Did you know that bodily fluids freeze on contact with ice that is -40F? I’ll leave what happens when you use the outhouse over a six week period up to you! Students learned a variety of facts which will add to their research on their question that they are studying at present. Thank you Peter for enriching our learning about Antarctica. The students enjoyed your persentation very much.

The outhouse poopsicle story always goes over well with the pre-teen crowd.

Chef Custom Resources: Here’s an Easy Example

| Comments

Chef custom resources are easy

Purpose: Demonstrate a simple custom resource cookbook with tests.

When it comes to programming, I’m an inductive learner. To me, a working example is worth a 1000 lines of documentation. The Chef Custom Resource model introduced with the release of 12.4 offers a way to write reusable resources that’s far simpler than the Chef LWRP and HWRP models (although those are still fully supported). And there’s strong documentation that came with it, to wit:

However, if you just want to see trivially simple example from which you can grow, then the motd_cr cookbook is for you: https://github.com/pburkholder/motd_custom_resource.

It’s a library cookbook, meaning there’s no default recipe, and it adds a motd resource to Chef when you add depends "motd_cr" to your metadata.rb any external cookbook.

The relevant custom resource code is in resources/motd.rb, and it’s merely this:

resource_name :motd
property :message, kind_of: String, name_property: true

action :create do
  file '/etc/motd' do
    content "#{message}\n"
    mode '0644'

Yes, it’s that’s short.

The example cookbooks for testing are in tests/fixtures/cookbooks/test. Basically you use the resource like this:

motd 'Welcome to my message of the day'

And working tests are in tests/ and spec/. Running the kitchen and chefspec tests should convince you it works

I hope this helps you write your own custom resources complete with ChefSpec and Test-Kitchen tests

– Peter


Integration testing

kitchen test

(If you want to use this with the ultra-fast kitchen-dokken framework, do this first:)

export KITCHEN_LOCAL_YAML=.kitchen.dokken.yml
eval $(docker-machine env default)

Unit testing

ChefSpec and custom resource:

rspec spec/unit/recipes/default_spec.rb

Anticipated Questions:

What about load_current_value and converge_by?

These aren’t generally needed if your custom resource comprises a collection of native chef resources, since each of them will compare the current state to the desired state and take no action when none is needed.

When you are building new resources from Ruby code then you’ll need the above methods so your resource follows the test-and-repair approach (idempotency).

Why doesn’t --why-run work as expected?

It’s a bug: https://github.com/chef/chef/issues/4537

Doesn’t the resource need an action :destroy?

Sure - fork this and write it. It’s a learning exercise, not something I expect anyone to use.

What about poise-style testing?

Oh that – to minimize confusion I’ve moved it to the poise tag and poise-dev branch of this repo. Look there for more info.

Lambda to Defer Chef Code Evaluation

| Comments

Draft Note

TKTK: needs explanation

Don’t do this, for examplar purposes only:

cookbook_file '/etc/chef/encrypted_data_bag_secret' do
  source 'encrypted_data_bag_secret'

# create an 'aws' lambda to call during converge phase
aws = lambda do
    'encrypted', 'aws', IO.read('/etc/chef/encrypted_data_bag_secret')

template '/home/ubuntu/.s3cfg' do
  source 's3cfg.erb'
  owner 'root'
  group 'root'
  mode 00744
  variables lazy {
      aws_secret_key: aws.call['aws_secret_key'],
      aws_access_key: aws.call['aws_access_key']

6 Ways to Resolve Chef Cookbook Version Conflicts

| Comments

Six ways to resolve Chef cookbook version conflicts


From one of my customers:

Our appdev role now depends on the perforce cookbook from Supermarket. The perforce cookbook’s metadata specifies the dependency depends windows, '~> 1.38'

Problem is, we’ve pinned our five environments to windows 1.36.1, and we’re not in a position to upgrade to 1.38.0 just now.

We’re using a current version of chef-client and chef-server. How can we include Perforce in our appdev role without inflicting too much pain on ourselves?


There’s no perfect solution for this, as they all have some tradeoffs, so you’ll need to decide with path works for you.

This answer refers to the perforce and windows cookbooks per the original answer, but substitute in whatever cookbook constraints you are working under.


Before we address the various approaches, let’s review cookbook dependency management. First, cookbook dependencies for Chef 11/12 can be specified in any of four places:

  • The cookbook’s metadata.rb
  • The node’s environment, e.g. in an environment.json
  • The node’s role, e.g. in a role.json. Note: Many chef user’s regard this use of roles as an anti-pattern.
  • The node’s runlist, e.g. "run_list": [ "recipe[foo::bar]@0.1.0" ] Note: This is largely undocumented and unsupported. Do not use.

Unlike attributes, there is no precedence for cookbook versions. Instead the chef-server’s depsolver will find the newest cookbook version to satisfy all the constraints, or throw a 412 error. You may want to read https://getchef.zendesk.com/hc/en-us/articles/204381030-Troubleshoot-Cookbook-Dependency-Issues if you’re trying to identify what cookbook constraint is breaking your chef run.

For newer Chef 12 (server & client) installs, one can use Policyfiles instead of the above approach to specify cookbook dependencies.


In brief, here are the options you have:

  1. Use an older version of the offending cookbook
  2. Fork the offending cookbook and yank the stuff that requires a version newer than you want to use. (Or don’t use the perforce cookbook at all, write one that does only what you need)
  3. Bump to Windows 1.38.0 everywhere
  4. Move the cookbook constraints from environments to cookbooks
  5. Create a microenvironment where you could pin to Windows 1.38.0
  6. Use policyfiles for the nodes in question

In more detail:

Use an older version of the offending cookbook

If you really want to keep your cookbooks as they are on the Supermarket, this may be an approach that works for you. Or the cookbook author may be able to respond to an issue submission or a pull-request if you have fix that works with wider cookbook version constraints.

Fork the offending cookbook and yank the stuff that requires a version newer than you want to use. (Or don’t use the perforce cookbook at all, write one that does only what you need)

In this case you are maintaining your own code, but that may be more maintainable than using the Supermarket cookbook

Bump to Windows 1.38.0 everywhere

If you have a solid cookbook development and testing pipeline, this is arguably the best solution as you want to keep your code on ‘master’ as much as possible. However, if your pipeline is still in the works, or you work under change management constraints, you may not have this option.

Move the cookbook constraints from environments to cookbooks

However, if you have lots of cookbooks, or have a workflow that leverages berks apply or otherwise is tuned to dependency constraints in environments, then this option is a non-starter.

Create a microenvironment where you could pin to 1.38.0

I don’t know if microenvironment is a common term for this, but if your currently have, say, sandbox, dev, qa, preprod and prod environments, then you may be okay adding just an appdev environment. However, if your role is ubiquitous, then you might have to go with five new envs: sandbox_appdev, dev_appdev, etc.

Use policyfiles for the nodes in question

Chef Policyfile was developed with just this sort of use-case in mind. You could apply a Policy for the appdev nodes in your orgs while leaving all the other nodes in the Chef 11/12 constraint solution world. The Chef Server 12.2.0 and higher can support both types of nodes just fine, so you might want to use this as an opportunity to start working with Policyfile for your applications and nodes.

See also: https://docs.chef.io/policy.html and https://www.chef.io/blog/2015/08/18/policyfiles-a-guided-tour/ and https://github.com/chef/chef-dk/blob/master/POLICYFILE_README.md

Related Articles

Chef Docs on Cookbook Versions

Troubleshoot Cookbook Dependency Issues


In this case, the problem was solved by the perforce cookbook getting an update to relax the constraints on the versions of the windows cookbook, a variation of number 1, above.

See http://blog.pburkholder.com for New Content

| Comments

Tumble and Medium and Octopress - oh my!

I have new content, it’s just not here.

I’m trying Octopress, which publishes at http://blog.pburkholder.com

I’ve not moved this over because I may want to move to Medium.

Recent posts:






Why Chef-vault and Autoscaling Don’t Mix

| Comments

[Note: The opinions here are mine and do not reflect a position of my employer, Chef Inc.]

I received this question through our support team:

We’ve got a handful of cookbooks that rely on Chef Vault to pick up some secrets – passwords, SSL certs, etc. When we bootstrap new nodes with vault-dependent cookbooks in the run list, the chef-client run fails because the node doesn’t have access to the vault. There seems to be a ‘chicken and egg’ scenario in that we can’t seem add a new node’s host key to a vault during bootstrap – adding an existing node to a vault happens with a knife search, which requires a node to have successfully completed a chef-client run and be registered with the server, but our fresh nodes haven’t done this (and fail due to this…). I think you can see where I’m going with this. I’ve found some discussions around this on the web and have seen that knife bootstrap appears to have some switches for vault, so I tend to believe what we’re trying to do is possible – just haven’t had any success.

Our new nodes are launched on [a cloud] by an automated process …, but during the customization step of provisioning (cloudinit perhaps?) the bootstrap is kicked off on the node and the run list is converged.

Here’s my response:

ChefVault and knife bootstrap

I’m going to assume you are already familiar with the theory behind chef-vault, as you’ll need that background when implementing chef-vault for an organization. For the first part of your use-case, how to bootstrap a node when using chef-vault, you can use features of validatorless-bootstrap when the bootstrapping client (e.g. your workstation) has an administrative key on the chef-server.

One of my chef-vault demos has commands like this:

knife bootstrap ec2-52-2-56-144.compute-1.amazonaws.com \
    -N testvault-i-35a4119d   -r 'role[sensu_chefvault]' \
    --bootstrap-vault-json '{"sensu_vault":["rabbitmq"]}'

which generates the output like:

Creating new client for testvault-i-35a4119d
Creating new node for testvault-i-35a4119d
Connecting to ec2-52-2-56-144.compute-1.amazonaws.com

What’s going on here is that the -N testvault-i-35a4119d -r 'role[sensu_chefvault]' options create a new node and client on the chef-server, with runlist ‘role[sensu_chefvault]’, even if the box in question doesn’t exist yet. Further, the the --bootstrap-vault-json '{"sensu_vault":["rabbitmq"]}' specifies a vault and item to rekey before continuing with the bootstrap.

When the knife bootstrap continues, the newly bootstrapped node will get its private key from the workstation, save it as /etc/chef/client.pem, and can use that to unlock the vault items it needs. This works – it has the key, and the vault has been re-keyed/refreshed from the workstation to allow vault access to testvault-i-35a4119d.

This works great for provisioning/bootstrapping a few nodes from your workstation, but the second part of your question is where things get hairy. In an autoscaling setup with chef-vault you would need to delegate a node in your environment with a large part of the authority that is usually reserved to administrative workstations:

About chef-vault and autoscaling

Before we go on, bear in mind what my colleague Nathan Cerney says when talking about secrets and Chef:

Secrets management is about a balance between how much you care about your secrets and how much work you’re willing to do.

Chef-vault is certainly a step up from generic encrypted data_bags; it’s an elegant design and a great fit for relatively static sites. But I find that chef-vault is a poor fit to autoscaling situations for the following reasons:

  1. You would need some privileged node that has the “keys to the kingdom” to rekey all your vaults for the clients as they come up, which may put some security conscious folks on edge.
  2. It’s completely on you to write an auditing/authorization framework around that privileged node, as there’s nothing in chef-vault that provides that
  3. The provisioning process would have to be single-threaded or use some locking around the vault databags or your vaults will get out of sync, or possibly corrupted
  4. If you’re not willing to delegate rekeying of the vaults to a provisioning node, then you need a human admin to rekeying the vaults with the new search results, and then you have to re-run chef (defeating the whole purpose of autoscaling and automaed provisioning)
  5. You’re potentially subject to node impersonation attacks, which I describe below
  6. Chef-Vault assumes that the set of Chef-Server administrators includes the set of secret administrators, but this is neither true, nor generally desirable. It’s not directly related to autoscaling, but is a point to bear in mind when managing secrets for larger organizations - as noted in this Chef vault blog post

So, now that the bad news is out of the way, how do you manage secrets with Chef in an autoscaling situation?

Noah Kantrowitz has a good 2014 survey of Chef & Secrets landscape here: https://coderanger.net/chef-secrets/ but there have been some interesting developments since then. Three tools you should definitely evaluate are:

  • Hashicorp Vault - open-source project from Hashicorp
  • Conjur - commercial project from Conjur; distributed as an appliance, either an AWS instance, VM or Docker image you deploy internally
  • Conjur Summons - - summon is an opensource secrets bus that interoperates with other secret backends. I happen to be evaluating it this week, based on this sample cookbook: (https://github.com/conjurinc/summon-chefapi)

Conjur has some features that I find pretty compelling, one that is relevant here is the hostfactory feature. Any host in a Conjur environment has to have a key to access its secrets. You can authorize hosts yourself as an admin, or you can assign a hostfactory token to an application to delegate host token creation to it, which is what you’d want in an autoscaling setup. I have a janky demo here if you’re interested.

Hashicorp Vault has also generated a lot of interest in the last year. The server is open source, and has an h/a mode with Consul. The folks at Bloomberg have a cookbook for management: https://github.com/johnbellone/vault-cookbook

A few other players in the secrets space that Noah didn’t mention that I’m aware of: - KeyWhiz - AWS KMS - Sneaker (AWS KMS backend) - Thycotic - CyberArk

General guidance around secrets with Chef

You can think about secrets-management with Chef in a hierarchy:

  1. Encrypted data bags(EDBs) with secrets common to all nodes
  2. EDBs with secrets shared only among nodes with a common role or run_list
  3. Chef-Vault, or EDBs with the EDB decryption key provided from Chef-Vault
  4. You may want to take this approach so you can evolve your secrets management from decryption keys coming from chef-vault, to some other service providing the keys.
  5. As Nathan pointed out, this approach means you “have an encrypted data bag, protected by a [key] that’s stored in an encrypted data bag protected by a secret that’s stored in an encrypted data bag protected by the client’s key,” so it’s less than ideal
  6. Secret storage services: Conjur or Hashicorp Vault or Red October or the like
  7. Hardware platforms (HSMs) that are FIPs-compliant and require N approvals and a time-lock

Each step up makes your infrastructure more complicated, but protects your secrets better. But even Step 1 is immeasurably better than storing secrets in clear-text, and you shouldn’t let the perfect be the enemy of the good. Choose an implementation you can realize, then iterate to make it better.

What is Chef Inc doing to help?

This is some of what’s going on here: [Again, not official]

  1. Providing better documentation and guidance
  2. This response is a first draft of new knowledge base article, which will incorporate Noah’s blog post, with updates
  3. Taking ownership of Chef Vault. The project was originally a Nordstrom project, and they’ve bequeathed it to us we can continue engineering on it while understanding its limitations
  4. Building better examples of integrating Chef and EDBs with third-party secret providers – this work is already underway on my team, and questions like yours help spur on that effort.

Hope that helps…


P.S. What’s a node impersonation attack?

Suppose you have: - a provisioning node that rekeys chef-vault based on updated search results - an unprivileged role that doesn’t have any access to interesting secrets - a database role that does have access to interesting secrets

If an attacker gains root on an unprivileged node, then s/he can use the client’s key to edit the node’s role and change it to database. Further, s/he can then just kill chef-client runs on that node so it doesn’t actually become a database node.

Next time the chef-vault is rekeyed, the compromised node (and the attacker) will access to all of the database secrets. Ugh.

This same attack works if you have humans doing the rekeying, because, seriously, do you really vet all of the nodes that show up in the search?

This attack against chef-vault will not be possible when Chef RFC 45 is fully implemented, since you can then administratively lock a node’s runlist. Pending that implementation, one can also use Chef-Guard to filter such attacks, or Chef Analytics to alert on attempts after-the-fact.

Update 2015.12.11: Root compromise is not necessary for impersonation. Filching any client.pem or a validation.pem will suffice to update or create client/node objects that match the search criteria, and access vaults. Protect your keys.

Other references: https://blog.conjur.net/lets-talk-encrypted-data-bags

DevOpsDays DC Quick Takeaways

| Comments

DevOpsDays DC key points

  • John Willis thinks we have lots of reading time on our hands
  • Lean Enterprise was focus of one session talk: “I read this book and here is how I implemented it and it worked for us”
  • Josh Corman’s talk on Rugged DevOps was one of the more interesting relevant ones
    • ‘root access is a bug’
    • ‘give devs a tool before taking one away’
  • OpenSCAP talk: A lot of the same design goals as Chef audit-mode. Lots of support scans for platforms already. Haven’t looked at it yet. Julian may also have thoughts on it.
    • Fed space loves opensource since procurement process for COTS is so painful.
  • Ken Johnson and Mike Nescott’s talk on DevOops security also pretty interesting/relevant to the fed space.

Open space on Windows Automation - Lots of Powershell - Success stories around https://octopusdeploy.com/ Octopus for deployments being used to automate release - WINDOWS ADMINS WANT CHEF SERVER TO RUN ON WINDOWS. - Just like Linux/Unix admins shudder when they’re told they have to run a core service that is Windows only.

RFC Report 2015 Jun 18

| Comments

Community updates:

Chef Community Summit - http://summit.chef.io Oct 14-15 in Seattle. London will be November 3 & 4. Registration for that should open by end of next week or so. Mark your calendars

ChefConf 2016 will be moved as May conflicted with partner conferences. New dates to be announced soon.

Mailing list migration to Google Groups not happening for technical reasons. May move to Discourse; discussions underway. Other stuff:

Product Feedback - http://feedback.chef.io - read, contribute, comment, and vote on things that fall outside of support requests, this RFC process, etc. and http://www.meetup.com/Chef-Office-Hours/ where we’ve been announcing regular Chef Office Hours

Also, https://github.com/chef/chef-rfc/blob/master/rfc029-governance-policy.md CBGB members need to be selected.

Product updates

Chef server 12.1, Ent Server 11.3.2 are in progress, patching Redis is a big push here. Manage and Supermarked updates coming soon too.


https://github.com/chef/chef-rfc/pull/118 garnered the most discussion. Consensus is that having Ohai behave on the CLI like it does during a Chef run is a nice win, but there are some details still being ironed out. Also, moving the hints systems into a central config will probably help our users.

The core desire is to have a way to configure ohai so that things like knife bootstrap can leave the system set up so that ohai run directly and ohai run from Chef will behave the same way. [the] most illustrative use cases so far: - I want to disable an ohai plugin, - I want to set the ohai plugin path. - I want to change the source of data for the hostname plugin. - I want to retrieve EC2 metadata in the ec2 plugin.

RFCs to move OpenBSD to tier 2 approved and changing resource attributes to resource properties also approved (https://github.com/chef/chef-rfc/pull/128) – This is a pretty interesting change. Ubuntu 10.04 is on track to be yanked.

Next time

https://github.com/chef/chef-rfc/pull/135 - some proposals to change resource property default behaviors. Mostly I’m opposed to adding more API keywords as they only have utility in some edge cases, and make the finding the useful API methods harder to find and use. https://github.com/chef/chef-rfc/pull/136 - clarify some Chef resoure naming precedence