Cloud Sidekick
News/Blog Posts

Chef and AWS CloudFormation

Written on April 10, 2012 by in Blog

AWS CloudFormation provides a powerful mechanism for starting a “Stack” in AWS, using any one of the hundreds of Amazon Machine Images available.Opscode Chef is a fantastic tool for provisioning a vanilla server instance into something more useful. Sure, there are many images available with a pre-configured setup, but in reality operating systems are patched frequently, as are the hundreds of packages they include. This means the AMI images start to get stale as soon as they’re created.So, if you need a way to start the latest Ubuntu, install the specific foundation you need and keep it that way, then Chef and Cloud Formation can help.

In my next article, I’ll show how to do all this using a Cato Task.

Chef makes use of ‘recipes’ and ‘cookbooks’ to provision software on a server, and maintain a specific configuration. The Chef server is a great way to manage a big infrastructure and keep all your systems in line. But, whether you use Chef Server or not, the easy to install Chef Solo allows you to use Cookbooks on just about any OS, with little fuss.

This document makes two assumptions:
1) the reader is familiar with the basic concepts of Amazon AWS and EC2
2) the reader has at least a conceptual understanding of Opscode Chef.

The example I’ll describe below is simple. We will:

  • Briefly introduce CloudFormation.
  • Talk a little bit about Chef Cookbooks.
  • Review the high points of an AWS Cloud Formation template file.
  • Describe in detail the ‘user data’ script in the template that bootstraps Chef Solo.
  • Put it all together to fire up a WordPress server using a vanilla Ubuntu AMI and Chef Solo.

CloudFormation

“AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.” (See http://aws.amazon.com/cloudformation/ for details about AWS Cloud Formation.)

Cookbooks

According to Opscode, “Cookbooks are the fundamental units of distribution in Chef.” (See http://wiki.opscode.com/display/chef/Home for all you could ever care to know about Chef.)

A CloudFormation Template

An AWS CloudFormation template is a JSON formatted declarative document that guides Cloud Formation to create resources in AWS. Most Cloud Formation templates include at least some Parameters (inputs to the process), Resources (a few ‘things’ to create) and ‘User Data’ (typically some shell commands or a complete script to execute when the instance is available.)

Here is a very simple example Cloud Formation template that would start up an EC2 instance, including the requisite Security Groups and IAM Access Key. This template starts a t1.micro instance in the us-east-1 region. (We’re not gonna pick it apart and explain every detail, this is just a basic working template as an example.)

CloudFormation Template
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Start an Ubuntu Server Instance.",
"Parameters": {
"KeyName": {
"Description": "Name of an existing EC2 KeyPair to enable SSH access to the instances",
"Type": "String"
}
},
"Resources": {
"CfnUser": {
"Type": "AWS::IAM::User",
"Properties": {
"Path": "/",
"Policies": [
{
"PolicyName": "root",
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": "cloudformation:DescribeStackResource",
"Resource": "*"
}
]
}
}
]
}
},
"HostKeys": {
"Type": "AWS::IAM::AccessKey",
"Properties": {
"UserName": { "Ref": "CfnUser" }
}
},
"WebServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": "ami-349b495d",
"InstanceType": "t1.micro",
"SecurityGroups": [
{ "Ref": "WebServerSecurityGroup" }
],
"KeyName": { "Ref": "KeyName" }
}
},
"WebServerSecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Enable HTTP access via port 80 and SSH access",
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"CidrIp": "0.0.0.0/0"
},
{
"IpProtocol": "tcp",
"FromPort": "22",
"ToPort": "22",
"CidrIp": "0.0.0.0/0"
}
]
}
}
}
}

So, if we feed this template to the Create Stack wizard in AWS, we will get a basic Ubuntu 10.04LTS instance. That’s nice and all, but how about some cool stuff?

User Data

AWS CloudFormation templates have the ability to install packages, create files and make configuration changes through several mechanisms: a tool called cfn-init which is available in the Amazon Linux images, and Canonical has created a bunch of AMI’s supporting their similar Cloud-init feature. Both are great tools, and if you’re good working with that smaller set of images, use the one that best fits your needs. The user data method works on any instance, and since an upcoming article will show how to do this on Eucalyptus and Openstack clouds, this example uses good old-fashioned User Data.

With the instructions on the Opscode site for installing Chef Client, I put together this bash script that:

  • installs Chef client (which includes Chef Solo)
  • sets up the Opscode community cookbook repository using git
  • grabs the WordPress cookbook (prerequisites automagically included – thank you Chef!)

(See the comments in the script for an explanation of what each command is doing.)

The Script
#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive

# tell apt-get about the opscode package repository
echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" | tee /etc/apt/sources.list.d/opscode.list

#get the opscode keys
mkdir -p /etc/apt/trusted.gpg.d
gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
gpg --export packages@opscode.com | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null

# update the apt library with the current versions
apt-get --yes --quiet update

# install the opscode key permanently
apt-get install opscode-keyring

# install Chef (the echo part passes in a few required variables, otherwise the chef installer prompts the user.)
echo "chef chef/chef_server_url string none" | debconf-set-selections && apt-get install chef -y

# install 'git' (required to get the cookbook repository)
apt-get --yes --quiet install git-core

# checkout the Opscode cookbook repository.
cd /var
git clone git://github.com/opscode/chef-repo.git

# use the chef 'knife' utility to grab the cookbook(s) we want
sudo knife cookbook site install wordpress --cookbook-path /var/chef-repo/cookbooks

# create the minimal config file needed for chef solo
echo -e 'file_cache_path "/var/chef-repo"\ncookbook_path "/var/chef-repo/cookbooks"' | tee /etc/chef/solo.rb

# create the 'node' config file (tells Chef what cookbooks to apply to 'me')
# note: the cookbooks themselves will look for these config settings.
echo -e '{
"wordpress": {
"db": {
"database": "wordpress",
"user": "admin",
"password": "admin",
"host": "localhost"
}
},
"run_list": ["recipe[wordpress]"]
}' | tee /etc/chef/node.json

# kick off chef-solo
sudo chef-solo -j /etc/chef/node.json &> /tmp/myscript.log

# create a 'done' file so other scripts can know we're all finished.
touch /home/ubuntu/complete

Since a CloudFormation template is a JSON document, we’ll need to escape quotes and backslashes in order to put this script in the template. Also, take a look at the “Ref” keywords near the bottom – that’s the CloudFormation template way of capturing input variables and then sticking them in the proper place. Here’s what it looks like, escaped and comments removed:

The Script – User Data’fied
"#!/bin/bash\n",
"set -e -x\n",
"export DEBIAN_FRONTEND=noninteractive\n",
"echo \"deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main\" | tee /etc/apt/sources.list.d/opscode.list\n",
"mkdir -p /etc/apt/trusted.gpg.d\n",
"gpg --keyserver keys.gnupg.net --recv-keys 83EF826A\n",
"gpg --export packages@opscode.com | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null\n",
"apt-get --yes --quiet update\n",
"apt-get install opscode-keyring\n",
"echo \"chef chef/chef_server_url string none\" | debconf-set-selections && apt-get install chef -y\n",
"apt-get --yes --quiet install git-core\n",
"cd /var\n",
"git clone git://github.com/opscode/chef-repo.git &> /tmp/myscript.log\n",
"sudo knife cookbook site install wordpress --cookbook-path /var/chef-repo/cookbooks &> /tmp/myscript.log\n",
"echo -e 'file_cache_path \"/var/chef-repo\"\\ncookbook_path \"/var/chef-repo/cookbooks\"' | tee /etc/chef/solo.rb\n",
"echo -e '{\n",
" \"wordpress\": {\n",
" \"db\": {\n",
" \"database\": \"", { "Ref": "DBName" }, "\",\n",
" \"user\": \"", { "Ref": "DBUsername" }, "\",\n",
" \"password\": \"", { "Ref": "DBPassword" }, "\",\n",
" \"host\": \"localhost\"\n",
" }\n",
" },\n",
" \"run_list\": [\"recipe[wordpress]\"]\n",
"}' | tee /etc/chef/node.json\n",
"sudo chef-solo -j /etc/chef/node.json &> /tmp/myscript.log\n",
"touch /home/ubuntu/complete\n"

That’s it! This snip of user data will install Chef and use the WordPress cookbook to get a WordPress site up and running in minutes. This is an easy, repeatable process using Cloud Formation. Simply change the knife command, and the ‘run_list’ to install anything Chef has to offer.

Coming soon… how to do this on Eucalyptus and Openstack!

Finally, here’s the entire template I used while writing this, just to save you the time!

The Whole Enchilada
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Create a basic WordPress server using Chef solo.",
"Parameters": {
"KeyName": {
"Default": "",
"Description": "Name of an existing EC2 KeyPair to enable SSH access to the instances",
"Type": "String"
},
"InstanceType": {
"Description": "WebServer EC2 instance type",
"Type": "String",
"Default": "t1.micro",
"AllowedValues": [
"t1.micro"
],
"ConstraintDescription": "must be a valid EC2 instance type."
},
"DBName": {
"Default": "wordpress",
"Description": "The WordPress database name",
"Type": "String",
"MinLength": "1",
"MaxLength": "64",
"AllowedPattern": "[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription": "must begin with a letter and contain only alphanumeric characters."
},
"DBUsername": {
"Default": "admin",
"NoEcho": "true",
"Description": "The WordPress database admin account username",
"Type": "String",
"MinLength": "1",
"MaxLength": "16",
"AllowedPattern": "[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription": "must begin with a letter and contain only alphanumeric characters."
},
"DBPassword": {
"Default": "admin",
"NoEcho": "true",
"Description": "The WordPress database admin account password",
"Type": "String",
"MinLength": "1",
"MaxLength": "41",
"AllowedPattern": "[a-zA-Z0-9]*",
"ConstraintDescription": "must contain only alphanumeric characters."
},
"DBRootPassword": {
"Default": "",
"NoEcho": "true",
"Description": "Root password for MySQL",
"Type": "String",
"MinLength": "1",
"MaxLength": "41",
"AllowedPattern": "[a-zA-Z0-9]*",
"ConstraintDescription": "must contain only alphanumeric characters."
}
},
"Mappings": {
"AWSInstanceType2Arch": {
"t1.micro": {
"Arch": "64"
}
},
"AWSRegionArch2AMI": {
"us-east-1": {
"64": "ami-349b495d"
}
}
},
"Resources": {
"CfnUser": {
"Type": "AWS::IAM::User",
"Properties": {
"Path": "/",
"Policies": [
{
"PolicyName": "root",
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": "cloudformation:DescribeStackResource",
"Resource": "*"
}
]
}
}
]
}
},
"HostKeys": {
"Type": "AWS::IAM::AccessKey",
"Properties": {
"UserName": {
"Ref": "CfnUser"
}
}
},
"WebServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI",
{
"Ref": "AWS::Region"
},
{
"Fn::FindInMap": [
"AWSInstanceType2Arch",
{
"Ref": "InstanceType"
},
"Arch"
]
}
]
},
"InstanceType": {
"Ref": "InstanceType"
},
"SecurityGroups": [
{
"Ref": "WebServerSecurityGroup"
}
],
"KeyName": {
"Ref": "KeyName"
},
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"set -e -x\n",
"export DEBIAN_FRONTEND=noninteractive\n",
"echo \"deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main\" | tee /etc/apt/sources.list.d/opscode.list\n",
"mkdir -p /etc/apt/trusted.gpg.d\n",
"gpg --keyserver keys.gnupg.net --recv-keys 83EF826A\n",
"gpg --export packages@opscode.com | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null\n",
"apt-get --yes --quiet update\n",
"apt-get install opscode-keyring\n",
"echo \"chef chef/chef_server_url string none\" | debconf-set-selections && apt-get install chef -y\n",
"apt-get --yes --quiet install git-core\n",
"cd /var\n",
"git clone git://github.com/opscode/chef-repo.git &> /tmp/myscript.log\n",
"sudo knife cookbook site install wordpress --cookbook-path /var/chef-repo/cookbooks &> /tmp/myscript.log\n",
"echo -e 'file_cache_path \"/var/chef-repo\"\\ncookbook_path \"/var/chef-repo/cookbooks\"' | tee /etc/chef/solo.rb\n",
"echo -e '{\n",
" \"wordpress\": {\n",
" \"db\": {\n",
" \"database\": \"", { "Ref": "DBName" }, "\",\n",
" \"user\": \"", { "Ref": "DBUsername" }, "\",\n",
" \"password\": \"", { "Ref": "DBPassword" }, "\",\n",
" \"host\": \"localhost\"\n",
" }\n",
" },\n",
" \"run_list\": [\"recipe[wordpress]\"]\n",
"}' | tee /etc/chef/node.json\n",
"chef-solo -j /etc/chef/node.json &> /tmp/myscript.log\n",
"touch /home/ubuntu/complete\n"
]
]
}
}
}
},
"WebServerSecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Enable HTTP access via port 80 and SSH access",
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"FromPort": "80",
"ToPort": "80",
"CidrIp": "0.0.0.0/0"
},
{
"IpProtocol": "tcp",
"FromPort": "22",
"ToPort": "22",
"CidrIp": "0.0.0.0/0"
}
]
}
}
},
"Outputs": {
"WebsiteURL": {
"Value": {
"Fn::Join": [
"",
[
"http://",
{
"Fn::GetAtt": [
"WebServer",
"PublicDnsName"
]
},
"/wordpress"
]
]
},
"Description": "WordPress Website"
}
}
}
-

Sorry, comments are closed for this post.