Bryan Helmkamp

At weplay, we recently switched to pure git version control from git-svn. Now that we've had a couple weeks for the dust to settle, I'd like to share our workflow for managing deployments to our staging and production clusters.

We started by outlining the goals of our system:

• All code that's pushed to our staging and production environments must be in GitHub. Nothing goes straight from a local repository to our servers.

• Any developer can deploy our most recent work to staging.

• Any developer can deploy the code on staging into production. We (try to) avoid deploying anything to production that hasn't been pushed to staging first.

• Any developer can see a diff between "What is deployed" and "What I'm about to deploy."

• Any developer can branch from the production codebase for time sensitive tweaks and fixes. These need to be staged before they're deployed to production too.

• That functionality is available in an automated, safe, easy to use form. (Hint: rake)

We ruled out having "production" and "staging" tags because updating tags across a tree of remotes doesn't seem to work smoothly. We also ruled out doing git merges from master into our staging branch. Our staging server/codebase jumps around from the latest changes to the production code (up to a week old) based on what we need to test, so we really just want to replace it.

We settled on using remote git branches for production and staging on origin and hard resetting them to other branches to simulate a copy. We treat these branches more like tags and never commit to them directly. This has worked great so far.

Here's what our rake tasks look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

class GitCommands

  def diff_staging
    `git fetch`
    puts `git diff origin/production origin/staging`
  end
  
  def tag_staging(branch_name)
    verify_working_directory_clean
    
    `git fetch`
    `git branch -f staging origin/staging`
    `git checkout staging`
    `git reset --hard origin/#{branch_name}`
    `git push -f origin staging`
    `git checkout master`
    `git branch -D staging`
  end
  
  def tag_production
    verify_working_directory_clean
    
    `git fetch`
    `git branch -f production origin/production`
    `git checkout production`
    `git reset --hard origin/staging`
    `git push -f origin production`
    `git checkout master`
    `git branch -D production`
  end

  def branch_production(branch_name)
    verify_working_directory_clean
    
    `git fetch`
    `git branch -f production origin/production`
    `git checkout production`
    `git branch #{branch_name}`
    `git checkout #{branch_name}`
    `git push origin #{branch_name}`
  end
  
protected

  def verify_working_directory_clean
    return if `git status` =~ /working directory clean/
    raise "Must have clean working directory"
  end
end


namespace :tag do
  desc <<-DESC
    Update the staging branch to prepare for a staging deploy.
    Defaults to master. Optionally specify a BRANCH=name
  DESC
  
  task :staging do
    branch_name = ENV['BRANCH'] || "master"
    GitCommands.new.tag_staging(branch_name)
  end

  desc "Update the remove production branch to prepare for a release"
  task :production => ['diff:staging'] do
    GitCommands.new.tag_production
  end
end

namespace :diff do
  desc "Show the differences between the staging branch and the production branch"
  task :staging do
    GitCommands.new.diff_staging
  end
end

namespace :branch do
  desc "Branch from production for tweaks or bug fixs. Specify BRANCH=name"
  task :production do
    branch_name = ENV['BRANCH']
    raise "You must specify a branch name using BRANCH=name" unless branch_name
    GitCommands.new.branch_production
  end
end

namespace :deploy do
  desc "Tag and deploy staging"
  task :staging => "tag:staging" do
    `cap staging deploy:long`
  end
end

The last one (rake deploy:staging) simply wraps up the common task of tagging our latest code to be pushed to staging and initiating a staging deploy.

Note: To use most of these commands, your local working directory must be clean. If we have outstanding changes in our tree when we need to run them, we use git-stash to temporarily move them out of the way.

Thanks to Scott Chacon for helping us work this out. Be sure to check out his GitCasts and Git Internals PDF.

Just wrapped up my Story Driven Development talk at GoRuCo 2008. There were some great questions at the end, and I'm looking forward to more hallway track conversations about SDD.

Download the slides as a PDF (1.6 MB)

Note: Confreaks is recording the talks today, so a full video of my presentation should be online soon.

The online community for youth sports I’ve been working on for the last month or so, weplay, is now in public beta. It’s been a roller coaster, but entirely worth it.

As part of this launch, we’re set to appear on the front page of the New York Times business section in tomorrow’s paper. The online version of the article is already available, and we’ve been busy all night responding to our first production load…. ever.

Check it out, don’t mind a few bumps and bruises as we settle in, and expect more to come soon. Oh, and be sure to checkout my weplay profile.

I just received word that my proposal to speak at NYC’s very own second annual GoRuCo has been accepted. The conference is set for Saturday, April 26th at Pace University downtown. My topic will be the same as my Scotland on Rails presentation.

I’m really excited to have the opportunity to present at home, in front of many of my friends and colleagues. Now I just need to make sure I don’t suck. Better get back to working on my slide deck…

I’m going to be presenting on Story Driven Development at the first Scottish Ruby on Rails conference, Scotland on Rails. I’ll be exploring the concept of Story-first development from both an agile process and implementation standpoint.

Topics will include:

• Collaborating with the product owner on stories
• What makes good stories and scenarios?
• Driving stories with a browser simulator like Webrat
• Potential pitfalls and ways to avoid them

If you’re going to be in Edinburgh for the conference, let me know.

Tonight I’m releasing version 0.1.0 of my new YUI Pages Rails plugin. Here’s a quick example of what it can do (from the README):

<% yui_page :width => YUIPages::PAGE_750, :secondary => YUIPages::RIGHT_180 do %>
  <% header do %>
    Page title
  <% end %>

  <% body do %>
    <% main do %>
      <% grid :columns => YUIPages::GRID_75_25 do %>
        <% unit do %>
          Content for the 75% width first column.
        <% end %>
        <% unit do %>
          Content for the 25% width second column.
        <% end %>
      <% end %>
    <% end %>

    <% secondary do %>
      Content for the 180px right sidebar.
    <% end %>      
  <% end %>

  <% footer do %>
    Footer links here
  <% end %>
<% end %>

This light (less than 90 lines of code) plugin saves me from having to remember or care about the YUI Grids CSS IDs and classes, and handles preparing correct markup regardless of my grid arrangement.

No more looking up that “yui-t3” means a 300px sidebar on the left or forgetting to class the first unit in a grid with “first” because I moved things around in my layout file. A small victory, but a victory nonetheless.

SVN is at http://svn.eastmedia.net/public/plugins/yui_pages/. Check out the full README.

Thanks to everyone who attended my “What’s new in Rails 2?” talk last night at NYC.rb. The questions were great, and I picked up a few tips myself from the discussions we had.

Download the slides as a PDF (1.4 MB).

On Tuesday, December 11th, I’ll be speaking at NYC.rb about “What’s new in Rails 2?”. Come by and watch or check back after then and I’ll be sure to have my slides up.

Last week I pushed Webrat 0.1.0 out the door. So far the response has been great, and I’ve already received a couple patched (Thanks, David!). Here’s a quick usage example (from the README):

1
2
3
4
5
6
7
8

def test_sign_up
    visits "/"
    clicks_link "Sign up"
    fills_in "Email", :with => "good@example.com"
    select "Free account"
    clicks_button "Register"
    ...
  end

Behind the scenes, this will perform the following work:

1. Verify that loading the home page is successful
2. Verify that a “Sign up” link exists on the home page
3. Verify that loading the URL pointed to by the “Sign up” link leads to a successful page
4. Verify that there is an “Email” input field on the Sign Up page
5. Verify that there is an select field on the Sign Up page with an option for “Free account”
6. Verify that there is a “Register” submit button on the page
7. Verify that submitting the Sign Up form with the values “good@example.com” and “Free account” leads to a successful page

Take special note of the things not specified in that test, that might cause tests to break unnecessarily as your application evolves:

• The input field IDs or names (e.g. “user_email” or “user[email]”), which could change if you rename a model
• The ID of the form element (Webrat can do a good job of guessing, even if there are multiple forms on the page.)
• The URLs of links followed
• The URL the form submission should be sent to, which could change if you adjust your routes or controllers
• The HTTP method for the login request

A test written with Webrat can handle these changes smoothly.

SVN is at http://svn.eastmedia.net/public/plugins/webrat/. Check out the full README.

Flog is a tool build by Ryan Davis to analyze Ruby code complexity. It’s dead simple to run, and immediately provides useful metrics in the form of a “Flog score” per method. Don’t get too hung up on the actual values, but high outliers are prime candidates for refactoring.

I ran this on a couple recent projects I worked on, and it proved quite accurate in identifying the areas of code that seem to be in the most pain. Flog doesn’t replace hands-on code review, but it is still a helpful aid.

Below is a rake task I threw together for running Flog on a Rails project. Seems like a great candidate to install via Chris Wanstrath’s Sake, a system for handling Rake tasks that you want to be available from anywhere.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

def flog(output, *directories)
  `find #{directories.join(" ")} -name \\*.rb|xargs flog > #{RAILS_ROOT}/tmp/flog/#{output}.txt`
end

desc "Flog models, controller, helpers and lib"
task :flog do
  flog "all", *%w[app/models app/controllers app/helpers lib]
end

namespace :flog do
  desc "Flog code in app/models"
  task :models do
    flog "models", "app/models"
  end
  
  desc "Flog code in app/controllers"  
  task :controllers do
    flog "controllers", "app/controllers"
  end
  
  desc "Flog code in app/helpers"  
  task :helpers do
    flog "helpers", "app/helpers"
  end
  
  desc "Flog code in lib"  
  task :lib do
    flog "lib", "lib"
  end
end

Let’s say you’re building a blogging application and you’ve got a Post model. Posts should not show up on the public site until they are approved. Approval is done by clicking an “Approve” button next to the post in the admin area.

Your first inclination (and mine, up until recently) might be to implement this with a boolean column:

1
2
3
4
5
6
7
8

class Post < ActiveRecord::Base
  # assuming posts.approved_at is a boolean column

  def approve!
    self.approved = true
  end

end

That works just fine, and may be all you need. On the other hand, my experience has shown that storing a little bit more data is generally worth the tiny incurred complexity. For example:

1
2
3
4
5
6
7
8
9
10
11
12

class Post < ActiveRecord::Base
  # assuming posts.approved_at is a datetime column

  def approve!
    self.approved_at = Time.now
  end

  def approved?
    !approved_at.nil?
  end

end

This has proven extremely valuable for debugging and audit trail purposes, even if the extra timestamp data is never exposed in a user interface. Approved posts can still be easily queried using SQL by using a WHERE approved_at IS NOT NULL condition.

At EastMedia, we generally run a development server which gets deployed from the trunk of each projects repository on a regular basis. One way we have improved the bug reports we get as part of our QA process is asking the testers to note the revision number of the build they were looking at when the bug occurred. To make this as easy as possible, during development we include the SVN revision number in the <title> tag of every page.

The code to determine the revision number lives in application_helper.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

def revision
  @revision ||= if svn_info_from_working_copy 
    svn_info_from_working_copy["Revision"].value
  else
    last_revision_in_log
  end
rescue
  @revision = "UNKNOWN"
end

def svn_info_from_working_copy
  @svn_info ||= YAML.parse(`svn info #{RAILS_ROOT}`)
end

def last_revision_in_log
  File.readlines(RAILS_ROOT + "/../../revisions.log").last.split[3]
end

And the header RHTML looks like this:

<title>r<%= revision  %> | ...Normal page title...</title>

I love tab completion for rake tasks, and tonight I got annoyed that I didn’t have the same thing for Capistrano tasks. Here’s what I came up with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/usr/bin/env ruby

# Save this somewhere, chmod 755 it, then add
#   complete -C path/to/this/script -o default cap
# to your ~/.bashrc or ~/.bash_login
#
# If you update your tasks, just $ rm ~/.captabs*
#
# Originally adapted from 
# http://onrails.org/articles/2006/08/30/namespaces-and-rake-command-completion

exit 0 unless File.file?(File.join(Dir.pwd, 'config', 'deploy.rb'))
exit 0 unless /^cap\b/ =~ ENV["COMP_LINE"]

def cap_tasks
  if File.exists?(dotcache = File.join(File.expand_path('~'), ".captabs-#{Dir.pwd.hash}"))
    File.read(dotcache)
  else
    tasks = `cap show_tasks 2>/dev/null|tail +3|cut -c 1-30|grep "[a-z]"|grep -v "^after_"|sort`
    File.open(dotcache, 'w') { |f| f.puts tasks }
    tasks
  end
end

after_match = $'
task_match = (after_match.empty? || after_match =~ /\s$/) ? nil : after_match.split.last
tasks = cap_tasks.split("\n").map { |t| t.gsub(/\s/, '') }
tasks = tasks.select { |t| /^#{Regexp.escape task_match}/ =~ t } if task_match

puts tasks
exit 0

Enjoy.

Update: I switched to Capistrano 2.0. Here’s the new tab completion code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

#!/usr/bin/env ruby

# Save this somewhere, chmod 755 it, then add
#   complete -C path/to/this/script -o default cap
# to your ~/.bashrc
#
# If you update your tasks, just $ rm ~/.captabs*
#
# Adapted from 
# http://onrails.org/articles/2006/08/30/namespaces-and-rake-command-completion

exit 0 unless File.file?(File.join(Dir.pwd, 'config', 'deploy.rb'))
exit 0 unless /^cap\b/ =~ ENV["COMP_LINE"]

def cap_tasks
  if File.exists?(dotcache = File.join(File.expand_path('~'), ".captabs-#{Dir.pwd.hash}"))
    File.read(dotcache)
  else
    tasks = `cap -T|grep "^cap"|cut -d " " -f 2`
    File.open(dotcache, 'w') { |f| f.puts tasks }
    tasks
  end
end

after_match = $'
task_match = (after_match.empty? || after_match =~ /\s$/) ? nil : after_match.split.last
tasks = cap_tasks.split("\n").map { |t| t.gsub(/\s/, '') }
tasks = tasks.select { |t| /^#{Regexp.escape task_match}/ =~ t } if task_match

# handle namespaces
if task_match =~ /^([-\w:]+:)/
  upto_last_colon = $1
  after_match = $'
  tasks = tasks.map { |t| (t =~ /^#{Regexp.escape upto_last_colon}([-\w:]+)$/) ? "#{$1}" : t }
end

puts tasks
exit 0

For a recent “enterprisey” project I’m working on, we had to offer a variety of CSV exports for many of the models in our system. Ruby’s FasterCSV library is great for raw parsing and generation of CSV data, so I used that as the basis for a quick and dirty system to easily provide customizable exports.

The main features are:

• Transform an array of exportable records into a whole CSV file. (By default, FasterCSV transforms arrays into a single CSV row.)
• Export a whole table my calling .to_csv on the ActiveRecord subclass.
• By default, export all of an ActiveRecord’s columns except for created_at and updated_at.
• Allow simple customization of exportable columns by overriding the export_columns method in your ActiveRecord class.
• Allow multiple CSV formats by conditionally branching inside the export_columns method depending on the format parameter.
• Allow complete customization of the export by overriding the to_row method. (I haven’t actually needed this much customization yet.)

The simplest example:

Address.to_csv

Customizing the columns included in the CSV:

1
2
3
4
5

class Address < ActiveRecord::Base
  def export_columns(format = nil)
    %w[city state postal_code]
  end
end

Multiple output formats:

1
2
3
4
5
6
7
8
9
10

class Address < ActiveRecord::Base
  def export_columns(format = nil)
    case format
    when :local
      %w[street1 street2 city state postal_code]
    else
      %w[city state postal_code]
    end
  end
end

Here’s the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

require "fastercsv"

class ActiveRecord::Base
  def self.to_csv(*args)
    find(:all).to_csv(*args)
  end
  
  def export_columns(format = nil)
    self.class.content_columns.map(&:name) - ['created_at', 'updated_at']
  end
  
  def to_row(format = nil)
    export_columns(format).map { |c| self.send(c) }
  end
end

class Array
  def to_csv(options = {})
    if all? { |e| e.respond_to?(:to_row) }
      header_row = first.export_columns(options[:format]).to_csv
      content_rows = map { |e| e.to_row(options[:format]) }.map(&:to_csv)
      ([header_row] + content_rows).join
    else
      FasterCSV.generate_line(self, options)
    end
  end
end

We’re using Ezra’s BackgrounDRb to handle batch processing of CSVs uploaded to our Rails application and have been very happy with the results. Our users now even get a progress indicator as we process their job.

When it came time to deploy the new functionality to our severs, however, we ran into a known issue where the BDRb process did not detach from Capistrano, and therefore got killed as Capistrano exits.

We solved this issue with the following BackgrounDRb Capistrano tasks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

desc "Stop the backgroundrb server"
task :stop_backgroundrb , :roles => :app do
  run "cd #{current_path} && ./script/backgroundrb/stop"
end

desc "Start the backgroundrb server"
task :start_backgroundrb , :roles => :app do
  run "cd #{current_path} && RAILS_ENV=production nohup ./script/backgroundrb/start -d  > /dev/null 2>&1"
end

desc "Start the backgroundrb server"
task :restart_backgroundrb, :roles => :app do
  stop_backgroundrb
  start_backgroundrb
end

And then we enhanced the restart task so that it would restart BackgrounDRb in addition to our Mongrel processes:

1
2
3
4
5

task :restart, :roles => :app do
  stop_mongrel_cluster
  restart_backgroundrb
  start_mongrel_cluster
end