Hi, I'm Samuel Cochran

on Twitter, Facebook, Google, LinkedIn, GitHub, Stack Overflow, and Xbox Live.

PaaS–ish Ubuntu

I've been doing git push production for a while now, but only recently has it become something akin to Heroku. The recent feature I added was foreman exporting user upstart configs, which brings it to an ideal deployment process for me. I can now deploy an app with thin runnings rails and websockets, rapns, sidekiq, and anything else I need using just git. Here's how.

This is on a fresh Ubuntu 12.04 LTS instance on Linode after creating and swapping to an account with sudo.

Installing PostgreSQL and PostGIS

I use PostgreSQL for pretty much any data–driven app. Combining with PostGIS makes it amazing. The latest PostGIS is lacking from 12.04 but makes installation much easier, so I'm adding a PPA to get it going.

$ sudo apt-get install python-software-properties
$ sudo apt-add-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo aptitude update
$ sudo aptitude install postgresql libpq-dev postgresql-9.1-postgis postgis
$ sudo -u postgres createuser -s $USER

Installing node.js

I use node.js as my asset pipeline’s javascript engine, used for precompilation. 12.04’s nodejs is still on the 0.4 series which is a bit old for my taste, so I use another PPA:

$ sudo add-apt-repository ppa:chris-lea/node.js
$ sudo aptitude update
$ sudo aptitude install nodejs

Installing nginx

The nginx-light package in ubuntu has everything I'm after, mainly proxying and gzip support. I really only want to serve my static assets and fall back to proxying a sanitized request to my app. I also like to trim the fat from the configs a little.

If you want SSL to work you'll need to generate some certificates and keys, too, or just remove the ssl bits. Notice I'm putting them in the Debian–recommended PKI locations.

$ sudo aptitude install nginx-light

/etc/nginx.conf

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
  worker_connections 1024;
}

http {
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  gzip on;
  gzip_disable "msie6";

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

/etc/sites-available/default

server {
  listen 80 default;
  listen 443 default ssl;
  ssl_certificate /etc/ssl/certs/myserver.com.crt;
  ssl_certificate_key /etc/ssl/private/myserver.com.key;
  server_name myserver.com;
}

Upstart

Seeing as we’re going whole–hog on upstart, I like replacing the distributed sysv init script with an upstart script. Remove the initial init script:

$ sudo rm /etc/init.d/nginx

and into /etc/init/nginx.conf:

description "nginx"

start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]

expect fork
respawn
respawn limit 10 5

exec /usr/sbin/nginx

Now:

$ sudo initctl start nginx

Installing Ruby

Brightbox distribute a deb of ruby 1.9.3 with the GC and tmalloc patches which works great:

$ sudo apt-add-repository ppa:brightbox/ruby-ng-experimental
$ sudo aptitude update
$ sudo aptitude install build-essential ruby1.9.1-dev ruby-switch

Installing Ruby Tools

There are only a couple of gems I will install globally—bundler and foreman:

$ sudo gem install bundler foreman

Patching dbus to allow upstart user sessions

Upstart in 12.04 is capable of running user–initiated sessions, but sending signals to upstart as a user is restricted via DBus. There is a known workaround which is to replace the DBus permissions for upstart to be more permissive:

$ sudo wget http://bazaar.launchpad.net/~upstart-devel/upstart/trunk/download/head:/upstart.conf-20080508231852-jw3hh1a1d02tcmj7-1/Upstart.conf -O /etc/dbus-1/system.d/Upstart.conf

Don't feel too uncomfortable about this—it's a change expected in the next version of Ubuntu, so is forward compatible, and security is still tightly controlled by upstart. Users can't play with your system processes, nor with other users' processes.

Updating foreman templates to support user upstart

The upstart templates distributed with foreman are a little broken. They also don't cater to user session processes. These versions also use some of the more recent features of upstart to provide feature parity with foreman, namely chdir, setuid and env.

At the moment I just override the ones inside the gem so I don’t have to copy them into every ~/.foreman or supply --template=... each time. To jump to the right directory for these files, try cd ${$(gem which foreman)%/*/*}/data/export/upstart.

master.conf.erb

pre-start script
  mkdir -p <%= log %>
  chown -R <%= user %> <%= log %>
end script
start on (started network-interface
          or started network-manager
          or started networking)

stop on (stopping network-interface
         or stopping network-manager
         or stopping networking)

process_master.conf.erb

start on starting <%= app %>
stop on stopping <%= app %>

process.conf.erb

start on starting <%= app %>-<%= name %>
stop on stopping <%= app %>-<%= name %>
respawn

setuid <%= shell_quote user %>
chdir <%= shell_quote engine.root %>

env PORT=<%= port %>
<%- engine.env.each_pair do |key, value| -%>
env <%= key.upcase %>=<%= shell_quote value %>
<% end %>

exec <%= process.command %> >> <%= log %>/<%=name%>-<%=num%>.log 2>&1

Setting up a site

Every site should have a user to run as. The home directory will store all of the app’s code in a git reposiory. I always put my app homes into /var/app, but you could put them wherever you like. /opt is a popular choice but can be used by third party packages.

I like to name my users app_environment like myapp_production.

Adding a user

$ sudo addgroup --system myapp_production
$ sudo adduser --system --home /var/app/myapp_production --shell /usr/bin/git-shell --gid myapp_production myapp_production

Creating a database

$ createuser --no-createdb --no-createrole --no-superuser myapp_production
$ createdb --owner myapp_production --encoding utf8 myapp_production

SSH Authorized Keys

So you don’t need a password but can do that lovely ssh–based git deployment you’ll need to pop your ssh key into ~myapp_production/.ssh/authorized_keys.

Setting up the git repository

This is where the magic happens. I use the directory current inside the app home:

$ sudo -u myapp_production git init ~myapp_production/current

Inside the git repo we add some git hooks which will perform the actual deployment. I add two hooks—pre– and post–receive. Credit for these goes to the ever–wonderful Ben Hoskings who originally placed me on the path to git deployment.

~myapp_production/current/.git/hooks/pre-receive

#!/usr/bin/env ruby

refspecs = STDIN.read.chomp.split("\n")

if refspecs.length != 1
  puts %Q{
remote says:
  Hi there, you just pushed #{refspecs.length} branches, but I only accept one branch
  at a time, because I use it to update the working copy.

  The best way to push just a single branch is like this:
    $ git push <remote-name> <branch-name>

}
  exit 1
end

~myapp_production/current/.git/hooks/post-receive

#!/usr/bin/env ruby

def fail message=nil
  puts message unless message.nil?
  exit 1
end

def shell cmd
  output = `#{cmd}`
  output.chomp if $? == 0
end

def log_shell message, cmd
  print "#{message}... "
  output = `#{cmd}`
  if $? == 0
    puts "done."
  else
    fail "failed!\n\n#{output.chomp}"
  end
end

STDOUT.sync = true

refspecs = STDIN.read.chomp.split("\n")
old_id, new_id, ref_name = refspecs.first.split(/\s+/)
new_branch = ref_name.scan(/^refs\/heads\/(.*)$/).flatten.first

# Otherwise operations in sub-gits fail
ENV.delete "GIT_DIR"

ENV["RAILS_ENV"] = "production"

if new_branch.nil?
  fail "Couldn't figure out what branch '#{ref_name}' refers to, not updating."
else
  env_git = "env -i #{`which git`.chomp}"
  Dir.chdir('..') do # change dir to .git/..
    branches = shell("#{env_git} branch").split("\n")
    star_branches = branches.grep(/^\*/)
    old_branch = star_branches.empty? ? nil : star_branches.first.split(/\s+/, 2)[-1]
    branches.map! { |branch| branch.split(/\s+/, 2).last }

    if !branches.include?(new_branch)
      log_shell "Creating the '#{new_branch}' branch", "#{env_git} checkout -b '#{new_branch}'"
    end

    if old_branch != new_branch
      log_shell "Switching to the '#{new_branch}' branch", "#{env_git} checkout '#{new_branch}'"
    end

    log_shell "Updating to #{new_id[0...7]}", "#{env_git} reset --hard '#{new_id}'"
    log_shell "Updating submodules", "#{env_git} submodule update --init"
    log_shell "Bundling", "bundle --deployment"
    log_shell "Pre-compiling assets", "bundle exec rake assets:precompile:primary RAILS_GROUPS=assets"
    log_shell "Migrating", "bundle exec rake db:migrate"
    log_shell "Regenerating upstart configuration", "foreman export upstart #{ENV["HOME"]}/.init --app '#{ENV["USER"]}' --log '#{ENV["HOME"]}/current/log'"
    log_shell "Restarting", "initctl restart #{ENV["USER"]} || initctl start #{ENV["USER"]}"
  end
end

They must be marked executable. I like to chown these to root so there can be no hanky panky, but even if the hooks were changed they couldn’t really cause any more damage than the app running could—they’re both sandboxed as the app user.

$ sudo chmod +x ~myapp_production/current/.git/hooks/*
$ sudo chown -R root:root ~myapp_production/current/.git/hooks

You’ll also need to:

$ sudo -u myapp_production git config receive.denyCurrentBranch ignore

which lets git-receive-pack receive updates to the non–bare repository which are applied by the post–receive hook.

Remember foreman lets you specify a .env file as well for supplying environment variables. I usually simply use:

RAILS_ENV=production

To actually serve the app I use thin. Add it to your Gemfile, then make sure you've got a Procfile setup in your project. Here's an example from one of my projects:

web: bundle exec thin --socket tmp/web.sock --rackup config.ru --environment ${RAILS_ENV:-development} start
sidekiq: bundle exec sidekiq --environment ${RAILS_ENV:-development}
rapns: bundle exec rapns ${RAILS_ENV:-development} --foreground

Adding site to nginx

Add /etc/nginx/sites-available/myapp_production:

server {
  listen 80;
  listen 443 ssl;
  ssl_certificate ...;
  ssl_certificate_key ...;
  server_name myapp.com;
  access_log /var/log/nginx/access-myapp_production.log;
  root /var/app/myapp_production/current/public;
  location @myapp_production-web {
    proxy_pass http://unix:/var/app/myapp_production/current/tmp/web.sock;
    proxy_set_header Host $host;
  }
  try_files $uri @myapp_production-web;
}

Notice I'm proxying to thin over a unix socket as configured in the Procfile above. You could choose to use a port if you liked, as suggested by foreman, but considering everything I'm doing is local I prefer the sockets. This also means I can take advantage of socket pipelining and sendfile.

Now enable the site:

$ cd /etc/nginx/sites-enabled && sudo ln -s ../sites-available/myapp_production

Give it a kick:

$ sudo initctl restart nginx

Deploying

We’re actually ready. Here’s deploying for the first time:

$ git remote add production myapp_production@myserver.com:current
$ git push production master
Counting objects: 520, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (481/481), done.
Writing objects: 100% (520/520), 62.72 KiB, done.
Total 520 (delta 270), reused 0 (delta 0)
remote: Updating to abcd1234... done.
remote: Updating submodules... done.
remote: Bundling... done.
remote: Pre-compiling assets... done.
remote: Migrating... done.
remote: Regenerating upstart configuration... done.
remote: Restarting... done.
To myapp_production@myserver.com:current
 * [new branch]      master -> master

Now go to your app’s website:

http://myapp.com

Voila!

Some next steps

It would be nice to create an app–specific skel which can be passed to adduser containing the ssh keys and git repo, ready to go. I’ve got a basic version of this going myself.

Foreman’s upstart templates are actually a bit broken. I’ll be submitting a pull request to use the ones included here as they should work for system upstart as well. Alternately, they could go into ~/.foreman instead of into the gem directly, and this could also be in the skel directory above to reduce management overhead.

To relieve some system resources on a box with many apps I'd like to investigate having thin exit after a period of inactivity and use upstart–socket–bridge for incoming proxy connections. This is some functionality that passenger handles really well, but there are too many trade–offs to use passenger at the moment for my liking. I'm eagerly awaiting the next version.

I'm using this process specifically for a Ruby on Rails deployment here, but it's quite easily tailored to Python, PHP or just plain old HTML files. I'm using it for this blog, a Rails app, just as easily for the MailCatcher website, which is just an HTML file. Something like Heroku build packs would be awesome, but perhaps a little overkill here.

Tell me!

Please let me know what you think. You might hate it, love it, or find a security hole, but let me know! Tweet @sj26.

Regardless, I hope this is useful to you somehow.