PaaS–ish Ubuntu
I've been doing git push production for a while now, but only recently has it become something akin to Heroku. The recent feature I added was foreman exporting user upstart configs, which brings it to an ideal deployment process for me. I can now deploy an app with thin runnings rails and websockets, rapns, sidekiq, and anything else I need using just git. Here's how.
This is on a fresh Ubuntu 12.04 LTS instance on Linode after creating and swapping to an account with sudo.
Installing PostgreSQL and PostGIS
I use PostgreSQL for pretty much any data–driven app. Combining with PostGIS makes it amazing. The latest PostGIS is lacking from 12.04 but makes installation much easier, so I'm adding a PPA to get it going.
$ sudo apt-get install python-software-properties
$ sudo apt-add-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo aptitude update
$ sudo aptitude install postgresql libpq-dev postgresql-9.1-postgis postgis
$ sudo -u postgres createuser -s $USER
Installing node.js
I use node.js as my asset pipeline’s javascript engine, used for precompilation. 12.04’s nodejs is still on the 0.4 series which is a bit old for my taste, so I use another PPA:
$ sudo add-apt-repository ppa:chris-lea/node.js
$ sudo aptitude update
$ sudo aptitude install nodejs
Installing nginx
The nginx-light package in ubuntu has everything I'm after, mainly proxying and gzip support. I really only want to serve my static assets and fall back to proxying a sanitized request to my app. I also like to trim the fat from the configs a little.
If you want SSL to work you'll need to generate some certificates and keys, too, or just remove the ssl bits. Notice I'm putting them in the Debian–recommended PKI locations.
$ sudo aptitude install nginx-light
/etc/nginx.conf
user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
/etc/sites-available/default
server {
listen 80 default;
listen 443 default ssl;
ssl_certificate /etc/ssl/certs/myserver.com.crt;
ssl_certificate_key /etc/ssl/private/myserver.com.key;
server_name myserver.com;
}
Upstart
Seeing as we’re going whole–hog on upstart, I like replacing the distributed sysv init script with an upstart script. Remove the initial init script:
$ sudo rm /etc/init.d/nginx
and into /etc/init/nginx.conf:
description "nginx"
start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]
expect fork
respawn
respawn limit 10 5
exec /usr/sbin/nginx
Now:
$ sudo initctl start nginx
Installing Ruby
Brightbox distribute a deb of ruby 1.9.3 with the GC and tmalloc patches which works great:
$ sudo apt-add-repository ppa:brightbox/ruby-ng-experimental
$ sudo aptitude update
$ sudo aptitude install build-essential ruby1.9.1-dev ruby-switch
Installing Ruby Tools
There are only a couple of gems I will install globally—bundler and foreman:
$ sudo gem install bundler foreman
Patching dbus to allow upstart user sessions
Upstart in 12.04 is capable of running user–initiated sessions, but sending signals to upstart as a user is restricted via DBus. There is a known workaround which is to replace the DBus permissions for upstart to be more permissive:
$ sudo wget http://bazaar.launchpad.net/~upstart-devel/upstart/trunk/download/head:/upstart.conf-20080508231852-jw3hh1a1d02tcmj7-1/Upstart.conf -O /etc/dbus-1/system.d/Upstart.conf
Don't feel too uncomfortable about this—it's a change expected in the next version of Ubuntu, so is forward compatible, and security is still tightly controlled by upstart. Users can't play with your system processes, nor with other users' processes.
Updating foreman templates to support user upstart
The upstart templates distributed with foreman are a little broken. They also don't cater to user session processes. These versions also use some of the more recent features of upstart to provide feature parity with foreman, namely chdir, setuid and env.
At the moment I just override the ones inside the gem so I don’t have to copy them into every ~/.foreman or supply --template=... each time. To jump to the right directory for these files, try cd ${$(gem which foreman)%/*/*}/data/export/upstart.
master.conf.erb
pre-start script
mkdir -p <%= log %>
chown -R <%= user %> <%= log %>
end script
start on (started network-interface
or started network-manager
or started networking)
stop on (stopping network-interface
or stopping network-manager
or stopping networking)
process_master.conf.erb
start on starting <%= app %>
stop on stopping <%= app %>
process.conf.erb
start on starting <%= app %>-<%= name %>
stop on stopping <%= app %>-<%= name %>
respawn
setuid <%= shell_quote user %>
chdir <%= shell_quote engine.root %>
env PORT=<%= port %>
<%- engine.env.each_pair do |key, value| -%>
env <%= key.upcase %>=<%= shell_quote value %>
<% end %>
exec <%= process.command %> >> <%= log %>/<%=name%>-<%=num%>.log 2>&1
Setting up a site
Every site should have a user to run as. The home directory will store all of the app’s code in a git reposiory. I always put my app homes into /var/app, but you could put them wherever you like. /opt is a popular choice but can be used by third party packages.
I like to name my users app_environment like myapp_production.
Adding a user
$ sudo addgroup --system myapp_production
$ sudo adduser --system --home /var/app/myapp_production --shell /usr/bin/git-shell --gid myapp_production myapp_production
Creating a database
$ createuser --no-createdb --no-createrole --no-superuser myapp_production
$ createdb --owner myapp_production --encoding utf8 myapp_production
SSH Authorized Keys
So you don’t need a password but can do that lovely ssh–based git deployment you’ll need to pop your ssh key into ~myapp_production/.ssh/authorized_keys.
Setting up the git repository
This is where the magic happens. I use the directory current inside the app home:
$ sudo -u myapp_production git init ~myapp_production/current
Inside the git repo we add some git hooks which will perform the actual deployment. I add two hooks—pre– and post–receive. Credit for these goes to the ever–wonderful Ben Hoskings who originally placed me on the path to git deployment.
~myapp_production/current/.git/hooks/pre-receive
#!/usr/bin/env ruby
refspecs = STDIN.read.chomp.split("\n")
if refspecs.length != 1
puts %Q{
remote says:
Hi there, you just pushed #{refspecs.length} branches, but I only accept one branch
at a time, because I use it to update the working copy.
The best way to push just a single branch is like this:
$ git push <remote-name> <branch-name>
}
exit 1
end
~myapp_production/current/.git/hooks/post-receive
#!/usr/bin/env ruby
def fail message=nil
puts message unless message.nil?
exit 1
end
def shell cmd
output = `#{cmd}`
output.chomp if $? == 0
end
def log_shell message, cmd
print "#{message}... "
output = `#{cmd}`
if $? == 0
puts "done."
else
fail "failed!\n\n#{output.chomp}"
end
end
STDOUT.sync = true
refspecs = STDIN.read.chomp.split("\n")
old_id, new_id, ref_name = refspecs.first.split(/\s+/)
new_branch = ref_name.scan(/^refs\/heads\/(.*)$/).flatten.first
# Otherwise operations in sub-gits fail
ENV.delete "GIT_DIR"
ENV["RAILS_ENV"] = "production"
if new_branch.nil?
fail "Couldn't figure out what branch '#{ref_name}' refers to, not updating."
else
env_git = "env -i #{`which git`.chomp}"
Dir.chdir('..') do # change dir to .git/..
branches = shell("#{env_git} branch").split("\n")
star_branches = branches.grep(/^\*/)
old_branch = star_branches.empty? ? nil : star_branches.first.split(/\s+/, 2)[-1]
branches.map! { |branch| branch.split(/\s+/, 2).last }
if !branches.include?(new_branch)
log_shell "Creating the '#{new_branch}' branch", "#{env_git} checkout -b '#{new_branch}'"
end
if old_branch != new_branch
log_shell "Switching to the '#{new_branch}' branch", "#{env_git} checkout '#{new_branch}'"
end
log_shell "Updating to #{new_id[0...7]}", "#{env_git} reset --hard '#{new_id}'"
log_shell "Updating submodules", "#{env_git} submodule update --init"
log_shell "Bundling", "bundle --deployment"
log_shell "Pre-compiling assets", "bundle exec rake assets:precompile:primary RAILS_GROUPS=assets"
log_shell "Migrating", "bundle exec rake db:migrate"
log_shell "Regenerating upstart configuration", "foreman export upstart #{ENV["HOME"]}/.init --app '#{ENV["USER"]}' --log '#{ENV["HOME"]}/current/log'"
log_shell "Restarting", "initctl restart #{ENV["USER"]} || initctl start #{ENV["USER"]}"
end
end
They must be marked executable. I like to chown these to root so there can be no hanky panky, but even if the hooks were changed they couldn’t really cause any more damage than the app running could—they’re both sandboxed as the app user.
$ sudo chmod +x ~myapp_production/current/.git/hooks/*
$ sudo chown -R root:root ~myapp_production/current/.git/hooks
You’ll also need to:
$ sudo -u myapp_production git config receive.denyCurrentBranch ignore
which lets git-receive-pack receive updates to the non–bare repository which are applied by the post–receive hook.
Remember foreman lets you specify a .env file as well for supplying environment variables. I usually simply use:
RAILS_ENV=production
To actually serve the app I use thin. Add it to your Gemfile, then make sure you've got a Procfile setup in your project. Here's an example from one of my projects:
web: bundle exec thin --socket tmp/web.sock --rackup config.ru --environment ${RAILS_ENV:-development} start
sidekiq: bundle exec sidekiq --environment ${RAILS_ENV:-development}
rapns: bundle exec rapns ${RAILS_ENV:-development} --foreground
Adding site to nginx
Add /etc/nginx/sites-available/myapp_production:
server {
listen 80;
listen 443 ssl;
ssl_certificate ...;
ssl_certificate_key ...;
server_name myapp.com;
access_log /var/log/nginx/access-myapp_production.log;
root /var/app/myapp_production/current/public;
location @myapp_production-web {
proxy_pass http://unix:/var/app/myapp_production/current/tmp/web.sock;
proxy_set_header Host $host;
}
try_files $uri @myapp_production-web;
}
Notice I'm proxying to thin over a unix socket as configured in the Procfile above. You could choose to use a port if you liked, as suggested by foreman, but considering everything I'm doing is local I prefer the sockets. This also means I can take advantage of socket pipelining and sendfile.
Now enable the site:
$ cd /etc/nginx/sites-enabled && sudo ln -s ../sites-available/myapp_production
Give it a kick:
$ sudo initctl restart nginx
Deploying
We’re actually ready. Here’s deploying for the first time:
$ git remote add production myapp_production@myserver.com:current
$ git push production master
Counting objects: 520, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (481/481), done.
Writing objects: 100% (520/520), 62.72 KiB, done.
Total 520 (delta 270), reused 0 (delta 0)
remote: Updating to abcd1234... done.
remote: Updating submodules... done.
remote: Bundling... done.
remote: Pre-compiling assets... done.
remote: Migrating... done.
remote: Regenerating upstart configuration... done.
remote: Restarting... done.
To myapp_production@myserver.com:current
* [new branch] master -> master
Now go to your app’s website:
Voila!
Some next steps
It would be nice to create an app–specific skel which can be passed to adduser containing the ssh keys and git repo, ready to go. I’ve got a basic version of this going myself.
Foreman’s upstart templates are actually a bit broken. I’ll be submitting a pull request to use the ones included here as they should work for system upstart as well. Alternately, they could go into ~/.foreman instead of into the gem directly, and this could also be in the skel directory above to reduce management overhead.
To relieve some system resources on a box with many apps I'd like to investigate having thin exit after a period of inactivity and use upstart–socket–bridge for incoming proxy connections. This is some functionality that passenger handles really well, but there are too many trade–offs to use passenger at the moment for my liking. I'm eagerly awaiting the next version.
I'm using this process specifically for a Ruby on Rails deployment here, but it's quite easily tailored to Python, PHP or just plain old HTML files. I'm using it for this blog, a Rails app, just as easily for the MailCatcher website, which is just an HTML file. Something like Heroku build packs would be awesome, but perhaps a little overkill here.
Tell me!
Please let me know what you think. You might hate it, love it, or find a security hole, but let me know! Tweet @sj26.
Regardless, I hope this is useful to you somehow.