Automated Rails App Deployment - hubot and heaven

Introduction

At my current company, deployments were done directly from local terminals using the cap staging deploy command. While Capistrano is a very convenient automation tool, several issues inevitably arise:

• Not everyone on the team has the exact same local environment.
• When everyone deploys on their own, nobody knows for sure which branch is currently running on staging.
• The deployment process is tied to local machines.

For a startup, having a more stable development efficiency and workflow allows the team to focus better on the product itself. Therefore, we wanted to achieve a few goals:

• Enable anyone on the dev team to deploy easily.
• Eliminate the need to run deploy commands locally or configure SSH keys on individual machines.
• Make deployments possible even without having your computer open.
• Keep track of deployment statuses and history.
• Quickly rollback to a previous version if something goes wrong.

Having grown tired of typing commands into the terminal and manually adding SSH keys, I decided to research whether there was a smoother deployment workflow.

Back at Sudo, we were lucky to have two “lazy” engineers, @ocowchun and @henry, who built such a thorough DevOps pipeline that we could focus entirely on building features instead of dealing with tedious configurations (though the service shut down right after development finished…).

The solution that currently seems most suitable is combining hubot-deploy with heaven.

However, Heaven’s documentation is honestly terrible.

After staring at it for ages and even diving into the source code to figure out how to configure it properly, I decided to share the entire setup process. Hopefully, this saves other DevOps engineers from going down the wrong path.

The Main Workflow

{% asset_img “process.png” “Github deployment process” %}

When Hubot receives a deployment command, it triggers a GitHub deployment via the API, which fires a deployment event. GitHub then sends a POST request to the webhook URL configured for the repo (in this case, the receiver is heaven). Once Heaven receives the payload, it begins the deployment and sends status updates back step by step.

hubot-deploy

hubot-deploy allows you to create GitHub deployment events by giving commands to a Slack bot.

heaven

Heaven is a Rails application. It primarily exposes an /events endpoint responsible for receiving deployments and payloads from GitHub deployments.

Setup Steps

heaven’s documentation is cryptic, and hubot-deploy’s is barely more than an overview. You basically have to rely on their workflow diagrams, endless trial-and-error, and intuition.

Setting Up hubot-deploy

• Getting started with Hubot

1. Use Yeoman to generate a Hubot project, choosing slack as the adapter.

2. Add hubot-deploy to package.json, or run npm install hubot-deploy --save-dev.

3. Add hubot-deploy to external-scripts.json.

4. Configure the repositories you want to deploy in apps.json:

   {
     "repo_name": {
       "provider": "capistrano",
       "auto_merge": false,
       "repository": "kjj6198/deploy101",
       "environments": ["production", "staging"]
     }
   }
   

   This data will be bundled into the payload when Hubot creates a deployment, looking something like this:

       payload: {
         "name": "repo_name",
         "robotName": "yourrobot",
         "hosts": "",
         "notify": {
           "adapter": "slack",
           "room": "123456789",
           "user": "123456789",
           "user_name": "kjj6198"
         },
         "config": {
           "provider": "capistrano",
           "auto_merge": false,
           "repository": "kjj6198/deploy101",
           "environments": [
             "production",
             "staging"
           ]
         }
       }

Note that the provider field will eventually be passed to Heaven, so its value must be supported by Heaven (discussed later), or you must implement a custom Provider.

At this point, our Hubot setup is complete. Let’s deploy it to Heroku to test. Deploying to Heroku is straightforward:

heroku login
git init
git add .
git commit "init"
heroku create
git push heroku master

Once deployed, here are the critical environment variables:

You can set environment variables in the Heroku dashboard or via the CLI:

heroku config:set HUBOT_GITHUB_TOKEN=abcccc
heroku config:set HUBOT_SLACK_TOKEN=abcccc

Let’s test if it works. Type hubot deploy:version in your configured channel:

{% asset_img “success.png” “success” %}

Replace hubot with your bot’s name. For instance, if the bot is named tripmomo, type tripmomo deploy:version.

If successful, Hubot will respond with the current version information.

5. Confirm that Hubot sends deployment events. Enter: hubot deploy app to staging

6. Run curl -H "Authorization: token YOUR_GITHUB_TOKEN" https://api.github.com/repos/my-github/my-repo/deployments to check whether the deployment was created successfully. If it succeeded, it will return:

   {
       "url": "https://api.github.com/repos/my-github/my-repo/deployments/28301325",
       "id": 123456,
       "sha": "2e3xxxxxxxaaaaaaabbbbbbb",
       "ref": "develop",
       "task": "deploy",
       "payload": { // from apps.json
         "name": "my-app",
         "robotName": "tripmomo",
         "hosts": "",
         "notify": {
           "adapter": "slack",
           "room": "aabbccdd",
           "user": "aabbccdd",
           "user_name": "kalan.chen"
         },
         "config": {
           "provider": "capistrano",
           "auto_merge": false,
           "repository": "my-github/my-repo",
           "environments": [
             "production",
             "staging"
           ]
         }
       },
       "environment": "staging",
       "description": "deploy on staging from hubot-deploy-v0.13.27",
       "creator": {
         "login": "kjj6198",
         "id": 123456,
         "avatar_url": "https://avatars2.githubusercontent.com/u/123456?v=3",
         "gravatar_id": "",
         "url": "https://api.github.com/users/kjj6198",
         "html_url": "https://github.com/kjj6198",
         "followers_url": "https://api.github.com/users/kjj6198/followers",
         "following_url": "https://api.github.com/users/kjj6198/following{/other_user}",
         "gists_url": "https://api.github.com/users/kjj6198/gists{/gist_id}",
         "starred_url": "https://api.github.com/users/kjj6198/starred{/owner}{/repo}",
         "subscriptions_url": "https://api.github.com/users/kjj6198/subscriptions",
         "organizations_url": "https://api.github.com/users/kjj6198/orgs",
         "repos_url": "https://api.github.com/users/kalanchen/repos",
         "events_url": "https://api.github.com/users/kjj6198/events{/privacy}",
         "received_events_url":"https://api.github.com/users/kjj6198/received_events",
         "type": "User",
         "site_admin": false
       },
       "created_at": "2017-03-01T12:24:20Z",
       "updated_at": "2017-03-01T12:24:20Z",
       "statuses_url": "https://api.github.com/repos/my-github/my-repo/deployments/12345667/statuses",
       "repository_url": "https://api.github.com/repos/my-github/my-repo"
     }

   For more deployment API details, check out the GitHub Deployment API documentation.

Setting Up heaven

• Clone the repository from heaven.
• Configure environment variables:

Other variables can be found here.

Additional note on DEPLOYMENT_PRIVATE_KEY: The original file looks like this:

--
MJVGa/WNT9aFs63ykxLCdGzav8CfQ5vKXrLrllHXUYFaB2yaN72L+fSsXAy9zMs2
vy6wV2fB6j3YrVNCnBwUUNGTX9Ka6eeK98dCvHVyyE9Iz3CJAWZxaI03Px/xX9ps
M4kDWe7IA6+mnuCVSzwQVWMdOoAXbQbhGdfeixbqljNhJrKW/jA9w4BNarwGYv4E
0MwdU9x7zpk826ytza87yXHSdNuTKcsGQk4XHMYxJECj4EM8vTlVlEyEXZtCeh2z
P4bjYkTcBom4nC/q7Ea7Pmy1iDJqs0qc1L/xtNMypMhx4iIaeDVawkvBaL6t9IPT
KVuC9Y1uw5nJP1gwxXa5qoazhcikzqRYmaeWIzsZrcVShZBrJO9/a/APxXY7qJpJ
0r1YYTykw7THYj2QYiv8cfF64/vh9cB0NELEp5hIuS82Mf6CjqRR7QYR+By3uIdD
hQ77NMpQlmIC+TCJsLoADqwmEEZCiQSejtkXXtN/mNl581jP8+ViNkWZfPYWe7g6
yUeXVN1cBPo6AIu+lStE+SlR8lbu7sdpn6lid1pJf50zeythabze81y/nrAdx+Jn
scACBJBrERkhm2wdULkqwMV2g0U53YpYVAs2fFU1hGzRcE5zF1sdy9RLLX45Mzrm
lRErTbSUcnoQJhhCso5uNY6MMnr/rQF920KA0Ufr40IBcQ8bOSX7lJucST5bZLDg
H7g16rimHgK4I9rrvKy4plvbolfpuKGMYJDS3Q7IW5cL5lWLU3HaVSn+VyZe8p3A
prVx0XmSCwpmUzbDI6FoqniVPVdgis2tV1uKdnJPVn0DoK0ersosGXmALytbYLeE
arH/cIlGGCoGbIX+Iv3u8aICBEG2eR8eXmQSlGI5rp9hGK/JrlkL3PywVmPw4Efi
atiS6Y12Tuu8bdpPxBTzXK3PoZ23Pc+1l7NXXIzBeGnj56bALOIbAY5kg+lIRdtP
NSTAW8IVgFJUl4uzy/NXn/ewiE093ZVs59I2x4OoS14S20mkM/ldWbvlVm4Z3JxC
xIWsIV8aLznttic5MJUGjGoqH1Brg0o1HyWdkoEcC1N0G57oO4pN4UTD5co5xY9j
Ai2NIcFCYzqrdTfSlPWJBZLhjZ5hOXIwuTeJfRxDAVphaUqfpXb3o3URGRWiGENA
kIYKiq4XeNguwrFBzg5CB7NEKvjbjJ31GI26yAPa7yrKpuNFAjPpO6JKdL8slvx8
GXCOSbhGPFxzmtYzEeMxmnHqOa0Z953XeheKfJoipqRAyENxPBvclDonqVfxuTvw
cZzqFD+XjDJCJ5INwuwk2WupVzQjzV6TagcIX63Kq1Z9HSoFIBiCrdLzTMDG4Ro3
2wpN1tFQFz6alvwKtifGwhvG3qqmsfcQqw56gGY0DWIqG5x/thdG7UzZT7iMVDJV
LAO5wNnBK6L+feov9LqP7ONAonBVawmTv0ArjVhhkYZEi6d+ymvPpL1ORFAymLne
dpk4VmmmQvkUu0KudRqulavTIrnXFkuv2va+5X9mHGoNNMo1TXk2XX1eM4Rc7nAY
6IwPyAuFEtT5ocWBklB/qUZtdu4fG876o0X87GklR9ZfPG+tWpH2F+1j1mMHKuiP
--

You need to convert it into a single line format with newlines escaped:

--\nMJVGa/WNT9aFs63ykxLCdGzav8CfQ5vKXrLrllHXUYFaB2yaN72L+fSsXAy9zMs2\nvy6wV2fB6j3YrVNCnBwUUNGTX9Ka6eeK98dCvHVyyE9Iz3CJAWZxaI03Px/xX9ps\nM4kDWe7IA6+mnuCVSzwQVWMdOoAXbQbhGdfeixbqljNhJrKW/jA9w4BNarwGYv4E\n0MwdU9x7zpk826ytza87yXHSdNuTKcsGQk4XHMYxJECj4EM8vTlVlEyEXZtCeh2z\nP4bjYkTcBom4nC/q7Ea7Pmy1iDJqs0qc1L/xtNMypMhx4iIaeDVawkvBaL6t9IPT\nKVuC9Y1uw5nJP1gwxXa5qoazhcikzqRYmaeWIzsZrcVShZBrJO9/a/APxXY7qJpJ\n0r1YYTykw7THYj2QYiv8cfF64/vh9cB0NELEp5hIuS82Mf6CjqRR7QYR+By3uIdD\nhQ77NMpQlmIC+TCJsLohtJEmEEZCiQSejtkXXtN/mNl581jP8+ViNkWZfPYWe7g6\nyUeXVN1cBPo6AIu+lStE+SlR8lbu7sdpn6lid1pJf50zeythabze81y/nrAdx+Jn\nscACBJBrERkhm2wdULkqwMV2g0U53YpYVAs2fFU1hGzRcE5zF1sdy9RLLX45Mzrm\nlRErTbSUcnoQJhhCso5uNY6MMnr/rQF920KA0Ufr40IBcQ8bOSX7lJucST5bZLDg\nH7g16rimHgK4I9rrvKy4plvbolfpuKGMYJDS3Q7IW5cL5lWLU3HaVSn+VyZe8p3A\nprVx0XmSCwpmUzbDI6FoqniVPVdgis2tV1uKdnJPVn0DoK0ersosGXmALytbYLeE\narH/cIlGGCoGbIX+Iv3u8aICBEG2eR8eXmQSlGI5rp9hGK/JrlkL3PywVmPw4Efi\natiS6Y12Tuu8bdpPxBTzXK3PoZ23Pc+1l7NXXIzBeGnj56bALOIbAY5kg+lIRdtP\nNSTAW8IVgFJUl4uzy/NXn/ewiE093ZVs59I2x4OoS14S20mkM/ldWbvlVm4Z3JxC\nxIWsIV8aLznttic5MJUGjGoqH1Brg0o1HyWdkoEcC1N0G57oO4pN4UTD5co5xY9j\nAi2NIcFCYzqrdTfSlPWJBZLhjZ5hOXIwuTeJfRxDAVphaUqfpXb3o3URGRWiGENA\nkIYKiq4XeNguwrFBzg5CB7NEKvjbjJ31GI26yAPa7yrKpuNFAjPpO6JKdL8slvx8\nGXCOSbhGPFxzmtYzEeMxmnHqOa0Z953XeheKfJoipqRAyENxPBvclDonqVfxuTvw\ncZzqFD+XjDJCJ5INwuwk2WupVzQjzV6TagcIX63Kq1Z9HSoFIBiCrdLzTMDG4Ro3\n2wpN1tFQFz6alvwKtifGwhvG3qqmsfcQqw56gGY0DWIqG5x/thdG7UzZT7iMVDJV\nLAO5wNnBK6L+feov9LqP7ONAonBVawmTv0ArjVhhkYZEi6d+ymvPpL1ORFAymLne\ndpk4VmmmQvkUu0KudRqulavTIrnXFkuv2va+5X9mHGoNNMo1TXk2XX1eM4Rc7nAY\n6IwPyAuFEtT5ocWBklB/qUZtdu4fG876o0X87GklR9ZfPG+tWpH2F+1j1mMHKuiP\n-----END RSA PRIVATE KEY-----

(Since it’s published here, this private key has obviously been revoked.)

Configuring the Gemfile

Heaven works by pulling the latest code and executing cap ... deploy. Therefore, the Capistrano version must match the version used by the target application. At the same time, any asset-related or deployment-related gems must also be added to Heaven. For example, if your Capfile uses:

gem 'capistrano', '3.4.0'
gem 'capistrano3-unicorn'
gem 'capistrano-rails'
gem 'sitemap_generator'
gem 'capistrano-rvm'

Then you must add these gems to Heaven’s Gemfile. Because Heaven pulls the target repo into a local directory and runs cap staging ... deploy from there, it won’t be able to deploy without the required gems installed.

Connecting to GitHub Deployments

• First, go to the repo’s Settings > Deploy keys and add the SSH public key.
• Go to the repo’s Settings > Webhooks > Add webhook.
• In Payload URL, enter your Heaven deployment host URL, e.g., https://yourapp.com.tw/events. If you want to change this path, you can edit routes.rb in Heaven.
• Set Content type to application/json.
• Secret is optional depending on your needs.
• Under “Which events would you like to trigger this webhook?”, select Deployments and Deployment statuses.

Deploying Heaven

If you are deploying Heaven to Heroku, it requires Redis and Resque. Remember to provision the appropriate add-on and set REDIS_URL.

Also, don’t forget to run database migrations: heroku run rake db:migrate.

Common hubot-deploy Commands

• hubot deploy:version: Shows the current version.
• hubot deploy repo: Deploys the specified repo name based on apps.json.
• hubot deploy repo/branch: Deploys a specific branch of the repo to the default environment (configured via HUBOT_DEPLOY_DEFAULT_ENVIRONMENT).
• hubot deploy repo/branch to staging: Deploys a specific branch of the repo to staging.

Notes

• While Heaven’s documentation leaves much to be desired, its codebase and tests are quite well-written. Developers familiar with Ruby could easily set up Heaven, modify the code, add routes, and build a web UI for one-click deployments.

• OptionParser::AmbiguousOption: ambiguous option: -s: Not sure if this was caused by command-line flag changes in newer Capistrano versions. The solution is to modify deploy_command in lib/heaven/provider/capistrano.rb:

  module Heaven
    # Top-level module for providers.
    module Provider
      # The capistrano provider.
      class Capistrano < DefaultProvider
   	.....
        def execute
          return execute_and_log(["/usr/bin/true"]) if Rails.env.test?
  
          unless File.exist?(checkout_directory)
            log "Cloning #{repository_url} into #{checkout_directory}"
            execute_and_log(["git", "clone", clone_url, checkout_directory])
          end
  
          Dir.chdir(checkout_directory) do
            log "Fetching the latest code"
            execute_and_log(%w{git fetch})
            execute_and_log(["git", "reset", "--hard", sha])
            deploy_command = [cap_path, environment, "YOUR_CAP_DEPLOY_COMMAND"]
            log "Executing capistrano: #{deploy_command.join(" ")}"
            execute_and_log(deploy_command)
          end
        end
      end
    end
  end

• Because Heaven uses Gists to capture stdout and stderr during deployment, make sure to check the gist scope when creating the GITHUB_TOKEN.

• Net::SSH::AuthenticationFailed: Authentication failed for user apps@staging.tripmoment.com: The SSH private key is misconfigured. Verify that the public key has been added to GitHub, confirm that the passphrase has been removed, and ensure the private key has been flattened into a single line with \n.

• ArgumentError: Could not parse PKey: no start line: The passphrase on the SSH private key was not removed.

Wrap-up

In smaller development teams, DevOps responsibilities usually fall on backend engineers, while frontend developers rarely get involved. However, using “I’m a frontend dev, DevOps isn’t my job” as an excuse not to learn didn’t sit right with me—building a healthy system takes far more than just frontend work.

This post attempts to pull together the steps that were either missing or glossed over in the documentation. Both Heaven and hubot-deploy omit many details in their docs, leading to a lot of time lost to trial-and-error. Hopefully, this saves you from stepping on the same landmines and digging through source code.

There are still many DevOps details not covered here, as building a comprehensive CI/CD pipeline takes time and I am still learning the ropes myself.

Reference Resources:

• ocowchun’s DevOps Notes
• ChatOps