Rails app automation - hubot vs heaven

Introduction

Currently, at my company, we run the cap staging deploy command directly on the local terminal. Capistrano is a very handy tool for automated deployment, but we inevitably encounter a few issues:

• Not everyone on the team has the same environment.
• Everyone is deploying, but it’s completely unclear which branch is currently on staging.
• The deployment process is stuck on the local machine.

For a startup, having a stable development efficiency and process allows us to focus more on the product. Therefore, we hope to achieve a few things:

• The development team can deploy easily.
• No need to run commands locally or set up SSH configurations.
• Deployment can be done effortlessly even without a computer.
• We can log the deployment status.
• If an issue arises, we can quickly roll back to the previous version.

I have grown weary of typing commands in the terminal and manually adding SSH keys, so I decided to research a smoother deployment process.

Previously at Sudo, I was fortunate to have two lazy colleagues, @ocowchun and @henry, who handled DevOps very thoroughly, allowing us to focus on developing features instead of dealing with a myriad of complex configurations. (Although the service was shut down right after development...)

I currently believe the best solution is to use hubot-deploy in conjunction with heaven to assist with deployments.

However, the documentation for heaven is quite poor.

After spending a long time looking through it, I even had to check the source code to figure out how to set it up. So, I've decided to share the entire setup process with everyone in hopes of reducing the time other DevOps professionals spend going down the wrong path.

Main Process

{% asset_img "process.png" "Github deployment process" %}

Once hubot receives the deployment command, it sends a github deployment request, triggering the deployment event. GitHub then sends a POST request to the URL set in the webhook (here, the receiver is heaven). After heaven receives the request, it will start the deployment and provide updates on the deployment status.

hubot-deploy

hubot-deploy allows you to create GitHub deployment events by sending commands to the slack-bot via Slack.

heaven

Heaven is a Rails application. It primarily has a /events endpoint responsible for receiving deployment and payload data from GitHub deployments.

Setup Steps

The documentation for heaven is confusing, and hubot-deploy is also briefly covered. We mostly have to rely on their flowchart, repeatedly trying things out and troubleshooting.

Setting Up hubot-deploy

• Getting started with hubot

1. Use Yeoman to generate a hubot and select adapter as slack.

2. Add hubot-deploy to package.json, or run npm install hubot-deploy --save-dev.

3. Add hubot-deploy to external-scripts.json.

4. In apps.json, specify the repositories you want to deploy:

   {
     "repo_name": {
       "provider": "capistrano",
       "auto_merge": false,
       "repository": "kjj6198/deploy101",
       "environments": ["production", "staging"]
     }
   }

   This information will be included in the payload when hubot sends the deployment. For example:

       payload: {
         "name": "repo_name",
         "robotName": "yourrobot",
         "hosts": "",
         "notify": {
           "adapter": "slack",
           "room": "123456789",
           "user": "123456789",
           "user_name": "kjj6198"
         },
         "config": {
           "provider": "capistrano",
           "auto_merge": false,
           "repository": "kjj6198/deploy101",
           "environments": [
             "production",
             "staging"
           ]
         }
       }

It's essential to note that the provider field will be sent to heaven, so the value must be one that heaven recognizes (which will be mentioned later), or you need to implement your own Provider.

With this, our hubot setup is complete. Let’s deploy to Heroku for testing; deploying to Heroku is straightforward:

heroku login
git init
git add .
git commit -m "init"
heroku create
git push heroku master

After a successful deployment, several important variables are:

Global variables can be set in the Heroku dashboard or directly via command line:

heroku config:set HUBOT_GITHUB_TOKEN=abcccc
heroku config:set HUBOT_SLACK_TOKEN=abcccc

Test if the setup was successful by typing hubot deploy:version in the channel you set up.

{% asset_img "success.png" "success" %}

Make sure the hubot part matches your bot's name; for example, if your bot's name is tripmomo, you should type tripmomo deploy:version.

If successful, hubot will respond with the current version information.

5. Confirm that hubot has sent out the deployment event. Type hubot deploy app to staging.

6. Use curl -H "Authorization: token YOUR_GITHUB_TOKEN" https://api.github.com/repos/my-github/my-repo/deployments to check if the deployment was successfully created. If successful, it will return:

   {
       "url": "https://api.github.com/repos/my-github/my-repo/deployments/28301325",
       "id": 123456,
       "sha": "2e3xxxxxxxaaaaaaabbbbbbb",
       "ref": "develop",
       "task": "deploy",
       "payload": { // from apps.json
         "name": "my-app",
         "robotName": "tripmomo",
         "hosts": "",
         "notify": {
           "adapter": "slack",
           "room": "aabbccdd",
           "user": "aabbccdd",
           "user_name": "kalan.chen"
         },
         "config": {
           "provider": "capistrano",
           "auto_merge": false,
           "repository": "my-github/my-repo",
           "environments": [
             "production",
             "staging"
           ]
         }
       },
       "environment": "staging",
       "description": "deploy on staging from hubot-deploy-v0.13.27",
       "creator": {
         "login": "kjj6198",
         "id": 123456,
         "avatar_url": "https://avatars2.githubusercontent.com/u/123456?v=3",
         "gravatar_id": "",
         "url": "https://api.github.com/users/kjj6198",
         "html_url": "https://github.com/kjj6198",
         "followers_url": "https://api.github.com/users/kjj6198/followers",
         "following_url": "https://api.github.com/users/kjj6198/following{/other_user}",
         "gists_url": "https://api.github.com/users/kjj6198/gists{/gist_id}",
         "starred_url": "https://api.github.com/users/kjj6198/starred{/owner}{/repo}",
         "subscriptions_url": "https://api.github.com/users/kjj6198/subscriptions",
         "organizations_url": "https://api.github.com/users/kjj6198/orgs",
         "repos_url": "https://api.github.com/users/kalanchen/repos",
         "events_url": "https://api.github.com/users/kjj6198/events{/privacy}",
         "received_events_url":"https://api.github.com/users/kjj6198/received_events",
         "type": "User",
         "site_admin": false
       },
       "created_at": "2017-03-01T12:24:20Z",
       "updated_at": "2017-03-01T12:24:20Z",
       "statuses_url": "https://api.github.com/repos/my-github/my-repo/deployments/12345667/statuses",
       "repository_url": "https://api.github.com/repos/my-github/my-repo"
     }

   For more deployment API details, check out the GitHub Deployment API.

Setting Up heaven

• Clone the repo from heaven.
• Set global variables

You can check other variables here.

To elaborate on DEPLOYMENT_PRIVATE_KEY: the raw file looks like this:

--
MJVGa/WNT9aFs63ykxLCdGzav8CfQ5vKXrLrllHXUYFaB2yaN72L+fSsXAy9zMs2
vy6wV2fB6j3YrVNCnBwUUNGTX9Ka6eeK98dCvHVyyE9Iz3CJAWZxaI03Px/xX9ps
M4kDWe7IA6+mnuCVSzwQVWMdOoAXbQbhGdfeixbqljNhJrKW/jA9w4BNarwGYv4E
0MwdU9x7zpk826ytza87yXHSdNuTKcsGQk4XHMYxJECj4EM8vTlVlEyEXZtCeh2z
P4bjYkTcBom4nC/q7Ea7Pmy1iDJqs0qc1L/xtNMypMhx4iIaeDVawkvBaL6t9IPT
KVuC9Y1uw5nJP1gwxXa5qoazhcikzqRYmaeWIzsZrcVShZBrJO9/a/APxXY7qJpJ
0r1YYTykw7THYj2QYiv8cfF64/vh9cB0NELEp5hIuS82Mf6CjqRR7QYR+By3uIdD
hQ77NMpQlmIC+TCJsLoADqwmEEZCiQSejtkXXtN/mNl581jP8+ViNkWZfPYWe7g6
yUeXVN1cBPo6AIu+lStE+SlR8lbu7sdpn6lid1pJf50zeythabze81y/nrAdx+Jn
scACBJBrERkhm2wdULkqwMV2g0U53YpYVAs2fFU1hGzRcE5zF1sdy9RLLX45Mzrm
lRErTbSUcnoQJhhCso5uNY6MMnr/rQF920KA0Ufr40IBcQ8bOSX7lJucST5bZLDg
H7g16rimHgK4I9rrvKy4plvbolfpuKGMYJDS3Q7IW5cL5lWLU3HaVSn+VyZe8p3A
prVx0XmSCwpmUzbDI6FoqniVPVdgis2tV1uKdnJPVn0DoK0ersosGXmALytbYLeE
arH/cIlGGCoGbIX+Iv3u8aICBEG2eR8eXmQSlGI5rp9hGK/JrlkL3PywVmPw4Efi
atiS6Y12Tuu8bdpPxBTzXK3PoZ23Pc+1l7NXXIzBeGnj56bALOIbAY5kg+lIRdtP
NSTAW8IVgFJUl4uzy/NXn/ewiE093ZVs59I2x4OoS14S20mkM/ldWbvlVm4Z3JxC
xIWsIV8aLznttic5MJUGjGoqH1Brg0o1HyWdkoEcC1N0G57oO4pN4UTD5co5xY9j
Ai2NIcFCYzqrdTfSlPWJBZLhjZ5hOXIwuTeJfRxDAVphaUqfpXb3o3URGRWiGENA
kIYKiq4XeNguwrFBzg5CB7NEKvjbjJ31GI26yAPa7yrKpuNFAjPpO6JKdL8slvx8
GXCOSbhGPFxzmtYzEeMxmnHqOa0Z953XeheKfJoipqRAyENxPBvclDonqVfxuTvw
cZzqFD+XjDJCJ5INwuwk2WupVzQjzV6TagcIX63Kq1Z9HSoFIBiCrdLzTMDG4Ro3
2wpN1tFQFz6alvwKtifGwhvG3qqmsfcQqw56gGY0DWIqG5x/thdG7UzZT7iMVDJV
LAO5wNnBK6L+feov9LqP7ONAonBVawmTv0ArjVhhkYZEi6d+ymvPpL1ORFAymLne
dpk4VmmmQvkUu0KudRqulavTIrnXFkuv2va+5X9mHGoNNMo1TXk2XX1eM4Rc7nAY
6IwPyAuFEtT5ocWBklB/qUZtdu4fG876o0X87GklR9ZfPG+tWpH2F+1j1mMHKuiP
--

It needs to be modified to:

--\nMJVGa/WNT9aFs63ykxLCdGzav8CfQ5vKXrLrllHXUYFaB2yaN72L+fSsXAy9zMs2\nvy6wV2fB6j3YrVNCnBwUUNGTX9Ka6eeK98dCvHVyyE9Iz3CJAWZxaI03Px/xX9ps\nM4kDWe7IA6+mnuCVSzwQVWMdOoAXbQbhGdfeixbqljNhJrKW/jA9w4BNarwGYv4E\n0MwdU9x7zpk826ytza87yXHSdNuTKcsGQk4XHMYxJECj4EM8vTlVlEyEXZtCeh2z\nP4bjYkTcBom4nC/q7Ea7Pmy1iDJqs0qc1L/xtNMypMhx4iIaeDVawkvBaL6t9IPT\nKVuC9Y1uw5nJP1gwxXa5qoazhcikzqRYmaeWIzsZrcVShZBrJO9/a/APxXY7qJpJ\n0r1YYTykw7THYj2QYiv8cfF64/vh9cB0NELEp5hIuS82Mf6CjqRR7QYR+By3uIdD\nhQ77NMpQlmIC+TCJsLoADqwmEEZCiQSejtkXXtN/mNl581jP8+ViNkWZfPYWe7g6\nyUeXVN1cBPo6AIu+lStE+SlR8lbu7sdpn6lid1pJf50zeythabze81y/nrAdx+Jn\nscACBJBrERkhm2wdULkqwMV2g0U53YpYVAs2fFU1hGzRcE5zF1sdy9RLLX45Mzrm\nlRErTbSUcnoQJhhCso5uNY6MMnr/rQF920KA0Ufr40IBcQ8bOSX7lJucST5bZLDg\nH7g16rimHgK4I9rrvKy4plvbolfpuKGMYJDS3Q7IW5cL5lWLU3HaVSn+VyZe8p3A\nprVx0XmSCwpmUzbDI6FoqniVPVdgis2tV1uKdnJPVn0DoK0ersosGXmALytbYLeE\narH/cIlGGCoGbIX+Iv3u8aICBEG2eR8eXmQSlGI5rp9hGK/JrlkL3PywVmPw4Efi\natiS6Y12Tuu8bdpPxBTzXK3PoZ23Pc+1l7NXXIzBeGnj56bALOIbAY5kg+lIRdtP\nNSTAW8IVgFJUl4uzy/NXn/ewiE093ZVs59I2x4OoS14S20mkM/ldWbvlVm4Z3JxC\nxIWsIV8aLznttic5MJUGjGoqH1Brg0o1HyWdkoEcC1N0G57oO4pN4UTD5co5xY9j\nAi2NIcFCYzqrdTfSlPWJBZLhjZ5hOXIwuTeJfRxDAVphaUqfpXb3o3URGRWiGENA\nkIYKiq4XeNguwrFBzg5CB7NEKvjbjJ31GI26yAPa7yrKpuNFAjPpO6JKdL8slvx8\nGXCOSbhGPFxzmtYzEeMxmnHqOa0Z953XeheKfJoipqRAyENxPBvclDonqVfxuTvw\ncZzqFD+XjDJCJ5INwuwk2WupVzQjzV6TagcIX63Kq1Z9HSoFIBiCrdLzTMDG4Ro3\n2wpN1tFQFz6alvwKtifGwhvG3qqmsfcQqw56gGY0DWIqG5x/thdG7UzZT7iMVDJV\nLAO5wNnBK6L+feov9LqP7ONAonBVawmTv0ArjVhhkYZEi6d+ymvPpL1ORFAymLne\ndpk4VmmmQvkUu0KudRqulavTIrnXFkuv2va+5X9mHGoNNMo1TXk2XX1eM4Rc7nAY\n6IwPyAuFEtT5ocWBklB/qUZtdu4fG876o0X87GklR9ZfPG+tWpH2F+1j1mMHKuiP\n-----END RSA PRIVATE KEY-----

Since this is public, this private key is obviously compromised.

Setting Up Gemfile

Since heaven will pull the latest repo and execute the cap ... deploy command, the version of Capistrano must match the version you want to deploy. Additionally, make sure to include any asset-related gems into heaven as well. For example, if my Capfile uses:

gem 'capistrano', '3.4.0'
gem 'capistrano3-unicorn'
gem 'capistrano-rails'
gem 'sitemap_generator'
gem 'capistrano-rvm'

Then these gems need to be added to heaven's Gemfile. Heaven will pull the repo to deploy, then enter the directory to run the cap staging ... deploy command, so if the corresponding gems are not installed, heaven won’t be able to deploy.

Integrating GitHub Deployment

• First, go to the repo's settings > deploy key to add the SSH key.
• Go to the repo's settings > webhooks > add webhook.
• Fill in the Payload URL with your heaven deployment host URL, for example: https://yourapp.com.tw/events. If you want to modify it, you can change it in heaven repo's routes.rb.
• Choose application/json for Content Type.
• Secret is optional depending on your needs.
• When asked which events this webhook should listen to, select deployment and deployment status since we use deployment for our deployments.

Deployment

If deploying to Heroku, keep in mind that heaven requires Redis and Resque. Don’t forget to add the corresponding add-ons and set REDIS_URL.

Also, remember to set up the database with heroku run rake db:migrate.

Common Commands for hubot-deploy

• hubot deploy:version - Check the current version.
• hubot deploy repo - Deploy the specified repo name based on apps.json.
• hubot deploy repo/branch - Deploy a specific branch of the designated repo to the default environment. You can set HUBOT_DEPLOY_DEFAULT_ENVIRONMENT to decide.
• hubot deploy repo/branch to staging - Deploy a specified branch of the repo to staging.

Notes

• Although heaven’s documentation is unclear, the code and tests are quite complete. Developers familiar with Ruby could set up heaven, modify the code a bit to add routes, and create a one-click deployment UI.

• OptionParser::AmbiguousOption: ambiguous option: -s: I'm not sure if there have been changes to commands after a Capistrano update. The solution is to modify deploy_command in lib/heaven/provider/capistrano.rb:

  module Heaven
    # Top-level module for providers.
    module Provider
      # The capistrano provider.
      class Capistrano < DefaultProvider
   	.....
        def execute
          return execute_and_log(["/usr/bin/true"]) if Rails.env.test?
  
          unless File.exist?(checkout_directory)
            log "Cloning #{repository_url} into #{checkout_directory}"
            execute_and_log(["git", "clone", clone_url, checkout_directory])
          end
  
          Dir.chdir(checkout_directory) do
            log "Fetching the latest code"
            execute_and_log(%w{git fetch})
            execute_and_log(["git", "reset", "--hard", sha])
            deploy_command = [cap_path, environment, "deploy command for cap"]
            log "Executing capistrano: #{deploy_command.join(" ")}"
            execute_and_log(deploy_command)
          end
        end
      end
    end
  end

• Since heaven uses Gist as stdout and stderr during deployment, make sure to check the gist scope when setting the GITHUB_TOKEN.

• Net::SSH::AuthenticationFailed: Authentication failed for user apps@staging.tripmoment.com: This indicates that the SSH private key setup is incorrect. First, ensure that this SSH key has been added to GitHub, then verify that the passphrase is removed, and that the SSH private key is formatted as a single line with \n.

• ArgumentError: Could not parse PKey: no start line: This error occurs when the SSH private key's passphrase has not been removed.

Conclusion

Typically, in companies where the development team is small, DevOps responsibilities are often handled by backend developers, with frontend developers having less exposure. However, using the excuse "I'm frontend; I don't need to worry about DevOps" doesn't hold water, as developing a robust system is impossible with just frontend work.

This article attempts to integrate steps not mentioned or glossed over in the documentation. There are many details omitted in the heaven and hubot-deploy documentation that can lead to significant time lost in troubleshooting. I hope this can save others from stepping on the same landmines and having to dig through the source code.

This article also leaves out many DevOps details, as building a complete DevOps pipeline takes time, and I am still not entirely familiar with CI/CD setups.

References:

• Keke's DevOps Notes
• ChatOps