POST — The Tech Behind Our Slackbot

Recently, as part of the Google’s DNI project, we built POST, a Slackbot to help journalists track article performance (and you can read our launch post here). As this was an in-house project, we used it as a learning opportunity to investigate some new technologies.

Always experimenting, always learning

Choosing the right tool for the job is one of the most important parts of web development, but when the set of tools is constantly changing this can be difficult. Here at Edenspiekermann we always try to keep ahead of the curve, staying up to date with the newest technologies and using them when it makes sense. For example, over two years ago we were early adopters of Facebook’s React.js framework when we built Red Bull’s Amaphiko project (read about it here), and now it’s one of the most popular front-end frameworks on the web.

While the Amaphiko project used some new front-end technologies, our Slackbot POST would primarily be a back-end application with very little for the end-user to see. We took this opportunity to investigate some new server-side technologies.

What our Slackbot needed

• Multiple real-time connections to communicate with teams via Slack
• Frequent calls to Google Analytics to retrieve analytic data for each article tracked
• Regular API calls to refresh team access token for data retrieval
• Background processes to analyze all data retrieved and make recommendations to users on article performance

At a very high level the project seemed quite simple, but after digging into the details it evolved into a complex set of technical requirements.

Enter Elixir & Phoenix

After our initial research, we knew the project would require a lot of concurrent processes and background jobs.

For many of the back-end-heavy projects, Ruby on Rails is our web framework of choice. It’s tried and trusted, and designed with developer productivity in mind—enabling developers to quickly build and evolve our projects in an agile manner. However, Ruby, the language Rails is built with, is single-threaded, meaning it can only process one instruction at a time. Today’s web servers usually have multi-core CPUs capable of processing multiple processes in parallel, but when running a Ruby web application, we’re limited to on process at a time. For our Slackbot, which would generate a large number of processes, this could be a potential bottleneck.

Elixir is a new programming language designed with concurrency in mind. It’s capable of processing multiple instructions in parallel, meaning we can take advantage of all cores on web server CPU. Instead of processing one instruction at a time, each core can process an instruction (in practice it's a little bit more complicated than this, you can find some in-depth information here if you're interested).

Not only is Elixir a language that handles concurrency, it can also be distributed - meaning that if using multiple cores on one CPU isn't enough, we can distribute web application over multiple machines and have them all work together to spread out the workload.

This language we be much more suitable for building our busy Slackbot - but what about the web framework?

Phoenix is a web framework built with Elixir. It makes use of elixir's concurrent and distributed capabilities to provide a performant web framework capable of dealing with high workloads. It was built from the ground up by Ruby on Rails contributors, taking many learnings from the 11 years Rails has been in use.

What we built

We started by creating a Slack App for our bot - this allowed us to add slash commands and interactive buttons for communication between the user, bot and web application.

Once our Phoenix web application was set up, we added OAuth2 integration with Slack and Google Anlaytics using two great elixir libraries, ueberauth_slack and ueberauth_google.

We then set up our Phoenix web application to receive commands and messages from teams using our bot. Once the information was received from slack, our web application handles storing the data, retrieving analytics, processing the data and notifying the user.

Here's how the general process for tracking an article looks:

1. The user adds an article URL to be tracked using the custom '/add' Slack slash command.
2. Once the Slackbot confirms the URL is correct it sends it to the Phoenix Web Application.
3. The Article URL is added to the PostgreSQL database.
4. A background job is enqueued in Redis to retrieve analytics for the article.
5. Analytics are retrieved using the Google Analytics Real Time API.
6. The web application processes the results from Google Analytics and decides if there significant performance improvement in the article.
7. The web application makes a call to the Slack API sending the user a Slack message from our Slackbot. It notifies the user that their article is performing particularly well and advises what action they should take.

How did we fare?

Learning curve

Building a web application in a new language is always going to be a challenge. It's like when our non-native German speakers try meetings 'auf Deutsch' - the work gets done, but it might not be as smooth as how it normally is.

Before we started the project we did some research and study to familiarize ourselves with Elixir. We were already reading some books on Elixir and Phoenix in our weekly ESPI Developer Book Club - so we had a head start.

We kept the scope of the project to a minimum, ensuring that we could both learn a new language and framework as well as complete the project. With the language written by a ruby contributor and the framework by a rails contributor, there are a lot of similarities that make the transition a bit easier. While we first week was a little slow, we quickly gained momentum with each passing week.

More than just a prototype

After just four weeks of development we built a working prototype. It handles multiples teams, each with numerous articles and uses a fraction of resources that a traditional Ruby on Rails application would use.

Not only did we complete our project be we added a new language and framework to our skill-set. We now have more tools in our toolbox when we're considering new projects.

We were very impressed with the performance and stability of such a new language and framework. Elixir and Phoenix were a pleasure to work with - like ruby and Ruby on Rails, they were built with developer productivity and it shows. After getting over the initial learning curve, development was quick to carry out and generally produced clean testable code.

A new tech stack?

We will certainly consider Elixir and Phoenix a great combination to tackle any backend-heavy projects in the future. The combination of excellent performance and developer productivity make for a great choice. It may be a while before they're as well known as ruby and Ruby on Rails, but we believe it's only a matter of time.