Automatic Signing and Publishing of Android Apps from Travis

As I discussed about preparing the apps in Play Store for automatic deployment and Google App Signing in previous blogs, in this blog, I’ll talk about how to use Travis Ci to automatically sign and publish the apps using fastlane, as well as how to upload sensitive information like signing keys and publishing JSON to the Open Source repository. This method will be used to publish the following Android Apps:

Current Project Structure

The example project I have used to set up the process has the following structure:

It’s a normal Android Project with some .travis.yml and some additional bash scripts in scripts folder. The update-apk.sh file is standard app build and repo push file found in FOSSASIA projects. The process used to develop it is documented in previous blogs. First, we’ll see how to upload our keys to the repo after encrypting them.

Encrypting keys using Travis

Travis provides a very nice documentation on encrypting files containing sensitive information, but a crucial information is buried below the page. As you’d normally want to upload two things to the repo – the app signing key, and API JSON file for release manager API of Google Play for Fastlane, you can’t do it separately by using standard file encryption command for travis as it will override the previous encrypted file’s secret. In order to do so, you need to create a tarball of all the files that need to be encrypted and encrypt that tar instead. Along with this, before you need to use the file, you’ll have to decrypt in in the travis build and also uncompress it for use.

So, first install Travis CLI tool and login using travis login (You should have right access to the repo and Travis CI in order to encrypt the files for it)

Then add the signing key and fastlane json in the scripts folder. Let’s assume the names of the files are key.jks and fastlane.json

Then, go to scripts folder and run this command to create a tar of these files:

tar cvf secrets.tar fastlane.json key.jks

 

secrets.tar will be created in the folder. Now, run this command to encrypt the file

travis encrypt-file secrets.tar

 

A new file secrets.tar.enc will be created in the folder. Now delete the original files and secrets tar so they do not get added to the repo by mistake. The output log will show the the command for decryption of the file to be added to the .travis.yml file.

Decrypting keys using Travis

But if we add it there, the keys will be decrypted for each commit on each branch. We want it to happen only for master branch as we only require publishing from that branch. So, we’ll create a bash script prep-key.sh for the task with following content

#!/bin/sh
set -e

export DEPLOY_BRANCH=${DEPLOY_BRANCH:-master}

if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_REPO_SLUG" != "iamareebjamal/android-test-fastlane" -o "$TRAVIS_BRANCH" != "$DEPLOY_BRANCH" ]; then
    echo "We decrypt key only for pushes to the master branch and not PRs. So, skip."
    exit 0
fi

openssl aes-256-cbc -K $encrypted_4dd7_key -iv $encrypted_4dd7_iv -in ./scripts/secrets.tar.enc -out ./scripts/secrets.tar -d
tar xvf ./scripts/secrets.tar -C scripts/

 

Of course, you’ll have to change the commands and arguments according to your need and repo. Specially, the decryption command keys ID

The script checks if the repo and branch are correct, and the commit is not of a PR, then decrypts the file and extracts them in appropriate directory

Before signing the app, you’ll need to store the keystore password, alias and key password in Travis Environment Variables. Once you have done that, you can proceed to signing the app. I’ll assume the variable names to be $STORE_PASS, $ALIAS and $KEY_PASS respectively

Signing App

Now, come to the part in upload-apk.sh script where you have the unsigned release app built. Let’s assume its name is app-release-unsigned.apk.Then run this command to sign it

cp app-release-unsigned.apk app-release-unaligned.apk
jarsigner -verbose -tsa http://timestamp.comodoca.com/rfc3161 -sigalg SHA1withRSA -digestalg SHA1 -keystore ../scripts/key.jks -storepass $STORE_PASS -keypass $KEY_PASS app-release-unaligned.apk $ALIAS

 

Then run this command to zipalign the app

${ANDROID_HOME}/build-tools/25.0.2/zipalign -v -p 4 app-release-unaligned.apk app-release.apk

 

Remember that the build tools version should be the same as the one specified in .travis.yml

This will create an apk named app-release.apk

Publishing App

This is the easiest step. First install fastlane using this command

gem install fastlane

 

Then run this command to publish the app to alpha channel on Play Store

fastlane supply --apk app-release.apk --track alpha --json_key ../scripts/fastlane.json --package_name com.iamareebjamal.fastlane

 

You can always configure the arguments according to your need. Also notice that you have to provide the package name for Fastlane to know which app to update. This can also be stored as an environment variable.

This is all for this blog, you can read more about travis CLI, fastlane features and signing process in these links below:

Automatic Signing and Publishing of Android Apps from Travis

Enabling Google App Signing for Android Project

Signing key management of Android Apps is a hectic procedure and can grow out of hand rather quickly for large organizations with several independent projects. We, at FOSSASIA also had to face similar difficulties in management of individual keys by project maintainers and wanted to gather all these Android Projects under singular key management platform:

To handle the complexities and security aspect of the process, this year Google announced App Signing optional program where Google takes your existing key’s encrypted file and stores it on their servers and asks you to create a new upload key which will be used to sign further updates of the app. It takes the certificates of your new upload key and maps it to the managed private key. Now, whenever there is a new upload of the app, it’s signing certificate is matched with the upload key certificate and after verification, the app is signed by the original private key on the server itself and delivered to the user. The advantage comes where you lose your key, its password or it is compromised. Before App Signing program, if your key got lost, you had to launch your app under a new package name, losing your existing user base. With Google managing your key, if you lose your upload key, then the account owner can request Google to reassign a new upload key as the private key is secure on their servers.

There is no difference in the delivered app from the previous one as it is still finally signed by the original private key as it was before, except that Google also optimizes the app by splitting it into multiple APKs according to hardware, demographic and other factors, resulting in a much smaller app! This blog will take you through the steps in how to enable the program for existing and new apps. A bit of a warning though, for security reasons, opting in the program is permanent and once you do it, it is not possible to back out, so think it through before committing.

For existing apps:

First you need to go to the particular app’s detail section and then into Release Management > App Releases. There you would see the Get Started button for App Signing.

The account owner must first agree to its terms and conditions and once it’s done, a page like this will be presented with information about app signing infrastructure at top.

So, as per the instructions, download the PEPK jar file to encrypt your private key. For this process, you need to have your existing private key and its alias and password. It is fine if you don’t know the key password but store password is needed to generate the encrypted file. Then execute this command in the terminal as written in Step 2 of your Play console:

java -jar pepk.jar –keystore={{keystore_path}} –alias={{alias}} –output={{encrypted_file_output_path}} –encryptionkey=eb10fe8f7c7c9df715022017b00c6471f8ba8170b13049a11e6c09ffe3056a104a3bbe4ac5a955f4ba4fe93fc8cef27558a3eb9d2a529a2092761fb833b656cd48b9de6a

You will have to change the bold text inside curly braces to the correct keystore path, alias and the output file path you want respectively.

Note: The encryption key has been same for me for 3 different Play Store accounts, but might be different for you. So please confirm in Play console first

When you execute the command, it will ask you for the keystore password, and once you enter it, the encrypted file will be generated on the path you specified. You can upload it using the button on console.

After this, you’ll need to generate a new upload key. You can do this using several methods listed here, but for demonstration we’ll be using command line to do so:

keytool -genkey -v -keystore {{keystore_path}} -alias {{alias_name}} -keyalg RSA -keysize 2048 -validity 10000

The command will ask you a couple of questions related to the passwords and signing information and then the key will be generated. This will be your public key and be used for further signing of your apps. So keep it and the password secure and handy (even if it is expendable now).

After this step, you need to create a PEM upload certificate for this key, and in order to do so, execute this command:

keytool -export -rfc -keystore {{keystore_path}} -alias {{alias_name}} -file {{upload_certificate.pem}}

After this is executed, it’ll ask you the keystore password, and once you enter it, the PEM file will be generated and you will have to upload it to the Play console.

If everything goes right, your Play console will look something like this:

 

Click enrol and you’re done! Now you can go to App Signing section of the Release Management console and see your app signing and new upload key certificates

 

You can use the SHA1 hash to confirm the keys as to which one corresponds to private and upload if ever in confusion.

For new apps:

For new apps, the process is like a walk in park. You just need to enable the App Signing, and you’ll get an option to continue, opt-out or re-use existing key.

 

If you re-use existing key, the process is finished then and there and an existing key is deployed as the upload key for this app. But if you choose to Continue, then App Signing will be enabled and Google will use an arbitrary key as private key for the app and the first app you upload will get its key registered as the upload key

 

This is the screenshot of the App Signing console when there is no first app uploaded and you can see that it still has an app signing certificate of a key which you did not upload or have access to.

If you want to know more about app signing program, check out these links:

Enabling Google App Signing for Android Project

Map Responses from SUSI Server

Giving user responses in Maps is a very common reply of various assistants. Android and iOS clients of SUSI.AI can Render their own custom map views but there is a challenge in doing that in Bots and other clients which can only render Images.

The MapServlet in SUSI-Server is a servlet that takes the constraints of a map as input parameters and returns PNG Image of the rendered map.

This is a simple HttpServlet. Since it does not requires any auth or user rights we do not extend AbstactAPIHandler in this class.

 
public class MapServlet extends HttpServlet {

This is to method which runs whenever the Servlet receives a GET Query. After getting the query the function call another function “process(…)” which processes the input parameters.

 
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse          response) throws ServletException, IOException {
Query post = RemoteAccess.evaluate(request);

This line is the for DDOS Protection.  It checks if the user query rate is too high and rate limit his queries. If Server suspects that the post is a DDOS then it returns a 503 error.

if (post.isDoS_blackout()) {
  response.sendError(503, "your request frequency is too high"); return;
} 
process(request, response, post);

The process method get the zoom, latitude, longitude, width and the height of the map and checks for the errors in them.

if (map.getHeight() > height || map.getWidth() > width) {
            BufferedImage bi = map.getImage();

            int xoff = (map.getWidth() - width) / 2;
            int yoff = (map.getHeight() - height) / 2;

            bi = bi.getSubimage(xoff, yoff, width, height);
            map = new RasterPlotter(width, height, RasterPlotter.DrawMode.MODE_REPLACE, "FFFFFF");
            map.insertBitmap(bi, 0, 0);
        }

Then we compute our image using RasterPlotter.

map.setDrawMode(DrawMode.MODE_SUB);
map.setColor(0xffffff);
if (text.length() > 0) PrintTool.print(map, 6, 12, 0, uppercase ? text.toUpperCase() : text, -1, false, 100);
PrintTool.print(map, map.getWidth() - 6, map.getHeight() - 6, 0, "MADE WITH LOKLAK.ORG", 1, false, 50);

Here we calculate the height of the map. If a user passed a height and width in get parameters, we scale the map according to that but if not its 256×256.

    int mx = (int) (map.getWidth() * (lon - west_lon) / (east_lon - west_lon));
    int my = (int) (map.getHeight() * (lat - north_lat) / (south_lat - north_lat));

// the marker has a height of 40 pixel and a width of 25 pixel
    final BufferedImage logo = ImageIO.read(FileSystems.getDefault().getPath("html").resolve("artwork").resolve("marker-red.png").toFile());
    map.insertBitmap(logo, Math.min(map.getWidth() - 25, Math.max(0, mx - 12)), Math.min(map.getHeight() - 40, Math.max(0, my - 40)), FilterMode.FILTER_ANTIALIASING);

We use the code below to set some text on the map for example the name of the place or the city etc.

map.setDrawMode(DrawMode.MODE_SUB);
map.setColor(0xffffff);
if (text.length() > 0) PrintTool.print(map, 6, 12, 0, uppercase ? text.toUpperCase() : text, -1, false, 100);
PrintTool.print(map, map.getWidth() - 6, map.getHeight() - 6, 0, "MADE WITH LOKLAK.ORG", 1, false, 50);

Finally we draw a marker on map

    int mx = (int) (map.getWidth() * (lon - west_lon) / (east_lon - west_lon));
    int my = (int) (map.getHeight() * (lat - north_lat) / (south_lat - north_lat));
      
    final BufferedImage logo = ImageIO.read(FileSystems.getDefault().getPath("html").resolve("artwork").resolve("marker-red.png").toFile());
    map.insertBitmap(logo, Math.min(map.getWidth() - 25, Math.max(0, mx - 12)), Math.min(map.getHeight() - 40, Math.max(0, my - 40)), FilterMode.FILTER_ANTIALIASING);

  At last we set the copyright message of OPENSTREETMAP at the bottom left.

PrintTool.print(map, 6, map.getHeight() - 6, 0, "(C) OPENSTREETMAP CONTRIBUTORS", -1, false, 100);

At the end we write the image and set the headers for Cross Origin Access.

    response.addHeader("Access-Control-Allow-Origin", "*");
        RemoteAccess.writeImage(fileType, response, post, map);
        post.finalize();
    }
}

 The servlet can be tested locally at

http://localhost:4000/vis/map.png?text=Test&mlat=1.28373&mlon=103.84379&zoom=18

Or on the live SUSI Server
http://api.susi.ai/vis/map.png?text=Test&mlat=1.28373&mlon=103.84379&zoom=18

Output:

References:

https://www.openstreetmap.org/

http://wiki.openstreetmap.org/wiki/Java_Access_Example

https://github.com/fossasia/susi_server/

Map Responses from SUSI Server

Keeping alive the messenger architecture on free heroku dynos

Heroku is a cloud application platform – a new way of building and deploying web apps and a PaaS service to host applications in various programming languages and frameworks on its cloud. The entire architecture behind the messengers has been written with nodejs and deployed to heroku as its production for one main reason which is that it provides a verified and signed SSL Certificate along with its deployment which is useful for facebook messenger integration of susi as well as useful for telegram to trigger webhooks on SSL.

A major problem with the dynos (apps deployment) is that for free users the deployment automatically goes to sleep if no one is using it for a short while and only wakes up when the endpoint is hit and also is available in a day for upto 18 hours. This could be problematic when the deployment is made, so the best way is to have maximum utilization of the resources and keep it awake as and when required. This was resolved with facebook messenger because of incoming event webhooks that are available which request the server to wake up in case they sleep but what about Slack ? The slack server doesn’t send a notification event to the server when a message is sent by a user mentioning susi which is just like every other user and can be added to a channel.

To fix this problem there are multiple approaches that we’ve taken, the first step is to ensure that every fixed time interval the server pings itself so that it’s kept alive. This is accomplished with this fraction of the code

setInterval(function() {
		http.get(heroku_deploy_url);
	}, 1800000); //pings the deployment url every 30 minutes

where the heroku_deploy_url is an environment variable that can be set by the user depending on the URL of the deployment.

Another option was to use the New Relic insights and their availability tracker to keep sending requests after every fixed interval of time and use it to keep the server alive. This can be accomplished by doing the following on the heroku toolbelt.

heroku addons:add newrelic:standard
heroku addons:open newrelic

then using the following ruby script and setting the PING_URL as heroku config:add PING_URL=http://longweburl.herokuapp.com

desc "Pings PING_URL to keep a dyno alive"
    task :dyno_ping do
      require "net/http"

      if ENV['PING_URL']
        uri = URI(ENV['PING_URL'])
        Net::HTTP.get_response(uri)
      end
    end

and then performing the execution of the task by doing

heroku addons:add scheduler:standard
heroku addons:open scheduler
rake dyno_ping

The last option was however to use the existing loklak.net server setup a cronjob on that to query the heroku instance periodically so that the instance quota doesn’t get over and at the same time has as much uptime as majorly required. The best option however would be to upgrade to a hobby plan and purchase a dyno to host the resource.

Keeping alive the messenger architecture on free heroku dynos

Setting up Susi for access from Telegram Messenger

Telegram is one of the popular applications for communication in the open source community and was one of the first apps to come out with end to encryption soon to be followed suit by whatsapp messenger and other messengers. Telegram is used by a large number of people and we the folks at loklak put on our thinking hats and decided why not provide susi’s capabilities to those using Telegram and the telegram integration to the ask susi messengers integration was born.

Consuming the Susi API with Telegram is fairly straightforward in telegram. The first step is the use of botfather in telegram ensures that the bots can be created with ease. So the first step is to login into telegram with your user account and search and talk to BotFather. Bot father would ask a few questions and then provide the required token. You need to save this token and the bot powered by susi is now available.

botfather1

botfather2

This sets up the bot and provides the token, the next step is to use the token and setup the way in which it responds. This was done by keeping the token as an environment variable

var telegramToken = process.env.TELEGRAM_ACCESS_TOKEN;

and continuing to build the response system around what should happen when a message event is received from telegram. To start the bot the standard entry to the bot happens using the /start message that’s sent to the telegram service.

bot.onText(/\/start/, function (msg, match) {
	var fromId = msg.from.id;
	var resp = 'Hi, I am Susi, You can ask me anything !';
	bot.sendMessage(fromId, resp);
});

This initiates the bot if it’s the first time a user is using the bot, here after every event is read and processed by susi and the response is returned

bot.on('message', function (msg) {
	var chatId = msg.chat.id;
	var queryUrl = 'http://loklak.org/api/susi.json?q='+encodeURI(msg.text);
	var message = '';
	// Wait until done and reply
	if (msg.text !== '/start') {
		request({
			url: queryUrl,
			json: true
		}, function (error, response, body) {
			if (!error && response.statusCode === 200) {
				message = body.answers[0].actions[0].expression;
				bot.sendMessage(chatId, message);
			} else {
				message = 'Oops, Looks like Susi is taking a break, She will be back soon';
				bot.sendMessage(chatId, message);
			}
		});
	}
});

Here’s how Susi’s capabilities are now available to all those users on telegram.

Setting up Susi for access from Telegram Messenger

Susi supports Map tiles

Susi chat client supports map tiles now. Try out the following query related to location and Susi responses with a internal map tile with the pin at the location.

Where is Singapore? 

Selection_310

You have internal zoom in and zoom out options for the map. It also provides you with a link to open street maps where you can get the whole view of the location. Other than the map tile it gives you information about the location’s population. Isn’t it awesome?

Implementation: Let’s get into the implementation part. There are multiple ways to display the map tiles. One way is we can use our own Loklak services for displaying the map. For example -> you can make a call to the /vis/map.png API for displaying the map. But the issue with this API is we cant dynamically zoom in or zoom out the map on the tile, which is moreover a static display. So to make it more interesting we used a js library called Leaflet, which provides interactive maps.

How is the data captured? Here is the sample response coming from the server.

Selection_307

This sample data which contains latitude, longitude, place and population helps us for drawing the map tiles.

  1. expression:“Berlin is a place with a population of 3426354. Here is a map: https://www.openstreetmap.org/#map=13/52.52436820069531/13.41053001275776”
  2. type:“answer”

The above two keys are under the actions object providing the us the answer along with the URL where it is linkified.

How is the map captured? First we included the following required library files.

<script src=”https://npmcdn.com/[email protected]/dist/leaflet.js”></script>

<link rel=”stylesheet” href=”https://npmcdn.com/[email protected]/dist/leaflet.css” />

Selection_312

The JSON response is parsed and the co ordinates (lat, lon) are captured. The variable type mapType initialized.

Selection_313

PS: The co ordinates are passed into the response, for telling the html that ‘hey the map is coming in the response!’ so that it prepares it’s space for the map. Here’s the html’s job. (Handlebars)

Selection_314

The extra div for the maps are loaded when intimated about the map.

Selection_316

When the mapType is set to true the drawMap method is called which initializes the id for the the html and paints the map tile to it. The object has attributes like maxZoom levels, Map marker and it’s tooltip on the map. And that’s how the map tiles are formed.

 

 

 

Susi supports Map tiles

Publicising your Slack Bot through Slack Apps + the Add to Slack button

In my previous blog posts on Slack bots, I spoke about making bots, both using a simple script, and using incoming webhooks. We are now well versed with how to code up a slack bot and code it according to our needs.

Now that our Slack bot is made. How do we get it through to everyone?

Slack offers an amazing feature for the same, the Add to Slack button. Using this button, other teams can simply add your bot to them, and this is one of the best ways of publicising, because this button can be kept anywhere: your website, your README.md on github, a blog post etc.

Add to Slack, however works with OAuth, so that only authorised teams can take in your bot. Also, for distributing your bot, you will have to package it into a Slack app. So, let’s get started!

First off, we’ll make a Slack app:

1. Login into your team on Slack.
2. Go to the Apps page here and click on “Create an App”.

3. Fill out the relevant details for your app. You should especially fill out the redirect_uri (it’s not compulsory but you’ll have to fill it sometime) since it is needed for OAuth (when other users use your bot). Once form filled, click on Add App.

4. Go to the main page of your app, and under Bot Integrations, add your bot (keep the name as @susi, or whatever you like. You’ll have to change the bot name then in the code).

5. Go to App Credentials, and save the client_id and the client_secret for reference. We need it for OAuth.

Don’t worry, we’ll handle the redirect_uri in a short while!

So the flow of this goes as follows:

1. When a team clicks on “Add to Slack” button, they are led to a page, where they have to verify that a bot is being added to their team. Once they have verified, you click on “Authorize”.

2. When one clicks on Authorize, Slack generates a code, and appends it to the redirect_uri as a GET parameter, and leads you there.

3. Your redirect_uri needs to handle this code, and then send the client_secret, client_id and this code as GET parameters to http://slack.com/api/oauth.access, so that your OAuth request is verified.

4. Once request is verified and the parameters match, the bot is successfully deployed onto your team. Additionally, a JSON is returned, specifying your access_token for the bot you just deployed, as well as the incoming webhook URL (incase you have incoming webhook in your code). You need to now use this very access token and the webhook URL to control your bot.

Let’s get started on implementing this then.

1. We first go to the Slack Button page. In the bottom of the page under the section “Add to Slack Button”, there is a box, where there’s a custom url so that you can add the bot to your website etc (where people will click on it). There are three checkboxes there as you can see. Check whichever one you need for your bot:

Screen Shot 2016-08-29 at 6.37.26 PM

2. Once you have selected this, you can embed this into your website / README file. That’s half the job done!

Now let’s dive into the code. We need to take in the code that’s sent as a GET parameter to our redirect_uri. Once we get this code, we need to send in a GET request to http://slack.com/api/oauth.access with the client_id, client_secret and this code. If the bot is approved, we take up the webhook url / bot token and use it for the deployed bot so that it runs properly.

Here, the redirect_uri I’ll use is the Slack deployment URL I have on Heroku (http://asksusisunode.herokuapp.com). I’ll just create a path on Express, named ‘/slackbot’, and get started from there. The entire process starts when you get the code as a GET parameter on the redirect_uri. So do the following:

1. Go to your Apps page on Slack, under App credentials, add http://yourherokuurl.com/slackbot (or obviously any other URL you have) as the redirect_uri. I used http://asksusisunode.herokuapp.com/slackbot as the redirect_uri.

2. Let’s dive into the code now. Below is the final code that handles the Add to Slack button:


'use strict';
/* global require, process, console */

var express = require('express');
var bodyParser = require('body-parser');
var request = require('request');
var SlackBot = require('slackbots');
var http = require("http");
var Slack = require('node-slackr')
var app = express();
var custom_slack_token;
var slack_token = process.env.SLACK_TOKEN
var payload;
var payload_url = process.env.PAYLOAD_URL //this is just the webhook URL
var custom_payload_url;
var slack; 

var slack_code;
var client_id = process.env.CLIENT_ID;
var client_secret = process.env.CLIENT_SECRET;

app.set('port', (process.env.PORT || 5000));

app.use(bodyParser.urlencoded({extended: false}));

app.use(bodyParser.json());

app.get('/', function (req, res) {
	res.send('Susi says Hello.');
});

app.get('/slackbot', function(req, res) {
	slack_code = req.param('code'); //getting the code GET parameter
	var queryurl = 'http://slack.com/api/oauth.access?client_id='+client_id+'&client_secret='+client_secret+'&code='+slack_code;
	console.log(queryurl);
	request(queryurl, {json:true}, function(error, response, body) { // we get a JSON response
		if(!error && response.statusCode == 200 && body.ok == 'true'){ //i.e if bot has been installed

//take in the slack token and webhook url
			custom_slack_token = body.bot.bot_access_token;
			custom_payload_url = body.incoming_webhook.url;
			console.log(body);
			console.log(slack_token);
			res.send('Susi has been installed to your team!');
		} else{
			res.send('Could not install');
		}
	});
});

function slackbot(){
	
	setInterval(function() {
		http.get("http://asksusisunode.herokuapp.com");
	}, 1800000); 

	if (custom_slack_token && custom_payload_url){
		slack_token = custom_slack_token;
		payload_url = custom_payload_url;
	}
	var slack_bot = new SlackBot({
		token: slack_token, 
		name: 'susi'
	})

	slack = new Slack(payload_url);

	slack_bot.on('message', function(data){
		var slackdata = data;
		var msg, channel, output, user;
		if(Object.keys(slackdata).length > 0){
			if('text' in slackdata && slackdata['username'] != 'susi'){
				msg = data['text'];
				channel = data['channel']
			}
			else {
				msg = null;
				channel = null;
			}
		}
		if(msg != null && channel !=null){
			var botid = '<@U1UK6DANT>' //need to change
			if (msg.split(" ")[0] != botid){
			//do nothing
		} else{
			var apiurl = 'http://loklak.org/api/susi.json?q=' + msg;
			var payload;
			request(apiurl, function (error, response, body) {
				if (!error && response.statusCode === 200) {
					var data = JSON.parse(body);
					if(data.answers[0].actions.length == 1){
						var susiresponse = data.answers[0].actions[0].expression;
						payload = {
							text: susiresponse,
							channel: channel
						}
						slack.notify(payload)

					} else if(data.answers[0].actions.length == 2 && data.answers[0].actions[1].type == "table"){
						payload = {
							text: data.answers[0].actions[0].expression + " (" + data.answers[0].data.length + " results)",
							channel: channel
						}
						slack.notify(payload)
						for(var i = 0; i < data.answers[0].data.length; ++i){
							var response = data.answers[0].data[i];
							var ansstring = "";
							for(var resp in response){
								ansstring += (resp + ": " + response[resp] + ", ");
							}
							payload = {
								text: ansstring,
								channel: channel
							}
							slack.notify(payload);
						}
					}
				}
			});
		}
	}
});
}

// Getting Susi up and running.
app.listen(app.get('port'), function() {
	console.log('running on port', app.get('port'));
	slackbot();
});

There’s just one small shortcoming: the bot id used above won’t be the same for the deployed bot, so there can be cases where you message the bot but it does not reply. So we need to actually use the RTM API to figure out the bot id directly. We’re in the process of fixing it. But the bot will definitely be installed into your team, just that in some cases it won’t message and will stay “Away”.

See? It was as simple as adding another path in Express, and the awesome Requests package does the rest. Your bot will successfully be added to your team as a result, and anyone can use it. 🙂

Apart from publicising using the Add to Slack button, additionally, you can also publicise your app on the Slack Apps directory by going here and filling out the form.

So now we know how to make a Slack bot from scratch, from two different methods, and how to effectively publicise it. This is another great way by which Susi will be publicised to everyone and more people can use it. Amazing, right?

By the way, please go to https://github.com/fossasia/asksusi_messengers and add more bots for Susi there. We wish to add Susi on as many platforms as possible. We really value your contributions 🙂

So that’s it for today! Feedback is welcome as always 🙂 See you later!

Publicising your Slack Bot through Slack Apps + the Add to Slack button

Making Slack Chatbots using Incoming Webhooks + The Idling problem

The last time I spoke about Chatbots, I spoke about the need of increasing Susi’s reach, how Slack is a great platform because of how it works within teams, and how to make a Slack bot yourself.

However, if you see the code snippet I posted in that blog post, you’ll see that the Slack bot I have is just a Python script, while the rest of the index.js code (which contains the Messenger and Telegram bots) is an Express application. We are basically just using a package (slackbots, if you remember), and it simply takes in your Slack token and POSTs to the Slack interface. Also, that is a custom bot, it will only be in use for us right now, we need to distribute it (which we do using Slack apps, we’ll talk about that later).

Today, I’ll be describing another method of making Slackbots: using Incoming Webhooks.

Incoming Webhook is a way by which you don’t directly POST to the Slack interface, but you POST to a webhook generated by Slack. It is a very convenient way of posting messages from external sources into Slack. Moreover, when you distribute your Slack bot, you can distribute your Webhook separately so that your reach can increase more (we’ll talk about distributions and OAuth in the next blog post). Incoming webhooks are seamlessly integrated within your Slack apps, so that your Slack bot can be distributed efficiently.

So let’s get started. To create an Incoming webhook integration:

1. Go to the Incoming Webhook Integration page here.

2. Fill in the details and select the channel you wish to post to.

3. Save the webhook URL for reference. We’ll need it.

Incoming Webhooks work with a payload. A payload is a JSON which contains all the information of the message (text, emojis, files etc). A normal payload looks like:

payload={"text":"This is a line of text.\nAnd this is another one."}

Now all we need to do is POST our message, AS a payload, to this URL, instead of directly posting to Slack. For easily handling payloads, we use a library named node-slackr. You can install it as follows:

npm install --save node-slackr

To post a payload to the URL, we first instantiate the node-slackr object using our webhook URL:

var slack = new Slack(webhook_url);

When we have the payload ready, all we need to POST to the webhook is simply do:

slack.notify(payload);

So here’s the final modified code that’s used for making bots using incoming webhooks. We just make a few changes to our original bot code in the last post on Slack bots on this blog:


'use strict';
/* global require, process, console */

var express = require('express');
var bodyParser = require('body-parser');
var request = require('request');
var SlackBot = require('slackbots');
var Slack = require('node-slackr')
var app = express();
var slack_token = process.env.SLACK_TOKEN
var webhook_url = process.env.WEBHOOK_URL
var heroku_url = process.env.HEROKU_URL
var slack; 

app.set('port', (process.env.PORT || 5000));

app.use(bodyParser.urlencoded({extended: false}));

app.use(bodyParser.json());

app.get('/', function (req, res) {
	res.send('Susi says Hello.');
});

function slackbot(){
	
	setInterval(function() {
		http.get(heroku_url);
	}, 1800000); 

	var slack_bot = new SlackBot({
		token: slack_token, 
		name: 'susi'
	})

	slack = new Slack(payload_url);

	slack_bot.on('message', function(data){
		var slackdata = data;
		var msg, channel, output, user;
		if(Object.keys(slackdata).length > 0){
			if('text' in slackdata && slackdata['username'] != 'susi'){
				msg = data['text'];
				channel = data['channel']
			}
			else {
				msg = null;
				channel = null;
			}
		}
		if(msg != null && channel !=null){
			var botid = ':' 
			if (msg.split(" ")[0] != botid){
			//do nothing
		} else{
			var apiurl = 'http://loklak.org/api/susi.json?q=' + msg;
			var payload;
			request(apiurl, function (error, response, body) {
				if (!error && response.statusCode === 200) {
					var data = JSON.parse(body);
					if(data.answers[0].actions.length == 1){
						var susiresponse = data.answers[0].actions[0].expression;
						payload = {
							text: susiresponse,
							channel: channel
						}
						slack.notify(payload)

					} else if(data.answers[0].actions.length == 2 && data.answers[0].actions[1].type == "table"){
						payload = {
							text: data.answers[0].actions[0].expression + " (" + data.answers[0].data.length + " results)",
							channel: channel
						}
						slack.notify(payload)
						for(var i = 0; i < data.answers[0].data.length; ++i){
							var response = data.answers[0].data[i];
							var ansstring = "";
							for(var resp in response){
								ansstring += (resp + ": " + response[resp] + ", ");
							}
							payload = {
								text: ansstring,
								channel: channel
							}
							slack.notify(payload);
						}
					}
				}
			});
		}
	}
});
}

// Getting Susi up and running.
app.listen(app.get('port'), function() {
	console.log('running on port', app.get('port'));
	slackbot();
});

All we did is set the webhook url as an environment variable, and used that, and just did slack.notify. Also, I encapsulated the function inside app.listen so that it runs up as soon as the app starts and stays alive.

But here comes another problem: We used heroku dynos for deployment. Heroku dynos have a sleep period of 6 hours. In those 6 hours, the bot would just be idle and would not work. We wish to circumvent this.

There are three ways of doing so. One way is to install the newrelic plugin of Heroku and using it (you can read more about it here). The second way is to simply use Kaffeine so that your heroku url is pinged every 30 minutes and the bot stays alive.

Or you can programatically solve it as well. Look at the code snippet above and notice:


setInterval(function() {
		http.get(heroku_url);
	}, 1800000); 

We’re basically pinging the Heroku URL (again stored as env var) every 1800000 milliseconds, i.e 30 minutes. This is a more convenient approach to solve this problem of idling too.

So now we know how to make our bot using two different methods, and how to solve the idling problem. To get this full circle, in the next blog post, I will talk about distribution of your bot, and how people can know about it. Feedback is welcome as always 🙂

Making Slack Chatbots using Incoming Webhooks + The Idling problem