Loklak Server just a click away – One Click Deployments

As a person with an Android background working on the loklak project for the first time last month, I really did have a lot to pick up, and somehow I had to get started off.

I checked out the loklak docs which listed the process of deploying the server on your local machine or on Heroku / DigitalOcean / AWS, and all of those processes were lengthy and needed a bit of time to get set up, and besides I had never worked with Heroku before so I had to read up how it worked. There was no other feature to speed up the deployment process at that time, and that is when we decided to add a new feature: One Click Deploy!

One Click Deploy is the easiest, most convenient method to start up your own loklak server. Instead of going through the long drawn setting up through the terminal, you can simply deploy your server and get loklak running with a single click. We extended our list of supported platforms, and today, loklak server can be started onto Heroku, Scalingo, Docker Cloud and Bluemix, just with a single click. And all you need to do? Give your deployed app a name. That’s it. Quite literally.

Screen Shot 2016-06-01 at 12.08.47 AM

Integrating One Click Deploy into loklak, at first, wasn’t too big of a task. For Heroku, I just created an app.json file and linked it to the custom buildpack we had here. For Scalingo, I similarly had to create a scalingo.json, however I had issues as they didn’t have a JDK for Heroku Buildpacks, so I contacted them and they mended their own codebase to accommodate Heroku, and it finally worked. The Bluemix deployment was a bit more tedious: we had to make a manifest.yml and a pipeline.yml file, and then added a Bluemix start shell script and a Bluemix buildpack to make it work.

However, we did end up facing a lot of challenges after a period of time. After some days, our deployed servers started crashing. I communicated with the Scalingo team over email and they told us that our servers used an Xmx (maximum memory) of 600 MB while Scalingo containers worked with 512 MB, thus increasing swapping and reducing performance hereby crashing the applications. This was sorted by keeping an environment variable $ENVXmx which set the Xmx level based on the environment.

Finally, we decided to update our existing codebase to support Java 8, and as a result, we modified our buildpacks (Bluemix and Heroku) to accommodate for this change. This took a couple of days but finally it was done.

So at the end of it all, loklak finally has a much needed feature with which you can experience it in a matter of seconds, and a way with which you can set up multiple servers in just a few minutes, to multiple platforms. We have plans for having a deployment to Azure as well, and will plan on implementing it in the days to come 🙂

Loklak Server just a click away – One Click Deployments

Know who tweeted the most!

It is obvious that all of us are curious to find out who is leading! My app, which is called ‘Twitter Leaderboard uses the search API provided by ‘Loklak’. Using my app you can collect the top five tweeters whose tweets contain the mentioned hashtag.

So here is a simple explanation of how ‘Twitter Leaderboard’ works. I built this app using AngularJS and the search API provided by Loklak. Primarily, I collected the tweets’ data and parse it for the required fields.

The leader board needed to have the tweeter’s name along with their tweet count. Also ‘Twitter Leaderboard’ needed to allow its users to search for a tweet based on the hashtag supplied to it in a query. 

I wrote a small controller to make a HTTP request to the Loklak search API, and parse the retrieved object accordingly.  I parsed the required data in the following manner:

  • data.statuses //The status object containing the requested hashtag.
  • user = statuses[i].screen_name // Get all the screen names.
  • counts = {} //Counts object to store the number of tweets count for each name.

After parsing for the required data from the search results, we have to sort the names according to the tweets count.  And here we go the results are pushed to the front end and displayed.

Selection_058

Another way to improve the search result is by using the Search Aggregations. This feature helps you to get the count results in a better by scaling up the search result. It will help you to count up all the tweets which are indexed(More than a billion tweets). This feature will not consider remote search results. Therefore the remote search is switched off by “source=cache” and the search results are switched off with “count=0”. The request for aggregation is simply hidden in “fields=…” which names the fields where you want to have an aggregation. You can even reduce the result size by restricting your search to certain period of time.

The upgrade to the above app will be provided soon using the Aggregations.

Know who tweeted the most!

Bubble & Bar charts for loklak

I have made 2 visualisations in the front-end apps section using D3.js and AngularJS, that anyone can use on loklak.org or locally.

The first type is a bubble chart, these are great for representing single values, such as the number of mentions or the frequency of certain words.

As you can see, when I search for the term loklak, one can see the number of times a user was mentioned and the relative frequency of words.Screenshot 2016-05-30 10.55.26

First, I had to solve the problem of getting the data. This can be done through the data service used for the tweet feed previously. This gives us an updated array of tweet objects in the local storage of the browser.

Next, I had to analyse the tweets. True to its name, D3.js handles a wide variety of data formats, so storing term-frequency pairs in an object is sufficient. Do take a look at the code if you are interested. The processed data is then stored in a separate array in local storage, for eg $storage.mentionFreq instead of $storage.tweets.

Next, I had to display the processed data. Thankfully, angularJS plays nicely with d3.js, so to create a directive in angular, I could use similar D3.js code in the $link function of the directive, as I was manipulating DOM elements with D3.js.

Finally, to make the chart update itself without a page refresh, I added a $watch function on the processed data in the local storage, which runs an update function. Within it, D3.js then uses array joins to render the new data with .data(), so it seamlessly updates itself. Do take a look at the code for the directives for a more in-depth explanation.

The beauty of directives is that they are components which are reusable, the above example is actually using the same directive but with different attributes passed in.

[code language=”css”]
<div ng-controller=’bubbleCloudCtrl’ >
<bubble-cloud flex data="$storage.mentionFreq" min ="1" title="Most Mentions"></bubble-cloud>
<bubble-cloud flex data="$storage.wordFreq" min="3" title="Word Frequency"></bubble-cloud>
</div>
[/code]

The same pattern can be applied to other D3.js charts, below we can see a stacked bar chart example using the general pattern as described above:

Screenshot 2016-05-30 10.55.19.png

The difference is in analaytics and directive code, which is a bit more involved, as I had to process by date, and then by quantity.

Hopefully this helps future developers to make their own, some ideas I have are making radar and force directed graph.

Bubble & Bar charts for loklak

A tweet feed for loklak.org

Loklak.org has a page for front-end apps that showcase the uses of loklak’s API, and in the first week I’ve tried to warm up for GSOC by making a ‘realtime’ twitter feed, as the existing front-end apps were mainly single http get requests and didn’t seem very appealing to me, and also to prepare for my front-end app data visualisations GSOC project.

To obtain updated data, the app polls the loklak API on a regular interval. This is done rather simply by using $interval in AngularJS, which is just a wrapper around window.setInterval(). For the realtime feed, I set the minimum rate for querying at an arbitrary 1min so as not to exceed the limits.

The next thing I needed to solve was where to store the data, as I didn’t have a database to work with, local storage was the only option. A useful module for this is “ngStorage“. At a maximum of 5MB it isn’t much, but decent enough to store a few thousand tweets and showcase the features of loklak. If developers wanted to create an external app for loklak, they could also use the front-end apps if they were well modularized.

Lastly, to display in a fancy pinterest-like grid, I’ve decided to use an angular module called “angulargrid“. To achieve the infinite scroll effect, I created another scope variable, to take a subset of the data in local storage. AngularJS then helps with rendering changes in the data model to the feed, by using $watch() on local storage.

Do checkout the tweetfeed:

tweetfeed

Some areas to improve:

  • Refactor services to make it more modular
  • Improve the regex to account for emojis and other tags
A tweet feed for loklak.org