Creating a Loklak App!

Hello everyone!

Today I will be shifting from course a bit. I will talk about how you can use Loklak API, create an app and show it off by deploying it on loklak.org .

Lets dive in!

Selection_039

This is how the loklak apps page looks like. If you want your creation to end up here, follow some simple steps.

Firstly clone loklak_server github repo.

git clone https://github.com/loklak/loklak_server.git

Deploy loklak server on localhost and you’ve got your “dev env” ready!

The next thing you need is an idea for an app which will use loklak API. When you have it, create a subdirectory under your loklak_server/html/apps/ folder and name your app. Now you start coding!! For easy quick-start, you can look at the boilerplate.

If you are planning to add some styles and scripts to your app, these must either be already present in loklak or you might have to add them explicitly. e.g. look at this app and notice the css and js folders which contain the custom scripts being used in the app.

Selection_041

index.html is the landing page for your app. The app.json file you create must have an json-Id format. You can use the json-Id file from an existing app for reference.

When you are sure about which Loklak API functions you want to use in your app, modify the "permissions" field in app.json to a comma-separated list of all api paths, that the app calls. This is used to apply authorization markers to the app, so it becomes visible if the app is actually usable for the user. e.g. look at this. Since this app just uses the search API to get the tweet feed. The permissions a user requires to use this app is    "/api/search.json" .

Selection_040
Social-Wall

Once you are done with coding your app, test it thoroughly on your local installation. See if your app appears under http://localhost:9000/api/apps.json in the app list. Test that your app is not browser specific and make sure to add a backlink to /apps/ to make it easier for user to browse to other apps from yours.

Since now you are sure that your app is “Loklak ready” all you have to do is send a pull request to https://github.com/loklak/loklak_server. Make sure that all your files are contained into one commit.

When your app gets merged, you would be able to see it here.

Thats all folks. For more info refer to the full documentation here.
Happy Coding!!!

Creating a Loklak App!

Loklak Walls Manual Moderation – tweet storage

Loklak walls are going to get a new feature called manual moderation! You may have seen it in the coming soon section of the content tab when you were creating a wall.

For the walls application we have decided to use Mongodb so we can query an external API like loklak_server, then store tweets in the Mongodb database. So the first question we have to think about before storing it is how to model it in our Mongoose schemas. To answer that we have to consider the access patterns of our web app, as it is critical in deciding how we model the collection in mongodb – whether we should embed the data in an existing collection, keep all tweets for a wall in an array in one document in the collection, or store each tweet as a a document itself and query them.

The main issue is that the number of tweets could keep growing, so we can’t embed nor store them in an array in a document. Mongodb has a document size limit of 16MB (for the unfamilar, a document is to a row as collection is to a table), so we could very realistically hit that if we stored all tweets in an array.

Another issue is the frequency of access, and updates of the tweet data. If we embedded the data, we could hit the size limit and it would not make sense to keep retrieving extra information that would not be needed if we only wanted the tweets array. Also to update individual tweets we would have to save the whole array, for just one tweet!

Thus in our case I have chosen the last option, and created a separate mongodb collection to store the tweets individually as I have modeled them as a one to many, (1 walls – many tweets) relationship. Although it could also be seen as a many to many relationship between walls and tweets, to control the approval status for each requires us to treat each tweet as separate from each other as one could be approved but another rejected. So in the schema I have userWallId used for querying all tweets belonging to a certain wall of a certain user, notice the index:true option which helps speed up the query by indexing the userWallId field.

[code language=”javascript”]

var TweetSchema = new Schema({
userWallId: { type: String, required: true, index: true },
approval: Boolean,
timestamp: String,
created_at: String,
,…
})

[/code]

In contrast I have chosen to embed the wall options themselves into the user’s own collection, as this is a “one-few” relation, and we can denormalize it by embedding the wall options within an array in the user object. Denormalizing as such in Mongodb allows us to query it faster as we do not have to perform another lookup for the wall options if it were in another collection.

[code language=”javascript”]
var UserSchema = new Schema({
name: { type: String, required: true },
local: {..},
isVerified: { type: Boolean, required: true },
twitter: {..},
apps: {
wall: [{
profanity: Boolean,
images: Boolean,
//…other wall options
}]
}
})
[/code]
To store the tweets we will then have to create a route to handle post requests:

[code language=”javascript”]
router.post (‘/tweets/:userId/:wallId’, auth, function (req, res) {
req.body.forEach(function(tweet){
var newTweet = new Tweet(tweet);
newTweet.save(function(err,result){

});
})
res.jsonp({message: "inserted "+ req.body.length});
});
[/code]

And in the front end angular app we can use these routes to store our new tweets with $http.post:

[code language=”javascript”]

var posturl= ‘/api/tweets/’+ $rootScope.root.currentUser._id + ‘/’ + $scope.userWalls[$scope.userWalls.length – 1].id;

SearchService.initData(searchParams).then(function(data) {

// get tweets array from loklak_server and set all tweets to default false for approval status

data.statuses.map(function(tweet){
tweet.userWallId = userWallId;
tweet.approval = false;
})

$http.post(posturl, data.statuses)

[/code]

The end result is that we can do a get request at the moderation tab page for that wall’s tweets and display it as such:

Screenshot 2016-06-18 07.02.04.png

Stay tuned for more updates on this new feature of loklak walls!

Loklak Walls Manual Moderation – tweet storage

Under the hood: Authentication (login)

In the second post of ‘Under the hood’, I’ll explain how our current login system works and how you can use it. This is mainly interesting if you want to know how to get API-access to recourses which need a login in the future.

AbstractAPIHandler

First a bit about the overall architecture.

Loklak uses an embedded Jetty HTTP server. It works fundamentaly different than for example, an Apache server with PHP.

Like in Apache, you can have a folder with static content, like hmtl, css and js files, which just get displayed. But for dynamic output, you don’t have just java-class-files in the same folder, which get interpretated on the fly like php-files do.

Instead we register so called servlets on the server before we start it. This servlets are java classes which get a http-request as input and send a response at the end. All classes in the api-packages are such servlets.

So the idea is to create a so called AAA-system:

  • Authentication (who are you?)
  • Authorization (what are you allowed to do?)
  • Accounting (what/how much did you do already?)

As of now, only the authentication part is ready to use, authorization will follow soon (hopefully early next week)

To make it easy for the servlets have access to that system, Michael created an abstract class called AbstactAPIHandler. This class is meant to contain all common code the servlets need, which in turn extend this class to implement the actual functionality.

So what does it offer so far? Well, as of the authentication, there’s alot of different use cases how we want to be able to authenticate a user:

  1. Login via a user id and a password
  2. For browsers: sessions and cookies, so the user stays logged in over multiple requests
  3. For api-access and links: access tokens
  4. Login via a key-pair (not yet implemented)

1., to 3. are implemented and ready to use. Here’s how:

User id and password

Just add

[email protected]&password=123456

to a request to any sevlet extending AbstractAPIHandler. This will log you in as the user [email protected]main.com.

As we want to use this for browser logins and logins via other tools (for example curl), this does not create a session.  A session means the server remembers you over multiple calls. We don’t want that for pure API calls, as APIs are better stateless, that means all calls are independent from each other.

Sessions and Cookies

If you use a browser, you probably want to do the login only once. Afterwards, your browser and the server should remember each other.

There are two different ways to archive that:

  • Sessions, which are short-lived and forgotten as soon as either the browser or the server shuts down
  • Cookies, which can live for a longer time and survice restarts (in the moment, server restarts also remove them, but this will change in the future)

Technically, these are very similar, but a bit different in how we have to use them.

Anyhow, you can request a session or a cookie with:

request_session=true

or

request_cookie=true

respectively. Cookie currently expire after one week if not used.

Access-tokens

In some use-cases you might want to log in without appending you password each time. For example you want to create link which logs a user in (this is done with the email validation link).

Or you want stateles API-access via a public-private key login (which is somewhat slow).

Then it’s useful to create a access-token. Currently you can do so by calling handshake-client.json. An example:

domain.com/api/[email protected]&password=123456&valid_seconds=3600

This returns a access-token for the user [email protected] with a livetime of one hour. You can then use that token to make further queries:

domain.com/api/hello.json?access_token=YYqs62uSqSYfZwufdZN1rPLbqGTyaA

 

That’s it for the login, but I’ll add more as soon as public-private-key login is ready.

Under the hood: Authentication (login)

Loklak gives some hackernews now !

It’s been befittingly said  “Well, news is anything that’s interesting, that relates to what’s happening in the world, what’s happening in areas of the culture that would be of interest to your audience.” by Kurt Loder, the famous American Journalist.

And what better than Hackernews : news.ycombinator.com for the tech community. It helps community by showing the important and latest buzz and sort them by popularity and their links.

Screenshot from 2016-06-17 08:01:42

LOKLAK next tried to include this important piece of information in its server by collecting data from this source. Instead of the usual scraping of HTML Pages we had been doing for other sources before, we have tried to read the RSS stream instead.

Simply put, RSS (Really Simple Syndication) uses a family of standard web feed formats to publish frequently updated information: blog entries, news headlines, audio, video. A standard XML file format ensures compatibility with many different machines/programs. RSS feeds also benefit users who want to receive timely updates from favorite websites or to aggregate data from many sites without signing-in and all.

Hackernews RSS Feed can be fetched via the URL https://news.ycombinator.com/rss and looks something like…

Screenshot from 2016-06-17 09:33:32

In order to keep things simple, I decided to use the ROME Framework to make a RSS Reader for Hackernews for Loklak.

Just for a quick introduction, ROME is a Java framework for RSS and Atom feeds. It’s open source and licensed under the Apache 2.0 license. ROME includes a set of parsers and generators for the various flavors of syndication feeds, as well as converters to convert from one format to another. The parsers can give you back Java objects that are either specific for the format you want to work with, or a generic normalized SyndFeed class that lets you work on with the data without bothering about the incoming or outgoing feed type.

So, I made a function hackernewsRSSReader which basically returns us a JSONObject of JSONArray “Hackernews RSS Feed[]” having JSONObjects each of which represents a ‘news headline’ from the source.

The structure of the JSONObject result obtained is something like:

{
   "Hackernews RSS Feed":[
      {
         "Description":"SyndContentImpl.value=....",
         "Updated-Date":"null",
         "Link":"http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.116.241103",
         "RSS Feed":"https://news.ycombinator.com/rss",
         "Published-Date":"Wed Jun 15 13:30:33 EDT 2016",
         "Hash-Code":"1365366114",
         "Title":"Second Gravitational Wave Detected at LIGO",
         "URI":"http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.116.241103"
      },
     ......
      {
         "Description":"SyndContentImpl.value=....",
         "Updated-Date":"null",
         "Link":"http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-410-principles-of-autonomy-and-decision-making-fall-2010/lecture-notes/MIT16_410F10_lec20.pdf",
         "RSS Feed":"https://news.ycombinator.com/rss",
         "Published-Date":"Wed Jun 15 08:37:36 EDT 2016",
         "Hash-Code":"1649214835",
         "Title":"Intro to Hidden Markov Models (2010) [pdf]",
         "URI":"http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-410-principles-of-autonomy-and-decision-making-fall-2010/lecture-notes/MIT16_410F10_lec20.pdf"
      }
   ]
}

It includes information like Title, Link, HashCode, Published Date, Updated Date, URI and the Description of each “news headline”.

The next step after extracting information is to write it to the back-end and then retrieve it whenever required and display it in the desired format as suitable to the Loklak Web Client after parsing it.

It requires JDOM and ROME jars to be configured into the build path before proceeding with implementation of the RSS Reader.

A look through the code for the HackernewsRSSReader.java :

/**
 *  Hacker News RSS Reader
 *  By Jigyasa Grover, @jig08
 **/

package org.loklak.harvester;

import java.net.MalformedURLException;
import java.net.URL;
import java.util.List;
import org.json.JSONArray;
import org.json.JSONObject;
import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.feed.synd.SyndFeed;
import com.sun.syndication.io.SyndFeedInput;
import com.sun.syndication.io.XmlReader;

public class HackernewsRSSReader {	
	
	/*
	 * For HackernewsRSS, simply pass URL: https://news.ycombinator.com/rss 
	 * in the function to obtain a corresponding JSON
	 */
	@SuppressWarnings({ "unchecked", "static-access" })
	public static JSONObject hackernewsRSSReader(String url){
		 
	        URL feedUrl = null;
			try {
				feedUrl = new URL(url);
			} catch (MalformedURLException e) {
				e.printStackTrace();
			}
	        
	        SyndFeedInput input = new SyndFeedInput();
	        
	        SyndFeed feed = null;
			try {
				feed = input.build(new XmlReader(feedUrl));
			} catch (Exception e) {
				e.printStackTrace();
			}
	        
	        String[][] result = new String[100][7];
	        //result[][0] = Title
	        //result[][1] = Link
	        //result[][2] = URI
	        //result[][3] = Hash Code
	        //result[][4] = PublishedDate
	        //result[][5] = Updated Date
	        //result[][6] = Description
	        
	        @SuppressWarnings("unused")
			int totalEntries = 0;
	        int i = 0;
	        
	        JSONArray jsonArray = new JSONArray();
	        
	        for (SyndEntry entry : (List)feed.getEntries()) {
	        	
	        	result[i][0] = entry.getTitle().toString();
	        	result[i][1] = entry.getLink().toString();
	        	result[i][2] = entry.getUri().toString();
	        	result[i][3] = Integer.toString(entry.hashCode()); 
	        	result[i][4] = entry.getPublishedDate().toString();
	        	result[i][5] = ( (entry.getUpdatedDate() == null) ? ("null") : (entry.getUpdatedDate().toString()) );
	        	result[i][6] = entry.getDescription().toString();
	        	
		        JSONObject jsonObject = new JSONObject();

	        	jsonObject.put("RSS Feed", url);
	        	jsonObject.put("Title", result[i][0]);
	        	jsonObject.put("Link", result[i][1]);
	        	jsonObject.put("URI", result[i][2]);
	        	jsonObject.put("Hash-Code", result[i][3]);
	        	jsonObject.put("Published-Date", result[i][4]);
	        	jsonObject.put("Updated-Date", result[i][5]);
	        	jsonObject.put("Description", result[i][6]);
	        	
	        	jsonArray.put(i, jsonObject);
	        	
	        	i++;
	        }
	        
	        totalEntries = i;
	        
	    JSONObject rssFeed = new JSONObject();
	    rssFeed.put("Hackernews RSS Feed", jsonArray);
	    System.out.println(rssFeed);
		return rssFeed;
		
	}

}

 

Feel free to ask questions regarding the above code snippet.

Also, Stay tuned for more posts on data crawling and parsing for Loklak.

Feedback and Suggestions welcome 🙂

Loklak gives some hackernews now !

Does tweets have emotions?

Tweets do intend some kind of emotion when they are tweeted. It can be happiness, sad, anticipation etc. Loklak does such classification and gives you the categorization of the tweet along with the probability of it being correct.

Selection_116

The sentiment analysis is one such tool which uses the classification provided by the Loklak Search API from the field called “classifier_emotion”. It displays the other metrics like the languages “classifier_language” in which the tweet is being tweeted along with the emotions. These results are visualized with the help of flash cards along with pie charts. Selection_117Selection_118

This has a clean display of the results which shows the recent 10 tweets being tweeted using the hashtag mentioned. One should simply enter the hashtag in to the search box for the results. The following image shows part of the search API results(JSONObject).

Selection_119

I used the above field names”classifier_emotion”, “classifier_language” to display the results. I have used Highcharts to display piecharts.

The internal churning and analysis of the data is happening in the following folder structure [org->loklak->tools->bayes]. Where an abstract Classifier class is being implemented in which the Bayes classifier implements a naive Bayes approach to classifying a given set of features:

classify(feat1,…,featN) = argmax(P(cat)*PROD(P(featI|cat)

Does tweets have emotions?

Dockerize the loklak server and publish docker images to IBM Containers on Bluemix Cloud

Docker is an open source platform which makes life easy for system developers and sysadmins to build, ship and run distributed applications. Unlike Virtual Machines (VMs) Docker containers allows you to package an application with all of its dependies into a standardized unit for software development.

docker_container_logo

To create an image of the loklak server, the first thing you need is to run the docker daemon process. This then boots up a blank docker instance (called a `default` image) such that the images could be stored. To run loklak in a container, we first need to create a new image for loklak server. This includes the base operating system, the required updates for that operating system as well as the build instructions for running the application. To run the loklak server we need to clone the loklak_server from `loklak/loklak_server` and then build and compile using `ant`.

Once the docker daemon’s default image has booted, Create an new directory called `Docker` and `cd` into that directory. Inside that create a directory `loklak` so that you could store the `loklak` image that you would create. To do this you need to create a Dockerfile mentioning the operating system and the different instructions that need to be executed.

FROM ubuntu:latest
MAINTAINER Ansgar Schmidt <[email protected]>
ENV DEBIAN_FRONTEND noninteractive

# update
RUN apt-get update
RUN apt-get upgrade -y

# add packages
RUN apt-get install -y git ant openjdk-8-jdk

# clone the github repo
RUN git clone https://github.com/loklak/loklak_server.git
WORKDIR loklak_server

# compile
RUN ant

# Expose the web interface ports
EXPOSE 80 443

# change config file
RUN sed -i.bak 's/^(port.http=).*/180/'                conf/config.properties
RUN sed -i.bak 's/^(port.https=).*/1443/'              conf/config.properties
RUN sed -i.bak 's/^(upgradeInterval=).*/186400000000/' conf/config.properties

# hack until loklak support no-daemon
RUN echo "while true; do sleep 10;done" >> bin/start.sh

# start loklak
CMD ["bin/start.sh"]

This fetches the latest ubuntu image and provides a non-interactive Debian front end. Docker files are executed step by step. So in the above docker file for loklak, the machine image is first pulled over and then the operting system image us updated and the distribution upgraded. Once this is done the dockerfile then runs `RUN apt-get install -y git ant openjdk-8-jdk` to install ant, git and java8 JDK. Once this is complete, the repository for the loklak server is closed and the container opens ports 80 and 443 to run it on HTTP and HTTPS port. The `sed` commands find and replace the settings for the HTTP port and HTTPS port on the configuration file of the loklak server. Once this is cloned, we run `ant` and execute `bin/start.sh` to start the loklak server.

Since now we understand how the docker file runs, lets find out how the docker is actually executed.
`docker images` to see the list of images that you have.

Now create the loklak image by doing `docker build -t loklak .` which uses the `Dockerfile` in the current directory to build the image in that directory.

The command takes several seconds to run and reports its outcome. During this process, it fetches each of the image layers and puts these image layers together to create your docker container. Once it’s installed, you can run it by doing `docker run loklak`. This has successfully created your image and it’s ready to push it to dockerhub after creating a namespace and renaming the image as `useraccountname/loklak` and then pushing to docker hub.

To push these to the IBM Bluemix cloud, you need to have `cloudfoundry` command line tools `cf`. Then install the plugin for containers `cf install-plugin https://static-ice.ng.bluemix.net/ibm-containers-linux_x64`

Once this is installed, connect to the IBM Bluemix cloud under a namespace and pull the required image you pushed to github as `cf ic cpi username/loklak loklak` which copies the image you pushed to dockerhub to the IBM Containers and run the container.

Containers make life really simple for devops and to create and deploy instantly as well as scale on demand.

Dockerize the loklak server and publish docker images to IBM Containers on Bluemix Cloud

Loklak enters WordPress Universe

In my last post, I briefly commented on how one could now use Loklak API in WordPress plugins. Lets discuss that here in detail.

The post is mostly focused on novice WP plugin/widget developers. If you ain’t novice (or don’t consider yourself to be) just a quick skim would be enough.

Suppose, you want to make a plugin with functionalities as given below:

  1. View tweets by a twitter account
  2. Between a time frame
  3. Containing specific “mentions” and “hashtags”

After deciding the design, you want best tools at your disposal. You search for the twitter API and find that you need to make a twitter app to use the API in your plugin.

Now you ask yourself “Is there a better way?”.

Yes there is! Its called Loklak!

You can use Loklak’s PHP API (refer to my previous blog post) and the included WordPress library to render tweets anonymously (No annoying ‘Create an app’ procedures!).

“Oh yes! that’s what I was looking for. How do I use it?”

So lets dive in!

When you are done with your plugin’s frontend, add loklak_php_api as a submodule (if using Github) or just download the zipped file and extract it in your plugin repo.

To add loklak PHP API as a submodule to your project, just execute the below-mentioned command.

cd path/to/your/project;
git submodule add https://github.com/loklak/loklak_php_api.git; 

Include loklak.php in your plugin file where you want to render your Loklak API results.

require_once('/path/to/loklak_php_api_installation/loklak.php');

Create an object of Loklak class.

$loklak = new Loklak($base_url);

Since we want our tweets filtered based on points we listed above, we could use the search API.

See the definition below.

Selection_028

You just need to create a form to input the arguments and you will be good to go!

For example: If the plugin user wants to see tweets by @lklknt which mention #loklak and were tweeted between 31st January, 2015 and 31st December, 2015.

$query = '#loklak';
$since = '2015-01-31';
$until = '2015-12-31';
$from_user = `lklknt';

Considering that we got these values from the frontend form where user could enter his choices. This is how the search function would look like.

$tweet_results = $loklak->search($query, $since, $until, $from_user);

Selection_027

You can use these results directly in your plugin.

For plugin developers who are currently using twitter API and want to provide Loklak as an option to their users.

Just include Lib/loklak-api-admin.php in your settings page to display loklak as an option.

require_once('/path/to/loklak_php_api_installation/Lib/loklak-api-admin.php');
Loklak enters WordPress Universe

Loklak: It’s not just an application

Loklak is not just an application. It is a server based application capable of scraping messages from Twitter unofficially and other sources. This server contains a search index and a peer-to-peer index sharing interface. It is available as a local host server, API and webclient.  The loklak server can be cloned from the repo present on github and can be run on localhost using Apache ANT.

Some of the highlights of loklak are:

  • It’s architecture is very generic and can be used for any twitter like platform.
  • Loklak is anonymous, as it doesn’t store client’s IP address when a search is done (thanks to the use of scrapers).
  •  Loklak can recursively de-shorten shortened links (so as to free up some more characters).
  • One can create an own search engine for tweets.

Next coming on to the loklak API, it is very much flexible and can be easily implemented. There are existing API libraries with support in Node.js, Python, C#.Net and Ruby in order to use the loklak API. The loklak being based on http is very much ubiquitous. One can access the loklak API without any authentication, but there exist three classes for access rights which are:

  1. open: access without any restrictions for all clients from any IP
  2. limited: localhost clients are granted more data than public clients
  3. restricted: only localhost clients are granted access

The loklak APIs that are currently available are:

  • status.json [with no access restrictions]
  • search.json [with no access restrictions]
  • suggest.json [with some access restictions]
  • crawler.json [with some access restrictions]
  • hello.json [with no access restrictions]
  • geocode.json [with no access restictions]
  • peers.json [with no access restrictions]
  • proxy.json [with some access restrctions]
  • push.json [with some access restrictions]
  • push/geojson.json [with no access restrcitions]
  • push/”.” [with no access restrictions]
  • import.json [with no access restrictions]
  • validate.json [with no access restrictions]
  • settings.json [with some access restrictions]
  • account.json [with some access restrictions]
  • user.json [with no access restrictions]
  • asset [with some access restrictions]
  • threaddump.txt [with some access restrictions]
  • vis/map.png [with no access restrictions]
  • vis/markdown.png [with no access restricions]

The search results on loklak can be retrieved in two formats: JSON and RSS. 

The following is a sample search result in JSON:

{
  "readme_0" : "THIS JSON IS THE RESULT OF YOUR SEARCH QUERY - THERE IS NO WEB PAGE WHICH SHOWS THE RESULT!",
  "readme_1" : "loklak.org is the framework for a message search system, not the portal, read: http://loklak.org/about.html#notasearchportal",
  "readme_2" : "This is supposed to be the back-end of a search portal. For the api, see http://loklak.org/api.html",
  "readme_3" : "Parameters q=(query), source=(cache|backend|twitter|all), callback=p for jsonp, maximumRecords=(message count), minified=(true|false)",
  "search_metadata" : {
    "itemsPerPage" : "1",
    "count" : "1",
    "count_twitter_all" : 0,
    "count_twitter_new" : 0,
    "count_backend" : 0,
    "count_cache" : 7336,
    "hits" : 7336,
    "period" : 9223372036854775807,
    "query" : "fossasia",
    "client" : "122.175.88.159",
    "time" : 1196,
    "servicereduction" : "false",
    "scraperInfo" : "local"
  },
  "statuses" : [ {
    "timestamp" : "2016-04-21T17:33:06.959Z",
    "created_at" : "2016-04-21T15:39:39.000Z",
    "screen_name" : "notrademark",
    "text" : "@fossasia @mariobehling sure, I am happy about every pull request on GitHub. What features do you plan?",
    "link" : "https://twitter.com/notrademark/status/723174364968005632",
    "id_str" : "723174364968005632",
    "source_type" : "USER",
    "provider_type" : "REMOTE",
    "provider_hash" : "1cadbfd3",
    "retweet_count" : 0,
    "favourites_count" : 0,
    "images" : [ ],
    "images_count" : 0,
    "audio" : [ ],
    "audio_count" : 0,
    "videos" : [ ],
    "videos_count" : 0,
    "place_name" : "Planá",
    "place_id" : "",
    "place_context" : "FROM",
    "location_point" : [ 12.743780125278874, 49.86816018348006 ],
    "location_radius" : 0,
    "location_mark" : [ 12.74180946741263, 49.86938948602838 ],
    "location_source" : "PLACE",
    "hosts" : [ ],
    "hosts_count" : 0,
    "links" : [ ],
    "links_count" : 0,
    "mentions" : [ "fossasia", "mariobehling" ],
    "mentions_count" : 2,
    "hashtags" : [ ],
    "hashtags_count" : 0,
    "classifier_language" : "english",
    "classifier_language_probability" : 2.7384035E-14,
    "without_l_len" : 103,
    "without_lu_len" : 79,
    "without_luh_len" : 79,
    "user" : {
      "screen_name" : "notrademark",
      "user_id" : "2374283574",
      "name" : "msquare",
      "profile_image_url_https" : "https://pbs.twimg.com/profile_images/514057179737772032/OPi1EgNA_bigger.png",
      "appearance_first" : "2015-08-17T02:42:20.874Z",
      "appearance_latest" : "2015-08-17T02:42:20.874Z"
    }
  } ],
  "aggregations" : { }
}

The following is a sample search result in RSS:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
  xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"
  xmlns:atom="http://www.w3.org/2005/Atom"
>
<channel>
<title>RSS feed for Twitter search for Divine</title>
<pubDate>Sat, 11 Jun 2016 08:48:49 +0000</pubDate>
<opensearch:startIndex>0</opensearch:startIndex>
<opensearch:itemsPerPage>20</opensearch:itemsPerPage>
<opensearch:totalResults>20</opensearch:totalResults>
<item>
<title>• Harley • @sunreymb</title>
<link>https://twitter.com/sunreymb/status/741552632200757249</link>
<description>@Sick_and_Divine YOU WERE SO SQUEAMISH</description>
<pubDate>Sat, 11 Jun 2016 08:48:19 +0000</pubDate>
<guid isPermaLink="false">741552632200757249</guid>
</item>
<item>
<title>siya nyengane @siya_nyengane</title>
<link>https://twitter.com/siya_nyengane/status/741552630502154240</link>
<description>6pm the divine encounter. I hope you will be a blessing again today. Love your show. Blessings to you. @Franklites</description>
<pubDate>Sat, 11 Jun 2016 08:48:19 +0000</pubDate>
<guid isPermaLink="false">741552630502154240</guid>
</item>
<item>
<title>Total OneEighty @Total_180</title>
<link>https://twitter.com/Total_180/status/741552626576297988</link>
<description>@WhisperNClamor&lt;/a</description>
<pubDate>Sat, 11 Jun 2016 08:48:18 +0000</pubDate>
<guid isPermaLink="false">741552626576297988</guid>
</item>
<item>
<title>悠巳@KHC愛知 @divine_KH8</title>
<link>https://twitter.com/divine_KH8/status/741552615213916160</link>
<description>三つ編み消した!!</description>
<pubDate>Sat, 11 Jun 2016 08:48:15 +0000</pubDate>
<guid isPermaLink="false">741552615213916160</guid>
</item>
<item>
<title>Uhoh It's Dad! @DigBeingADad</title>
<link>https://twitter.com/DigBeingADad/status/741552554660659201</link>
<description>To err is human, to compute divine. Trust your computer but not its programmer. ― Morris Kingston</description>
<pubDate>Sat, 11 Jun 2016 08:48:01 +0000</pubDate>
<guid isPermaLink="false">741552554660659201</guid>
</item>
<item>
<title>ActusCulture @Actus_Culture</title>
<link>https://twitter.com/Actus_Culture/status/741552549149474817</link>
<description>Musique sacrée: Gildas Avegozard Onina innove avec son opus « Divine Bonté » #culture</description>
<pubDate>Sat, 11 Jun 2016 08:47:59 +0000</pubDate>
<guid isPermaLink="false">741552549149474817</guid>
</item>
<item>
<title>John @Sick_and_Divine</title>
<link>https://twitter.com/Sick_and_Divine/status/741552464055308288</link>
<description>Me: I love gore. It can be pretty fucking hot</description>
<pubDate>Sat, 11 Jun 2016 08:47:39 +0000</pubDate>
<guid isPermaLink="false">741552464055308288</guid>
</item>
<item>
<title>悠巳@KHC愛知 @divine_KH8</title>
<link>https://twitter.com/divine_KH8/status/741552462008459264</link>
<description>ほんとすみちゃんなんで私って気付いたの………おかしい………</description>
<pubDate>Sat, 11 Jun 2016 08:47:38 +0000</pubDate>
<guid isPermaLink="false">741552462008459264</guid>
</item>
<item>
<title>paul tims @1lengua222</title>
<link>https://twitter.com/1lengua222/status/741552442958045184</link>
<description>@FoxxxBabesLive @candydeevine candy really is DIVINE</description>
<pubDate>Sat, 11 Jun 2016 08:47:34 +0000</pubDate>
<guid isPermaLink="false">741552442958045184</guid>
</item>
<item>
<title>AQUARIUS Astrology @AquariusUnite</title>
<link>https://twitter.com/AquariusUnite/status/741552316143173632</link>
<description>When an #Aquarian is confident, they radiate a divine energy that is unstoppable.</description>
<pubDate>Sat, 11 Jun 2016 08:47:04 +0000</pubDate>
<guid isPermaLink="false">741552316143173632</guid>
</item>
<item>
<title>Xander @XanderGems</title>
<link>https://twitter.com/XanderGems/status/741552299789721601</link>
<description>Loved this song for a whole year now! This will never get old to me</description>
<pubDate>Sat, 11 Jun 2016 08:47:00 +0000</pubDate>
<guid isPermaLink="false">741552299789721601</guid>
</item>
<item>
<title>Esmeralda♡ @divine__es</title>
<link>https://twitter.com/divine__es/status/741552277559738368</link>
<description>Where you're locked out of you own room &amp; have to go in from outside. #sketchmuch</description>
<pubDate>Sat, 11 Jun 2016 08:46:55 +0000</pubDate>
<guid isPermaLink="false">741552277559738368</guid>
</item>
<item>
<title>★ Hope Estheim 彡 @TheHopefulBot</title>
<link>https://twitter.com/TheHopefulBot/status/741552234559766531</link>
<description>“Clear the way.” — Upon using Divine Judgement</description>
<pubDate>Sat, 11 Jun 2016 08:46:44 +0000</pubDate>
<guid isPermaLink="false">741552234559766531</guid>
</item>
<item>
<title>Lucifer news @Lucifenews24</title>
<link>https://twitter.com/Lucifenews24/status/741552141559439360</link>
<description>Comic Book Review - The Wicked + The Divine #20... http://prt.news/amfcL7</description>
<pubDate>Sat, 11 Jun 2016 08:46:22 +0000</pubDate>
<guid isPermaLink="false">741552141559439360</guid>
</item>
<item>
<title>Piyush Puri @piyushpuri12</title>
<link>https://twitter.com/piyushpuri12/status/741552132986306565</link>
<description>#DCShoesIndia #NewStore #CityOfJoy&lt;/a</description>
<pubDate>Sat, 11 Jun 2016 08:46:20 +0000</pubDate>
<guid isPermaLink="false">741552132986306565</guid>
</item>
<item>
<title>John @Sick_and_Divine</title>
<link>https://twitter.com/Sick_and_Divine/status/741552053055459328</link>
<description>Never thought I'd ever see the day I'd watch somebody finger a heart but that day came</description>
<pubDate>Sat, 11 Jun 2016 08:46:01 +0000</pubDate>
<guid isPermaLink="false">741552053055459328</guid>
</item>
<item>
<title>Roxyychixx @YasminEntice</title>
<link>https://twitter.com/YasminEntice/status/741552050794729472</link>
<description>tatak divine hahah choss https://twitter.com/angtanongkosayo/status/741543513196630017</description>
<pubDate>Sat, 11 Jun 2016 08:46:00 +0000</pubDate>
<guid isPermaLink="false">741552050794729472</guid>
</item>
<item>
<title>Shruti :) @Divine_Shruti</title>
<link>https://twitter.com/Divine_Shruti/status/741552026870386688</link>
<description>@WestlandBooks @storytellersavi i'm liking this !!!</description>
<pubDate>Sat, 11 Jun 2016 08:45:55 +0000</pubDate>
<guid isPermaLink="false">741552026870386688</guid>
</item>
<item>
<title>悠巳@KHC愛知 @divine_KH8</title>
<link>https://twitter.com/divine_KH8/status/741551946872426496</link>
<description>そういえばすみちゃんには顔バレてるじゃん!!!!!!!!!!!!めっちゃ恥ずかしい!!!!!!!!!!!!</description>
<pubDate>Sat, 11 Jun 2016 08:45:36 +0000</pubDate>
<guid isPermaLink="false">741551946872426496</guid>
</item>
<item>
<title>da13thsun™ @da13thsun</title>
<link>https://twitter.com/da13thsun/status/741551903033659392</link>
<description>The evolving soul is not made divine by what it does, but by what it strives to do</description>
<pubDate>Sat, 11 Jun 2016 08:45:25 +0000</pubDate>
<guid isPermaLink="false">741551903033659392</guid>
</item>
</channel>
</rss>

 

While harvesting tweets, the Twitter will act as the source, the loklak API will provide with the distributed peer – to – peer infrastructure and ElasticSearch will work as the search index. Whenever scraping is done,  the JSON dump becomes a very large collection of archived tweets. This can be used with a search portal like Kibana which provides the front-end experience.

Loklak has been already implemented for Twitter and in the near future, we are planning to implement it on other platforms such as Instagram, Wikipedia and possibly wherever  good amount of data dumps are available.

Loklak !!

 

 

 

Loklak: It’s not just an application