Loklak: It’s not just an application

Loklak is not just an application. It is a server based application capable of scraping messages from Twitter unofficially and other sources. This server contains a search index and a peer-to-peer index sharing interface. It is available as a local host server, API and webclient.  The loklak server can be cloned from the repo present on github and can be run on localhost using Apache ANT.

Some of the highlights of loklak are:

  • It’s architecture is very generic and can be used for any twitter like platform.
  • Loklak is anonymous, as it doesn’t store client’s IP address when a search is done (thanks to the use of scrapers).
  •  Loklak can recursively de-shorten shortened links (so as to free up some more characters).
  • One can create an own search engine for tweets.

Next coming on to the loklak API, it is very much flexible and can be easily implemented. There are existing API libraries with support in Node.js, Python, C#.Net and Ruby in order to use the loklak API. The loklak being based on http is very much ubiquitous. One can access the loklak API without any authentication, but there exist three classes for access rights which are:

  1. open: access without any restrictions for all clients from any IP
  2. limited: localhost clients are granted more data than public clients
  3. restricted: only localhost clients are granted access

The loklak APIs that are currently available are:

  • status.json [with no access restrictions]
  • search.json [with no access restrictions]
  • suggest.json [with some access restictions]
  • crawler.json [with some access restrictions]
  • hello.json [with no access restrictions]
  • geocode.json [with no access restictions]
  • peers.json [with no access restrictions]
  • proxy.json [with some access restrctions]
  • push.json [with some access restrictions]
  • push/geojson.json [with no access restrcitions]
  • push/”.” [with no access restrictions]
  • import.json [with no access restrictions]
  • validate.json [with no access restrictions]
  • settings.json [with some access restrictions]
  • account.json [with some access restrictions]
  • user.json [with no access restrictions]
  • asset [with some access restrictions]
  • threaddump.txt [with some access restrictions]
  • vis/map.png [with no access restrictions]
  • vis/markdown.png [with no access restricions]

The search results on loklak can be retrieved in two formats: JSON and RSS. 

The following is a sample search result in JSON:

{
  "readme_0" : "THIS JSON IS THE RESULT OF YOUR SEARCH QUERY - THERE IS NO WEB PAGE WHICH SHOWS THE RESULT!",
  "readme_1" : "loklak.org is the framework for a message search system, not the portal, read: http://loklak.org/about.html#notasearchportal",
  "readme_2" : "This is supposed to be the back-end of a search portal. For the api, see http://loklak.org/api.html",
  "readme_3" : "Parameters q=(query), source=(cache|backend|twitter|all), callback=p for jsonp, maximumRecords=(message count), minified=(true|false)",
  "search_metadata" : {
    "itemsPerPage" : "1",
    "count" : "1",
    "count_twitter_all" : 0,
    "count_twitter_new" : 0,
    "count_backend" : 0,
    "count_cache" : 7336,
    "hits" : 7336,
    "period" : 9223372036854775807,
    "query" : "fossasia",
    "client" : "122.175.88.159",
    "time" : 1196,
    "servicereduction" : "false",
    "scraperInfo" : "local"
  },
  "statuses" : [ {
    "timestamp" : "2016-04-21T17:33:06.959Z",
    "created_at" : "2016-04-21T15:39:39.000Z",
    "screen_name" : "notrademark",
    "text" : "@fossasia @mariobehling sure, I am happy about every pull request on GitHub. What features do you plan?",
    "link" : "https://twitter.com/notrademark/status/723174364968005632",
    "id_str" : "723174364968005632",
    "source_type" : "USER",
    "provider_type" : "REMOTE",
    "provider_hash" : "1cadbfd3",
    "retweet_count" : 0,
    "favourites_count" : 0,
    "images" : [ ],
    "images_count" : 0,
    "audio" : [ ],
    "audio_count" : 0,
    "videos" : [ ],
    "videos_count" : 0,
    "place_name" : "Planá",
    "place_id" : "",
    "place_context" : "FROM",
    "location_point" : [ 12.743780125278874, 49.86816018348006 ],
    "location_radius" : 0,
    "location_mark" : [ 12.74180946741263, 49.86938948602838 ],
    "location_source" : "PLACE",
    "hosts" : [ ],
    "hosts_count" : 0,
    "links" : [ ],
    "links_count" : 0,
    "mentions" : [ "fossasia", "mariobehling" ],
    "mentions_count" : 2,
    "hashtags" : [ ],
    "hashtags_count" : 0,
    "classifier_language" : "english",
    "classifier_language_probability" : 2.7384035E-14,
    "without_l_len" : 103,
    "without_lu_len" : 79,
    "without_luh_len" : 79,
    "user" : {
      "screen_name" : "notrademark",
      "user_id" : "2374283574",
      "name" : "msquare",
      "profile_image_url_https" : "https://pbs.twimg.com/profile_images/514057179737772032/OPi1EgNA_bigger.png",
      "appearance_first" : "2015-08-17T02:42:20.874Z",
      "appearance_latest" : "2015-08-17T02:42:20.874Z"
    }
  } ],
  "aggregations" : { }
}

The following is a sample search result in RSS:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
  xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"
  xmlns:atom="http://www.w3.org/2005/Atom"
>
<channel>
<title>RSS feed for Twitter search for Divine</title>
<pubDate>Sat, 11 Jun 2016 08:48:49 +0000</pubDate>
<opensearch:startIndex>0</opensearch:startIndex>
<opensearch:itemsPerPage>20</opensearch:itemsPerPage>
<opensearch:totalResults>20</opensearch:totalResults>
<item>
<title>• Harley • @sunreymb</title>
<link>https://twitter.com/sunreymb/status/741552632200757249</link>
<description>@Sick_and_Divine YOU WERE SO SQUEAMISH</description>
<pubDate>Sat, 11 Jun 2016 08:48:19 +0000</pubDate>
<guid isPermaLink="false">741552632200757249</guid>
</item>
<item>
<title>siya nyengane @siya_nyengane</title>
<link>https://twitter.com/siya_nyengane/status/741552630502154240</link>
<description>6pm the divine encounter. I hope you will be a blessing again today. Love your show. Blessings to you. @Franklites</description>
<pubDate>Sat, 11 Jun 2016 08:48:19 +0000</pubDate>
<guid isPermaLink="false">741552630502154240</guid>
</item>
<item>
<title>Total OneEighty @Total_180</title>
<link>https://twitter.com/Total_180/status/741552626576297988</link>
<description>@WhisperNClamor&lt;/a</description>
<pubDate>Sat, 11 Jun 2016 08:48:18 +0000</pubDate>
<guid isPermaLink="false">741552626576297988</guid>
</item>
<item>
<title>悠巳@KHC愛知 @divine_KH8</title>
<link>https://twitter.com/divine_KH8/status/741552615213916160</link>
<description>三つ編み消した!!</description>
<pubDate>Sat, 11 Jun 2016 08:48:15 +0000</pubDate>
<guid isPermaLink="false">741552615213916160</guid>
</item>
<item>
<title>Uhoh It's Dad! @DigBeingADad</title>
<link>https://twitter.com/DigBeingADad/status/741552554660659201</link>
<description>To err is human, to compute divine. Trust your computer but not its programmer. ― Morris Kingston</description>
<pubDate>Sat, 11 Jun 2016 08:48:01 +0000</pubDate>
<guid isPermaLink="false">741552554660659201</guid>
</item>
<item>
<title>ActusCulture @Actus_Culture</title>
<link>https://twitter.com/Actus_Culture/status/741552549149474817</link>
<description>Musique sacrée: Gildas Avegozard Onina innove avec son opus « Divine Bonté » #culture</description>
<pubDate>Sat, 11 Jun 2016 08:47:59 +0000</pubDate>
<guid isPermaLink="false">741552549149474817</guid>
</item>
<item>
<title>John @Sick_and_Divine</title>
<link>https://twitter.com/Sick_and_Divine/status/741552464055308288</link>
<description>Me: I love gore. It can be pretty fucking hot</description>
<pubDate>Sat, 11 Jun 2016 08:47:39 +0000</pubDate>
<guid isPermaLink="false">741552464055308288</guid>
</item>
<item>
<title>悠巳@KHC愛知 @divine_KH8</title>
<link>https://twitter.com/divine_KH8/status/741552462008459264</link>
<description>ほんとすみちゃんなんで私って気付いたの………おかしい………</description>
<pubDate>Sat, 11 Jun 2016 08:47:38 +0000</pubDate>
<guid isPermaLink="false">741552462008459264</guid>
</item>
<item>
<title>paul tims @1lengua222</title>
<link>https://twitter.com/1lengua222/status/741552442958045184</link>
<description>@FoxxxBabesLive @candydeevine candy really is DIVINE</description>
<pubDate>Sat, 11 Jun 2016 08:47:34 +0000</pubDate>
<guid isPermaLink="false">741552442958045184</guid>
</item>
<item>
<title>AQUARIUS Astrology @AquariusUnite</title>
<link>https://twitter.com/AquariusUnite/status/741552316143173632</link>
<description>When an #Aquarian is confident, they radiate a divine energy that is unstoppable.</description>
<pubDate>Sat, 11 Jun 2016 08:47:04 +0000</pubDate>
<guid isPermaLink="false">741552316143173632</guid>
</item>
<item>
<title>Xander @XanderGems</title>
<link>https://twitter.com/XanderGems/status/741552299789721601</link>
<description>Loved this song for a whole year now! This will never get old to me</description>
<pubDate>Sat, 11 Jun 2016 08:47:00 +0000</pubDate>
<guid isPermaLink="false">741552299789721601</guid>
</item>
<item>
<title>Esmeralda♡ @divine__es</title>
<link>https://twitter.com/divine__es/status/741552277559738368</link>
<description>Where you're locked out of you own room &amp; have to go in from outside. #sketchmuch</description>
<pubDate>Sat, 11 Jun 2016 08:46:55 +0000</pubDate>
<guid isPermaLink="false">741552277559738368</guid>
</item>
<item>
<title>★ Hope Estheim 彡 @TheHopefulBot</title>
<link>https://twitter.com/TheHopefulBot/status/741552234559766531</link>
<description>“Clear the way.” — Upon using Divine Judgement</description>
<pubDate>Sat, 11 Jun 2016 08:46:44 +0000</pubDate>
<guid isPermaLink="false">741552234559766531</guid>
</item>
<item>
<title>Lucifer news @Lucifenews24</title>
<link>https://twitter.com/Lucifenews24/status/741552141559439360</link>
<description>Comic Book Review - The Wicked + The Divine #20... http://prt.news/amfcL7</description>
<pubDate>Sat, 11 Jun 2016 08:46:22 +0000</pubDate>
<guid isPermaLink="false">741552141559439360</guid>
</item>
<item>
<title>Piyush Puri @piyushpuri12</title>
<link>https://twitter.com/piyushpuri12/status/741552132986306565</link>
<description>#DCShoesIndia #NewStore #CityOfJoy&lt;/a</description>
<pubDate>Sat, 11 Jun 2016 08:46:20 +0000</pubDate>
<guid isPermaLink="false">741552132986306565</guid>
</item>
<item>
<title>John @Sick_and_Divine</title>
<link>https://twitter.com/Sick_and_Divine/status/741552053055459328</link>
<description>Never thought I'd ever see the day I'd watch somebody finger a heart but that day came</description>
<pubDate>Sat, 11 Jun 2016 08:46:01 +0000</pubDate>
<guid isPermaLink="false">741552053055459328</guid>
</item>
<item>
<title>Roxyychixx @YasminEntice</title>
<link>https://twitter.com/YasminEntice/status/741552050794729472</link>
<description>tatak divine hahah choss https://twitter.com/angtanongkosayo/status/741543513196630017</description>
<pubDate>Sat, 11 Jun 2016 08:46:00 +0000</pubDate>
<guid isPermaLink="false">741552050794729472</guid>
</item>
<item>
<title>Shruti :) @Divine_Shruti</title>
<link>https://twitter.com/Divine_Shruti/status/741552026870386688</link>
<description>@WestlandBooks @storytellersavi i'm liking this !!!</description>
<pubDate>Sat, 11 Jun 2016 08:45:55 +0000</pubDate>
<guid isPermaLink="false">741552026870386688</guid>
</item>
<item>
<title>悠巳@KHC愛知 @divine_KH8</title>
<link>https://twitter.com/divine_KH8/status/741551946872426496</link>
<description>そういえばすみちゃんには顔バレてるじゃん!!!!!!!!!!!!めっちゃ恥ずかしい!!!!!!!!!!!!</description>
<pubDate>Sat, 11 Jun 2016 08:45:36 +0000</pubDate>
<guid isPermaLink="false">741551946872426496</guid>
</item>
<item>
<title>da13thsun™ @da13thsun</title>
<link>https://twitter.com/da13thsun/status/741551903033659392</link>
<description>The evolving soul is not made divine by what it does, but by what it strives to do</description>
<pubDate>Sat, 11 Jun 2016 08:45:25 +0000</pubDate>
<guid isPermaLink="false">741551903033659392</guid>
</item>
</channel>
</rss>

 

While harvesting tweets, the Twitter will act as the source, the loklak API will provide with the distributed peer – to – peer infrastructure and ElasticSearch will work as the search index. Whenever scraping is done,  the JSON dump becomes a very large collection of archived tweets. This can be used with a search portal like Kibana which provides the front-end experience.

Loklak has been already implemented for Twitter and in the near future, we are planning to implement it on other platforms such as Instagram, Wikipedia and possibly wherever  good amount of data dumps are available.

Loklak !!

 

 

 

Loklak: It’s not just an application