Search vs Aggregations

    The search aggregations also know as facets(Field aggregations) are often ignored while using the Search API provided by Loklak. This post aims at clearing up the difference and bring out the clear usage. This extends the previous blog, titled “Know who tweeted the most!” where we discussed about the Search API. Using the same example, the results were viewed using the search aggregations. The following two pictures illustrates the difference.

Selection_074

Selection_058

    The above results are achieved by scraping the aggregation field called mentions, which contains the list of user names who mentioned the search hashtag in their tweets. The result is from the one billion tweets(~1,377,367,341) which Loklak had scraped and indexed. The following screenshots show the JSON object of the search aggregations.

Selection_076

    The above search metadata object shows some important fields which can alter the result. The count is set to zero and the cache is disabled so that the aggregations do not consider the remote search results. The query fields shows the constraints on which the query must be performed. It shows the hashtag for which it must query. Since the search aggregations are going to query the whole Loklak index(over one billion tweets) we can limit the result payload by giving time constraints such as “since” and “until” fields. We can even limit the result display by mentioning a value to the limit field. This helps you to form a search query which gives you a much accurate result. The following is a sample query where you can check out the JSON result. Edit the “q” variable to your search requirement and the time fields too.

http://loklak.org/api/search.json?q=loklak since:2016-01-01 until:2016-03-01&source=cache&count=0&fields=mentions,hashtags

The following shows the aggregations field and the benefits you can get out of it.

Selection_078

    The aggregations field consists of the hashtags and the mentions. The mentions contains all user names denoted by a ‘@’-prefix from the message, which is being listed without the leading ‘@’. The hashtags contains all hashtags appearing in the message, without the leading ‘#’. You have more fields which you can mention in the query according to your need such as limit, source, timezoneOffset, minified etc. For more info on the search aggregations visit the docs page.

You can even visit to the app “knowTheDiff” which guides you through the difference with step by step intro.

Selection_075

The app is still under construction and needs to get more fields into the picture, but can try out the basic difference. Here are some sample queries where you can try out.

Search vs Aggregations

Know who tweeted the most!

It is obvious that all of us are curious to find out who is leading! My app, which is called ‘Twitter Leaderboard uses the search API provided by ‘Loklak’. Using my app you can collect the top five tweeters whose tweets contain the mentioned hashtag.

So here is a simple explanation of how ‘Twitter Leaderboard’ works. I built this app using AngularJS and the search API provided by Loklak. Primarily, I collected the tweets’ data and parse it for the required fields.

The leader board needed to have the tweeter’s name along with their tweet count. Also ‘Twitter Leaderboard’ needed to allow its users to search for a tweet based on the hashtag supplied to it in a query. 

I wrote a small controller to make a HTTP request to the Loklak search API, and parse the retrieved object accordingly.  I parsed the required data in the following manner:

  • data.statuses //The status object containing the requested hashtag.
  • user = statuses[i].screen_name // Get all the screen names.
  • counts = {} //Counts object to store the number of tweets count for each name.

After parsing for the required data from the search results, we have to sort the names according to the tweets count.  And here we go the results are pushed to the front end and displayed.

Selection_058

Another way to improve the search result is by using the Search Aggregations. This feature helps you to get the count results in a better by scaling up the search result. It will help you to count up all the tweets which are indexed(More than a billion tweets). This feature will not consider remote search results. Therefore the remote search is switched off by “source=cache” and the search results are switched off with “count=0”. The request for aggregation is simply hidden in “fields=…” which names the fields where you want to have an aggregation. You can even reduce the result size by restricting your search to certain period of time.

The upgrade to the above app will be provided soon using the Aggregations.

Know who tweeted the most!

Loklak : Map your Tweets!

The heading must have aroused your interest what it is about, but let me first tell you about Loklak.Loklak is a tool which you can use to anonymously scrap the tweets from twitter.Such a great back-end API needs an awesome enough front-end tool!That’s what we are developing this year as GSOC Students.The Loklak Front End, currently in its pre-alpha stage just completed its 1st phase out of 4, so I will be talking what we have achieved till now.I am working on the Maps feature so I will mostly speak about it

The Tweet Map feature allows the user to visualise the tweets as they are tweeted by the users on world maps as markers.The markers will show the tweet content in a popup with other related content like the username, time, location and links.The current stage is updated regularly and can be seen here. You can also try the wall feature thats another dimension.

ad_converted

Technologies I am using

Loklak Frontend is built on AngularJS as most of us developers share a love for JS here at Loklak.I am using Leaflet libraries to handle the Map related features.Currently I am using a inline geoJSON Variable to plot the markers on the map, soon it will be plotted from a jsonp request to server.

Its just a start to an awesome ending, once the whole feature is complete a user can select a region to show the tweets in, put in a name of a place and retrieve its tweet on the world map.Just imagine you want to see whats trending where the most? Just put in a #tag and you can see the density or may be heatmap! Its just way too cool! It has lots of potential and we are just scratching the surface. If you would like to join the development. Join us on Github, chat with us on Gitter.

Thats all for now! Next week new developments new post!

Loklak : Map your Tweets!