First Loklak plugin added to WordPress directory

Recently first Loklak plugin was accepted in WordPress directory. We are planning to add more and more plugins with loklak integration and it is an awesome start. The plugin Tweets Widget is a minimalist tweet feed which can render tweets from Loklak.org API or Twitter API v1.1 .

The widget uses Twitter Auth WordPress API (wp-twitter-api) by _timwhitlock_ and Loklak’s PHP API for tweet rendering purposes. Both the APIs are added as submodules.

Tweets Widget shows tweet feed on your wordpress website. You can add its shortcode or just add it as a Widget using wordpress’s drag and drop feature.

It is compatible with both Twitter as well as Loklak API. The settings page allows you to select one of the two modes for your current feed (as shown below).

screenshot-2

After you provide the default settings, you have to provide a tweet title, handle, number of tweets and some more custom feature

screenshot-3

See the screenshot of a sample feed below.

screenshot-1

It uses loklak search API (code sample shown below).

Selection_020

The main widget code can be seen here.

First Loklak plugin added to WordPress directory

Improving Code Coverage for Loklak PHP API

Tests


This week, I added tests for suggest, map, markdown, push and susi APIs and fixed tests for user topology related queries. PHPUnit was used as the testing framework since it provides code-coverage support.

Loklak PHP API now has a good test-suite support and associated examples to help you use our services

Below are the Tests that were added.

testPushto test Push API.Selection_001

testSusito test newly added Susi API. Selection_003

Refer to this and this for more info about Susi

testMap – to test map API.

testMarkdown – to test markdown API.Selection_002

For more detailed information regarding the entire Loklak PHP API test-suite, refer to this.

In above process, code-coverage was increased from 33% to 61%. Test-suite continuously updated as new APIs are added to Loklak.

Source support for Search API


Apart from that, since Loklak is scaling beyond Twitter. Source argument has been added to the search API to define the source of search results. As far as wordpress plugins are concerned, since they only require twitter results for now, source has been added as a default argument. See below code: Selection_004

Improving Code Coverage for Loklak PHP API

User Information via Loklak API

While working on “adding loklak API support to  wordpress plugins”, I found that a lot plugins require Twitter user information e.g. profile_pic, followers_count, tweets by the user etc.

One can get an exhaustive user information using Loklak search and user APIs. In this blog post I would show how one can combine the results from Loklak’s Search and User API to provide all kinds of data required by a WordPress plugin (e.g.).

Starting with Search API!

Typing

http://loklak.org/api/search.json?q=from:KhoslaSopan

in url would give you results like this:

Selection_072

As you can see, it provides all tweet (tweeted by the <user>) related information and a small user profile like this:

Selection_073

But what if you require his location, followers_count, friends_count, no. of tweets he has posted till now etc, you would have to take help from the Loklak User API. Sample results for

http://loklak.org/api/user.json?screen_name=KhoslaSopan

are given below:

Selection_074

But somehow you also require the information of his followers and people he is following. How do you achieve that? Well Loklak API has provision for that as well. All you need to do is add followers=<count> or following=<count> as need be.

http://loklak.org/api/user.json?screen_name=KhoslaSopan&following=100&followers=100

Following data would be added to your user.json results.

Selection_075

It would give you the user information of twitter users followed by that user. Recursively you can get their tweet data, their followers and on and on!!

In order to get complete user profile, you can merge the results from Search and User API and create wonderful applications.

To do all this using Loklak PHP API, refer to the code snippet below.

$connection = new Loklak();
$tweets = $connection->search('', null, null, $username, $no_of_tweets);

$tweets = json_decode($tweets, true);
$tweets = json_decode($tweets['body'], true);

$user = $connection->user($username);
$user = json_decode($user, true);
$user = json_decode($user['body'], true);

$tweet_content = $tweets['statuses'];
for ($i=0; $i < sizeof($tweet_content); $i++) {
     $tweet_content[$i] = array_merge($tweet_content[$i], $user);
}
User Information via Loklak API

Does tweets have emotions?

Tweets do intend some kind of emotion when they are tweeted. It can be happiness, sad, anticipation etc. Loklak does such classification and gives you the categorization of the tweet along with the probability of it being correct.

Selection_116

The sentiment analysis is one such tool which uses the classification provided by the Loklak Search API from the field called “classifier_emotion”. It displays the other metrics like the languages “classifier_language” in which the tweet is being tweeted along with the emotions. These results are visualized with the help of flash cards along with pie charts. Selection_117Selection_118

This has a clean display of the results which shows the recent 10 tweets being tweeted using the hashtag mentioned. One should simply enter the hashtag in to the search box for the results. The following image shows part of the search API results(JSONObject).

Selection_119

I used the above field names”classifier_emotion”, “classifier_language” to display the results. I have used Highcharts to display piecharts.

The internal churning and analysis of the data is happening in the following folder structure [org->loklak->tools->bayes]. Where an abstract Classifier class is being implemented in which the Bayes classifier implements a naive Bayes approach to classifying a given set of features:

classify(feat1,…,featN) = argmax(P(cat)*PROD(P(featI|cat)

Does tweets have emotions?

Let’s ‘Meetup’ with Loklak

Loklak has already started to expand beyond the realms of Twitter and working its way to build an extensive social network.

Now, Loklak aims to bring in data crawled from meetups.com to create a close-knit community.

Chiming together with Meetup’s mission to revitalize local community and help people around the world self-organize, Loklak strives to revolutionize the social networking scenario of the present world.

Screenshot from 2016-06-10 09:23:10
Screenshot from 2016-06-10 09:24:05

In order to extract information viz. Group Name, Location, Description, Group Topics/Tags, Recent Meetups (Day, Date, Time, RSVP, Reviews, Introductory lines etc.) about a specific group on meetups.com I have used the URL http://www.meetup.com/<group-name>/ and then scraped information from the HTML page itself.

Just like previous experiments with other webpages, I have made use of JSoup: The Java HTML parser library in the loklak_server. It provides a very convenient API for extracting and manipulating data, scrape and parse HTML from a URL. The suggested use of JSoup is designed to deal with all varieties of HTML, hence as of now it is being considered a suitable choice.

The information scraped is stored in a multi-dimensional array of recentMeetupsResult[][] and then the data inside can be used accordingly.

A sample code-snippet for reference is as:

/**
 *  Meetups Scraper
 *  By Jigyasa Grover, @jig08
 **/

package org.loklak.harvester;

import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.*;
import org.jsoup.select.Elements;

public class MeetupsScraper {
	public static void main(String args[]){
		
		Document meetupHTML = null;
		String meetupGroupName = "Women-Who-Code-Delhi";
		// fetch group name here
		Element groupDescription = null;
		String groupDescriptionString = null;
		Element topicList = null;
		Elements topicListStrings = null;
		String[] topicListArray = new String[100];
		Integer numberOfTopics = 0;
		Element recentMeetupsSection = null;
		Elements recentMeetupsList = null;
		Integer numberOfRecentMeetupsShown = 0;
		Integer i = 0, j = 0;
		String recentMeetupsResult[][] = new String[100][3];
		
		// recentMeetupsResult[i][0] == date && time
		// recentMeetupsResult[i][1] == Attendance && Review
		// recentMeetupsResult[i][2] == Information
				
		try{
			meetupHTML = Jsoup.connect("http://www.meetup.com/" + meetupGroupName).userAgent("Mozilla)").get();
			
			groupDescription = meetupHTML.getElementById("groupDesc");
			groupDescriptionString = groupDescription.text();
			System.out.println(meetupGroupName + "ntGroup Description: ntt" + groupDescriptionString);
			
			topicList = meetupHTML.getElementById("topic-box-2012");
			topicListStrings = topicList.getElementsByTag("a");
			
			int p = 0;
			for(Element topicListStringsIterator : topicListStrings){
				topicListArray[p] = topicListStringsIterator.text().toString();
				p++;
			}
			numberOfTopics = p;
			
			System.out.println("nGroup Topics:");
			for(int l = 0; l<numberOfTopics; l++){
				System.out.println("ntTopic Number "+ l + " : " + topicListArray[l]);
			}
			
			recentMeetupsSection = meetupHTML.getElementById("recentMeetups");
			recentMeetupsList = recentMeetupsSection.getElementsByTag("p");
			
			i = 0;
			j = 0;
			
			for(Element recentMeetups : recentMeetupsList ){				
				if(j%3==0){
					j = 0;
					i++;
				}
				
				recentMeetupsResult[i][j] = recentMeetups.text().toString();
				j++;
				
			}
			
			numberOfRecentMeetupsShown = i;
			
			for(int k = 1; k < numberOfRecentMeetupsShown; k++){
				System.out.println("nnRecent Meetup Number" + k + " : n" + 
						"nt Date & Time: " + recentMeetupsResult[k][0] + 
						"nt Attendance: " + recentMeetupsResult[k][1] + 
						"nt Information: " + recentMeetupsResult[k][2]);
			}

		}catch (IOException e) {
            e.printStackTrace();
        }
		
	}
}

In this, simply a HTTP Connection was established and text extracted using "element_name".text()from inside the specific tags using identifiers like classes or ids. The tags from which the information was to be extracted were identified after exploring the web page’s HTML source code.

The above yields results as:

Women-Who-Code-Delhi
	Group Description: 
		Mission: Women Who Code is a global nonprofit organization dedicated to inspiring women to excel in technology careers by creating a global, connected community of women in technology. The organization tripled in 2013 and has grown to be one of the largest communities of women engineers in the world. Empowerment: Women Who code is a professional community for women in tech. We provide an avenue for women to pursue a career in technology, help them gain new skills and hone existing skills for professional advancement, and foster environments where networking and mentorship are valued. Key Initiatives: - Free technical study groups - Events featuring influential tech industry experts and investors - Hack events - Career and leadership development Current and aspiring coders are welcome.  Bring your laptop and a friend!  Support Women Who Code: Donating to Women Who Code, Inc. (#46-4218859) directly impacts our ability to efficiently run this growing organization, helps us produce new programs that will increase our reach, and enables us to expand into new cities around the world ensuring that women and girls everywhere have the opportunity to pursue a career in technology. Women Who Code (WWCode) is dedicated to providing an empowering experience for everyone who participates in or supports our community, regardless of gender, gender identity and expression, sexual orientation, ability, physical appearance, body size, race, ethnicity, age, religion, or socioeconomic status. Because we value the safety and security of our members and strive to have an inclusive community, we do not tolerate harassment of members or event participants in any form. Our Code of Conduct applies to all events run by Women Who Code, Inc. If you would like to report an incident or contact our leadership team, please submit an incident report form. WomenWhoCode.com

Group Topics:

	Topic Number 0 : Django

	Topic Number 1 : Web Design

	Topic Number 2 : Ruby

	Topic Number 3 : HTML5

	Topic Number 4 : Women Programmers

	Topic Number 5 : JavaScript

	Topic Number 6 : Python

	Topic Number 7 : Women in Technology

	Topic Number 8 : Android Development

	Topic Number 9 : Mobile Technology

	Topic Number 10 : iOS Development

	Topic Number 11 : Women Who Code

	Topic Number 12 : Ruby On Rails

	Topic Number 13 : Computer programming

	Topic Number 14 : WWC


Recent Meetup Number1 : 

	 Date & Time: April 2 · 10:30 AM
	 Attendance: 13 Women Who Code-rs | 5.001
	 Information: Brought to you in collaboration with Women Techmakers Delhi.According to a survey, only 11% of open source participants are women. People find it intimidating to get... Learn more


Recent Meetup Number2 : 

	 Date & Time: March 3 · 3:00 PM
	 Attendance: 21 Women Who Code-rs | 5.001
	 Information: “Behold, the number five is at hand. Grab it and shake and harness the power of networking.” Women Who Code Delhi is proud to present Social Hack Eve, a networking... Learn more


Recent Meetup Number3 : 

	 Date & Time: Oct 18, 2015 · 9:00 AM
	 Attendance: 20 Women Who Code-rs | 4.502
	 Information: Hello Ladies :) Google Women Techmakers is looking for women techies to present a talk in one of the segments of Google DevFest Delhi 2015 planned for October 18, 2015... Learn more


Recent Meetup Number4 : 

	 Date & Time: Jul 5, 2015 · 12:00 PM
	 Attendance: 24 Women Who Code-rs | 4.001 | 1 Photo
	 Information: Agenda: Learning how to use and develop open source software, and contribute to huge existing open source projects.A series of talks by some of this year’s GSoC... Learn more

In this sample, simply text has been retrieved . Advanced versions could include the hyperlinks and multimedia embedded in the web-page and integrating the extracted information in a suitable format.


Check out this space for more details and implementation details of crawlers.
Feedback and Suggestions welcome.

Let’s ‘Meetup’ with Loklak

Loklak Weibo: Now going beyond Twitter !

As of now, Loklak has done a wonderful job in collecting billion(s) of tweets especially from Twitter. The highlight of this service has been anonymous search of Twitter without the use of any authentication key.

The next step is to go beyond Twitter and collect data from Chinese twitter like services for instance Weibo.com.

Screenshot from 2016-06-02 01:39:19.png

The major challenge however is to understand the Chinese annotations especially being from a non-Chinese background, but now that I have support of a Chinese friend from the Loklak community,  we hope it shall be an easier task to achieve now 🙂

The trick shall be simple, to parse the HTML page of the search results. It has been suggested to use the JSoup: The Java HTML parser library in the loklak_server. It provides a very convenient API for extracting and manipulating data, scrape and parse HTML from a URL. The suggested use of JSoup is designed to deal with all varieties of HTML, hence as of now it is being considered a suitable choice.

In our approach, the HTML page generated by the search query http://s.weibo.com/weibo/<search-string> shall be parsed instead of going the traditional way using the API call by authenticating via the key.

Screenshot from 2016-06-02 01:48:34.png
Sample code snippet to extract the title of the page:

String q = "Berlin";
//Get the Search Query in "q" here
Document doc = null;
String title = null;
try {
doc = Jsoup.connect("http://s.weibo.com/weibo/"+q).get();
title = doc.title();
} catch (IOException e) {
e.printStackTrace();
}

Check out this space for upcoming detail on implementing this technique to parse the entire page and get desired results…

Feedback and Suggestions welcome.

Loklak Weibo: Now going beyond Twitter !

Know who tweeted the most!

It is obvious that all of us are curious to find out who is leading! My app, which is called ‘Twitter Leaderboard uses the search API provided by ‘Loklak’. Using my app you can collect the top five tweeters whose tweets contain the mentioned hashtag.

So here is a simple explanation of how ‘Twitter Leaderboard’ works. I built this app using AngularJS and the search API provided by Loklak. Primarily, I collected the tweets’ data and parse it for the required fields.

The leader board needed to have the tweeter’s name along with their tweet count. Also ‘Twitter Leaderboard’ needed to allow its users to search for a tweet based on the hashtag supplied to it in a query. 

I wrote a small controller to make a HTTP request to the Loklak search API, and parse the retrieved object accordingly.  I parsed the required data in the following manner:

  • data.statuses //The status object containing the requested hashtag.
  • user = statuses[i].screen_name // Get all the screen names.
  • counts = {} //Counts object to store the number of tweets count for each name.

After parsing for the required data from the search results, we have to sort the names according to the tweets count.  And here we go the results are pushed to the front end and displayed.

Selection_058

Another way to improve the search result is by using the Search Aggregations. This feature helps you to get the count results in a better by scaling up the search result. It will help you to count up all the tweets which are indexed(More than a billion tweets). This feature will not consider remote search results. Therefore the remote search is switched off by “source=cache” and the search results are switched off with “count=0”. The request for aggregation is simply hidden in “fields=…” which names the fields where you want to have an aggregation. You can even reduce the result size by restricting your search to certain period of time.

The upgrade to the above app will be provided soon using the Aggregations.

Know who tweeted the most!

Android Twitter Search App with loklak

Everyone can create an app using the loklak_wok_android libraries. We now have an Android Tweet Search App, that fetches results using the TwitterScraper class.

Check out the code: https://github.com/loklak/LoklakAndroidApp

Requirements

  • Devices running Android-KitKat 4.4 or greater are supported.
  • Android Studio

Project setup

  • Download and setup Android Studio
  • Clone/ download this project. Cloning is recommended if you plan to contribute
  • Navigate to the directory where you saved this project and select the root folder ,and hit OK.
  • Wait for Android Studio to build the project with Gradle.
  • Once the build is complete, you can start playing around!
  • You can test it by running it on either a real device or an emulated one by going to Run>Run ‘app’ or presing the Run icon in the toolbar.

Working

Type a query and boom!

loklak_android_app

Watch out for the WiFi! This App only operates under WiFi

loklak_android_app2

Android Twitter Search App with loklak