Improved documentation for Loklak repos

Its the final week of GSoC 2016. All the projects are nearing their completion stage. Since one of the plugins from FOSSASIA (https://wordpress.org/plugins/tweets-widget/) is already in WordPress directory, I took this opportunity to write some documentation for other plugins and the plugin maintenance repo.

The documentation now verbosely describes the complete Heroku deployment procedure directly from Github as well as using the Git-Heroku toolbelt (see this).

Selection_023

Docs for updating to a newer version of WordPress have also been added.

Selection_022

Screenshots and relevant documentation regarding Loklak API was added to several plugins.

 

Selection_025
readme.txt of https://github.com/fossasia/wp-recent-tweet

Some screenshots of the plugin (wp-recent-tweet) added in the readme

Selection_026

Improved documentation for Loklak repos

Setting up plugins for test!

I spent most of this summer working on wordpress plugins. So when we were up with a substantial amount, we wanted to test them online. I discussed in a previous blog-post regarding putting an online wordpress implementation through Heroku. Once I was done with internal testing, the plugins were supposed to be released for common testing. Now, since we did not want to risk our service, we couldn’t provide admin rights to new users. So in order to overcome this problem I had the following options at my disposal:

  1. Create a script which would automatically activate and configure all plugins and show a basic plugin interface to users who are not logged in; OR
  2. Create a user with lesser privileges than administrator, but enough to view and modify plugin settings.

The problem with the first approach was its static nature. A user would not be able to test your service if he is not leveraged with all options your program provides. So, in order to ensure rigorous testing, I used the second approach.

WordPress, by default, provides 5 user-types:

  1. Administrator
  2. Editor
  3. Author
  4. Contributor
  5. Subscriber

As none of these user profiles fit the required job specification, I had to create my own user-type. After some brainstorming and searching, I found a pretty useful WordPress plugin (User Role Editor) which creates custom user profiles based on actions already present in WP suite. Once I installed the plugin on our WP installation, I used following steps to create my own user-profile called Loklak Tester. 

  • Click on ‘Users’ menu and then click on ‘User Role Editor’.

Selection_010

  • Here I selected the privileges I wanted for my user. Some of them are shown in the figure.

Selection_009

  • Once I was done, I clicked on Add Role and provided the required user-type name.
  • Below screenshot shows the menu for our new user-profile ‘Loklak Tester’.

Selection_011

Creation of this new user-profile would allow users to login using its credentials. Activate/deactivate, modify, edit plugins and change their settings. This would later act as a demo testing user which could be used by our audience to test our plugins on variety of levels.

Setting up plugins for test!

Using Heroku-WordPress Buildpack to test Loklak integration in WordPress plugins

Loklak support was added to several wordpress plugins this summer. In order to properly handle and structure further development in this area, a common repository has been created to host the incumbent plugins. Please refer to this repository. To ensure rigorous testing in the internet environment, all plugins have been installed (loklak_wordpress_plugins/wp-content/plugins) in a sample WordPress site deployed on Heroku.

Heroku is a platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud.

Heroku provides several buildpacks for different deployment patterns. After careful research I found a useful WordPress+Heroku buildpack here. To create your own wordpress instance to test the plugins follow the below instructions:

Deploy from github directly:

  1. Click on Deploy to Heroku button in the Readme file of your fork of loklak_wordpress_plugins repo. This step would install wordpress on Heroku and setup your database.
  2. Give your website a name and input your time-zone and add authentication information for .htpasswd to access wp-admin page (admin privileges). See the screenshot below. blog2_1
  3. Click on Deploy for Free. Once the app is deployed, Click on ‘Manage App’. Go to ‘Deploy’ tab and choose ‘Deployment method’ as Github
  4. Connect your forked loklak_wordpress_plugins repo to Heroku. blog2_2
  5. To automate the deployment process when the github repo is updated, enable automatic deploys. Now deploy the master branch of your repo and you are good to go. blog2_3

Deploy using Heroku Toolbelt:

To deploy using Heroku-git toolbelt, please refer to this.

After we are done with the deployment, we need to setup our plugins for using and testing:

  1. Follow the steps to install WordPress (as they appear on your computer screen) on your chosen app/domain name.
  2. Once WP is installed, change the language, time-zone and preferred date-time format in ‘general settings’.blog2_4
  3. Now you can activate plugins from ‘plugins’ as per your need and test their functionality!blog2_5
Using Heroku-WordPress Buildpack to test Loklak integration in WordPress plugins

User Information via Loklak API

While working on “adding loklak API support to  wordpress plugins”, I found that a lot plugins require Twitter user information e.g. profile_pic, followers_count, tweets by the user etc.

One can get an exhaustive user information using Loklak search and user APIs. In this blog post I would show how one can combine the results from Loklak’s Search and User API to provide all kinds of data required by a WordPress plugin (e.g.).

Starting with Search API!

Typing

http://loklak.org/api/search.json?q=from:KhoslaSopan

in url would give you results like this:

Selection_072

As you can see, it provides all tweet (tweeted by the <user>) related information and a small user profile like this:

Selection_073

But what if you require his location, followers_count, friends_count, no. of tweets he has posted till now etc, you would have to take help from the Loklak User API. Sample results for

http://loklak.org/api/user.json?screen_name=KhoslaSopan

are given below:

Selection_074

But somehow you also require the information of his followers and people he is following. How do you achieve that? Well Loklak API has provision for that as well. All you need to do is add followers=<count> or following=<count> as need be.

http://loklak.org/api/user.json?screen_name=KhoslaSopan&following=100&followers=100

Following data would be added to your user.json results.

Selection_075

It would give you the user information of twitter users followed by that user. Recursively you can get their tweet data, their followers and on and on!!

In order to get complete user profile, you can merge the results from Search and User API and create wonderful applications.

To do all this using Loklak PHP API, refer to the code snippet below.

$connection = new Loklak();
$tweets = $connection->search('', null, null, $username, $no_of_tweets);

$tweets = json_decode($tweets, true);
$tweets = json_decode($tweets['body'], true);

$user = $connection->user($username);
$user = json_decode($user, true);
$user = json_decode($user['body'], true);

$tweet_content = $tweets['statuses'];
for ($i=0; $i < sizeof($tweet_content); $i++) {
     $tweet_content[$i] = array_merge($tweet_content[$i], $user);
}
User Information via Loklak API

Now get wordpress blog updates with Loklak !

Loklak shall soon be spoiling its users !

Next, it will be bringing in tiny tweet-like cards showing the blog-posts (title, publishing date, author and content) from the given WordPress Blog URL.

This feature is certain to expand the realm of Loklak’s missive of building a comprehensive and an extensive social network dispensing useful information.

Screenshot from 2016-06-22 04:48:28

In order to implement this feature, I have again made the use of JSoup: The Java HTML parser library as it provides a very convenient API for extracting and manipulating data, scrape and parse HTML from a URL.

The information is scraped making use of JSoup after the corresponding URL in the format "https://[username].wordpress.com/" is passed as an argument to the function scrapeWordpress(String blogURL){..} which returns a JSONObject as the result.

A look at the code snippet :

/**
 *  WordPress Blog Scraper
 *  By Jigyasa Grover, @jig08
 **/

package org.loklak.harvester;

import java.io.IOException;

import org.json.JSONArray;
import org.json.JSONObject;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

public class WordPressBlogScraper {
	public static void main(String args[]){
		
		String blogURL = "https://loklaknet.wordpress.com/";
		scrapeWordpress(blogURL);		
	}
	
	public static JSONObject scrapeWordpress(String blogURL) {
		
                Document blogHTML = null;
		
		Elements articles = null;
		Elements articleList_title = null;
		Elements articleList_content = null;
		Elements articleList_dateTime = null;
		Elements articleList_author = null;

		String[][] blogPosts = new String[100][4];
		
		//blogPosts[][0] = Blog Title
		//blogPosts[][1] = Posted On
		//blogPosts[][2] = Author
		//blogPosts[][3] = Blog Content
		
		Integer numberOfBlogs = 0;
		Integer iterator = 0;
		
		try{			
			blogHTML = Jsoup.connect(blogURL).get();
		}catch (IOException e) {
            e.printStackTrace();
        }
			
			articles = blogHTML.getElementsByTag("article");
			
			iterator = 0;
			for(Element article : articles){
				
				articleList_title = article.getElementsByClass("entry-title");				
				for(Element blogs : articleList_title){
					blogPosts[iterator][0] = blogs.text().toString();
				}
				
				articleList_dateTime = article.getElementsByClass("posted-on");				
				for(Element blogs : articleList_dateTime){
					blogPosts[iterator][1] = blogs.text().toString();
				}
				
				articleList_author = article.getElementsByClass("byline");				
				for(Element blogs : articleList_author){
					blogPosts[iterator][2] = blogs.text().toString();
				}
				
				articleList_content = article.getElementsByClass("entry-content");				
				for(Element blogs : articleList_content){
					blogPosts[iterator][3] = blogs.text().toString();
				}
				
				iterator++;
				
			}
			
			numberOfBlogs = iterator;
			
			JSONArray blog = new JSONArray();
			
			for(int k = 0; k<numberOfBlogs; k++){
				JSONObject blogpost = new JSONObject();
				blogpost.put("blog_url", blogURL);
				blogpost.put("title", blogPosts[k][0]);
				blogpost.put("posted_on", blogPosts[k][1]);
				blogpost.put("author", blogPosts[k][2]);
				blogpost.put("content", blogPosts[k][3]);
				blog.put(blogpost);
			}			
			
			JSONObject final_blog_info = new JSONObject();
			
			final_blog_info.put("Wordpress blog: " + blogURL, blog);			

			System.out.println(final_blog_info);
			
			return final_blog_info;
		
	}
}

 

In this, simply a HTTP Connection was established and text extracted using “element_name”.text() from inside the specific tags using identifiers like classes or ids. The tags from which the information was to be extracted were identified after exploring the web page’s HTML source code.

The result thus obtained is in the form of a JSON Object

{
  "Wordpress blog: https://loklaknet.wordpress.com/": [
    {
      "posted_on": "June 19, 2016",
      "blog_url": "https://loklaknet.wordpress.com/",
      "author": "shivenmian",
      "title": "loklak_depot u2013 The Beginning: Accounts (Part 3)",
      "content": "So this is my third post in this five part series on loklak_depo... As always, feedback is duly welcome."
    },
    {
      "posted_on": "June 19, 2016",
      "blog_url": "https://loklaknet.wordpress.com/",
      "author": "sopankhosla",
      "title": "Creating a Loklak App!",
      "content": "Hello everyone! Today I will be shifting from course a...ore info refer to the full documentation here. Happy Coding!!!"
    },
    {
      "posted_on": "June 17, 2016",
      "blog_url": "https://loklaknet.wordpress.com/",
      "author": "leonmakk",
      "title": "Loklak Walls Manual Moderation u2013 tweet storage",
      "content": "Loklak walls are going to....Stay tuned for more updates on this new feature of loklak walls!"
    },
    {
      "posted_on": "June 17, 2016",
      "blog_url": "https://loklaknet.wordpress.com/",
      "author": "Robert",
      "title": "Under the hood: Authentication (login)",
      "content": "In the second post of .....key login is ready."
    },
    {
      "posted_on": "June 17, 2016",
      "blog_url": "https://loklaknet.wordpress.com/",
      "author": "jigyasa",
      "title": "Loklak gives some hackernews now !",
      "content": "It's been befittingly said  u... Also, Stay tuned for more posts on data crawling and parsing for Loklak. Feedback and Suggestions welcome"
    },
    {
      "posted_on": "June 16, 2016",
      "blog_url": "https://loklaknet.wordpress.com/",
      "author": "Damini",
      "title": "Does tweets have emotions?",
      "content": "Tweets do intend some kind o...t of features: classify(feat1,u2026,featN) = argmax(P(cat)*PROD(P(featI|cat)"
    },
    {
      "posted_on": "June 15, 2016",
      "blog_url": "https://loklaknet.wordpress.com/",
      "author": "sudheesh001",
      "title": "Dockerize the loklak server and publish docker images to IBM Containers on Bluemix Cloud",
      "content": "Docker is an open source...nd to create and deploy instantly as well as scale on demand."
    }
  ]
}

 

The next step now would include "writeToBackend"-ing and then parsing the JSONObject as desired.

Feel free to ask questions regarding the above code snippet, shall be happy to assist.

Feedback and Suggestions welcome 🙂

Now get wordpress blog updates with Loklak !