Time across seven seas…

It has been rightly said:

Time is of your own making
Its clock ticks in your head.
The moment you stop thought
Time too stops dead.

loklak_org_sticker

Hence to keep up with evolving times, Loklak has now introduced a new service for “time”.

The recently developed API provides the current time and day at the location queried by the user.

The /api/locationwisetime.json API scrapes the results from timeanddate.com using our favourite JSoup as it provides a very convenient API for extracting and manipulating data, scrape and parse HTML from a given URL.

In case of multiple locations with the same name, countries are then also provided along-with corresponding day and time wrapped up as a JSONObject.

A sample query could then be something like: http://loklak.org/api/locationwisetime.json?query=london

Screenshot from 2016-08-17 14:28:28

 

When implemented as a console service, this API can be used along-with our our dear SUSI by utilising the API Endpoints like: http://loklak.org/api/console.json?q=SELECT * FROM locationwisetime WHERE query=’berlin’;

Screenshot from 2016-08-17 14:50:58

LocationWiseTimeService.java for reference:


/**
 *  Location Wise Time
 *  timeanddate.com scraper
 *  Copyright 27.07.2016 by Jigyasa Grover, @jig08
 *
 *  This library is free software; you can redistribute it and/or
 *  modify it under the terms of the GNU Lesser General Public
 *  License as published by the Free Software Foundation; either
 *  version 2.1 of the License, or (at your option) any later version.
 *  
 *  This library is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 *  Lesser General Public License for more details.
 *  
 *  You should have received a copy of the GNU Lesser General Public License
 *  along with this program in the file lgpl21.txt
 *  If not, see <http://www.gnu.org/licenses/>.
 */

package org.loklak.api.search;

import java.io.IOException;

import javax.servlet.http.HttpServletResponse;

import org.json.JSONArray;
import org.json.JSONObject;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.loklak.server.APIException;
import org.loklak.server.APIHandler;
import org.loklak.server.AbstractAPIHandler;
import org.loklak.server.Authorization;
import org.loklak.server.BaseUserRole;
import org.loklak.server.Query;
import org.loklak.susi.SusiThought;
import org.loklak.tools.storage.JSONObjectWithDefault;

public class LocationWiseTimeService extends AbstractAPIHandler implements APIHandler {

	private static final long serialVersionUID = -1495493690406247295L;

	@Override
	public String getAPIPath() {
		return "/api/locationwisetime.json";
	}

	@Override
	public BaseUserRole getMinimalBaseUserRole() {
		return BaseUserRole.ANONYMOUS;

	}

	@Override
	public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
		return null;
	}

	@Override
	public JSONObject serviceImpl(Query call, HttpServletResponse response, Authorization rights,
			JSONObjectWithDefault permissions) throws APIException {
		String query = call.get("query", "");
		return locationWiseTime(query);
	}

	public static SusiThought locationWiseTime(String query) {
		
		Document html = null;

		JSONArray arr = new JSONArray();

		try {
			html = Jsoup.connect("http://www.timeanddate.com/worldclock/results.html?query=" + query).get();
		} catch (IOException e) {
			e.printStackTrace();
		}

		Elements locations = html.select("td");
		int i = 0;
		for (Element e : locations) {
			if (i % 2 == 0) {
				JSONObject obj = new JSONObject();
				String l = e.getElementsByTag("a").text();
				obj.put("location", l);
				String t = e.nextElementSibling().text();
				obj.put("time", t);
				arr.put(obj);
			}
			i++;
		}
		
		SusiThought json = new SusiThought();
		json.setData(arr);
		return json;
	}

}

 

Hope this helps, and worth the “time” 😛

Feel free to ask questions regarding the above code snippet, shall be happy to assist.

Feedback and Suggestions welcome 🙂

Time across seven seas…

Welcoming Wiki GeoData to Loklak !

Loklak has grown vast with due course of time and it’s capabilities have extended manifold especially by the inclusion of sundry website scraper services and data provider services.

loklak_org_sticker

The recent addition includes a special service which would provide the user with a list of Wikipedia articles tagged with the location when supplied with a specific name of the place.

Thanks to the Media Wiki GeoData API, this service was smoothly integrated in the Loklak Server and SUSI (our very own cute and quirky personal digital assistant)

When the name of the place is sent in the query , firstly the home-grown API loklak.org/api/geocode.json was utilized to get the location co-ordinates i.e. Latitude and Longitude.


URL getCoordURL = null;

String path = "data={\"places\":[\"" + place + "\"]}";

try {
    getCoordURL = new URL("http://loklak.org/api/geocode.json?" + path);
} catch (MalformedURLException e) {
    e.printStackTrace();
}

JSONTokener tokener = null;
try {
    tokener = new JSONTokener(getCoordURL.openStream());
} catch (Exception e1) {
    e1.printStackTrace();
}

JSONObject obj = new JSONObject(tokener);

String longitude = obj.getJSONObject("locations").getJSONObject(place).getJSONArray("location").get(0)
				.toString();
String lattitude = obj.getJSONObject("locations").getJSONObject(place).getJSONArray("location").get(1)
				.toString();

The resultant geographical co-ordinates were then passed on to the Media Wiki GeoData API with other parameters like the radius of the geographical bound to be considered and format of the resultant data along-with the co-ordinates to obtain a list of Page IDs of the corresponding Wikipedia Articles besides Title and Distance.


URL getWikiURL = null;

try {
    getWikiURL = new URL(
                      "https://en.wikipedia.org/w/api.php?action=query&list=geosearch&gsradius=10000&gscoord=" + latitude
			+ "|" + longitude + "&format=json");
} catch (MalformedURLException e) {
    e.printStackTrace();
}

JSONTokener wikiTokener = null;

try {
    wikiTokener = new JSONTokener(getWikiURL.openStream());
} catch (Exception e1) {
    e1.printStackTrace();
}

JSONObject wikiGeoResult = new JSONObject(wikiTokener);

When implemented as a Console Service for Loklak, the servlet was registered as /api/wikigeodata.json?place={place-name} and the API endpoint for example goes like http://localhost:9000/api/console.json?q=SELECT * FROM wikigeodata WHERE place=’Singapore’;

Presto !!

We have a JSON Object as the result with a list of Wikipedia Articles as:

56976976-5e41-11e6-95b6-19e570099739

The Page-IDs thus obtained can now be utilized very easily to diaply the articles by using the placeholder https://en.wikipedia.org/?curid={pageid} for the purpose.

And this way, another facility was included in our diversifying Loklak server.

Questions, feedback, suggestions appreciated 🙂

Welcoming Wiki GeoData to Loklak !

The Making of the Console Service

SUSI , our very own personal digital assistant has been up and running giving quirky answers.

But behind all these are rules which train our cute bot to assist her and decide what answers to provide after parsing the question asked by the users.

The questions could range from any formal-informal greetings, general queries about name, weather, date, time to specific ones like details about some random Github profile or Tweets and Replies  from Twitter, or Weibo or election/football score predictions or simply asking her to read a RSS feed or a WordPress blog for you.

The rules for her training are written after that specific service is implemented which shall help her fetch the particular website/social network in question and scrape data out of it to present it to her operator.

And to help us expand the scope and ability of this naive being, it shall be helpful if users could extend her rule set. For this, it is required to make console service for sites which do not provide access to information without OAuth.

To begin with, let us see how a console service can be made.

Starting with a SampleService class which shall basically include the rudimentary scraper or code fetching the data is defined in the package org.loklak.api.search.
This is made by extending the AbstractAPIHandler class which itself extends the javax.servlet.http.HttpServlet class.
SampleService class further implements APIHandler class.

A placeholder for SampleService class can be as:


package org.loklak.api.search;

/**
* import statements
**/

public class SampleService extends AbstractAPIHandler 
    implements APIHandler{

    private static final long serialVersionUID = 2142441326498450416L;
    /**
     * serialVersionUID could be 
     * auto-generated by the IDE used
    **/

    @Override
    public String getAPIPath() {
        return "/api/service.json";
        /**
         *Choose API path for the service in question
        **/
    }

    @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.ANONYMOUS;
    }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    @Override
    public JSONObject serviceImpl(Query call, HttpServletResponse response, 
        Authorization rights, JSONObjectWithDefault permissions) 
        throws APIException {

        String url = call.get("url", "");
        /**
         *This would extract the argument that will be supplied
         * to the "url" parameter in the "call"
        **/
        return crawlerForService(url);

    }

    public SusiThought crawlerForService(String url) {
        JSONArray arr = new JSONArray();
        
        /**
         * Crawler code or any other function which
         * returns a JSON Array **arr** goes in here 
        **/

        SusiThought json = new SusiThought();
        json.setData(arr);
        return json;
    }

}

 

The JSONArray in the key function crawlerForService is wrapped up in a SusiThought which is nothing but a piece of data that can be remembered. The structure or the thought can be modeled as a table which may be created using the retrieval of information from elsewhere of the current argument.

Now to implement it as a Console Service we include it in the ConsoleService class which is defined in the same package org.loklak.api.search and similarly extends AbstractAPIHandler class and implements APIHandler class.

Here, dbAccess is a static variable of the type SusiSkills where a skill is defined as the ability to inspire, to create thoughts from perception. The data structure of a skill set is a mapping from perception patterns to lambda expressions which induce thoughts.


package org.loklak.api.search;

/**
 * import statements go here
**/

public class ConsoleService extends AbstractAPIHandler 
    implements APIHandler {

    private static final long serialVersionUID = 8578478303032749879L;
    /**
     * serialVersionUID could be 
     * auto-generated by the IDE used
    **/

    @Override
    public BaseUserRole getMinimalBaseUserRole() { 
        return BaseUserRole.ANONYMOUS; 
    }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    public String getAPIPath() {
        return "/api/console.json";
    }

    public final static SusiSkills dbAccess = new SusiSkills();
        static {

            /**
             * Other "skills" are defined here
             * by "putting" them in "dbAccess"
            **/
    
    dbAccess.put(Pattern.compile("SELECT\\h+?(.*?)\\h+?FROM\\h+?
        sampleservice\\h+?WHERE\\h+?url\\h??=\\h??'(.*?)'\\h??;"), 
            (flow, matcher) -> {
                /**
                 * SusiThought-s are fetched from the Services
                 * implemented as above
                **/
                SusiThought json = SampleService.crawlerForService(matcher.group(2));
                SusiTransfer transfer = new SusiTransfer(matcher.group(1));
                json.setData(transfer.conclude(json.getData()));
                return json;
                });
    }

    @Override
    public JSONObject serviceImpl(Query post, HttpServletResponse response, 
        Authorization rights, final JSONObjectWithDefault permissions) 
        throws APIException {

            String q = post.get("q", "");
            /**
             *This would extract the argument that will be supplied
             * to the "q" parameter in the "post" query
            **/
            

            return dbAccess.inspire(q);
        }

}

 

Now that the console service is made, an API endpoint for the same can correspond to: http://localhost:9000/api/console.json?q=SELECT * FROM sampleservice WHERE url = ‘ … ‘;

The above can serve as a placeholder for creating Console Service which shall enable SUSI widen her horizon and become intelligent.

So, Go ahead and make Susi rules using it and you are done !

If any aid is required in making SUSI Rules, stay tuned for the next post.

Come, contribute to Loklak and SUSI !

The Making of the Console Service