How Susi linkifies the link?

Susi responses contains links to open street maps as of now. Previously the UI was not able to handle the links gracefully.

0c013146-4cf6-11e6-91a7-57015406412e

The links are not linkified properly. Suppose if i want to check the contents on the link, i need to copy the link text and check out the contents in it, which is a very bad UX. This blog post explains on how Susi deals with linkifying the links gracefully without using any third party plugins or JS libs.

When I started off with the process of fixing this, i searched for exisiting js library which can help me do this job. I came across Linkify JS library for adding the Hyperlinks to texts and avoiding the plain URL displays. But found that the library is pretty huge and quite convoluted to implement it. Instead of using this, i tried out solving this issue with simple regular expressions.

How I implemented this? 

I made a generic check for the linkification. Most of the URLs start with the following protocols.

  • http://
  • https://
  • ftp://
  • www.
  • mailto::

The following regular expression checks for URLs starting with http://, https://, or ftp://

/(\b(https?|ftp):\/\/[-A-Z0-9+&@#\/%?=~_|!:,.;]*[-A-Z0-9+&@#\/%=~_|])/gim;

The following regular expression for URLs starting with “www.” (without // before it, or it’d re-link the ones done above)

/(^|[^\/])(www\.[\S]+(\b|$))/gim;

The following regular expressions change email addresses to mailto:: links

/(([a-zA-Z0-9\-\_\.])[email protected][a-zA-Z\_]+?(\.[a-zA-Z]{2,6})+)/gim;

These regular expressions follow the format for each of the different URL formats mentioned, places the link into the hrefs.

//URLs starting with http://, https://, or ftp://
replacePattern1 = /(\b(https?|ftp):\/\/[-A-Z0-9+&@#\/%?=~_|!:,.;]*[-A-Z0-9+&@#\/%=~_|])/gim;
replacedText = replacedText.replace(replacePattern1, '<a href="$1" target="_blank">Click Here!</a>');

//URLs starting with "www." (without // before it, or it'd re-link the ones done above).
replacePattern2 = /(^|[^\/])(www\.[\S]+(\b|$))/gim;
replacedText = replacedText.replace(replacePattern2, '$1<a href="http://$2" target="_blank">$3$</a>');

//Change email addresses to mailto:: links.
replacePattern3 = /(([a-zA-Z0-9\-\_\.])[email protected][a-zA-Z\_]+?(\.[a-zA-Z]{2,6})+)/gim;
replacedText = replacedText.replace(replacePattern3, '<a href="mailto:$1">$3$</a>');

At the same time the text is being wrapped inside the pre tags. Here is the final result without using any libraries and simply fixing the issue using three different regular expressions.

b9dd6c06-4d39-11e6-907d-29945fc8f4c5

 

How Susi linkifies the link?

Susi supports Map tiles

Susi chat client supports map tiles now. Try out the following query related to location and Susi responses with a internal map tile with the pin at the location.

Where is Singapore? 

Selection_310

You have internal zoom in and zoom out options for the map. It also provides you with a link to open street maps where you can get the whole view of the location. Other than the map tile it gives you information about the location’s population. Isn’t it awesome?

Implementation: Let’s get into the implementation part. There are multiple ways to display the map tiles. One way is we can use our own Loklak services for displaying the map. For example -> you can make a call to the /vis/map.png API for displaying the map. But the issue with this API is we cant dynamically zoom in or zoom out the map on the tile, which is moreover a static display. So to make it more interesting we used a js library called Leaflet, which provides interactive maps.

How is the data captured? Here is the sample response coming from the server.

Selection_307

This sample data which contains latitude, longitude, place and population helps us for drawing the map tiles.

  1. expression:“Berlin is a place with a population of 3426354. Here is a map: https://www.openstreetmap.org/#map=13/52.52436820069531/13.41053001275776”
  2. type:“answer”

The above two keys are under the actions object providing the us the answer along with the URL where it is linkified.

How is the map captured? First we included the following required library files.

<script src=”https://npmcdn.com/[email protected]/dist/leaflet.js”></script>

<link rel=”stylesheet” href=”https://npmcdn.com/[email protected]/dist/leaflet.css” />

Selection_312

The JSON response is parsed and the co ordinates (lat, lon) are captured. The variable type mapType initialized.

Selection_313

PS: The co ordinates are passed into the response, for telling the html that ‘hey the map is coming in the response!’ so that it prepares it’s space for the map. Here’s the html’s job. (Handlebars)

Selection_314

The extra div for the maps are loaded when intimated about the map.

Selection_316

When the mapType is set to true the drawMap method is called which initializes the id for the the html and paints the map tile to it. The object has attributes like maxZoom levels, Map marker and it’s tooltip on the map. And that’s how the map tiles are formed.

 

 

 

Susi supports Map tiles

Time across seven seas…

It has been rightly said:

Time is of your own making
Its clock ticks in your head.
The moment you stop thought
Time too stops dead.

loklak_org_sticker

Hence to keep up with evolving times, Loklak has now introduced a new service for “time”.

The recently developed API provides the current time and day at the location queried by the user.

The /api/locationwisetime.json API scrapes the results from timeanddate.com using our favourite JSoup as it provides a very convenient API for extracting and manipulating data, scrape and parse HTML from a given URL.

In case of multiple locations with the same name, countries are then also provided along-with corresponding day and time wrapped up as a JSONObject.

A sample query could then be something like: http://loklak.org/api/locationwisetime.json?query=london

Screenshot from 2016-08-17 14:28:28

 

When implemented as a console service, this API can be used along-with our our dear SUSI by utilising the API Endpoints like: http://loklak.org/api/console.json?q=SELECT * FROM locationwisetime WHERE query=’berlin’;

Screenshot from 2016-08-17 14:50:58

LocationWiseTimeService.java for reference:


/**
 *  Location Wise Time
 *  timeanddate.com scraper
 *  Copyright 27.07.2016 by Jigyasa Grover, @jig08
 *
 *  This library is free software; you can redistribute it and/or
 *  modify it under the terms of the GNU Lesser General Public
 *  License as published by the Free Software Foundation; either
 *  version 2.1 of the License, or (at your option) any later version.
 *  
 *  This library is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 *  Lesser General Public License for more details.
 *  
 *  You should have received a copy of the GNU Lesser General Public License
 *  along with this program in the file lgpl21.txt
 *  If not, see <http://www.gnu.org/licenses/>.
 */

package org.loklak.api.search;

import java.io.IOException;

import javax.servlet.http.HttpServletResponse;

import org.json.JSONArray;
import org.json.JSONObject;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.loklak.server.APIException;
import org.loklak.server.APIHandler;
import org.loklak.server.AbstractAPIHandler;
import org.loklak.server.Authorization;
import org.loklak.server.BaseUserRole;
import org.loklak.server.Query;
import org.loklak.susi.SusiThought;
import org.loklak.tools.storage.JSONObjectWithDefault;

public class LocationWiseTimeService extends AbstractAPIHandler implements APIHandler {

	private static final long serialVersionUID = -1495493690406247295L;

	@Override
	public String getAPIPath() {
		return "/api/locationwisetime.json";
	}

	@Override
	public BaseUserRole getMinimalBaseUserRole() {
		return BaseUserRole.ANONYMOUS;

	}

	@Override
	public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
		return null;
	}

	@Override
	public JSONObject serviceImpl(Query call, HttpServletResponse response, Authorization rights,
			JSONObjectWithDefault permissions) throws APIException {
		String query = call.get("query", "");
		return locationWiseTime(query);
	}

	public static SusiThought locationWiseTime(String query) {
		
		Document html = null;

		JSONArray arr = new JSONArray();

		try {
			html = Jsoup.connect("http://www.timeanddate.com/worldclock/results.html?query=" + query).get();
		} catch (IOException e) {
			e.printStackTrace();
		}

		Elements locations = html.select("td");
		int i = 0;
		for (Element e : locations) {
			if (i % 2 == 0) {
				JSONObject obj = new JSONObject();
				String l = e.getElementsByTag("a").text();
				obj.put("location", l);
				String t = e.nextElementSibling().text();
				obj.put("time", t);
				arr.put(obj);
			}
			i++;
		}
		
		SusiThought json = new SusiThought();
		json.setData(arr);
		return json;
	}

}

 

Hope this helps, and worth the “time” 😛

Feel free to ask questions regarding the above code snippet, shall be happy to assist.

Feedback and Suggestions welcome 🙂

Time across seven seas…

Susi support for Loklak APIs

Here at Loklak, we are striving continuously for innovation. Continuing with this trend, we recently launched ‘Susi – The chat bot’. Please refer to this previous blog post by Damini.

Along with the chat bot, Susi query support was added to Loklak Python and PHP APIs. Susi can be queried from localhost as well as other online loklak peers.

Susi API function added to Python API(as shown below). See full implementation here.

def susi(self, query=None):
   """Hits Susi with the required query and returns back the susi response"""
   susi_application = 'api/susi.json'
   url_to_give = self.baseUrl + susi_application
   self.query = query
   if query:
      params = {}
      params['q'] = self.query
      return_to_user = requests.get(url_to_give, params=params)
      if return_to_user.status_code == 200:
          return return_to_user.json()
      else:
          return_to_user = {}
          return_to_user['error'] = ('Looks like there is a problem in susi replying.')
          return json.dumps(return_to_user)
      else:
          return_to_user = {}
          return_to_user['error'] = ('Please ask susi something.')
          return json.dumps(return_to_user)

A sample usage of Susi API in python could be:

from loklak import Loklak
query = "Hi I am Zeus"
l = Loklak()
result = l.susi(query)
print result

Susi integration with PHP API(see below). See full implementation here.

public function susi($query=null) {
	$this->requestURL = $this->baseUrl . '/api/susi.json';
	$this->query = $query;
	if($query) {
		$params = array('q'=>$this->query);
		$request = Requests::request($this->requestURL, array('Accept' => 'application.json'), $params);
		if ($request->status_code == 200) {
			return json_encode($request, true);
		}
		else {
			$request = array();
			$error = "Looks like Susi is not replying.";
			$request['error'] = array_push($request, $error);
			return json_encode($request, true);
		}
	}
	else {
		$request = array();
		$error = "Please ask Susi something.";
		$request['error'] = array_push($request, $error);
		return json_encode($request, true);
	}
}

Sample usage of Susi API in PHP:

include('loklak.php');
$loklak = new Loklak(); 
$result = $loklak->susi('Hi I am Zeus');
$susiResponse = json_decode($result);
$susiResponse = $susiResponse->body;
$susiResponse = json_decode($susiResponse, true);
var_dump($susiResponse);

Tests for above-mentioned functions have been added to the respective API suite. Refer to this and this.

Try Social Universe Super Intelligence!

Ask questions, interact with it. I am pretty sure that you would like it!

Susi support for Loklak APIs

Susi Rule-Score Hierarchy Brainstorming

For Susi’s score system, we need a hierarchy to assign good score values to the rules. To do so we should develop a hierarchy to find an easy system that can be used to assign scores to new rules.

Please add your suggestions below, as you add your ideas we will change a score hierarchy suggestion below.

Preliminary Consideration: Patterns

We have two kinds of rules: such with patterns and others without. The meansing of such rules are:

with pattern(s):

  • (P+LR) variables in pattern should be used for retrieval in internal Susi’s log (reflection memory)
  • (P+IR) variables in pattern should be used for retrieval in internal databases
  • (P+ER) variables in pattern should be used for retrieval in external databases
  • (P+LS) variables in pattern should be stored in Susi’s memory to be used for reflection later
  • (P+IS) variables in pattern should be stored in internal databases to be used for retrieval later
  • (P+ES) variables in pattern should be stored in external databases to be used for retrieval later

without any pattern:

  • (P-D) default answers if no other rule applies
  • (P-O) overruling of rules which would apply, but should not

Secondary Consideration: Purpose

We have three kinds of purposes for Susi answers:

  • (/A) to answer on the users question
  • (/Q) to ask a question to the user in the context of an objective within Susi’s planning to do a conversation
  • (/R) to answer on an answer of the user within the planning of Susi to do a conversation. It appears clear that answers in the context of a Susi conversation strategy should have higher priority.

Combinations of Pattern and Purpose Considerations:

To combine the various Pattern and Purpose types, we write the abbreviations of these items together. For example, we want to answer on a question of the user “Are you happy” with “Yes!, Are you happy as well?” which would be an rule of type P-O/Q. The combination of the both consideration types give 8×3=24 possibilities.

Score Hierarchy

I believe there should be
– score(R) > score(Q) > score(A):
to do a steering of conversations within a conversation plan.
– score(P-O) > score(P+?) > score(P-D):
overruling of pattern-directed answers and default answers in case of pattern fail
– score(P+?S) > score(P+?R):
storing of information (= learning) is more important than answering
– score(P+L?) > score(P+I?) > score(P+E?):
using local information is primary above external information. Reflection is most important.

This produces the following order (with decreasing score, first line has highest sore):

– Overruling of patterns:
– R/P-O
– Q/P-O
– A/P-O

– Answer on an Answer of the user using patterns, possibly learning, otherwise retrieving data
– R/P+LS
– R/P+IS
– R/P+ES
– R/P+LR
– R/P+IR
– R/P+ER

– Asking the user a question with the purpose of learning with the users answer
– Q/P+LS
– Q/P+IS
– Q/P+ES
– Q/P+LR
– Q/P+IR
– Q/P+ER

– Just giving an answer to the question of the user
– A/P+LS
– A/P+IS
– A/P+ES
– A/P+LR
– A/P+IR
– A/P+ER

– Fail-over if no other rule apply to just answer anything, but try to start a new conversation
– R/P-D
– Q/P-D
– A/P-D

Susi Rule-Score Hierarchy Brainstorming

Susi chat interface with visualizations

Susi got few capabilities to visualize it’s response. She can respond by sharing links, showing analytics on pie charts and give you a list of bulleted data. So this post shows you on how these components are integrated into Susi.

The rules which are defined can give data in various compatible forms. It can give links, share some analytics in the form of percentages and certain list of data. For example, in the previous blog post on adding susi rules, we added a sample rule showing on how to add types of responses to susi. If you want more context on it, you can click here.

  • Susi taking responses from data: This type of response is in the form of a table. Susi can take the extra data under the data.answers[0].data , where the type if of table. The below is a sample JSON format from which the tabular data can be parsed.

Selection_251

From the above JSON the data under the answers object is being traced for the tabulated answers. This expression will get you the titles of the reddit articles.

Selection_252

The above response is for the following query

What are the reddit articles about loklak

This is Susi’s response on asksusi.

Selection_249

  • Susi answering using piecharts: Susi rules can also be defined in such a way that the response can give out a well formed pie chart. The data required for the piechart is defined and this can easily be interpreted using highcharts for giving a clear pie chart response. Here is sample JSON response for the following query.
Who will win the 2016 presidential election

Selection_234

The above JSON defines data for piecharts giving percentage and relevant name for that particular object. This is easy to interpret the json for defining the piecharts using highchart.js . The below is the sample code which was used to define the piecharts.

Selection_253

This is how the interface answers with piecharts.

Selection_247

Selection_248

  • So susi can also interpret links from the response and linkify them accordingly.

Here is the sample code on how Susi interprets the links from the response.

Selection_254

The links are linkified and this how Susi responses.

Selection_255

Stay tuned for more updates on Susi.

 

 

Susi chat interface with visualizations