The Making of the Console Service

SUSI , our very own personal digital assistant has been up and running giving quirky answers.

But behind all these are rules which train our cute bot to assist her and decide what answers to provide after parsing the question asked by the users.

The questions could range from any formal-informal greetings, general queries about name, weather, date, time to specific ones like details about some random Github profile or Tweets and Replies  from Twitter, or Weibo or election/football score predictions or simply asking her to read a RSS feed or a WordPress blog for you.

The rules for her training are written after that specific service is implemented which shall help her fetch the particular website/social network in question and scrape data out of it to present it to her operator.

And to help us expand the scope and ability of this naive being, it shall be helpful if users could extend her rule set. For this, it is required to make console service for sites which do not provide access to information without OAuth.

To begin with, let us see how a console service can be made.

Starting with a SampleService class which shall basically include the rudimentary scraper or code fetching the data is defined in the package org.loklak.api.search.
This is made by extending the AbstractAPIHandler class which itself extends the javax.servlet.http.HttpServlet class.
SampleService class further implements APIHandler class.

A placeholder for SampleService class can be as:


package org.loklak.api.search;

/**
* import statements
**/

public class SampleService extends AbstractAPIHandler 
    implements APIHandler{

    private static final long serialVersionUID = 2142441326498450416L;
    /**
     * serialVersionUID could be 
     * auto-generated by the IDE used
    **/

    @Override
    public String getAPIPath() {
        return "/api/service.json";
        /**
         *Choose API path for the service in question
        **/
    }

    @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.ANONYMOUS;
    }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    @Override
    public JSONObject serviceImpl(Query call, HttpServletResponse response, 
        Authorization rights, JSONObjectWithDefault permissions) 
        throws APIException {

        String url = call.get("url", "");
        /**
         *This would extract the argument that will be supplied
         * to the "url" parameter in the "call"
        **/
        return crawlerForService(url);

    }

    public SusiThought crawlerForService(String url) {
        JSONArray arr = new JSONArray();
        
        /**
         * Crawler code or any other function which
         * returns a JSON Array **arr** goes in here 
        **/

        SusiThought json = new SusiThought();
        json.setData(arr);
        return json;
    }

}

 

The JSONArray in the key function crawlerForService is wrapped up in a SusiThought which is nothing but a piece of data that can be remembered. The structure or the thought can be modeled as a table which may be created using the retrieval of information from elsewhere of the current argument.

Now to implement it as a Console Service we include it in the ConsoleService class which is defined in the same package org.loklak.api.search and similarly extends AbstractAPIHandler class and implements APIHandler class.

Here, dbAccess is a static variable of the type SusiSkills where a skill is defined as the ability to inspire, to create thoughts from perception. The data structure of a skill set is a mapping from perception patterns to lambda expressions which induce thoughts.


package org.loklak.api.search;

/**
 * import statements go here
**/

public class ConsoleService extends AbstractAPIHandler 
    implements APIHandler {

    private static final long serialVersionUID = 8578478303032749879L;
    /**
     * serialVersionUID could be 
     * auto-generated by the IDE used
    **/

    @Override
    public BaseUserRole getMinimalBaseUserRole() { 
        return BaseUserRole.ANONYMOUS; 
    }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    public String getAPIPath() {
        return "/api/console.json";
    }

    public final static SusiSkills dbAccess = new SusiSkills();
        static {

            /**
             * Other "skills" are defined here
             * by "putting" them in "dbAccess"
            **/
    
    dbAccess.put(Pattern.compile("SELECT\\h+?(.*?)\\h+?FROM\\h+?
        sampleservice\\h+?WHERE\\h+?url\\h??=\\h??'(.*?)'\\h??;"), 
            (flow, matcher) -> {
                /**
                 * SusiThought-s are fetched from the Services
                 * implemented as above
                **/
                SusiThought json = SampleService.crawlerForService(matcher.group(2));
                SusiTransfer transfer = new SusiTransfer(matcher.group(1));
                json.setData(transfer.conclude(json.getData()));
                return json;
                });
    }

    @Override
    public JSONObject serviceImpl(Query post, HttpServletResponse response, 
        Authorization rights, final JSONObjectWithDefault permissions) 
        throws APIException {

            String q = post.get("q", "");
            /**
             *This would extract the argument that will be supplied
             * to the "q" parameter in the "post" query
            **/
            

            return dbAccess.inspire(q);
        }

}

 

Now that the console service is made, an API endpoint for the same can correspond to: http://localhost:9000/api/console.json?q=SELECT * FROM sampleservice WHERE url = ‘ … ‘;

The above can serve as a placeholder for creating Console Service which shall enable SUSI widen her horizon and become intelligent.

So, Go ahead and make Susi rules using it and you are done !

If any aid is required in making SUSI Rules, stay tuned for the next post.

Come, contribute to Loklak and SUSI !

The Making of the Console Service

Spin-Off: Loklak fuels Open Event

Continuing with the Loklak & Open Event Partnership (check out Loklak fuels Open Event ), we can now easily in clicks create our very own web-app for the event with details imported from eventbrite.com powered by Loklak.

The scraping of data done using JSoup, Java HTML parsing was explained in the previous post of this series.

Next, a console service was implemented as the single point for information retrieval from various social networks and websites (a post coming for it soon 😉 ) especially for SUSI (our very own personal digital assistant, a cute one indeed !)

The JSONArray result of the EventBriteCrawler was set in SusiThought, which is nothing but a piece of data that can be remembered. The structure or the thought can be modelled as a table which may be created using the retrieval of information from elsewhere of the current argument.


/** Defining SusiThought as a class 
 * which extends JSONObject
 */

public class SusiThought extends JSONObject {

/* details coming soon.... */

}

/** Modifications in EventBriteCrawler
 *  Returning SusiThought instead of 
 * a simple JSONObject/JSONArray.
 */
public static SusiThought crawlEventBrite(String url) {
    ...
    ...    
    SusiThought json = new SusiThought();
    json.setData(jsonArray);
    return json;
}

 

The API EndPoint was thus created.
A sample is as: http://loklak.org/api/console.json?q=SELECT * FROM eventbrite WHERE url=’https://www.eventbrite.fr/e/billets-europeade-2016-concert-de-musique-vocale-25592599153′;

Screenshot from 2016-07-15 13:22:00

 

The files generated were next imported in the Open Event Web App generator, using simple steps.

screenshot-from-2016-07-04-075700

Screenshot from 2016-07-15 13:25:39 Screenshot from 2016-07-15 13:36:19

It’s amazing to see how a great visual platform is provided to edit details parsed from the plain JSONObject and deploy the personalized web-app !

Screenshot from 2016-07-15 13:59:47

Screenshot from 2016-07-15 12:55:06Screenshot from 2016-07-15 12:55:18Screenshot from 2016-07-15 12:55:29Screenshot from 2016-07-15 12:54:24
Screenshot from 2016-07-15 12:54:33
Tadaa !
We have our very own event web-app with all the information imported from eventbrite.com in a single (well, very few 😛 ) click (s) !

With this, we conclude the Loklak – Open Event – EventBrite series.

Stay tuned for detailed post on SUSI and Console Services 🙂

Spin-Off: Loklak fuels Open Event

Teach Susi some rules

“I’m a young, naive being, sometimes unsure about my own statements, sometimes quirky and maybe rough, provocative in cases where the person who talks to me is provocative as well. In other cases I’m a friendly and a bit cute being”, says Susi.

She needs some help to train her personality and give the best assistance to it’s users, at the same time offers her help. So this blog post helps you on how to can get her help according to your needs.

Here is the data stub, susi_cognition_000.json, where you can teach susi all the manners. Like, produce some rules. There are few interesting things one must know before teaching her the rules.

{
keys :[belief],
score :1,
phrases:[ {type:pattern, expression:*I feel *} ],
process:[],
actions:[ {type:answer, select:random, phrases:[
Why do you feel $2$?
]}]
},

“keys” – These are the words list which helps a rule to get categorized. Like a particular rule can be categorized into happy when the keys are happy, joyous, festive.  The above is an example of a rule(JSON object), which has the following keys.

  • “score” The score determines the priority of the rule. When there are multiple rules under same category, these gets prioritized based on the score. So when there are similar rules out there, the score tags play it’s required role.
  • “phrases” – This sub object has more fields namely type and expression. Type can be a pattern or a regular expression, defining the kind of expression which you are giving. Suppose the expression is a pattern like the above example *I feel * then the type will be a pattern. similarly for regular expressions.
  • “process” – Will explain about the process in the next example rule.
  • “actions” – This has fields like type , select and phrases. The type here defines the type of the answer to be given such as text(answer), pie charts and tables. (the current supported features). select tag ensures the random select of the answer and the phrases contains the list of possible answer styles.

The above example gives you the answer, on using the Susi service.

http://loklak.org/api/susi.json?q=i feel super happy

{
  "session": {"identity": {
    "type": "host",
    "name": "127.0.0.1",
    "anonymous": true
  }},
  "count": 1,
  "answers": [{
    "metadata": {
      "hits": 1,
      "offset": 0,
      "count": 1
    },
    "data": [{
      "0": "i feel super happy",
      "1": "",
      "2": "super happy"
    }],
    "actions": [{
      "expression": "Why do you feel super happy?",
      "type": "answer"
    }]
  }]
}

The above JSON is the result of the susi result. Woah this is how susi replies back. So the answer is here.

"actions": [{
      "expression": "Why do you feel super happy?",
      "type": "answer"
}]

The above answer format has a data

This is a sample example. Let’s dive into some more rules with different formats.

{
keys :[how],
score :100,
example:How many mentions are on reddit about loklak,
phrases:[ {type:pattern, expression:How many mentions are on reddit about *}
],
process:[ {type:console, expression:SELECT COUNT(*) AS count FROM rss WHERE url=’https://www.reddit.com/search.rss?q=$1$’;}],
actions:[ {type:answer, select:random, phrases:[
Here you go, $count$ times!
]}]
},

The above example is a score 100 rule with a high priority range than the previous example. This rule will try to answer the query related to Reddit mentions of a particular tag or a name. Here are few key differences to be noted.

  • “example” – The example key gives a sample query which is useful for understanding about the rule’s input.
  • “process” – This key is being used when you have to get the data from an external service. The above example is trying to query the tables being present using the SQL syntax. It is a console type of querying where the count is being calculated from the rss table from where the data can be retrieved from the URL. (Standard format for rss reader which is being supported by loklak: https://www.reddit.com/search.rss?q=term; 

For the above rule, susi response is this,

http://loklak.org/api/susi.json?q=How many mentions are on reddit about loklak

{
  "session": {"identity": {
    "type": "host",
    "name": "127.0.0.1",
    "anonymous": true
  }},
  "count": 1,
  "answers": [{
    "data": [{"count": 5}],
    "metadata": {"count": 1},
    "actions": [{
      "expression": "Here you go, 5 times!",
      "type": "answer"
    }]
  }]
}

And here’s the required answer :

"actions": [{
      "expression": "Here you go, 5 times!",
      "type": "answer"
 }]

So till now we have seen examples where we can retrieve text answers, Here comes an example which can return a table or a list of answers.

{
keys :[reddit],
score :100,
example:What are the reddit articles about loklak,
phrases:[ {type:pattern, expression:* reddit * about *}
],
process:[ {type:console, expression:SELECT title FROM rss WHERE url=’https://www.reddit.com/search.rss?q=$3$’;}],
actions:[ {type:answer, select:random, phrases:[
Here is the list of articles about $3$
]}, {type:table}]
},

For the above result you can even mention about type being table for getting the list of data. So here is the Susi’s response

http://loklak.org/api/susi.json q=What are the reddit articles about loklak

{
  "session": {"identity": {
    "type": "host",
    "name": "127.0.0.1",
    "anonymous": true
  }},
  "count": 1,
  "answers": [{
    "data": [
      {"title": "programming"},
      {"title": "Datasets Archive"},
      {"title": "We collected 1.3 billion tweets with a distributed, peer-to-peer based free, open source twitter scraper that has a nice API for your self-made apps to evaluate the data: loklak.org"},
      {"title": "A look at Loklak.org and the Webclient progress"},
      {"title": "#Loklak Web client Test A Twitter search engine loklak.org"}
    ],
    "metadata": {"count": 5},
    "actions": [
      {
        "expression": "Here is the list of articles about loklak",
        "type": "answer"
      },
      {"type": "table"}
    ]
  }]
}

So the data object under answers gives you the list of titles tagged under loklak (PS: This is from Reddit)

Another example for getting data, using which one can form pie charts.

{
keys :[president,election,america],
score :100,
comment:a statistical app which tries to predict the american presidential election,
phrases:[ {type:regex, expression:Who will win the 2016 presidential election},
{type:regex, expression:Who will (?:be|become) the next us president}
],
process:[ {type:console, expression:SELECT PERCENT(count) AS percent, hashtag AS president FROM (SELECT COUNT(*) AS count, hashtags AS hashtag FROM messages WHERE query=’election2016′ GROUP BY hashtags) WHERE hashtag IN (‘hillaryclinton’,’berniesanders’,’donaldtrump’);}],
actions:[ {type:answer, select:random, phrases:[
I believe the next president will be $president$ with a likelihood of $percent$ percent but I a not sure.,
I can calculate a likelihood, here is my guess:
]},
{type:piechart, total:100, key: country, value:percent, unit:%}]
},

So the above example gives you the predictions for the US elections. Here you go, Susi’s response.

http://loklak.org/api/susi.json?q=Who will win the 2016 presidential election

{
  "session": {"identity": {
    "type": "host",
    "name": "10.67.93.57",
    "anonymous": true
  }},
  "count": 1,
  "answers": [{
    "metadata": {
      "hits": 21,
      "offset": 0,
      "count": 2
    },
    "data": [
      {
        "percent": 50,
        "president": "hillary2016"
      },
      {
        "percent": 50,
        "president": "trump2016"
      }
    ],
    "actions": [
      {
        "expression": "I believe the next president will be hillary2016 with a likelihood of 50.0 percent but I a not sure.",
        "type": "answer"
      },
      {
        "total": 100,
        "unit": "%",
        "type": "piechart",
        "value": "percent",
        "key": "country"
      }
    ]
  }]
}

Important Nomenclature for defining the variables

Suppose the expression is of the form

* reddit * about *

Using the above expression the following sentences can be formed. “What are the Reddit articles about Loklak” So the first set of words before reddit can be defined as $1$ variable, the set of words between reddit and about can be defined as $2$ variable and the words after about can be $3$ variable, so on. This is how the variable system works. As a whole, the complete sentence can be considered as a $0$ variable.

Some important data sources for console querying

 * http://loklak.org/api/console.json?q=SELECT%20text,%20screen_name,%20user.name%20AS%20user%20FROM%20messages%20WHERE%20query=%271%27;
 * http://loklak.org/api/console.json?q=SELECT%20*%20FROM%20messages%20WHERE%20id=%27742384468560912386%27;
 * http://loklak.org/api/console.json?q=SELECT%20link,screen_name%20FROM%20messages%20WHERE%20id=%27742384468560912386%27;
 * http://loklak.org/api/console.json?q=SELECT%20COUNT(*)%20AS%20count,%20screen_name%20AS%20twitterer%20FROM%20messages%20WHERE%20query=%27loklak%27%20GROUP%20BY%20screen_name;
 * http://loklak.org/api/console.json?q=SELECT%20PERCENT(count)%20AS%20percent,%20screen_name%20FROM%20(SELECT%20COUNT(*)%20AS%20count,%20screen_name%20FROM%20messages%20WHERE%20query=%27loklak%27%20GROUP%20BY%20screen_name)%20WHERE%20screen_name%20IN%20(%27leonmakk%27,%27Daminisatya%27,%27sudheesh001%27,%27shiven_mian%27);
 * http://loklak.org/api/console.json?q=SELECT%20query,%20query_count%20AS%20count%20FROM%20queries%20WHERE%20query=%27auto%27;
 * http://loklak.org/api/console.json?q=SELECT%20*%20FROM%20users%20WHERE%20screen_name=%270rb1t3r%27;
 * http://loklak.org/api/console.json?q=SELECT%20place[0]%20AS%20place,%20population,%20location[0]%20AS%20lon,%20location[1]%20AS%20lat%20FROM%20locations%20WHERE%20location=%27Berlin%27;
 * http://loklak.org/api/console.json?q=SELECT%20*%20FROM%20locations%20WHERE%20location=%2753.1,13.1%27;
 * http://loklak.org/api/console.json?q=SELECT%20description%20FROM%20wikidata%20WHERE%20query=%27football%27;
 * http://loklak.org/api/console.json?q=SELECT%20*%20FROM%20meetup%20WHERE%20url=%27http://www.meetup.com/?q=Women-Who-Code-Delhi%27;
 * http://loklak.org/api/console.json?q=SELECT%20*%20FROM%20rss%20WHERE%20url=%27https://www.reddit.com/search.rss?q=loklak%27;
 * http://loklak.org/api/console.json?q=SELECT%20*%20FROM%20eventbrite%20WHERE%20url=%27url=https://www.eventbrite.com/e/?q=global-health-security-focus-africa-tickets-25740798421%27;

The above are the supported data stores and queries. You can have a look at ConsoleService.java

That’s it and this is how you can train Susi and teach her rules !

These would be useful for knowing more about Susi.

What is Susi?

Susi is an artificial intelligence combining pattern matching, internet data, data flow principles and inference engine principles. It will have some reflection abilities and it will be able to remember the users input to produce deductions and a personalized feed-back.

Why is it built?

To create a personal assistant software like Siri and Cortana, made by the people for the people. Its an free software open mind which expands with the help of it’s users.
What is it’s purpose?

To explore the abilities of an artificial companion and to answer the remaining unanswered questions.

 

Teach Susi some rules