Adding Unit Tests for Services in loklak search

In Loklak search, it can be tricky to write tests for services as these services are customizable and not fixed. Therefore, we need to test every query parameter of the URL. Moreover, we need to test if service is parsing data in a correct manner and returns only data of type ApiResponse.

In this blog here, we are going to see how to build different components for unit testing services. We will be going to test Search service in loklak search which makes Jsonp request to get the response from the loklak search.json API which are displayed as feeds on loklak search. We need to test if the service handles the response in a correct way and if the request parameters are exactly according to customization.

Service to test

Search service in loklak search is one of the most important component in the loklak search. SearchService is a class with a method fetchQuery() which takes parameter and sets up URL parameters for the search.json API of loklak. Now, it makes a JSONP request and maps the API response. The Method fetchQuery() can be called from other components with parameters query and lastRecord to get the response from the server based on a certain search query and the last record to implement pagination feature in loklak search. Now as the data is retrieved, a callback function is called to access the response returned by the API. Now, the response received from the server is parsed to JSON format data to extract data from the response easily.

@Injectable()
export class SearchService {
private static readonly apiUrl: URL = new URL(‘http://api.loklak.org/api/search.json’);
private static maximum_records_fetch = 20;
private static minified_results = true;
private static source = ‘all’;
private static fields = ‘created_at,screen_name,mentions,hashtags’;
private static limit = 10;
private static timezoneOffset: string = new Date().getTimezoneOffset().toString();constructor(
private jsonp: Jsonp
) { }// TODO: make the searchParams as configureable model rather than this approach.
public fetchQuery(query: string, lastRecord = 0): Observable<ApiResponse> {
const searchParams = new URLSearchParams();
searchParams.set(‘q’, query);
searchParams.set(‘callback’, ‘JSONP_CALLBACK’);
searchParams.set(‘minified’, SearchService.minified_results.toString());
searchParams.set(‘source’, SearchService.source);
searchParams.set(‘maximumRecords’, SearchService.maximum_records_fetch.toString());
searchParams.set(‘timezoneOffset’, SearchService.timezoneOffset);
searchParams.set(‘startRecord’, (lastRecord + 1).toString());
searchParams.set(‘fields’, SearchService.fields);
searchParams.set(‘limit’, SearchService.limit.toString());
return this.jsonp.get(SearchService.apiUrl.toString(), { search: searchParams })
.map(this.extractData)}private extractData(res: Response): ApiResponse {
try {
return <ApiResponse>res.json();
} catch (error) {
console.error(error);
}
}

Testing the service

  • Create a mock backend to assure that we are not making any Jsonp request. We need to use Mock Jsonp provider for this. This provider sets up MockBackend and wires up all the dependencies to override the Request Options used by the JSONP request.

const mockJsonpProvider = {
provide: Jsonp,
deps: [MockBackend, BaseRequestOptions],
useFactory: (backend: MockBackend, defaultOptions: BaseRequestOptions) => {
return new Jsonp(backend, defaultOptions);
}
};

 

  • Now, we need to configure the testing module to isolate service from other dependencies. With this, we can instantiate services manually. We have to use TestBed for unit testing and provide all necessary imports/providers for creating and testing services in the unit test.

describe(‘Service: Search’, () => {
let service: SearchService = null;
let backend: MockBackend = null;
beforeEach(() => {
TestBed.configureTestingModule({
providers: [
MockBackend,
BaseRequestOptions,
mockJsonpProvider,
SearchService
]
});
});

 

  • Now, we will inject Service (to be tested) and MockBackend into the Testing module. As all the dependencies are injected, we can now initiate the connections and start testing the service.

beforeEach(inject([SearchService, MockBackend], (searchService: SearchService, mockBackend: MockBackend) => {
service = searchService;
backend = mockBackend;
}));

 

  • We will be using it() block to mention about what property/feature we are going to test in the block. All the tests will be included in this block. One of the most important part is to induce callback function done which will close the connection as soon the testing is over.

it(‘should call the search api and return the search results’, (done)=>{
// test goes here
});

 

  • Now, we will create a connection to the MockBackend and subscribe to this connection. We need to configure ResponseOptions so that mock response is JSONified and returned when the request is made.  Now, the MockBackend is set up and we can proceed to make assertions and test the service.

const result = MockResponse;
backend.connections.subscribe((connection: MockConnection) => {
const options = new ResponseOptions({
body: JSON.stringify(result)
});
connection.mockRespond(new Response(options));

 

  • We can now add test by using expect() block to check if the assertion is true or false. We will now test:
    • Request method: We will be testing if the request method used by the connection created is GET.

expect(connection.request.method).toEqual(RequestMethod.Get);
    • Request Url: We will be testing if all the URL Search Parameters are correct and according to what we provide as a parameter to the method fetchQuery().

expect(connection.request.url).toEqual(
`http://api.loklak.org/api/search.json` +
`?q=${query}` +
`&callback=JSONP_CALLBACK` +
`&minified=true&source=all` +
`&maximumRecords=20&timezoneOffset=${timezoneOffset}` +
`&startRecord=${lastRecord + 1}` +
`&fields=created_at,screen_name,mentions,hashtags&limit=10`);
});
);

 

  • Response:  Now, we need to call the service to make a request to the backend and subscribe to the response returned. Next, we will make an assertion to check if the response returned and parsed by the service is equal the Mock Response that should be returned. At the end, we need to call the callback function done() to close the connection.

service
.fetchQuery(query, lastRecord)
.subscribe((res) => {
expect(res).toEqual(result);
done();
});
});

Reference

Adding Unit Tests for Services in loklak search

Setting up Susi’s capabilities on Facebook Messenger

Facebook’s messenger platform is a great way to reach out to a lot of people from a page that one owns on facebook. The messenger services reach out to almost 900 million people who use the system, that’s a humongous set of people to which Susi’s capabilities could be reached out to if integrated with the facebook messenger and that’s exactly what has been done. Susi is the AI System running in Loklak which contains rules to run the required scrapers or fetch the information directly from facebook. To set this up, we created a new repository deployed on heroku called asksusi_messengers which is going to be a collection of such messenger integrations i.e. to Facebook, Slack, Whatsapp, Telegram etc.., So that the power of mobile and messaging services can be used to make Susi smarter and ways in which people can consume Susi’s capabilities.

Considering the real time nature of people and messages, we have used node.js to take requests by using a webhook on facebook which subscribes to the changes or any event that triggers from the asksusisu facebook page. So here’s the way they really work. Messenger bots uses a web server to process messages it receives or to figure out what messages to send. You also need to have the bot be authenticated to speak with the web server and the bot approved by Facebook to speak with the public. This means that we create the messenger service with facebook apps and then register the endpoint so that facebook can trigger that URL endpoint more like a webhook so that the messages that were sent by the user can be sent to Susi’s AI Service.

Using express this is pretty simple and can be accomplished by the following piece of code that listens to the root of the application.
app.get('/', function (req, res) {
res.send('Susi says Hello.');
});

This ensures that the application is active for a user trying to hit the GET endpoint of the service and as a security method, the rest of the application endpoints need to be on a POST endpoints so that the application’s responses are secure and not anyone can send requests without the required tokens.

Facebook needs an SSL based hostname so that the service can be binded for this. The fastest way this can be spun up is by using the heroku deployments, Hence we use a ProcFile with the contents in it as

web: node index.js

Then configure the application on facebook developers with the given name and setup a messenger webhook over there to send an event to the heroku server that you just deployed, Added to this add your own token which you need to remember and put up on heroku too. After this you will receive a page access token which you can temporarily save somewhere and later push to heroku as a configuration. You can read this configuration by doing and facebook’s verification works.


var token = process.env.FB_PAGE_ACCESS_TOKEN;

// for facebook verification
app.get('/webhook/', function (req, res) {
if (req.query['hub.verify_token'] === 'this_is_my_top_secret_token_that_i_know') {
res.send(req.query['hub.challenge']);
}
res.send('Error, wrong token');
});

You can then receive the information from facebook’s events as follows on a POST request endpoint to the webhook

// to post data
app.post('/webhook/', function (req, res) {
var messaging_events = req.body.entry[0].messaging;
}

The messaging_events gets the required information from facebook in

event.message

and

event.message.text

objects of the triggered webhook event. The query to susi is then constructed


// Construct the query for susi
var queryUrl = 'http://loklak.org/api/susi.json?q='+encodeURI(text);
var message = '';
// Wait until done and reply
request({
url: queryUrl,
json: true
}, function (error, response, body) {
if (!error && response.statusCode === 200) {
message = body.answers[0].actions[0].expression;
sendTextMessage(sender, message);
} else {
message = 'Oops, Looks like Susi is taking a break, She will be back soon';
sendTextMessage(sender, message);
}
});

And voila we have the susi facebook page automatically replying powered by Loklak’s capabilities of Susi. Currently susi can reply with the text messages as well as image responses.

Susi's facebook integration
Susi’s facebook integration
Setting up Susi’s capabilities on Facebook Messenger

The Making of the Console Service

SUSI , our very own personal digital assistant has been up and running giving quirky answers.

But behind all these are rules which train our cute bot to assist her and decide what answers to provide after parsing the question asked by the users.

The questions could range from any formal-informal greetings, general queries about name, weather, date, time to specific ones like details about some random Github profile or Tweets and Replies  from Twitter, or Weibo or election/football score predictions or simply asking her to read a RSS feed or a WordPress blog for you.

The rules for her training are written after that specific service is implemented which shall help her fetch the particular website/social network in question and scrape data out of it to present it to her operator.

And to help us expand the scope and ability of this naive being, it shall be helpful if users could extend her rule set. For this, it is required to make console service for sites which do not provide access to information without OAuth.

To begin with, let us see how a console service can be made.

Starting with a SampleService class which shall basically include the rudimentary scraper or code fetching the data is defined in the package org.loklak.api.search.
This is made by extending the AbstractAPIHandler class which itself extends the javax.servlet.http.HttpServlet class.
SampleService class further implements APIHandler class.

A placeholder for SampleService class can be as:


package org.loklak.api.search;

/**
* import statements
**/

public class SampleService extends AbstractAPIHandler 
    implements APIHandler{

    private static final long serialVersionUID = 2142441326498450416L;
    /**
     * serialVersionUID could be 
     * auto-generated by the IDE used
    **/

    @Override
    public String getAPIPath() {
        return "/api/service.json";
        /**
         *Choose API path for the service in question
        **/
    }

    @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.ANONYMOUS;
    }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    @Override
    public JSONObject serviceImpl(Query call, HttpServletResponse response, 
        Authorization rights, JSONObjectWithDefault permissions) 
        throws APIException {

        String url = call.get("url", "");
        /**
         *This would extract the argument that will be supplied
         * to the "url" parameter in the "call"
        **/
        return crawlerForService(url);

    }

    public SusiThought crawlerForService(String url) {
        JSONArray arr = new JSONArray();
        
        /**
         * Crawler code or any other function which
         * returns a JSON Array **arr** goes in here 
        **/

        SusiThought json = new SusiThought();
        json.setData(arr);
        return json;
    }

}

 

The JSONArray in the key function crawlerForService is wrapped up in a SusiThought which is nothing but a piece of data that can be remembered. The structure or the thought can be modeled as a table which may be created using the retrieval of information from elsewhere of the current argument.

Now to implement it as a Console Service we include it in the ConsoleService class which is defined in the same package org.loklak.api.search and similarly extends AbstractAPIHandler class and implements APIHandler class.

Here, dbAccess is a static variable of the type SusiSkills where a skill is defined as the ability to inspire, to create thoughts from perception. The data structure of a skill set is a mapping from perception patterns to lambda expressions which induce thoughts.


package org.loklak.api.search;

/**
 * import statements go here
**/

public class ConsoleService extends AbstractAPIHandler 
    implements APIHandler {

    private static final long serialVersionUID = 8578478303032749879L;
    /**
     * serialVersionUID could be 
     * auto-generated by the IDE used
    **/

    @Override
    public BaseUserRole getMinimalBaseUserRole() { 
        return BaseUserRole.ANONYMOUS; 
    }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    public String getAPIPath() {
        return "/api/console.json";
    }

    public final static SusiSkills dbAccess = new SusiSkills();
        static {

            /**
             * Other "skills" are defined here
             * by "putting" them in "dbAccess"
            **/
    
    dbAccess.put(Pattern.compile("SELECT\\h+?(.*?)\\h+?FROM\\h+?
        sampleservice\\h+?WHERE\\h+?url\\h??=\\h??'(.*?)'\\h??;"), 
            (flow, matcher) -> {
                /**
                 * SusiThought-s are fetched from the Services
                 * implemented as above
                **/
                SusiThought json = SampleService.crawlerForService(matcher.group(2));
                SusiTransfer transfer = new SusiTransfer(matcher.group(1));
                json.setData(transfer.conclude(json.getData()));
                return json;
                });
    }

    @Override
    public JSONObject serviceImpl(Query post, HttpServletResponse response, 
        Authorization rights, final JSONObjectWithDefault permissions) 
        throws APIException {

            String q = post.get("q", "");
            /**
             *This would extract the argument that will be supplied
             * to the "q" parameter in the "post" query
            **/
            

            return dbAccess.inspire(q);
        }

}

 

Now that the console service is made, an API endpoint for the same can correspond to: http://localhost:9000/api/console.json?q=SELECT * FROM sampleservice WHERE url = ‘ … ‘;

The above can serve as a placeholder for creating Console Service which shall enable SUSI widen her horizon and become intelligent.

So, Go ahead and make Susi rules using it and you are done !

If any aid is required in making SUSI Rules, stay tuned for the next post.

Come, contribute to Loklak and SUSI !

The Making of the Console Service