Improving Loklak apps site

In this blog I will be describing some of the recent improvements made to the Loklak apps site. A new utility script has been added to automatically update the loklak app wall after a new app has been made. Invalid app query in app details page has been handled gracefully.

A proper message is shown when a user enters an invalid app name in the url of the details page. Tests has been added for details page.

Developing updatewall script

This is a small utility script to update Loklak wall in order to expose a newly created app or update an existing app. Before moving into the working of this script let us discuss how Loklak apps site tracks all the apps and their details. In the root of the project there is a file names apps.json. This file contains an aggregation of all the app.json files present in the individual apps. Now when the site is loaded, index.html loads the Javascript code present in app_list.js. This app_list.js file makes an ajax call to root apps.json files, loads all the app details in a list and attaches this list to the AngularJS scope variable. After this the app wall consisting of various app details is rendered using html. So whenever a new app is created, in order to expose the app on the wall, the developer needs to copy the contents of the application’s app.json and paste it in the root apps.json file. This is quite tedious on the part of the developer as for making a new app he will first have to know how the site works which is not all directly related to his development work. Next, whenever he updates the app.json of his app, he needs to again update apps.json file with the new data.

This newly added script (updatewall) automates this entire process. After creating a new app all that the developer needs to do is run this script from within his app directory and the app wall will be updated automatically.

Now, let us move into the working of this script. The basic workflow of the updatewall script can be described as follows. The script loads the json data present in the app.json file of the app under consideration. Next it loads the json data present in the root apps.json file.

if __name__ == '__main__':

    #open file containg json object
    json_list_file = open(PATH_TO_ROOT_JSON, 'r')

    #load json object
    json_list = json.load(json_list_file,  object_pairs_hook=OrderedDict)
    json_list_file.close()

    app_json_file = open(PATH_TO_APP_JSON, 'r')
    app_json = json.load(app_json_file,  object_pairs_hook=OrderedDict)
    app_json_file.close()

    #method to update Loklak app wall
    expose_app(json_list, app_json)

When we are loading the json data we are using object_pairs_hook in order to load the data into an OrderedDict rather than a normal python dictionary. We are doing this so that the order of the dictionary items are maintained. Once the data is loaded we invoke the expose method.

def expose_app(json_list, app_json):
    #if app is already present in list then fetch that app
    app = getAppIfPesent(json_list, app_json)

    #if app is not present then add a new entry
    if app == None:
        json_list['apps'].append(app_json)
        update_list_file(json_list)
        print colors.BOLD + colors.OKGREEN + 'App exposed on app wall' + colors.ENDC

    #else update the existing app entry
    else:
        for key in app_json:
            app[key] = app_json[key]
        update_list_file(json_list)
        print colors.BOLD + colors.OKGREEN + 'App updated on app wall' + colors.ENDC

The apps.json file contain a key called apps. This value of this key is a list of json objects, each object being the json data of an individual app’s app.json file. In the above function we iterate over all the json objects present in the list. If we are unable to find a json object whose name value is same as that of the newly created app then we simply append the new app’s app.json object to that list. However if we find an object containing the same name value as that of the newly created app, then we simply update its properties. In short, if the app is a new one, its data gets added to apps.json otherwise the corresponding app data is updated.

Handling invalid app names in the URL of details page

The url of the app details page takes the app name as parameter. If any user wants to see the store listing of an app then he has to use the following url.

https://apps.loklak.org/details.html?q=<app_name>

Here app name is a url parameter used to load the store listing information. Now if anyone enters an invalid app name, that is an app which does not exists, then a proper error message has to be shown to the user. This can be done by checking whether the given app name is present in the root apps.json file or not. If not present if simply set a flag so that the error message can be conditionally rendered.

$scope.getSelectedApp = function() {
        for (var i = 0; i < $scope.apps.length; i++) {
            if ($scope.apps[i].name === $scope.appName) {
                $scope.selectedApp = $scope.apps[i];
                $scope.found = true;
                $("nav").show();
                break;
            }
        }
        if ($scope.found == false) {
            $scope.notFound = true;
        }
    }

In the above snippet if the app is not found then we set notFound to true. This causes the error message to appear on the page.

<div ng-if="notFound" class="not-found">
        <span class="brand-and-image">
          <img src="images/loklak_icon.png">
          <span class="loklak-brand"> <span class="loklak-header">
            loklak </span> <span>apps</span>
          </span>
        </span>
        <span class="error-404">
          Error: Requested app not found
        </span>
        <span class="go-back">
          <a href="/"> Go Back to Home Page >> </a>
        </span>
      </div>

The code renders the error message if notFound is set to true.

Writing tests for store listing page

Almost the entire content of the store listing is loaded dynamically by Javascript logic. So it is very important to write tests for store listing page. Protractor framework has been used to write automated browser test. The tests make sure that for a given app, the content of the middle section is loaded correctly.

it("should have basic information", function() {
    expect(element(by.css(".app-name")).getText()).toEqual("MultiLinePlotter");
    expect(element(by.css(".app-headline")).getText()).toEqual("App to plot tweet aggregations and statistics");
    expect(element(by.css(".author")).getText()).toEqual("by Deepjyoti Mondal");
    expect(element(by.css(".short-desc")).getText()).toEqual("An applicaton to visually compare tweet statistics");
  });

The above tests make sure that the top section is loaded properly. Next we check that getting started section and app use section are not empty.

it("main content should not be empty", function() {
    expect(element(by.css(".get-started-md")).getText()).not.toBe("");
    expect(element(by.css(".app-use-md")).getText()).not.toBe("");
  });

Apart from these, two more tests are performed to check the behaviour of the side bar menu items on click event and the functionality of the Try now button.

Future roadmap

There is still a lot of scope for the site’s improvement and enhancement. Some of the features which can be implemented next are given below.

  • Add more tests to make the site stable and add tests to travis build.
  • Make the apps independent. Work on this has already been started and can be viewed here – issue, PR
  • Optimise the site for mobile using services workers and caching (making a progressive web app).
  • Add a splash screen and home screen icon for mobile.

Important resources

Improving Loklak apps site

Using Protractor for UI Tests in Angular JS for Loklak Apps Site

Loklak apps site’s home page and app details page have sections where data is dynamically loaded from external javascript and json files. Data is fetched from json files using angular js, processed and then rendered to the corresponding views by controllers. Any erroneous modification to the controller functions might cause discrepancies in the frontend. Since Loklak apps is a frontend project, any bug in the home page or details page will lead to poor UI/UX. How do we deal with this? One way is to write unit tests for the various controller functions and check their behaviours. Now how do we test the behaviours of the site. Most of the controller functions render something on the view. One thing we can do is simulate the various browser actions and test site against known, accepted behaviours with Protractor.

What is Protractor

Protractor is end to end test framework for Angular and AngularJS apps. It runs tests against our app running in browser as if a real user is interacting with our browser. It uses browser specific drivers to interact with our web application as any user would.

Using Protractor to write tests for Loklak apps site

First we need to install Protractor and its dependencies. Let us begin by creating an empty json file in the project directory using the following command.

echo {} > package.json

Next we will have to install Protractor.

The above command installs protractor and webdriver-manager. After this we need to get the necessary binaries to set up our selenium server. This can be done using the following.

./node_modules/protractor/bin/webdriver-manager update
./node_modules/protractor/bin/webdriver-manager start

Let us tidy up things a bit. We will include these commands in package.json file under scripts section so that we can shorten our commands.

Given below is the current state of package.json

{
    "scripts": {
        "start": "./node_modules/http-server/bin/http-server",
        "update-driver": "./node_modules/protractor/bin/webdriver-manager update",
        "start-driver": "./node_modules/protractor/bin/webdriver-manager start",
        "test": "./node_modules/protractor/bin/protractor conf.js"
    },
    "dependencies": {
        "http-server": "^0.10.0",
        "protractor": "^5.1.2"
    }
}

The package.json file currently holds our dependencies and scripts. It contains command for starting development server, updating webdriver and starting webdriver (mentioned just before this) and command to run test.

Next we need to include a configuration file for protractor. The configuration file should contain the test framework to be used, the address at which selenium is running and path to specs file.

// conf.js
exports.config = {
    framework: "jasmine",
    seleniumAddress: "http://localhost:4444/wd/hub",
    specs: ["tests/home-spec.js"]
};

We have set the framework as jasmine and selenium address as http://localhost:4444/wd/hub. Next we need to define our actual file. But before writing tests we need to find out what are the things that we need to test. We will mostly be testing dynamic content loaded by Javascript files. Let us define a spec. A spec is a collection of tests. We will start by testing the category name. Initially when the page loads it should be equal to All apps. Next we test the top right hand side menu which is loaded by javascript using topmenu.json file.

it("should have a category name", function() {
    expect(element(by.id("categoryName")).getText()).toEqual("All apps");
  });

  it("should have top menu", function() {
    let list = element.all(by.css(".topmenu li a"));
    expect(list.count()).toBe(5);
  });

As mentioned earlier, we are using jasmine framework for writing our specs. In the above code snippet ‘it’ describes a particular test. It takes a test description and a callback function thereby providing a very efficient way to document our tests white write the test code itself. In the first test we use expect function to check whether the category name is equal to All apps or not. Here we select the div containing the category name by its id.

Next we write a test for top menu. There should be five menu options in total for the top menu. We select all the list items that are supposed to contain the top menu items and check whether the number of such items are five or not using expect function. As it can be seen from the snippet, the process of selecting a node is almost similar to that of Jquery library.

Next we test the left hand side category list. This list is loaded by AngularJS controller from apps,json file. We should make sure the list is loaded properly and all the options are present.

it("should have a category list", function() {
    let categoryIds = ["All", "Scraper", "Search", "Visualizer", "LoklakLibraries", "InternetOfThings", "Misc"];
    let categoryNames = ["All", "Scraper", "Search", "Visualizer", "Loklak Libraries", "Internet Of Things", "Misc"];

    expect(element(by.css("#catTitle")).getText()).toBe("Categories");

    let categoryList = element.all(by.css(".category-main"));
    expect(categoryList.count()).toBe(7);

    categoryIds.forEach(function(id, index) {
      element(by.css("#" + id)).isPresent().then(function(present) {
        expect(present).toBe(true);
      });

      element(by.css("#" + id)).getText().then(function(text) {
        expect(text).toBe(categoryNames[index]);
      });
    });
  });

At first we maintain two lists of category id and category names. We begin by confirming that Category title is equal to Categories. Next we get the list of categories and iterate over them, For each category we check whether the corresponding id is present in the DOM or not. After confirming this, we match the names of the categories with the expected names. Elements.all function allows us to get a list of selected nodes.

Finally we check the click functionality of the left side menu. Expected behaviour is, on clicking a menu item, the category name should get replaced with the selected category name. For this we need to simulate the click event. Protractor allows us to do it very easily using click function.

it("category list should respond to click", function() {
    let categoryIds = ["All", "Scraper", "Search", "Visualizer", "LoklakLibraries", "InternetOfThings", "Misc"];
    let categoryNames = ["All apps", "Scraper", "Search", "Visualizer", "Loklak Libraries", "Internet Of Things", "Misc"];

    categoryIds.forEach(function(id, index) {
      element(by.id(categoryIds[index])).click().then(function() {
        browser.getCurrentUrl().then(function(url) {
          expect(url).toBe("http://127.0.0.1:8080/#/" + categoryIds[index]);
        });
        element(by.id("categoryName")).getText().then(function(text) {
          expect(text).toBe(categoryNames[index]);
        });
      });
    });
  });



Once again we maintain two lists, category id and category names. We obtain the present list of categories and iterate over them. For each category link we simulate a click event. For each click event we check two values. We check the new browser URL which should now contain the category id. Next we check the value of category name. It should be equal to the category selected.
FInally after all the tests are over we get the final report on our terminal.
In order to run the tests, use the following command.

npm test

This will start executing the tests.

Important resources

Using Protractor for UI Tests in Angular JS for Loklak Apps Site

Adding unit tests for effects in Loklak Search

Loklak search uses @ngrx/effects to listen to actions dispatched by the user and sending API request to the loklak server. Loklak search, currently, has seven effects such as Search Effects,  Suggest Effects which runs to make the application reactive. It is important to test these effects to ensure that effects make API calls at the right time and then map the response to send it back to the reducer.

I will  explain here how I added unit tests for the effects. Surprisingly, the test coverage increased from 43% to 55% after adding these tests.

Effects to test

We are going to test effects for user search. This effect listens to the event of type USER_SEARCH and makes a call to the user-search service with the query as a parameter. After a response is received, it maps the response and passes it on the UserSearchCompleteSuccessAction action which performs the later operation. If the service fails to get a response, it makes a call to the UserSearchCompleteFailAction.

Code

ApiUserSearchEffects is the effect which detects if the USER_SEARCH action is dispatched from some component of the application and consequently, it makes a call to the UserSearchService and handles the JSON response received from the server. The effects then, dispatch the action new UserSearchCompleteSuccessAction if response is received from server or either dispatch the action new UserSearchCompleteFailAction if no response is received. The debounce time is set to 400 so that response can be flushed if a new USER_SEARCH is dispatched within the next 400ms.

For this effect, we need to test if the effects actually runs when USER_SEARCH action is made. Further, we need to test if the correct parameters are supplied to the service and response is handled carefully. We also, need to check if the response if really flushed out within the certain debounce time limit.

@Injectable()
export class ApiUserSearchEffects {@Effect()
search$: Observable<Action>
= this.actions$
.ofType(userApiAction.ActionTypes.USER_SEARCH)
.debounceTime(400)
.map((action: userApiAction.UserSearchAction) => action.payload)
.switchMap(query => {
const nextSearch$ = this.actions$.ofType(userApiAction.ActionTypes.USER_SEARCH).skip(1);const follow_count = 10;return this.apiUserService.fetchQuery(query.screen_name, follow_count)
.takeUntil(nextSearch$)
.map(response => new userApiAction.UserSearchCompleteSuccessAction(response))
.catch(() => of(new userApiAction.UserSearchCompleteFailAction()));
});constructor(
private actions$: Actions,
private apiUserService: UserService
) { }

}

Unit test for effects

  • Configure the TestBed class before starting the unit test and add all the necessary imports (most important being the EffectsTestingModule) and providers. This step will help to isolate the effects completely from all other components and testing it independently. We also need to create spy object which spies on the method userService.fetchQuery with provider being UserService.

beforeEach(() => TestBed.configureTestingModule({
imports: [
EffectsTestingModule,
RouterTestingModule
],
providers: [
ApiUserSearchEffects,
{
provide: UserService,
useValue: jasmine.createSpyObj(‘userService’, [‘fetchQuery’])
}
]
}));
  • Now, we will be needing a function setup which takes params which are the data to be returned by the Mock User Service. We can now configure the response to returned by the service. Moreover, this function will be initializing EffectsRunner and returning ApiUserSearchEffects so that it can be used for unit testing.

function setup(params?: {userApiReturnValue: any}) {
const userService = TestBed.get(UserService);
if (params) { userService.fetchQuery.and.returnValue(params.userApiReturnValue);
}return {
runner: TestBed.get(EffectsRunner),
apiUserSearchEffects: TestBed.get(ApiUserSearchEffects)
};
}

 

  • Now we will be adding unit tests for the effects. In these tests, we are going to test if the effects recognise the action and return some new action based on the response we want and if it maps the response only after a certain debounce time.We have used fakeAsync() which gives us access to the tick() function. Next, We are calling the function setup and pass on the Mock Response so that whenever User Service is called it returns the Mock Response and never runs the service actually. We will now queue the action UserSearchAction in the runner and subscribe to the value returned by the effects class. We can now test the value returned using expect() block and that the value is returned only after a certain debounce time using tick() block.

it(‘should return a new userApiAction.UserSearchCompleteSuccessAction, ‘ +
‘with the response, on success, after the de-bounce’, fakeAsync(() => {
const response = MockUserResponse;const {runner, apiUserSearchEffects} = setup({userApiReturnValue: Observable.of(response)});

const expectedResult = new userApiAction.UserSearchCompleteSuccessAction(response);

runner.queue(new userApiAction.UserSearchAction(MockUserQuery));

let result = null;
apiUserSearchEffects.search$.subscribe(_result => result = _result);
tick(399); // test debounce
expect(result).toBe(null);
tick(401);
expect(result).toEqual(expectedResult);
}));

it(‘should return a new userApiAction.UserSearchCompleteFailAction,’ +
‘if the SearchService throws’, fakeAsync(() => {
const { runner, apiUserSearchEffects } = setup({ userApiReturnValue: Observable.throw(new Error()) });

const expectedResult = new userApiAction.UserSearchCompleteFailAction();

runner.queue(new userApiAction.UserSearchAction(MockUserQuery));

let result = null;
apiUserSearchEffects.search$.subscribe(_result => result = _result );

tick(399); // Test debounce
expect(result).toBe(null);
tick(401);
expect(result).toEqual(expectedResult);
}));
});

Reference

Adding unit tests for effects in Loklak Search