Improving Loklak apps site

In this blog I will be describing some of the recent improvements made to the Loklak apps site. A new utility script has been added to automatically update the loklak app wall after a new app has been made. Invalid app query in app details page has been handled gracefully.

A proper message is shown when a user enters an invalid app name in the url of the details page. Tests has been added for details page.

Developing updatewall script

This is a small utility script to update Loklak wall in order to expose a newly created app or update an existing app. Before moving into the working of this script let us discuss how Loklak apps site tracks all the apps and their details. In the root of the project there is a file names apps.json. This file contains an aggregation of all the app.json files present in the individual apps. Now when the site is loaded, index.html loads the Javascript code present in app_list.js. This app_list.js file makes an ajax call to root apps.json files, loads all the app details in a list and attaches this list to the AngularJS scope variable. After this the app wall consisting of various app details is rendered using html. So whenever a new app is created, in order to expose the app on the wall, the developer needs to copy the contents of the application’s app.json and paste it in the root apps.json file. This is quite tedious on the part of the developer as for making a new app he will first have to know how the site works which is not all directly related to his development work. Next, whenever he updates the app.json of his app, he needs to again update apps.json file with the new data.

This newly added script (updatewall) automates this entire process. After creating a new app all that the developer needs to do is run this script from within his app directory and the app wall will be updated automatically.

Now, let us move into the working of this script. The basic workflow of the updatewall script can be described as follows. The script loads the json data present in the app.json file of the app under consideration. Next it loads the json data present in the root apps.json file.

if __name__ == '__main__':

    #open file containg json object
    json_list_file = open(PATH_TO_ROOT_JSON, 'r')

    #load json object
    json_list = json.load(json_list_file,  object_pairs_hook=OrderedDict)
    json_list_file.close()

    app_json_file = open(PATH_TO_APP_JSON, 'r')
    app_json = json.load(app_json_file,  object_pairs_hook=OrderedDict)
    app_json_file.close()

    #method to update Loklak app wall
    expose_app(json_list, app_json)

When we are loading the json data we are using object_pairs_hook in order to load the data into an OrderedDict rather than a normal python dictionary. We are doing this so that the order of the dictionary items are maintained. Once the data is loaded we invoke the expose method.

def expose_app(json_list, app_json):
    #if app is already present in list then fetch that app
    app = getAppIfPesent(json_list, app_json)

    #if app is not present then add a new entry
    if app == None:
        json_list['apps'].append(app_json)
        update_list_file(json_list)
        print colors.BOLD + colors.OKGREEN + 'App exposed on app wall' + colors.ENDC

    #else update the existing app entry
    else:
        for key in app_json:
            app[key] = app_json[key]
        update_list_file(json_list)
        print colors.BOLD + colors.OKGREEN + 'App updated on app wall' + colors.ENDC

The apps.json file contain a key called apps. This value of this key is a list of json objects, each object being the json data of an individual app’s app.json file. In the above function we iterate over all the json objects present in the list. If we are unable to find a json object whose name value is same as that of the newly created app then we simply append the new app’s app.json object to that list. However if we find an object containing the same name value as that of the newly created app, then we simply update its properties. In short, if the app is a new one, its data gets added to apps.json otherwise the corresponding app data is updated.

Handling invalid app names in the URL of details page

The url of the app details page takes the app name as parameter. If any user wants to see the store listing of an app then he has to use the following url.

https://apps.loklak.org/details.html?q=<app_name>

Here app name is a url parameter used to load the store listing information. Now if anyone enters an invalid app name, that is an app which does not exists, then a proper error message has to be shown to the user. This can be done by checking whether the given app name is present in the root apps.json file or not. If not present if simply set a flag so that the error message can be conditionally rendered.

$scope.getSelectedApp = function() {
        for (var i = 0; i < $scope.apps.length; i++) {
            if ($scope.apps[i].name === $scope.appName) {
                $scope.selectedApp = $scope.apps[i];
                $scope.found = true;
                $("nav").show();
                break;
            }
        }
        if ($scope.found == false) {
            $scope.notFound = true;
        }
    }

In the above snippet if the app is not found then we set notFound to true. This causes the error message to appear on the page.

<div ng-if="notFound" class="not-found">
        <span class="brand-and-image">
          <img src="images/loklak_icon.png">
          <span class="loklak-brand"> <span class="loklak-header">
            loklak </span> <span>apps</span>
          </span>
        </span>
        <span class="error-404">
          Error: Requested app not found
        </span>
        <span class="go-back">
          <a href="/"> Go Back to Home Page >> </a>
        </span>
      </div>

The code renders the error message if notFound is set to true.

Writing tests for store listing page

Almost the entire content of the store listing is loaded dynamically by Javascript logic. So it is very important to write tests for store listing page. Protractor framework has been used to write automated browser test. The tests make sure that for a given app, the content of the middle section is loaded correctly.

it("should have basic information", function() {
    expect(element(by.css(".app-name")).getText()).toEqual("MultiLinePlotter");
    expect(element(by.css(".app-headline")).getText()).toEqual("App to plot tweet aggregations and statistics");
    expect(element(by.css(".author")).getText()).toEqual("by Deepjyoti Mondal");
    expect(element(by.css(".short-desc")).getText()).toEqual("An applicaton to visually compare tweet statistics");
  });

The above tests make sure that the top section is loaded properly. Next we check that getting started section and app use section are not empty.

it("main content should not be empty", function() {
    expect(element(by.css(".get-started-md")).getText()).not.toBe("");
    expect(element(by.css(".app-use-md")).getText()).not.toBe("");
  });

Apart from these, two more tests are performed to check the behaviour of the side bar menu items on click event and the functionality of the Try now button.

Future roadmap

There is still a lot of scope for the site’s improvement and enhancement. Some of the features which can be implemented next are given below.

  • Add more tests to make the site stable and add tests to travis build.
  • Make the apps independent. Work on this has already been started and can be viewed here – issue, PR
  • Optimise the site for mobile using services workers and caching (making a progressive web app).
  • Add a splash screen and home screen icon for mobile.

Important resources

Improving Loklak apps site

Using Protractor for UI Tests in Angular JS for Loklak Apps Site

Loklak apps site’s home page and app details page have sections where data is dynamically loaded from external javascript and json files. Data is fetched from json files using angular js, processed and then rendered to the corresponding views by controllers. Any erroneous modification to the controller functions might cause discrepancies in the frontend. Since Loklak apps is a frontend project, any bug in the home page or details page will lead to poor UI/UX. How do we deal with this? One way is to write unit tests for the various controller functions and check their behaviours. Now how do we test the behaviours of the site. Most of the controller functions render something on the view. One thing we can do is simulate the various browser actions and test site against known, accepted behaviours with Protractor.

What is Protractor

Protractor is end to end test framework for Angular and AngularJS apps. It runs tests against our app running in browser as if a real user is interacting with our browser. It uses browser specific drivers to interact with our web application as any user would.

Using Protractor to write tests for Loklak apps site

First we need to install Protractor and its dependencies. Let us begin by creating an empty json file in the project directory using the following command.

echo {} > package.json

Next we will have to install Protractor.

The above command installs protractor and webdriver-manager. After this we need to get the necessary binaries to set up our selenium server. This can be done using the following.

./node_modules/protractor/bin/webdriver-manager update
./node_modules/protractor/bin/webdriver-manager start

Let us tidy up things a bit. We will include these commands in package.json file under scripts section so that we can shorten our commands.

Given below is the current state of package.json

{
    "scripts": {
        "start": "./node_modules/http-server/bin/http-server",
        "update-driver": "./node_modules/protractor/bin/webdriver-manager update",
        "start-driver": "./node_modules/protractor/bin/webdriver-manager start",
        "test": "./node_modules/protractor/bin/protractor conf.js"
    },
    "dependencies": {
        "http-server": "^0.10.0",
        "protractor": "^5.1.2"
    }
}

The package.json file currently holds our dependencies and scripts. It contains command for starting development server, updating webdriver and starting webdriver (mentioned just before this) and command to run test.

Next we need to include a configuration file for protractor. The configuration file should contain the test framework to be used, the address at which selenium is running and path to specs file.

// conf.js
exports.config = {
    framework: "jasmine",
    seleniumAddress: "http://localhost:4444/wd/hub",
    specs: ["tests/home-spec.js"]
};

We have set the framework as jasmine and selenium address as http://localhost:4444/wd/hub. Next we need to define our actual file. But before writing tests we need to find out what are the things that we need to test. We will mostly be testing dynamic content loaded by Javascript files. Let us define a spec. A spec is a collection of tests. We will start by testing the category name. Initially when the page loads it should be equal to All apps. Next we test the top right hand side menu which is loaded by javascript using topmenu.json file.

it("should have a category name", function() {
    expect(element(by.id("categoryName")).getText()).toEqual("All apps");
  });

  it("should have top menu", function() {
    let list = element.all(by.css(".topmenu li a"));
    expect(list.count()).toBe(5);
  });

As mentioned earlier, we are using jasmine framework for writing our specs. In the above code snippet ‘it’ describes a particular test. It takes a test description and a callback function thereby providing a very efficient way to document our tests white write the test code itself. In the first test we use expect function to check whether the category name is equal to All apps or not. Here we select the div containing the category name by its id.

Next we write a test for top menu. There should be five menu options in total for the top menu. We select all the list items that are supposed to contain the top menu items and check whether the number of such items are five or not using expect function. As it can be seen from the snippet, the process of selecting a node is almost similar to that of Jquery library.

Next we test the left hand side category list. This list is loaded by AngularJS controller from apps,json file. We should make sure the list is loaded properly and all the options are present.

it("should have a category list", function() {
    let categoryIds = ["All", "Scraper", "Search", "Visualizer", "LoklakLibraries", "InternetOfThings", "Misc"];
    let categoryNames = ["All", "Scraper", "Search", "Visualizer", "Loklak Libraries", "Internet Of Things", "Misc"];

    expect(element(by.css("#catTitle")).getText()).toBe("Categories");

    let categoryList = element.all(by.css(".category-main"));
    expect(categoryList.count()).toBe(7);

    categoryIds.forEach(function(id, index) {
      element(by.css("#" + id)).isPresent().then(function(present) {
        expect(present).toBe(true);
      });

      element(by.css("#" + id)).getText().then(function(text) {
        expect(text).toBe(categoryNames[index]);
      });
    });
  });

At first we maintain two lists of category id and category names. We begin by confirming that Category title is equal to Categories. Next we get the list of categories and iterate over them, For each category we check whether the corresponding id is present in the DOM or not. After confirming this, we match the names of the categories with the expected names. Elements.all function allows us to get a list of selected nodes.

Finally we check the click functionality of the left side menu. Expected behaviour is, on clicking a menu item, the category name should get replaced with the selected category name. For this we need to simulate the click event. Protractor allows us to do it very easily using click function.

it("category list should respond to click", function() {
    let categoryIds = ["All", "Scraper", "Search", "Visualizer", "LoklakLibraries", "InternetOfThings", "Misc"];
    let categoryNames = ["All apps", "Scraper", "Search", "Visualizer", "Loklak Libraries", "Internet Of Things", "Misc"];

    categoryIds.forEach(function(id, index) {
      element(by.id(categoryIds[index])).click().then(function() {
        browser.getCurrentUrl().then(function(url) {
          expect(url).toBe("http://127.0.0.1:8080/#/" + categoryIds[index]);
        });
        element(by.id("categoryName")).getText().then(function(text) {
          expect(text).toBe(categoryNames[index]);
        });
      });
    });
  });



Once again we maintain two lists, category id and category names. We obtain the present list of categories and iterate over them. For each category link we simulate a click event. For each click event we check two values. We check the new browser URL which should now contain the category id. Next we check the value of category name. It should be equal to the category selected.
FInally after all the tests are over we get the final report on our terminal.
In order to run the tests, use the following command.

npm test

This will start executing the tests.

Important resources

Using Protractor for UI Tests in Angular JS for Loklak Apps Site

Preparing for Automatic Publishing of Android Apps in Play Store

I spent this week searching through libraries and services which provide a way to publish built apks directly through API so that the repositories for Android apps can trigger publishing automatically after each push on master branch. The projects to be auto-deployed are:

I had eyes on fastlane for a couple of months and it came out to be the best solution for the task. The tool not only allows publishing of APK files, but also Play Store listings, screenshots, and changelogs. And that is only a subset of its capabilities bundled in a subservice supply.

There is a process before getting started to use this service, which I will go through step by step in this blog. The process is also outlined in the README of the supply project.

Enabling API Access

The first step in the process is to enable API access in your Play Store Developer account if you haven’t done so. For that, you have to open the Play Dev Console and go to Settings > Developer Account > API access.

If this is the first time you are opening it, you’ll be presented with a confirmation dialog detailing about the ramifications of the action and if you agree to do so. Read carefully about the terms and click accept if you agree with them. Once you do, you’ll be presented with a setting panel like this:

Creating Service Account

As you can see there is no registered service account here and we need to create one. So, click on CREATE SERVICE ACCOUNT button and this dialog will pop up giving you the instructions on how to do so:

So, open the highlighted link in the new tab and Google API Console will open up, which will look something like this:

Click on Create Service Account and fill in these details:

Account Name: Any name you want

Role: Project > Service Account Actor

And then, select Furnish a new private key and select JSON. Click CREATE.

A new JSON key will be created and downloaded on your device. Keep this secret as anyone with access to it can at least change play store listings of your apps if not upload new apps in place of existing ones (as they are protected by signing keys).

Granting Access

Now return to the Play Console tab (we were there in Figure 2 at the start of Creating Service Account), and click done as you have created the Service Account now. And you should see the created service account listed like this:

Now click on grant access, choose Release Manager from Role dropdown, and select these PERMISSIONS:

Of course you don’t want the fastlane API to access financial data or manage orders. Other than that it is up to you on what to allow or disallow. Same choice with expiry date as we have left it to never expire. Click on ADD USER and you’ll see the Release Manager created in the user list like below:

Now you are ready to use the fastlane service, or any other release management service for that matter.

Using fastlane

Install fastlane by

sudo gem install fastlane

Go to your project folder and run

fastlane supply init

First it will ask the location of the private key JSON file you downloaded, and then the package name of the application you are trying to initialize fastlane for.

Then it will create metadata folder with listing information excluding the images. So you’ll have to download and place the images manually for the first time

After modifying the listing, images or APK, run the command:

fastlane supply run

That’s it. Your app along with the store listing has been updated!

This is a very brief introduction to the capabilities of the supply service. All interactive options can be supplied via command line arguments, certain parts of the metadata can be omitted and alpha beta management along with release rollout can be done in steps! Make sure to check out the links below:

Preparing for Automatic Publishing of Android Apps in Play Store