Custom colours for loklak walls

You can now customize the background and card colors on loklak walls!

Here’s how we did it:

First, we had to add extra fields to the wall schema:

var UserSchema = new Schema({
  apps: {
    wall: [{
      // other options
      cardBgColour: String,
      cardForeColour: String,
      wallBgColour: String,

Next, we had to add these extra options in the angular controller (wall.js) for the creation modal:

var initWallOptions = function() {
  $scope.newWallOptions.wallBgColour = '#ecf0f5';
  $scope.newWallOptions.cardBgColour = '#ffffff';
}

$scope.$watch('newWallOptions.cardBgColour', function() {
  if ($scope.newWallOptions.cardBgColour) {
    $scope.newWallOptions.cardForeColour = colourCalculator(hexToRgb($scope.newWallOptions.cardBgColour));
  }
});

The $watch function watches for any changes in the card background color and changes the cardForeColour / text color to be black or white depending on the bg color.

Now, we have to use the saved data in the wall display pages (display.html):


   <div ng-style="{'background-color': wall.wallOptions.wallBgColour}" class="wall-container container-fluid">
        <div class="container content-container wall-body">
            <div ng-switch on="wall.wallOptions.layoutStyle" ng-show="wall.statuses.length>0" ng-class="wall.wallOptions.showStatistics || wall.currentAnnoucement?'col-md-8':'col-md-12'" masonry>
                <!-- 1. Linear -->
                <div ng-switch-when="1" linear ng-repeat="status in wall.statuses" open="wall.open" data="status" 
                cardbgcolor="wall.wallOptions.cardBgColour" cardtxtcolor="wall.wallOptions.cardForeColour"></div>
                <!-- 2. Masonry -->
                <div ng-switch-when="2" card ng-repeat="status in wall.statuses" open="wall.open" data="status" 
                cardbgcolor="wall.wallOptions.cardBgColour" cardtxtcolor="wall.wallOptions.cardForeColour"
                leaderboardEnabled="{{wall.wallOptions.showStatistics}}"></div>
                <!-- 3. Single -->
                <div ng-switch-when="3" coa ng-repeat="status in wall.statuses" open="wall.open" data="status"
                cardbgcolor="wall.wallOptions.cardBgColour" cardtxtcolor="wall.wallOptions.cardForeColour"
                ></div>
            </div>
         </div>
     </div>

We pass the saved wall options into each directive using the attributes cardbgcolor, cardtxtcolor, and we use ng-style to evaluate the expression with wallBgColour.

In the linear layout directive file, we use the ‘=’ sign to signal 2-way-binding.

function linearLayoutDirective() {
  return {
    scope: {
      data: '=',
      cardbgcolor:'=',
      cardtxtcolor:'=',
    },
    templateUrl: 'wall/templates/linear.html',
  };
}

Then we can use it in our template (linear.html):

<div class="linear linear-simple" style="background-color: {{cardbgcolor}};">
  <!-- Main content -->
  <p class="linear-content-text" style="color: {{cardtxtcolor}};"></p>
</div>

I have passed the cardbgcolour into the filter

|tweetTextLink:cardbgcolor

so we can also change the colours of the links:

filtersModule.filter('tweetTextLink', function() {
  return function(input, cardBgColour) {
  var textClassName = cardBgColour ? colourCalculator(hexToRgb(cardBgColour)) : '';
  }
}

Have fun customizing your walls at: loklak-wall.herokuapp.com
Screenshot 2016-07-10 09.25.14

Custom colours for loklak walls

Manual Moderation – multiple users

This is the final post on the manual moderation feature, where we will cover how to deal with the case of multiple users logged in moderating or viewing the same wall. The problem was that whenever two pages of the wall were open at the same time, there was no way to tell if another page was polling for tweets, so there would be duplicate tweets added to the database, which would then show up on both walls. This had to be solved as it is expected to have multiple displays or moderators for each wall.

To solve this, we needed to be able to store the user-wall id that was currently polling from the server. Not only did an open page have to check if there was another page polling, but also when the page was closed, the store would have to remove the user-wall id, so that when another open page checked it could start polling and store it’s user-wall id.

This connecting and disconnecting behavior can be detected through websocket events, and socket.io has a convenient way for us to listen to these events on the server:

[code language=”javascript”]
var server = express();
var s = http.createServer(server);
var io = require(‘socket.io’)(s);
var pollingWalls = {};
var clientIds = {};

io.on(‘connection’, function (socket) {

// Create and join UserwallId room when user opens page
socket.on(‘create’, function(userWallId) {
socket.join(userWallId);
});

// When close window, use the socket.id to remove from maps so another poll can pass
socket.on(‘disconnect’, function() {
var clientWallPolls = clientIds[socket.id];
clientWallPolls.forEach(function(wallId){
pollingWalls[wallId] = null;
})
delete clientIds[socket.id];
})

// Check duplicate, start if no one else polling
socket.on(‘checkDup’, function(data){
var clients_in_the_room = io.sockets.adapter.rooms[data.userWallId];
var isNoOneElsePolling = pollingWalls[data.userWallId] === socket.id || !pollingWalls[data.userWallId];
if(clients_in_the_room){
var result = clients_in_the_room.length === 1 || isNoOneElsePolling;
var responseEmit = ‘checkDupSuccess’+ data.userWallId+ data.socketId;
socket.emit(responseEmit, result);
}
})

// Start polling and mark poller
// Pre-cond: no one else polling / previous poller leaves
socket.on(‘addPollingWalls’, function(userWallId){
pollingWalls[userWallId] = socket.id;
var clientWalls = clientIds[socket.id];
if(clientWalls.indexOf(userWallId) === -1){
clientWalls.push(userWallId);
}
})
[/code]

I have pasted the server events(above) and client events(below) so that it is easier to view and explain:

[code language=”javascript”]
var init = function() {
// … other config
socket.emit(‘create’, $stateParams.user + $stateParams.id);
}

// Timeout that checks for multiple users on the same wall
vm.update2 = function(refreshTime) {
return $timeout(function() {
socket.emit(‘checkDup’, {userWallId:userWallId, socketId:socketId});
}, refreshTime);
};

// Event listener that polls if there are no duplicates
socket.on(‘checkDupSuccess’+userWallId+socketId, function(result){
if(result){
SearchService.initData(searchParams).then(successCb, errorCb);
socket.emit(‘addPollingWalls’, userWallId);
}
})
[/code]

In server.js, we use two objects – ‘pollingWalls‘ to map the walls that are currently polling to the userId, and ‘clientIds‘ to map each user to the walls that has the user has opened, so we do not have to traverse the whole object.

We also use socket.io rooms to check if there is no one else polling on the same wall. socket.join allows the connected user to join an existing room or create one if it does not exist. Each room is marked by their user-wall id. In the client code, (in wallDisplay.js the controller for the wall display page), the ‘create‘ event is emitted when a user opens a wall.

The other 2 event listeners on the server are to check if there are duplicate users.

Before each polling interval, in the timeout function below, the ‘checkDup‘ event is emitted, carrying with it the userwallId and the socketId. The event listener on server (as seen above) checks if there are no other users in the room with ‘io.sockets.adapter.rooms[data.userWallId]’ or nobody polling ‘var isNoOneElsePolling = pollingWalls[data.userWallId] === socket.id || !pollingWalls[data.userWallId];’. It then emits an event ‘checkDupSuccess’ carrying the boolean result to the specific ‘socket.id’ of the client that requested the check.

If the check passes then the angular ‘SearchService’ is called, and another event ‘addPollingWalls‘ is emitted to store the ‘userWallId’ and associate it with the client’s socket.id in the ‘pollingWalls‘ and ‘clientIds‘ object.

Create your own walls at: loklak-wall.herokuapp.com

 

Manual Moderation – multiple users

Loklak Walls Manual Moderation – approving tweets

This is a continuation from the previous post – Loklak walls manual moderation – tweet storage. In this section, I will show the changes I made to enable the user to approve or reject a tweet, and make that change happen across all walls that are opened.

First, I had to examine how loklak.net displays it’s tweets previously. On each wall page, a timeout would be called every interval to retrieve new tweets from loklak_server, these tweets would then be stored on the client’s browser window, in the view model, or $scope in angular.

[code language=”javascript”]

vm.update2 = function(refreshTime) {
return $timeout(function() {
SearchService.initData(searchParams).then(function(data) {

vm.statuses = data.statuses.splice(0, searchParams.count);

}, refreshTime);
};
[/code]

 

Having this kind of storage for each browser makes the data inconsistent across the same wall open in different browser windows, as they start querying loklak_server at different times and intervals. This also makes it tough to implement manual moderation previously as the tweets on each open page could not be controlled from the dashboard, since they all lived in their own page.

Now that we have shifted the storage of tweets to mongoDB, we are now able to control how tweets are displayed for all open walls! First, I shifted the calls to loklak_server to the dashboard page instead of the walls page as it made more sense to control the interval from the dashboard rather than the display pages themselves.

Next, I needed a way to sync changes in the database across the dashboard as well as the display pages. At first, I tried to use the same method of http calls, but I soon found them too complicated to sync, having 3 components with interconnected actions. Actions from the dashboard and new entries from the database would have to affect the display, and new entries from the database would have to affect the dashboard and display. Also having an interval for updating the wall after changes were made to the database made it seem very unresponsive and resulted in a bad user experience.

The solution to this was: WebSockets! This allows us to listen for new events like addition of new tweets. When first initialized, the display pages and the dashboard just had to load the existing tweets in the database, when new tweets are loaded, they’ll be added into the database AND the displays and dashboard, making it update in real time.

websocket-small

I chose socket.io as it made integrating WebSockets into the MEAN stack relatively easy. After the http request for new tweets from loklak_server is returned, the app then sends a POST request to the node server, which then emits an event to update the display and the dashboard. Below is the route controller, which posts the tweet array received from loklak_server.

[code language=”javascript”]
module.exports.storeTweet = function (req, res) {
req.body.tweetArr.forEach(function(tweet){
var newTweet = new Tweet(tweet);
newTweet.save(function(err,tweet){
// EMIT DASHBOARD EVENT
io.emit("addNewTweet", tweet);
// EMIT WALL DISPLAY EVENT
io.emit("addNewTweet"+req.body.userWallId, tweet);
}
})
});

[/code]

On the wall display page controller, it listens for the emitted event and adds the data to the display.

[code language=”javascript”]
socket.on(‘addNewTweets’ + $stateParams.user + $stateParams.id, function(tweet){
vm.statuses.splice(0,0, tweet);
})
[/code]

The toggle events are similar in that instead of POST requests,  now we are sending PUT requests from the dashboard to update the tweet in mongoDB, and then changing the data attribute on the wall display. Using AngularJS’s ng-hide we can show/hide the tweet depending on it’s approval field.

Inside the angular directive on the dashboard we attach a toggle function to the click:

[code language=”javascript”]
$scope.toggle = function(){
$scope.data.approval = !$scope.data.approval;
$http.put(‘/api/tweets/’+$scope.data._id, $scope.data);
}
[/code]

[code language=”html”]
<div ng-show="data.approval" ng-attr-id="{{data.id_str}}" class="linear linear-simple" /&amp;gt;
[/code]

Similarly on the server we emit an event:

[code language=”javascript”]

module.exports.updateTweet = function (req, res) {
Tweet
.findById(req.params.tweetId)
.exec(function(err, tweet) {
tweet.approval = !tweet.approval;
tweet.save(function(err) {
res.json({tweet: tweet});
});
});

// EMIT TOGGLE EVENT
io.emit("toggle", req.params.tweetId);
}
}
[/code]

And on the wallDisplay controller we can listen to that toggle event:

[code language=”javascript”]

socket.on(‘toggle’,function(tweetId){
var tweetIdx = vm.statuses.findIndex(function(tweet){
return tweet._id === tweetId;
});
vm.statuses[tweetIdx].approval = !vm.statuses[tweetIdx].approval;
});

[/code]

The end result is manual moderation from the dashboard!

icLfu4KZE9

 

 

Loklak Walls Manual Moderation – approving tweets

Loklak Walls Manual Moderation – tweet storage

Loklak walls are going to get a new feature called manual moderation! You may have seen it in the coming soon section of the content tab when you were creating a wall.

For the walls application we have decided to use Mongodb so we can query an external API like loklak_server, then store tweets in the Mongodb database. So the first question we have to think about before storing it is how to model it in our Mongoose schemas. To answer that we have to consider the access patterns of our web app, as it is critical in deciding how we model the collection in mongodb – whether we should embed the data in an existing collection, keep all tweets for a wall in an array in one document in the collection, or store each tweet as a a document itself and query them.

The main issue is that the number of tweets could keep growing, so we can’t embed nor store them in an array in a document. Mongodb has a document size limit of 16MB (for the unfamilar, a document is to a row as collection is to a table), so we could very realistically hit that if we stored all tweets in an array.

Another issue is the frequency of access, and updates of the tweet data. If we embedded the data, we could hit the size limit and it would not make sense to keep retrieving extra information that would not be needed if we only wanted the tweets array. Also to update individual tweets we would have to save the whole array, for just one tweet!

Thus in our case I have chosen the last option, and created a separate mongodb collection to store the tweets individually as I have modeled them as a one to many, (1 walls – many tweets) relationship. Although it could also be seen as a many to many relationship between walls and tweets, to control the approval status for each requires us to treat each tweet as separate from each other as one could be approved but another rejected. So in the schema I have userWallId used for querying all tweets belonging to a certain wall of a certain user, notice the index:true option which helps speed up the query by indexing the userWallId field.

[code language=”javascript”]

var TweetSchema = new Schema({
userWallId: { type: String, required: true, index: true },
approval: Boolean,
timestamp: String,
created_at: String,
,…
})

[/code]

In contrast I have chosen to embed the wall options themselves into the user’s own collection, as this is a “one-few” relation, and we can denormalize it by embedding the wall options within an array in the user object. Denormalizing as such in Mongodb allows us to query it faster as we do not have to perform another lookup for the wall options if it were in another collection.

[code language=”javascript”]
var UserSchema = new Schema({
name: { type: String, required: true },
local: {..},
isVerified: { type: Boolean, required: true },
twitter: {..},
apps: {
wall: [{
profanity: Boolean,
images: Boolean,
//…other wall options
}]
}
})
[/code]
To store the tweets we will then have to create a route to handle post requests:

[code language=”javascript”]
router.post (‘/tweets/:userId/:wallId’, auth, function (req, res) {
req.body.forEach(function(tweet){
var newTweet = new Tweet(tweet);
newTweet.save(function(err,result){

});
})
res.jsonp({message: "inserted "+ req.body.length});
});
[/code]

And in the front end angular app we can use these routes to store our new tweets with $http.post:

[code language=”javascript”]

var posturl= ‘/api/tweets/’+ $rootScope.root.currentUser._id + ‘/’ + $scope.userWalls[$scope.userWalls.length – 1].id;

SearchService.initData(searchParams).then(function(data) {

// get tweets array from loklak_server and set all tweets to default false for approval status

data.statuses.map(function(tweet){
tweet.userWallId = userWallId;
tweet.approval = false;
})

$http.post(posturl, data.statuses)

[/code]

The end result is that we can do a get request at the moderation tab page for that wall’s tweets and display it as such:

Screenshot 2016-06-18 07.02.04.png

Stay tuned for more updates on this new feature of loklak walls!

Loklak Walls Manual Moderation – tweet storage

Protecting API routes in loklak web client using JWTs

This is a continuation from the previous post on how I used JWTs (JSON Web Tokens) for local authentication. Signing in locally is great but what we want is to make sure the right person has access to the right routes. This was done for the loklak user walls, as we have to check if the user is signed in and is authorized to see the current walls, and edit them.

Server side – protecting routes

To verify the identity of the logged in user we have to check for the JWT. To setup and use the route authentication we use express-jwt as middleware, which checks if the JWT is present before going to the route.

[code language=”javascript”]
var jwt = require(‘express-jwt’);
var config = require(‘../../custom_configFile.json’);

var auth = jwt({
secret: config.jwtsecret,
userProperty: ‘payload’
});

var ctrlAuth = require(‘../controllers/authentication’);
var ctrlMailer = require(‘../controllers/email’);
var ctrlWalls = require(‘../controllers/walls’);

// WALL API
router.get (‘/:user/:app/:id’, ctrlWalls.getWallById);
router.get (‘/:user/:app’, auth, ctrlWalls.getUserWalls);
router.post (‘/:user/:app’, auth, ctrlWalls.createWall);
router.put (‘/:user/:app/:id’, auth, ctrlWalls.updateWall);
router.delete(‘/:user/:app/:id’, auth, ctrlWalls.deleteWall);
[/code]

Notice that we do not use authentication for the wall route as we want anyone to be able to access the wall.

The route controllers then handle manipulating the mongoose user models. We check for the JWT payload in the getUserWalls controller but not the getWallById controller.

[code language=”javascript”]
module.exports.getWallById = function (req, res) {
User
.findById(req.params.user)
.exec(function(err, user) {
if (user.apps[req.params.app]) {
for (i = 0; i < user.apps[req.params.app].length; i = i + 1) {
if (user.apps[req.params.app][i].id === req.params.id) {
return res.jsonp(user.apps[req.params.app][i]);
}
}
res.jsonp({});
} else {
res.jsonp({});
}
});
}

module.exports.getUserWalls = function (req, res) {
// If no user ID exists in the JWT return a 401
if (!req.payload._id) {
res.status(401).json({
"message" : "UnauthorizedError: private wall page"
});
} else {
User
.findById(req.params.user)
.exec(function(err, user) {
if (user.apps && user.apps[req.params.app]) {
res.jsonp(user.apps[req.params.app]);
} else {
res.jsonp([]);
}
});
}
}
[/code]

One error I got stuck on was the “Mongoose: TypeError: doc.validate is not a function”, in the update and delete methods. Mutating the array with splice instead of assigning it a new object solves this.

[code language=”javascript”]

// appData[req.params.app][i] = req.body;
appData[req.params.app].splice(i, 1, req.body);

[/code]

Model Schema

I have chosen to embed the wall options into the User model schema itself instead of creating another collection, as it is one user-few walls, so there would be lesser queries to the API as there’s mostly reading wall options, and relatively less updating of wall options.

[code language=”javascript”]
var UserSchema = new Schema({
email: { type: String, unique: true, required: true },
name: { type: String, required: true },
hash: String,
salt: String,
isVerified: { type: Boolean, required: true },
apps: {
wall: [{
profanity: Boolean,

}]
}
});
[/code]

Client side – consuming the API.

We use $resource, a factory built on $http, to interact with the routes, through an angular service.

[code language=”javascript”]
function AppsService($q, $http, $resource, AppSettings, AuthService) {
return $resource(‘/api/:user/:app/:id’, {
user: ‘@user’,
app: ‘@app’,
id: ‘@id’
}, {
query: {
method: ‘GET’,
isArray: true
},
save: {
method: ‘POST’,
transformRequest: function(data) {
delete data.user;
delete data.app;
delete data.showLoading;
return JSON.stringify(data);
},
params: {
user: ‘@user’,
app: ‘@app’,
id: ‘@id’
}
}, …
[/code]

Then in our controller, WallCtrl, we use this service, for eg:

[code language=”javascript”]
var init = function() {
if ($scope.isLoggedIn) {
$scope.userWalls = AppsService.query({
user: $scope.currentUser._id,
app: ‘wall’
}, function(result) {
if ($scope.userWalls.length === 0) {
$scope.wallsPresent = false;
}
});
}
};
[/code]

To attach the JWT in the header we will get the JWT from local storage and attach it using an httpInterceptor through the client side routes which uses UI-router. If you do not do this you will get the UnauthorizedError, as no token is found.

[code language=”javascript”]
function tokenInjectorService($window) {
var tokenInjector = {
request: function(config) {
config.headers[‘Authorization’] = ‘Bearer ‘+ $window.localStorage[‘jwt-token’];
return config;
}
};
return tokenInjector;
}
[/code]

[code language=”javascript”]
function Routes($stateProvider, $locationProvider, $httpProvider) {
$locationProvider.html5Mode(true);

$stateProvider
.state(‘Home’, {
url: ‘/’,
controller: ‘MapCtrl as map’,
templateUrl: ‘home.html’,
title: ‘Home’
}) // and other routes

$httpProvider.interceptors.push(‘tokenInjector’);
}
[/code]

Hope that helps anyone who has trouble with their MEAN app and JWT auth.

 

Protecting API routes in loklak web client using JWTs

Authentication with JWT for loklak.net

In the loklak_webclient, the plan is to store the accounts locally on mongodb, instead of using loklak_server to store accounts. Here’s how I used JWTs (JSON Web Tokens) for basic user authentication.

Token based authentication is often compared to the traditional method of saving a session id in a cookie, and checking the user information in the session object for every request. Instead of saving user information on the server and checking it every request, a token, which allows access to the server, is given in exchange for valid credentials and is sent in every request. There are various advantages to this, including security and scaling.

As you can see from the mongoose user schema the hash and salt are stored instead of a password.

[code language=”javascript”]
var UserSchema = new Schema({
email: {
type: String,
unique: true,
required: true
},
name: {
type: String,
required: true
},
hash: String,
salt: String
});
[/code]

PassportJS is a popular middleware for handling authentication in Node.js, and since the web client already uses Express, this was a natural choice. Various authentication ‘strategies’ will also be used to connect other social login providers, like twitter, or weibo.

For now, since we are using email to store the username and hashed password, we will choose the passport-local package. The configuration is easy can be found on their website, and is almost the same, except that we use email as username, rather than just any string.

[code language=”javascript”]
passport.use(new LocalStrategy({
usernameField: ’email’
},
function(username, password, done) {
// … same as example config in passportjs.org
}
})
[/code]
Express.js then handles the register routes:

[code language=”javascript”]

router.post(‘/register’, function(req,res){
passport.authenticate(‘local’, function(err, user, info){
var token;
// If Passport throws/catches an error
if (err) {
res.status(404).json(err);
return;
}
// If a user is found, log in, ie. return a token
if(user){
token = user.generateJwt();
res.status(200);
res.json({
"token" : token
});
} else {
// If user is not found, register the user, then return a token
var user = new User();
user.name = req.body.email;
user.email = req.body.email;
user.setPassword(req.body.password);
user.save(function(err) {
var token;
token = user.generateJwt();
res.status(200);
res.json({
"token" : token
});
});
}
});
[/code]

So that’s it for the server side code, and similar logic can be applied to the login route. Now we move on to AngularJS, where we use a service to access the above route. As you can see below the JWT is stored in local storage when the user logs in.

[code language=”javascript”]
function AuthenticationService($http, $window) {

var saveToken = function (token) {
$window.localStorage[‘jwt-token’] = token;
};

var getToken = function () {
return $window.localStorage[‘jwt-token’];
};

var isLoggedIn = function() {
var token = getToken();
var payload;

if(token){
payload = token.split(‘.’)[1];
payload = $window.atob(payload); // .atob decodes the base64 String
payload = JSON.parse(payload);
return payload.exp > Date.now() / 1000;
} else {
return false;
}
};

var currentUser = function() {
if(isLoggedIn()){
var token = getToken();
var payload = token.split(‘.’)[1];
payload = $window.atob(payload);
payload = JSON.parse(payload);
return {
email : payload.email,
name : payload.name
};
}
};

var register = function(user) {
return $http.post(‘/api/register’, user).success(function(data){
saveToken(data.token);
});
};
[/code]

The last step is to use these services, which in turn uses our routes, in our controllers:

[code language=”javascript”]
$rootScope.root.onSubmit = function () {
AuthService
.register($rootScope.root.credentials)
.error(function(err){
alert(err);
})
.then(function(){
$location.path(‘/’);
$rootScope.root.isLoggedIn = AuthService.isLoggedIn();
$rootScope.root.currentUser = AuthService.currentUser();
});
};

$rootScope.root.onLogout = function () {
AuthService.logout();
$location.path(‘/’);
$rootScope.root.isLoggedIn = AuthService.isLoggedIn();
};
[/code]

And here’s what the form looks like:

3eM8YtQdYC

As you can see, there is still more work to be done to connect this with the rest of the app, but hopefully those reading can better understand how to implement local auth using JWTs on their SPA.

Authentication with JWT for loklak.net

Bubble & Bar charts for loklak

I have made 2 visualisations in the front-end apps section using D3.js and AngularJS, that anyone can use on loklak.org or locally.

The first type is a bubble chart, these are great for representing single values, such as the number of mentions or the frequency of certain words.

As you can see, when I search for the term loklak, one can see the number of times a user was mentioned and the relative frequency of words.Screenshot 2016-05-30 10.55.26

First, I had to solve the problem of getting the data. This can be done through the data service used for the tweet feed previously. This gives us an updated array of tweet objects in the local storage of the browser.

Next, I had to analyse the tweets. True to its name, D3.js handles a wide variety of data formats, so storing term-frequency pairs in an object is sufficient. Do take a look at the code if you are interested. The processed data is then stored in a separate array in local storage, for eg $storage.mentionFreq instead of $storage.tweets.

Next, I had to display the processed data. Thankfully, angularJS plays nicely with d3.js, so to create a directive in angular, I could use similar D3.js code in the $link function of the directive, as I was manipulating DOM elements with D3.js.

Finally, to make the chart update itself without a page refresh, I added a $watch function on the processed data in the local storage, which runs an update function. Within it, D3.js then uses array joins to render the new data with .data(), so it seamlessly updates itself. Do take a look at the code for the directives for a more in-depth explanation.

The beauty of directives is that they are components which are reusable, the above example is actually using the same directive but with different attributes passed in.

[code language=”css”]
<div ng-controller=’bubbleCloudCtrl’ >
<bubble-cloud flex data="$storage.mentionFreq" min ="1" title="Most Mentions"></bubble-cloud>
<bubble-cloud flex data="$storage.wordFreq" min="3" title="Word Frequency"></bubble-cloud>
</div>
[/code]

The same pattern can be applied to other D3.js charts, below we can see a stacked bar chart example using the general pattern as described above:

Screenshot 2016-05-30 10.55.19.png

The difference is in analaytics and directive code, which is a bit more involved, as I had to process by date, and then by quantity.

Hopefully this helps future developers to make their own, some ideas I have are making radar and force directed graph.

Bubble & Bar charts for loklak

A tweet feed for loklak.org

Loklak.org has a page for front-end apps that showcase the uses of loklak’s API, and in the first week I’ve tried to warm up for GSOC by making a ‘realtime’ twitter feed, as the existing front-end apps were mainly single http get requests and didn’t seem very appealing to me, and also to prepare for my front-end app data visualisations GSOC project.

To obtain updated data, the app polls the loklak API on a regular interval. This is done rather simply by using $interval in AngularJS, which is just a wrapper around window.setInterval(). For the realtime feed, I set the minimum rate for querying at an arbitrary 1min so as not to exceed the limits.

The next thing I needed to solve was where to store the data, as I didn’t have a database to work with, local storage was the only option. A useful module for this is “ngStorage“. At a maximum of 5MB it isn’t much, but decent enough to store a few thousand tweets and showcase the features of loklak. If developers wanted to create an external app for loklak, they could also use the front-end apps if they were well modularized.

Lastly, to display in a fancy pinterest-like grid, I’ve decided to use an angular module called “angulargrid“. To achieve the infinite scroll effect, I created another scope variable, to take a subset of the data in local storage. AngularJS then helps with rendering changes in the data model to the feed, by using $watch() on local storage.

Do checkout the tweetfeed:

tweetfeed

Some areas to improve:

  • Refactor services to make it more modular
  • Improve the regex to account for emojis and other tags
A tweet feed for loklak.org