Rolling out Freifunk Router data IOT to Loklak

Freifunk is a non-commercial initiative for free wireless networks. The vision of freifunk is distributing free networks, to democratize the communication infrastructure and promoting social structures locally. There are a lot of routers on the freifunk network that are available across the entire country of germany and a few other countries. Previously there was an IOT system to push data to loklak about each of the freifunk nodes that are available.

This time we’re stretching it a little further, each of the nodes collected are packaged into objects and can be given back to the user in the JSON format so that the user can use this information for visualizations or other tasks needed. This was done using the fetch servlet and each of the given data looks somewhat like this

"communities": {
    "aachen": {
      "name": "Freifunk Aachen",
      "url": "http://www.Freifunk-Aachen.de",
      "meta": "Freifunk Regio Aachen"
    }...,
}

"allTheRouters": [
    {
      "id": "60e327366bfe",
      "lat": "50.564485",
      "long": "6.359705",
      "name": "ffac-FeWo-Zum-Sonnenschein-2",
      "community": "aachen",
      "status": "online",
      "clients": 1
    }...,
}

The complete JSON dumps can be read by querying the Freifunk network and can be used to populate the data available in loklak push from the locations stored on the router network and then fetched to loklak. This information can be harvested every 24 hours to fetch updates of the entire entire and update the results accordingly.

Each of this data is available at the /api/freifunkfetch.json which is queried as follows

	private static String readAll(Reader rd) throws IOException {
		StringBuilder sb = new StringBuilder();
		int cp;
		while ((cp = rd.read()) != -1) {
			sb.append((char) cp);
		}
		return sb.toString();
	}

	public static JSONObject readJsonFromUrl(String url) throws IOException, JSONException {
		InputStream is = new URL(url).openStream();
		try {
			BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
			String jsonText = readAll(rd);
			JSONObject json = new JSONObject(jsonText);
			return json;
		} finally {
			is.close();
		}
	}

In this way, each of the freifunk node data is available on loklak server and being harvested, thus adding one more IOT Service to the loklak server and harvester.

Rolling out Freifunk Router data IOT to Loklak

Scraping and Feeding IOT data sets into Loklak – Part 2

As the integrations for the IOT services had begun, there are challenges especially with scraping multiple pages at once, such was the case with the NOAA Alerts and Weather information of the US Government. To scrape this information for the live updates that happen every 5 minutes it was necessary to simplify the process with which this process could be completed without running a complex web scraper work on them every time taking up precious cycles, What’s really interesting about the website is the way in which the data can be modeled into XML for any given page. Using this and leveraging the XML and other data conversion logic implemented previously for such task, I started digging deeper into the total working of the website and realized that appending &y=0 to the alerts URL resulted in XML generation, here’s an example of how this works
https://alerts.weather.gov/cap/wwaatmget.php?x=AKC013&y=0
and
https://alerts.weather.gov/cap/wwaatmget.php?x=AKC013

Screen Shot 2016-08-21 at 8.21.08 PM

Equivalent XML being
XML from Source NOAA

So extracting this has become quite a challenge because this poses two different challenges , one is how we can efficiently retrieve the information of the counties and how we can construct the alert urls. Perl to the rescue here !

sub process_statelist {
    my $html = `wget -O- -q https://alerts.weather.gov/`;
    $html =~ [email protected]*summary="Table [email protected]@s;
    $html =~ [email protected]*\s*@@s;
    $html =~ [email protected]\s*.*@@s;
    $html =~ [email protected]\s*@@s;
    %seen = ();

    while ( $html =~ [email protected]/(\w+?)\.php\?x=1">([^<]+)@sg ) {
        my $code = $1;
        my $name = $2;
        $name =~ s/'/\\'/g;
        $name =~ [email protected]\[email protected] @g;
        if (!exists($seen{$code})) {
            push @states_entries, $name;
            push @states_entryValues, $code;
        }
        $seen{$code} = 1;
    }
    open STATE, ">", "states.xml";
    print STATE <<EOF1;




    
EOF1
    foreach my $entry (@states_entries) {
        my $temp = $entry;
        $temp =~ s/'/\\'/g;
        $temp = escapeHTML($temp);
        print STATE "        $temp\n";
    }
    print STATE <<EOF2;
    
    
EOF2
    foreach my $entryValue (@states_entryValues) {
        my $temp = $entryValue;
        print STATE "        $temp\n";
    }
    print STATE <<EOF3;
    


EOF3
    close STATE;
    print "Wrote states.xml.\n";
}

Makes a request to the website and constructs the states list of all the states present in the USA. Now it’s time to construct it’s counties.

sub process_state {
    my $state = shift @_;
    if ( $state !~ /^[a-z]+$/ ) {
        print "Invalid state code: $state (skipped)\n";
        return;
    }

    my $html = `wget -O- -q https://alerts.weather.gov/cap/${state}.php?x=3`;

    my @entries     = ();
    my @entryValues = ();

    $html =~ [email protected].*@@s;
    while ( $html =~
[email protected]\s*?]+>\s*?]+>\s*?]+>\s*?\s*?]+>\s*?]+>([^<]+)\s*?\s*?]+>([^<]+)\s*?\s*@mg
      )
    {
        push @entries,     $2;
        push @entryValues, $1;
    }
    my $unittype = "Entire State";
    if ($state =~ /^mz/) {
        $unittype = "Entire Marine Zone";
    }
    if ($state eq "dc") {
        $unittype = "Entire District";
    }
    if (grep { $_ eq $state } qw(as gu mp um vi) ) {
        $unittype = "Entire Territory";
    }
    if ($state eq "us") {
        $unittype = "Entire Country";
    }
    if ($state eq "mzus") {
        $unittype = "All Marine Zones";
    }
    print COUNTIES <<EOF1;
    
        $unittype
EOF1
    foreach my $entry (@entries) {
        my $temp = $entry;
        $temp =~ s/'/\\'/g;
        $temp = escapeHTML($temp);
        print COUNTIES "        $temp\n";
    }
    print COUNTIES <<EOF2;
    
    
        https://alerts.weather.gov/cap/$state.php?x=0
EOF2
    foreach my $entryValue (@entryValues) {
        my $temp = $entryValue;
        $temp =~ s/'/\\'/g;
        $temp = escapeHTML($temp);
        print COUNTIES "        https://alerts.weather.gov/cap/wwaatmget.php?x=$temp&y=0\n";
    }
    print COUNTIES <<EOF3;
    
EOF3
    print "Processed counties from $state.\n";

}

There we go voila, we now have a perfect mapping in between every single county and the alert URL requirement for that particular county. The NOAA scraper and parser has been quite a challenge but provides the data in real-time from the loklak server. The information can be passed via the XML Parser written as a service at /api/xml2json.json and the developers can receive the information in their required format.

Scraping and Feeding IOT data sets into Loklak – Part 2

Scraping and Feeding IOT datasets into Loklak – Part 1

There’s a lot of open data that’s available online be it a government website providing opendata or different portals having a lot of information in various formats. Many data portals and IOT devices support XML, CSV and JSON type data queries and hence previously in Loklak the integration for type conversion has been made making it a very simple method call so that each source format can be converted into a destination format. This makes it really easy for other parts of the entire code to also reuse the components of the program. For example to convert from XML to JSONML, the requirement is to have a well structured XML document after which performing a method call like this below makes the conversion.

XML.toJSONObject(xmlDataString);

Since we now have all the required data conversion logic in place it was time to start scraping and fetching the information from various data sources that we had targeted and IOT devices. In the previous weeks, the support for tracking the GPS datasets from different GPS devices to render the location has been complete and this time we looked ahead and started with the Earthquake data sets available from the government. The data available here are classified based on duration and magnitude with the fixed values possible for each one of them as

duration : hour, day, week, month
magnitude: significant, 1.0, 2.5, 4.5

So different set of queries can be constructed which roughly translate as follows
1. Data of significant earthquakes in the last hour
2. Data of significant earthquakes in the last day
3. Data of significant earthquakes in the last week
4. Data of significant earthquakes in the last month
5. Data of earthquakes less than 1.0 richters in the last hour
6. Data of earthquakes less than 1.0 richters in the last day
7. Data of earthquakes less than 1.0 richters in the last week
8. Data of earthquakes less than 1.0 richters in the last month

Similarly other queries can also be constructed. All of this data is realtime and refreshes at 5 minute intervals enabling loklak server to harvest this data for use with Susi or providing this information to researchers and scientists / data visualization experts to perform their tasks.

Once this stream has been implemented, it was time to look at similar data structures and integrate them, most of the IOT devices sending out weather related information generally send out similar structure of information, So the next target was to integrate Yahi the haze index, these devices in singapore monitor the air quality. The data which looks like this

[
  {
    "hardwareid": "hardwareid",
    "centerLat": "1.3132409",
    "centerLong": "103.8878271"
  },
  {
    "hardwareid": "48ff73065067555017271587",
    "centerLat": "1.348852",
    "centerLong": "103.926314"
  },
  {
    "hardwareid": "53ff6f066667574829482467",
    "centerLat": "1.3734711",
    "centerLong": "103.9950669"
  },
  {
    "hardwareid": "53ff72065075535141071387",
    "centerLat": "1.3028249",
    "centerLong": "103.762174"
  },
  {
    "hardwareid": "55ff6a065075555332151787",
    "centerLat": "1.2982054",
    "centerLong": "103.8335754"
  },
  {
    "hardwareid": "55ff6b065075555351381887",
    "centerLat": "1.296721",
    "centerLong": "103.787217",
    "lastUpdate": "2015-10-14T16:00:25.550Z"
  },
  {
    "hardwareid": "55ff6b065075555340221787",
    "centerLat": "1.3444644",
    "centerLong": "103.7046901",
    "lastUpdate": "2016-05-19T16:43:03.704Z"
  },
  {
    "hardwareid": "53ff72065075535133531587",
    "centerLat": "1.324921",
    "centerLong": "103.838749",
    "lastUpdate": "2015-11-29T01:45:44.985Z"
  },
  {
    "hardwareid": "53ff72065075535122521387",
    "centerLat": "1.317937",
    "centerLong": "103.911654",
    "lastUpdate": "2015-12-04T09:23:48.912Z"
  },
  {
    "hardwareid": "53ff75065075535117181487",
    "centerLat": "1.372952",
    "centerLong": "103.856987",
    "lastUpdate": "2015-01-22T02:06:23.470Z"
  },
  {
    "hardwareid": "55ff71065075555323451487",
    "centerLat": "1.3132409",
    "fillColor": "green",
    "centerLong": "103.8878271",
    "lastUpdate": "2016-08-21T13:39:01.047Z"
  },
  {
    "hardwareid": "53ff7b065075535156261587",
    "centerLat": "1.289199",
    "fillColor": "blue",
    "centerLong": "103.848112",
    "lastUpdate": "2016-08-21T13:39:06.981Z"
  },
  {
    "hardwareid": "55ff6c065075555332381787",
    "centerLat": "1.2854769",
    "centerLong": "103.8481097",
    "lastUpdate": "2015-03-19T02:31:18.738Z"
  },
  {
    "hardwareid": "55ff70065075555333491887",
    "centerLat": "1.308429",
    "centerLong": "103.796707",
    "lastUpdate": "2015-03-31T00:48:49.772Z"
  },
  {
    "hardwareid": "55ff6d065075555312471787",
    "centerLat": "1.4399071",
    "centerLong": "103.8030919",
    "lastUpdate": "2015-11-15T04:04:41.907Z"
  },
  {
    "hardwareid": "53ff6a065075535139311587",
    "centerLat": "1.310398",
    "fillColor": "green",
    "centerLong": "103.862517",
    "lastUpdate": "2016-08-21T13:38:56.147Z"
  }
]

There’s more that’s happened with IOT and interesting scenarios that happened, I’ll detail this in the next follow up blog post on IOT.

Scraping and Feeding IOT datasets into Loklak – Part 1

Visualizing NMEA Datasets from GPS Tracking devices with Loklak

Loklak now supports the NMEA format and gives the developers access to the format in a much more friendlier JSON format which can be readily plugged in into the required map visualizer and usable on the web to create dashboards.

The stream URL is the data URL to which the GPS devices are streaming the data which need to be read by Loklak converted and then reused. For a given stream the response is as follows

{
"1": {
"lat": 0,
"long": 0,
"time": 0,
"Q": 0,
"dir": 0,
"alt": 0,
"vel": 0
},
"2": {
"lat": 0,
"long": 0,
"time": 0,
"Q": 0,
"dir": 0,
"alt": 0,
"vel": 0
},
"3": {
"lat": 0,
"long": 0,
"time": 0,
"Q": 0,
"dir": 0,
"alt": 0,
"vel": 0
},
"4": {
"lat": 0,
"long": 0,
"time": 0,
"Q": 0,
"dir": 0,
"alt": 0,
"vel": 0
},
"5": {
"lat": 0,
"long": 0,
"time": 0,
"Q": 0,
"dir": 0,
"alt": 0,
"vel": 0
},
"6": {
"lat": 0,
"long": 0,
"time": 0,
"Q": 0,
"dir": 0,
"alt": 0,
"vel": 0
}
}

We now need to visualize this information. Loklak now has a built in tool which can make this happen present here. You’ll see a screen which asks for the stream URL, Provide the URL where the GPS device is sending the information
NMEA App Loklak
This visualizes the information
Visualized points on Map
This is possible with the data format the NMEA.txt servlet is performing and the client side packaging of the objects while loading them onto a map.

function getTracking() {
var url = document.getElementById('url').value;
var cUrl = window.location.href;
var pName = window.location.pathname;
var baseUrl = cUrl.split(pName)[0]
var urlComplete = baseUrl+'/api/nmea.txt?stream='+url;
var centerlat = 52;
var centerlon = 0;
// set default zoom level
var zoomLevel = 2;
$.getJSON(urlComplete, function (data) {
var GPSObjects = [];
for (var key in data) {
var obj = data[key];
var latitudeObject, longitudeObject
for (var prop in obj) {
if (prop == 'lat') {
latitudeObject = obj[prop];
}
if (prop == 'long') {
longitudeObject = obj[prop];
}
}
var marker = L.marker([latitudeObject, longitudeObject]).addTo(map);
spiralCoords = connectTheDots(marker);
var spiralLine = L.polyline(spiralCoords).addTo(map)
}
});
}

A request is made to the required NMEA data stream and stored into the required latitude and longitude which is then each individually pushed onto the map.

Loklak is now slowly moving towards supporting multiple devices and visualizing the data that it obtains from the device data streams. The NMEA is a global standard for GPS and any tracking devices therefore loklak actively supports more than 10 devices Garmin 15, 15H etc.., from the Garmin series, Xexun 10X series and the Navbie series.

Visualizing NMEA Datasets from GPS Tracking devices with Loklak

Loklak now supports IOT streams from GPS Devices

GPS is one of the most widely used techniques for geolocation. There are a lot of commercial GPS tracking and reporting devices that are available in the market like the Xexun 10X series, Garmin GPS Devices and lots of other open source devices like GPS Cookie and GPS Sandwich. GPS is a satellite based navigation system which accurately positions a person. These tracking devices generally have 2 functionalities STORE and STREAM.

The STORE stores the NMEA GPS data onto the SD Card / storage device present on the device. This information can later be visualized or converted into a specific format as the NMEA.txt call does on the loklak server. The STREAM service takes the data and uses a GPRS mode of communication to send the NMEA Sentences to the designated server to which it has been configured. There has to be a node listening on the port so that the devices can stream the information to that port, or it can stream to a specific URL and Loklak can use the `stream=` in its GET request to make the required conversions into visualizer data so that the NMEA Sentences can be visualized.

A sample NMEA sentence looks as follows

/*
Sample Read Format
==================
  1      2    3        4 5         6 7     8     9      10    11
$xxxxx,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,003.1,W*6A
  |      |    |        | |         | |     |     |      |     |
  |      |    |        | |         | |     |     |      |     Variation sign
  |      |    |        | |         | |     |     |      Variation value
  |      |    |        | |         | |     |     Date DDMMYY
  |      |    |        | |         | |     COG
  |      |    |        | |         | SOG
  |      |    |        | |         Longitude Sign
  |      |    |        | Longitude Value
  |      |    |        Latitude Sign
  |      |    Latitude value
  |      Active or Void
  UTC HHMMSS
 */

Making a query on the given stream of such sentences on loklak returns a response as follows

{
		  "1": {
		    "lat": 60.073490142822266,
		    "long": 19.67548179626465,
		    "time": 94154,
		    "Q": 2,
		    "dir": 82.0999984741211,
		    "alt": 0.699999988079071,
		    "vel": 1.7999999523162842
		  },
		  "2": {
		    "lat": 60.07347106933594,
		    "long": 19.675262451171875,
		    "time": 94140,
		    "Q": 2,
		    "dir": 82.0999984741211,
		    "alt": 0.10000000149011612,
		    "vel": 1.7999999523162842
		  }
		}

Some devices give only a few variables in the sentence whereas some provide even information such as direction , heading and velocity. Hence there’s a need for handling multiple types of sentences in the NMEA Parsers. This is done by implementing an overloaded function parse() in the NMEAServlet which works as follows

class GPVTG implements SentenceParser {
		public boolean parse(String [] tokens, GPSPosition position) {
			position.dir = Float.parseFloat(tokens[3]);
			return true;
		}
	}

For a new NMEA Sentence format for example XXXXX We need to add a few lines to enhance the parser as follows by using the SentenceParser interfaces.

class XXXXX implements SentenceParser {
		public boolean parse(String [] tokens, GPSPosition position) {
			position.dir = Float.parseFloat(tokens[X]);
			return true;
		}
	}

This is a great milestone for loklak because it now supports IOT devices to stream to it and has the required conversion logic which converts the Sentence streams/logs into the requires JSON format so that it can be visualized. The NMEA Visualizer is coming soon where you can enter a stream URL or attach a log and it performs the required operations.

Cheers !

Loklak now supports IOT streams from GPS Devices

IOT Data push in different formats now supported

Lots of IOT devices, that are out there currently log a lot of information as CSV files. At the same time there’s a lot of data generated to data stream services from devices and satellites alike that are present in XML format. A lot of the structured storage data is exported by developers into formats like XML on most IOT devices and hence these dashboards contain options like export to XML/CSV/JSON formats.

The loklak server can store JSON formats but since the devices can send out any type of information, it becomes important for the server to support multiple data formats for both the streams as well as device pushes to the server. Over the last week we’ve had integrations to parse XML contents and convert these to JSON and publically expose them as an API endpoint available at /api/xml2json.json and data being the GET/POST parameter that needs to be sent out. This allows a lot of clients to be built around this which convert XML to JSON. We’ll soon have an app to show how this is possible and integrations into the LQL application which is an application meant for making loklak API easier for developers who are trying to use and access loklak data and the server.

Similarly the support for CSV to JSON formats is also now supported. A major problem that we hit here while development is the conversion schemes, the way special characters like carriage returns, new lines etc.., r, n etc.., are parsed and converted, Directly passing these in the parameter data string makes it hard to replace and takes up valuable string searching and processing. To overcome this, the client itself encodes the URI fields r as %0A%0D  and n as %0A accordingly and pass the data strings to the publically exposed /api/csv2json.json and data being it’s GET/POST parameters with CSV encoded information. Every CSV sent is converted and returned as a JSON which can be caught by the corresponding .success() equivalent functions of the programming language being used by the developer of the IOT device / service owner and retrigger a push response to the loklak server making this data harvesting as a completely synchronous and step by step process because of data format conversions.

This data API is publicly accessible which means it can potentially support a lot of applications that are currently trying out to use their own lexical parsers and tokenizers for making the data conversions.

For eg.

twittername, name, org
0rb1t3r, Michael Peter Christen, FOSSASIA
sudheesh001, Sudheesh Singanamalla, FOSSASIA

in the URL

http://localhost:9000/api/csv2json.json?data=twittername,name,org%0A0rb1t3r,Michael%20Peter%20Christen,FOSSASIA%0Asudheesh001,Sudheesh%20Singanamalla,FOSSASIA%0A

Resulting in the json format as follows

[
  {
    "org": "FOSSASIA",
    "name": "Michael Peter Christen",
    "twittername": "0rb1t3r"
  },
  {
    "org": "FOSSASIA",
    "name": "Sudheesh Singanamalla",
    "twittername": "sudheesh001"
  }
]

The CDL.java, XML.java files contain the required helper classes and methods needed for the required operations, this makes string conversions anywhere in loklak server if needed as simple as String jsonData = XML.toJSONObject(data).toString(); and JSONArray array = CDL.toJSONArray(json); for the corresponding XML and CSV conversions respectively.

The Tokeners, parse and pick out tokens matching

static {
       entity = new java.util.HashMap(8);
       entity.put("amp",  XML.AMP);
       entity.put("apos", XML.APOS);
       entity.put("gt",   XML.GT);
       entity.put("lt",   XML.LT);
       entity.put("quot", XML.QUOT);
   }

The XML parser looks for the starting angle brackets
< and ending angle brackets > and parses the content in between them finding out the contents in between the XML tags, Once this information is obtained as the key value, it looks for the format of >...</ and retrieves the value of the given XML tag. It gets pushed into a JSON object and forms a key:value pair. A collection of such objects are created for every complete XML structure given the DTD Schema.

These additions prove very useful to the IOT devices and services whose data is going to be integrated into the loklak server. The next task is to take up the Home automation system from IBM Watson cloud and make it parallely push the data to loklak now that we support so many data formats. We are very close to supporting real time applications streaming data to the loklak server.

IOT Data push in different formats now supported