Big Data Collection with Desktop version of loklak wok

A desktop version of loklak wok is now available. The goal of the wok is to enable users to collect and parse data from social services like twitter and enable users, citizen scientists and companies to analyze big data.

The origin of the project is a tweet by @Frank_gamefreak. Thank you!

Please join the development on GitHub: https://github.com/loklak/loklak_wok_desktop

desktop_wok_heap_up_display

How to compile and run

  • import required lib by running setup.sh
  • compile with mvn clean install -Pexecutable-jar
  • run artifact in target dircetory: java -jar wok-desktop-0.0.1-SNAPSHOT-jar-with-all-dependencies.jar
  • stop program with ESC key

To be done

  • The code has been hacked and butchered and is some kind of Frankenstein. It needs cleanup.
  • Font size is hardcoded. How ugly is that?
  • It would be cool to have a project for code shared between Android and Desktop version.
  • The only dependency which can not be resolved via Maven is loklakj. Wouldn’t it be cool to change that?
  • The used font does not seem to support Asian characters.
Big Data Collection with Desktop version of loklak wok

Elasticsearch node built into loklak

loklak has a built-in elasticsearch node, bit it can also connect as a transport client to a elasticsearch cluster. Here are some screenshots of elastic-hq of a 16-shard 8-disk 2-server 16-core loklak cluster.

elastic_plugin_hq_cluster_overview_screenshot

elastic_plugin_hq_node_diagnostic_graph_screenshot

elastic_plugin_head_cluster_screenshot

elastic_plugin_head_browser_screenshot

 

Elasticsearch node built into loklak

Android Twitter Search App with loklak

Everyone can create an app using the loklak_wok_android libraries. We now have an Android Tweet Search App, that fetches results using the TwitterScraper class.

Check out the code: https://github.com/loklak/LoklakAndroidApp

Requirements

  • Devices running Android-KitKat 4.4 or greater are supported.
  • Android Studio

Project setup

  • Download and setup Android Studio
  • Clone/ download this project. Cloning is recommended if you plan to contribute
  • Navigate to the directory where you saved this project and select the root folder ,and hit OK.
  • Wait for Android Studio to build the project with Gradle.
  • Once the build is complete, you can start playing around!
  • You can test it by running it on either a real device or an emulated one by going to Run>Run ‘app’ or presing the Run icon in the toolbar.

Working

Type a query and boom!

loklak_android_app

Watch out for the WiFi! This App only operates under WiFi

loklak_android_app2

Android Twitter Search App with loklak

Growing list of API libraries for loklak

We are very happy that the list of API libraries for loklak is constantly growing. Please check out the following project to create applications with loklak:

Growing list of API libraries for loklak

Data Collection and Parsing on Android with loklak wok

We now have a data parser to collect data, that you want to analyze. It is called loklak wok and runs on your android phone. The showcase collects tweet data for loklak.

Please check it out and test development!

Github: https://github.com/loklak/loklak_wok_android

android_wok_welcome_screenshot

Data Collection and Parsing on Android with loklak wok

Telegram Chatbot using loklak

It is now possible to retrieve single tweets within telegram using a telegram loklak bot. This is a kind of ‘first try’ to make an AI out of the tweet database.

Enjoy!

telegram_bot_screenshot

Telegram Chatbot using loklak

Tweet analytics with loklak and Kibana as a search front-end

You can use Kibana to analyze large amounts of Tweet data as a source for statistical data. Please find more info on http://loklak.org/download.html#kibana

Kibana is a tool to “explore and visualize your data”. It is not actually a search front-end but you can use it as such. Because Kibana is made for elasticsearch, it will instantly fit on loklak without any modification or configuration. Here is what you need to do:

kibana_screenshot

Kibana is pre-configured with default values to attach to an elasticsearch index containing logstash data. We will use a differnt index name than logstash: the loklak index names are ‘messages’ and ‘users’. When the Kibana Settings page is visible in your browser, do:

  • On the ‘Configure an index pattern’ Settings-page of the kibana interface, enter “messages” (without the quotes) in the field “Index name or pattern”.
  • As soon as you typed this in, another field “Time-field name” appears, with a red border and empty. Use the selectbox-arrows on the right side of the empty field to select one entry which is there: “created_at”.
  • Push the ‘Create’ button.

A page with the name “messages” appears and shows all index fields of the loklak messages index. If you want to search the index from Kibana, do:

  • Click on “Discover” in the upper menu bar.
  • You may probably see a page with the headline “No results found”. If your loklak index is not empty, this may be caused by a too tight time range; therefore the next step should solve that:
  • Click on the time picker in the top right corner of the window and select (i.e.) “This month”.
  • A ‘searching’ Message appears, followed with a search result page and a histogram at the top.
  • replace the wild-card symbol ‘*’ in the query input line with a word which you want to search, i.e. ‘fossasia’
  • You can also select a time period using a click-drag over the histogram to narrow the search result.
  • You can click on the field names on the left border to show a field facet. Click on the ‘+’-sign at the facet item to activate the facet.

The remote search to twitter with the twitter scraper is not done using the elasticsearch ‘river’ method to prevent that a user-frontend like Kibana constantly triggers a remote search. Therefore this search method with kibana will not help to enrich your search index with remote search results. This also means that you won’t see any results in Kibana until you searched with the /api/search.json api.

Tweet analytics with loklak and Kibana as a search front-end