Is your Big Data an Asset or Liability?

Balance sheets are funny things. In one stroke, they tell your positives and negatives. And it takes no time for assets to convert into liabilities also. The best examples of assets and liabilities can be taken from our daily life. If the kid turn out to be good (whatever it means to one), the kids are assets otherwise they become liability. A marriage or a partnership can become both asset or liability. 

Companies should also look into their data from a similar perspective. Data in itself has no meaning if you can't mint it. You can boast of having Terabytes and Petabytes of data, having clusters and farms and ranches  and villas and islands of servers to store them.  However if you are not able to make any money about of it, is it even worth the salt?

Cassandra Installation and Introduction

Cassandra is a no sql database and known for massive horizontal scaling with replication feature built in. You can install Cassandra from the instruction at http://cassandra.apache.org/download/ . Cassandra can be installed either fetching it from the repositories or a tar package can be downloaded from the above download link. I will proceed with downloading the tar package.

Steps to Install Cassandra:

• Download the tar file from one of the mirrors.   wget  <Cassandra Mirror location>
• Untar the file - tar -xzvf apache-cassandra-2.1.3-bin.tar.gz 
• Go to extracted directory of cassandra.
• Run Cassandra  sudo bin/cassandra -f  This will run cassandra as a forward process. Running it without -f switch will make to run as daemon.

Big and Fast Data - Lambda Architecture

In the previous post we looked into the architecture of Big Data framework. In this post, we will look into Lambda architecture. Lambda architecture is an architectural framework proposed by Nathan Marz to deal with the need of long term analytics of big data and at the same time processing the data for real time needs. More details can be seen at http://lambda-architecture.net/. The basic building blocks of Lambda architecture are:

Big and Fast Data - Introduction

Big data is the most popular term in software industry. There are companies build around this notion. Generally big data is characterized by three attributes:

• Volume - It's the amount of  data that one has to deal with. Big data landscape is full of stories of terabytes and petabytes of data and even more than that. The volume is only increasing with time. A quick search in the internet about number of messages Facebook handles in a second or number of search queries Google handles can give the idea of the size of data we are dealing with.

In Memory Databases

In memory databases are different from traditional databases in the sense that in memory database sits in memory and traditional databases stores the data in files system. As the data sits in in-memory, in memory databases are much faster than traditional databases. The primary reason for performance is that in memory databases do not have to seek the disk. 

In terms of ACID properties, in -memory databases satisfy all the requirements apart from durability. As the database exist in memory, in case of power failure, the database can loose the data. However their are many techniques which are employed to handle the situation. The involves from maintaining

Map Reduce in simple terms

The one concept that is quite popular today is Map Reduce. A perception is created that all the software problems can be solved by using Map Reduce. People have created cult around Map Reduce. Ok..ok..jokes apart, Map Reduce is an important concept to understand when you are dealing with large scale data processing. In this post we will try to look into Map Reduce in simple terms and try to unravel the mystery behind it.

Map-Reduce is made up of two words. Map and Reduce. Let's take each term and see what they mean. Map is a key value pair. Here in Map Reduce concept, Map word basically refers to making a Map. This is the first step in Map Reduce. The second word is Reduce, which mean to shrink. So what we do is we shrink the number of maps, so that for one key only one store exists. In the context of this discussion, let's look into an example.

Let's say we have 3 files and each file contains some set of words.

Hadoop Installation

The steps are tested on Linux Ubuntu 10 using user 'yarn'. You can use any other user for the same.

• Download Hadoop from http://www.apache.org/dyn/closer.cgi/hadoop/common/. This document is done with hadoop-2.3.0/ release.
• Unzip Hadoop in a directory. In this case I have created hadoop in the home directory and unzipped there.

           tar -xvzf hadoop-2.3.0.tar.gz

•  This will create a direcotry "/home/yarn/hadoop/hadoop-2.3.0"

Hadoop Introduction

Persistence is the ability for data to live beyond the life cycle of a process. Software as part of interaction generate a lot of data which needs to live beyond them. In the beginning, the data storage systems were file oriented. However files did little to provide a good structure to the stored data. The other main issue with files was there proprietary format, which made it impossible to read them in the absence of certain programs. Also processing of data in file across versions of software was another challenge to deal with, as run time structures to hold the data do change across releases. Also in terms of scalability, files were not very good in terms of searching and doing local updates on data. To solve this, industry came up with RDBMS type of system.

RDBMS is a very structured way of defining the data. The rigorous structuring and ability to

SQL vs No SQL

With the advent of many no-sql databases, this has become a matter of debate for many organizations that which data persistence strategy to follow. In the past, in relational world the debate used to be which database to use. The relational world landscape is pretty matured with good choices in both commercial and open source world. With the advent of SQL, for many applications changing the database along the way also remained an option. Frameworks like Hibernate helped in bringing neutrality to that choice, though there are still features of databases which makes migration hard.

In the past, computing technology basically belonged to corporates and academics, who were more